\section{Newton's method}

\subsection{Implementation}

We are going to use the class algorithm:

\includegraphics[width=\linewidth]{3_hessian_formula.png}

\subsubsection{Hessian matrix}

For function $f(x,y) = (x+2)^2 + 2(y-2)^2 + 2 \sin (2 \pi x) \sin (2 \pi y)$

\begin{equation*}
  \frac{\partial }{\partial x} f = 2 (x + 2) + 2 \sin (2 \pi y) \cos ( 2 \pi x) 2 \pi =  2 (x + 2) +  4 \pi \sin (2 \pi y) \cos ( 2 \pi x)
\end{equation*}

\begin{equation*}
  \frac{\partial }{\partial y} f = 4 (y - 2) +  4 \pi \sin (2 \pi x) \cos ( 2 \pi y)
\end{equation*}

\begin{equation*}
  \frac{\partial ^2 }{\partial y \partial x} f =  \frac{\partial ^2 }{\partial x \partial y } f =  8 \pi^2 \cos (2 \pi x) \cos ( 2 \pi y)
\end{equation*}

\begin{equation*}
  \frac{\partial^2 }{\partial y^2} f = 4  -  8 \pi^2 \sin (2 \pi x) \sin ( 2 \pi y)
\end{equation*}

\begin{equation*}
  \frac{\partial^2 }{\partial x^2} f = 2  -  8 \pi^2 \sin (2 \pi x) \sin ( 2 \pi y)
\end{equation*}

\subsubsection{Implementation}

One implementation (really similar to the \texttt{gradient\_descent\_trace}

\begin{minted}{python}
def newton_trace(initial_point, fun, grad_fun, hessian, eta, max_iter):
    """ Newton method
    INPUT
    - initial_point:
    - f: differential function
    - grad_fun: Gradient
    - hessian: hessian
    - eta: learning rate
    - max_iter: number of iterations

    OUTPUT
    w trace
    """

    w = initial_point
    w_list = [initial_point]
    iterations = 0

    while iterations < max_iter:
        w = w - eta *np.linalg.inv(hessian(w[0],w[1])).dot(grad_fun(w[0], w[1]))
        w_list.append(w)
        iterations += 1

    return np.array(w_list)

 \end{minted}

 The results after run are

 \begin{verbatim}
With eta = 0.01,
coordenates (x,y)= (-0.9793498427941949, 0.9893767407575305),
the number of iterations: 50 and the image is f(x,y) = 3.0671859515488302

With eta = 0.1,
coordenates (x,y)=(-0.9463274028190676, 0.9717082373563009),
the number of iterations: 50 and the image is f(x,y) = 3.1079767229661335
\end{verbatim}

 \includegraphics[width=\linewidth]{3_1_eta_01.png}
 \includegraphics[width=\linewidth]{3_1_eta_001.png}
 \includegraphics[width=\linewidth]{3_1_comparatives_etas.png}

 As we see this method does not minimize the function independently of the learning rate and even worse than the initial point. This is because this algorithm is sensible about where the gradient is zero.

 So in this case the algorithm get trapped in a saddle point.

 However if the function is convex it would be more precise even though the selection of the learning rating was bigger.

\subsubsection{Different initial points }

\begin{center}
  \begin{tabular}{ |c|c|c| }
    \hline
    Initial point  & Final coordinates & Final value  \\
    \hline

    (-0.5 -0.5) & (-0.9463274,  0.97170824)  &  3.10798 \\
    (1, 1)    & [1.067, 0.911] &  11.345 \\
( 2.1, -2.1)   &  ( 3.261, -3.118)  & 78.711 \\
    (-3 , 3)  &  (-3.0536726 ,  3.02829176) &   3.108\\
(-2 , 2)  &  (-2,  2) &  0 \\

 \hline
\end{tabular}
\end{center}

If we compare with the gradient descendant's algorithm

\begin{center}
  \begin{tabular}{ |c|c|c| }
    \hline
    Initial point  & Final coordinates & Final value  \\
    \hline

    (-0.5 -0.5) &  (-0.793 -0.126) &   9.125 \\
(1 1) &  (0.677 1.29) &   6.437 \\
( 2.1 -2.1) &  ( 0.149 -0.096 ) &   12.491 \\
(-3  3) &  (-2.7315  2.713) &  -0.381 \\
(-2  2) &  (-2.  2.) &  0 \\

 \hline
\end{tabular}
\end{center}

So apart form the first one and the last point (which is a solution) the Newton's method is worse.

Let's see what is happening by plotting their traces.

\includegraphics[width=\linewidth]{3_2_comparation_1.png}
\includegraphics[width=\linewidth]{3_2_comparation_2.png}

Analysing the results we have observed that the points get easily trapped in a point, this may be because this algorithm is sensitive to the hessian.

If we compute these points' gradients and hessian we see that:

\begin{verbatim}

For (-0.38067878, -0.52778221)
	The gradient value is [ 1.64133153 -1.67813744]
	The inverse hessian is [[-0.00432244  0.01843361]
 [ 0.01843361 -0.00367461]]

For (1.06677195, 0.91078249)
	The gradient value is [ 0.03180694 -0.02149455]
	The inverse hessian is [[-0.00634209  0.01835718]
 [ 0.01835718 -0.00574093]]

For (3.26077803, -3.11750721)
	The gradient value is [ 11.09387996 -11.19723696]
	The inverse hessian is [[0.0182662  0.00126589]
 [0.00126589 0.0176255 ]]

For (-2.0, 2.0)
	The gradient value is [-6.15574622e-15  6.15574622e-15]
	The inverse hessian is [[-0.00064245  0.01268142]
 [ 0.01268142 -0.00032122]]

\end{verbatim}

As we thought the hessians of all these points are \textit{close} the null matrix.

\subsubsection{Conclusions}

This algorithm minimizes correctly the gradient but not the function. Therefore, if a function have saddle points this algorithm would not be as useful as the gradient descendent. Moreover, other problem is that the function should be twice differentiable.


\section{Supervised learning on image classification}\label{sec:MLbasics}
We consider a basic machine learning problem for classifying a
collection of images into $\kappa$ distinctive classes.  As an
example, we consider a two-dimensional image which is usually
represented by a tensor
$$
f\in  {\mathcal D} := \mathbb{R}^{c \times m\times n}.
$$
Here
\begin{equation}
\label{data-c}
c=
\left\{
\begin{array}[rl]{rl}
1 & \mbox{for grayscale image},\\
3 & \mbox{for color image}.
\end{array}
\right.
\end{equation}

A typical supervised machine learning problem begins with a data set (training data)
$$
D := \{(f_i, y_i)\}_{i=1}^N,
$$
with
$$
\{f_i\}_{i=1}^N \subset {\mathcal D},
$$
and $y_i \in \mathbb{R}^{\kappa}$ is the label for data $f_i$, with
$[y_i]_j$ as the probability for $f_i$ in classes $j$.

Roughly speaking, a supervised learning problem can be thought as a data fitting
problem in a high dimensional space ${\mathcal D}$.
Namely, we need to find a mapping
$H:  \mathbb R^{c\times m\times n}\mapsto \mathbb R^\kappa,$
such that, for a given $f\in  {\mathcal D}$,
\begin{equation}\label{eq:idealouput}
H(f)\approx e_i\in \mathbb R^\kappa,
\end{equation}
if $f$ is in class $i$, for $1\le i\le \kappa$.
For the general setting above, we use a probatilistic model for understanding the
output $H(f) \in \mathbb{R}^{\kappa}$ as a discrete
distribution on $\{1, \cdots,\kappa\}$, with $[H (f)]_i$ as the probability
for $f$ in the class $i$, namely
\begin{equation}
\label{distrib}
0 \le [H(f)]_i \le 1,\quad
\sum_{i=1}^\kappa  [H(f)]_i=1.
\end{equation}
At last, we finish our model with a simple strategy to choose
\begin{equation}\label{eq:maxchoose}
\mathop{\arg\max}_{i}\{[H(f)]_i~:~ i = 1:\kappa\},
\end{equation}
as the label for a test data $f$, which ideally is close to
\eqref{eq:idealouput}.  The remaining key issue is the construction of
the classification mapping $H$.

%\subsection{Model construction}
%In many CNN type models for image classification, there are usually
%three major steps to construct $H$.
The main step in the construction of $H$ is to
construct a nonlinear mapping
\begin{equation}
\label{linearize}
H_0: {\mathcal D} \mapsto V_J,
\end{equation}
with
\begin{equation}
\label{VJ}
V_J = \mathbb R^{c_J \times m_J\times n_J}.
\end{equation}
To be consistent with the notation for CNN which will be described below,
here the subscript $J$ refers to the number of
coarsening girds in CNN.
Roughly speaking, the map $H_0$ plays two roles.  The first role
is to conduct a dimensionality reduction, namely
$$
c_Jm_Jn_J\ll  mnc.
$$
The second role is to map a complicated set of data into a set of data
that are linearly separable. As a result, the simple logistic regression
procedure can be applied.

The first step in a logistic regression is to introduce a linear mapping:
\begin{equation*}
\Theta: {\mathcal D} \to\mathbb{R}^{\kappa} ,
\end{equation*}as
\begin{equation}\label{thetamap}
\Theta(x)=Wx+b,
\end{equation}
where $W=(w_{ij})\in\mathbb{R}^{(c_J\times m_J \times n_J)\times \kappa}$,
$b\in\mathbb{R}^{\kappa}$.

We then use the soft-max function.
\begin{equation}
\label{softmax}
[S(z)]_i=[{\rm Solftmax}(z)]_i= \frac{e^{z_i}}{\sum_{j} e^{z_j}},
\end{equation}
to obtain a logistic regression model
\begin{equation}
\label{eq:log_reg}
S \circ \Theta: \mathbb R^{c_J \times m_J\times n_J}\mapsto \mathbb R^\kappa.
\end{equation}

By combining the nonlinear mapping $H$ in \eqref{linearize}
and the logistic regression \eqref{eq:log_reg}, we obtain the following classifier:
\begin{equation}
\label{classifier}
H=  S\circ \Theta\circ H_0.
\end{equation}

%\subsection{Loss function and training}
Given the model \eqref{classifier},
we finish the training phase with solving the next optimization
problem:
\begin{equation}
\label{eq:3}
\min \sum_{j=1}^Nl(y_j, H(f_j)),
\end{equation}
where
Here $l(y_j, H(f_j))$ is a  loss function that measures the
predicted result $H(f_j)$ and the real label $y_j$.
In logistic regression,
the following cross-entropy loss function is often used
$$
l(y_j, H(f)) = \sum_{i=1}^\kappa -[y]_i \log [H(f)]_i.
$$


Artificial neural networks (ANN) were inspired by the idea of developing programs that are able to learn how to perform certain tasks by getting examples. Neural networks generally don't have any task-specific rules implemented, they learn the important features by themselves.

The basic idea was to model the neurons in a biological brain. Such artificial neurons (also called connected units or nodes) are linked to each other and transmit signals. The neuron receives the signal and combines it with a certain weight and bias before sending it to the next connected unit. The weight and the bias are free parameters which have to be optimized by the network itself.
In ANNs, nodes are combined into layers. The input data is processed by several layers connected to each other. Each layer is typically trained to extract a certain feature of the input data to yield the required information at the end.

Neural networks are mostly used for two different kinds of analysis. The first one is the prediction of a continuous variable, which is called a regression problem. The second, which is specified here, is the problem of identifying an input as part of a class, called a classification model.
In classification problems the training is typically supervised. The network is given input data and corresponding label vectors which state to which category the respective input belongs. False labelling is penalized and leads to weight and bias adjustments.

Training a network means giving examples of the task and evaluating the performance of the network. This evaluating is done through backpropagation. The input propagates through the network until it reaches the output layer. There it is compared to the actual output vector. The difference between these two or the classification error is given by a loss function. This error is then propagated backwards through the network until every neuron has an associated error value. Through this value the weight and bias of the corresponding neuron is then changed to improve the performance of the network.

As updating the weights after every input sample is inefficient and makes the loss function very noisy, several samples are combined in one batch or minibatch. The training values are averaged over the sample and the network learns more general features.

Typically the available input data is limited and can not be easily generated. The network therefore uses the same data several times during the training. One training session which uses the entire data set exactly once is called an epoch. To fully train a ANN several hundred epochs are needed. To prevent the network only being able to classify the training data correctly a small part of it is being split off. This data set is called the validation data. After every epoch the network evaluates its progress on the validation data which has not been used during training. This secures an independent evaluation.

In contrast to the free parameters set by the network, hyperparameters have to be specified before training starts. Different kinds of networks have different hyperparameters, one for example being the size of the minibatch. Optimized hyperparameters contribute significantly to the success of the network. \cite{Bishop2016}

\section{Image recognition}
Image recognition has always played a huge part in the development of neural networks. Current state-of-the-art networks in image recognition have now reached human or even superhuman performance. Image recognition with neural networks is normally placed within the \enquote{deep learning} category. Such architectures include successive layers with nonlinear processing units. Each layer is connected to its predecessor and uses the output of the latter as input. This enables the network to learn different features, higher layers then correspond to higher levels of abstraction.

 The problem with conventional multilayer perceptrons are the computing power needed. In conventional ANNs the neurons are fully connected, meaning that every node in one layer is connected to every node in the layer before and after. The number of weights per node is equivalent to the product of width, height and colours of the picture. They therefore scale very badly to higher resolution images. Fully connected layers also do not take localized characteristics into account. In image recognition neighbouring pixels are more important than those far apart.
Convolutional neural networks (CNNs) can solve both problems and are currently the best performing networks \cite{imagenet}.

\section{Convolutional neural networks}

\begin{figure}
\centering
\includegraphics[scale=1.5]{convolve.png}
\caption{A convolutional layer with its kernel. The elements of the kernel are also called a filter. The kernel moves across the whole input data on the left. The right matrix is the product of input and kernel. \cite{convwindow}}
\label{convwindow}
\end{figure}
A convolutional neural network contains several layers in which one input sample is processed in small receptive fields (as seen in fig. \ref{convwindow}). These receptive fields are most often called kernel or filter. One neuron in the layer corresponds to only this part of the input image in width and height but gets the whole depth of the input volume. The dot product between the entries of the kernel and the input is computed and sent to the neuron. The filter values or weights are free parameters of the network that have to be optimized. For different sets of parameters different features can be detected.
The map of neurons is called the activation map. As several input pixels are passed on to one neuron, the size of the input data gets gradually smaller while it is being processed by the network.

Convolutional neural networks are dominated by three different features which make them ideal for image recognition. The first is the 3D volume of neurons, where they are classified in three dimensions, width, height and depth of colour. This ideally represents a picture. The second is the application of spatial locality. One neuron of a following layer is only connected to a small part of neurons in its predecessor. Therefore the network learns to recognize local patterns. With many such layers stacked the patterns learned get increasingly more global, the first one only recognizing lines, while the last one identifies complete features such as \enquote{cat}, \enquote{dog} or \enquote{mouse}.
At last a convolutional neural network takes translational invariance into account. To classify an image correctly, the position of the object in the image is not important. Therefore all neurons in one convolutional layer share the same weights and bias, that means, the same filter is applied while forwarding the signal. Thus one layer always recognizes the same feature, regardless of its position in the image \cite{lecunimage}.

The hyperparameters for this kind of ANN, which have to be manually tuned, are firstly the size of the convolutional window. If a size of one in width and height is chosen, the window only takes one pixel into account. A bigger size means more information is being observed. The bigger the window the less localized the features detected are. Smaller windows mean more computing power needed, and may not be able to detect larger objects, especially in high resolution pictures. At times it may be advantageous to choose a non-symmetrical kernel size.

The second hyperparameter is the depth of the output volume. It is possible to choose how often a kernel with a new set of weights will be used on the input data or further on in the network. As one kernel with a set of weights is also called a filter, the number of filters can be manually tuned. The more filters are being used, the more free parameters the network has. This can be beneficial as more details can be learned by the network but also disadvantageous as overfitting, which is explained later, is more likely to occur.

If the kernel size is chosen, the stride must be finetuned as well. If the stride is one, then the window only moves one pixel in one direction each time. A high overlap of kernel windows occurs which also produces a large output volume. This can significantly increase the computing time. If the kernel size is large, than a higher overlap is helpful.

When the size of the input volume is not a multiple of the kernel size window, then the output dimensions differ from those of the input. Most of the time it is helpful to pad the input volume with zeros at the edges to preserve the spatial size of the input while going through layers. It is also beneficial to minimize edge effects. \cite{lecun-89e}

\section{Overfitting}
Overfitting is a potential problem of deep-learning networks. By choosing layers with a high number of learnable weights and biases, the number of free parameters is very high compared to the actual number of features learnable. Therefore a network is prone to overfit the training data, performing very well during the training but failing while validating on other data (fig. \ref{accuracy}).

\begin{figure}
\centering
\includegraphics[scale=0.35]{overfittingacc.png}
\caption{Training and validation accuracy. The validation accuracy follows the training accuracy well if there is little or no overfitting. In the case of strong overfitting the validation accuracy increases very little or even decreases during the training.}
\label{accuracy}
\end{figure}
One method to address the problem of overfitting is \enquote{max pooling}. Max pooling also reduces the spatial size, thus limiting the amount of computational power needed. The general idea is that only the rough location in comparison to others of the feature detected is needed, so the input is downsampled with the help of a non-linear function. The working principle is the same as in a convolutional layer. A kernel window moves over the input without overlapping. In every region the activation numbers of the included pixels are compared. The maximum is determined and only this pixel is forwarded to the next layer. For example, in the case of a 2x2 kernel window, the spatial size is reduced by 75 \%, while the depth dimension stays the same. As the network cannot fit parameters to very detailed features, it has to pick up general features, which help correct classification.

Another method to reduce overfitting is dropout. With dropout a input unit is set to zero before being forwarded to the next layer with a possibility $p$. This is the same as to set every weight of one neuron to zero. This is done with after every minibatch of samples, which is tantamount to every parameter update of the network. This forces the CNN to learn more robust features to better generalize the data analysis. Dropout also cuts down on computational time.

\section{Activation functions}
Deep neural networks need two different kinds of activation functions. The first one is to propagate the signal through the network. After the product between the input signal and the weight of one neuron is computed, the outcome is put into the activation function. Only if a nonlinear activation function is used, non trivial problems can be solved with only few nodes. The activation function is applied after every layer, meaning that every output signal is put into relation with the help of the function. The current best working activation function is the ReLu function. ReLu stands for rectified linear unit. It is calculated by
\begin{equation}
f(x) = \mathrm{max} (0,x)
\end{equation}
with x as the input signal. Negative signals are therefore always set to zero while propagating through the network. Other activation functions are for example the sigmoid function or the tangens hyperbolicus. Unlike the ReLu function, those functions saturate when dealing with very high input signals and therefore perform worse.

The second activation function is needed at the output layer, the last layer in the network. Most deep neural networks are multi-class networks. Multi-class means that the output is not simply signal or background but identifies the input as belonging to one class of several. That means that the output activation function calculates the probability of belonging in each class in correlation to every other probability. A usual activation function in this case is the softmax function. The softmax function calculates a n-dimensional vector of random real values to a n-dimensional vector, where every number lies between 0 and 1 and the sum of all entries is 1. It is calculated by
\begin{equation}
\sigma (\vec{z})_{j} = \frac{\mathrm{e}^{z_j}}{\sum_{k=1}^{n} \mathrm{e}^{z_k} } \quad .
\end{equation}
The vector $\vec{z}$ is the output vector of the neural network, which corresponds after the application of the activation function to $\sigma (\vec{z})_{j}$, the probability of the input belonging to the corresponding classes.
The softmax function cannot be applied if the network is not only multi-class, but also multi-label. The categories in a multi-label network are not mutually exlusive. One input can therefore be classified into several different categories, their probability to fit is not correlated to each other. In this case the beforementioned sigmoid activation function can be used. This function also squashes a vector of real numbers to a vector of real numbers between 0 and 1. In this case the sum of all numbers does not have to be 1. The output $\sigma (\vec{z})_{j}$ is the independent probability of the input data belonging to class j without taking the other classes into account. The function looks like
\begin{equation}
\sigma (\vec{z})_{j} = \frac{1}{1 + \mathrm{e}^{z_j}}
\end{equation}
with $\mathrm{e}^{z_j}$ as the output vector of the network.
Assigning a class to the input data is normally done by selecting those entries which have a number higher than 0.5, meaning a probability higher than 50\% of belonging to that class.
\section{Loss functions}
Loss functions are a way to track the progress while training a neural network. A loss function assigns a real number to the difference between real and estimated class labels of the input, which should be minimized. During training the network changes the weights of the neurons according to the increase or decrease of the loss function. For a specific problem it is important to use a suitable loss function. For classification problems such as image recognition with more than two classes, either categorical or binary crossentropy can be used. In machine learning cross entropy is the same as logistic loss. It is calculated by
\begin{equation}
f(x) = y_{\mathrm{true}} \cdot \mathrm{ln} (y_{\mathrm{pred}}) + (1 - y_{\mathrm{true}}) \cdot \mathrm{ln} (1 - y_{\mathrm{pred}})
\end{equation}
where $y_{\mathrm{true}}$ is the true label vector of the input and $y_{\mathrm{pred}}$ the label vector of the prediction of the network.

The difference between categorical and binary crossentropy is only the way in which it is applied. With categorical crossentropy, the whole vector is compared. If only one entry differs, it is considered falsely labeled. This is useful in multi-class problems, where the labels are mutually exclusive. With multi-label problems, binary crossentropy works better. Binary crossentropy compares every label separately, therefore not penalizing one incorrect label so much.

\section{Optimization algorithms}
To minimize the loss function the neural network has to update its weights and biases during training. This updating is done through an optimization algorithm. To find a minimum of the loss function it is possible to evaluate the gradient of the function. The updating of the network is then proportional to the negative of the gradient. This is an iterative algorithm which finds the steepest descent to a local minimum. The size of the steps taken in the direction of the descent is also called the learning rate. The learning rate is a hyperparameter which has to be optimized to get best training results.

To compute the gradient for the whole training data takes very long. To shorten the time and computing power needed stochastic gradient descent (SDG) is implemented. SDG uses only one stochastically chosen example to compute the gradient and update the parameters. This can of course result in very varying parameter updates. To prevent this the gradient is computed not over one sample, but over a whole batch. This works because the samples in the dataset are correlated. They all depict the same or similar things, so an update computed for one batch is a good approximation for the complete set. It also can be done much more often, therefore achieving faster convergence for the loss function.

Most optimization algorithms also use momentum. Momentum can be understood by making an analogy to classical physics. If a ball at the top of a hill rolls down, it gains momentum if the direction downhill always stays the same. The hill here represents our loss function. If the gradient of a step points in the same \enquote{direction} as the step before, the \enquote{speed} or step size increases. If the direction changes, the step size gets smaller. If a minimum is found and the imaginary ball oversteps it, the gradient points in the other direction and momentum decreases. The minimum can so be found iteratively.

To further prevent the network from overstepping the minimum which is still a possibility even with momentum, the Nesterov Accelerated Gradient can be implemented. Instead of evaluating the gradient, changing the loss function and then applying momentum, the network anticipates what is going the happen. At first the momentum is applied to the current position. This is an approximation of the future position of the function. At this position the gradient is computed. The network there \enquote{sees} where its going to end up in the next step and can therefore move in the right direction (see fig. \ref{nesterov}). This minimizes the possibility of overstepping and decreases the convergence time. The strength of momentum is also a hyperparameter which needs to be tuned.

\begin{figure}
\centering
\resizebox{!}{5cm}{
\begin{tikzpicture}
%% Linke Hälfte

%Pfeile
\draw [-triangle 60, very thick, red](0,0) -- (5,0);
\draw [-triangle 60, very thick, gruen](0,0) -- (2,4.5);
\draw [-triangle 60, dashed, red, opacity= 0.7] (2,4.5) -- (7,4.5);
\draw[-triangle 60, very thick, blue] (0,0) -- (7,4.5);

%Kugel
\draw[ball color=red] (0,0) node (v1) {} circle (.2);
%\fill[red]  (0,0) ellipse (0.5 and 0.5); Alternative Kugel, mehr ein Kreis

%% Texte
\node[red] at (2.5,-0.5) { \large{gradient step}};
\node[gruen] at (0,3.5) {\large{momentum}};
\node[gruen] at (-0.6,3) {\large{step}};
\node[blue] at (5,2) {\large{actual step}};
\node at (3,6) {\LARGE{Momentum update}};

% Trennlinie
\draw [gray, very thick](9,6.5) -- (9,-1);

%% Rechte Hälfte

%Pfeile
\draw [-triangle 60, gruen, very thick](11,0) node (v2) {} -- (13,4.5) node (v3) {};
\draw [blue, -triangle 60, very thick](11,0) -- (17,3) node (v4) {};
\draw [red, -triangle 60, very thick](13,4.5) -- (17,3);

%Kugel
\draw[ball color=red] (11,0) circle (.2);

%%Texte
\node at (13.5,6) {\LARGE{Nesterov momentum update}};
\node[gruen] at (10.5,3.5) {\large{momentum}};
\node[gruen] at (9.9,3) {\large{step}};
\node[blue] at (15,1) {\large{actual step}};
\node[red] at (15.5,5) {\large{'lookahead'}};
\node[red] at (15.5,4.5) {\large{gradient step}};

\end{tikzpicture}
}
\caption{Classical momentum compared to Nesterov momentum. The network anticipates its future position through momentum and calculates the gradient based on that. It ends up on a slightly different position, thus converging faster. \cite{nesterov}}
\label{nesterov}
\end{figure}

During the training it is helpful to decrease the learning rate over time. As the loss gets smaller, the gradient also decreases. Too high learning rates can start to behave chaotically and the network cannot settle into the minimum of the loss function. To manually decrease the learning rate the common types of decay are step decay, which reduces the learning rate by a factor every few epochs, exponential decay or 1/t decay, where t is the number of epochs.

Such decays act globally and are difficult to tune. Setting the decay rate too high means severely slowing down the network, setting it too low risks erratic jumps of the loss function. Several more complicated algorithms have been developed to be used for optimizing functions. One of the current best working algorithms is the Adam algorithm (\cite{adam}). Adam is derived from adaptive moment estimation. The learning rate is adjusted for each parameter separately and based on first and second moments of the gradient. It is bounded by a set step size and naturally annealed during the training. Adam performs well on sparse and noisy gradients and needs very little memory, as only first order gradients have to be calculated.


\section{The Diffusion Equation}
\subsection{Physical Origin}
We want to understand physical phenomena that ``diffuses'' due to spatial gradient.
An early example was Fick's Law $\underline{J}=-D\nabla c$ where $J$ is the flux, $c$ is the concentration and $D$ is the diffusion coefficient.
For heat flow, we also have Fourier's Law saying $\underline{q}=-k\nabla\Theta$ where $\underline{q}$ is the heat flux, $k$ is the thermal conductivity and $\Theta$ is the temperature.\\
In a volume $V$, the overall heat energy $Q$ is
$$Q=\int_V c_V\rho\Theta\,\mathrm dV$$
where $c_V$ is the specific heat capacity of the matertial of the volume $V$, and $\rho_V$ is the mass density.
Te rate of change of it would then be, making use of Fourier's law,
$$\frac{\mathrm dQ}{\mathrm dt}=\int_Vc_V\rho\frac{\partial\Theta}{\partial t}\,\mathrm dV$$
On the other hand, integrating Fick's Law over $S=\partial V$,
$$-\frac{\mathrm dQ}{\mathrm dt}=\int_S\underline{q}\cdot\underline{\hat{n}}\,\mathrm dS=\int_S(-k\nabla\Theta)\cdot\underline{\hat{n}}\,\mathrm dS=\int_V(-k\nabla^2\Theta)\,\mathrm dV$$
by Fourier's Law.
Therefore,
$$\int_Vc_V\rho\frac{\partial\Theta}{\partial t}-k\nabla^2\Theta\,\mathrm dV=0$$
for all volumn $V$.
So the integrand must vanish everywhere (assuming it is continuous), which gives
$$\frac{\partial\Theta}{\partial t}-D\nabla^2\Theta=0$$
where $D=k/(c_V\rho)$.
This is known as the diffusion equation.\\
We can also derive this from a more fundamental point of view.
Consider gas particles diffuse by scattering.
So for every small time change $\Delta t$ in time a particle moves for a distance $\xi$ with probability (PDF) $p(\xi)$.
Let $\langle X\rangle$ be the mean of $X$.
We assume $\langle\xi\rangle=0$.
Suppose the PDF after $N\Delta t$ steps is $P_{N\Delta t}(x)$, then for $(N+1)\Delta t$ step,
\begin{align*}
    P_{(N+1)\Delta t}(x)&=\int_{-\infty}^\infty p(\xi)P_{N\Delta t}(x-\xi)\,\mathrm d\xi\\
    &\approx \int_{-\infty}^\infty p(\xi)\left(P_{N\Delta t}(x)-\xi P_{N\Delta t}^\prime(x)+\frac{\xi^2}{2}P_{N\Delta t}^{\prime\prime}(x)\right)\,\mathrm d\xi\\
    &=P_{N\Delta t}(x)+P_{N\Delta t}^{\prime\prime}(x)\frac{\langle\xi^2\rangle}{2}
\end{align*}
Identify $P_{N\Delta t}(x)=P(x,N\Delta t)$, then we have
$$P(x,(N+1)\Delta t)-P(x,N\Delta t)=\frac{\partial^2P}{\partial x^2}(x,N\Delta t)\frac{\langle\xi^2\rangle}{2}$$
By some probabilistic argument, $\langle\xi^2\rangle\propto\Delta t$, so this gives
$$\frac{\partial P}{\partial t}=D\frac{\partial^2P}{\partial x^2}$$
for a constant $D$.
\subsection{Similarity Solution}
The characteristic relation between variance and time suggests that we may start seeking a solution in terms of the dimensionless parameter $\eta=x/(2\sqrt{Dt})$.
That is, we want to find solutions of the form $\Theta(x,t)=\Theta(\eta)$.
Change variables in this way,
$$\frac{\partial\Theta}{\partial t}=\frac{\partial\eta}{\partial t}\frac{\mathrm d\Theta}{\mathrm d\eta}=-\frac{1}{2}\frac{x}{\sqrt{D}t^{3/2}}\Theta^\prime=-\frac{\eta}{2t}\Theta^\prime$$
$$D\frac{\partial^2\Theta}{\partial x^2}=D\frac{\partial}{\partial x}\left(\frac{\partial\eta}{\partial x}\frac{\mathrm d\Theta}{\mathrm d\eta}\right)=D\frac{\partial}{\partial x}\left( \frac{1}{2\sqrt{Dt}}\Theta^\prime \right)=\frac{D}{4Dt}\Theta^{\prime\prime}=\frac{1}{4t}\Theta^{\prime\prime}$$
Putting them all together gives $\Theta^{\prime\prime}=-2\eta\Theta^\prime$, which gives $\Theta^\prime\propto e^{-\eta^2}$, therefore
$$\Theta(\eta)=\Theta(0)+\frac{2C}{\sqrt{\pi}}\int_0^\eta e^{-u^2}\,\mathrm du=\Theta(0)+C\operatorname{erf}(\eta)=\Theta(0)+C\operatorname{erf}\left( \frac{x}{2\sqrt{Dt}} \right)$$
where $C$ is a constant and $\operatorname{erf}$ is the error function defined by
$$\operatorname{erf}(z)=\frac{2}{\sqrt{\pi}}\int_0^ze^{-u^2}\,\mathrm du$$
This can describes discontinuous initial conditions (e.g. step functions) that spreads over time.
\subsection{Heat Conduction in a Finite Bar}
Suppose we have a bar of length $2L$ at $[-L,L]$ and initial temperature
$$\Theta(x,0)=H(x)=\begin{cases}
    1\text{, for $x\in [0,L]$}\\
    0\text{, for $x\in [-L,0)$}
\end{cases}$$
with boundary conditions $\Theta(L,t)=1,\Theta(-L,t)=0$.
We want to use Sturm-Liouville theory, but there is a problem here:
Our boundary conditions is not homogeneous.
So we must make the condition homogeneous by a suitable superposition.
But there is an obvious choice of this, namely $\Theta_s(x,t)=(x+L)/(2L)$.
Therefore we with a transformation $\hat\Theta=\Theta-\Theta_s$ the problem becomes
$$\frac{\partial\hat\Theta}{\partial t}=D\frac{\partial^2\hat\Theta}{\partial x^2},\hat\Theta(-L,t)=\hat\Theta(L,t)=0,\hat\Theta(x,0)=H(x)-\frac{x+L}{2L}$$
Now we do seperation of variables $\hat\Theta(x,t)=X(x)T(t)$, which gives $X^{\prime\prime}=-\lambda X,\dot{T}=-D\lambda T$ where $\lambda$ is the seperation constant.
The boundary conditions imply that $\lambda>0$ and
$$X(x)=A\cos{\sqrt{\lambda}x}+B\sin(\sqrt{\lambda}x)$$
where $A,B$ are constants.
The initial condition is odd, so $A=0$ and consequently the eigenvalues are $\lambda_n=(n\pi/L)^2$ for $n=1,2,3,\ldots$, so we obtained the family of solutions
$$X_n=B_n\sin\frac{n\pi x}{L},\lambda_n=\frac{n^2\pi^2}{L^2},n=1,2,3,\ldots$$
Put $\lambda_n$ in the temporal equation gives
$$T_n(t)=C_n\exp\left( -\frac{Dn^2\pi^2}{L^2}t \right)$$
So
$$\hat\Theta(x,t)=\sum_{n=1}^\infty b_n\sin\left(\frac{n\pi x}{L}\right)\exp\left( -\frac{Dn^2\pi^2}{L^2}t \right)$$
Now we impose initial conditions at $t=0$ which gives
\begin{align*}
    b_n&=\frac{1}{L}\int_{-L}^L\hat\phi(x,0)\sin\frac{n\pi x}{L}\,\mathrm dx\\
    &=\frac{2}{L}\int_0^L\hat\phi(x,0)\sin\frac{n\pi x}{L}\,\mathrm dx\\
    &=\frac{2}{L}\int_0^L\left( H(x)-\frac{x+L}{2L} \right)\sin\frac{n\pi x}{L}\,\mathrm dx\\
    &=\frac{2}{L}\int_0^L\left( H(x)-\frac{1}{2} \right)\sin\frac{n\pi x}{L}\,\mathrm dx-\frac{2}{L}\int_0^L\frac{x}{2L}\sin\frac{n\pi x}{L}\,\mathrm dx\\
    &=\begin{cases}
        2/[(2m-1)\pi]-1/(n\pi)\text{, for $n=2m-1$ odd}\\
        1/n\pi\text{, for $n$ even}
    \end{cases}\\
    &=\frac{1}{n\pi}
\end{align*}
Therefore we get the final solution
$$\Theta(x,t)=\frac{2+L}{2L}+\hat\Theta(x,t)=\frac{x+L}{2L}+\sum_{n=1}^\infty \frac{1}{n\pi}\sin\left(\frac{n\pi x}{L}\right)\exp\left( -\frac{Dn^2\pi^2}{L^2}t \right)$$
A plot reveals that this solution is very similar to the similarity solution we got earlier especially for small $t$.


\section{Hamilton's Principle of Least Action}
3 equivalent formulations
\begin{itemize}
    \item Newton's laws
    \item Lagrange's equation
    \item Hamiton's principle
\end{itemize}
Let us generalize to non-Cartesian coordinates
\begin{align}
    \left( x, y, z \right) \to \left( q_1, q_2, q_3 \right)
\end{align}
For example, $(q_1, q_2, q_3)$
could be
$(r, \theta, \phi)$
or
$(\rho, \phi, z)$.

Necessary that each position $\vec{r}$
uniquely specifies $(q_1, q_2, q_3)$
\begin{align}
    q_i &= q_i(x, y, z)\\
    \vec{r} &= \vec{r}\left( q_1, q_2, q_3 \right).
\end{align}
We can rewrite $x,y,z$
and also
$\dot{x},\dot{y},\dot{z}$
in terms of the
$q_1,q_2,q_3$
and
$\dot{q}_1,\dot{q}_2,\dot{q}_3$.
Then
\begin{align}
    L &=
    \frac{1}{2}m\dot{\vec{r}}^2 - U(\vec{r})
    = L(q_i, \dot{q}_i)
\end{align}
Let's write the action integral
\begin{align}
    S &=
    \int_{t_1}^{t_2} dt\,
    L\left( q_i, \dot{q}_i \right).
\end{align}
Let's draw the picture again.

The value of the action integral for every single coordinate system,
so the action is minimized for all paths.
It's going to be the same path in the extremum value.
And so that means $L$ will satisfy an Euler-Lagrange equation in these
coordinates.

The value of $L$ at any point and therefore the integral $S$ for any path is
unaltered by the change of variables.
Thus it must be true that $S$ is stationary at the correct path even in the new
coordinates system.
This implicitly implies that the Euler-Lagrange equations
\begin{align}
    \frac{d}{dt}\left( \frac{\partial L}{\partial \dot{q}_i} \right)
    - \frac{\partial L}{\partial q_i} &= 0
\end{align}
holds for any set of generalized coordaintes.

\section{Generalized Force and Momentum}
force a particle in 1D,
\begin{align}
    L &= \frac{1}{2} m\doc{x}^2 - U(x)
\end{align}
and
\begin{align}
    \frac{d}{dt}\underbrace{%
        \left(
            \frac{\partial L}{\partial \dot{x}}
        \right)
    }_{m\dot{x} = p}
    =
    \underbrace{%
        \frac{\partial L}{\partial x}
    }_{%
        -\frac{\partial U}{\partial x}
    }
\end{align}
So that gives
\begin{align}
    \frac{dp}{dt} &= -\frac{\partial U}{\partial x}.
\end{align}
So in general.
\begin{align}
    \text{generalized momentum} &:= \frac{\partial L}{\partial \dot{q}_i}\\
    \text{generalized force} &:= \frac{\partial L}{\partial q_i}.
\end{align}
The Euler-Lagrange equation says that
the rate of change of generalized momentum equals the generalized force.

\begin{example}[Single particle in polar coordinates]
    The kinetic energy is
    \begin{align}
        T &= \frac{1}{2}mv^2 =
        \frac{1}{2} m\left( \dot{r}^2 + r^2 \dot{\phi}^2 \right)
    \end{align}
    and the Lagrangian is
    \begin{align}
        L &= T - U\\
        &=
        \frac{1}{2}m\left( \dot{r}^2 + r^2\dot{\phi}^2 \right)
        - U(r, \phi)
    \end{align}
\end{example}
Then the Euler-Lagrange equation is
\begin{align}
    \frac{d}{dt}
    \underbrace{%
        \left(
            \frac{\partial L}{\partial\dot{r}}
        \right)
    }_{%
        m\dot{r}
    }
    =
    \underbrace{%
        \frac{\partial L}{\partial r}
    }_{%
        -\frac{\partial U}{\partial r} + mr\dot{\phi}^2
    }
\end{align}
which leads to the radial equation of motion.
\begin{align}
    m\underbrace{\ddot{r}}_{\text{radial acceleration}}
    - \underbrace{m r\dot{\phi}^2}_{\text{centripetal force}}
    &=
    \underbrace{- \frac{\partial U}{\partial r}}_{%
        \text{radial component of force}
    }
\end{align}
Then the angular equation of motion is
\begin{align}
    \frac{d}{dt}\underbrace{%
        \left(
            \frac{\partial L}{\partial\dot{\phi}}
        \right)
    }_{mr^2\dot{\phi}}
    &=
    \underbrace{\frac{\partial L}{\partial \phi}}_{
        -\frac{\partial U}{\partial \phi}
    }
\end{align}
so you get
\begin{align}
    \frac{d}{dt}\underbrace{\left( mr^2\dot{\phi} \right)}_{\text{angular
    momentum}}
    &=
    \underbrace{- \frac{\partial U}{\partial \phi}}_{r F_{\phi}=\text{torque}}
\end{align}
Then the force is
\begin{align}
    \vec{F} = - \vec{\nabla} U
    = -\frac{\partial U}{\partial r} \hat{e}_r
    - \frac{1}{r} \frac{\partial U}{\partial \phi} \hat{e}_\phi
\end{align}

\section{Constrained system}
Consider a pendulum of length $l$.
The position is given by $(x,y)$
but there is a constraint
\begin{align}
    \sqrt{x^2 + y^2} &= l.
\end{align}
So you only need one degree of freedom.

The system only has one degree of freedom because location of bob is specified
once we know the value of $x$.
Alternatively,
location is specified by the value of generalized coordinate $\theta$.

The coordinate transform is
\begin{align}
    x &= l \sin\theta\\
    y &= l \cos\theta.
\end{align}

The kinetic energy is
\begin{align}
    T &= \frac{1}{2}m \left( \dot{x}^2 + \dot{y}^2 \right)\\
    &= \frac{1}{2}m l^2 \dot{\theta}^2
\end{align}
and the potential energy
is
\begin{align}
    U &= mgl(1 - \cos\theta)
\end{align}
Then the Lagrangian is
\begin{align}
    L &= T - U\\
    &= \frac{1}{2} ml^2 \dot{\theta}^2
    - mgl(1 - \cos\theta)
\end{align}
The Euler-Lagrange equation is
\begin{align}
    \frac{d}{dt}\left(\frac{\partial L}{\partial \dot{\theta}} \right)
    -
    \frac{\partial L}{\partial \theta} = 0
\end{align}
which gives the equation of motion
\begin{align}
    ml^2\ddot{\theta} + mgl\sin\theta &= 0\\
    \ddot{\theta} + \frac{g}{l}\sin\theta &= 0.
\end{align}

\section{Generalized Coordinates for a Constrained System}
We say that the parameters
$q_1,q_2,\ldots,q_n$ are generalized coodrinates for the system of
$N$ particles
$\alpha = 1,2,\ldots,N$
with psoitions $\vec{r_}_{\alpha}$
if
\begin{enumerate}
    \item Each position $\vec{r}_{\alpha}$ can be expanded as a function of
        $\left( q_1,q_2,\ldots,q_n \right)$
        and (possibly) $t$.
        \begin{align}
            \vec{r}_{\alpha} &=
            \left( q_1, q_2, \ldots, q_n, t \right)
        \end{align}
        for $\alpha = 1,2,\ldots,N$.
    \item Each $q_i$ can e expressed in terms of the $\vec{r}_{\alpha}$
        and (possibly) $t$.
        \begin{align}
            q_i &= q_i\left( \vec{r}_1, \vec{r}_2,\ldots,\vec{r}_N, t \right)
        \end{align}
    \item The number of generazed coordinates is the smallest number that allows
        the system to be parameterized in this way
        \begin{align}
            n &= 3N & \text{for unconstrained system}\\
            n &< 3N & \text{for constrained system}
        \end{align}
\end{enumerate}
\begin{example}[Double pendulum]
    There is a pivot point to which a rigid bar of length $l_1$ is free to
    swing,
    at the end of which is a mass $m_1$.
    It makes an anble $\phi_1$ with the veritical.
    Another rigid bar is attached to where $m_2$ is,
    but is also free to swing,
    of length $l_2$ which makes an anble $\phi_2$ with the vertical,
    at the end of which is a mass $m_2$.
\end{example}
The position of the first mass is
\begin{align}
    \vec{r}_1 &= \vec{r}_1(\phi_1)\\
    \vec{r}_2 &= \vec{r}_2(\phi_1, \phi_2)
\end{align}


\section{Composition of transformations}

%-------------------------------------------------------------------------------------------
\subsection{Order of composition}
\label{subsec:tools:composition:order}

Let us consider two images $I_F$ and $I_M$ (resp. fixed and moving images). We want to register $I_M$ to $I_F$ and a common way to do it is to apply successively different registration methods. In this example, we use three registration methods (e.g. rigid, affine, and non-linear). First, we apply the first registration method between images $I_F$ and $I_M$. We obtain a transformation (form the fixed to the moving image) $T_1$ and an image $I_1$ ($I_M$ re-sampled in the geometry of $I_F$):

\begin{equation}
\textrm{method}_1(I_F,I_M) \rightarrow \{T_1,I_1\}
\end{equation}

Then we apply the second registration method between images $I_F$ and $I_1$. By considering the image $I_1$ instead of $I_M$, we initialize the registration process by starting the algorithm \textit{closer} to the optimal transformation. We obtain a transformation $T_2$ and an image $I_2$ ($I_1$ re-sampled in the geometry of $I_F$):

\begin{equation}
\textrm{method}_2(I_F,I_1) \rightarrow \{T_2,I_2\}
\end{equation}

Finally, we apply the third registration method between images $I_F$ and $I_2$. We obtain a transformation $T_3$ and an image $I_3$ ($I_2$ re-sampled in the geometry of $I_F$):

\begin{equation}
\textrm{method}_3(I_F,I_2) \rightarrow \{T_3,I_3\}
\end{equation}

If we assume that the successive registration processes improved the image registration, the image $I_3$ should be the image the most similar to $I_F$, then $I_2$, $I_1$, and finally $I_M$. An important issue here is that $I_3$ is the result of three consecutive re-samplings of the initial moving image $I_M$. In order to avoid the multiple approximations due to these re-samplings, one may want to re-sample only once $I_M$ using the global transformation from $I_F$ to $I_M$. However, this global transformation is not $T_3$ but the composition of $T_1$, $T_2$, and $T_3$. Indeed, $T_3$ is the transformation from $I_F$ to $I_2$ ; the image $I_M$ re-sampled using $T_3$ is probably not correctly registered onto $I_F$.
\\

Let us consider the image re-sampling of $I_M$ using $T_1$. Let $\mathbf{x}=(x,y,z)$ be a point (in the real world coordinates) in the real grid of the fixed image. The re-sampling is defined as:

\begin{equation}
I_1(\mathbf{x}) = I_M(T_1(\mathbf{x}))
\end{equation}

Now, let us now consider the image re-sampling of $I_1$ using $T_2$:

\begin{align}
I_2(\mathbf{x}) & = I_1(T_2(\mathbf{x})) \\
                & = I_M(T_1(T_2(\mathbf{x}))
\end{align}

Finally, we re-sample $I_2$ using $T_3$:

\begin{align}
I_3(\mathbf{x}) & = I_2(T_3(\mathbf{x})) \\
                & = I_1(T_2(T_3(\mathbf{x})) \\
                & = I_M(T_1(T_2(T_3(\mathbf{x}))) \\
                & \triangleq I_M( T_1 \circ T_2 \circ T_3(\mathbf{x}) ) \label{eq:composition}
\end{align}

Eq.~\ref{eq:composition} shows that the global transformation allowing to re-sample the image $I_M$ onto $I_F$ using a single global transformation $T$ which is the composition of $T_1$, $T_2$, and $T_3$:

\begin{equation}
T = T_1 \circ T_2 \circ T_3
\end{equation}

%-------------------------------------------------------------------------------------------
\subsection{Composing transformations using \texttt{rpiFuseTransformation}}

The \texttt{rpiFuseTransformation} executable is a program that allow to fuse (compose) a list of transformation into a unique transformation. Since it is not possible yet -- at least in a general case -- to store a list of transformations into a single file, we propose to store the list as a XML file. This XML lists the type of each transformation and the path to the files containing the transformations :
%
\begin{lstlisting}
<?xml version="1.0" encoding="UTF-8"?>

<listoftransformations>

    <transformation>
        <type>linear</type>
        <path>t0.txt</path>
        <invert>1</invert>
    </transformation>

    <transformation>
        <type>displacementfield</type>
        <path>t1.nii.gz</path>
        <invert>0</invert>
    </transformation>

</listoftransformations>
\end{lstlisting}
%

In this example, we have 2 transformations. The first one (denoted $T_0$) is a linear transformation stored into the text file ``t0.txt''. The second transformation (denoted $T_1$) is a displacement field stored into the -- image -- file ``t1.nii.gz''. The computed transformation is given by $T_0 \circ T_1$. Thus, the order the transformations written into the XML file is important since the composition of two transformations is usually not commutative (see section~\ref{subsec:tools:composition:order}). The XML file can contain any strictely positive number of transformations. The meaning of the different tags is:
%
\begin{itemize}
%
\item Tag \texttt{<listoftransformations>} contains a set of transformations. This tag is mandatory.
%
\item Tag \texttt{<transformation>} contains a transformation. This tag must be an element of the tag \texttt{<listoftransformations>}. This tag is mandatory.
%
\item Tag \texttt{<type>} contains the type of the considered transformation. The available types are ``linear'', ``displacemendfield'', and ``stiaionaryvelocityfield''. This tag must be an element of the tag \texttt{<transformation>}. This tag is mandatory.
%
\item Tag \texttt{<path>} contains the path to the file containing the transformation. A displacement field or a stationary velocity field must be stored into an image file (e.g. Nifty, Analyse, etc.). This tag must be an element of the tag \texttt{<transformation>}. This tag is mandatory.
%
\item Tag \texttt{<invert>} indicates if the considered transformation has to be inverted before the fusion. The value ``1'' induces the transformation inversion ; the value ``0'' means no inversion. This tag is optional. By default, no inversion is performed.
%
\end{itemize}
%
The type of the output transformation depends on the transformations of the list. A list of linear transformations will be fused into a linear transformation. If the list contains at least one displacement field, the output transformation will be a displacement field. If the option "--force-displacement-field" is used, the ouput transformation will be a displacement field.
For now on, if the list contains linear transformations and stationary velocity fields, the computed transformation will be a displacement field. However, this will change in the near future in order to compute a unique stationary velocity field (thanks to Marco Lorenzi).
\\
One should choose the output file extension according to the transformation computed. Indeed, if the output transformation is a displacement field, only image format are supported (e.g. .nii). No checking on the file extension will be performed since the output transformation type is easily predictable.


\chapter{Probability Distributions \label{chapter:probabilitydistributions}}

Many of the methods we will examine in these workshops depend on basic concepts from probability theory. For example, linear and logistic regression are members of a class of supervised learning algorithms called \textbf{generalized linear models} (see Chapter~\ref{chapter:glms}) which make assumptions about the type of probability distribution followed by the outcome variable. Decision trees use a concept called \textbf{entropy} (see Chapter~\ref{chapter:decisiontrees}), whose mathematical formulation depends on the probability distribution underlying the outcome. Many \textbf{hypothesis tests} (see Chapter~\ref{chapter:hypothesistesting}) likewise rely on probabilistic assumptions about the data. Probability is everywhere.

The following sections review some key probability concepts -- in an extremely hand-wavey and non-rigorous way -- and the properties of some of the most common probability distributions you will encounter in machine learning and statistics.

\section{Definitions}

A \textbf{probability distribution} is just a mathematical function that provides the relative likelihoods of various possible outcomes of an observation. We call the quantity that is being observed a \textbf{random variable}. Probability distributions can be discrete or continuous. The random variable involved can be a number, a vector of numbers, a category/class, etc. The \textbf{sample space} is the set of all possible outcomes. The integral (or sum) of the probability distribution over the entire sample space is $1.0$. You will often hear probability distributions for continuous random variables referred to as \textbf{probability densities}.

Probability distributions are grouped into families that are characterized by their overall shapes. These families contain \textbf{parameters} that, when varied, produce different distributions. Specific probability distributions from within a single family can often look quite different.

We use the notation $E[x|\theta]$ to refer to the \textbf{expected value}, or mean, of a distribution, given its parameter(s), $\theta$. There can be more than one parameter, and it will not always be called $\theta$; this is just an example. We use the notation $\text{var}(x|\theta)$ to refer to the \textbf{variance}, or spread, of a distribution around its mean.

\section{Normal Distribution \label{sect:normal}}

Also called the \textbf{Gaussian distribution}, the normal distribution is probably the most well-known continuous probability distribution. It has the following properties:
\begin{equation*} p(x | \mu, \sigma) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2 \sigma^2}} \qquad  E[x| \mu, \sigma] = \mu \qquad \text{var}(x | \mu, \sigma) = \sigma^2 \end{equation*}
where $x \in \mathbb{R}$. We will abbreviate the normal distribution as $\mathcal{N}(\mu, \sigma)$.  The value of $\mu$ changes the position of the center of the normal distribution.
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure1-normal-mean-change.png}
\end{center}
The value of $\sigma$ changes the width of the normal distribution.
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure2-normal-sd-change.png}
\end{center}

\begin{question}{question:normalex}
List 5 random variables from medicine or biology that should follow normal distributions.
\end{question}

\section{Bernoulli Distribution \label{sect:bernoulli}}

The \textbf{Bernoulli distribution} is a discrete probability distribution with the following properties:
$$ p(x|\mu) = \mu^x (1 - \mu) ^ {1-x} \qquad E[x| \mu] = \mu \qquad \text{var}(x | \mu) = \mu (1 - \mu) $$
where $x \in \{0, 1\}$. It is used to model events where the outcome is yes/no. Think of it as a weighted coin, with $\mu$ the probability that the coin comes up ``heads'' on a single toss. Here are three Bernoulli distributions with (from left to right) $\mu = 1.0, 0.7, 0.2$. The number along the bottom is $x$, which can only be $0$ or $1$.
\begin{center}
\includegraphics[width=0.3\textwidth]{img/l01-figure2a-bernoulli-1-0.png}
\includegraphics[width=0.3\textwidth]{img/l01-figure2b-bernoulli-0-7.png}
\includegraphics[width=0.3\textwidth]{img/l01-figure2c-bernoulli-0-2.png}
\end{center}

The \textbf{categorical distribution} is a generalization of the Bernoulli distribution to an outcome with more than two levels. The categorical distribution looks like this:
$$ p(x|\phi_1, \dots, \phi_K) = \phi_1^{\mathbb{I}(x=1)} \phi_2^{\mathbb{I}(x=2)} \cdots \phi_K^{\mathbb{I}(x=K)} $$
where $\sum_{k=1}^K \phi_k = 1$. The term $\mathbb{I}(x=j)$ is an \textbf{indicator}. It equals 1 if $x=j$ and 0 otherwise. For example, $\mathbb{I}(x=2)$ is 1 if $x=2$ and 0 otherwise.

\begin{question}{}
List 5 random variables from medicine or biology that should follow Bernoulli distributions.
\end{question}

\section{Binomial Distribution \label{sect:binomial}}

The \textbf{binomial distribution} models the number of positive outcomes, $x$, out of $n$ independent\footnote{The word \textbf{independent} just means that the outcome of one trial does not influence the outcome of any other trial.} Bernoulli trials, each of which is positive with probability $\mu$. This distribution has the following properties, with $x \in \{0, \dots, n\}$:
$$ p(x|n,\mu) = {n\choose x} \mu^x (1 - \mu) ^ {n-x} \qquad E[x| \mu] = n \mu \qquad \text{var}(x | \mu) = n \mu (1 - \mu) $$
where the notation ${n \choose x}$ is defined as:
$$ {n \choose x} = \frac{n!}{x!(n-x)!}. $$
This notation denotes the number of ways it is possible to choose $x$ things out of a group of $n$ things, where the ordering doesn't matter. The exclamation point denotes the \textbf{factorial function}: $x! = x(x-1)(x-2)\cdots(2)(1)$.

The shape of the binomial distribution is governed by the values of $n$ and $\mu$. Here, we vary $n$ but keep $\mu$ constant at $0.5$:
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure3-binom-n-change.png}
\end{center}
And here we vary $\mu$ but keep $n$ constant at $50$:
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure4-binom-p-change.png}
\end{center}

\begin{question}{question:binomialex}
List 5 random variables from medicine or biology that should follow binomial distributions.
\end{question}

\section{Poisson Distribution \label{sect:poisson}}

The \textbf{Poisson distribution} is a probability distribution that is often used to model discrete quantitative data, such as counts. It has the following properties:
$$ p(x | \lambda) = \frac{e^{-\lambda} \lambda^x}{x!} \qquad E[x|\lambda] = \lambda \qquad \text{var}(x|\lambda) = \lambda $$
where $x \in \left\{0, 1, 2, \dots \right\}$. Below are four examples of Poisson distributions. If events of a particular type occur continuously and independently at a constant rate (\textbf{Poisson process}), the number of events within a time window of fixed width will be distributed according to the Poisson distribution, with rate parameter $\lambda$ proportional to the width of the window.

Situations where the population size, $n$, is large, the probability of an individual event, $p$, is small, but the expected number of events, $np$, is moderate (say five or more) can generally be modeled using a Poisson distribution with $\lambda = np$.

\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure3a-poisson-lambda-change.png}
\end{center}

\begin{question}{question:poissonex}
List 5 random variables from medicine or biology that should follow Poisson distributions.
\end{question}

\section{Geometric \label{sect:geometric}}

The \textbf{geometric distribution} models the number of failures in a sequence of Bernoulli trials before the first success. It has the following properties:
$$ p(x|\mu) = (1-\mu)^x \mu \qquad E[x|\mu] = \frac{1-\mu}{\mu} \qquad \text{var}(x|\mu) = \frac{1-\mu}{\mu^2}  $$
for $x \in \left\{0, 1, 2, \dots \right\}$, where $\mu$ refers to the probability (in the Bernoulli trial) that the trial is a success. Some examples of geometric distributions with different $\mu$ are shown below:
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure4-geometric-mu-change.png}
\end{center}

\begin{question}{question:geometricex}
List 5 random variables from medicine or biology that should follow geometric distributions.
\end{question}

\section{Exponential \label{sect:exponential}}

The \textbf{exponential distribution} is a continuous probability distribution that models waiting times between events that happen independently and continuously at a constant rate (Poisson process), as well as many other random variables\footnote{For example, in an epidemiologic model of an infectious process like COVID-19 community spread, exponential waiting times are often used to model transitions between the susceptible, exposed, infectious, and recovered compartments in the model.}. It has the following properties:
$$ p(x|\lambda) = \lambda e^{-\lambda x} \qquad E[x|\lambda] = \frac{1}{\lambda} \qquad \text{var}(x|\lambda) = \frac{1}{\lambda^2} $$
where $x \in \mathbb{R}^+$ ($x$ is a positive real number, or zero). The exponential distribution is the continuous analogue of the geometric distribution. It is memoryless, which means that the distribution of a waiting time until an event does not depend on how much time has elapsed already.

Here are some different exponential distributions. Compare them to the geometric distribution, above.
\begin{center}
\includegraphics[width=0.9\textwidth]{img/l01-figure5-exponential-lambda-change.png}
\end{center}

\begin{question}{question:exponentialex}
List 5 random variables from medicine or biology that should follow exponential distributions.
\end{question}

\section{Chi-Squared Distribution \label{sect:chisqdist}}

How this distribution arises:
\begin{enumerate}
\item If $Z \sim \mathcal{N}(0, 1)$, the distribution of $U = Z^2$ is called the chi-squared distribution with one degree of freedom.
\item If $U_1, U_2, \dots, U_k$ are independent $\chi_1^2$ random variables, their sum,
$ V = \sum_{i=1}^k U_i $
follows $\chi_k^2$, a chi-squared distribution with $k$ degrees of freedom.
\end{enumerate}

You'll often see the chi-squared distribution used as the sampling distribution for the sample variance in a variety of statistical hypothesis tests. It looks like this:
\begin{center}
\includegraphics[width=0.9\textwidth]{img/hyp-example-chisq-distribution.png}
\end{center}

The parameter $k$, the \textbf{degrees of freedom}, controls the shape of the chi-squared distribution. The actual formula for the chi-squared distribution looks a bit intimidating, but I'm including it here so you can compare it to the other distributions we've seen:
$$ p(x|k) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{k/2 - 1} e^{-x/2} $$
$$ E[x | k] = k \qquad \text{var}(x | k) = 2k $$
The gamma function shown in the denominator of the probability density,
$$ \Gamma(z) = \int_0^{\infty} x^{z-1} e^{-x} dx, $$
is a generalization of the factorial function to complex numbers. For any positive integer $n$, $\Gamma(n) = (n-1)!$.

\section{Student's T Distribution \label{sect:tdist}}

If $Z \sim \mathcal{N}(0, 1)$ and $U \sim \chi_k^2$ and $Z$ and $U$ are independent,
$$ T = \frac{Z}{\sqrt{U/k}} \sim t_k $$
or in words, the statistic $T$ follows a $t$-distribution with $k$ degrees of freedom. The T distribution plays an important role in a family of statistical hypothesis tests called T-tests.

\begin{center}
\includegraphics[width=0.9\textwidth]{img/hyp-example-t-distribution.png}
\end{center}

Again, the functional form of the T distribution is a bit intimidating, but I'm including it for completeness:

$$ p(x|k) = \frac{\Gamma \left(\frac{k+1}{2} \right)} {\sqrt{k\pi}\,\Gamma \left(\frac{k}{2} \right)} \left(1+\frac{x^2}{k} \right)^{-\frac{k+1}{2}} $$
$$ E[x|k] = 0~~\text{ for }k>1; \text{ otherwise undefined} $$
$$ \text{var}(x|k) = \left\{ \begin{array}{cl} \frac{k}{k-2} & k>2 \\
                                               \infty & 1 < k \leq 2 \\
                                               \text{undefined} & \text{otherwise} \end{array} \right. $$

\section{F Distribution \label{sect:fdist}}

If $U$ and $V$ are independent $\chi^2$ random variables with $m$ and $n$ degrees of freedom,
$$ W = \frac{U/m}{V/n} \sim F_{m, n} $$
or in words, the statistic $W$ follows an $F$ distribution with $m$ and $n$ degrees of freedom. I'm not writing out the functional form of the F distribution here because it's too awful-looking, but graphically it looks like this:
\begin{center}
\includegraphics[width=0.9\textwidth]{img/hyp-example-f-distribution.png}
\end{center}

Note that if $T \sim t_k$, then $T^2 \sim F_{1,k}$. The $F$-distribution plays an important role in a class of statistical analysis techniques called \textbf{ANalysis Of VAriance}, or \textbf{ANOVA}.

\vspace{2mm}

\begin{question}{question:likexamples}
For each of the following experimental conditions, which distribution (from those listed above) provides the best model for how the data $x^{(1)},\dots,x^{(n)}$ are generated?
    \begin{enumerate}
    \item[(a)] You are observing several patients' skin in a clinical study to see how long it takes them to develop a rash. You take a picture each day. Let $x^{(i)}$ be the number of days of \emph{no rash} before the rash occurs.

\begin{center}{\small
\begin{tabular}{cc}
\toprule
Patient ID ($i$) & $x^{(i)}$ \\
\midrule
1 & 4 \\
2 & 1 \\
3 & 0 \\
4 & 2 \\
5 & 2 \\
6 & 4 \\
7 & 3 \\
8 & 1 \\
9 & 0 \\
10 & 1 \\
\end{tabular}}
\end{center}

    \item[(b)] Same situation as above except that instead of taking a picture each day, the patient texts you at the moment he/she observes a rash. The data look like this, where $x^{(i)}$ is the time (in days) at which patient $i$ develops a rash:

\begin{center}{\small
\begin{tabular}{cc}
\toprule
Patient ID ($i$) & $x^{(i)}$ \\
\midrule
1 & 2.25 \\
2 & 3.43\\
3 & 0.68\\
4 & 0.04\\
5 & 3.78\\
6 & 5.65\\
7 & 2.88\\
8 & 3.88\\
9 & 2.83\\
10 & 1.87\\
\end{tabular}}
\end{center}

    \item[(c)] Imagine you are Ladislaus Bortkiewicz, and you are modeling the number of persons killed by mule or horse kicks in the Prussian army per year. You have data from the late 1800s over the course of 20 years. Let $x^{(i)}$ be the number of people killed in year $i$.

\begin{center}{\small
\begin{tabular}{cc|cc}
\toprule
Year ($i$) & $x^{(i)}$ & Year ($i$) & $x^{(i)}$ \\
\midrule
1 & 8 & 11 & 9 \\
2 & 10 & 12 & 7 \\
3 & 5 & 13 & 10 \\
4 & 3 & 14 & 12 \\
5 & 10 & 15 & 8 \\
6 & 8 & 16 & 7 \\
7 & 7 & 17 & 8 \\
8 & 2 & 18 & 8 \\
9 & 6 & 19 & 10 \\
10 & 11 & 20 & 7 \\
\end{tabular}}
\end{center}

    \item[(d)] Every year, $10$ scientists go to the same geographic area (same Lyme prevalence) and they each collect $40$ ticks. They test each tick for Lyme disease and record the number of ticks that have Lyme. Let $x^{(i)}$ be the number of ticks with Lyme in the $i$th scientist's bunch.

\begin{center}{\small
\begin{tabular}{cc}
\toprule
Scientist ID ($i$) & $x^{(i)}$ \\
\midrule
1 & 8 \\
2 & 9 \\
3 & 14 \\
4 & 15 \\
5 & 12 \\
6 & 7 \\
7 & 6 \\
8 & 8 \\
9 & 8 \\
10 & 14 \\
\end{tabular}}
\end{center}

    \item[(e)] You have waist circumference data on 1045 men aged 70 and above (see Dey's 2002 paper in the Journal of the American Geriatric Society). It looks like this:

\begin{center}
\includegraphics[width=0.5\textwidth]{img/l01-problem5.png}
\end{center}

    \end{enumerate}
\end{question}


\section{Linear Programming}
Recall that a linear program is the maximisation of $c^\top x$ subject to $Ax\le b,x\ge 0$, where $c\in\mathbb R^n$, $A$ is an $m\times n$ matrix and $b\in\mathbb R^m$.
We also know its dual problem in the form of minimising $b^\top\lambda$ subject to $A^\top\lambda\ge c,\lambda\ge 0$.
\subsection{The Fundamental Theorem}
\begin{theorem}[Fundamental Theorem of Linear Programming]
    Consider the primal problem stated.
    A vector $x^\star\in\mathbb R^n$ is optimal for the primal problem iff there is some vector $\lambda^\star\in\mathbb R^m$ such that:\\
    1. $x^\star$ is feasible for the primal problem.\\
    2. $\lambda^\star$ is feasible for the dual problem.\\
    3. $(b-Ax^\star)^\top\lambda^\star=0=(c-A^\top\lambda^\star)^\top x^\star$ (i.e. the complementary slackness).\\
    Also, if these are true, then $\lambda^\star$ is optimal for the dual problem, and $c^\top x^\star=b^\top\lambda^\star$.
\end{theorem}
\begin{proof}
    For the `if' direction, note that if $x,\lambda$ are feasible for the primal and dual problems, then by Theorem \ref{weak_dual}, $c^\top x\le b^\top\lambda$.
    But if $x^\star$ and $\lambda^\star$ satisfy complementary slackness, $c^\top x^\star=b^\top\lambda^\star$, consequently $c^\top x\le b^\top\lambda^\star\le c^\top x^\star$ for all feasible $x$ and $b^\top\lambda\ge c^\top x^\star=b^\top\lambda^\star$ for all feasible $\lambda$.
    Hence $x^\star$ and $\lambda^\star$ are optimal for their respective problems.\\
    For the ``only if'' direction, note that the objective function $f(x)=c^\top x$ and constraint function $g(x)=Ax$ are linear, hence are both concave and convex at the same time.
    In particular, $f$ is concave while $g$ is convex.
    Hence the value function of the primal function is concave, therefore Theorem \ref{lagrange_nece} guarantees the existence of $\lambda^\star$.
\end{proof}
\subsection{Extreme Points}
Consider the maximisation of $\psi(x)$ subject to $x\in X$ where $X$ is convex and $\psi$ is also convex.
In particular, for any $p\in(0,1)$,
$$\psi(px+(1-p)y)\le p\psi(x)+(1-p)\psi(y)\le\max\{\psi(x),\psi(y)\}$$
Hence the maximum of $\psi$ on any segment occurs at one of the endpoints.
Therefore it is sufficient to consider the maximisation problem in the set of points in $X$ that is not contained properly in a line segment.
\begin{definition}
    Let $X\subset\mathbb R^n$ be a convex set.
    A point $x\in X$ is an extreme point if the only case where there can be $y,z\in X,p\in (0,1)$ with $x=py+(1-p)z$ is when $x=y=z$.
\end{definition}
Extreme points are of course very useful in linear programs as linear functions are both concave and convex.
The concavity tells us we can characterise the optimal solution via duality.
As the set of feasible solutions is convex, the convexity tells us we only need to search for optimal solutions to the extreme points of the set of feasible solutions.
In particular, if the linear program has an optimal solution, then it has a optimiser that is an extreme point.
\subsection{Basic Feasible Solutions}
As we have seen, it is helpful if we formulate linear programs in the form of inequality functional constraints.
There is another form which we can write the linear programs in order to make it more convenient to optimise.
\begin{definition}
    A linear program is in standard form if the constraint can be written as $Ax=b,x\ge 0$.
\end{definition}
All linear programs can be put into the standard form.
For instance, $Ax\le b,x\ge 0$ can be transformed to $Ax+z=b,x,z\ge 0$ by our previous idea of slack variables.
Simply $Ax=b$ can be modified to $A(x-y)=b,x,y\ge 0$.
Combining the ideas, $Ax\le b$ can be transformed to $A(x-y)+z,x,y,z\ge 0$.
\begin{definition}
    Given an $m\times n$ matrix $A$ with $n>m$ and $b\in\mathbb R^m$.
    A solution $x\in\mathbb R^n$ of the equation $Ax=b$ is called basic if at most $m$ entries of $x$ are nonzero, i.e. $x_i\neq 0$ holds for at most $m$ different $i\in\{1,\ldots,n\}$.\\
    If $x$ is a basic solution and $x\ge 0$, it is called a basic feasible solution.
\end{definition}
Now denote the set of feasible solutions by $C=\{x\ge 0:Ax=b\}$, then we have
\begin{theorem}
    If $x$ is an extreme point of $C$, then $x$ is a basic feasible solution.
\end{theorem}
\begin{proof}
    We will prove the contrapositive.
    Let $x$ be a point in $C$ that is not basic, then there are at least $m+1$ indices $i_1,\ldots,i_r$ with $x_{i_k}>0$.
    Then the set $\{A_{i_1},\dots,A_{i_r}\}$ of columns of the matrix $A$ would be linearly dependent as $t\ge m+1$, therefore there exists $(w_1,\ldots,w_r)\neq 0$ such that $w_aA_{i_a}=0$ (summation implied).
    We construct
    $$z_i=\begin{cases}
        w_k\text{, if $i=i_k$ for some $k=1,\ldots,r$}\\
        0\text{, otherwise}
    \end{cases}$$
    Then $Az=0$ and $z\neq 0$, so $A(x\pm\epsilon z)=Ax=b$ for any $\epsilon$ for any $\epsilon$.
    As $x_{i_k}>0$ for any $k$, it is possible to choose some $\epsilon>0$ such that $x\pm\epsilon z\ge 0$.
    So $x\pm\epsilon z\in C$ and
    $$x=\frac{1}{2}(x+\epsilon z)+\frac{1}{2}(x-\epsilon_z)$$
    Hence $x$ is not extreme.
\end{proof}
This theorem has a partial converse.
\begin{theorem}
    Suppose that every set of $m$ columns of $A$ is linearly independent and $x$ is a basic feasible solution, then $x$ is necessarily an extreme point of $C$.
\end{theorem}
\begin{proof}
    By definition, at most $m$ many indices $i$ satisfy $x_i>0$.
    Suppose there is some $y,z\in C$ and $p\in (0,1)$ such that $x=py+(1-p)z$, then $y,z\ge 0$, so whenever $x_i=0$ we have $y_i=z_i=0$.
    Thus there are at least $n-m$ indices $i$ satisfying $y_i=z_i=0$.
    But also $Ay=b=Az$, so
    $$0=A(y-z)=\sum_{i}w_iA_i$$
    where $w=y-z$.
    But at most $m$ entries of $w$ are nonzero while any set of $m$ columns of $A$ is linearly independent, whence we conclude $w=0$.
    It follows that $x=y=z$.
\end{proof}
In conclusion, we can find the optimal solution of a linear program by checking each of the basic feasible solutions, whcih are easier to find and, more importantly, finite.
\subsection{Algorithms for Linear Programs}
Consider a linear program in standard form of maximising $c^\top x$ subject to $Ax=b,x\ge 0$ where $c\in\mathbb R^n,b\in\mathbb R^m$ and $A$ is an $m\times n$ matrix.
Assume that $n>m$ and every set of $m$ columns of $A$ is linearly independent, and that the problem is non-degenerate, i.e. every basic feasible solution $x$ has exactly $m$ indices $i$ with $x_i\neq 0$.
Our goal is to find a mechanical method of finding the optimal solution to the problem.\\
A na\"ive method to do this is to check all extreme points of $C$.
But we also know that extreme points are precisely the basic feasible solutions.
Thus we can roll out an algorithm as follows:\\
First, fix $B\subset\{1,\ldots,n\}$ with $|B|=m$ write and $N=\{1,\ldots,n\}\setminus B$.
Such a $B$ is called a basis of the solution.
Suppose $B=\{i_1,\ldots,i_m\}$, then we define $A_B=(A_{i_1}\ldots A_{i_m})$ which is an invertible matrix by assumption.
For any $x\in\mathbb R^n$, denote $x_B=(x_{i_1},\ldots,x_{i_m})^\top$.
The quantities $c_B,A_N,x_N,c_N$ are defined similarly.\\
Our aim is to find a basic solution of $Ax=b$.
We can write the problem instead in the form $A_Bx_B+A_Nx_N=b$.
Set $x_N=0$ shall yield $x_B=A_B^{-1}b$ as $A_B$ is invertible.
By a rearrangement of coordinate we can write the solution as
$$x=\begin{pmatrix}
    x_B\\
    x_N
\end{pmatrix}=\begin{pmatrix}
    A_B^{-1}b\\
    0
\end{pmatrix}$$
which is basic.
Now $x$ is feasible iff $A_B^{-1}b\ge0$ (which can be checked).
Assuming $x$ is a basic feasible solution, then we can just compute the objective function $c^\top x=c_B^\top x_B$.
Repeat this procedure for all possible choices of $B$ gives a list of all basic feasible solutions and the value of objective function evaluated there.
This is a finite set, so we can find its maximiser which is the solution to the problem.\\
There are some issues with this approach.
Firstly, if $n$ is large, the number of choices of $B$, that is $\binom{n}{m}$, would grow very fast, making it more computationally complex.
Also, this method assumed the existence of maximiser which is not always the case.
Our idea algorithm should certainly detect it when a maximiser does not exist, i.e. the problem is unbounded.\\
There is a better method to accomplish what we wanted.
We also use the idea of computing the basic feasible solutions like above, but we will try not to evaluate all candidate solutions.
By calculation we can show that the dual problem is to minimise $b^\top\lambda$ subject to $A^\top\lambda\ge c$.
So by the fundamental theorem, a feasible $x^\star$ is optimal for our primal problem iff there is a feasible (in the dual problem) Lagrange multiplier $\lambda^\star\in\mathbb R^m$ such that complementary slackness $(c-A^\top\lambda^\star)^\top x^\star=0$ is satisfied.\\
Consider a basic feasible solution $x=(A_B^{-1}b,0)^\top$, then if it is optimal, then there is some $\lambda$ satisfying complementary slackness
$$0=(c-A^\top\lambda)^\top x=(c_B-A_B^\top\lambda)^\top x_B$$
But no index of $x_B$ is zero by assumption, hence $\lambda=(A_B^\top)^{-1}c_B$.
So we can roll out our algorithm in the same way as before, but each time we check as well the feasibility of $\lambda$.
We can stop if such $\lambda$ is feasible, because by the fundamental theorem we have already found the optimal solution.
We also know when the problem is unbounded, that is when none of the basic feasible solution gives a feasible $\lambda$.
\subsection{The Simplex Algorithm}
A third way, which is known as the simplex algorithm, is a yet faster method.
So far our algorithms depend on iterating all candidates for basic feasible solutions randomly, which inspires us to find the solutions in a particular order to speed up the process.
In particular, if our current basic feasible solution does not work, we will want to find the next basic feasible solution that increases the objective function.\\
We start with an initial basic feasible solution $x_0$ whose components with indices in $B_0\subset\{1,\ldots,n\}$ are nonzero.
Write $N_0=\{1,\ldots,n\}\setminus B_0$.
We can associate a Lagrange multiplier $\lambda_0$ by complementary slackness, i.e. one that satisfies $(c-A^\top\lambda_0)_{B_0}=0$.
In particular, $(c-A^\top\lambda_0)^\top x_0=0$, so $c^\top x_0=b^\top\lambda_0$.
Let $\mu_0=c-A^\top\lambda_0$.
If $\mu_0\le 0$ then we are done and $x_0$ is optimal, otherwise we choose $x_1$ by the following:
\begin{claim}
    Suppose there is some $j\in N_0$ such that $\mu_{j,0}>0$ and there is an $i\in B_0$ such that there is a basic feasible solution $x_1$ with basis $B_1=B_0\cup\{j\}\setminus\{i\}$.
    Then
    $$c^\top x_1=c^\top x_0+\mu_{j,0}x_{j,1}>c^\top x_0$$
\end{claim}
\begin{proof}
    If $x$ is any feasible solution, then
    $$c^\top x=c^\top x+\lambda_0^\top(b-Ax)=b^\top\lambda_0+(c-A^\top\lambda_0)^\top x=c^\top x_0+\sum_{k\in N_0}\mu_{k,0}x_k$$
    as $\mu_{k,0}=0$ for any $k\in B_0$.
    The sum is then
    $$\sum_{k\in N_0}\mu_k x_{k,1}=\sum_{k\in N_1}\mu_kx_{k,1}+\mu_jx_{j,1}-\mu_ix_{i,1}=\mu_jx_{j,1}$$
    Also $x_{j,1}>0$ by non-degeneracy assumption.
    The claim follows.
\end{proof}
\begin{proposition}\label{simplex_unbounded}
    Suppose there is a $j\in N_0$ such that $\mu_{j,0}>0$ and for every $i\in B_0$ the basis solution with basis $B_1=B_0\cup\{j\}\setminus\{i\}$ is infeasible, then the problem is unbounded.
\end{proposition}
So if no feasible $x_1$ exists, then we can stop the algorithm and conclude that the problem is unbounded.
Otherwise, we can repeat the algorithm again and again, and stop either at a basic feasible solution whose accompanied Lagrange multiplier is feasible in the dual problem or at a point where we can conclude the unboundedness of the problem.
We may, however, have to go through all basic feasible solutions if we start with a bad guess, but at least we know after each step we shall obtain a better solution candidate.
\subsection{Simplex Algorithm in Action}
We are still considering the problem of maximising $c^\top x$ subject to $Ax=b,x\ge 0$, assuming the same things we assumed previously.
Suppose we know one basic feasible solution $x_0$, we want to use the simplex algorithm.
Before that, we need to do some pre-processing of the problem.
Let $B$ be the basis of $x_0$ and $N=\{1,\ldots,n\}\setminus B$ as usual.
For $x\in\mathbb R^n$, we rearrange the components $x=(x_B,x_N)^\top$ as usual.
Furthermore, the objective function can be written as
$$c^\top x=c^\top x_0+\mu_{N,0}^\top x_N$$
for feasible $x$.
where $\mu_0=c-A^\top\lambda_0$ and $\lambda_0=(A_B^\top)^{-1}c_B$ is the Lagrange multiplier obtained by complementary slackness.
The set of feasible solutions is then subject to the condition
$$x_B+A_B^{-1}A_Nx_N=x_{B,0},x_B,x_N\ge 0$$
To use the simplex algorithm, we first formulate the initial \text{simplex tableau} as the $(m+1)\times(n+1)$ matrix (in practice some of the columns are shuffled due to the position of the basis)
$$\Gamma=\begin{pmatrix}
    I&A_B^{-1}A_N&x_{B,0}\\
    0&\mu_{N,0}^\top&-c^\top x_0
\end{pmatrix}$$
where $I$ is the $m\times m$ identity.
We first test for optimality of $x_0$.
If $\mu_0\le 0$, then we can stop as it would be optimal due to the fundamental theorem.
Otherwise, we choose the pivot column, i.e. $j\in N$ such that $\mu_{j,0}>0$.
a rule of thumb is to pick $j$ such that $\mu_{j,0}$ is the largest.
We also pick the pivot row by looking within the pivot column $j$ and find the $i\in B$ which minimises $x_{i,0}/\Gamma_{i,j}$ over all $i\in B$ with $\Gamma_{i,j}>0$.
If this cannot be done, i.e. $\Gamma_{i,j}\le 0$ for all $i$, then we can stop as it indicates the unboundedness of the problem.\\
Assume we go through these steps without stopping the algorithm, we perform the pivot operation to move to the next basic feasible solution.
To do this, we scale row $i$ with $1/\Gamma_{i,j}$ and replace row $k$ with
$$(\text{old row }k)-(\text{old row }i)\times\frac{\Gamma_{k,j}}{\Gamma_{i,j}}$$
for all $k\neq i$ and run the algorithm on this new simplex tableau.
\begin{remark}
    1. For the initial basic feasible solution we have $x_{i,0}>0$ and $x_{j,0}=0$, whilst for the next basic feasible solution $x_{i,1}=0$ and $x_{j,1}=x_{i,0}/\Gamma_{i,j}>0$.\\
    2. The pivot operation is actually just a Gaussian elimination.\\
    3. The $\Gamma$ we obtained after the pivot operation is the same $\Gamma$ had we start with $x_1$ instead.\\
    4. Suppose $\Gamma_{i,j}\le 0$ for all $i\in B$, then for $i\in B$ and for $r>0$ let $x_r=x_0+r(\delta_j-\Gamma_{i,j}\delta_i)$ where $\delta_k$ is the $k^{th}$ standard basis.
    Then the transformation $x_0\mapsto x_r$ replaces $x_{i,0}$ with $x_{i,r}=x_{i,0}-r\Gamma_{i,j}$ and $x_{j,0}=0$ with $x_{j,r}=r$ and leaves other entries unchanged.
    Now $x_r$ is feasible since $x_r\ge 0$ and
    $$\begin{pmatrix}
        I&A_B^{-1}A_N
    \end{pmatrix}x_r=x_0$$
    But $c^\top x_r=c^\top x_0+r\mu_{j,0}\to\infty$ as $r\to\infty$, so the problem is unbounded.
    This proves Proposition \ref{simplex_unbounded}.
\end{remark}
\begin{example}
    Consider the linear program to maximise $3x_1+2x_2$ subject to
    $$\begin{cases}
        2x_1+x_2\le 4\\
        2x_1+3x_2\le 6\\
        x_1,x_2\ge 0
    \end{cases}$$
    Then its dual problem is to minimise $4\lambda_1+6\lambda_2$ subject to
    $$\begin{cases}
        2\lambda_1+2\lambda_2\ge 3\\
        \lambda_1+3\lambda_2\ge 2\\
        \lambda_1,\lambda_2\ge 0
    \end{cases}$$
    By introducing slack variables, the primal problem has constraint
    $$\begin{cases}
        2x_1+x_2+z_1=4\\
        2x_1+3x_2+z_2=6\\
        x_1,x_2,z_1,z_2\ge 0
    \end{cases}$$
    and the dual problem has constraint
    $$\begin{cases}
        2\lambda_1+2\lambda_2-v_1= 3\\
        \lambda_1+3\lambda_2-v_2= 2\\
        \lambda_1,\lambda_2,v_1,v_2\ge 0
    \end{cases}$$
    So the complementary slackness translates to
    $$x_1v_1=x_2v_2=\lambda_1z_1=\lambda_2z_2=0$$
    Running the simplex algorithm shall yield the maximiser $x_1=3/2,x_2=1$ with then corresponding dual solution $\lambda_1=5/4,\lambda_2=1/4$.
    We start with the simplex tableau (asterisked columns are where the identity matrix and the zero vectors are at)
    $$\begin{array}{c|cccc|c}
        &&&\ast&\ast&\\
        &x_1&x_2&z_1&z_2&\\ \hline
        z_1&2&1&1&0&4\\
        z_2&2&3&0&1&6\\ \hline
        \text{payoff}&3&2&0&0&0
    \end{array}$$
    Now the payoff row is not non-positive, hence we have not arrived at the optimal solution.
    As $3$ is the largest payoff entry, we let $x_1$ enter the basis, so the first column is the pivot column.
    We choose the first row to be the pivot row.
    The the pivot operation yields the next simplex tableau:
    $$\begin{array}{c|cccc|c}
        &\ast&&&\ast&\\
        &x_1&x_2&z_1&z_2&\\ \hline
        x_1&1&1/2&1/2&0&2\\
        z_2&0&2&-1&1&2\\ \hline
        \text{payoff}&0&1/2&-3/2&0&-6
    \end{array}$$
    It is still not optimal due to the positive $1/2$ entry in the payoff row.
    So this time the pivot column is the second column and we choose the second row as the pivot row.
    This gives us
    $$\begin{array}{c|cccc|c}
        &\ast&\ast&&&\\
        &x_1&x_2&z_1&z_2&\\ \hline
        x_1&1&0&3/4&-1/4&3/2\\
        x_2&0&1&-1/2&1/2&1\\ \hline
        \text{payoff}&0&0&-5/4&-1/4&-13/2
    \end{array}$$
    which we realise is optimal as the payoff is nonpositive.
\end{example}
\begin{remark}
    1. If the linear program is a minimisation instead of a maximisation problem, then the stopping criterion would be the tableau being nonnegative.\\
    2. When the problem is of the form like above, i.e. maximising $c^\top x$ subject to $Ax\le b,x\ge 0$, one observe that the coefficient of the payoff row under the slack variables are minus the corresponding dual variables (via complementary slackness).
    For the final tableau, the feasible dual variables are the Lagrange multipliers for the problems.\\
    3. In the example, we have
    $$\begin{pmatrix}
        3/4&-1/4\\
        -1/2&1/2
    \end{pmatrix}\begin{pmatrix}
        2&1&1&0&4\\
        2&3&0&1&6
    \end{pmatrix}=\begin{pmatrix}
        1&0&3/4&-1/4&3/2\\
        0&1&-1/2&1/2&1
    \end{pmatrix}$$
    Hence the perturbed problem of maximising $3x_1+2x_2$ subject to
    $$\begin{cases}
        2x_1+x_2\le 4+\epsilon_1\\
        2x_1+3x_2\le 6+\epsilon_2\\
        x_1,x_2\ge 0
    \end{cases}$$
    with $\epsilon_{1,2}$ small yields the final tableau
    $$\begin{array}{c|cccc|c}
        &\ast&\ast&&&\\
        &x_1&x_2&z_1&z_2&\\ \hline
        x_1&1&0&3/4&-1/4&3/2+3\epsilon_1/4-\epsilon_2/4\\
        x_2&0&1&-1/2&1/2&1-\epsilon_1/2+\epsilon_2/2\\ \hline
        \text{payoff}&0&0&-5/4&-1/4&-13/2-5\epsilon_1/4-\epsilon_2/4
    \end{array}$$
    Therefore it has maximiser at
    $$x_1=\frac{3}{2}+\frac{3}{4}\epsilon_1-\frac{1}{4}\epsilon_1,x_2=1-\frac{1}{2}\epsilon_1+\frac{1}{2}\epsilon_2, 3x_1+2x_2=\frac{13}{2}+\frac{5}{4}\epsilon_1+\frac{1}{4}\epsilon_2$$
    given that they are feasible (possible for small enough $\epsilon_{1,2}$).
\end{remark}
\subsection{The Two-Phase Algorithm}
Again our problem is to maximise $c^\top x$ subject to $Ax=b,x\ge 0$.
In order to roll out the simplex algorithm, we need at least one basic feasible solution.
It is certainly tempting if we can know a way that can yield one efficiently.\\
There are a lot of approaches, for example just pick a basis and computed $A_B^{-1}b$ (using the notations we introduced earlier).
This is certainly a sufficient algorithm, but this might not be very efficient.\\
Consider a new problem of minimising $e^\top y$ subject to $Ax+y=b,x,y\ge 0$ where $e=(1,\ldots,1)^\top$.
The variable $y$ here is called an artificial variable.
Now $(x,y)=(0,b)$ is a basic feasible solution for this problem, so we can apply the simplex algorithm which must terminate at some $(x,y)=(x_0,0)$ given that the original problem is feasible.
Then $x_0$ has to be basic feasible for the original problem.
So we can modify the algorithm by adding a first phase to compute one basic feasible solution first (called Phase I) and then apply the simplex algorithm to it (which is Phase II).
\begin{example}
    Consider the problem to maximise $x_1-3x_2+5x_3$ subject to
    $$\begin{cases}
        x_1+x_2+x_3\le 30\\
        -x_2+2x_3=20\\
        -x_1+2x_2+x_3\ge 40\\
        x_1,x_2,x_3\ge 0
    \end{cases}$$
    we introduce slack variables as usual to transform the constraint to
    $$\begin{cases}
        x_1+x_2+x_3+z_1=30\\
        -x_2+2x_3=20\\
        -x_1+2x_2+x_3-z_2=40\\
        x_1,x_2,x_3,z_1,z_2\ge 0
    \end{cases}$$
    Add in artificial variable $y_1,y_2$,
    $$\begin{cases}
        x_1+x_2+x_3+z_1=30\\
        -x_2+2x_3+y_1=20\\
        -x_1+2x_2+x_3-z_2+y_2=40\\
        x_1,x_2,x_3,z_1,z_2,y_1,y_2\ge 0
    \end{cases}$$
    In some sense we should have added a variable $y_3$ on the first expression as well, but it is actually redundant as we already have an nonnegative expression.
    We solve this new problem with the simplex algorithm, with the initial tableau:
    $$\begin{array}{c|ccccccc|c}
        &&&&\ast&&\ast&\ast&\\
        &x_1&x_2&x_3&z_1&z_2&y_1&y_2&\\ \hline
        z_1&1&1&1&1&0&0&0&30\\
        y_1&0&-1&2&0&0&1&0&20\\
        y_2&-1&2&1&0&-1&0&1&40\\ \hline
        \text{Phase II}&1&-3&5&0&0&0&0&0\\ \hline
        \text{Phase I}&0&0&0&0&0&1&1&0
    \end{array}$$
    We have put the objective functions of both phases so that we can manipulate both simultaneously so that when Phase I is completed, we can immediately go to Phase II.\\
    Now the identity part of the upper part of the tableau does not correspond to the zero part of the Phase I payoff, so we do a transformation to modify the tableau:
    $$\begin{array}{c|ccccccc|c}
        &&&&\ast&&\ast&\ast&\\
        &x_1&x_2&x_3&z_1&z_2&y_1&y_2&\\ \hline
        z_1&1&1&1&1&0&0&0&30\\
        y_1&0&-1&2&0&0&1&0&20\\
        y_2&-1&2&1&0&-1&0&1&40\\ \hline
        \text{Phase II}&1&-3&5&0&0&0&0&0\\ \hline
        \text{Phase I}&1&-1&-3&0&1&0&0&-60
    \end{array}$$
    Note that as we are encountering a minimisation problem in Phase I, our aim is to make the last row nonnegative.
    Pivot once:
    $$\begin{array}{c|ccccccc|c}
        &&&\ast&\ast&&&\ast&\\
        &x_1&x_2&x_3&z_1&z_2&y_1&y_2&\\ \hline
        z_1&1&3/2&0&1&0&-1/2&0&20\\
        x_3&0&-1/2&1&0&0&1/2&0&10\\
        y_2&-1&5/2&0&0&-1&-1/2&1&30\\ \hline
        \text{Phase II}&1&-1/2&0&0&0&-5/2&0&-50\\ \hline
        \text{Phase I}&1&-5/2&0&0&1&3/2&0&-30
    \end{array}$$
    Pivot twice:
    $$\begin{array}{c|ccccccc|c}
        &&\ast&\ast&\ast&&&&\\
        &x_1&x_2&x_3&z_1&z_2&y_1&y_2&\\ \hline
        z_1&8/5&0&0&1&3/5&-1/5&-3/5&2\\
        x_3&-1/5&0&1&0&-1/5&2/5&1/5&16\\
        x_2&-2/5&1&0&0&-2/5&-1/5&2/5&12\\ \hline
        \text{Phase II}&4/5&0&0&0&-1/5&-13/5&1/5&-44\\ \hline
        \text{Phase I}&0&0&0&0&0&1&1&0
    \end{array}$$
    The last row is nonnegative and the objective function attains zero, so we have already achieved our aim for Phase I with the basic feasible solution shown in the tableau.
    This produces our initial tableau for Phase II:
    $$\begin{array}{c|ccccc|c}
        &&\ast&\ast&\ast&&\\
        &x_1&x_2&x_3&z_1&z_2&\\ \hline
        z_1&8/5&0&0&1&3/5&2\\
        x_3&-1/5&0&1&0&-1/5&16\\
        x_2&-2/5&1&0&0&-2/5&12\\ \hline
        \text{Phase II}&4/5&0&0&0&-1/5&-44\\
    \end{array}$$
    Now we are maximising, so we need to make the payoff nonpositive.
    After the first pivoting we have
    $$\begin{array}{c|ccccc|c}
        &\ast&\ast&\ast&&&\\
        &x_1&x_2&x_3&z_1&z_2&\\ \hline
        x_1&1&0&0&5/8&3/8&5/4\\
        x_3&0&0&1&1/8&-1/8&65/4\\
        x_2&0&1&0&1/4&-1/4&25/2\\ \hline
        \text{Phase II}&0&0&0&-1/2&-1/2&-45\\
    \end{array}$$
    which has nonpositive payoff, hence the optimal solution is at $(x_1,x_2,x_3)=(5/4,25/2,65/4)$.
\end{example}


% Series expansion (Differentiate between the two approaches being used)
\subsection{Series Expansion}

SymPy is able to calculate the symbolic series expansion of an arbitrary series
or expression involving elementary and special functions and multiple
variables. For this it has two different implementations: the \texttt{series}
method and Ring Series.

The first approach stores a series as an instance of the \texttt{Expr} class.
Each function has its specific implementation of its expansion, which is able to
evaluate the Puiseux series expansion about a specified point. For example,
consider a Taylor expansion about 0:

\begin{verbatim}
>>> series(sin(x+y) + cos(x*y), x, 0, 2)
1 + sin(y) + x*cos(y) + O(x**2)
\end{verbatim}

The newer and much faster approach called Ring Series makes use of the
fact that a truncated Taylor series is simply a polynomial. Correspondingly, it
may be represented by a sparse polynomial, which performs
well in a under a wide range of cases. Ring Series also gives
the user the freedom to choose the type of coefficients to use, resulting in
faster operations on certain types.

For this, several low-level methods for expansion of trigonometric, hyperbolic
and other elementary operations (like series inversion, calculating the $n$th
root, etc.) are implemented using variants of the Newton Method~\cite{zimmerman}.
All these support Puiseux series expansion. The following example demonstrates
the use of an elementary function that calculates the Taylor expansion of the
sine of a series.

\begin{verbatim}
>>> from sympy.polys.ring_series import rs_sin
>>> R, t = ring('t', QQ)
>>> rs_sin(t**2 + t, t, 5)
-1/2*t**4 - 1/6*t**3 + t**2 + t
\end{verbatim}

The function \texttt{sympy.polys.rs\_series} makes use of these elementary
functions to expand an arbitrary SymPy expression. It does so by following a
recursive strategy of expanding the lowermost functions first and then
composing them recursively to calculate the desired expansion. Currently, it
only supports expansion about 0 and is under active development. Ring Series
is several times faster than the default implementation with the speed
difference increasing with the size of the series. The
\texttt{sympy.polys.rs\_series} takes as input any SymPy expression and hence
there is no need to explicitly create a polynomial \texttt{ring}. An example
demonstrating its use:

% rs_series bug, output sometimes has a factored out
% no-doctest
\begin{verbatim}
>>> from sympy.polys.ring_series import rs_series
>>> from sympy.abc import a, b
>>> rs_series(sin(a + b), a, 4)
-1/2*(sin(b))*a**2 + (sin(b)) - 1/6*a**3*(cos(b)) + a*(cos(b))
\end{verbatim}

\subsection{Formal Power Series}

SymPy can be used for computing the formal power series of a function.
The implementation is based on the algorithm described in the paper on
formal power series~\cite{Gruntz93formalpower}.  The advantage of this approach is
that an explicit formula for the coefficients of the series expansion is generated
rather than just computing a few terms.

The following example shows how to use \texttt{fps}:

\begin{verbatim}
>>> f = fps(sin(x), x, x0=0)
>>> f.truncate(6)
x - x**3/6 + x**5/120 + O(x**6)
>>> f[15]
-x**15/1307674368000
\end{verbatim}

\subsection{Fourier Series}

SymPy provides functionality to compute Fourier series of a function using the
\texttt{fourier\_series} function:

\begin{verbatim}
>>> L = symbols('L')
>>> expr = 2 * (Heaviside(x/L) - Heaviside(x/L - 1)) - 1
>>> f = fourier_series(expr, (x, 0, 2*L))
>>> f.truncate(3)
4*sin(pi*x/L)/pi + 4*sin(3*pi*x/L)/(3*pi) + 4*sin(5*pi*x/L)/(5*pi)
\end{verbatim}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\subsection{Worksheet - Variational Principle Applied to Free Fall}
As a second example for the variational principle, let’s consider the (one-dimensional) free fall of a particle of mass $m$ due to gravity starting at zero and from rest for a duration T.

\begin{p}
Write down the Lagrange-Function $\LL = T - U$ using a generalized coordinate q.
\end{p}
\begin{s}
Let us use the height of particle as our generalized coordinate $q$. In this case the velocity of the particle is simply $\dot{q}$, so we have that:
\[T = \frac{1}{2}m\dot{q}^2, \quad U = mgq\]
Hence the Lagrangian is:
\[\LL = T - U = \frac{1}{2}m\dot{q}^2 - mgq\]
\end{s}

\begin{p}
Using Newton’s 2\textsuperscript{nd} law, write down the equation of motion for $q$ and the solution $q(t)$.
\end{p}
\begin{s}
By Newton's second law, we have that there is only a gravitational force $-mg$ acting on the particle, and hence it reads:
\[\sum F = -mg = m\ddot{q} \implies -g = \ddot{q}\]
We solve this second order differential equation by integrating twice, which gives us:
\[q(t) = -\frac{1}{2}gt^2 + \dot{q}(t=0)t + q(t=0)\]
Since the particle starts at rest, we have that $\dot{q}(0) = 0$ and hence:
\[q(t) = -\frac{1}{2}gt^2 + q(t=0)\]
\end{s}

\begin{p}
Write down the “boundary conditions” $q(t=0)$ and $q(t=T)$.
\end{p}
\begin{s}
We define our coordinate system such that $q(t=0) = 0$, and then by the solution above, $q(t=T) = -\frac{1}{2}gT^2$.
\end{s}

\begin{p}
A possible 'trial trajectory' could be the from $q_{\text {trial }, \alpha}(t)=-\frac{1}{2} g t^{2}+\alpha \sin \left(\frac{\pi t}{T}\right)$. (This is how variational calculus is carried out!) Check that this function satisfies the boundary conditions at $\mathrm{t}=0$ and $\mathrm{t}=\mathrm{T}$.
\end{p}
\begin{s}
We see that:
\[q(t=0) = -\frac{1}{2}g(0)^2 + \alpha \sin \left(\frac{\pi (0)}{T}\right) = 0\]
and
\[q(t=T) = -\frac{1}{2}gT^2 + \alpha \sin \left(\frac{\pi T}{T}\right) = -\frac{1}{2}gT^2 \]
as desired.
\end{s}

\begin{p}
Instead of considering all possible trajectories that obey the boundary conditions, we consider here only a specific family parameterized by $\alpha .$ Thus $S\left[q_{\text {trial}, \alpha}(t)\right]=S(\alpha)$ is a function of $\alpha$ Write down $S\left[q_{\text {trial}, \alpha}(t)\right]$ and show that it can be written as
\[ S\left[q_{\text {trial}, \alpha}(t)\right]=S[q(t)]+m \alpha^{2} \pi^{2} / 4 T \]
In the above form it is evident that the true trajectory $(\alpha=0)$ minimizes $S$.
\end{p}
\begin{s}
Plugging in the trial trajectory into the Lagrangian determined in question 1, we have:
\begin{align*}
S\left[q_{\text {trial}, \alpha}(t)\right] &= \int_{0}^T \LL(q,\dot{q}, t)dt
\\ &= \int_{0}^T\frac{1}{2}m\left(\dod{}{t}\left(-\frac{1}{2} g t^{2}+\alpha \sin \left(\frac{\pi t}{T}\right)\right)\right)^2 - mg\left(-\frac{1}{2} g t^{2}+\alpha \sin \left(\frac{\pi t}{T}\right)\right)dt
\\ &= \int_{0}^T\frac{1}{2}m\left(-gt + \frac{\alpha \pi}{T}\cos(\frac{\pi t}{T})\right)^2 + \frac{mg^2}{2}t^2 - mg\alpha\sin(\frac{\pi t}{T}) dt
\\ &= \int_0^T \frac{1}{2}m\left(g^2t^2 - 2\frac{gt\alpha\pi}{T}\cos(\frac{\pi t}{T}) + \frac{\alpha^2\pi^2}{T^2}\cos[2](\frac{\pi t}{T})\right) + \frac{mg^2}{2}t^2 - mg\alpha\sin(\frac{\pi t}{T}) dt
\\ &= \int_0^T mg^2t^2 + \frac{m\alpha^2\pi^2}{2T^2}\cos[2](\frac{\pi t}{T}) dt \text{ (Integrals over full period of sine/cosine are zero)}
\\ &= \int_0^T mg^2t^2 + \frac{m\alpha^2 \pi^2}{4T}
\\ &= S[q(t)] + \frac{m\alpha^2 \pi^2}{4T}
\end{align*}
Clearly this is minimized for $\alpha = 0$, and for any $\alpha > 0$ the action is larger. The one the action that makes the action minimal is the true physical path.
\end{s}

\begin{p}
Show that if $S = \int_{t_1}^{t_2} f(x, y, \dot{x}, \dot{y}) dt$ that there are \textbf{two} Euler-Lagrange equations for the stationary curves $x(t), y(t)$. Write the Euler-Lagrange equations.
\end{p}
\begin{s}
Let the correct path be given by $x = x(u)$ and by $y = y(u)$, and the wrong/varied path be given by $x = x(u) + \alpha\xi(u)$ and $y = y(u) + \beta\eta(u)$. By a very similar process to worksheet 1, we impose the requirement that $\left. \od{S}{\alpha}\right|_{\alpha = 0} = 0$ and $\left. \od{S}{\beta}\right|_{\beta = 0} = 0$. This leads to the two Euler Lagrange equations:
\[\dpd{f}{x} = \dod{}{u}\dpd{f}{x'} \text{ and } \dpd{f}{y} = \dod{}{u}\dpd{f}{y'}\]
\end{s}

\begin{p}
Construct Lagrange’s equations for a particle in a two-dimensional potential $U(x,y)$, and show that these are equivalent to a particle obeying Newton’s equations.
\end{p}
\begin{s}
We have that the Lagrangian is given by:
\[\LL = \frac{1}{2}m\left(\dot{x}^2+\dot{y}^2\right) - U(x,y)\]
So by the two Euler-Lagrange equations above:
\[\dpd{L}{x} = \dod{}{t}\dpd{L}{\dot{x}} \implies -\dod{U(x,y)}{x} = \dod{}{t}m\dot{x} \implies F_x = m \ddot{x} \]
\[\dpd{L}{y} = \dod{}{t}\dpd{L}{\dot{y}} \implies -\dod{U(x,y)}{y} = \dod{}{t}m\dot{y} \implies F_y = m \ddot{y} \]
So we recover Newton's second law of:
\[\v{F} = m\ddot{\v{r}}\]
\end{s}
\end{document}