\documentclass[revision-guide.tex]{subfiles}

%% Current Author: SEQ
\setcounter{chapter}{10}
\begin{document}
\tikzset{
  every pin/.style={
      font=\scriptsize,
      pin distance=4ex},
  small dot/.style={
      fill=gray,
      circle,
      scale=0.1}
}
\chapter{Oscillations}
\section{Simple Harmonic Motion}
\spec{Recall the condition for simple harmonic motion and hence identify situations in which simple harmonic motion will occur}

Any oscillation where the acceleration is proportional to the displacement from an equilibrium position and in the opposite direction to the displacement is described as Simple Harmonic Motion (SHM).

These two conditions can be expressed in equation form as:
\[ a \propto -x \]
Notice the minus sign to signify that the acceleration is in the opposite direction to the displacement.

We will mainly be looking at idealised springs and pendulums to understand the maths behind it but the real reason for studying SHM is that it is an excellent approximation for many of the oscillations that we come across in the natural world.

So while the topic is introduced with some rather prosaic examples this provides the building blocks to understanding earthquakes or how atoms vibrate in lattices as well as any musical instrument you can think of.

%\pagebreak

So without further ado let's look at our first and perhaps simplest example.

A mass on a spring on a smooth horizontal surface.
\vspace{0.2in}

\begin{tikzpicture}
\node[rectangle,scale = 1.8, fill=blue,inner sep=2.5mm, text = white] (a) at (3.75,3) {$m$};

\draw[decoration={aspect=0.3, segment length=2mm, amplitude=3mm,coil},decorate] (0,3) -- (a);

\fill [pattern = north east lines] (0,2) rectangle (6,2.4);
\fill [pattern = north east lines] (-.5,2) rectangle (0,4);
\draw[thick] (0,2.4) -- (6,2.4);
\draw[thick] (0,2.4) -- (0,4);

\end{tikzpicture}

What happens if we move the mass by a distance x and then let it go?

\begin{tikzpicture}
\node[rectangle,scale = 1.8, fill=blue,inner sep=2.5mm, text = white] (a) at (5,3) {$m$};

\draw[decoration={aspect=0.3, segment length=3mm, amplitude=3mm,coil},decorate] (0,3) -- (a);

\fill [pattern = north east lines] (0,2) rectangle (6,2.4);
\fill [pattern = north east lines] (-.5,2) rectangle (0,4);
\draw[thick] (0,2.4) -- (6,2.4);
\draw[thick] (0,2.4) -- (0,4);
\draw[thick, |-latex] (3,3.5) -- (4.25,3.5);
\node at (3.5,3.7) {$x$};
\end{tikzpicture}

Before going into any mathematical detail we can think about what will happen to the mass.
\begin{itemize}
\item We know that there is now a force from the stretched string which is pulling the mass back to its original position.
\item This will cause the mass to accelerate to the left.
\item Once the mass reaches its starting point it will overshoot and start compressing the spring.
\item This creates a resultant force which will again restore the mass to its original position.
\item This will cause the mass to accelerate to the right.
\item The mass will overshoot again and the cycle will continue.
\end{itemize}
Note that the direction of the acceleration is always opposite to the displacement.

The mathematical treatment for this starts very simply with a basic knowledge of Hooke's law.

The force on a stretched or compressed spring is given by
\[
F=-kx
\]
where k is the spring constant.

Newton's second law tells us that
\[
F=ma
\]

Putting these together we get
\[
a = \dfrac{F}{m}=-\dfrac{k}{m}x
\]
Because k and m are constant this satisfies the condition for SHM,
\[
a \propto -x
\]

\begin{example}
A  2kg mass attached to a horizontal spring of spring constant 0.3Nm$^{-1}$ is stretched by 10cm and then released.

Find the maximum acceleration of the mass

	\vspace{1cm}

		\textbf{Answer}

Using the formula $a = -\dfrac{k}{m}x$
The maximum acceleration will occur when x is a maximum so
\[
acceleration_{max} = - \dfrac{0.3}{2} \times 0.1
\]
\[
acceleration_{max} = -0.015 ms^{-2}
\]

\end{example}

\spec{* show that the condition for simple harmonic motion leads to a differential equation of the form
\[
\dfrac{d^2x}{dt^2} = - \omega^2 x
\]
and that
\[
x = A cos \omega t
\]
is a solution to this equation}

Acceleration is the rate of change of velocity

\[
a = \dfrac{dv}{dt}
\]

and velocity is the rate of change of displacement

\[
v = \dfrac{dx}{dt}
\]

Putting these two together gives the condition for simple harmonic motion as.

\[
\dfrac{d^2x}{dt^2}=-\omega ^2 x
\]

where $\omega ^2$ is a (strange choice of) constant.

This is a second order differential equation and to solve it we need to find a function which when differentiated twice gives us the negative of the original function.

By inspection we can see that functions with $\sin \omega t$ and $\cos \omega t$ are both possible solutions.

e.g.

\begin{align*}
x &= A cos \omega t \\
\dfrac{dx}{dt} &= - A\omega sin \omega t \\
\dfrac{d^2x}{dt^2} &= -A\omega^2 cos \omega t = -\omega ^2 x \\
\end{align*}

So $x= Acos \omega t$ is a solution (using sin is equivalent but with a phase offset.)

Looking at this function we can see that it will give us a cosine wave with an Amplitude of A and a time period of $ \dfrac{2 \pi}{\omega}$.

This gives us a frequency on $\dfrac{1}{T} = \dfrac{\omega}{2 \pi}$

So $\omega$ is the angular frequency (which explains our strange choice of constant).

\begin{tikzpicture}
	%curve
		\draw (0.5,1) cos (1.5,0) sin (2.5,-1) cos (3.5,0) sin (4.5,1) cos
			(5.5,0) sin (6.5,-1) cos (7.5,0) sin (8.5,1) cos (9.5,0)
			sin (10.5,-1) cos (11.5,0);
	% zero crossing
		\draw[dotted] (0.5,0) -- (11.5,0);
	%axis + labels
		\draw [<->] (0.5,1) -- (0.5,-1);
		\draw [->] (0.5,0) -- (11.5,0);
		\node at (11.7,0) {$t$};
		\node at (0.5,1.2) {$+A$};
		\node at (0.5,-1.2) {$-A$};
		\node at (0.3,0) {$0$};
	%Time period description
		\draw [<->] (4.5,1.15) -- (8.5,1.15);
		\node at (6.5,1.6) {Time period, $\dfrac{2 \pi}{\omega}$};
	%Amplitude description
		\draw [<->] (6.5,0) -- (6.5,-1);
		%to stop line overlap
		\draw [fill=white,ultra thick,white] (7,-0.3) rectangle (9,-0.7);
		\node at (8, -0.5) {Amplitude, $A$};
\end{tikzpicture}

We now have a general expression for the displacement x of the object after a time t.

The value of $\omega$ will be given by the physical properties of the system.

e.g.
For the mass on a spring we looked at earlier

\begin{align*}
F &= -kx && \text {Force on mass by Hooke's Law}\\
a &= -\dfrac{k}{m} x  && \text{gives acceleration}\\
\dfrac{d^2x}{dt^2} &= -\omega ^2 x && \text{comparing with condition for SHM} \\
\omega ^2 &= \dfrac{k}{m} && \text{gives value for constant } \omega \\
\omega &= \sqrt{\dfrac{k}{m}} \\
\end{align*}

\spec{* use differential calculus to derive the expressions
\[
v=-A \omega sin \omega t
\] and
\[a = -A\omega^2 cos \omega t
\]
for simple harmonic motion.}

This is achieved very simply by differentiating our expression for displacement with respect to time as velocity =$\dfrac{dx}{dt}$ and acceleration = $\dfrac{dv}{dt}$

Remember that the differential of cos is -sin.

so
\[
x = A cos \omega t
\]
\[
v = \dfrac{dx}{dt}= -A \omega sin \omega t
\]
\[
a = \dfrac{dv}{dt}= -A \omega^2 cos \omega t
\]

\spec{ *recall and use the expressions $x = A cos\omega t, v = -A \omega sin \omega t, a = -A \omega^2 cos \omega t$ and $F = -m \omega^2x$ to solve problems}

We can use these equations to solve problems by identifying the variables and substituting.

The last equation is a combination of $a = -\omega^2 x$ and $F = ma$

\begin{example}

A mass attached to a spring is set into motion on a smooth horizontal surface. The amplitude of oscillation is 15mm and it takes 5 seconds to perform 20 oscillations.

Calculate the time period and frequency.

Hence calculate the velocity and acceleration after 9.3 seconds.

	\vspace{1cm}

\textbf{Answer}

The first part doesn't require any knowledge of SHM. If there are 20 oscillations in 5 seconds this means $\dfrac{20}{5}$ oscillations in one second.

So $f=4Hz$

and $T = \dfrac{1}{f} = 0.25s$

Now that we have the frequency we can calculate the angular frequency $\omega = 2 \pi f = 8 \pi$

We are given the amplitude A as 15mm and the time is 9.3 seconds so we can now use our SHM formulae.

So for the velocity

\[
v=-A \omega sin \omega t
\]

\[
v = -15 \times 8 \pi sin (8 \times \pi \times 9.3)
\]
\[
v = -360 mms^{-1} = .36ms^{-1}
\]

and for acceleration

\[
a = \dfrac{dv}{dt}= -A \omega^2 cos \omega t
\]

\[
a = 15 (8 \times \pi)^2 cos (8 \times \pi \times 9.3)
\]

\[
a = 2900 mms^{-2} = 2.9ms^{-2}
\]

\end{example}

\spec{ recall and use $T = \dfrac{2 \pi}{\omega}$
as applied to a simple harmonic oscillator}

Angular frequency $w$ is how many radians per second the oscillator goes through with one complete cycle being $2 \pi$ radians.

So $\dfrac{1}{\omega}$ is how many seconds is takes to complete one radian.

and the time for one complete cycle is $T = \dfrac{2 \pi}{\omega}$

\spec{ understand the phase differences between displacement, velocity and acceleration in simple harmonic
motion}

Plotting the equations \[x = A cos \omega t
\]
\[
v =  -A \omega sin \omega t
\]
\[
a = -A \omega^2 cos \omega t
\]
onto a graph gives the following.

\begin{tikzpicture}
	%curve 1
		\draw[blue] (0.5,0) sin (1.5,-1) cos (2.5,0) sin (3.5,1) cos (4.5,0)
			sin (5.5,-1) cos (6.5,0) sin (7.5,1) cos (8.5,0) sin (9.5,-1)
			cos (10.5,0) sin (11.5,1);
	%curve 2
    \draw[red] (0.5,-1) cos (1.5,0) sin (2.5,1) cos (3.5,0) sin (4.5,-1) cos
			(5.5,0) sin (6.5,1) cos (7.5,0) sin (8.5,-1) cos (9.5,0)
			sin (10.5,1) cos (11.5,0);
		cos (10.5,0) sin (11.5,1);
    %curve 3
        	\draw (0.5,1) cos (1.5,0) sin (2.5,-1) cos (3.5,0) sin (4.5,1) cos
			(5.5,0) sin (6.5,-1) cos (7.5,0) sin (8.5,1) cos (9.5,0)
			sin (10.5,-1) cos (11.5,0);
	% zero crossing
		\draw[dotted] (0.5,0) -- (11.5,0);
	%axis + labels
		\draw [<->] (0.5,1) -- (0.5,-1);
		\draw [->] (0.5,0) -- (11.5,0);
		\node at (11.7,0) {$t$};
		\node[rotate=90] at (0.2,0) {displacement};
        \node[rotate=90, blue] at (-0.12,0) {velocity};
        \node[rotate=90, red] at (-0.5,0) {acceleration};
	%Phase difference description
		\draw [<->] (0.5,1.2) -- (2.5,1.2);
		\node at (1.5,1.5) {$180^\circ$};
\end{tikzpicture}

Note the phase shift between displacement, velocity and acceleration.

Velocity is $\dfrac{\pi}{2}$ or $90^\circ$out of phase with displacement.

Acceleration is $\pi$ or $180^\circ$ out of phase with displacement.

So when acceleration or displacement has a maximum magnitude, the velocity is zero

\spec{ *show that the total energy of an undamped simple harmonic system is given by $E = \dfrac{1}{2}mA^2\omega^2$ and
recognise that this is a constant}

For an undamped system no energy is lost to the surroundings so the total energy is conserved.

The total energy will take the form of Potential Energy  and Kinetic Energy.

Considering our mass on a spring at maximum displacement the spring will be at maximum extension and the mass will be stationary. So the total energy will be in the form of P.E.

Similarly when the displacement is zero all the energy will be K.E.

At any point in between the total energy will be a mixture of K.E. and P.E.

As the total energy is the same at any point we can pick what's easiest to derive an equation for it.

So at zero displacement.

\begin{align*}
T.E. &= K.E. \\
&= \dfrac{1}{2}mv_{max}^2 \\
&= \dfrac{1}{2}mA^2\omega^2 \\
\end{align*}

\spec{ recall and use $E = \dfrac{1}{2}mA^2\omega^2$ to solve problems}

We can look at example problem to see how this works.

\begin{example}

A mass of 8kg is attached to a spring with a spring constant of $5 Nm^{-1}$ on a smooth horizontal surface.
It is pulled back a distance of 20 cm and then released.

Calculate the total energy of the system, the time period of oscillation and the velocity of the mass when the displacement is 10 cm.

\end{example}

\textbf{Answer}

One way to approach this question is by using our equation for the energy stored in a spring.

\[
P.E = \dfrac{1}{2}kx^2
\]

\[
P.E = \dfrac{1}{2} \times 5 \times .2^2 = 0.1J
\]

We also know that all the energy at this point is in the form of potential as the mass is not moving so this is the total energy of the system and will not change.

For the time period we can first calculate the angular frequency using our energy equation.

\begin{align*}
E = \dfrac{1}{2}mA^2\omega^2 &= 0.1 J\\
\therefore \omega &= \sqrt{\dfrac{2 E}{m A^2}}\\
&= \sqrt{\dfrac{2 \times 0.1}{8 \times 0.2^2}}\\
&= 0.79 rads^{-1}\\
\omega &= \dfrac{2 \pi}{T} \\
\therefore T &= 2 \pi = 8.0 seconds \\
\end{align*}

When the mass has a displacement of 10cm, the total energy will be the kinetic and potential energy.

\begin{align*}
T.E. = P.E + K.E. &= 0.1J \\
 \dfrac{1}{2} \times 5 \times .1^2 + K.E. &= 0.1J\\
 0.025 + K.E. &= 0.1J  \\
 \implies K.E. &= 0.075J \\
 \dfrac{1}{2}mv^2 &= 0.075J \\
 v &= 0.14 ms^{-1} \\
 \end{align*}

\pagebreak

\spec{ distinguish between free, damped and forced oscillations}

A free oscillator is one which is set in motion and left to oscillate without any external forces or damping.

e.g. an ideal pendulum with no friction.

A forced oscillation is one where an external periodic force is applied.

e.g. pushing a child on a swing.

Damping is where frictional forces remove energy from the system and the oscillations die down.

This can be increased intentionally for example adding thick oil to car suspension to prevent you bouncing around every time the car hits a bump.

If a system is lightly damped it will gradually come to rest after a number of cycles.

An important case is critical damping where the system comes to rest without overshooting and in the shortest possible time.

\spec{ recall how the amplitude of a forced oscillation changes at and around the natural frequency of a system
and describe, qualitatively, how damping affects resonance.}

Every system has a natural frequency of oscillation which will depend on its physical properties. In the case of a mass on a spring it will depend on the mass and spring constant but more complex examples e.g. washing machines or the millennium bridge will also have natural frequencies.

If we apply a driving force to an oscillator, the effect it has will depend on the amplitude of the driving force and its frequency compared to the natural frequency of the system.

For a simple example try holding a spring with a mass on the end of it and move your hand up and down rhythmically.

If you move your hand very slowly the spring won't stretch or contract and the mass will simply follow the movement of your hand. So the amplitude will be the same as the amplitude of the driving frequency.

As you start to increase the frequency of your hand movement the spring will start to stretch and contract and the mass will move more than your hand. So the amplitude of oscillation increases.

Once the frequency of your hand is close to the natural frequency you are adding energy at just the right point in the cycle to increase the amplitude with each cycle and the amplitude gets very large. We call this resonance.

If we know move our hand with a very high frequency, the spring stretches and compresses but the mass doesn't much have time to accelerate hence the amplitude starts to get very small.

\begin{itemize}
\item Low frequency driving force, amplitude of oscillator = amplitude of driver.
\item Driving frequency around natural frequency, resonance - very high amplitude oscillations.
\item High frequency driving force, low amplitude oscillations.
\end{itemize}

In the real world we need to consider damping which will affect all systems to some degree. Damping will come from frictional forces and will remove energy from the system every cycle. If the energy added by the driving force is more than is removed by damping then the amplitude will increase. As the amplitude increases, the energy removed each cycle will also increase until a maximum amplitude is reached.

From this it is clear that the more damping there is, the lower the amplitude.

Another feature is that as we have more damping, the frequency at which resonance occurs is less. You can think of this as damping slowing down the system and so reducing the natural frequency (although strictly speaking the natural frequency refers to an undamped system).

This is all expressed in the following graph where I have used the damping ratio $\zeta$ as a measure of how much damping there is. Its not on the syllabus, you will only need to know qualitatively how damping affects resonance e.g. light damping and heavy damping but I though it would be good to introduce you to another Greek letter.

\begin{tikzpicture}
  \begin{axis}[
    % labels
    tick label style={font=\scriptsize},
    xlabel={Driving Frequency},
    ylabel={Amplitude / Driving Amplitude},
    ytick={0,1,2,3,4,5},
    xtick={0,1,2,3},
    xticklabels={0,$\omega_\mathrm{f}$,,},
    %
    % plot lines property
    no markers,
    line width=0.3pt,
    cycle list={{black,solid}},
    %
    % dominio 2D
    samples=200,
    smooth,
    domain=0:2.5,
    xmin=0, xmax=2.5,
    ymin=0, ymax=5.0,
    %
    % canvas dimensions
    width=12cm, height=12cm
    ]

    % horizontal help line
    \draw[help lines] (axis cs:0,1) -- (axis cs:2.5,1);
    % vertical help line
    \draw[help lines] (axis cs:1,0) -- (axis cs:1,5);

    % draw curves
    \addplot {1/sqrt((1-x^2)^2+4*0.05^2*x^2)};
    \addplot {1/sqrt((1-x^2)^2+4*0.10^2*x^2)};
    \addplot {1/sqrt((1-x^2)^2+4*0.20^2*x^2)};
    \addplot {1/sqrt((1-x^2)^2+4*0.30^2*x^2)};
    \addplot {1/sqrt((1-x^2)^2+4*0.40^2*x^2)};
    \addplot {1/sqrt((1-x^2)^2+4*0.50^2*x^2)};
   % \addplot {1/sqrt((1-x^2)^2+4*1.00^2*x^2)};
   % \addplot {1/sqrt((1-x^2)^2+4*2.00^2*x^2)};

    % draw maximum curve
    \addplot[dashed,domain=0:0.99] {1/sqrt(1-x^4)};

    %  curve labels
    \node[small dot,pin=30:{$\zeta_0=0{,}05$}] at
      (axis cs:1.10,4.22) {};
    \node[small dot,pin=30:{$\zeta_0=0{,}10$}] at
      (axis cs:1.10,3.29) {};
    \node[small dot,pin=30:{$\zeta_0=0{,}20$}] at
      (axis cs:1.10,2.05) {};
    \node[small dot,pin=30:{$\zeta_0=0{,}30$}] at
      (axis cs:1.20,1.19) {};
    \node[small dot,pin=30:{$\zeta_0=0{,}40$}] at
      (axis cs:1.28,0.83) {};
    \node[small dot,pin=30:{$\zeta_0=0{,}50$}] at
      (axis cs:1.50,0.51) {};

   % \node[small dot,pin=210:{$\zeta_0=1{,}00$}] at
      (axis cs:0.52,0.79) {};
   % \node[small dot,pin=210:{$\zeta_0=2{,}00$}] at
      (axis cs:0.60,0.40) {};
  \end{axis}
\end{tikzpicture}

\begin{itemize}
\item At very low driving frequencies the amplitude is the same as the driving amplitude.
\item With low damping we get a sharp resonance peak.
\item More damping creates a broader resonance peak.

\item Increasing damping reduces the amplitude at all frequencies.

\item The resonance peak shifts lower with increased damping.

\item As the frequency gets very high the amplitude will tend to zero regardless of damping or driving amplitude.

\end{itemize}

\end{document}

%sagemathcloud={"latex_command":"latexmk -xelatex -f -g -bibtex -synctex=1 -interaction=nonstopmode '11-oscillations.tex'"}


% ******************************* Thesis Appendix A ****************************
\chapter{Appendix B: An introduction to neural networks}\label{sec:primer-nn}

\ifpdf
    \graphicspath{{Appendix2/Figs/Raster/}{Appendix2/Figs/PDF/}{Appendix2/Figs/}}
\else
    \graphicspath{{Appendix2/Figs/Vector/}{Appendix2/Figs/}}
\fi

This appendix chapter provides background at a more elementary level than
\vref{sec:bg-rnn}. Its goal is to sufficiently educate readers unfamiliar with
recurrent neural networks such that the remainder of our work can be
understood.

\section{Neurons: the basic computation unit}

Neurons are the basic abstraction which are combined together to form
neural networks. A \emph{neuron} is a parametric model of a function $f : \RR^D \to
\RR$ from its $D$-dimensional input $\x$ to its output $y$. Our neurons will be
defined as
\begin{equation}
    f(\x) \coloneqq \sigma( \langle \vec{w}, \x \rangle)
\end{equation}
which can be viewed as an inner product with \emph{weights} $\vec{w}$ to
produce an \emph{activation} $z \coloneqq \langle \vec{w}, \x \rangle
\in \RR$ which is then squashed to a bounded domain by a non-linear
\textbf{activation function} $\sigma : \RR \to [L, U]$. This is visually
depicted in \cref{fig:nn-single}, which also makes apparent the
interpretation of weight $w_i$ as the sensitivity of the output $y$ to the
input $x_i$.

\begin{figure}[tb]
    \centering
    \input{Appendix2/Figs/nn-single.pdf_tex}
    \caption{A single neuron first computes an activation $z$ and then passes it through an activation function $\sigma(\cdot)$}
    \label{fig:nn-single}
\end{figure}

\section{Feedforward neural networks}

Multiple neurons may share inputs and have their outputs concatenated together
to form a \emph{layer} modelling a multivariate functions $f :
\RR^{D_\text{in}} \to \RR^{D_\text{out}}$. Multiple layers can then
be composed together to form a \emph{feedforwd neural network}.

\begin{figure}[tb]
    \centering
    \input{Appendix2/Figs/nn-ffw.pdf_tex}
    \caption{Graph depiction of a feedforward neural network with $2$ hidden layers}
    \label{fig:nn-ffw}
\end{figure}

Although a single hidden layer is theoretically sufficient for a universal
function approximator \citep{Cybenko1993}, the number of hidden units to
guarantee reported theoretical bounds are usually infeasibly large. Instead,
recent work in \emph{deep learning} has shown that deep models which contain
many hidden layers can achieve strong performance across a variety of
tasks \citep{Bengio2011}.

The improved modeling capacity gained by composing multiple layers is due to
the composition of multiple non-linear activation functions.
In fact, it is easy to show that removing activation functions would make
a deep network equivalent to a single matrix transform: let $\W_{l,l+1}$
denote the weights between layers $l$ and $l+1$. The original neural network
computes the function
\begin{equation}
    \sigma\left(
        \W_{L,L-1} \sigma \left(
            \W_{L-1,L-2}\cdots \sigma \left(
                \W_{2,1} \x
            \right) \cdots
        \right)
    \right)
\end{equation}
After removing the activation functions $\sigma$, we are left with
\begin{equation}
    \W_{L,L-1} \W_{L-1,L-2}\cdots \W_{2,1} \x
    = \x
    = \tilde{\W} \x
\end{equation}
where $\tilde{\W} = \left(\prod_{i=1}^{L-1} \W_{i,i+1} \right)$
is a matrix transform computing the same function as the neural network with
activation functions removed.

\section{Recurrent neural networks}

While feedforward neural networks provide a flexible model for approximating
arbitrary functions, they require a fixed-dimension input $\x$ and hence
cannot be directly applied to sequential data $\x = (\x_t)_{t=1}^T$ where $T$ may
vary.

A naive method for extending feedforward networks would be to independently
apply a feedforward network to compute $\y_t = f(\x_t \vec{\theta})$ at each timestep
$1 \leq t \leq T$. However, this approach is only correct when each output
$\y_t$ depends only on the input at the current time $\x_t$ and is independent of
all prior inputs $\{\x_k\}_{k < t}$. This assumption is false in musical data:
the current musical note usually is highly dependent on the sequence of notes
leading up to it.

This shortcoming motivates \emph{recurrent neural networks} (RNNs), which
generalize feedforward networks by introducing time-delayed recurrent
connections between hidden layers (Elman networks \citep{elman1990finding}) or
from the output layers to the hidden layers (Jordan networks
\citep{jordan1997serial}). Mathematically, a linear Elman-type RNN is a discrete time
dynamical system commonly parameterized as:
\begin{equation}
 \left.\begin{aligned}
          \h_t &=& \W_{xh} \sigma_{xh} \left( \x_t \right) + \W_{hh} \sigma_{hh} \left( \h_{t-1} \right)\\
          \y_t &=& \W_{hy} \sigma_{hy} \left( \h_t \right)
       \end{aligned}
 \right\}
 \qquad \text{Linear Elman-Type RNN Dynamics}
\end{equation}
\begin{align}
\end{align}
where $\sigma_{\cdot \cdot}(\cdot)$ are activation functions acting
element-wise and $\theta = \{ \W_{xh}, \W_{hh}, \W_{hy}\}$ are the learned
parameters. \cref{fig:nn-rnn} provides a graphical illustration of such a
network. Notice that apart from the edges between hidden nodes, the network is
identical to a regular feedforward network (\cref{fig:nn-ffw}).

\begin{figure}[tb]
    \centering
    \input{Appendix2/Figs/nn-rnn.pdf_tex}
    \caption{Graph representation of an Elman-type RNN.}
    \label{fig:nn-rnn}
\end{figure}

To apply the RNN over an input sequence $\x$, the activations of the hidden
states are first initialized to an initial value $\h \in \RR^{D_{h}}$. Next,
for each timestep $t$ the hidden layer activations are computed using the
current input $\x_t$ and the previous hidden state activations $\h_{t-1}$.
This motivates an alternative perspective on RNNs as a template consisting
of a feedforward network with inputs $\{\x_t, \h_{t-1}\}$ (see
\vref{fig:rnn-elman}) replicated across time $t$.


Here we are going to talk about the \textbf{Dropout} method for training DL model. This is a kind of training algorithm but also a kind of parameter regularization method in some sense.

\section{Bagging and Dropout}
\begin{itemize}
	\item \textbf{Bootstrap} comes from a common saying "pull up by your own bootstraps", which means use your won resource to solve problem.
	\item \textbf{Bagging} (short for \textbf{bootstrap aggregating}) is a technique for reducing generalization error by combining several models. The idea is to train several different models separately, then have all of the models vote on the	output for test examples.This is an example of a general strategy in machine learning called	model averaging. Techniques employing this strategy are known as ensemble methods.
	\item The reason that model averaging works is that different models will usually
	not make all the same errors on the test set.
\end{itemize}
Neural networks reach a wide enough variety of solution points that they can often benefit from model averaging even if all of the models are trained on the same dataset. Differences in random initialization, random selection of minibatches,
differences in hyperparameters, or different outcomes of non-deterministic implementations of neural networks are often enough to cause different members of the ensemble to make partially independent errors.

\textbf{Dropout} provides a computationally inexpensive but	powerful method of regularizing a broad family of models. To a first approximation, dropout can be thought of as a method of making bagging practical for ensembles of very many large neural networks. Specifically, dropout trains the ensemble consisting of all sub-networks that can be formed by removing non-output units from an underlying base network.

\newpage
\begin{figure}
	\includegraphics[width=\linewidth]{figures/dropout}
	\caption{Dropout trains an ensemble consisting of all sub-networks that can be
		constructed by removing non-output units from an underlying base network.}
	\label{fig:dropout}
\end{figure}

\begin{wrapfigure}{r}{0.4\textwidth}
	\vspace{-20pt}
	\begin{center}
		\includegraphics[width=0.38\textwidth]{figures/dropoutexample}\label{dropoutexample}
	\end{center}
	\vspace{-20pt}
	\vspace{-10pt}
\end{wrapfigure}

To perform forward propagation with dropout, we randomly sample a vector $\bm \mu$ with one entry for each input
or hidden unit in the network. The entries of $\bm \mu$ are binary and are sampled independently from each other. The probability of each entry being 1 is a hyperparameter, usually $0.5$ for the hidden layers and $0.8$ for the input. Each unit in the network is multiplied by the corresponding mask, and then forward propagation continues through the rest of the	network as usual. This is equivalent to randomly selecting one of the sub-networks from figure and running forward propagation through it.

Recall that to learn with bagging, we define $k$ different models, construct $k$ different datasets by sampling from the training set with replacement, and then train model $i$ on dataset $i$. Dropout aims to approximate this process, but with an exponentially large number of neural networks. Specifically, to train with dropout, we use a minibatch-based learning algorithm that makes small steps, such as stochastic gradient descent. Each time we load an example into a minibatch, we randomly sample a different binary mask to apply to all of the input and hidden units in the network.

Dropout training is not quite the same as bagging training. In the case of
bagging, the models are all independent. In the case of dropout, the models share
parameters, with each model inheriting a different subset of parameters from the
parent neural network. This parameter sharing makes it possible to represent an
exponential number of models with a tractable amount of memory. In the case of
bagging, each model is trained to convergence on its respective training set. In the
case of dropout, typically most models are not explicitly trained at all-usually,
the model is large enough that it would be infeasible to sample all possible sub-
networks within the lifetime of the universe. Instead, a tiny fraction of the possible
sub-networks are each trained for a single step, and the parameter sharing causes
the remaining sub-networks to arrive at good settings of the parameters. These
are the only differences. Beyond these, dropout follows the bagging algorithm.

\section{Mathematical form}

Suppose $\mathbf X=(x_1,...,x_n)$ is dataset, $\mathbf Y=(y_1,...,y_n)$ is label set. Our goal is find the parameter $\Theta=(\theta_1,...,\theta_j)$ of a neural network $f$ minimize
\begin{equation}\label{1}
\mathcal L(\Theta)=\sum_{i=1}^n L(f(x_i;\Theta),y_i)
\end{equation}
where $L(\cdot, \cdot)$ is the loss function like $L^2$ normal or cross-entropy, $f=f^j\circ\cdots\circ f^1$ is the neural network. $f^s(\mathbf x)=g(\theta_s \begin{pmatrix}
\mathbf x\\ 1
\end{pmatrix})$.

Recall the traditional Mini-Batch SGD training algorithm, for every step, we choose a subset $B_t\subset \{1,...,n\}$ and update the parameters $\Theta$ by:
\begin{equation}
\Theta^{t+1} = \Theta^t + \eta_t\nabla \sum_{i \in B_t} L(f(x_i ;\Theta) ,y_i).
\end{equation}

So the dropout means that you need to change our model just in this step, and get:
\begin{equation}
\tilde{f}^{j} = \theta^j \circ P^j \circ g^j \circ \tilde f^{j-1},
\end{equation}
with $P^j$ is a diagonal matrix for
\begin{equation}
P^j_{ii} \simeq P,
\end{equation}
with
\begin{equation}
P\{x = 0 \} = 1-\gamma_j, \quad P\{x = 1 \} = \gamma_j.
\end{equation}
and then get the update
\begin{equation}
\Theta^{t+1} = \Theta^{t} - \eta \nabla_{\Theta} \sum_{i \in B_t} L(\tilde f^J(x_i, \Theta), y_i).
\end{equation}

\subsection{Prediction}
When we finish training, we get a solution $\Theta^\#=(\theta_1^\#,...,\theta_n^\#)$.
And we can simply set the solution of \eqref{1} is $\Theta = (\gamma_1 \theta_1^\#,...,\gamma_n \theta_n^\# )$. This called \textbf{weight scaling inference rule}.
%To make the weight of each layer should be scaling to make the expectation be same as dropout.
We use a linear problem to explain this.
Suppose our problem is minimize
\begin{equation}\label{7}
\mathcal{L}(\mathbf{W})=\|\mathbf{Y}-\mathbf W\mathbf X\|_F^2
\end{equation}
so the solution of \eqref{7} $\mathbf W^*$ satisfy vanish gradient, i.e.
\begin{equation}\label{gradient}
(\mathbf Y-\mathbf W^*\mathbf X)\mathbf{X}^T=0
\end{equation}
%If $\mathbf X$ is full rank, we can get
%\begin{equation}
%\mathbf Y-\mathbf W^*\mathbf X=0
%\end{equation}

While we use dropout technology, actually we solve a series of  problems like
\begin{equation}\label{10}
\mathcal{L}(\mathbf{W})=\|\mathbf{Y}-\mathbf W(\mathbf D\mathbf X)\|_F^2
\end{equation}
where $\mathbf D=\text{diag}(\mathbf d)$, $\mathbf d$ is a random matrix with $P(d_{i}=1)=\gamma,P(d_{i}=0  )=1-\gamma$.

Its gradient is
\begin{equation}
g_t=g(\mathbf D_t)=(\mathbf Y-\mathbf W(\mathbf D_t\mathbf X))(\mathbf D_t\mathbf X)^T
\end{equation}
So the iteration step is
\begin{equation}\label{iter}
\mathbf W^{t+1}=\mathbf W^t-\eta_t g_t
\end{equation}
%We also want
%\begin{equation}
%g(\mathbf D)=\mathbf Y-\mathbf W(\mathbf D\otimes\mathbf X)=0
%\end{equation}
If \eqref{iter} is convergent, the expectation of $g$ should equals zero, so we have
\begin{eqnarray}
0=\mathbb E_{\mathbf D}(\mathbf Y-\mathbf W(\mathbf D\mathbf X))(\mathbf D\mathbf X)^T\\
=\gamma\mathbf Y\mathbf X^T-\mathbf W \gamma^2\mathbf{X}\mathbf X^T+(\gamma^2-\gamma)\mathbf W\text{diag}(\mathbf X\mathbf X^T)\\
\approx\gamma\mathbf Y\mathbf X^T-\mathbf W \gamma^2\mathbf{X}\mathbf X^T
\end{eqnarray}
So we get the solution of \eqref{10} $\mathbf W^\#$ satisfy
\begin{equation}
(\mathbf Y-\mathbf W^\#(\gamma\mathbf X))\mathbf X^T\approx 0
\end{equation}

Combine with \eqref{gradient} we can set $\mathbf W^*=\gamma\mathbf W^\#$

In other opinion,
\begin{align}
&\mathbb E_{\mathbf D}[\|\mathbf Y-\mathbf W(\mathbf D\mathbf X) \|^2_F]\\
=&\mathbb E_{\mathbf D}\text{tr}((\mathbf Y-\mathbf W \mathbf D \mathbf X)(\mathbf Y-\mathbf W \mathbf D \mathbf X)^T)\\
=&\mathbb E_{\mathbf D}\text{tr}(\mathbf Y \mathbf Y^T-\mathbf W \mathbf D \mathbf X \mathbf Y^T-\mathbf Y(\mathbf W \mathbf D \mathbf X)^T+(\mathbf W \mathbf D \mathbf X)(\mathbf W \mathbf D \mathbf X)^T)\\
=&\text{tr}(\mathbf Y \mathbf Y^T-\gamma\mathbf W \mathbf X \mathbf Y^T-\gamma \mathbf Y(\mathbf W  \mathbf X)^T+\gamma^2(\mathbf W \mathbf X)(\mathbf W  \mathbf X)^T+(\gamma-\gamma^2)\mathbf W \text{diag}(\mathbf X\mathbf X^T)\mathbf W^T )\\
=&\text{tr}((\mathbf Y-\gamma \mathbf W  \mathbf X)(\mathbf Y-\gamma \mathbf W \mathbf X)^T)+(\gamma-\gamma^2)\text{tr}(\mathbf W \text{diag}(\mathbf X\mathbf X^T)\mathbf W^T )\\
=&\|\mathbf Y-\gamma \mathbf W  \mathbf X\|^2_F+(\gamma-\gamma^2)\text{tr}( \text{diag}(\mathbf X\mathbf X^T)\mathbf W^T\mathbf W )\\
=&\|\mathbf Y-\gamma \mathbf W  \mathbf X\|^2_F+(\gamma-\gamma^2)\|(\mathbf X\mathbf X^T)^{\frac 12}\mathbf W\|_F^2\\
\end{align}
Let $ \tilde{\mathbf W}=\gamma \mathbf W$, the loss function became to
\begin{equation}
\|\mathbf Y-\tilde{\mathbf W} \mathbf X\|^2_F+\dfrac {(1-\gamma)}\gamma\|\tilde{\mathbf W}\text{diag}(\mathbf X\mathbf X^T)^{\frac 12}\|_F^2
\end{equation}
Thus, you can think dropout for linear problem is to add a regularization term $\dfrac {(1-\gamma)}\gamma\|\tilde{\mathbf W}\text{diag}(\mathbf X\mathbf X^T)^{\frac 12}\|_F^2$.

\begin{remark}
	The analyze of linear problem only try to explain why \textbf{weight scaling inference rule} is work. It doesn't mean dropout can solve the linear problem better than not using dropout. In my(Yuyan) opinion, dropout takes the advantages of bagging method, but disadvantages of solving the problem inexactly. In most case the benefit of bagging method is greater than inexact solution. This is means the dropout has larger loss on training dataset but high accuracy on test dataset.
\end{remark}

\subsection{Inverted Dropout}

When we implement dropout, usually we use inverted dropout instead.

In tensorflow, the annotation of tf.nn.dropout shows "With probability $\gamma$, outputs the input element scaled up by $\dfrac{1}{\gamma}$, otherwise outputs 0. The scaling is so that the expected sum is unchanged."

\subsection{Numerical Experiment for Linear System}
If we consider the linear problem as a machine learning problem. We say a method is work if we get higher accuracy on test data by using this method.  Meanwhile if we consider the linear problem as a optimal problem, we say a method is work if we get less value of loss function on training dataset.

\begin{table}
	\begin{tabular}{c|c|c}
		&with dropout& without dropout\\\hline
		accuracy&86.3\%&86.3\%\\\hline
		loss &0.407&0.385\\\hline
	\end{tabular}
\end{table}


\subsection{Free groups}\label{subsec:free_groups}

\begin{definition}\label{def:free_monoid}
  We associate with every set \( A \) its \term{free monoid} \( F(A) \coloneqq (A^*, \cdot) \), where \( A^* \) is the \hyperref[def:formal_language/kleene_star]{Kleene star} and \( \cdot \) is \hyperref[def:formal_language/concatenation]{concatenation}.

  Denote by \( \iota_A: A \to F(A) \) the canonical inclusion function, which sends every member of \( A \) into the corresponding single-symbol word in the \hyperref[def:free_monoid]{free monoid} \( F(A) \).
\end{definition}
\begin{proof}
  Concatenation is clearly associative and the empty word \( \varepsilon \) is an \hyperref[def:monoid]{identity} under concatenation.
\end{proof}

\begin{theorem}[Free monoid universal property]\label{thm:free_monoid_universal_property}
  We associate with every set \( A \) its \hyperref[def:formal_language/kleene_star]{Kleene star} \( A^* \). Denote by \( \iota_A: A \to F(A) \) the canonical inclusion function, which sends every member of \( A \) into the corresponding single-symbol word in \( A^* \).

  The Kleene star \( A^* \) with \hyperref[def:formal_language/concatenation]{concatenation} is the unique up to an isomorphism monoid that satisfies the following \hyperref[rem:universal_mapping_property]{universal mapping property}:
  \begin{displayquote}
    For every monoid \( M \) and every function \( f: A \to M \), there exists a unique monoid homomorphism \( \widetilde{f}: A^* \to M \) such that the following diagram commutes:
    \begin{equation}\label{eq:thm:free_monoid_universal_property/diagram}
      \begin{aligned}
        \includegraphics[page=1]{output/thm__free_monoid_universal_property.pdf}
      \end{aligned}
    \end{equation}
  \end{displayquote}

  Via \fullref{rem:universal_mapping_property}, \( (\Anon*)^* \) becomes \hyperref[def:category_adjunction]{left adjoint} to the \hyperref[def:concrete_category]{forgetful functor}
  \begin{equation*}
    U: \cat{Mon} \to \cat{Set}.
  \end{equation*}
\end{theorem}
\begin{proof}
  For every function \( f: A \to M \), define the monoid homomorphism
  \begin{equation*}
    \begin{aligned}
      &\widetilde{f}: A^* \to M, \\
      &\widetilde{f}(x_1 x_2 \ldots x_n) \coloneqq f(x_1) \cdot f(x_2) \cdot \ldots \cdot f(x_n)
    \end{aligned}
  \end{equation*}
  obtained by applying the monoid operation \( \cdot \) recursively to the pointwise image
  \begin{equation*}
    f(x_1) f(x_2) \ldots f(x_n)
  \end{equation*}
  of the word
  \begin{equation*}
    x_1 x_2 \ldots x_n.
  \end{equation*}

  The homomorphism \( \widetilde{f} \) is uniquely determined by the action of \( f \) on single-symbol words.
\end{proof}

\begin{definition}\label{def:abstract_reduction_system}\mcite[def. 1.1.2]{Book1993}
  Fix an arbitrary \hyperref[def:set]{set} \( A \) and an \hyperref[rem:first_order_formula_conventions/infix]{infix} \hyperref[def:binary_relation]{binary relation} \( \to \) on \( A \). We call the operation \( \to \) a \term{reduction relation}, and the pair \( (A, \to) \) --- an \term{abstract reduction system}.

  In the case where \( A \) is the \hyperref[def:formal_language/kleene_star]{Kleene star} of some set, we also call \( (A, \to) \) a \term{string rewriting system}.

  \begin{thmenum}
    \thmitem{def:abstract_reduction_system/relations}\mcite[not. 1.1.1]{Book1993} We also introduce the following auxiliary relations:
    \begin{itemize}
      \item \( \reloset n \to \) is the \( n \)-th \hyperref[def:binary_relation/composition]{iterated composition} of \( \to \), where \( n \) is a nonnegative integer.

      \item \( \reloset + \to \) is the \hyperref[def:relation_closures/transitive]{transitive closure} of \( \to \).

      \item \( \reloset {*} \to \) is the \hyperref[def:relation_closures/reflexive]{reflexive closure} of \( \reloset + \to \).

      \item \( {\leftrightarrow} \) is the \hyperref[def:relation_closures/symmetric]{symmetric closure} of \( \to \).

      \item \( \reloset + \leftrightarrow \) is the \hyperref[def:relation_closures/transitive]{transitive closure} of \( \leftrightarrow \).

      \item \( \reloset {*} \leftrightarrow \) is the \hyperref[def:relation_closures/reflexive]{reflexive closure} of \( \reloset + \leftrightarrow \), the smallest equivalence relation containing \( \to \).
    \end{itemize}

    \thmitem{def:abstract_reduction_system/hierarchy}\mcite[def. 1.1.3]{Book1993} In analogy with \fullref{def:arborescence/ancestry}, if \( x \reloset {*} \to y \), we say that \( y \) is a \term{descendant} of \( x \) and that \( x \) is an \term{ancestor} of \( y \). If \( x \to y \), we say that \( y \) is an \term{immediate descendant}, or a \term{successor}, especially when considering \( (A, \to) \) as a \hyperref[def:quiver]{quiver}.

    We say that an element is \term{reducible} if it has an immediate descendant and \term{irreducible} otherwise.

    \thmitem{def:abstract_reduction_system/equivalent}\mcite[def. 1.1.3]{Book1993} We say that \( x \) and \( y \) are \term{equivalent} if \( x \reloset {*} \leftrightarrow y \). This is a stronger condition than \( x \reloset {*} \to y \) and \( y \reloset {*} \to x \). Indeed, the latter corresponds to \hyperref[def:quiver_connectedness/weak]{weak connectedness} of quivers and equivalence corresponds to \hyperref[def:quiver_connectedness/strong]{strong connectedness}.

    \thmitem{def:abstract_reduction_system/normal_form}\mcite[def. 1.1.5]{Book1993} Given an \hyperref[def:equivalence_relation]{equivalence relation} \( \cong \), if \( x \cong y \) and \( y \) is irreducible, we say that \( y \) is a \term{normal form} of \( x \) modulo \( \cong \), and that \( x \) is \term{normalizable} modulo \( \cong \).

    Without further context, we assume that \( \cong \) is the equivalence relation \( \reloset {*} \leftrightarrow \).

    If \( x \) has a unique normal form, we denote it by \( \op{nf} \).
  \end{thmenum}
\end{definition}

\begin{definition}\label{def:abstract_rewriting_convergence}
  Let \( (A, \to) \) be an \hyperref[def:abstract_reduction_system]{abstract reduction system}. We will list several conditions ensuring existence and uniqueness of \hyperref[def:abstract_reduction_system/normal_form]{normal forms}.

  \begin{thmenum}
    \thmitem{def:abstract_rewriting_convergence/confluence}\mcite[def. 1.1.6]{Book1993} We call the system \term{confluent} if, whenever \( x \) and \( y \) are descendants of \( w \), they have a common descendant \( z \). That is,
    \begin{equation}\label{eq:def:abstract_rewriting_convergence/confluence/diagram}
      \begin{aligned}
        \includegraphics[page=1]{output/def__abstract_rewriting_convergence.pdf}
      \end{aligned}
    \end{equation}

    If there always exists an \hi{immediate descendant} \( z \), we say that the system is \term{locally confluent}.

    \thmitem{def:abstract_rewriting_convergence/noetherian}\mcite[def. 1.1.9]{Book1993} We call the system \term{noetherian} or \term{strongly normalizable} if there exists no \term{infinitely ascending sequence}
    \begin{equation*}
      x_1 \to x_2 \to x_3 \to \cdots.
    \end{equation*}

    This is dual to \hyperref[def:well_founded_relation]{well-foundedness}.

    \thmitem{def:abstract_rewriting_convergence/convergent}\mcite[def. 1.1.11]{Book1993} Finally, we call the system \term{convergent} if it is both confluent and noetherian.

    By \fullref{thm:noetherian_rewriting_system_local_confluence}, this is equivalent to the system being \hi{locally} confluent and noetherian.
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:noetherian_rewriting_system_local_confluence}\mcite[thm. 1.1.13]{Book1993}
  A \hyperref[def:abstract_rewriting_convergence/noetherian]{noetherian}  \hyperref[def:abstract_reduction_system]{abstract reduction system} is \hyperref[def:abstract_rewriting_convergence/confluence]{confluent} if and only if it is \hyperref[def:abstract_rewriting_convergence/confluence]{locally confluent}.
\end{proposition}

\begin{proposition}\label{thm:convergent_reduction_system_normal_forms}
  In a \hyperref[def:abstract_rewriting_convergence/convergent]{convergent} \hyperref[def:abstract_reduction_system]{abstract reduction system}, every element has a unique \hyperref[def:abstract_reduction_system/normal_form]{normal form}.
\end{proposition}
\begin{proof}
  Let \( (A, \to) \) be a convergent abstract reduction system.

  \SubProof{Proof of existence} Suppose that \( x \) has no normal form. Hence, it is reducible and has an immediate descendant \( x_1 \). Proceeding by \hyperref[rem:natural_number_recursion]{natural number recursion}, we can build a sequence
  \begin{equation*}
    x \to x_1 \to x_2 \to \cdots.
  \end{equation*}

  Such a sequence cannot exist, however, since the system is \hyperref[def:abstract_rewriting_convergence/noetherian]{noetherian}.

  \SubProof{Proof of uniqueness} Suppose that both \( y \) and \( z \) are normal forms of \( w \).

  Since the system is confluent, there exists an element \( z \) that is a descendant of both \( x \) and \( y \). But \( x \) and \( y \) are irreducible, therefore \( x = y \).
\end{proof}

\begin{definition}\label{def:free_group}
  Let \( A \) be an arbitrary set. We will now construct the \term{free group} \( F(A) \) of \( A \).

  Let \( \anon \) be an arbitrary set not in \( A \). Consider the \hyperref[def:disjoint_union]{disjoint union} \( U \coloneqq A \times \set{ +, - } \), whose members we will denote by \( a^+ \) and \( a^- \).

  On the \hyperref[def:formal_language/kleene_star]{Kleene star} \( U \), define the relation \( \to \) to hold for \( x \to y \) if there exists \( a \in A \) and words \( p \) and \( s \) such that \( y = ps \) and either \( x = p a^+ a^- s \) or \( x = p a^+ a^- s \).

  Then \( (U, \to) \) is a \hyperref[def:abstract_rewriting_convergence/convergent]{convergent} \hyperref[def:abstract_reduction_system]{abstract reduction system}. The \term{free group} \( F(A) \) is the set of \hyperref[def:abstract_reduction_system/hierarchy]{irreducible elements} in \( U \), with the operation \( v \star w \coloneqq \op{nf}(vw) \) that gives the \hyperref[def:abstract_reduction_system/normal_form]{normal form} of the concatenated string \( vw \).

  The identity is the empty word and the inverse \( v^{-1} \) of \( v \) can be characterized recursively as
  \begin{equation*}
    v^{-1} \coloneqq \begin{cases}
      \varepsilon &v = \varepsilon \\
      u^{-1} a^-  &v = u a^+ \\
      u^{-1} a^+  &v = u a^-
    \end{cases}
  \end{equation*}

  The canonical inclusion is
  \begin{equation*}
    \begin{aligned}
      &\iota_A: A \to F(A) \\
      &\iota_A(a) \coloneqq a^+.
    \end{aligned}
  \end{equation*}

  Compare this definition to free abelian groups defined in \fullref{def:free_semimodule}.
\end{definition}
\begin{defproof}
  We need to prove that the system \( (U, \to) \) is convergent. Then from \fullref{thm:convergent_reduction_system_normal_forms} it will follow that every word has a unique normal form.

  \SubProofOf[def:abstract_rewriting_convergence/confluence]{local confluence} Fix two reductions \( w \to x \) and \( w \to y \).

  If \( x = y \), define \( z \coloneqq x \). Otherwise, let \( p_x \) and \( s_x \) be words such that \( w = p_x a^+ a^- s_x \) and \( x = p_x s_x \) (the case where \( w = p_x a^- a^+ s_x \) is analogous). Similarly, let \( p_y \) and \( s_y \) be words such that \( w = p_y b^+ b^- s_y \) and \( y = p_y s_y \).

  Without loss of generality, suppose that \( p_x \) is a subword of \( p_y \). Then there exists a word \( v \) such that
  \begin{equation*}
    w = p_x a^+ a^- v b^+ b^- s_y.
  \end{equation*}

  Then both \( x \) and \( y \) are reducible to \( z \coloneqq p_x v s_y \).

  This shows that the rewriting system is locally confluent.

  \SubProofOf[def:abstract_rewriting_convergence/noetherian]{noetherianity} All words are finite, and a reduction always removes two symbols. Hence, there cannot exist an infinite reduction path.
\end{defproof}

\begin{theorem}[Free group universal property]\label{thm:free_group_universal_property}
  The \hyperref[def:free_group]{free group} \( F(A) \) is the unique up to an isomorphism group that satisfies the following \hyperref[rem:universal_mapping_property]{universal mapping property}:
  \begin{displayquote}
    For every group \( G \) and every function \( f: A \to G \), there exists a unique group homomorphism \( \widetilde{f}: F(A) \to G \) such that the following diagram commutes:
    \begin{equation}\label{eq:thm:free_group_universal_property/diagram}
      \begin{aligned}
        \includegraphics[page=1]{output/thm__free_group_universal_property.pdf}
      \end{aligned}
    \end{equation}
  \end{displayquote}

  Via \fullref{rem:universal_mapping_property}, \( F \) becomes \hyperref[def:category_adjunction]{left adjoint} to the \hyperref[def:concrete_category]{forgetful functor}
  \begin{equation*}
    U: \cat{Grp} \to \cat{Set}.
  \end{equation*}
\end{theorem}
\begin{proof}
  The free group operation is more complicated than the free monoid operation, however the proof of the universal mapping property is identical to the property in \fullref{thm:free_monoid_universal_property}.
\end{proof}

\begin{definition}\label{def:group_presentation}\mcite[314]{Knapp2016BasicAlgebra}
  Let \( A \) be a \hyperref[def:set]{plain set} with \hyperref[def:free_group]{free group} \( F(A) \) and let \( R \subseteq F(A) \) be a set of words in \( F(A) \). Denote by \( N(R) \) the \hyperref[thm:normal_subgroup_equivalences]{normal subgroup} \hyperref[def:first_order_generated_substructure]{generated} by \( R \).

  Then we define the group with \term{generators} \( A \) and \term{relators} \( R \) as
  \begin{equation}\label{eq:def:group_presentation/presentation}
    \braket{ A \mid R } \coloneqq F(A) / N(R).
  \end{equation}

  If \( R = \varnothing \), there are no relators, and we use the following notation for the free group:
  \begin{equation}\label{eq:def:group_presentation/free}
    \braket{ A } \coloneqq F(A)
  \end{equation}

  Note that we use similar notation compared to generated subgroups defined in \fullref{def:group/submodel}. The former defines a new group operation from scratch, while the latter uses an existing group operation and is restricted by this operation.

  If, for a given group \( G \) and subsets \( A \) and \( R \) of \( G \), we have
  \begin{equation*}
    G \cong \braket{ A \given R },
  \end{equation*}
  we call the pair \( (A, R) \) a \term{presentation} of \( G \).

  We say that \( G \) is \term{finitely generated} if both \( A \) and \( R \) are \hyperref[def:set_finiteness]{finite sets}; if only \( R \) is finite, we call \( G \) \term{finitely presented}.

  Compare this to \hyperref[def:module_presentation]{module presentations} and \hyperref[def:algebra_presentation]{algebra presentations}.
\end{definition}

\begin{proposition}\label{thm:group_presentation_existence}\mcite[prop. 7.7]{Knapp2016BasicAlgebra}
  Every group has at least one \hyperref[def:group_presentation]{presentation}.

  Compare this to \fullref{thm:module_presentation_existence} and \fullref{thm:algebra_presentation_existence}.
\end{proposition}
\begin{proof}
  Fix an arbitrary group \( G \) and let \( A \coloneqq U(G) \) be the underlying set.

  By \fullref{thm:free_group_universal_property}, there exists a unique group homomorphism \( \varphi: F(A) \to G \) such that
  \begin{equation*}
    U(\varphi) \bincirc \iota_A = \id_A.
  \end{equation*}

  By \fullref{thm:def:group/kernel_is_normal_subgroup}, the kernel \( \ker \varphi \) is a normal subgroup of \( A \), hence by \fullref{thm:quotient_group_universal_property},
  \begin{equation*}
    G = \varphi(F(A)) \cong F(A) / \ker \varphi = \braket{ S \mid \ker \varphi }.
  \end{equation*}
\end{proof}

\begin{definition}\label{def:group_free_product}\mcite[323]{Knapp2016BasicAlgebra}
  The \term{free product} of a nonempty pairwise disjoint family of groups \( \seq{ X_k }_{k \in \mscrK} \) with \hyperref[def:group_presentation]{presentations} \( \braket{S_k \mid R_k}, k \in \mscrK \) is the group with presentation
  \begin{equation*}
    \Ast_{k \in \mscrK} X_k \coloneqq \braket*{ \bigcup_{k \in \mscrK} S_k \given[\Big] \bigcup_{k \in \mscrK} R_k }.
  \end{equation*}

  If the constituent groups are not disjoint, we may instead use \hyperref[def:disjoint_union]{disjoint unions} as
  \small
  \begin{equation*}
    \Ast_{k \in \mscrK} X_k \coloneqq \braket*{ \set[\Big]{ (k, x) \given k \in \mscrK \T{and} x \in S_k } \given[\Big] \set[\Big]{ (k, x_1) (k, x_2) \ldots (k, x_n) \colon k \in \mscrK \T{and} x_1 x_2 \ldots x_n \in R_k } }.
  \end{equation*}
  \normalsize

  For every index \( m \in \mscrK \), we define the canonical embedding
  \begin{equation*}
    \begin{aligned}
       &\iota_m: X_m \to \Ast_{k \in \mscrK} X_k \\
       &\iota_m(x) \coloneqq (m, x).
    \end{aligned}
  \end{equation*}
\end{definition}

\begin{definition}\label{def:cyclic_group}
  For a singleton alphabet \( \set{ a } \), we define the \term{infinite cyclic group}
  \begin{equation*}
    C_\infty \coloneqq \braket{ a }
  \end{equation*}
  and, for positive integers \( n \), the \term{finite cyclic group} of \term{order} \( n \) as
  \begin{equation*}
    C_n \coloneqq \braket{ a \given a^n }.
  \end{equation*}

  We use the same notation independent of \( a \) because all cyclic groups of the same order are obviously \hyperref[def:group/homomorphism]{isomorphic}.

  Given an ambient group \( G \) and some element \( g \in G \), the \term{cyclic subgroup} of \( g \) is the cyclic group isomorphic to the \hyperref[def:group/submodel]{generated subgroup} of \( G \).

  As shown in \fullref{thm:cyclic_group_isomorphic_to_integers_modulo_n}, cyclic groups are isomorphic to certain groups of integers, however it is still useful to have cyclic groups as a separate concept.
\end{definition}

\begin{definition}\label{def:group_order}
  The \term{order} \( \ord(G) \) of a group \( G \) is its \hyperref[thm:cardinality_existence]{cardinality}.

  The \term{order} \( \ord(x) \) of a member \( x \) of a group is the smallest positive integer \( n \) such that \( x^n = e \), i.e. order of the \hyperref[def:cyclic_group]{cyclic subgroup} \hyperref[def:group/submodel]{generated} by \( x \).
\end{definition}

\begin{proposition}\label{thm:def:group_order}
  \hyperref[def:group_order]{Group orders} have the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:group_order/divides} For finite groups, the order of a group element divides the order of the group.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:group_order/divides} Follows from \fullref{thm:lagranges_theorem_for_groups}.
\end{proof}

\begin{proposition}\label{thn:product_of_cyclic_groups}
  The \hyperref[def:monoid_direct_product]{direct product} \( C_m \times C_n \) of two \hyperref[def:cyclic_group]{cyclic groups} is cyclic if and only if \( m \) and \( n \) are \hyperref[def:coprime_elements]{coprime}.
\end{proposition}
\begin{proof}
  The \hyperref[def:group_order]{order} of the element \( (a, e) \) is \( m \) and the order of \( (e, a) \) is \( n \). The order of
  \begin{equation*}
    (a, a) = (a, e) \cdot (e, a)
  \end{equation*}
  is the least common multiple of \( m \) and \( n \), which equals \( mn \) if and only if \( m \) and \( n \) are coprime.
\end{proof}

\begin{proposition}\label{thm:cyclic_groups_are_simple}
  Groups of \hyperref[def:prime_number]{prime} \hyperref[def:group_order]{order} are \hyperref[def:group/simple]{simple}.
\end{proposition}
\begin{proof}
  Let \( N \) be a proper normal subgroup of \( G \), where \( \ord(G) \) is a prime number.

  From \fullref{thm:lagranges_theorem_for_groups} it follows that \( \ord(N) \) divides \( p \). But \( p \) is prime, hence \( N \) is either the trivial group or the full group.

  Therefore, \( G \) is a simple group.
\end{proof}

\begin{proposition}\label{thm:group_categorical_limits}
  \hfill
  \begin{thmenum}
    \thmitem{thm:group_categorical_limits/product} The \hyperref[def:discrete_category_limits]{categorical product} of the family \( \seq{ G_k }_{k \in \mscrK} \) in the category \hyperref[def:group/category]{\( \cat{Grp} \)} of groups is their \hyperref[def:monoid_direct_product]{monoid direct product} \( \prod_{k \in \mscrK} G_k \).

    \thmitem{thm:group_categorical_limits/coproduct} The \hyperref[def:discrete_category_limits]{categorical coproduct} of the family \( \seq{ G_k }_{k \in \mscrK} \) in the category \hyperref[def:monoid/category]{\( \cat{CMon} \)} of \hi{commutative} monoids is their \hyperref[def:group_free_product]{group free product} \( \Ast_{k \in \mscrK} G_k \).

    Compare this to the commutative case discussed in \fullref{thm:monoid_categorical_limits/coproduct}.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:group_categorical_limits/product} Follows from \fullref{thm:monoid_categorical_limits/product}.

  \SubProofOf{thm:group_categorical_limits/coproduct}  Let \( (A, \alpha) \) be a \hyperref[def:category_of_cones/cocone]{cocone} for the discrete diagram \( \seq{ G_k }_{k \in \mscrK} \). We want to define a group homomorphism \( l: \Ast_{k \in \mscrK} G_k \to A \) such that, for every \( m \in \mscrK \),
  \begin{equation*}
    \alpha_m(x) = l_A(\iota_m(x)).
  \end{equation*}

  This suggests the definition
  \begin{equation*}
    l_A\parens[\Big]{ (k_1, x_1) (k_2, x_2) \ldots (k_n, x_n) } \coloneqq \alpha_{k_1}(x_1) \cdot \alpha_{k_2}(x_k) \cdot \ldots \cdot \alpha_{k_n}(x_n).
  \end{equation*}
\end{proof}


\section{The one-pass distributed algorithm}

The essence of the distributed strategy is to achieve almost perfect
parallelism, by splitting the input matrix into several smaller
matrices called \emph{jobs}. \\

\[
A^{m \times n} =
\begin{bmatrix}
A_1^{m \times c_1} \mid A_2^{m \times c_2} \mid \cdots \mid A_k^{m \times c_k}
\end{bmatrix}
\suchthat \sum_{i=1}^k c_i = n
\]
\\

A subset of these smaller matrices or \emph{jobs} is assigned to each
node in the cluster, depending on their capabilities; the
objective is to assign matrices that fit into the node's RAM
memory. Each node will calculate the SVD factorization of the
submatrices assigned, but merging those results into a single
SVD approximation that covers all the input data it received. At the
end, a global merge step across all the nodes is performed, giving the
global SVD approximation for original matrix $A$. The
\cref{alg:svd-dist} describes the overall distributed algorithm: \\

\begin{algorithm}
  \label{alg:svd-dist}
  \caption{Distributed-SVD: Distributed SVD for LSI (global)}
%
  \setstretch{1.35}
  \SetKwInOut{Input}{Input}
  \SetKwInOut{Output}{Output}
  \DontPrintSemicolon
%
    \Input{Truncation factor $k$, queue of jobs $A= [A_1, A_2, \dots ]$}
%
    \Output{Matrices $U^{m \times k}$ and $\Sigma^{k \times k}$,
      from the SVD decomp. of $A$}
%
    \For {\textbf{all} (node $i$ in cluster)}
    {
      $B_i \gets \text{subset of the queue of jobs } [A_1,A_2,\dots]$ \;
%
      $P_i = (U_i,\Sigma_i) \gets \func{SVD-Node}(k,B_i)$ \;
    }
    $(U,\Sigma) \gets \func{Reduce}(\func{Merge-SVD},[P_1,P_2,\dots])$ \;
%
    return $(U, \Sigma)$ \;
\end{algorithm}
\hfill

The first important detail from the algorithm just shown, is that we
are not calculating the matrix $V$ from the SVD factorization, how
come! Such detail is explained at the end of the last section. For the
moment, let us just say that such matrix is not required for our
purposes. \\

We can also observe the map-reduce pattern in this algorithm, with the map
part being the iteration done over $p$ nodes (in parallel); and the
reduce part being the final merge of those partial results. The
\cref{alg:svd-dist-node} describes the part done inside each node.

\begin{algorithm}
  \label{alg:svd-dist-node}
  \caption{SVD-Node: Distributed SVD for LSI (node)}
%
  \setstretch{1.35}
  \SetKwInOut{Input}{Input}
  \SetKwInOut{Output}{Output}
  \DontPrintSemicolon
%
  \Input{Truncation factor $k$, queue of jobs $A_1,A_2,\dots$}
%
  \Output{Matrices $U^{m \times k}$ and $\Sigma^{k
        \times k}$, from the SVD  of $[A_1,A_2,\dots]$}
%
  $P = (U,\Sigma) \gets 0^{m \times k} 0^{k \times k}$ \;
%
  \For {each job $A_i$}
  {
    $\prim{P} = (\prim{U},\prim{\Sigma}) \gets \func{Basecase-SVD}(k,A_i)$ \;
%
    $P = (U^{m \times k},\Sigma^{k \times k}) \gets \func{Merge-SVD}(k, P, \prim{P})$ \;
  }
%
  return $(U,\Sigma)$ \;
\end{algorithm}
\hfill

It is important to realize that the iteration in this
\cref{alg:svd-dist-node} is done serially, but that the procedure
$\func{Basecase-SVD}$ that resolves the SVD of a
matrix that fits in memory (base case), internally may exploit the
multicore or vectorial capabilities of the node computer. This
procedure serves as a black box SVD calculator, and \Rehurek mentions
at least two algorithms which can be plugged on its place: \\

\begin{enumerate}
\item The Lanczos algorithm as implemented by SVDLIBC (\cite{svdlibc}),
  which in turn is based on SVDPACKC written by Berry et al
  (\cite{svdpackc}), which in turn is based on its Fortran77
  predecessor SVDPACK (\cite{svdpack}). All of them  ultimately based
  on seminal paper by  Berry \cite{berry92} (which in turn comes from
  his PhD thesis \cite{berry91}). \\
\item A custom stochastic algorithm based on the work of Halko et al
  (see \cite{halko11}).
\end{enumerate}
\hfill

For the scope of this project, we considered appropriate to focus only
on the Lanczos based algorithm; as that is essentially what we
described in the previous chapter. In that sense, the work of \Rehurek
is interesting because by using the divide and conquer strategy for
the SVD problem, he is leveraging on the decades of research and
numerical accuracy of the work done by Berry et al. At the same time,
his key contribution becomes the procedure $\func{Merge-SVD}$, which
we will describe in further sections. \\


\section{A Refined Relational Database}\label{sec:database}

Next, we use bounded refinements to develop a library
for relational algebra, which we use to enable generic,
type safe database queries.
%
A relational database stores data in \emph{tables},
that are a collection of \emph{rows}, which in turn
are \emph{records} that represent a unit of data
stored in the table.
The tables's \textit{schema} describes the types of
the values in each row of the table.
For example, the table in Figure~\ref{fig:moviedb} organizes
information about movies, and has the schema:
%
\begin{code}
 Title:String, Dir:String, Year:Int, Star:Double
\end{code}

\begin{table}[h]
\captionsetup{justification=centering}
\caption{Example entries for Movies Database.}
\label{fig:moviedb}
\centering
$$
\begin{tabular}{| l | l| r | r |}
  \hline
  \textbf{Title} & \textbf{Director} & \textbf{Year} & \textbf{Star} \\
  \hline
  ``Birdman'' & ``I\~{n}\'{a}rritu''   & 2014 & 8.1\\
  ``Persepolis''  & ``Paronnaud'' & 2007 & 8.0 \\
  \hline
\end{tabular}
$$
\end{table}

First, we show how to write type safe extensible
records  that represent rows, and use them to
implement database tables~(\S~\ref{subsec:records}).
%
Next, we show how bounds let us specify type
safe relational operations and how they may be
used to write safe database queries~(\S~\ref{subsec:relational}).

\subsection{Rows and Tables}\label{subsec:records}

We represent the rows of a database with dictionaries,
which are maps from a set of keys to values.
In the sequel, each key corresponds to a column and
the mapped value corresponds to a valuation of the column
in a particular row.

\mypara{A dictionary} @Dict <r> k v@ maps a key @x@ of
type @k@ to a value of type @v@ that satisfies the property @r x@
%
% \NV CUT
\begin{code}
  type Range k v = k -> v -> Bool

  data Dict k v <r :: Range k v> = D {
      dkeys :: [k]
    , dfun  :: x:{k | x SetMem elts dkeys} -> v <r x>
    }
\end{code}
%
Each dictionary @d@ has a domain @dkeys@
\ie the list of keys for which @d@ is defined
and a function @dfun@ that is defined only on
elements @x@ of the domain @dkeys@.
%
For each such element @x@, @dfun@ returns a value that satisfies the
property @r x@.

\mypara{Propositions about the theory of sets} can be decided
efficiently by modern SMT solvers. Hence we use such
propositions within refinements as demonstrated in chapter~\ref{chapter:tool}.
%
The measures (logical functions) @elts@ and @keys@
specify the set of keys in a list and a dictionary
respectively:
%
\begin{code}
  elts        :: [a] -> Set a
  elts ([])   = Set_emp
  elts (x:xs) = {x} Set_cup elts xs

  keys        :: Dict k v -> Set k
  keys d      = elts (dkeys d)
\end{code}

\mypara{Domain and Range of dictionaries.}
%
In order to precisely define the domain (\eg columns) and range (\eg values)
of a dictionary (\eg row), we define the following aliases:
%
% NV CUT
\begin{code}
  type RD k v <dom :: Dom k v, rng :: Range k v>
    = {v:Dict <rng> k v | dom v}

  type Dom k v = Dict k v -> Bool
\end{code}
%
We may instantiate @dom@ and @rng@ with predicates that precisely describe
the values contained with the dictionary.
%
For example,
%
\begin{code}
  RD < \d -> keys d = {"x"}, \k v-> 0 < v> String Int
\end{code}
%
%% instantiating @dom@ with
%% \hbox{@\d -> keys d = {"x"}@}
%% and @rng@ with the predicate
%% \hbox{@\k v -> k = "x" => v > 0@}
%
describes dictionaries with a single field @"x"@
whose value (as determined by @dfun@) is stricly
greater than 0.
%
We will define schemas by appropriately
instantiating the abstract refinements
@dom@ and @rng@.

\mypara{An empty dictionary} has an empty domain
and a function that will never be called:
%
\begin{code}
  empty   :: RD <emptyRD, rFalse> k v
  empty   = D [] (\x -> error "calling empty")

  emptyRD = \d -> keys d == Set_emp
  rFalse  = \k v -> false
\end{code}
%% %
%% Thus, @empty@'s range satisfies \textit{any} predicate -- that is,
%% it satisfies @false@.

\mypara{We define singleton maps} as dependent pairs
@x := y@ which denote the mapping from @x@ to @y@:
%
\begin{code}
  data P k v <r :: Range k v>
    = (:=) {pk :: k, pv :: v <r pk>}
\end{code}
%
Thus, @key := val@ has type \hbox{@P <r> k v@} only if
@r key val@.

\paragraph{A dictionary may be extended} with a singleton
binding (which maps the new key to its new value).
%
\begin{code}
  (+=)   :: bind:P<r> k v
         -> dict:RD<pTrue, r> k v
         -> RD <addKey (pk bind) dict, r> k v

  (k := v) += (D ks f)
         = D (k:ks) (\i -> if i == k then v else f i)

  addKey = \k d d' -> keys d' == {k} Set_cup keys d
  pTrue  = \_ -> True
\end{code}
%
Thus, @(k := v)  += d@ evaluates to
a dictionary @d'@ that extends @d@
with the mapping from @k@ to @v@.
%
The type of @(+=)@ constrains the new binding @bind@,
the old dictionary @dict@ and the returned value to have
the same range invariant @r@.
%
The return type states that the output dictionary's
domain is that of the domain of @dict@ extended by
the new key @(pk bind)@.

\mypara{To model a row in a table} \ie a schema,
we define the unrefined (Haskell) type @Schema@,
which is a dictionary mapping @String@s, \ie the
names of the fields of the row, to elements of
some universe @Univ@ containing @Int@, @String@
and @Double@.
%
(A closed universe is not a practical restriction;
most databases support a fixed set of types).
%
\begin{code}
  data Univ   = I Int | S String | D Double

  type Schema = RD String Univ
\end{code}

\mypara{We refine Schema} with concrete instantiations
for @dom@ and @rng@, in order to recover precise
specifications for a particular database. For example,
@MovieSchema@ is a refined @Schema@ that describes the
rows of the Movie table in Figure~\ref{fig:moviedb}:
%
%
\begin{code}
  type MovieSchema = RD <md, mr> String Univ

  md = \d -> keys d={"year","star","dir","title"}
  mr = \k v ->
       (k = "year"  => isI v && 1888 < toI v)
     && (k = "star"  => isD v && 0 <= toD v <= 10)
     && (k = "dir"   => isS v)
     && (k = "title" => isS v)

  isI (I _)   = True
  isI _       = False

  toI       :: {v: Univ | isI v} -> Int
  toI (I n) = n
\end{code}
%
The predicate @md@ describes the \emph{domain} of the movie schema,
restricting the keys to exactly @"year"@, @"star"@, @"dir"@, and @"title"@.
%
The range predicate @mr@ describes the types of the values in the schema:
%
a dictionary of type @MovieSchema@ must map
@"year"@ to an @Int@,
@"star"@ to a @Double@,
and @"dir"@ and @"title"@ to @String@s.
%
The range predicate may be used to impose additional constraints on the values
stored in the dictionary.
%
For instance, @mr@ restricts the year to be not only an integer but
also greater than @1888@.
%
%%%Because refinements in \toolname are drawn from a decidable logic,
%%%refining the range with logical predicates comes ``for free''.
%%%%
%%%A more heavyweight dependent type system, on the other hand, would
%%%require the programmer to manually thread proofs of these range
%%%predicates throughout the code.

\mypara{We populate the Movie Schema} by extending the
empty dictionary with the appropriate pairs of fields and
values. For example, here are the rows from the table
in Figure~\ref{fig:moviedb}
%
\begin{code}
  movie1, movie2 :: MovieSchema
  movie1 = ("title" := S "Persepolis")
        += ("dir"   := S "Paronnaud")
        += ("star"  := D 8)
        += ("year"  := I 2007)
        += empty

  movie2 = ("title" := S "Birdman")
        += ("star"  := D 8.1)
        += ("dir"   := S "Inarritu")
        += ("year"  := I 2014)
        += empty
\end{code}
%
Typing @movie1@ (and @movie2@) as @MovieSchema@
boils down to proving
%
that @keys movie1 = {"year", "star", "dir", "title"}@
and that each key is mapped to an appropriate value
as determined by @mr@.
%
For example, declaring @movie1@'s year to be @I 1888@
or even misspelling @"dir"@ as @"Dir"@
will cause the @movie1@ to become ill-typed.
%
As the (sub)typing relation depends on logical
implication (unlike in @HList@ based approaches
\cite{heterogeneous}) key ordering \emph{does not}
affect type-checking;
%
in @movie1@ the star field is added before the
director, while @movie2@ follows the opposite order.
%%%On the contrary, with dependent types, proving that two heterogeneous
%%%collections constructed with different ordering have the same type
%%%would require an additional (manually-supplied) equality proof.

\mypara{Database Tables} are collections of rows,
\ie collections of refined dictionaries.
%
We define a type alias @RT s@ (Refined Table) for
the list of refined dictionaries from the field
type @String@ to the @Univ@erse.
%
\begin{code}
  type RT (s :: {dom::TDom, rng::TRange})
    = [RD <s.dom, s.rng> String Univ]

  type TDom   = Dom   String Univ
  type TRange = Range String Univ
\end{code}
%
For brevity we pack both the domain and the range
refinements into a record @s@ that describes the
schema refinement; \ie each row dictionary has
domain @s.dom@ and range @s.rng@.

For example, the table from Figure~\ref{fig:moviedb}
can be represented as a type @MoviesTable@ which
is an @RT@ refined with the domain and range @md@
and @mr@ described earlier (\S~\ref{subsec:records}):
%
\begin{code}
  type MoviesTable = RT {dom = md, rng = mr}

  movies :: MoviesTable
  movies = [movie1, movie2]
\end{code}

\subsection{Relational Algebra}\label{subsec:relational}

Next, we describe the types of the relational algebra operators
which can be used to manipulate refined rows and tables.
For space reasons, we show the \emph{types} of the basic
relational operators; their (verified) implementations
can be found online~\cite{liquidhaskellgithub}.
%
\begin{code}
  union   :: RT s -> RT s -> RT s
  diff    :: RT s -> RT s -> RT s
  select  :: (RD s -> Bool) -> RT s -> RT s
  project :: ks:[String] -> RTSubEqFlds ks s
          -> RTEqFlds ks s
  product :: ( Disjoint s1 s2, Union s1 s2 s
             , Range s1 s, Range s2 s)
          => RT s1 -> RT s2 -> RT s
\end{code}

\mypara{{Union} and {diff}} compute the union
and difference, respectively of the (rows of) two tables.
%
The types of @union@ and @diff@ state that the
operators work on tables with the same schema
@s@ and return a table with the same schema.

\mypara{{select}} takes a predicate @p@
and a table @t@ and filters the rows of @t@
to those which that satisfy @p@.
%
The type of @select@ ensures that @p@ will
not reference columns (fields) that are
not mapped in @t@, as the predicate @p@
is constrained to require a dictionary
with schema @s@.

\mypara{{project}} takes
a list of @String@ fields @ks@
and a table @t@ and projects
exactly the fields @ks@ at
each row of @t@.
%
@project@'s type states that for
any schema @s@, the input table
has type @RTSubEqFlds ks s@
\ie its domain should have at
least the fields @ks@ and the
result table has type @RTEqFlds ks s@,
\ie its domain has exactly the elements @ks@.
%
\begin{code}
  type RTSubEqFlds ks s = RT s{dom = \z -> elts ks Set_sub  keys z}

  type RTEqFlds ks s    = RT s{dom = \z -> elts ks = keys z}
\end{code}
%
The range of the argument and the result tables
is the same and equal to @s.rng@.

\mypara{{product}} takes two tables
as input and returns their (Cartesian)
product.
%
It takes two Refined Tables with schemata
@s1@ and @s2@ and returns a Refined Table
with schema @s@. Intuitively, the output
schema is the ``concatenation'' of the input
schema; we formalize this notion using bounds:
%
(1)~@Disjoint s1 s2@ says the domains of
    @s1@ and @s2@ should be disjoint,
%
(2)~@Union s1 s2 s@ says the domain of @s@
    is the union of the domains of @s1@ and @s2@,
%
(3)~@Range s1 s@ (\resp @Range s2 s2@) says
    the range of @s1@ should imply the result
    range @s@; together the two imply the output
    schema @s@ preserves the type of each key in
    @s1@ or @s2@.
%
\begin{code}
  bound Disjoint s1 s2 = \x y ->
    s1.dom x => s2.dom y => keys x Set_cap keys y == Set_emp

  bound Union s1 s2 s = \x y v ->
    s1.dom x => s2.dom y
             => keys v == keys x Set_cup keys y
             => s.dom v

  bound Range si s = \x k v ->
    si.dom x => k SetMem keys x => si.rng k v => s.rng k v
\end{code}

%% We note that none of these restrictions can be
%% expressed using unbounded Abstract Refinement Types.

Thus, bounded refinements  enable the precise
typing of  relational algebra operations.
They let us describe precisely when union,
intersection, selection, projection and products
can be computed, and let us determine, at compile
time the exact ``shape'' of the resulting tables.

% \subsection{Data Base Queries}\label{subsec:dbclient}

\mypara{We can query Databases} by writing functions
that use the relational algebra combinators.
%
For example, here is a query that returns the
``good'' titles -- with more than 8 stars --
from the @movies@ table
\footnote{More example queries can be found online~\cite{liquidhaskellgithub}}
%
\begin{code}
  good_titles = project ["title"] $ select (\d ->
                  toDouble (dfun d $ "star") > 8
                ) movies
\end{code}
%
%% The above query selects the movies that have more
%% than 8 stars and projects only their @"title"@ field.
%

Finally, note that our entire library -- including
records, tables, and relational combinators -- is
built using vanilla Haskell \ie without \emph{any}
type level computation.
%
All schema reasoning happens at the granularity of
the logical refinements. That is if the refinements
are erased from the source, we still have a well-typed
Haskell program but of course, lose the safety
guarantees about operations (\eg ``dynamic'' key lookup)
never failing at run-time.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\chapter{OPTIMIZATION}
\section{Introduction.}Definition: an act, process, or methodology of making something (as a design, system, or decision) as fully perfect, functional, or effective as possible. \\
\textbf{In Business:}\\
Finding an alternative with the most cost effective or highest achievable performance under the given constraints, by maximizing desired factors and minimizing undesired ones.Practice of optimization is restricted by the lack of full information, and the lack of time to evaluate what information is available.\\
\textbf{In Science:}\\
In mathematics, computer science and operations research, mathematical optimization (alternatively, optimization or mathematical programming) is the selection of a best element (with regard to some criteria) from some set of available alternatives.In the simplest case, an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. The generalization of optimization theory and techniques to other formulations comprises a large area of applied mathematics.More generally, optimization includes finding "best available" values of some objective function given a defined domain (or a set of constraints), including a variety of different types of objective functions and different types of domains.\\
\textbf{In Engineering:}\\
Engineering Optimization is the subject which uses optimization techniques to achieve design goals in engineering. It is sometimes referred to as design optimization.\\
Let us take an example.\\
Suppose we want to buy a car. If we want the car with maximum luxury and minimum cost, this means we are considering both the factors at the time of purchase. So, this is optimization.If we are looking for the car only with maximum luxury, then this is 	maximization.And if, we are looking for a car with minimum cost, then this is minimization.\\
\section{Types of optimization.}
\subsection{Single-objective optimization.}The term single-objective corresponds to the fact that we have just one objective to optimize i.e. either maximize it or minimize it.\\Considering it the other way single-objective optimization can also we viewed as a combination of various objectives into an overall objective,where this overall objective is optimized. So, here the individual identity of the objectives is lost and there is no guarantee on the performance of the individual objectives as they are not considered separately. So, for these reasons we go for multi-objective optimization.\\For example,getting a job. Here,if we consider just that the job should offer maximum salary and no other constraints and objectives, then we go for maximization. Such cases are single-objective based.
\subsection{Multi-objective optimization.}
\subsubsection{4.2.2.1 Real-life situation.}In real-life we face situations where increasing value of one parameter decreases the value of the other parameter. So, actually there is a contradictory situation. For example, we want to buy a house. The house should be well-furnished with high-class facilities and at the same time the purchasing price should be less. So, here we find an inverse situation where the comparison is between luxury vs. cost, where we want increased luxury and decreased cost.\\
But, in actual practice it is not possible. So, we have to make a compromise between luxury and cost so that to a certain limit both are optimal. We need to find the optimal luxury at the optimal price. Some objectives can be in competition, so a simultaneous minimization is not possible, but only a trade-off among them.\\
So, such situations have multiple objectives associated like luxury and cost in the above example, so we study multiple-objective optimization.
\subsubsection{4.2.2.2 Principles of MOO.}A multi-objective optimization problem involves a number of objective functions which are to be either minimized or maximized. As in the single-objective optimization problem, the multi-objective optimization problem usually has a number of constraints which any feasible solution (including the optimal solution) must satisfy. In the following, we state the multi-objective optimization problem (MOOP) in its general form:\\
				Minimize/Maximize $f_{m}$(x),                    m=1, 2,…, M;\\
				subject to    $g_{j}$ (x) ≥ 0,	                j=1, 2,...,J;\\
							  $h_{k}$ (x) = 0.	                K=1, 2,..., K;\\
\subsubsection{4.2.2.3 Example of MOO.}Minimize ZDT1 Function:  \\
It is one of benchmark function which is commonly use to test the performance of a algorithm for MOO.
The ZDT1 function has a convex Pareto-optimal front. The objective functions are: \\
$f_{1}$(x) = x1\\
%%$f_{2}$(x) = g(x) 1- \sqrt{$x_1$/g(x)\} \\
Where g(x) is defined as: g(x )= 1+ 9($ $$\sum_{i=2}^{n}$$ $  $x_{i}$)/ (n-1)\\
In this ZDT1 function, thirty design variables xi were chosen (n=30). Each design variable ranged in value from 0 to 1. The Pareto-optimal front appears when g = 1.0.\\
\subsubsection{4.2.2.4 Dominated and Non-dominated set.  }
\subsubsection{4.2.2.5 MOO approaches.}
\subsubsection{4.2.2.6 Basic MOPSO algorithm.}
\subsubsection{4.2.2.7 Applications of MOO.}


\subsection{Free Damped Vibrations ($b > 0$)}
We need to consider three cases where the discriminant $\Delta = b^2 - 4mk$ is positive, zero, and negative.

\input{./higherOrder/freeVibrs/overdamped.tex}
\input{./higherOrder/freeVibrs/critically_damped.tex}
\input{./higherOrder/freeVibrs/underdamped.tex}

\ifodd\includeHigherOrderExamples\input{./higherOrder/freeVibrs/freeDamped_example.tex}\fi


% Copyright 2018-2021 Melvin Eloy Irizarry-Gelpí
\setcounter{chapter}{5}
\chapter{Curved Mirrors}
%
In this experiment you will learn about the properties of images produced by curved mirrors.
%
\section{Preliminary}
%
A mirror reflects light. Mirrors can be flat or curved. Flat mirrors are commonly found in bathrooms. Curved mirrors can either be \textbf{concave} or \textbf{convex}. As you will see, a \textbf{concave mirror} can produce a \textbf{real image}, in the sense that the image can be projected on a screen.

Light is emitted from an \textbf{object}. This light travels through space and then bounces of a mirror, potentially producing an \textbf{image} somewhere in space. There is a relationship between the \textbf{distance between the object and the mirror} ($d_{\text{O}}$) and the \textbf{distance between the image and the mirror} ($d_{\text{I}}$). This relationship involves a property of the mirror called the \textbf{focal length} ($f$):
\begin{equation}
    \frac{1}{d_{\text{O}}} + \frac{1}{d_{\text{I}}} = \frac{1}{f}
\end{equation}
This is a curious relation because it involves the inverse quantities, and not the direct quantities themselves. Solving for the inverse image distance provides another way of writing this relation:
\begin{equation}
    \frac{1}{d_{\text{I}}} = -\frac{1}{d_{\text{O}}} + \frac{1}{f}
\end{equation}
If you measure $d_{\text{O}}$ and $d_{\text{I}}$, then the chart with $1/d_{\text{I}}$ in the vertical axis and $1/d_{\text{O}}$ in the horizontal axis should have a \textbf{linear shape}. The \textbf{slope} of the line should be close to $-1$, and the value of the \textbf{intercept} should be close to the inverse of the focal length of the mirror used.

In practice it is easier to measure \textbf{position} than to measure distance. You have three things: the \textbf{object}, the \textbf{image}, and the \textbf{mirror}. Thus you have three positions along the track: the position $x_{\text{O}}$ of the object, the position $x_{\text{I}}$ of the image, and the position $x_{\text{M}}$ of the mirror. The distance between the object and the mirror is simply
\begin{equation}
    d_{\text{O}} = \left\vert x_{\text{O}} - x_{\text{M}} \right\vert
\end{equation}
Similarly, the distance between the image and the mirror is given by
\begin{equation}
    d_{\text{I}} = \left\vert x_{\text{I}} - x_{\text{M}} \right\vert
\end{equation}
The absolute value is used because distance should be positive.
%
\subsection{Magnification}
%
Curved mirrors produce images that look very different from the object that is emitting the light. Some differences can involve a change in \textbf{orientation}. For example, left and right can be switched, as is common with plane mirrors. Other differences can be a change in \textbf{size}. That is, the size of an image can be smaller or larger than the object. There is a quantity called \textbf{magnification} that is used to describe how the size of an image compares to the size of the object. Let $h_{\text{I}}$ be the \textbf{height of the image}, and $h_{\text{O}}$ be the \textbf{height of the object}. The magnification $m$ of the mirror is given by the following ratio:
\begin{equation}
    m = \frac{h_{\text{I}}}{h_{\text{O}}}
\end{equation}
Note that magnification has no units because it is the ratio of two lengths. There is another definition of the magnification that involves the distances from the mirror to the object and the image:
\begin{equation}
    m = \frac{d_{\text{I}}}{d_{\text{O}}}
\end{equation}
This particular definition is missing a minus sign that is traditionally used to denote a change in orientation. For practical purposes, you can ignore that minus sign.
%
\section{Experiment}
%
The experiment with the concave mirror consisted of recording the position of the mirror, the object, and the screen where the sharp image is projected. You also used a ruler to record the height of the object and the image.
%
\section{Analysis}
%
With the position of the object, mirror, and image you can calculate the distances $d_{\text{O}}$ and $d_{\text{I}}$. Then you can make a chart of $1 / d_{\text{I}}$ versus $1 / d_{\text{O}}$. The expected relation is
\begin{equation}
    \frac{1}{d_{\text{I}}} = -\frac{1}{d_{\text{O}}} + \frac{1}{f}
\end{equation}
The slope of the best-fit line should be:
\begin{equation}
    \text{slope} = -1
\end{equation}
Note that the slope does not have units. The intercept of the best-fit line should be:
\begin{equation}
    \text{intercept} = \frac{1}{\text{focal length}}
\end{equation}
From the intercept you can find an experimental estimate of the focal length of the mirror:
\begin{equation}
    \text{focal length} = \frac{1}{\text{intercept}}
\end{equation}
The expected value for the focal length of the concave mirror used is 20 cm.

With the distances $d_{\text{O}}$ and $d_{\text{I}}$ you can calculate the magnification:
\begin{equation}
    m = \frac{d_{\text{I}}}{d_{\text{O}}}
\end{equation}
You can also calculate the magnification using the measured heights $h_{\text{I}}$ and $h_{\text{O}}$:
\begin{equation}
    m = \frac{h_{\text{I}}}{h_{\text{O}}}
\end{equation}
both of these methods should be (relatively) consistent.
%
\section{My Data}
%
My data consist of ten distinct configurations. As can be seen from Figure \ref{figure.07.chart}, the expected behavior is observed between the inverse distances to the mirror. Further more, the slope and intercept agree with the expected values.

You can also see from Table \ref{table.07.magnification} that the two ways of computing the magnification of the mirror give somewhat consistent results.
%
\section{Your Data}
%
You should have positions and heights for eight distinct configurations.
%
% \newpage
% \section{Your Lab Report}
% %
% Your laboratory report should include the following:
% \begin{itemize}
%     \item A table like Table \ref{table.07.magnification} with the magnification computed in two different ways.
%     \item A chart with $d_{\text{I}}$ in the vertical axis, and $d_{\text{O}}$ in the horizontal axis.
%     \item A chart with $1/d_{\text{I}}$ in the vertical axis, and $1/d_{\text{O}}$ in the horizontal axis. Include the best linear fit, and display the equation. See Figure \ref{figure.07.chart}.
%     \item A table like Table \ref{table.07.results} with your results and the percent differences.
%     \item Answer the following question: which method of calculating the magnification do you think is more accurate?
% \end{itemize}
%
\newpage
\section{Tables}
%
\begin{table}[ht]
    \centering
    \begin{tabular}{r|r|r}
        $x_{\text{O}}$ (cm) & $x_{\text{M}}$ (cm) & $x_{\text{I}}$ (cm) \\
        \hline
        10 & 120 & 96.05 \\
        10 & 112 & 87.35 \\
        10 & 104 & 79.1 \\
        10 & 96 & 70.15 \\
        10 & 88 & 61.35 \\
        10 & 80 & 52.55 \\
        10 & 72 & 43.05 \\
        10 & 64 & 32.5 \\
        10 & 56 & 21.35 \\
        60 & 90 & 36.2 \\
        \hline
    \end{tabular}
    \caption{Raw Position Data}
    \label{table.07.position}
\end{table}
%
\begin{table}[ht]
    \centering
    \begin{tabular}{r|r}
        $h_{\text{O}}$ (cm) & $h_{\text{I}}$ (cm) \\
        \hline
        2 & 0.5 \\
        2 & 0.5 \\
        2 & 0.5 \\
        2 & 0.7 \\
        2 & 0.7 \\
        2 & 0.8 \\
        2 & 0.9 \\
        2 & 1.1 \\
        2 & 1.4 \\
        2 & 3.5 \\
        \hline
    \end{tabular}
    \caption{Raw Height Data}
    \label{table.07.height}
\end{table}
%
\begin{table}[ht]
    \centering
    \begin{tabular}{r|r}
        $d_{\text{O}}$ (cm) & $d_{\text{I}}$ (cm) \\
        \hline
        110 & 23.95 \\
        102 & 24.65 \\
        94 & 24.9 \\
        86 & 25.85 \\
        78 & 26.65 \\
        70 & 27.45 \\
        62 & 28.95 \\
        54 & 31.5 \\
        46 & 34.65 \\
        30 & 53.8 \\
        \hline
    \end{tabular}
    \caption{Distances to the mirror}
    \label{table.07.distance}
\end{table}
%
\begin{table}[ht]
    \centering
    \begin{tabular}{r|r}
        $h_{\text{I}} / h_{\text{O}}$ & $d_{\text{I}} / d_{\text{O}}$ \\
        \hline
        0.25 & 0.22 \\
        0.25 & 0.24 \\
        0.25 & 0.26 \\
        0.35 & 0.30 \\
        0.35 & 0.34 \\
        0.4 & 0.39 \\
        0.45 & 0.45 \\
        0.55 & 0.58 \\
        0.7 & 0.75 \\
        1.75 & 1.79 \\
        \hline
    \end{tabular}
    \caption{Magnification}
    \label{table.07.magnification}
\end{table}
%
\begin{table}[ht]
    \centering
    \begin{tabular}{l|r|r|r}
        & \textbf{Expected Value} & \textbf{Observed Value} & \textbf{P.D.} (\%) \\
        \hline
        \textbf{Slope} & \textminus 1 & \textminus 0.95 & \textminus 4.68 \\
        \textbf{Focal Length} & 20 cm & 20.02 cm & 0.11 \\
        \hline
    \end{tabular}
    \caption{Results}
    \label{table.07.results}
\end{table}
%
\newpage
\FloatBarrier
\section{Figures}
%
\begin{figure}[ht]
    \centering
    \includegraphics[scale=0.74]{image/07-mirrors/chart.pdf}
    \caption{Linear fit}
    \label{figure.07.chart}
\end{figure}
%


% !TEX root = ../main.tex
% chktex-file 21
% chktex-file 46
\section{Conclusion}%
\label{sec:conclusion}

We have now discussed the three main topics of this paper:
\begin{enumerate*}
	\item How the structural properties of graphs can be described via spectral graph theory.
	\item How graphs can be coarsened via the REC algorithm.
	\item How to bound the effects of coarsening via RSS and what this implies for spectral clustering.
\end{enumerate*}

Based on the work we presented, there are two main open questions for future research:
\begin{enumerate*}
	\item The described RSS bound assumes a single application of the REC algorithm.
		For multiple REC applications with a total reduction ratio of $r > \frac{1}{2}$, the RSS bound still needs to be generalized.
	\item Currently the RSS bound has only been applied to the analysis of spectral clustering.
		To make the results more generally applicable, the implications for other graph algorithms, e.g.\ graph convolutional neural networks, are still to be considered.
\end{enumerate*}