\chapter{Cross Correlations}\label{chp:1}

Cross correlation is powerful statistical tool for computing the degree to which two signals are correlated (or similar), and also for computing lags. In this part, we first implement the cross correlation in 1 dimension and apply it to a signal processing task. Then we extend it to 2 dimensions and do some image processing with it.

 After implementing these basic models, we focus on the efficiency of calculating cross correlation. To be specific, large signals and images may result in time consumption. Then there is an alternative way to solve this problem, that is, finding spectral cross correlation with a Fourier transform and inverse Fourier transform. The result shows this new method achieve better performance in scale data.

\section{Normalized Spatial Cross Correlation in 1d}

In signal processing, cross-correlation is a measure of similarity of two series as a function of the displacement of one relative to the other. In this project, we only consider discrete signals rather than continue signals.

For any one dimension signal, we can represent it by a function. For discrete functions f and g, the cross-correlation is defined as\cite{rabiner1975theory}\cite{rabiner1978digital}:

\begin{equation*}
r=\frac{1}{N}
\sum_{i=1}^{i=N}(f(i)-\bar{f})(g(i)-\bar{g})
\end{equation*}

Which computes the correlation between two signals. However, the value of $r$ is somewhat arbitrary because of the variant amplitude of $f$ and $g$, which makes the position of maximum value of $r$ does not represents the offset that makes two signals get maximum similarity. One way around this is to normalize signal with the root-mean-quare. Normalized cross correlation is typically done by subtracting the mean and dividing by the standard deviation. As:

\begin{equation*}
R=\frac{1}{N}
\sum_{i=1}^{i=N}
\frac{(f(i)-\bar{f})(g(i)-\bar{g})}{\sigma _{f}\sigma _{g}}
\end{equation*}

where

\begin{equation*}
{\sigma _{f}}=
\sqrt{\frac{1}{N}\sum_{i=1}^{i=N}(f(i)-\bar{f})^{2}}
\hspace{4em}
{\sigma _{g}}=
\sqrt{\frac{1}{N}\sum_{i=1}^{i=N}(g(i)-\bar{g})^{2}}
\end{equation*}

Source code of one dimensional cross correlation and normalized one dimensional cross correlation are in Appendix \ref{code:1.1_1} and\ref{code:1.1_2} separately.

\section{Signal Offset}

Consider two signals (denote by vector) come from the same source and they are just offset by some time since they are captured by two sensors with constant distance. Cross correlation can be used to find the offset time and distance between two sensors. First find the max value's position of the cross correlation vector computed from two signals. Then find corresponding positions of the two signals and compute the offset of the signal vector. Through the given sample rate and propagation speed, the offset time and distance can be calculated as follow:

\begin{equation*}
\text{offset time}=\frac{\text{offset}}{\text{sample rate}}
\end{equation*}

\begin{equation*}
\text{distance}=\text{(offset time)} * \text{(propagation speed)}
\end{equation*}

In this task, we take two signal files and compute the cross correlation between them. Figure \ref{fig:crr_vis0} shows the result of the cross correlation.

\begin{figure}[h!]
	\centering
		\includegraphics[width=0.45\linewidth]{figures/part1/crr_vis0.eps}
		\caption{Cross correlation of two signals.}
		\label{fig:crr_vis0}
\end{figure}

The x coordinate of maximum correlation is 50081, the length of signal is 176401 and sample rate is 44100. Applying the function above can get the result of offset time equals 50081 and sensor distance equals 378.17 meters.

Source code of signal offset computing is in Appendix \ref{code:1.2}.

\section{Normalized Spatial Cross Correlation in 2d}

Till now, the cross correlation we used can only computer one dimensional signal. But in later work, for comparing images and get the correlation of them smoothly, it is necessary to extend the cross correlation to two dimensions. Similar with signals can be represented by vector, images can represented by matrix. Consider two matrices, t (template) and A (search region), The matrix A will always larger than the matrix t. We can use two nested for-loops to ``lag'' t over A, compute for each ``lag'' the cross-correlation. Normalized cross correlation of two matrices defines as:

\begin{equation*}
R(lag_{x},lag_{y})=
\frac{\sum_{x,y}[A(x,y)-\overline{A_{lag_{x},lag_{y}}})][t(x-lag_{x},y-lag_{y})-\bar{t}]}
{\{\sum_{x,y}[A(x,y)-\overline{A_{lag_{x},lag_{y}}})]^2
	\sum_{x,y}[t(x-lag_{x},y-lag_{y})-\bar{t}]^2
	\}^{0.5}}
\end{equation*}

Where $\bar{t}$ is the mean of t, $\overline{A_{lag_{x},lag_{y}}}$ is the mean of A in the region under t.

The source code of implementation of normalized 2d cross correlation is in Appendix \ref{code:1.3}.

\section{Image Alignment}

Images are just a matrix of pixel values in most image processing. It is naturally think of computing cross correlation between two images to get more relation informations of them. In this task, there are two images, one of them is a section of the other, so it is easy to find where the section of the image fits in the whole through cross correlation.

Here, we have a section (rocket man) show in Figure \ref{fig:rocketman} and a search region (maze) as Figure \ref{fig:maze}.

\begin{figure}[h!]
	\centering
		\begin{subfigure}[t]{0.4\linewidth}
			\centering
			\includegraphics[width=0.6\linewidth]{figures/part1/wallypuzzle_rocketman.png}
			\caption{Rocket Man (section). }
			\label{fig:rocketman}
		\end{subfigure}
		\begin{subfigure}[t]{0.55\linewidth}
			\centering
			\includegraphics[width=1\linewidth]{figures/part1/wallypuzzle_result}
			\caption{Maze (search region). }
			\label{fig:maze}
		\end{subfigure}
		\caption{Source images for image alignment.}
\end{figure}

The visualization of cross correlation is shown in Figure \ref{fig:crr_vis1} and Figure \ref{fig:crr_vis2}. The maximum of the cross correlation corresponds to the estimated location of the section. Which means the most similar position of the section and search region. We mark it with a blue circle in Figure \ref{fig:maze}.

\begin{figure}[h!]
	\centering
	\begin{subfigure}[t]{0.45\linewidth}
		\includegraphics[width=1\linewidth]{figures/part1/crr_vis1.eps}
		\caption{2D visualization}
		\label{fig:crr_vis1}
	\end{subfigure}
	\begin{subfigure}[t]{0.45\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{figures/part1/crr_vis2.eps}
		\caption{3D visualization}
		\label{fig:crr_vis2}
	\end{subfigure}
	\caption{Cross correlation result matrix visualization}
\end{figure}

Source code for image alignment is in Appendix \ref{code:1.4}.

\section{Spectral Cross Correlation}

Cross correlation can also be done in the spectral domain by completing a Fourier transform, multiplying signals, and doing an inverse Fourier transform.

The fast Fourier transform (FFT) algorithm can convert a signal from its original domain to a representation in the frequency domain and inverse fast Fourier transform (IFFT) vice versa. It is an advanced algorithm of the discrete Fourier transform (DFT), and it manages to reduce the complexity of computing the DFT from $O(n^2)$, which arises if one simply applies the definition of DFT, to $O(n\log n)$ \cite{heiderman84}.

Fast Fourier transform is a very useful and powerful algorithm in computing convolution and cross correlation. It can be shown that the discrete convolution of signal $u$ and $v$ as defined by,

\begin{equation*}
(u*v)(\tau)=\sum_{m=1}^{N}u(m)v(\tau-m)
\end{equation*}

can also be expressed in terms of Fourier transform

\begin{equation*}
(u*v)(\tau)=\mathcal{F}^{-1}\{\mathcal{F}(u)\cdot \mathcal{F}(v)\}
\end{equation*}

where $\mathcal{F}(u)$ is the Fourier transform of $u$, $\mathcal{F}(v)$ is the Fourier transform of $v$, and $\mathcal{F}^{-1}$ is the inverse Fourier transform.

As for cross correlation, it can be calculated through summation of a product.

\begin{equation*}
(u*v)(\tau)=\sum_{m=1}^{N}u^*(m)v(\tau+m)
\end{equation*}

or using FFTs.

\begin{equation*}
	(u*v)(\tau)=\mathcal{F}^{-1}\{(\mathcal{F}(u))^{*}\cdot \mathcal{F}(v)\}
\end{equation*}

Where the $*$ refers to the complex conjugate.

Table \ref{tab:run_time} compare the run time of two different method of task in section 2. From which we can find that spectral method using Fourier transform achieves better performance.

\begin{table}[h!]
	\centering
	\caption{Run time compare of spatial and spectral methods.}
	\begin{tabular}{c|c|c}
		\hline
		method & Spatial & Spectral \\
		\hline
		time(s) & 4.6741 & 0.1103 \\
		\hline
	\end{tabular}
	\label{tab:run_time}
\end{table}

Source code of spectral cross correlation is in Appendix \ref{code:1.5}.

\section{Pattern Finder}

In this task, We pick a piece of music and cut a small piece of it (a drum) through listening. Figure \ref{fig:drum} and Figure \ref{fig:music} shows two pieces of waveforms separately. Figure \ref{fig:crr_vis3} presents all occurrence of the element, as it is not a normalized correlation picture, all positions with the y-coordinates larger than 1000 is the appearance of the drum. In addition, other background noise have an effect on our calculation but not serious.

\begin{figure}[h!]
	\centering
	\begin{subfigure}[t]{0.32\linewidth}
		\includegraphics[width=1\linewidth]{figures/part1/drum.eps}
		\caption{drum waveform}
		\label{fig:drum}
	\end{subfigure}
	\begin{subfigure}[t]{0.32\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{figures/part1/music.eps}
		\caption{music waveform}
		\label{fig:music}
	\end{subfigure}
	\begin{subfigure}[t]{0.32\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{figures/part1/crr_vis3.eps}
		\caption{cross correlation}
		\label{fig:crr_vis3}
	\end{subfigure}
	\caption{Two oscillographs}
	\label{fig:wave}
\end{figure}

From the above plots, the signal length of one single drum is 7000, then cutting a 4.5 seconds music which length in Matlab is about $2*10^5$. There are 7 occurrences of the dum in this episode.

Source code for pattern finder is in Appendix \ref{code:1.6}.

\section{Cross correlation in three dimensions}

This part is just some thinking by myself.

From the above research, we know that 1d cross correlation can be used to process acoustical signal and 2d cross correlation is very useful for image processing. Then, how about 3d cross correlation? If 3d cross correlation can be computed, we can easily know the space structure's similarity of two objects, theoretically. This task may have its value in microcosmos' research like molecular physics and microbiology. Consider there are one cell with a particular disease as specimen and no one know the reason of its pathogenicity. In the meanwhile, for several cells, some of them have the particular disease. If we can find spatial cross correlation between the specimen and other cell, base on the hypothesis that same diseased cell have the similar space structure, we can distinguish the healthy cell and diseased cell.

Computer vision is one of the most popular research today. One important reason is people believe 2d object is easy to understand and image compared with 3d object. The 2d technology today have been more and more mature. However, we live in a 3d (even higher dimensions) world, research in 3d will become mainstream someday, I believe 3d cross correlation will become a basic concept and tool like today's 2d cross correlation in the future.

Well, there are already some research about 3D cross correlation\cite{Li2002Auto}\cite{article2013}.


\begin{abox}
	Laplace Transforms
	\end{abox}

The Laplace transform $f(s)$ of a function $F(t)$ is defined by
\begin{equation}
f(s)=\mathcal{L}\{F(t)\}=\int_{0}^{\infty} e^{-s t} F(t) d t .
\end{equation}
A few comments on the existence of the integral are in order. The infinite integral of $F(t)$,
\begin{equation*}
\int_{0}^{\infty} F(t) d t,
\end{equation*}
need not exist. For instance, $F(t)$ may diverge exponentially for large $t$. However, if there are some constants $s_{0}, M$, and $t_{0} \geq 0$ such that for all $t>t_{0}$
\begin{equation}
\left|e^{-s_{0} t} F(t)\right| \leq M,\label{LT-01}
\end{equation}
the Laplace transform will exist for $s>s_{0} $. $F(t)$ is then said to be of exponential order. As a counter example, $F(t)=e^{t^{2}}$ does not satisfy the condition given by Eq. (\ref{LT-01}) and is not of exponential order. Thus, $\mathcal{L}\left\{e^{t^{2}}\right\}$ does not exist.
The Laplace transform may also fail to exist because of a sufficiently strong singularity in the function $F(t)$ as $t \rightarrow 0$. For example,
\begin{equation*}
\int_{0}^{\infty} e^{-s t} t^{n} d t
\end{equation*}
diverges at the origin for $n \leq-1$. The Laplace transform $\mathcal{L}\left\{t^{n}\right\}$ does not exist for $n \leq-1$. Since, for two functions $F(t)$ and $G(t)$ for which the integrals exist,
\begin{equation}
\mathcal{L}\{a F(t)+b G(t)\}=a \mathcal{L}\{F(t)\}+b \mathcal{L}\{G(t)\}
\end{equation}
the operation denoted by $\mathcal{L}$ is linear.\\
\subsection{Important Laplace Transforms:}
\begin{enumerate}
	\item $L(1)=\frac{1}{s}$
	\item $L\left(x^{n}\right)=\frac{n !}{s^{n+1}}(n=0,1,2, \ldots \ldots . .)$
	\item $L\left(e^{a x}\right)=\frac{1}{s-a}(s>a)$
	\item $L(\cos a x)=\frac{s}{s^{2}+a^{2}}$
	$(s>0)$
	\item $L(\sin a x)=\frac{a}{s^{2}+a^{2}} \quad(s>0)$
	\item $L(\cosh a x)=\frac{s}{s^{2}-a^{2}}\left(s^{2}>a^{2}\right)$
	\item $L(\sinh a x)=\frac{a}{s^{2}-a^{2}}\left(s^{2}>a^{2}\right)$
\end{enumerate}

\subsubsection{Important properties:}
\begin{enumerate}
	\item Linear Property: $L\left[a_{1} f_{1}(x)+a_{2} f_{2}(x)\right]=a_{1} L\left[f_{1}(x)\right]+a_{2} L\left[f_{2}(x)\right]$
	\item Shifting Property: $L\left[e^{a x} f(x)\right]=f(s-a)$
	\item Scaling Property: $L[f(a x)]=\frac{1}{a} f\left(\frac{s}{a}\right)$
	\item $L\left[x^{n} f(x)\right]=(-1)^{n} \frac{d^{n}}{d s^{n}}(f(s))$
	\item $L\left[\frac{f(x)}{x}\right]=\int_{-\infty}^{\infty} f(s) d s$
	\item $L\left[\int_{0}^{t} f(x) d x\right]=\frac{1}{s} f(s)$
	\item $L\left[f^{\prime}(x)\right]=s L[f(x)]-f(0)$
	\item  $L\left[f^{\prime \prime}(x)\right]=s^{2} L[f(x)]-f^{\prime}(0)-s f(0)$
	\item Laplace transform of a periodic function $f(x)$ having period ' $T$ ' is $$L[f({x})]=\frac{1}{1-e^{-s T}} \int_{0}^{T} e^{-s x} f({x}) d {x}$$
	\item Laplace transform of the unit step function $u(x-a)=u_{a}(x)=\left\{\begin{array}{lll}0 & \text { for } & x<a \\ 1 & \text { for } & x \geq a\end{array}\right.$ is $\frac{e^{-a s}}{s}$.
	\item Laplace transform of the dirac delta function $\delta(x-a)$ is $e^{-a s}$.
\end{enumerate}
\begin{exercise}
	Find the laplace transform of $f(x)=(1+\cos 2 x)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\begin{aligned}
	L[f(x)] &=\int_{0}^{\infty} e^{-s x}\left(1+\cos 2 x \cdot d x=\left(\frac{e^{-s x}}{-s}\right)_{0}^{\infty}+\int_{0}^{\infty} e^{-s x} \cos 2 x d x\right.\\
	&=\frac{1}{s}+\frac{s}{4+s}=\frac{2 s^{2}+4}{s\left(s^{2}+4\right)}
	\end{aligned}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of $f(x)=2 \sin 2 x \cos 4 x$
\end{exercise}
\begin{answer}
	\begin{align*}
	L[f(x)]=L[2 \sin 2 x \cdot \cos 4 x]=L[\sin 6 x-\sin 2 x]=\left(\frac{6}{s^{2}+36}-\frac{2}{s^{2}+4}\right)
	\end{align*}
\end{answer}
\begin{exercise}
Find the laplace transform of $f(x)=e^{-x}(3 \sinh 2 x-5 \cosh 2 x)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\begin{aligned}
	L[f(x)] &=L\left(3 e^{-x} \sinh 2 x\right)-L\left(5 e^{-x} \cosh 2 x\right) \\
	&=3 \frac{2}{(s+1)^{2}-4}-5 \cdot \frac{(s+1)}{(s+1)^{2}-4}=\frac{(1-5 \mathrm{~s})}{\left(\mathrm{s}^{2}+2 \mathrm{~s}-3\right)} \quad \text { (Using 2nd property) }
	\end{aligned}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of the fuction\\
	$f(x)=1$ for $2 n \leq x \leq 2 n+1$\\
	$=0 \quad$ for $2 n+1 \leq x \leq 2 n+2 \quad(n=0,1,2, \ldots . .)$
\end{exercise}
\begin{answer}
	\begin{align*}
	L[f(x)] &=\frac{1}{1-e^{-s T}} \int_{0}^{T} e^{-s x} f(x) d x=\frac{1}{1-e^{-2 s}} \int_{0}^{2} e^{-s x} d x \\
	&=\frac{1}{1-e^{-2 s}}\left[\int_{0}^{1} e^{-s x} d x\right]=\frac{1}{1-e^{-2 s}}\left(\frac{1-e^{-s}}{s}\right)\\
	&=\frac{\left(1-e^{-s}\right)}{s\left(1+e^{-s}\right)\left(1-e^{-s}\right)}=\frac{1}{s\left(1+e^{-s}\right)}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of $f(x)=\left(1+x e^{-x}\right)^{3}$
\end{exercise}
\begin{answer}
	\begin{align*}
	f(x)&=\left(1+x e^{-x}\right)^{3}=1+x^{3} e^{-3 x}+3 x e^{-x}+3 x^{2} e^{-2 x} \\
	L[f(x)]&=L(1)+L\left(x^{3} e^{-3 x}\right)+L\left(3 x e^{-x}\right)+L\left(3 x^{2} e^{-2 x}\right)\\
	&=\frac{1}{s}+(-1)^{3} \frac{d^{3}}{d s^{3}}\left(\frac{1}{s+3}\right)+3(-1)^{1} \frac{d}{d s}\left(\frac{1}{s+1}\right)+3(-1)^{2} \frac{d^{2}}{d s^{2}}\left(\frac{1}{s+2}\right)\text{ (Using 4th property)}\\
	&=\frac{1}{s}+\frac{6}{(s+3)^{4}}+\frac{3}{(s+1)^{2}}+\frac{6}{(s+2)^{3}} \text {. }
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of $f(x)=\frac{1-e^{x}}{x}$
\end{exercise}
\begin{answer}
	\begin{align*}
	L\left(\frac{1-e^{x}}{x}\right)&=\int_{s}^{\infty}\left(\frac{1}{s}-\frac{1}{s-1}\right) d s=[\ln s-\ln (s-1)]_{s}^{\infty}=\left[\ln \frac{s}{s-1}\right]_{s}^{\infty}:\text{ (Using 5th property)}\\
	&=[\ln s-\ln (s-1)]_{s}^{\infty}=\left[\ln \frac{s}{s-1}\right]_{s}^{\infty}=\ln \left(\frac{1}{\left.1-\frac{1}{s}\right)_{s}}\right)^{\infty} \\
	&=0-\ln \frac{s}{s-1}=\ln \frac{(s-1)}{s}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the inverse laplace transform of
	$$\mathrm{f}(\mathrm{s}) =\frac{\mathrm{s}-2}{(\mathrm{~s}-2)^{2}+25}+\frac{\mathrm{s}+4}{(\mathrm{~s}+4)^{2}+81}+\frac{1}{(\mathrm{~s}+2)^{2}+9} $$
\end{exercise}
\begin{answer}
	\begin{align*}
	\mathrm{~L}^{-1}[\mathrm{f}(\mathrm{s})] &=\mathrm{L}^{-1}\left[\frac{\mathrm{s}-2}{(\mathrm{~s}-2)^{2}+25}\right]+\mathrm{L}^{-1}\left[\frac{\mathrm{s}+4}{(\mathrm{~s}+4)^{2}+81}\right]+\mathrm{L}^{-1}\left[\frac{1}{(\mathrm{~s}+2)^{2}+9}\right] \\
	&=\mathrm{e}^{2 \mathrm{x}} \cos 5 \mathrm{x}+\mathrm{e}^{-4 \mathrm{x}} \cos 9 \mathrm{x}+\frac{\mathrm{e}^{-2 \mathrm{x}}}{3} \sin 3 \mathrm{x}
	\end{align*}
\end{answer}
\begin{exercise}
	The graph of the function
	$$f(x)= \begin{cases}1 & \text { for } 2 n \leq x \leq 2 n+1 \\ 0 & \text { for } 2 n+1 \leq x \leq 2 n+2\end{cases}$$
	(Where $n=0,1,2,........$)is shown below.
	\begin{figure}[H]
		\centering
		\includegraphics[height=2.5cm,width=6cm]{LT-01}
	\end{figure}
	Its Laplace transform $\tilde{f}(\mathrm{s})$ is
	 \begin{tasks}(2)
		\task[\textbf{a.}]$\frac{1+e^{-s}}{s}$
		\task[\textbf{b.}]$\frac{1-e^{-s}}{s}$
		\task[\textbf{c.}]$\frac{1}{s\left(1+e^{-s}\right)}$
		\task[\textbf{d.}]  $\frac{1}{s\left(1-e^{-s}\right)}$
	\end{tasks}
\end{exercise}
\begin{answer}
	Given function is a periodic function of period $T=2$
	\begin{align*}
	L[f(x)] &=\frac{1}{1-e^{-s T}} \int_{0}^{T} e^{-s x} f(x) d x=\frac{1}{1-e^{-2 s}} \int_{0}^{2} e^{-s x} d x \\
	&=\frac{1}{1-e^{-2 s}}\left[\int_{0}^{1} e^{-s x} d x\right]=\frac{1}{1-e^{-2 s}}\left(\frac{1-e^{-s}}{s}\right) \\
	&=\frac{\left(1-e^{-s}\right)}{s\left(1+e^{-s}\right)\left(1-e^{-s}\right)}=\frac{1}{s\left(1+e^{-s}\right)}
	\end{align*}
	So the correct answer is \textbf{option (c)}
\end{answer}
\begin{exercise}
	The inverse transform of $\frac{1}{s^2(s+1)}$ is
	 \begin{tasks}(2)
		\task[\textbf{a.}]$\frac{1}{2} t^{2} e^{-t}$
		\task[\textbf{b.}]$\frac{1}{2} t^{2}+1-e^{-t}$
		\task[\textbf{c.}]$t-1+e^{-t}$
		\task[\textbf{d.}] $\frac{1}{2} t^{2}\left(1-e^{-t}\right)$
	\end{tasks}
\end{exercise}
\begin{answer}
	\begin{align*}
	&f(s)=\frac{1}{s^{2}(s+1)}=\frac{(s+1)-s(s+1)+s^{2}}{s^{2}(s+1)}=\frac{1}{s^{2}}-\frac{1}{s}+\frac{1}{s+1} \\
	&\Rightarrow L^{-1}[f(s)]=t-1+e^{-t}
	\end{align*}
		So the correct answer is \textbf{option (c)}
\end{answer}

CSIR- Mathematical Physics/chapter/Laplace Transforms.tex


\chapter{Introduction}
\label{chap:introduction}

\section{Image Compression}

Image compression is to decrease the size of the image file without dramatically downgarding the quality of the image.

In this courework, I set out to implement a simulation of the JPEG (Joint Photographic Experts Group) image compression process\citep{wallace1992jpeg}. The implementation uses MATLAB as frontend interface and Python as backend.

\section{JPEG Standard}

The JPEG Still Picture Compression Standard is widely used on modern digital devices, ranging from computers, cameras to smartphones. For a grayscale image, a JPEG CODEC (Encoder and Decoder) typically involves 6 main steps as illustrated in Figure \ref{fig:codec}.

\begin{figure}
\centering
\includegraphics{codec}
\caption{Six Main Steps of a JPEG CODEC.}
\label{fig:codec}
\end{figure}

In my application, I implement a simplified JPEG CODEC in Python programming language. The CODEC consists of both foward and inverse steps of Discrete Cosine Transform (DCT) and Quantization and gets rid of both forward and inverse steps of Entropy Coding.

Since Entropy Codeing is reversible through Inverse Entropy Coding and does not affect the quality of the image, my implementation should be able to simulate the full effects of JPEG coding on any given still image despite its simplicity.

As shown in Figure \ref{fig:app}, given an input grayscale image in uint8 matrix format, the application first crops the image array to multiply of $8$ in both column (width) and row (height). Then, a value of $128$ is subtracted from the image for each pixel value. After that, the matrix is divided into 8 by 8 blocks. Each block then goes through FDCT, Quantization, Inverse Quantization and IDCT individually.  All resulted blocks are placed in their previous position and a new matrix is thus formed. Finally, a value of $128$ is added back to the new matrix to get the compressed grayscale image.

\begin{figure}
\centering
\includegraphics{app}
\caption{Main Steps of the Application.}
\label{fig:app}
\end{figure}

For a color RGB image, the same process is carried out on each color channel respectively.

\subsection{Discrete Cosine Transform (DCT)}

The forward step of DCT is computed using the following equation.

\begin{equation}
F(u, v) = \frac{1}{4} C(u) C(v) [\sum_{x=0}^7 \sum_{y=0}^7 f(x,y) * cos\frac{(2x+1)u\pi}{16} cos\frac{(2y+1)v\pi}{16}]
\label{equ:fdct}
\end{equation}

where $C(t) = 1/\sqrt(2)$ for $t = 0$; $C(t) = 1$ otherwise. Both input $f$ and output $F$ is an 8 by 8 block.

The inverse step of DCT is computed using the following equation.

\begin{equation}
f(x, y) = \frac{1}{4} [\sum_{u=0}^7 \sum_{v=0}^7 C(u) C(v) F(u, v) * cos\frac{(2x+1)u\pi}{16} cos\frac{(2y+1)v\pi}{16}]
\label{equ:idct}
\end{equation}

where $C(t) = 1/\sqrt(2)$ for $t = 0$; $C(t) = 1$ otherwise. Both input $F$ and output $f$ is an 8 by 8 block.

\subsection{Quantization}

The forward step of Quantization is computed using the following equation.

\begin{equation}
F^Q(u, v) = round(\frac{F(u, v)}{Q(u, v)})
\label{equ:fq}
\end{equation}

where $Q$ is the Quantization Table specified in the JPEG standard. Both input $F$ and ouput $F^Q$ is an 8 by 8 block.

The inverse step of Quantization is computed using the following equation.

\begin{equation}
F(u, v) = F^Q(u, v) * Q(u, v))
\label{equ:iq}
\end{equation}

where $Q$ should be the same as in the forward step. Both input $F^Q$ and ouput $F$ is an 8 by 8 block.

By default, the value of $Q$ is specfied as followed.

\begin{equation}
Q =
\begin{bmatrix}
  16& 11& 10& 16& 24& 40& 51& 61 \\
  12& 12& 14& 19& 26& 58& 60& 55 \\
  14& 13& 16& 24& 40& 57& 69& 56 \\
  14& 17& 22& 29& 51& 87& 80& 62 \\
  18& 22& 37& 56& 68& 109& 103& 77 \\
  24& 35& 55& 64& 81& 104& 113& 92 \\
  49& 64& 78& 87& 103& 121& 120& 101 \\
  72& 92& 95& 98& 112& 100& 103& 99
\end{bmatrix}
\label{equ:q}
\end{equation}


\section{Non-deterministic finite automata}

A \emph{non-deterministic finite automaton} (\emph{NFA}) has the
same definition as a DFA except that \(\delta\) returns a set of
states instead of one state. Let us consider
\begin{center}
\includegraphics[bb=48 711 188 758]{nfa_01_suffix}
\label{nfa_01_suffix}
\end{center}
There are two out-going edges from state~\(q_0\) which are
labelled~\(0\), hence two states can be reached when~\(0\) is input:
\(q_0\) (loop) and~\(q_1\). This NFA recognises the language of words
on the binary alphabet whose suffix is~01.

Before describing formally what is a recognisable language by a NFA,
let us consider as an example the previous NFA and the input
\(00101\). Let us represent each transition for this input by an edge
in a tree where nodes are states of the NFA:
\begin{center}
\includegraphics[bb=71 651 395 732]{nfa_01_suffix_trace}
\end{center}
A NFA is represented essentially like a DFA: \(\mathcal{N} = (Q_N,
\Sigma, \delta_N, q_0, F_N)\), where the names have the same
interpretation as for DFA, except~\(\delta_N\), which returns a subset
of~\(Q\) --~not an element of~\(Q\). For example, the NFA above can be
specified formally as
\begin{equation*}
\mathcal{N} = (\{q_0, q_1, q_2\}, \{0, 1\}, \delta_N, q_0, \{q_2\})
\end{equation*}
where the transition function~\(\delta_N\) is given by the transition
table
\begin{equation*}
\begin{array}{r@{}l||c|c}
\multicolumn{2}{c||}{\mathcal{N}} & 0 & 1\\
\hhline{==::==}
\rightarrow & q_0 & \{q_0, q_1\} & \{q_0\}\\
            & q_1 & \varnothing  & \{q_2\}\\
\#          & q_2 & \varnothing  & \varnothing
\end{array}
\end{equation*}
Note that, in the transition table of a NFA, all the cells are filled:
there is no transition between two states if and only if the
corresponding cell contains~\(\varnothing\). In case of a DFA, the
cell would remain empty. It is common also to set that in case of the
empty word input, \(\varepsilon\), both for the DFA and NFA, the state
remains the same:
\begin{itemize*}

  \item for DFA: \(\forall q \in Q.\delta_D (q, \varepsilon) = q\);

  \item for NFA: \(\forall q \in Q.\delta_N (q, \varepsilon) = \{q\}\).

\end{itemize*}
As we did for the DFAs, we can \emph{extend the transition function}
\(\delta_N\) to accept words and not just letters (labels). The
extended function is noted \(\hat{\delta}_N\) and defined as
\begin{itemize*}

  \item for all state \(q \in Q\), let \(\hat{\delta}_N (q,
    \varepsilon) = \{q\}\);

  \item for all state \(q\ \in Q\), all words \(w \in \Sigma^{*}\),
    all input \(a \in \Sigma\), let
    \begin{equation*}
      \hat{\delta}_N (q, wa) =
        \bigcup_{q' \in \hat{\delta}_N (q, w)}{\delta_N  (q', a)}.
    \end{equation*}

\end{itemize*}
The language \(L(\mathcal{N})\) recognised by a NFA~\(\mathcal{N}\) is
defined as
\begin{equation*}
L(\mathcal{N}) = \{w \in \Sigma^{*} \; \lvert \; \hat{\delta}_N (q_0, w)
\cap F \neq \varnothing\},
\end{equation*}
which means that the processing of the input stops successfully as
soon as at least one current state belongs to~\(F\).

For example, let us use~\(\hat{\delta}_N\) to describe the processing
of the input \verb+00101+ by the NFA \vpageref{nfa_01_suffix}:
\begin{enumerate*}

  \item \(\hat{\delta}_N (q_0, \varepsilon) = q_0\),

  \item \(\hat{\delta}_N (q_0, \verb+0+) = \delta_N (q_0, \verb+0+) =
    \{q_0, q_1\}\),

  \item \(\hat{\delta}_N (q_0, \verb+00+) = \delta_N (q_0, \verb+0+)
    \cup \delta_N (q_1, \verb+0+) = \{q_0, q_1\} \cup \varnothing =
    \{q_0, q_1\}\),

  \item \(\hat{\delta}_N (q_0, \verb+001+) = \delta_N (q_0, \verb+1+)
    \cup \delta_N (q_1, \verb+1+) = \{q_0\} \cup \{q_2\} = \{q_0,
    q_2\}\),

  \item \(\hat{\delta}_N (q_0, \verb+0010+) = \delta_N (q_0, \verb+0+)
    \cup \delta_N (q_2, \verb+0+) = \{q_0, q_1\} \cup \varnothing =
    \{q_0, q_1\}\),

  \item \(\hat{\delta}_N (q_0, \verb+00101+) = \delta_N (q_0,
    \verb+1+) \cup \delta_N (q_1, \verb+1+) = \{q_0\} \cup \{q_2\} =
    \{q_0, q_2\} \ni q_2\).

\end{enumerate*}
Since~\(q_2\) is a final state, in fact \(F = \{q_2\}\), we get
\(\hat{\delta}_N (q_0, \verb+00101+) \cap F \neq \varnothing\) thus
the string \verb+00101+ is recognised by the NFA.


\chapter{Brownian Motion}

\newpage

\newcommand{\cov}{\mathrm{Cov}}
\newcommand{\var}{\mathrm{Var}}

\section{Definition of Brownian Motions}
    \subsection{Introduction}
        We start from a random walk on real axis, but this time we move every $\Delta t$ time. So in time interval $t$ we take $t/\Delta t$ steps. Let $\delta$ be the step size, then
        \[ X(t) = \delta(X_1 + X_2 + \cdots + X_{t/\Delta t}) \]
        where $X_i \in \{0,1\}$ with mean $0$ and variance $1$.

        Then
        \[ \mathbb{E}[X(t)] = 0 \qquad Var[X(t)] = \delta^2\frac{t}{\Delta t} \]

        What happens if $\Delta t \to 0$? Notice that if $\Delta t$ exists in $Var[X(t)]$, then it either goes to 0 or goes to infinity and the problem is meaningless. So we set $\delta = \sigma \sqrt{\Delta t}$ and therefore
        \[ Var[X(t)] = \sigma^2 t \]

        Recall the Central Limit Theorem in \textsc{Probability and Statistics}
        \begin{theorem}[Central Limit Theorem]\label{thm:CentralLimitTheorem}
            Let $X_1,X_2,\dots$ be a sequence of i.i.d. random variables, with mean $\mu$ and variance $\sigma^2$. Then
            \[ \frac{\sum_iX_i - n\mu}{\sigma\sqrt{n}} \sim \mathcal{N}(0,1) \]
        \end{theorem}

        Let $Y_i = \delta X_i$. $Var[Y_i] = \sigma^2 \Delta t$. Then
        \[ X(t) = \sum_{k=0}^{t/\Delta t} Y_k \sim \sigma \sqrt{\Delta t}\cdot\sqrt{\frac{t}{\Delta t}} \mathcal{N}(0,1) = \mathcal{N}(0, \sigma^2 t) \]

        \begin{remark}
            This result implies that the motion $X(t)$ in different time intervals are independent, i.e. $\forall t_1 < t_2 < \cdots < t_n$,
            \[ X(t_n)-X(t_{n-1}), \dots, X(t_2)-X(t_1), X(t_1) \]
            are independent.

            Further, $X(t+s)-X(t)$ depends only on $s$.

            This is very similar to the property of a Poisson Process.
        \end{remark}

    \subsection{Formal Definition}
        \begin{definition}[Standard Brownian Motion]\label{def:BrownianMotionStd}
            $\{W(t):t>0\}$ is a \textbf{Standard Brownian Motion} Process if
            \begin{enumerate}
                \item $W(0) = 0$.
                \item (Independent Increment) $\forall t_1 < t_2 < \cdots < t_n$, $W(t_n)-W(t_{n-1}), \dots, W(t_2)-W(t_1)$, $W(t_1)$ are independent.
                \item (Stationary) $\forall t,s \ge 0$, $W(t+s) - W(t)$ only depends on $s$, and has a normal distribution $\mathcal{N}(0, s)$
                \item Continuous Path
            \end{enumerate}
        \end{definition}

    \subsection{Another Characterizationo of SBM}
        \begin{definition}[Multivariate Gaussian Distribution]
            A vector $(X_1, \dots, X_n)$ has a Gaussian Distribution if (and only if)
            \[ \forall a_1, \dots, a_n \qquad \sum_{i=1}^n a_iX_i \]
            has a (Univariate) Gaussian Distribution.

            Or equivalently
            \[ f(X) = (2\pi)^{-n/2}|\det\Sigma|^{-1/2}\exp\left\{(x-\mu)^T\Sigma^{-1}(x-\mu)\right\} \]
        \end{definition}
        \begin{remark}
            The marginal probability given $X_i$ is also a Gaussian Distribution.
        \end{remark}

        \begin{definition}[Gaussian Process]\label{def:GaussianProcess}
            A stochastic process $\{W(t)\}_{t \ge 0}$ is a \textbf{Gaussian Process} if
            \[ \forall n, \forall t_1 \le t_2 \le \cdots \le t_n, \qquad (W(t_1), W(t_2), \dots, W(t_n))\]
            has a Guassian Distribution.
        \end{definition}

        \begin{definition}[Definition of SBM via Gaussian Process]\label{def:BrownianMotionStdAlt}
            A stochastic process $\{W(t)\}$ is a Standard Brownian Motion if
            \begin{enumerate}
                \item $W(t)$ is a Guassian Process.
                \item $W(t)$ has mean zero. $\forall s$, $\mathbb{E}[W(s)] = 0$.
                \item $\forall s \le t$, $Cov(W(s), W(t)) = s$. ($Cov(W(s), W(t))$ equals to $s$ if $s < t$ else $t$)
            \end{enumerate}
        \end{definition}
        \begin{remark}
            This alternative definition is more often used to justify that a process is a Brownian motion. The only difficulty is to compute the covariance.

            As a reminder, the covariance is given by
            \[ Cov(X,Y) = \mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])] \]
        \end{remark}
        \begin{proof}
            We now verify that Definition \ref{def:BrownianMotionStdAlt} is equivalent to \ref{def:BrownianMotionStd}.

            $\Rightarrow$: A SBM satisfies definition \ref{def:BrownianMotionStdAlt}. To see this, we show that any linear combination of $W(s)$ and $W(t)$ is a Gaussian. $\forall a,b$,
            \begin{align*}
                aW(s) + bW(t) &= aW(s) + b(W(t) - W(s) + W(s))\\
                &= (a+b)W(s) + b(W(t)-W(s))
            \end{align*}
            This is the sum of two independent Gaussians, so the sum is still a Gaussian, and we are done.

            $W(t)$ has mean zero is straightforward by definition of SBM. So we only need to check condition 3.

            \begin{align*}
                Cov(W(s), W(t)) &= Cov(W(s), W(t) - W(s) + W(s))\\
                &= Var(W(s)) + Cov(W(s), W(t) - W(s))\\
                &=s
            \end{align*}
            Notice that the latter term is zero by the independent increment property.
        \end{proof}

        \subsubsection{A Simple Example}
            If $W(t)$ is a SBM, then $X(t)=tW(1/t)$ is also a SBM.
            \begin{proof}
                We only need to check definition \ref{def:BrownianMotionStdAlt}. The first two conditions are easy to check. We consider the covariance.
                \begin{align*}
                    Cov(X(s), X(t)) &= Cov(sW(1/s),tW(1/t))\\
                    &= stCov(W(1/s), W(1/t))\\
                    &= st\cdot\frac{1}{t} = s
                \end{align*}
            \end{proof}

    \subsection{Non-Standard Brownian Motions}
        \begin{definition}[General Brownian Motion]
            $\{X(t)\}$ is a $(\mu, \sigma^2)$ Brownian Motion if
            \[ X(t) = X(0) + \mu t + \sigma W(t) \]
        \end{definition}

\section{First Passage Time and Reflection Principle}
    Let $W(t)$ be a Brownian Motion. Let $\tau_b$ be the first time for the motion to pass some point $b$,
    \[ \tau_b \triangleq \inf\left\{ t \ge 0 : W(t) > b \right\} \]

    We are interested in $\mathbb{P}[\tau_b < t]$. $\forall t$
    \begin{align*}
        \mathbb{P}[\tau_b < t] &= \mathbb{P}[\tau_b < t, W(t) > b] + \mathbb{P}[\tau_b < t, W(t) \le b] \quad \text{(Total Probability)}\\
        &= \mathbb{P}[W(t) > b] + \mathbb{P}[W(t) < b | \tau_b < t]\cdot\mathbb{P}[\tau_b < t]\\
    \end{align*}

    Notice that $W(t) > b$ implies $\tau_b < t$, so
    \[ \mathbb{P}[\tau_b < t, W(t) > b] = \mathbb{P}[W(t) > b] = 1 - \Phi(\frac{b}{\sqrt{t}}) \]

    Further notice that $\mathbb{P}[W(t) < b | \tau_b < t]$ equals exactly to $1/2$, because the path is continuous, so that starting from $W(\tau_b) = b$, the motion is \emph{equally likely} to go beyond or below $b$. This property is referred to as the \textbf{Reflection Principle}.

    Plugging in all the results and solving for $\mathbb{P}[\tau_b < t]$ yields
    \[ \mathbb{P}[\tau_b < t] = 2 \left( 1 - \Phi\left(\frac{b}{\sqrt{t}}\right) \right) \]

\section{Brownian Bridge}
    \subsection{Conditional Distributions for Brownian Motion}
        In this section, we fix $W(u) = x$, i.e. we add a restriction that at time $u$ the process must be at $x$. We are interested in the distribution of $W(t)$ in time interval $[0,u]$, conditioned on $W(u)=x$.

        The resulting process still has a Gaussian distribution, so we only need to compute the mean and covariance.

        Before we formally start, we first introduce a tool.
        \begin{proposition}\label{prop:LemmaOfBrownianBridge}
            \[ W(t) - (t/u)W(u) \]
            is independent of $W(u)$
        \end{proposition}
        \begin{proof}
            To see this, we only need to show that the covariance $\cov[W(t)-(t/u)W(u), W(u)] = 0$.
            \begin{align*}
                \cov[W(t) - (t/u)W(u), W(u)] &= \cov[W(t),W(u)] - \frac{t}{u}\cov[W(u), W(u)]\\
                &= t - \frac{t}{u} \cdot u = 0
            \end{align*}
            So we are done.
        \end{proof}

        With Proposition \ref{prop:LemmaOfBrownianBridge} we shall continue.

        \begin{align*}
            0 &= \mathbb{E}\left[ W(t) - \frac{t}{u}W(u) \right]\\
            &= \mathbb{E}\left[ W(t) - \frac{t}{u} W(u) \middle| W(u) \right]\\
            &= \mathbb{E}\left[ W(t) \middle| W(u) \right] - \frac{t}{u}\mathbb{E}[W(u) | W(u)]\\
            &= \mathbb{E}\left(W(t) | W(u)\right) - \frac{t}{u}W(u)
        \end{align*}

        Therefore
        \[ \mu = \mathbb{E}[W(t)|W(u)] = \frac{t}{u}W(u) \]

        We then move on to compute the covariance, $\forall t \in [0, u]$,
        \begin{align*}
            \var[W(t) | W(u)] &= \mathbb{E}[(W(t) - \mathbb{E}[W(t)|W(u)])^2 | W(u)]\\
            &= \mathbb{E}[(W(t)- \frac{t}{u}W(u))^2 | W(u)]\\
            &= \mathbb{E}[(W(t) - \frac{t}{u}W(u))^2] \quad \text{(By Proposition \ref{prop:LemmaOfBrownianBridge})}\\
            &= \mathbb{E}[W^2(t)] - \frac{2t}{u}\mathbb{E}[W(t)W(u)] + \frac{t^2}{u^2}\mathbb{E}[W^2(u)]\\
            &= t - \frac{2t}{u}t + \frac{t^2}{u^2}u\\
            &= \frac{ut - t^2}{u} = \frac{t(u-t)}{u}
        \end{align*}
        \begin{remark}
            Since the process is constrained at $t=0$ and $t=u$, the result has a straightforward intuition that the process is more likely to have large variances in the middle of the process.
        \end{remark}

        \begin{align*}
            \cov[W(s), W(t) | W(u)] &= \mathbb{E}[W(s) \cdot W(t) | W(u)] - \mathbb{E}[W(s) | W(u)] \cdot \mathbb{E}[W(t) | W(u)]\\
        \end{align*}
        The last two terms are exactly the expectation (mean value) computed previously, so we only need to deal with the first term.

        \begin{align*}
            \mathbb{E}[W(s)W(t)|W(u)] &= \int y \mathbb{E}[W(s)|W(t)=y, W(u)]\cdot p_{W(t)}(y|W(u)) \mathrm{d}y\\
            &= \int y \mathbb{E}[W(s) | W(t) = y] \cdot p[y|W(u)]\mathrm{d}y \quad \text{(By Independent Increment)}\\
            &= \int y \cdot \frac{s}{t}y p(y|W(u))\mathrm{d}y\\
            &= \frac{s}{t}\mathbb{E}[W^2(t)|W(u)]
        \end{align*}

        Plug this term back,
        \[
            \cov[W(s), W(t) | W(u)] = \frac{s}{t}\mathbb{E}[W^2(t)|W(u)] - \frac{st}{u^2}W^2(u)
        \]

        where
        \[ \mathbb{E}[W^2(t)|W(u)] = \mathbb{E}[W(t)|W(u)]^2 + \var[W(t)|W(u)] = \frac{t(u-t)}{u} + \frac{t^2}{u^2}W^2(u) \]

        so the final result is
        \[ \cov[W(s), W(u) | W(u)] = \frac{s(u-t)}{u} \]

    \subsection{Standard Brownian Bridge}
        \begin{definition}[Standard Brownian Bridge]\label{def:BrownianBridgeStd}
            A Gaussian Process is called a \textbf{Standard Brownian Bridge} if
            \begin{enumerate}
                \item $X(0) = X(1) = 0$
                \item $\cov[X(s), X(t)] = s(1-t)$
            \end{enumerate}
        \end{definition}
        \begin{remark}
            The Brownian Bridge can be directly constructed from a Brownian Motion by
            \[ X(t) = W(t) - tW(1) \]
        \end{remark}

    \subsection{First Passage Time of Brownian Bridge}
        In this section we consider the first passage time of a Brownian Bridge.
        \[ \tau_b \triangleq \inf\left\{ t \ge 0 : W(t) > b \right\} \]

        We assume $x < b$, or otherwise $\mathbb{P}[\tau_b < t] = 1$. Consider $\mathbb{P}[\tau_b < t | W(t) = x]$, that is, the probability that we ever reached $b$ before we arrive at $x$ at time $t$.

        By definition of conditional probability,
        \[ \mathbb{P}[\tau_b < t | W(t) =  x] = \frac{\mathbb{P}[\tau_b < t, W(t) = x]}{\mathbb{P}[W(t) = x]} \]

        We then get a $0/0$ because the probability that a continuous random variable gets a certain value is 0. Instead, we apply some informal notations. Let $\mathrm{d}x$ be a small interval around $x$, we compute
        \[ \frac{\mathbb{P}[\tau_b < t, W(t) \in \mathrm{d}x]}{\mathbb{P}[W(t) \in \mathrm{d}x]} \]

        Denominator
        \begin{align*}
            \mathbb{P}[W(t) \in \mathrm{d}x] &= \mathbb{P}[\frac{1}{\sqrt{u}}W(t) \in \frac{\mathrm{d}x}{\sqrt{u}}]\\
            &= \Phi(\frac{\mathrm{d}x}{\sqrt{u}})\\
            &= \frac{1}{\sqrt{u}}\phi(\frac{x}{\sqrt{u}})\mathrm{d}x
        \end{align*}

        Numerator
        \begin{align*}
            \mathbb{P}[\tau_b < t, W(t) \in \mathrm{d}x] &= \mathbb{P}[\tau_b < t] \mathbb{P}[W(t) \in \mathrm{d}x | \tau_b < t]\\
            &= \mathbb{P}[\tau_b < t]\mathbb{P}[W(t) \in 2b - \mathrm{d}x | \tau_b < t]\\
            &= \mathbb{P}[W(t) \in 2b-\mathrm{d}x, \tau_b < t]
        \end{align*}

        Notice that $W(t) \in 2b-\mathrm{d}x$ implies $\tau_b < t$.
        \begin{align*}
            \mathbb{P}[\tau_b < t, W(t) \in \mathrm{d}x] &= \mathbb{P}[W(t) \in 2b-\mathrm{d}x] = \frac{1}{\sqrt{u}}\phi\left( \frac{2b-x}{\sqrt{u}}\mathrm{d}x \right)
        \end{align*}

        The result is
        \[ \mathbb{P}[\tau_b < u | W(u) = x] = \exp\left\{ \frac{-2b(b-x)}{u} \right\} \]

    \subsection{Application to Test for Uniformity}
        Suppose we have a series of i.i.d. random variables $U_1, \dots, U_n$ having some distribution $F$ on $[0,1]$. We want to test if $F(t)=\mathbb{P}[U < t]=t$, i.e. whether $F$ is a uniform distribution.

        We define
        \[ F_n(t) = \frac{1}{n} \sum_{i=1}^n \mathbb{I}[U_i < t] \]

        \subsubsection{The Kolmogorov-Smirnov Test}
        For some large enough $n$, reject if
        \[ F_n(t) - t \ge b \]
        for some $t \in [0,1]$ and accept otherwise.

        \subsubsection{Choice of Parameters}
        We want to choose a proper $b$ so that the probability that we mistakenly reject a uniform distribution is small enough.

        Assume $F(t)=t$, we hope the algorithm outputs accept. By Central Limit Theorem, $F_n(t)$ has a normal distribution. For a fixed $t$, we have
        \[ \var[\mathbb{I}[U_i < t]] = t(1-t) \]

        Let
        \[ X_n(t) = \sqrt{n}(F_n(t)-t) \]

        Then by Central Limit Theorem,
        \[ X_n(t) \longrightarrow \mathcal{N}(0, t(1-t)) \]

        Using a Multivariate CLT, we actually have
        \[ (X_n(t_1), X_n(t_2)) \longrightarrow \mathcal{N}(0,t_1(1-t_2)) \]
        where $t_1(1-t_2)$ is a covariance matrix.

        Therefore
        \[ \cov[\mathbb{I}[U < t_1], \mathbb{I}[U < t_2]] = t_1(1-t_2) \]

        Which implies that $(X_n(t_1), X_n(t_2), \dots, X_n(t_k))$ converges (in distribution) to a standard Brownian Bridge. Using the conclusion in previous section, we can conclude that
        \[ \mathbb{P}[F_n(t)-t \ge b] = e^{-2b^2} \]


% !TEX root = main.tex

%-------------------------------------------------
\section{One-sample tests}\label{sec:np_one_sample_tests}

%-------------------------------------------------
\subsection{The sign test}\label{sec:signtest}

Let $X$ be a continuous random variable with an unknown median $\eta$. The sign test is a non-parametric method for testing hypotheses about $\eta$.

%%-----------------------------
%\subsection{The median}
%The non-parametric methods we consider are based on the \emph{median} of an unknown distribution.
%
%\begin{definition}
%A \emph{median} of a random variable $X$ is a real number $\eta$ satisfying
%\[
%\prob(X\leq\eta)\,\geq\frac{1}{2} \quad\text{and}\quad \prob(X\geq\eta)\,\geq\frac{1}{2}.
%\]
%If $X$ is a continuous random variable, its median is uniquely defined,
%\[
%\prob(X\leq\eta) = \prob(X\geq\eta) = \frac{1}{2}.
%\]
%\end{definition}
%
%\begin{example}
%Consider the random sample $\{1, 2, 2, 2, 3, 14\}$.
%\bit
%\it The sample median is $\hat{\eta} = 2$; the sample mean is $\hat{\mu}  = 4$.
%\eit
%In this case, $\hat{\eta}$ provides a better indicator of centrality than $\hat{\mu}$.
%\end{example}

%-----------------------------
\subsubsection{The test statistic}

Let $X_1,X_2,\ldots,X_n$ be a random sample from the distribution of $X$ and consider the null hypothesis $H_0:\eta=\eta_0$ against a suitable alternative. If $H_0$ is correct then approximately half of the observations should be smaller than $\eta_0$ and approximately half should be larger than $\eta_0$.

\begin{definition}
The sign test statistic $S^{+}_n$ is the number of observations larger than $\eta_0$:
\[
S^{+}_n = \sum_{i=1}^n Z_i \quad\text{where}\quad Z_i=I(X_i>\eta_0).
\]
\end{definition}

We also define the complementary statistic $S^{-}_n = \displaystyle\sum_{i=1}^n (1-Z_i)$ which is the number of observations smaller than $\eta_0$. Note that $S^{+}_n + S^{-}_n = n$.

\bigskip
Because the observations are independent, the distribution of our test statistic under $H_0$ is
\[
S^{+}_n\sim\text{Binomial}(n,0.5).
\]
\bit
\it Small values of $S^{+}_n$ support the alternative $H_1:\eta < \eta_0$.
\it Large values of $S^{+}_n$ support the alternative $H_1:\eta > \eta_0$.
\eit

% example: sign test (one sample)
\begin{example}
The following are measurements of the breaking strength of a certain kind of two-inch cotton ribbon.
\[\begin{array}{cccccccccc}
163 & 165 & 158 & 189 & 161 & 171 & 158 & 151 & 169 & 162 \\
163 & 139 & 172 & 165 & 148 & 166 & 172 & 163 & 187 & 173 \\
\end{array}\]
Conduct a sign test to decide between $H_0:\eta=160$ and $H_1:\eta>160$ at significance level $\alpha=0.05$.
\end{example}

\begin{solution}
First we must assume that the distribution of the breaking strength is continuous.
\small
\[\begin{array}{cccccccccc} \hline
163 & 165 & 160 & 189 & 161 & 171 & 158 & 151 & 169 & 162 \\
+ & + & - & + & + & + & - & - & + & + \\ \hline
163 & 139 & 172 & 165 & 148 & 166 & 172 & 163 & 187 & 173 \\
+ & - & + & + & - & + & + & + & + & + \\ \hline
\end{array}\]
\normalsize
\bit
\it We have $n=20$ signs: under the null hypothesis, $S^{+}_n\sim\text{Binomial}(20,0.5)$.
\it The observed value of the test statistic $s^{+}_n=15$.
\it From tables we find that $\prob_{H_0}(S^{+}_n\geq 15) = 1 - 0.9793 = 0.0207$ approx.
\it Thus we reject $H_0$ at significance level $\alpha=0.05$.
\eit
\end{solution}

% example: sign test (paired samples)
\begin{example}[Sign test for paired samples]
To evaluate a new traffic-control system, the number of accidents that occurred at 12 dangerous junctions were recorded during the four weeks prior to the installation of the new system, and for the four weeks after its installation. The following data were obtained.
\small
\[\begin{array}{|l|rrrrrrrrrrrr|} \hline
\text{Junction}	& \phantom{1}1 & \phantom{1}2 & \phantom{1}3 & \phantom{1}4 & \phantom{1}5 & \phantom{1}6 & \phantom{1}7 & \phantom{1}8 & \phantom{1}9 & 10 & 11 & 12 \\ \hline
\text{Before}	& 3 & 5 & 2 & 3 & 3 & 3 & 0 & 4 & 1 &  6 &  4 &  1 \\
\text{After}	& 1 & 2 & 2 & 2 & 2 & 0 & 2 & 3 & 3 &  4 &  1 &  0 \\ \hline
\end{array}\]
\normalsize
Use a sign test to evaluate the claim that the new system is more effective than the old system.
\end{example}

\begin{solution}
Let $\eta_1$ and $\eta_2$ denote the median number of accidents before and after the new system was installed, respectively. We test $H_0:\eta_1=\eta_2$ against $H_1:\eta_1 > \eta_2$.
\[\begin{array}{|l|rrrrrrrrrrrr|} \hline
\text{Junction}	& \phantom{1}1 & \phantom{1}2 & \phantom{1}3 & \phantom{1}4 & \phantom{1}5 & \phantom{1}6 & \phantom{1}7 & \phantom{1}8 & \phantom{1}9 & 10 & 11 & 12 \\ \hline
%\text{Before}		& 3 & 5 & 2 & 3 & 3 & 3 & 0 & 4 & 1 &  6 &  4 &  1 \\
%\text{After}		& 1 & 2 & 2 & 2 & 2 & 0 & 2 & 3 & 3 &  4 &  1 &  0 \\ \hline
\text{Difference}	& + & + & 0 & + & + & + & - & + & - &  + &  + &  + \\ \hline
\end{array}\]

\bit
\it We have $n=11$ observations (one discarded) so $S^{+}_n\sim\text{Binomial}(11,\theta)$ under $H_0$.
\it The value of the test statistic is $s^{+}_n = 9$.
\it Under $H_0:\theta=0.5$, from tables we obtain $\prob_{0.5}(S^{+}_n\geq 9) = 1 - 0.9673 = 0.0327$.
\it At $\alpha=0.05$ we reject $H_0$ and conclude that the system has indeed reduced the number of accidents.
\it At $\alpha=0.01$ we retain $H_0$ and conclude that there is insufficient evidence to support the claim.
\eit
\end{solution}

%-----------------------------
\subsubsection{Normal approximation}

By the central limit theorem, if $X\sim\text{Binomial}(n,\theta)$ then for large $n$,
\[
X\sim N\big(n\theta,n\theta(1-\theta)\big) \text{\quad approx.}
\]
If $H_0:\theta=0.5$ is correct, the distribution of the test statistic is $S^{+}_n\sim N(n/2,n/4)$ approx.

% continuity correction
\begin{definition}[The continuity correction]
Let $X$ be a discrete random variable, taking values in the set $\{0,\pm 1,\pm 2,\ldots\}$. If the distribution of a continuous random variable $Y$ is taken as an approximation of the distribution of $X$, we set
\[
\prob(X=k) = \prob\left(k - \frac{1}{2} < Y < k + \frac{1}{2}\right).
\]
\end{definition}
This means that
\bit
\it $\prob(X < k) = \prob(Y\leq k-1/2)$ and $\prob(X\leq k) = \prob(Y\leq k+1/2)$,
\it $\prob(X\geq k) = \prob(Y\geq k-1/2)$ and $\prob(X > k) = \prob(Y\geq k+1/2)$
\eit

% example: sign test (large sample)
\begin{example}[Sign test for large samples]
The following data are the amounts of sulphur oxide (in tonnes) emitted by a large industrial plant over a period of 40 days.
\[\begin{array}{cccccccccc}
17 & 15 & 20 & 29 & 19 & 18 & 22 & 25 & 27 &  9 \\
24 & 20 & 17 &  6 & 24 & 14 & 15 & 23 & 24 & 26 \\
19 & 23 & 28 & 19 & 16 & 22 & 24 & 17 & 20 & 13 \\
19 & 10 & 23 & 18 & 31 & 13 & 20 & 17 & 24 & 14
\end{array}\]
Construct a sign test of size $\alpha=0.01$ to evaluate $H_0:\eta=21.5$ against $H_1:\eta<21.5$.
\end{example}

\begin{solution}
Assume that sulphur oxide emissions per day has a continuous distribution. The sign test statistic is
\[
S^{+}_n = \sum_{i=1}^n I(X_i>21.5).
\]
and because the sample is relatively large, $S^{+}_n\sim N(n/2,n/4)$ approx.

\bigskip
Using the continuity correction (for a lower-tailed test), we have the test statistic
\[
Z = \displaystyle\frac{(S^{+}_n+1/2)-n/2}{\sqrt{n/4}} \sim N(0,1) \text{ approx.}
\]
Here, we have $n=40$ and $s^{+}_n=16$ (the number values exceeding $\mu_0 = 21.5$), so the value of the test statistic is
\[
z = \frac{16.5 - 20}{\sqrt{10}} = -1.1068.
\]
From tables, the (lower-tail) critical value of $N(0,1)$ at $\alpha=0.01$ is $z_c = \Phi^{-1}(0.01) = -2.33$ approx. Thus we retain $H_0$ and conclude that the median amount of sulphur oxide emitted by the plant is not less than $21.5$ tons per day.
\end{solution}

\begin{exercise}
\begin{questions}

\question % flies
The biting rate of a particular species of fly was investigated. The biting rate is defined as the number of flies biting a volunteer during $15$ minutes of exposure. The species is known to have a median biting rate of $5$ bites per $15$ minutes. It is hypothesized that the median biting range is higher in bright, sunny weather. To test the hypothesis, a total of $122$ volunteers were exposed to flies on a sunny day, of which $95$ experienced biting rates greater than $5$.
State the null and alternative hypotheses for the test, and state your conclusion for $\alpha = 0.01$.
\begin{answer}
Let $\eta$ deonte the (true) median biting rate.
\par
The hypothesis test is $H_0:\eta=5$ against, $H_1:\eta>5$.
\par
The test statistic is $S_n^{+} = 95$ where $n=122$.

\bigskip
Under the null hypothesis, $\prob(S_{122}^{+}\geq 95) = \prob\big[\text{Binomial}(122,0.5) \geq 95\big]$.

\bigskip
Since $n$ is large, we use the normal approximation: under the null hypothesis,
\[
Z = \frac{(S_n^{+}-1/2) - n/2}{\sqrt{n/4}} \sim N (0,1) \quad\text{approx.}
\]
In this case, the test statistic is
\[
z = \frac{94.5 - 61}{\sqrt{30.5}} = 6.0659.
\]
An approximate $p$-value is $\prob(Z>6.6059) < 0.001$, so we reject $H_0$ at $\alpha=0.01$.
\end{answer}

\question % sign-test v z-test
Let $X\sim N (\mu,1)$ where $\mu$ is unknown and suppose we wish to test the simple null hypothesis $H_0:\mu=0$ against the simple alternative $H_1:\mu=0.5$. A random sample of 9 observations is taken from the distribution of $X$ and the number $S^{+}$ of positive values is counted.
\ben
\it % << (i)
A sign test rejects the null hypothesis if $S^{+}$ exceeds $6$. Find the size and power of the test.
\it % << (ii)
Construct a test based on the sample mean of the observations which has the same significance level as the sign test described above. Find the power of the test, and explain why this is higher than the power of the sign test.
\een

\begin{answer}
\ben
\it % << (i)
Under the null hypothesis, $S^{+}\sim\text{Binomial}(9, 0.5)$. The size of the test is
\[
\alpha = \prob_{\mu_0}(S^{+}>6) = \prob\big(\text{Binomial}(9,0.5)>6\big) \approx 0.0898 \text{ (from tables)}.
\]
Under $H_1:\mu=0.5$ we have $X\sim N(0.5,1)$, so the probability that an observataion takes a positive value under $H_1$ is
\[
\prob_{\mu_1}(X > 0)
	= \prob\big(X > 0 \text{ where } X\sim N(0.5,1)\big)
	= \prob\big(Z > -0.5 \text{ where } Z\sim N(0,1)\big)
	\approx 0.69146 \quad\text{(from tables).}
\]
The power of the test to detect the alternative $H_1:\mu=0.5$ is therefore
\[
\gamma(0.5) = \prob_{\mu_1}(S^{+}>6) = \prob\big[\text{Binomial}(9,0.6915) > 6\big] = \prob\big[\text{Binomial}(9,0.0.3085) < 3\big]
\]
where the second equality follows by the fact that
\[
\prob\big[\text{Binomial}(n,p) > k\big] = \prob\big[\text{Binomial}(n,1-p) < n-k\big].
\]
From tables, we find that
\[
\prob[\text{Binomial}(9,0.30)\leq 2\big] \approx 0.4628  \quad\text{and}\quad \prob[\text{Binomial}(9,0.35)\leq 2\big] \approx 0.3373.
\]
To find the required probability, we interpolate between these values:
\begin{align*}
\prob\big[\text{Binomial}(9,0.3085)\leq 2\big]
	& \approx \prob\big[\text{Binomial}(9,0.30)\leq 2\big] \\
	& \qquad + \left(\frac{0.3085 - 0.30}{0.35 - 0.30}\right)\left(\prob\big[\text{Bino}(9,0.35)\leq 2\big]-\prob\big[\text{Bino}(9,0.30)\leq 2\big]\right) \\
	& = 0.4628 - (0.1708\times 0.1255) \\
	& = 0.4414.
\end{align*}
Alternatively, we can use the normal approximation: if $Y\sim \text{Binomial}(9,0.6915)$ then $\expe(Y) = 6.2231$ and $\var(Y) = 1.9201$. Using the continuity correction,
\[
\prob\big[\text{Binomial}(9,0.6915) > 6\big]
	\approx \prob\left[ N (0,1) > \frac{6.5-6.2231}{\sqrt{1.9201}}\right]
	= \prob\big(N(0,1)> 0.1998\big)
%	= 1 - 0.5793
	\approx 0.4207.
\]
\item % (ii)
Let $\bar{X}$ denote the sample mean, and consider the $z$-test, where $H_0:\mu=0$ is rejected in favour of $H_1:\mu=0.5$ if $\bar{X} > c$, with $c$ chosen to give the required significance level. Here we require $\alpha = 0.0898$, so we need
\[
\prob_{\mu_0}(\bar{X} > c) = 0.0898.
\]
Under $H_0$ we have $X_i\sim N (0,1)$, so $\expe(\bar{X})=0$ and $\var(\bar{X})=1/9$. Thus $\bar{X}\sim N (0,1/9)$, so the critical value $c$ satisfies
\[
\prob\left( N (0,1) > \frac{c}{\sqrt{1/9}}\right) = 0.0898,
\]
i.e.\ $1-\Phi(3c) = 0.0898$, or $\Phi(3c) = 0.9102$. From tables, $\Phi(1.34076) = 0.91$ and $\Phi(1.40507)= 0.92$. Interpolating linearly between these values gives
\[
3c = 1.34076 + 0.02(1.40507 - 1.34076) = 1.342,
\]
so $c\approx 0.447$. The statistical power is
\begin{align*}
\gamma(0.5) = \prob_{H_1}(\bar{X} > 0.447)
	& = \prob\big(N(0.5,1/9) > 0.447\big) \\
	& = \prob\big(N(0,1) > 3(0.447 - 0.5)\big) \\
	& = \prob\big(N(0,1) > -0.159)\big) \\
	& \approx 0.564 \quad\text{(from tables).}
\end{align*}
This is higher than the power of the sign test (which is approximately $0.43$), because the $z$-test takes account of the magnitude of the observations, as well as their signs. If the data really do come from a normal distribution, the $z$-test is more powerful than the sign test for detecting $H_1:\mu=0.5$ against $H_0:\mu=0$.
\een
\end{answer}
\end{questions}
\end{exercise}

%-------------------------------------------------
\subsection{The Wilcoxon signed-rank test}\label{sec:wsr_test}

The Wilcoxon signed rank test is an extension of the sign test.
\bit
\it The sign test counts the \emph{number} of observations which are larger than some fixed value.
\it The WSR test also considers the \emph{relative size} of these observations.
\eit

%-----------------------------
%\subsubsection{Ranks}

\begin{definition}
For a random sample $X_1,\ldots,X_n$, the \emph{rank} of observation $X_i$ is its position in the sequence of observations sorted in ascending order:
\[
R(X_i) = \sum_{j=1}^n I(X_j \leq X_i)
\]
\end{definition}

Note that the sum of the ranks is always equal to the sum of the first $n$ positive integers:
\[
\displaystyle \sum_{i=1}^n R(X_i) = \sum_{i=1}^n i = \frac{1}{2}n(n+1).
\]

\begin{remark}[Ties]
If two or more observations have the same value, they are said to be \emph{tied}. If the distribution is continuous, ties occur with probability zero. Ties often occur in practical applications however, due to the limited precision of measurements. To deal with ties, the usual approach is to assign an \emph{average rank} to each of the tied observations. For example, if there are $m-1$ observations strictly smaller than $X_i=X_j$, we set
\[
R(X_i) = R(X_j) = \frac{m+(m+1)}{2} = m+\frac{1}{2}.
\]
\end{remark}

%-----------------------------
\subsubsection{The test statistic}

Let $X$ be a continuous random variable whose distribution is \emph{symmetric} and whose median $\eta$ is unknown, and let $X_1,X_2,\ldots,X_n$ be a random sample from the distribution of $X$.
Without loss of generality, we consider the null hypothesis $H_0:\eta=0$:
\bit
\it for a single sample $X_1,X_2,\ldots,X_n$, the null hypothesis $H_0:\eta=\eta_0$ reduces to $H_0:\eta=0$ by looking at the differences $D_i = X_i - \eta_0$;
\it For paired samples $X_1,X_2,\ldots,X_n$ and $Y_1,Y_2,\ldots,Y_n$, the null hypothesis $H_0:\eta_1=\eta_2$ reduces to $H_0:\eta=0$ by looking at the differences $D_i = X_i - Y_i$.
\eit

% definition: wsr statistic
\begin{definition}
For the null hypothesis $H_0:\eta=0$, the \emph{Wilcoxon signed rank} (WSR) statistic is
\[
W^{+}_n = \sum_{i=1}^n R_i Z_i \quad\text{where}\quad R_i = \sum_{j=1}^n I(|X_j|\leq |X_i|) \quad\text{and}\quad Z_i=I(X_i>0).
\]
\end{definition}
\bit
\it $R_i$ is the rank of $|X_i|$ in the sample of absolute values $|X_1|,|X_2|,\ldots,|X_n|$;
\it $Z_i=1$ if $X_i>0$, otherwise $Z_i=0$.
\eit

We also define the complementary statistic $W^{-}_n = \displaystyle\sum_{i=1}^n R_i(1-Z_i)$. Note that
\[
W^{+}_n + W^{-}_n = \sum_{i=1}^n R_i = \sum_{i=1}^n i = \frac{1}{2}n(n+1).
\]

%% test procedure
%The WSR test procedure is as follows:
%\ben
%\it Compute the differences $D_i = X_i - \eta_0$ (single sample) or $D_i=X_i-X'_i$ (matched pairs).
%\it Compute absolute differences $|D_i|$ and record the sign of $D_i$.
%\it Compute the ranks $R_i$ of the absolute differences $|D_i|$.
%\it Compute the the sum of the ranks of those $|D_i|$ having positive sign.
%\een
%The final step yields the test statistic $W^{+}_n$.

% example
\begin{example}
A study of the effects of smoking by mothers on the birthweight of their children involved 12 pairs of mothers. The pairs of mothers were selected so that they were as similar as possible, except that one mother of each pair was a smoker and the other was not. The birthweights (in kilograms) of the babies were as follows.
\small
\[\begin{array}{l|cccccccccccc} \hline
\text{Pair}			& 1      &  2     & 3      & 4      & 5      & 6      & 7      & 8      & 9      & 10     & 11     & 12 		\\ \hline
\text{Non-smoker}	& 3.22   & 4.48   & 3.90   & 3.47   & 3.07   & 3.23   & 4.25   & 3.31   & 3.33   & 3.78   & 3.18   & 4.60  	\\
\text{Smoker}		& 3.00   & 4.27   & 3.95   & 3.32   & 2.51   & 2.77   & 4.02   & 3.41   & 3.39   & 3.88   & 3.18   & 4.37 	\\ \hline
\end{array}\]
\normalsize
Do these data support the claim that mothers who smoke tend to have babies with smaller birthweights than those who do not?
\end{example}

\begin{solution}
The data are in matched pairs so we compute the differences $D_i=X_i-Y_i$, where $X_1,X_2,\ldots,X_n$ are the birthweights for non-smokers and $Y_1,Y_2,\ldots,Y_n$ are the corresponding birthweights for smokers. The test then becomes
\[
H_0:\eta = 0 \quad\text{against}\quad H_1:\eta > 0,
\]
where $\eta$ is the (true) median difference between birthweights for the non-smoking group and birthweights for the smoking group.

\small
\[\begin{array}{|c|cccccccccccc|} \hline
i			& 1      &  2     & 3      & 4      & 5      & 6      & 7      & 8      & 9      & 10     & 11     & 12   	\\ \hline
X_i			& 3.22   & 4.48   & 3.90   & 3.47   & 3.07   & 3.23   & 4.25   & 3.31   & 3.33   & 3.78   & 3.18   & 4.60 	\\
X'_i		& 3.00   & 4.27   & 3.95   & 3.32   & 2.51   & 2.77   & 4.02   & 3.41   & 3.39   & 3.88   & 3.18   & 4.37	\\ \hline
D_i			& 0.22   & 0.21   & -0.05  & 0.15   & 0.56   & 0.46   & 0.23   & -0.10  & -0.06  & -0.10  & 0.00  	& 0.23	\\
|D_i|		& 0.22   & 0.21   &  0.05  & 0.15   & 0.56   & 0.46   & 0.23   &  0.10  &  0.06  &  0.10  & 0.00  	& 0.23	\\ \hline
\text{sgn}(D_i)	& +      & +      & -      & +      & +      & +      & +      & -      & -      & -      &      	& +  	\\
%R(|D_i|)		& 8      & 7      & 2      & 6      & 12     & 11     & 9.5    &  4.5   &  3     &  4.5   & 0     	& 9.5  	\\ \hline
R(|D_i|)		& 7      & 6      & 1      & 5      & 11     & 10     & 8.5    &  3.5   &  2     &  3.5   &      	& 8.5  	\\ \hline
\end{array}\]
\vspace*{-2ex}\normalsize
\bit
\it The zero difference is discarded, leaving $n=11$ non-zero differences.
\it Ties are replaced by the average of the corresponding ranks.
\eit
From the data, $W^{+}_n = 56$ and $W^{-}_n = 10$. \Big[Check: $W^{+}_n + W^{-}_n = 66 = \frac{1}{2}n(n+1)$.\Big]
\bit
\it Large values of $W^{+}_n$ support the alternative hypothesis, $H_1:\eta > 0$.
\it From tables, the upper-tail critical value of $W^{+}_n$ for $n=11$ at $\alpha=0.05$ is $w_c = 52$.
\eit
Since $W^{+}_n > w_c$, we reject the null hypothesis and conclude that mothers who smoke tend to have babies with smaller birthweights than those who do not.
\end{solution}

%-----------------------------
\subsubsection{Mean and variance of $W^{+}_n$}

% theorem: mean and variance of W+
\begin{theorem}
Let $X_1,X_2,\ldots,X_n$ be a random sample from a continuous distribution whose density function is symmetric about its mean. Under the null hypothesis $H_0:\eta=0$,
\[
\expe(W^{+}_n) = \frac{n(n+1)}{4} \qquad\text{and}\qquad \var(W^{+}_n)	= \frac{n(n+1)(2n+1)}{24}.
\]
\end{theorem}

\begin{proof}
Recall that
\[
W^{+}_n = \sum_{i=1}^n R_i Z_i \quad\text{where}\quad R_i = \sum_{j=1}^n I(|X_j|\leq |X_i|) \quad\text{and}\quad Z_i=I(X_i>0).
\]
%
%$W^{+}_n = \sum_{i=1}^n R_iZ_i$, where $R_i=R(|X_i|)$ and $Z_i = I(X_i > 0)$.
Under $H_0:\eta=0$ we have $Z_i\sim\text{Bernoulli}(1/2)$ and hence $\expe(Z_i)=1/2$ and $\var(Z_i)=1/4$.

\begin{align*}
\expe(W^{+}_n)
	& = \sum_{i=1}^n R_i\expe(Z_i)
	= \frac{1}{2}\sum_{i=1}^n R_i
	= \frac{1}{2}\sum_{i=1}^n i
	= \frac{1}{4}n(n+1). \\
\var(W^{+}_n)
	& = \sum_{i=1}^n R_i^2 \var(Z_i)
	= \frac{1}{4}\sum_{i=1}^n R_i^2
	= \frac{1}{4}\sum_{i=1}^n i^2
	= \frac{1}{24}n(n+1)(2n+1).
\end{align*}
\end{proof}

%\begin{remark}
%\bit
%\it
%In practice, the sum of the squares of the ranks is equal to the sum of the squares of the first $n$ natural numbers only if there are no ties and no zeros (i.e.\ observations that are exactly equal to $\eta_0$).
%\it A correction for ties is
%\[
%\var(W^{+}_n) = \frac{1}{24}n(n+1)(2n+1) - \frac{1}{48}\sum t(t^2-1)
%\]
%where $t$ is the number of tied observations (e.g.\ $t=2$ if there are two identical observations) and the summation is over the number of sets of tied observations.
%\eit
%\end{remark}

%-----------------------------
\subsubsection{Normal approximation}
$W^{+}_n$ is the sum of random variables and (although the random variables are not independent) it satisfies the central limit theorem. Under $H_0:\eta=0$, and provided $n$ is sufficiently large, the distribution of $W^{+}_n$ is
\[
W^{+}_n \sim N\left(\frac{1}{4}n(n+1), \frac{1}{24}n(n+1)(2n+1)\right) \text{\quad approx.}
\]

The continuity correction should be applied when defining the test statistic.
\bit
\it Lower-talied test:
\[
Z = \frac{(W^{+}_n +\frac{1}{2})- \frac{1}{4}n(n+1)}{\sqrt{\frac{1}{24}n(n+1)(2n+1)}}\sim N(0,1) \quad\text{approx.}
\]
\it Upper-talied test:
\[
Z = \frac{(W^{+}_n -\frac{1}{2})- \frac{1}{4}n(n+1)}{\sqrt{\frac{1}{24}n(n+1)(2n+1)}}\sim N(0,1) \quad\text{approx.}
\]
\eit

%-----------------------------
\subsubsection{Exact distribution of $W^{+}_n$ under $H_0$}

The exact distribution of $W^{+}_n$ under $H_0$ can be obtained for small $n$ by a recurrence relation. The base case is $n=1$ (a single observation), for which $W^{+}_1\in\{0,1\}$ and under $H_0:\eta=0$,
\[
\prob(W^{+}_1 = 0) = \frac{1}{2} \quad\text{and}\quad \prob(W^{+}_1 = 1) = \frac{1}{2}.
\]
%
%
%\bit
%\it For $n = 2$, the set of ranks is $\{1,2\}$ so $W^{+}_2$ takes values in the set $\{0,1,2,3\}$. Under $H_0:\eta=0$,
%%\small
%\[\begin{array}{|c|cccc|}\hline
%\text{Ranks}			& \{-1,-2\}	& \{+1,-2\}	& \{-1,+2\}	& \{+1,+2\}	\\ \hline
%k					& 0	& 1	& 2	& 3 \\ \hline
%\prob(W^{+}_2=k)		& 0.25		& 0.25		&  0.25		& 0.25 		\\ \hline
%\end{array}\]
%%\normalsize
%
%\it For $n=3$, the set of ranks is $\{1,2,3\}$, so $W^{+}_3$ takes values in the set $\{0,1,2,3,4,5,6\}$. Under $H_0:\eta=0$,
%%\small
%\[\begin{array}{|c|ccccccc|}\hline
%k				& 0		& 1		& 2		& 3 		& 4 		& 5 		& 6		\\ \hline
%\prob(W^{+}_3=k)	& 0.125	& 0.125	& 0.125	& 0.25	& 0.125	& 0.125	& 0.125	\\ \hline
%\end{array}\]
%%\normalsize
%\eit

% theorem
\begin{theorem}
Under $H_0:\eta=0$,
\[
\prob\big(W^{+}_{n+1}=k\big) = \frac{1}{2}\prob\big(W^{+}_n=k\big) + \frac{1}{2}\prob\big(W^{+}_n=k-(n+1)\big).
\]
%for $k=0,1,\ldots,\displaystyle\frac{1}{2}n(n+1)$ (and zero otherwise).
for $k=0,1,\ldots,\frac{1}{2}n(n+1)$, and zero otherwise.
\end{theorem}

\begin{proof}
\bit
\it Let $X_1,X_2,\ldots,X_n$ be a random sample from a continuous and symmetric distribution.
\it Let $R_i$ be the rank of $|X_i|$ among the absolute values $|X_1|,|X_2|,\ldots,|X_n|$:
\eit

Then $W^{+}_n$ can be written as
\[
W^{+}_n = \sum_{r=1}^n rU_r \qquad\text{where}\qquad U_{R_i} = \begin{cases} 1 & X_i > 0, \\ 0 & \text{otherwise.}\end{cases}
\]

\bit
\it $U_r$ indicates that the observation associated with rank $r$ has a positive sign.
\it Under the null hypothesis, $U_1,U_2,\ldots,U_n$ is a random sample from the $\text{Bernoulli}(1/2)$ distribution.
\eit

Let $G_n(t)$ denote the probability generating function of $W^{+}_n$:
\begin{align*}
G_n(t)
	= \expe(t^{W^{+}_n})
	= \expe(t^{\sum_{r=1}^n rU_r})
	& = \expe(t^{U_1}t^{2U_2}\cdots t^{nU_n}) \\
	& = \prod_{r=1}^n \expe(t^{rU_r}) \qquad\text{(by independence)} \\
	& = \prod_{r=1}^n \Big[t^0\,\prob(U_r=0) + t^r\,\prob(U_r=1)\Big] \\
	& = \prod_{r=1}^n \frac{1}{2}(1 + t^r). \\
\intertext{Hence,}
G_{n+1}(t)
	& = \frac{1}{2}(1 + t^{n+1})G_n(t).
\end{align*}

By definition,
\begin{align*}
G_n(t)		= \sum_{k=0}^{\frac{1}{2}n(n+1)}\,\prob(W^{+}_n=k)t^k
\qquad\text{and}\qquad
G_{n+1}(t)	= \sum_{k=0}^{\frac{1}{2}(n+1)(n+2)}\,\prob(W^{+}_{n+1}=k)t^k.
\end{align*}

Thus,
\[
\sum_{k=0}^{\frac{1}{2}(n+1)(n+2)}\,\prob(W^{+}_{n+1}=k)t^k
	= \frac{1}{2}(1+t^{n+1})\sum_{k=0}^{\frac{1}{2}n(n+1)}\,\prob(W^{+}_n=k)t^k,
\]

Comparing the coefficients of $t^k$ yields the required recurrence relation:
\[
\prob\big(W^{+}_{n+1}=k\big) = \frac{1}{2}\prob\big(W^{+}_n=k\big) + \frac{1}{2}\prob\big(W^{+}_n=k-(n+1)\big).
\]
This recurrence relation is used to compute the quantiles of $W^{+}_n$ listed in statistical tables.
\end{proof}

\begin{exercise}
\begin{questions}

\question
We wish to test the hypothesis that two treatments $A$ and $B$ are equivalent, against the alternative hypothesis that the responses to treatment $A$ tend to be larger than the responses to treatment $B$. We perform a paired difference experiment, and analyse the resulting data using the Wilcoxon signed rank test. The data is shown in the following table.
\[\begin{array}{|l|cccccccccc|}\hline
\text{Pair}	&  1 &  2 &  3 &  4 &  5 &  6 &  7 &  8 &  9 & 10 \\ \hline
$A$ 			& 54 & 60 & 98 & 43 & 82 & 77 & 74 & 29 & 63 & 80 \\
$B$ 			& 45 & 45 & 87 & 31 & 71 & 75 & 63 & 30 & 59 & 82 \\ \hline
\end{array}\]
State the null and alternative hypotheses for the test, and conduct the test at significance level $\alpha=0.05$.

\begin{answer}
\ben
\it % << (i)
Let $D_i=A_i-B_i$ denote the differences, and let $\eta$ denote the (true) median difference. We test the null hypothesis $H_0:\eta=0$ against the one-sided alternative $H_1:\eta>0$.
\it % << (ii)
The signed ranks are computed as follows:
\[\begin{array}{|l|cccccccccc|}\hline
\text{Pair}			&  1 &  2 &  3 &  4 &  5 &  6 &  7 &  8 &  9 & 10 \\ \hline
\text{Difference}	&  9 & 15 & 11 & 12 & 11 &  2 & 11 & -1 &  4 & -2 \\ \hline
\text{Signed Rank}	&  5 & 10 &  7 &  9 &  7 &2.5 &  7 & -1 &  4 & -2.5 \\ \hline
\end{array}\]
\bit
\it The Wilcoxon signed rank statistics are $W^{+} = 51.5$ and $W^{-}=3.5$.
\it Check: $\frac{1}{2}n(n+1) = 55 = W^{+}+W^{-}$.
\it From tables, for a one-tailed test at $\alpha=0.05$ and $n=10$, the critical value is $w^{+}_c = 44$.
\it Thus we reject $H_0:\eta=0$ in favour of $H_1:\eta>0$, and conclude that responses to treatment $A$ tend to be larger than responses to treatment $B$.
\eit
\een
\end{answer}

\question
In a comparison of two populations $A$ and $B$, a paired difference experiment with $n=30$ pairs yields the Wilcoxon signed-rank statistic $w^{+}=354$.
\ben
\it Construct a test to determine whether or not population $A$ is located to the right of population $B$.
\it Conduct the test at $\alpha=0.05$.
\it Repeat part (2) using a normal approximation to the distribution of $W^{+}$.
\een

\begin{answer}
\ben
\it % << (i)
Let $\eta_A$ and $\eta_B$ denote the (true) medians of populations $A$ and $B$ respectively. To determine whether population $A$ is located to the right of population $B$, we test the null hypothesis $H_0:\eta_A=\eta_B$ against the alternative $H_1:\eta_A > \eta_B$. We could also define $\eta=\eta_A-\eta_B$ to be the difference between the two medians, in which case we would test $H_0:\eta=0$ agaianst $H_1:\eta>0$.

\it % << (ii)
From tables, for a one-tailed test at $\alpha=0.05$ and $n=30$, the critical value is $w^{+}_c = 313$, so we reject $H_0$ in favour of $H_1$.

To compute an approximate $p$-value for the test, from tables (for $n=30$) we see that $w^{+}_c = 345$ at $\alpha=0.01$ and $w^{+}_c = 356$ at $\alpha=0.005$. Interpolating between these values,
\[
\prob_{H_0}(W^{+}\geq 354) \approx 0.01 + \left(\frac{354-345}{356-345}\right)(0.005 - 0.01) = 0.0059.
\]

\it % << (ii)
The normal approximation is $\prob(W^{+}\geq w^{+}_c) \approx \prob(Z\geq z_c)$ where
\[
Z = \frac{(W^{+}-1/2) - \expe(W^{+})}{\sqrt{\var(W^{+})}} \sim N (0,1) \quad\text{approx.},
\]
Here, $\expe(W^{+})$ and $\var(W^{+})$ are respectively the mean and variance of $W^{+}$ under $H_0$.
\par
In this case,
\[
\expe(W^{+}) = \frac{1}{4}n(n+1) = 232
\quad\text{and}\quad
\var(W^{+})  = \frac{1}{24}n(n+1)(2n+1) = 2363,
\]
so the test statistic is
\[
z = \frac{353.5 - 232}{\sqrt{2363}} = 2.4994.
\]
\bit
\it At $\alpha=0.05$ we have $z_c = 1.645$, so we reject $H_0$.
\it From tables, an approximate $p$-value for the test is $1-0.99379 = 0.0062$.
\eit
\een
\end{answer}

\question
To test whether or not a new diet actually results in weight loss, a researcher recruited nine subjects, measured their weight (in kilograms) before starting the diet, and then again after two months on the diet. The results are shown in the following table.
\[\begin{array}{|l|ccccccccc|} \hline
\text{Case}			&  1 &  2 &  3 &  4 &  5 &  6 &  7 &  8 &  9 \\ \hline
\text{Weight before }	& 55 & 50 & 54 & 43 & 62 & 61 & 56 & 48 & 53 \\
\text{Weight after }		& 47 & 40 & 52 & 43 & 51 & 62 & 50 & 47 & 56 \\ \hline
\end{array}\]

\begin{parts}
\part
Use a sign test with $\alpha=0.05$ to decide whether or not the diet results in weight loss.
\begin{answer}
Let $\eta$ denote the (true) median weight loss.
\par
We evaluate $H_0:\eta = 0$ against the alternative $H_0:\eta > 0$.
\[\begin{array}{|l|ccccccccc|} \hline
\text{Case } (i)			&  1 &  2 &  3 &  4 &  5 &  6 &  7 &  8 &  9 \\ \hline
\text{Weight before }	& 55 & 50 & 54 & 43 & 62 & 61 & 56 & 48 & 53 \\
\text{Weight after }		& 47 & 40 & 52 & 43 & 51 & 62 & 50 & 47 & 56 \\ \hline
\text{Difference } (D_i)	&  8 & 10 &  2 &  0 & 11 & -1 &  6 &  1 & -3 \\ \hline
\text{Sign }				&  + &  + &  + &    &  + &  - &  + &  + &  - \\ \hline
%\text{Rank } (R_i)		&  6 &  7 &  3 &    &  8 &1.5 &  5 &1.5 &  4 \\ \hline
\end{array}\]
\bit
\it Exclude the zero difference, and take the sample size to be $n=8$.
\it $S_n^{+} = \sum_i I(D_i>0) = 6$.
\it $S_n^{-} = \sum_i I(D_i<0) = 2$.
\it Check: $S_n^{+} + S_n^{-} = n$.
\eit
Under $H_0$, $S_n^{+}\sim\text{Binomial}(n,0.5)$. From tables,
\[
\prob_{H_0}(S_n^{+}\geq 6) = 1 - \prob_{H_0}(S_n^{+}\leq 5) = 1 - 0.8555 = 0.1445.
\]
At $\alpha=0.05$, we would decide to retain the null hypothesis $H_0:\eta=0$, and conclude that the diet results in no weight loss.
\end{answer}

\part
Repeat the analysis using the Wilcoxon signed rank test.
\begin{answer}
\[\begin{array}{|l|ccccccccc|} \hline
\text{Difference } (D_i)	&  8 & 10 &  2 &  0 & 11 & -1 &  6 &  1 & -3 \\ \hline
\text{Sign }				&  + &  + &  + &    &  + &  - &  + &  + &  - \\ \hline
\text{Rank } (R_i)		&  6 &  7 &  3 &    &  8 &1.5 &  5 &1.5 &  4 \\ \hline
\end{array}\]
\bit
\it $W^{+} = \sum_i R_i I(D_i>0) = 6 + 7 + 3 + 8 + 5 + 1.5 = 30.5$.
\it $W^{-} = \sum_i R_i I(D_i<0) = 1.5 + 4 = 5.5$.
\it Check: $W^{+} + W^{-} = \frac{1}{2}n(n+1)$.
\eit
From tables, critical values for an upper-tail test are
\bit
\it $w^{+}_c = 30$ at $\alpha=0.05$, and
\it $w^{+}_c = 32$ at $\alpha=0.025$.
\eit
Thus at $\alpha=0.05$ we would reject the null hypothesis $H_0:\eta = 0$ in favour of the alternative $H_0:\eta > 0$, and conclude that the diet does indeed result in weight loss.
\end{answer}

\part
Briefly discuss the reasons why the tests lead to different conclusions.
\begin{answer}
The sign test takes no account of the fact that the average amount of weight lost by those whose weight decreased over the study period, is significantly greater than the average amount of weight gained by those whose weight increased over the study period.
\end{answer}
\end{parts}

\end{questions}
\end{exercise}


If you haven't downloaded and unzipped \href{https://libaoj.in/courses/2021f/MATH3341/zip/Math.3341.zip}{\texttt{Math.3341.zip}}. Download and unzip it under \verb|H:| (H Drive if you are working on the Remote Lab). Change the current working directory by typing \verb|cd H:\Math.3341\Math.3341.Lab.14| in the Command Window, and type \verb|edit lab_14_script| in the Command Window to edit \verb|lab_14_script.m|.

%---------------------------------------------
\section{Direction Fields and Solution Curves}
%---------------------------------------------
Given the following ODE and the initial condition,
$$
\frac{dy}{dt} = -y - 5 e^{-t} \sin(5t), \quad y(0) = -2, \quad t \in [0, 3].
$$
\begin{enumerate}[(a)]
    \item Define anoymous function \verb`dydt` to be the right-hand side of the ODE.
    \item Define \verb`a`, \verb`b` to be the left and right endpoint of the interval $[0, 3]$, respectively.
    \item Define \verb`t_step` to be the step size $\Delta t = 0.01$.
    \item Define \verb`t_span` to be a vector starting from \verb`a` to \verb`b` with step size \verb`t_step` using colon notation.
    \item Use \verb`ode23` to solve the ODE.
    \item Plot \verb`y_sol(:, i)` versus \verb`t_sol(:, i)` with line style specified in the cell array \verb`LineStyle`.
    \item Run the script and see whether it works. If it does work, add more initial conditions to \verb|y0|: $y(0) = 0$, $y(0) = 2$, $y(0) = 4$.
\end{enumerate}
%---------------------------------------------
\section{System of ODEs}
%---------------------------------------------
Next, use the built-in ODE solver \verb`ode45` to solve the following system of ODEs:
$$
\begin{cases}
    y'_1(t) = y_3, \\
    y'_2(t) = y_4, \\
    y'_3(t) = -2 y_1 + (3/2) y_2, \\
    y'_4(t) = (4/3) y_1 - 3 y_2, \\
\end{cases}
\quad
\mathbf{y}(0) =
\begin{bmatrix}
    -1 \\ 4 \\ 1 \\ 1
\end{bmatrix},
\quad
t \in [0, 15].
$$
\begin{enumerate}[(a)]
    \item Define an anoymous function (you can refer to the example in reference page for \verb`ode45`. To open the reference page, type \verb`doc ode45` in the Command Window).
    \item Repeat the steps in Part 1 to define \verb`a`, \verb`b`, \verb`t_step`, \verb`t_span`, and \verb`y0`.
    \item Use \verb`ode45` to solve the system of ODEs.
    \item Plot \verb`y(:, i)` versus \verb`t` with line style specified in the cell array \verb`LineStyle`.
    \item Plot \verb`y(:, 3)` versus \verb`y(:, 1)`.
\end{enumerate}
%---------------------------------------------
At last, run the script \verb|lab_14_script.m|. Upload the script file \verb|lab_14_script.m|, and figure files \verb|lab_14_figure_1.pdf|, \verb|lab_14_figure_2.pdf|, \verb|lab_14_figure_3.pdf| to Overleaf. Recompile, and submit the generated .pdf file on WyoCourses.

\begin{figure}[!hbtp]
  \centering
  \includegraphics[height=0.3\textheight]{../Math.3341.Lab.14.ans/lab_14_figure_1.pdf}
  \caption{Expected result for Direction Fields and Solution Curves}
  \label{fig:}
\end{figure}

\begin{figure}[!hbtp]
  \centering
  \includegraphics[height=0.3\textheight]{../Math.3341.Lab.14.ans/lab_14_figure_2.pdf}
  \includegraphics[height=0.3\textheight]{../Math.3341.Lab.14.ans/lab_14_figure_3.pdf}
  \caption{Expected result for System of ODEs}
  \label{fig:}
\end{figure}


\lab{Applications}{Image Segmentation}{Image Segmentation}

\objective{Understand some basic applications of Eigenvalues to graph theory}
\label{lab:ImgSeg_eigenvalues}

\section*{Graph Theory}
\begin{figure}[h]
\includegraphics[width=\textwidth/2]{graphExample}
\caption{A simple graph}
\label{fig:example_graph}
\end{figure}

Graphs are often used to represent relationships between objects.
They are represented by a set of nodes (or vertices) and a set of edges where each edge connects exactly two nodes.
We denote the number of vertices in a graph by $|V|$, and the number of edges by $|E|$.
A graph is \emph{directed} if connections are uni-directional, and \emph{undirected} if they are bi-directional.
One way to encode all the information in this picture is to use what is called an adjacency matrix.
\begin{definition} An adjacency matrix $A$ is a $|V| \times |V|$ matrix where the $(i,j)$-th entry $a_{ij}$ is
\begin{center}
	$a_{ij} = \begin{cases} 1 & \mbox{If an edge connects vertex i to vertex j} \\ 0 & \mbox{otherwise} \end{cases}$
\end{center}
%technically, the definition does not apply to all graphs (according to Wikipedia)
%however, this definition works if we are just considering simple graphs, i.e. at most one edge between any two vertices, no loops
\end{definition}

Every undirected edge can be represented as two directed edges.
Figure \ref{fig:example_graph} shows a simple undirected graph.
We can represent the connections of nodes as a matrix called an \emph{adjacency matrix}.
The graph in Figure \ref{fig:example_graph} can be represented as the following adjacency matrix.
\[
A = \begin{pmatrix}
0 & 1 & 0 & 0 & 1 & 0\\
1 & 0 & 1 & 0 & 1 & 0\\
0 & 1 & 0 & 1 & 0 & 0\\
0 & 0 & 1 & 0 & 1 & 1\\
1 & 1 & 0 & 1 & 0 & 0\\
0 & 0 & 0 & 1 & 0 & 0
\end{pmatrix}
\]
We order the nodes and each node represents one column and one row of the adjacency matrix.
If an edge exists between node $i$ and node $j$ then the $A_{ij}$ entry is 1.
For undirected graphs, the adjacency matrix will always be symmetric.
The diagonal represents self-edges, or edges that connect a node to itself.
While adjacency matrices are useful, many graph algorithms depend on another representation of a graph called an \emph{adjacency list}.

Raising the adjacency matrix to a power yields some very interesting information.
We can discover the number of paths of length $n$ between two nodes by raising a graph's adjacency matrix to the $n$th power.
For example, by squaring $A$, we can find the number of paths of length 2 between every pair of nodes.
\begin{lstlisting}
>>> np.linalg.matrix_power(A,2)
array([[2, 1, 1, 1, 1, 0],
       [1, 3, 0, 2, 1, 0],
       [1, 0, 2, 0, 2, 1],
       [1, 2, 0, 3, 0, 0],
       [1, 1, 2, 0, 3, 1],
       [0, 0, 1, 0, 1, 1]])
\end{lstlisting}
We can see that no paths of length 2 exist between node 0 and node 5 because $A^2_{0,5} = 0$.
By calculating $A^6$ we can find the number of paths length 6 from node 3 to itself.
\begin{lstlisting}
>>> np.linalg.matrix_power(A, 6)
array([[45, 54, 38, 45, 54, 16],
       [54, 86, 29, 77, 51, 11],
       [38, 29, 55, 15, 70, 27],
       [45, 77, 15, 75, 31,  4],
       [54, 51, 70, 31, 93, 34],
       [16, 11, 27,  4, 34, 14]])
\end{lstlisting}
We see that there are 55 unique paths of length 6 from node 3 to itself.
Imagine trying to count all of those paths by hand!
It would be very easy to count incorrectly.
However, this method makes it very simple to count paths without any mistakes.

Adjacency matrices can also be composed of \li{True} and \li{False} values.
In this case, the $n$th power of such a matrix (using boolean arithmetic)
is again a matrix of
boolean values which simply indicate whether there exists a path of length $n$ between the given pair of nodes, rather than indicating the number of such
paths.

\begin{problem}
Let the following matrix represent a directed graph
\[
\begin{pmatrix}
0 & 0 & 1 & 0 & 1 & 0 & 1 \\
1 & 0 & 0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 & 0 & 1 & 0 \\
1 & 0 & 0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 & 0 & 1 \\
0 & 1 & 0 & 0 & 0 & 0 & 0
\end{pmatrix}
\]
Between which pair of nodes does there exist the greatest number of paths
of length five?
From which node to which node is there no path of length seven?
\end{problem}

Another useful way of representing an undirected graph is a Laplacian matrix.
This special matrix can reveal a lot of information about a graph.
\begin{definition}
For a simple graph (an unweighted, undirected graph without self-edges), the Laplacian of a graph $G$ is
\[ L_G = D_G - A_G \]
where $D_G$ is the degree matrix of $G$ and $A_G$ is the adjacency matrix of $G$.
\end{definition}
An important question in working with graphs is the degree of its nodes.
The degree of a node is the number of edges that connect to the node.
For a directed graph, each node has an \emph{out-degree} (the number of edges directed away from a node) and an \emph{in-degree} (the number edges directed toward a node).
The degree matrix of a graph is a diagonal matrix that has the degree of each node as the diagonal entries (thus, for a directed graph there in an in-degree matrix and an out-degree matrix).

\begin{problem}
Calculate the Laplacian matrix of the graph in Figure \ref{fig:example_graph} by calculating $D$ and $A$.
Think about how to find the number of neighboring nodes using an adjacency matrix.

Write a general solution that will calculate the Laplacian of any small graph.
Use the adjacency matrix to check that the graph is unweighted, undirected, and contains no self-edges.
\label{prob:laplacian}
\end{problem}

We may also define weighted graphs so that each edge has a weight or cost attached to it.
In this case the adjacency matrix will have the cost in each entry, rather than just a 1 or 0.
An example of a weighted may be a collection of cities as vertices, roads connecting them as edges, and the distance of the connecting roads as the cost attached to each edge.

In this lab we will be investigating the sorts of things that we can discover about a graph by examining the spectrum of its adjacency matrix.
While the formulation of the adjacency matrix seems to be very simple, its structure allows us to learn surprising things by examining its eigenvalues.

For a weighted graph the definition of the Laplacian of a graph is:

\begin{definition}  The \emph{degree} of a vertex of a weighted graph is the sum of the weights of the edges connected to that vertex.
In undirected graphs, which is what we will be dealing with, this is the sum of all the weights moving into that vertex.
This definition varies slightly for directed graphs.
Let $D$ be a diagonal matrix with
\[
D_{ii} = \mbox{ Degree of vertex $i$}
\]
and let $A$ be the adjacency matrix of the graph.
The graph \emph{Laplacian} $Q$ is given by
\[
Q = D-A
\]
\end{definition}

The Laplacian matrix of a graph will typically be very sparse.
We will be examining the spectrum of the Laplacian matrix.

A connected graph is a graph where every vertex is connected to every other vertex by at least one path.
An important question one might ask about a graph is whether or not it is connected.
What is the best way to do this?
A naive approach would be to exhaustively map every possible path from each vertex.
While this would be feasible for very small graphs, most interesting graphs (for example, the internet) will have thousands of vertices and such an approach becomes essentially impossible to execute.

It turns out there is a better way.
If the second smallest eigenvalue of the Laplacian matrix associated with a graph is positive, then this graph is connected.
The mathematics behind this is quite involved, so we omit a proof for now.
In many applications the Laplacian matrix will be very sparse, and thus with optimizations made for sparsity, we can discover the connectivity of a graph relatively cheaply.

\begin{problem}Write a function \li{laplacian} that accepts an adjacency matrix as an argument and returns the Laplacian matrix and its second smallest eigenvalue.
Use the \li{scipy.linalg} package to compute this eigenvalue.
Using {\tt numpy.rand(n,n)}, generate several random matrices of various sizes.
Use masking to change the sparsity of said matrices, i.e. {\tt numpy.rand(n,n) > c} for {\tt c = 0.25,0.5,0.75}.
Try different distributions and $c$ values.
What can we expect about the connectivity of random matrices?
\end{problem}

A related problem is found in image processing.
An image is a collection of coordinates and light intensities.
We call each coordinate and associated brightness a pixel.
We let every pixel in an image be a vertex in a graph that is connected to its neighbors within a certain radius.
For an $N \times N$ image, we define an adjacency matrix as follows:

\begin{equation}
\label{eq:adjacency}
w_{ij} = e^{-\frac{|I(i) - I(j)|}{\sigma_I^2}} \cdot \begin{cases} e^{-\frac{d(i,j)}{\sigma_d^2}} & \mbox{ for $d(i,j) < r$} \\ 0 & \mbox{ otherwise} \end{cases}
\end{equation}
for $i = 1 \hdots N^2$ and $j = 1 \hdots N^2$ (traveling through each pixel in the image), where
\begin{itemize}
	\item$d(i,j)$ is the Euclidean distance between pixel $i$ and pixel $j$
	\item $|I(i) - I(j)|$ is the difference in brightness of pixels $i$ and $j$
	\item $\sigma_I$ and $\sigma_d$ are constants
\end{itemize}
Thus, given a $N\times N$ image, we will produce an adjacency matrix of size $N^2\times N^2$.
Even for smaller images, this will become very large, but we have sparsity on our side.
Even so, only do this for images smaller than 50 by 50.
Since each pixel is only connected to the pixels within radius $r$ around it, relatively few entries will be non-zero.

\begin{problem}
Write a function \li{adjacency} that takes an $N \times N$ image array, radius $r$, and values for
$\sigma_I^2$ and $\sigma_d^2$, and returns the adjacency matrix defined in \eqref{eq:adjacency}.
Notice that for each pixel you can save time by only checking the pixels $r$ rows and columns away.
%For that you'll have to handle the pixels on the edges and corners of the image carefully.
%I gave them new helper code, which abstracts away the edge cases.
Make use of sparsity in order to feasibly store such a large matrix.
We have provided code to help out with this problem.
\end{problem}

An important problem in image processing is that of image segmentation.
When humans observe an image, they can easily pick out portions of an image that ``belong together.''
For example, if we saw the picture of a person against a black background, the pixels making up the background would make up one segment of an image, and the person would make up the other part.
While this seems simple for a human, how can we program a computer to do so?

Once again, we turn to the eigenvalues of the Laplacian matrix.
We examine the second smallest eigenvalue of $D^{-\frac{1}{2}}QD^{-\frac{1}{2}}$, where $D$ is the degree matrix and $Q$ is the Laplacian matrix of the adjacency matrix defined in \eqref{eq:adjacency}.
The associated $N^2 \times 1$ eigenvector will have positive and negative entries, splitting the image into two parts.
Thanks to some very fancy mathematics and the way we have defined the weights between nodes, this does a pretty good job of breaking an image into two segments.
Recursively executing this function, we can find segments within segments.
%Figure \ref{segmentation:example} shows a simple example of an image and its segments.

%\begin{figure}
%\includegraphics[scale=0.2]{monument}
%\includegraphics[scale=0.2]{segment1}
%\includegraphics[scale=0.2]{segment2}
%\caption{An image and it's segments.}
%\label{segmentation:example}
%\end{figure}

\begin{problem}  Write a function \li{segment} that solves the segmentation problem for small images.
Accept an image array as an argument and return the two segments.
Use $r = 5, \sigma_I^2 = 0.02,$ and $\sigma_d^2 = 3.0$.
Remember that the Laplacian matrix will be very large but also very sparse.
Because of the way we defined $\sigma_I^2$ the image matrix intensity values need to be between $0.0$ and $1.0$.
\end{problem}


\section{Stochastic gradient descent method and convergence theory}
The next optimization problem is the most common case in machine learning.
\begin{problem}
	\begin{equation}\label{SGDproblem}
 	\min_{x \in \mathbb{R}^n} f(x)\quad \mbox{and}\quad f(x) = \frac{1}{N} \sum_{i=1}^N f_i(x).
	\end{equation}
\end{problem}
One version of stochastic gradient descent (SGD) algorithm is:
\begin{algorithm}\caption{SGD}
\label{alg:SGD}
{\bf Input}: initialization $x_0$, learning rate $\eta_t$.

{\bf For}: t = 0,1,2,$\dots$

Randomly pick $i_t \in \{1, 2, \cdots, N\}$ independently with probability $\frac{1}{N}$
\begin{equation}\label{equ:sgd-iteration}
x_{t+1} = x_{t} - \eta_t \nabla f_{i_t}(x_t).
\end{equation}
\end{algorithm}

\subsection{Convergence of SGD}
\begin{theorem}Assume that each $f_i(x)$ is $\lambda$-strongly convex and $\|\nabla f_i(x)\| \le M$ for some $M >0$.
	If we take $\eta_t = \frac{a}{\lambda (t+1)}$ with sufficiently large $a$ such that
	\begin{equation}\label{key}
	\|x_0 - x^*\|^2 \le \frac{a^2M^2}{(a-1)\lambda^2}
	\end{equation}
then
\begin{equation}
\mathbb{E}e_{t}^2 \le \frac{a^2M^2}{(a-1)\lambda^2 (t+1)}, \quad  t\ge 1,
\end{equation}
where $e_t = \|x_t - x^*\|$.
\end{theorem}

\begin{proof}
The $L^2$ error of SGD can be written as
\begin{equation}
      \label{equ:L2SGD}
      \begin{split}
            \mathbb{E} \|x_{t+1} - x^*\|^2 &\le \mathbb{E}\| x_{t} - \eta_t \nabla f_{i_t}(x_t) - x^* \|^2 \\
            &\le \mathbb{E} \|x_t - x^*\|^2
            - 2 \eta_t \mathbb{E} (\nabla f_{i_t}(x_t) \cdot (x_t - x^*))
            + \eta_t^2 \mathbb{E} \|\nabla f_{i_t}(x_t)\|^2 \\
            & \le \mathbb{E} \|x_t - x^*\|^2 - 2 \eta_t \mathbb{E} (\nabla f (x_t) \cdot (x_t - x^*))
            + \eta_t^2 M^2 \\
            & \le \mathbb{E} \|x_t - x^*\|^2 -  \eta_t \lambda \mathbb{E} \|x_t - x^*\|^2 + \eta_t^2 M^2 \\
            & = (1 - \eta_t\lambda) \mathbb{E} \|x_t - x^*\|^2 + \eta_t^2 M^2
      \end{split}
\end{equation}
The third line comes from the fact that
\begin{equation}\label{key}
\begin{aligned}
\mathbb{E} (\nabla f_{i_t}(x_t) \cdot (x_t - x^*))  &= \mathbb{E}_{i_1i_2\cdots i_t} (\nabla f_{i_t}(x_t) \cdot (x_t - x^*)) \\
&= \mathbb{E}_{i_1i_2\cdots i_{t-1}} \frac{1}{N} \sum_{i=1}^N \nabla f_i(x_t)\cdot (x_t - x^*) \\
&= \mathbb{E}_{i_1i_2\cdots i_{t-1}}  \nabla f(x_t)\cdot (x_t - x^*) \\
&= \mathbb{E}\nabla f(x_t)\cdot (x_t - x^*),
\end{aligned}
\end{equation}
and
\begin{equation}
\mathbb{E} \|\nabla f_{i_t}(x_t)\|^2 \le \mathbb{E} M^2 = M^2.
\end{equation}
Note when $t=0$, we have
\begin{equation}\label{key}
\mathbb{E} e_0^2 = \|x_0 - x^*\|^2 \le \frac{a^2M^2}{(a-1)\lambda},
\end{equation}
based on the assumption.

In the case of SDG, by the inductive hypothesis,
\begin{equation}
      \begin{split}
            \mathbb{E}e_{t+1}^2 & \le (1 - \eta_t\lambda)\mathbb{E}e_{t}^2  + \eta_t^2 M^2\\
            &\le  (1 - \frac{a}{t+1}) \frac{a^2M^2}{(a-1)\lambda^2 (t+1)} + \frac{a^2M^2}{\lambda^2 (t+1)^2} \\
            & \le \frac{a^2M^2}{(a-1)\lambda^2} \frac{1}{(t+1)^2}(t+1 -a + a-1) \\
            & = \frac{a^2M^2}{(a-1)\lambda^2} \frac{t}{(t+1)^2} \\
            & \le \frac{a^2M^2}{(a-1)\lambda^2(t+2)}. \quad \left(\frac{t}{(t+1)^2} \le \frac{1}{t+2}\right),
      \end{split}
\end{equation}
which completes the proof.
\end{proof}

\subsection{SGD with mini-batch}
Firstly, we will introduce a natural extended version of the SGD discussed above with introducing mini-batch.

\begin{algorithm}[H]\caption{SGD with mini-batch}
	\label{alg:SGD}
	{\bf Input}: initialization $x_0$, learning rate $\eta_t$.

	{\bf For}: t = 0,1,2,$\dots$

	\begin{center}
	Randomly pick $B_t \subset \{1, 2, \cdots, N\}$ independently
	with probability $\frac{m!(N-m)!}{N!}$ \\
	and $\# B_t = m$.% with $\frac{1}{\tbinom{N}{m}} = $.
	\end{center}
	\begin{equation}\label{equ:sgd-iteration}
	x_{t+1} = x_{t} - \eta_t g_t(x_t).
	\end{equation}
	\quad where
	$$
	g_{t}(x_t) = \frac{1}{m} \sum_{i \in B_{t}}  \nabla f_i(x_t)
	$$
\end{algorithm}

Now we introduce the SGD algorithm with mini-batch without replacement which is the most commonly used version of SGD in machine learning.
\begin{algorithm}[H]
	\caption{Shuffle SGD with mini-batch}
	\label{alg:mini-batch}
	{\bf Input}: learning rate $\eta_k$, mini-batch size $m$, parameter initialization $ x_{0}$ and denote $M = \lceil \frac{N}{m} \rceil$.

	{\bf For} Epoch $k = 1,2,\dots$

	\begin{center}
	Randomly pick $B_t \subset \{1, 2, \cdots, N \}$ without replacement \\
	with $\# B_t = m$ for $t = 1,2,\cdots,M$.
	\end{center}

	\quad{\bf For} mini-batch $t = 1:M$

	\quad Compute the gradient on $B_{t}$:
	$$
	g_{t}(x) = \frac{1}{m} \sum_{i \in B_{t}}  \nabla f_i(x)
	$$
	\quad Update $x$:
	\begin{equation*}
	x  \leftarrow  x - \eta_k g_t(x),
	\end{equation*}
	\quad {\bf EndFor}

	{\bf EndFor}
\end{algorithm}

To "randomly pick $B_i \subset \{1, 2, \cdots, N \}$ without replacement
with $\# B_i = m$ for $i = 1,2,\cdots,t$'', we usually just randomly shuffle the index set first and
then consecutively pick every $m$ elements in the shuffled index set. That is the reason why we
would like to call the algorithm as shuffled SGD while this is the mostly used version of SGD in machine learning.

\begin{remark}
Let us recall a general machine learning loss function
\begin{equation}\label{key}
L(\theta) = \frac{1}{N}\sum_{i=1}^N \ell(h(X_i; \theta), Y_i),
\end{equation}
where $\{(X_i, Y_i)\}_{i=1}^N$ correspond to these data pairs.
For example, $\ell(\cdot, \cdot)$ takes cross-entropy and $h(x; \theta) = \bm p(x;\theta)$ as we discussed in Section \ref{sec:LR}.
Thus, we have the following corresponding relation
\begin{equation*}
f(x) \leftrightsquigarrow L(\theta), \quad
f_i(x) \leftrightsquigarrow \ell(h(X_i; \theta), Y_i).
\end{equation*}
\end{remark}

\endinput
Here we can take the expectation of $i_t$ independently from $\{x_i, i = 1,\ldots,t\}$
to obtain the second line of (\ref{equ:L2SGD}) since $i_t$ is independent from
$\{x_i, i = 1,\ldots,t\}$ which is completely determined by $\{i_j, j = 1,\ldots,t - 1\}$.
And the third line of (\ref{equ:L2SGD}) is obtained from the boundness of the gradients.

Here shuffle  index and get mini-batch $B_1, \cdots, B_{\frac{N}{m}}$ means that $\# B_i = m$ and
\begin{equation}\label{key}
\{1, 2, \cdots, N\} = \cup_{i = 1}^{\frac{N}{m}} B_i,  \quad B_i \cap B_j = \emptyset
\end{equation}
where $B_i$ is chosen from $\{1, 2, \cdots, N\}$ randomly.


\subsection{Stacking and SuperLearner}

\subsubsection{Introduction}

With stacking we take the predictions from each of our classifiers, and then train a new model using these predictions as inputs.

\subsubsection{Hard and soft inputs}

Hard classification (\(0\) or \(1\)) and soft classification (between \(0\) and \(1\)) can be used as inputs.

\subsubsection{Cross validation}

We can select hyper-parameters using cross validation, however there is an issue of using cross validation twice on the same data. Once for the underyling classifiers, and again for the stacked model.

\subsubsection{SuperLearner}

We have \(m\) models.

Part 1: Train each of them on all data.

Part 2: Split the data into k sets

For each set, associate all other data as training

For each fold:
\begin{itemize}
\item Fit each model on the training data
\item Predict on other
\item Create weighted predictor. choose weightings to minimise error
\end{itemize}

Part 3: Use these weightings on the original (unrestricted data) model


\subsection{$LU$ factorization, multiplier method}

Remember that for a matrix $A$ to be written in the form $A=LU$, you must be able to reduce it to its {\ef} without interchanging rows. The following method gives a process for calculating the $LU$ factorization of such a matrix $A$.

\begin{example}{$LU$ factorization}{lu-factorization}
Find an $LU$ factorization for
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 2 & 3 \\
2 & 3 & 1 \\
-2 & 3 & -2
\end{mymatrix}
\end{equation*}
\end{example}

\begin{solution}

Write the matrix as the following product.
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 2 & 3 \\
2 & 3 & 1 \\
-2 & 3 & -2
\end{mymatrix}
\end{equation*}

In the matrix on the right, begin with the left row and zero
out the entries below the top using the row operation which involves adding
a multiple of a row to another row. You do this and also update the matrix
on the left so that the product will be unchanged. Here is the first step.
Take $-2$ times the top row and add to the second. Then take $2$ times the
top row and add to the second in the matrix on the left.
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 0 \\
2 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 2 & 3 \\
0 & -1 & -5 \\
-2 & 3 & -2
\end{mymatrix}
\end{equation*}
The next step is to take $2$ times the top row and add to the bottom in the
matrix on the right. To ensure that the product is unchanged, you place a $%
-2 $ in the bottom left in the matrix on the left. Thus the next step yields
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 0 \\
2 & 1 & 0 \\
-2 & 0 & 1
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 2 & 3 \\
0 & -1 & -5 \\
0 & 7 & 4
\end{mymatrix}
\end{equation*}
Next take $7$ times the middle row on right and add to bottom row. Updating
the matrix on the left in a similar manner to what was done earlier,
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 0 \\
2 & 1 & 0 \\
-2 & -7 & 1
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 2 & 3 \\
0 & -1 & -5 \\
0 & 0 & -31
\end{mymatrix}
\end{equation*}
At this point, stop. You are done.
\end{solution}

The method just described is called the
multiplier method.