\section{Subgradients}
\frame{\tableofcontents[currentsection, hideothersubsections]}

\begin{frame}
\frametitle{Subgradients: Intro}

WHAT:\\
apply GD to \textbf{non}differentiable convex function
\vspace{5mm}

HOW:\\
use subgradient of $f(\mathbf{w})$ at $\mathbf{w}^{(t)}$, instead of the gradient;\\
(the analysis of the convergence rate remains unchanged)

\end{frame}

\begin{frame}
\frametitle{Subgradients: Intro}

\begin{figure}
    \centering
    \includegraphics[scale=0.225]{lemma_14_3}
\end{figure}

\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}

\begin{figure}
    \centering
    \includegraphics[scale=0.225]{def_14_4}
\end{figure}

\end{frame}

\begin{frame}
\frametitle{Subgradients: Intro}

\begin{figure}
    \centering
    \includegraphics[scale=0.25]{fig_14_2}
\end{figure}

\end{frame}

\begin{frame}
\frametitle{Subgradients: Calculating Subgradients}

How do we construct subgradients of a given convex function?
\vspace{5mm}

For pointwise maximum functions:
\begin{figure}
    \centering
    \includegraphics[scale=0.225]{claim_14_6}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[scale=0.225]{example_14_2}
\end{figure}

\end{frame}


\section*{Related Work \& Background}

\subsection*{Static Scene Reconstruction}

Static scene reconstruction is the problem of reconstructing a 3D model from a set of 2D views.
Neural Radiance Fields (NeRFs)~\cite{mildenhall2020nerf} are a recent technique for reconstructing new views of a scene from new camera positions. To represent highly-detailed scenes, NeRFs model a scene as a continuous volume of varying density, which emits different colored light depending on the view direction. This is based on traditional volume rendering techniques:
\[ I(r) = \int_{t_n}^{t_f} T(t, r) \sigma(r(t)) c(r(t), r_d)dt, \]
where
\[ T(t, r) = \exp(-\int_{t_n}^{t} \sigma(r(s))ds), \]
\noindent
where $I(r)$ is the illumination along camera ray $r(t) = r_o + r_d t$, $r_o, r_d$ are the ray origin and direction respectively, and $t$ is some positive distance along the ray. NeRFs are able to accurately reconstruct high-frequency features by recovering $\sigma$, the density at a given point, and $c$, the view-dependent color at a given point by modelling them as MLPs with an additional encoding scheme that can differentiate between extremely close points. NeRFs evaluate the above equations by performing ray-marching and computing $T(j,r) = \Sigma -\exp\sigma_i c_i$, by partitioning the ray into evenly spaced bins and sampling randomly from within each bin.

There have been a significant number of extensions to NeRF, including optimizations on the encoding for differentiating positions in space~\cite{tancik2020fourfeat}, better sampling approaches~\cite{barron2021mipnerf}, and faster training~\cite{yu2021plenoxels}.

\subsection*{Dynamic Scene Reconstruction}

Dynamic scene reconstruction is removing the assumption in static scene reconstruction that all views are under the same condition, such as the same lighting and that nothing has moved.
NeRFs were designed to only handle static scenes, and thus cannot accurately reconstruct scenes which contain movement, alternative lighting conditions, or other changes between frames.
In order to model dynamic scenes, there have been two diverging approaches.

One kind of approach directly models the transformation in the time domain, by learning a function $\sigma(x,t)=f(x\in\mathbb{R}^3, t\in[0,1])$, which include works such as HyperNeRF~\cite{park2021hypernerf}, NeRFies~\cite{park2021nerfies}, and Space-Time Invariant Irradiance Fields~\cite{xian2021space}. By directly modelling the variation of the density, these methods are able to reconstruct large deformations in latent spaces and reconstruct a wide variety of transformations from a single radiance field. These often allow for fun transformations in some learned space between many similar scenes, which allow novelty warping and interpolation.

The other kind of approach models movement directly as translation. NeRFs are not able to move the objects inside the scene since we can only evaluate the NeRF at a given $x$. Instead we bend the rays, warping what is visible from a given view. This is essentially a perspective shift of a transformation of the space being rendered. Instead of moving an object that is seen by ray $r$, we warp ray $r$ such that it is sees the object. The equation for density is defined as $\sigma(x,t)=f(x+\Delta(x,t))$. This formulation enforces a coherent canonical representation, while directly modelling movement, and has been shown to be able to reconstruct both synthetic scenes with D-NeRF~\cite{pumarola2020dnerf} and real scenes in NR-NeRF~\cite{tretschk2021nonrigid}.

The pros of directly including time as a function in the MLP are that we are able to represent a much broader class of functions, in theory every frame may be fully distinct from the previous, but the canonical formulation lends itself to smoothness between frames. Our approach falls into the canonization category, since we are interested in accurately reconstructing smooth movement as opposed to generalizing over many classes of movement.

\subsection*{Bezier Curves}

Bezier curves refer to a specific set of polynomials parametrized by a set of control points. They are most commonly used as cubic polynomials: $f(x) = ax^3 + bx^2 + cx + d$,
where $x$ is the variable we are interested in interpolating over. An example of a Bezier Spline is shown in Figure~\ref{fig:bezier_diagram}. The general formulation for
the Bezier basis functions is defined as $B^n(t) = \Sigma^n_{i=0}
{n \choose i} (1-t)^{n-i} t^i$ where $n$
is the degree of the Bezier polynomial. In order to give control of the Bezier curve, we
introduce "control points", which weigh different points along the curve differently:
$B^n(t) = \Sigma^n_{i=0} P_i {n \choose i} (1-t)^{n-i} t^i$, where $P_i\in\mathbb{R}^3$ for 3D
movement. For a more comprehensive guide on Bezier splines, we refer the reader to a more
\href{https://pomax.github.io/bezierinfo/index.html}{complete reference}~\cite{bezier_primer}.

\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{bezier_curve.png}
    \caption{
        Bezier Curves are a low-dimensional polynomial representation that allows for smooth interpolation between a few control points. We reconstruct control points to produce smooth movement and induce a prior on continuity. Credit to Wikipedia~\cite{bezier_diagram} for diagram.
    }
    \label{fig:bezier_diagram}
\end{figure}


\subsection{Scripts}
Scripting is a way of automatically running a sequence of calculations.
A script is entered in the left-hand field of the Eigenmath window.

\begin{center}
\begin{tikzpicture}
\node at (0,0) {\includegraphics[scale=0.2]{face.png}};
\draw (-2.4,0.1) node {Scripts go here.};
\end{tikzpicture}
\end{center}

To create a script, enter one calculation per line in the script field.
Nothing happens until the Run button is clicked. When the Run button is
clicked, Eigenmath evaluates the script line by line. After a script runs,
all of its symbols are available for immediate mode calculation.
Scripts can be saved and loaded using the File menu.

Here is an example script that can be pasted into the script field
and then run by clicking the Run button.

\begin{Verbatim}[formatcom=\color{blue}]
"Solve for vector X in AX = B"
A = ((1,2),(3,4))
B = (5,6)
X = dot(inv(A),B)
X
\end{Verbatim}

After clicking the Run button, the following result is displayed.

\verb$Solve for vector X in AX = B$

$\displaystyle X=\begin{bmatrix}-4\\ \frac{9}{2}\end{bmatrix}$

A handy debugging aid is to include the line $trace=1$ in the script.
When $trace=1$ each line of the script is displayed as it is evaluated.
For example, here is the previous script with the addition of
$trace=1$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
"Solve for vector X in AX = B"
trace = 1
A = ((1,2),(3,4))
B = (5,6)
X = dot(inv(A),B)
X
\end{Verbatim}

The result is

\begin{Verbatim}
Solve for vector X in AX = B
A = ((1,2),(3,4))
B = (5,6)
X = dot(inv(A),B)
X
\end{Verbatim}

$X=\begin{bmatrix}-4\\ \frac{9}{2}\end{bmatrix}$


\subsection{lag compensation}
$$
G_c(s) = K_c\dfrac{s + \dfrac{1}{T}}{s+\dfrac{1}{\beta T}}, \quad \beta > 1
$$
In lag compensation we find out $K$ for steady state error.
$$K = 5.1300$$
$$\gamma_d = 45^{\circ} \to \bar{\gamma_d} = 45 + 5 = 50^\circ$$
$$
\phi(\omega_g) = 50 - 180 = -130^{\circ}
$$
The phase angle:

$$
\angle G(j\omega) = 0^{\circ} + \tan^{-1}\dfrac{\omega}{0.5} - \tan^{-1}\dfrac{\omega}{1} - 3\tan^{-1}\dfrac{\omega}{1.5} - \tan^{-1}\dfrac{\omega}{2}
$$
$$
\angle G(j\omega_g) =
0^{\circ} + \tan^{-1}\dfrac{\omega_g}{0.5} - \tan^{-1}\dfrac{\omega_g}{1} - 3\tan^{-1}\dfrac{\omega_g}{1.5} - \tan^{-1}\dfrac{\omega_g}{2} = -130^{\circ}
$$
This equation solved with MATLAB and code has attacked (Q1\_b.m)
$$
\omega_g =  1.2025
$$
The amplitude ratio:
$$
\left\vert G_1(j\omega) \right\vert = \left\vert KG(j\omega) \right\vert = 5.1300\dfrac{50\sqrt{\omega^2+0.5^2}}{\sqrt{\omega^2+1^2}\times(\sqrt{\omega^2 + 1.5^2})^3\times\sqrt{\omega^2 + 2^2}}
$$
$$
\beta = \left\vert G_1(j\omega_g) \right\vert =
5.1300\dfrac{50\sqrt{\omega_g^2+0.5^2}}{\sqrt{\omega_g^2+1^2}\times(\sqrt{\omega_g^2 + 1.5^2})^3\times\sqrt{\omega_g^2 + 2^2}}
$$

$$
\beta = 5.1300\dfrac{50\sqrt{1.2025^2+0.5^2}}{\sqrt{1.2025^2+1^2}\times(\sqrt{1.2025^2 + 1.5^2})^3\times\sqrt{1.2025^2 + 2^2}} = 12.8807
$$
Assume:
$$
\dfrac{1}{T} = \dfrac{\omega_g}{2}
\to \dfrac{1}{T}  = 0.6012 \to T = 1.6632
$$
$$
\to \dfrac{1}{\beta T} = 0.0467, \quad K_c = \dfrac{K}{\beta} = \dfrac{5.1300}{12.8807} = 0.3983
$$
$$
G_c(s) = K_c \dfrac{s + \dfrac{1}{T}}{s + \dfrac{1}{\beta T}}
= 0.3983 \dfrac{s + 0.6012}{s + 0.0467}
$$
Bode diagram for lag compensation using MATLAB.
\begin{figure}[H]
	\caption{lag compensation Bode diagram using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/controller.png}
\end{figure}
Nyquist plot for lag compensation using MATLAB.
\begin{figure}[H]
	\caption{lag compensation Nyquist plot using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/controller_nyquist.png}
\end{figure}

Now add lag compensation to system.
$$
G_c(s)G(s) = 0.3983 \dfrac{s + 0.6012}{s + 0.0467}\dfrac{50(s+0.5)}{(s+1)(s+1.5)^{3}(s+2)}
$$
Bode diagram for system with adding lag compensation.
\begin{figure}[H]
	\caption{Bode diagram for system with lag compensation using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_margin.png}
\end{figure}
Nyquist plot for system with adding lag compensation.
\begin{figure}[H]
	\caption{Nyquist plot for system with lag compensation using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_nyquist.png}
\end{figure}
All bode diagram in one figure:
\begin{figure}[H]
	\caption{all bode diagram using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/all_in_one.png}
\end{figure}
All Nyquist plot in one figure:
\begin{figure}[H]
	\caption{all Nyquist plot using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/all_in_one_nyquist.png}
\end{figure}
We didn't get what we want in equation so we change our before assumption.
assume:
$$
\dfrac{1}{T} = \dfrac{\omega_g}{10}
\to \dfrac{1}{T}  = 0.1202 \to T = 8.3161
$$
$$
\to \dfrac{1}{\beta T} = 0.0093, \quad K_c = \dfrac{K}{\beta} = \dfrac{5.1300}{12.8807} = 0.3983
$$
$$
G_c(s) = K_c \dfrac{s + \dfrac{1}{T}}{s + \dfrac{1}{\beta T}}
= 0.3983 \dfrac{s + 0.1202}{s + 0.0093}
$$
Bode diagram for lag compensation using MATLAB.
\begin{figure}[H]
	\caption{new lag compensation Bode diagram using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_controller.png}
\end{figure}
Nyquist plot for lag compensation using MATLAB.
\begin{figure}[H]
	\caption{new lag compensation Nyquist plot using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_controller_nyquist.png}
\end{figure}
Now add new lag compensation to system.
$$
G_c(s)G(s) = 0.3983 \dfrac{s + 0.1202}{s + 0.0093}\dfrac{50(s+0.5)}{(s+1)(s+1.5)^{3}(s+2)}
$$
Bode diagram for system with adding lag compensation.
\begin{figure}[H]
	\caption{Bode diagram for system with new lag compensation using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new2_margin.png}
\end{figure}
Nyquist plot for system with adding lag compensation.
\begin{figure}[H]
	\caption{Nyquist plot for system with new lag compensation using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new2_nyquist.png}
\end{figure}
All bode diagram in one figure:
\begin{figure}[H]
	\caption{all bode diagram using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_all_in_one.png}
\end{figure}
All Nyquist plot in one figure:
\begin{figure}[H]
	\caption{all Nyquist plot using MATLAB}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/new_all_in_one_nyquist.png}
\end{figure}
In new lag compensation phase margin is $45.3^{\circ}$ and we are near to question requirement.
Step respond for close loop system.
\begin{figure}[H]
	\caption{Step respond with lag compensation}
	\centering
	\includegraphics[width=12cm]{../Figure/Q1/b/step.png}
\end{figure}


\chapter{Randomized Algorithms in Numerical Linear Algebra}

%------------------------------------------------------------------------------------------------------
\section{Least Squares Problems}
Consider the linear system
\begin{equation}\label{AXB}
A x =b ,
\end{equation}
where $A \in \mathbb{R}^{m \times n}, b \in \mathbb{R}^{m}, m \gg n, $  and $rank(A) = n$.
System (\ref{AXB}) has the least squares solution
\begin{equation}\label{OPT}
x_{opt} = argmin_{x} \| Ax -b\|_2^2 .
\end{equation}
$x_{opt}$ is also the solution of the normal equation
\begin{equation}\label{NOR}
A^T A x = A^T b.
\end{equation}
Suppose $\bold{y} = (y_1,\cdots,y_n)'$  is the response or target variable.

$$\bold{X} = \begin{bmatrix} x_{11} ,&\cdots,& x_{1p}\\
                                                \cdots,&\cdots,&\cdots \\
                                                x_{n1},&\cdots,&x_{np} \end{bmatrix}
                 = (\bold{x_1},\cdots,\bold{x_p})$$
is the $n \times p$ data matrix of the $n$ observations on the $p$ variables.

The linear regression is to find the parameter $\bold{\beta}$ for a 'best' prediction of $\bold{y}$, which is to minimize the 2 norm of error
$$\| \bold{y} - \bold{X} \bold{\beta} \|_2^2.$$

The least squares estimator $\beta_{opt}$ is given by
$$
\beta_{opt} = (\bold{X}^T \bold{X})^{-1} \bold{X}^T \bold{y}.
$$
%-----------------------------------------------------------------------------------------------------------------
\section{Traditional Methods}
%------------------------------------------------

\subsection{QR Decomposition}

Compute
$$A = QR,$$
where $Q$ is an orthogonal matrix and $R$ is an upper triangular matrix.

 \begin{figure}[htbp]
    \includegraphics[width=1\textwidth]{qrd}
    \caption{QR Decomposition}
\end{figure}

Complexity of QR decomposition is {\color{red} $O(mn^2)$}.

%------------------------------------------------
\subsection{SVD Decomposition}

Compute
$$
A = U \Sigma V^T,
$$
where
\begin{itemize}
\item $U$ is a $m \times m $ orthogonal matrix \\
\item $\Sigma$ is a diagonal $m \times n$ matrix with non-negative real numbers on the diagonal and \\
\item $V$ is a $n \times n $ orthogonal matrix.
\end{itemize}
The diagonal entries are $\sigma_i$ of $\Sigma$ are known as the singular values of $A$. \\
 \begin{figure}[htbp]
    \includegraphics[width=1\textwidth]{svd2}
\end{figure}
Complexity of SVD decomposition is {\color{red} $O(mn^2)$}.

%------------------------------------------------

\subsection{Conjugate Gradient Method}

Apply the conjugate gradient method to the normal equation (\ref{NOR})
$$
A^T A x = A^T b.
$$
Complexity:
\begin{itemize}
\item The computation cost at each step is  $O(mn)$  for $A^T(Ax)$. \\
\item To achieve accuracy $\epsilon$, we need $ k = log_{\frac{\sqrt{\kappa(A^T A)} -1}{\sqrt{\kappa(A^T A)} +1}} \frac{\epsilon}{2} \approx 2 |log(\epsilon)|\kappa(A)$ steps. \\
\item The total complexity is
{\color{red}
$$
O(m n \kappa(A) |log(\epsilon)|).
$$
}
\end{itemize}

%------------------------------------------------

\subsection{Kaczmarz Method}

Denote the rows and columns of $A$ by $A^{(1)}, \cdots, A^{(m)}$ and $A_{(1)},\cdots,A_{(n)},$ respectively (both viewed as columns vectors). The Kaczmarz Scheme is given by
\begin{tcolorbox}
procedure $(A,b,T)$ \\
\quad Set\quad $ x^{(0)} $ \quad to by any vector in the row space of $A$\\
\quad for $k = 0,1,2,\cdots, T-1$ do \\
\quad \quad $ik \equiv k \Mod{m}$ \\
\quad \quad Set $x^{(k+1)} = \mathcal{P}_{A^{(ik)},b_{ik}}(x^{(k)}) $\\
\quad end for \\
\quad Output $x^{(T)}$  \\
end procedure
\end{tcolorbox}

\subsubsection{Orthogonal Projection}
We define the orthogonal projection of $\mathbf{x}$ onto the hyperplane by \quad $\mathbf{c}^T \mathbf{x} = d$:
$$
\mathcal{P}_{\mathbf{c},d}(\mathbf{x}) = \mathbf{x} - \frac{\mathbf{c}}{\| \mathbf{c} \|_2} (\mathbf{c} ^T \mathbf{x} - d).
$$
The operator does an orthogonal projection of the current estimate vector $\mathbf{x}$ onto the hyperplane $\mathbf{c}^T \mathbf{x} =d$.
\begin{figure}[htbp]
    \includegraphics[width=\textwidth]{kaczmarz-method1.png}
    \caption{Kaczmarz Method}
    \label{fig:kaczmarz method}
\end{figure}

\subsubsection{Convergence Analysis and Complexity}
\begin{theorem}
The cylic Kaczmarz iteration with approriately chosen relaxation parameter $\omega$ and row scaling induced by $D$ for solving a linear system $Ax = b$ with $rank(A) = r \leq min(m,n)$ possesses an error bound
$$
\| x^{(k)} - x_{LS} \|_2^2 \leq \left[ 1 - \frac{C}{(ln(r)+1) \kappa(A^T D^{-1} A)}\right] ^k \| x^{(0)} - x_{LS} \|_2^2, \quad k \geq 1,
$$
where $C$ is an absolute constant.
\end{theorem}

The complexity of kaczmarz method is computed by the following steps.
\begin{itemize}
\item The complexity at each projection is {\color{red} $O(n)$}.
\item To achieve accuracy $\epsilon$, we need
$$\| x^{(k)} - x_{LS} \|_2^2 \leq \epsilon^2 \| x^{(0)} - x_{LS} \|_2^2.$$
\item The expected number of iterations $k_{\epsilon}$ to achieve the accuracy $\epsilon$ is
{\color{red}$$  k_{\epsilon} \approx O ((ln(r)+1) \kappa(A^TD^{-1}A) | log(\epsilon)| ). $$}
\item The complexity of  Kaczmarz method is thus
{\color{red}
$$  O (n (ln(r)+1) \kappa(A^TD^{-1}A) | log(\epsilon)| ). $$}
\item The complexity of CG applied to normal equation is
{\color{red}
$$
O(m n \kappa(A) |log(\epsilon)|).
$$
}
\end{itemize}

%------------------------------------------------
\section{Randomized Algorithms}
\subsection{Sampling based randomized algorithm (SRA)}
The sampling based randomized algorithm (SRA) and projection based randomized algorithm (PRA) are from the paper \cite{FLSA2011}.

%------------------------------------------------
Instead of solving the original problem is
\begin{equation}\label{ORG}
\mathcal{Z} = min_{x \in \mathbb{R}^n} \| Ax - b\|_2
\quad \text{with solution} \quad
x_{opt} = (A^T A)^{-1} A^T b = A^{\dagger} b,
\end{equation}
actually we solve
\begin{equation}\label{ACT}
\tilde{\mathcal{Z}} = min_{x \in \mathbb{R}^n} \| X Ax - X b\|_2
\quad \text{with solution} \quad
\tilde{x}_{opt} = (XA)^{\dagger} X b.
\end{equation}

%------------------------------------------------
\subsubsection{Theory}
\begin{lemma}[Drineas, Mahoney, Muthukrishnan and Sarlos(2011)]
Consider the overconstrained least squares problem of (\ref{ORG}) and let the matrix $U_A \in \mathbb{R}^{m\times n}$ contain the top $n$ left singular vectors of $A$. Assume $X$ satisfies the two conditions
\begin{enumerate}
\item $\sigma_{min}^2(X U_A) \geq 1/\sqrt{2}; $ \\
\item $\| U_A^T X^T X b^{\perp}\|_2^2 \leq \epsilon \mathcal{Z}^2 /2, \quad (b^{\perp} = b - A A^T b), $
\end{enumerate}
for some $\epsilon \in (0,1).$

Then the solution $\tilde{x}_{opt}$ to the least squares approximation problem (\ref{ACT}) satisfies:
\begin{enumerate}
\item $\| A \tilde{x}_{opt} - b\|_2 \leq (1+ \epsilon) \mathcal{Z} $ , and \\
\item $\| x_{opt} - \tilde{x}_{opt} \|_2 \leq \frac{1}{\sigma_{min}(A)} \sqrt{\epsilon} \mathcal{Z}. $
\end{enumerate}
\end{lemma}

%------------------------------------------------
\subsubsection{Implementation: A sampling-based randomized algorithm}
A sampling-based randomized algorithm (SRA) is to construct
$$
X = S^T H D,
$$

\begin{itemize}
\item $S \in \mathbb{R}^{r \times m}$ is the uniform sampling matrix, where $S_{r} = (\sqrt{m/r}) e_{ir},$ where $ir$ is uniformly chosen from $[m],$
and $e_{ir}$ is the standard basis. \\
\item $H \in \mathbb{R}^{m \times m}$ is the Hadamard transform matrix defined recursively by
$$
H_m = \begin{bmatrix} H_{m/2} & H_{m/2} \\ H_{m/2} & -H_{m/2} \end{bmatrix}, \quad \text{with} \quad H_2 = \begin{bmatrix} +1 & +1 \\ +1 & -1 \end{bmatrix}.
$$\\
\item $D \in \mathbb{R}^{m \times m}$ is the diagonal matrix with
$$
D_{ii} = \begin{cases} &+1, \quad \text{with probability} \quad 1/2;\\ &-1 \quad \text{with probability} \quad 1/2. \end{cases}
$$
\end{itemize}

$$
\tilde{x}_{opt} = (S^T H D A)^{\dagger} S^T H D b.
$$

%------------------------------------------------
\subsubsection{Effect of the Randomized Hadamard Transform}
{\color{red} HD approximately 'uniformizes' information in the left singular subspace of the matrix $A$.}
\begin{lemma}
Let $U$ be an $m \times n$ orthogonal matrix and let the product $HD$ be the $m \times m$ Randomized Hadamard Transform. Then with probability at least $0.95$,
$$
\| (HDU)_{i} \|_2^2 \leq \frac{2n \ln(40 m n)}{m}, \quad \forall i \in [m].
$$
\end{lemma}

%------------------------------------------------

\begin{theorem}[Convergence and Complexity of SRA]
Suppose $A,b$ and $\epsilon$ satisfy the input requirement of SRA. Run SRA with
\begin{equation}\label{eq22}
r = max \left( 48^2 n \ln(40 mn) \ln(100^2 n \ln(40 mn)) , 40n \ln(40 mn) /\epsilon \right)
\approx O(n/\epsilon).
\end{equation}
and return $\tilde{x}_{opt}.$ Then, with probability at least $0.8$, the following claims hold:
\begin{enumerate}
\item $\tilde{x}_{opt}$ satisfies $\| A \tilde{x}_{opt} - b \|_2  \leq (1 + \epsilon) \mathcal{Z}; $ \\
\item If we assume that $\| U_A  U_A^T b \|_2 \geq \gamma \|b\|_2 $ for some $0< \gamma \leq 1$ then
$$
\| x_{opt} - \tilde{x}_{opt}\|_2 \leq \sqrt{\epsilon} \left( \kappa(A) \sqrt{\gamma^{-2} -1} \right) \| x_{opt}\|_2.
$$\\
\item $m(n+1) + 2m(n+1)log_2(r+1) + O(rn^2)$ time suffices to compute the solution $\tilde{x}_{opt}. $
\end{enumerate}
\end{theorem}
The cost is reduced since $XA \in \mathbb{R}^{r \times n}$ while $A \in \mathbb{R}^{m \times n}.$

%------------------------------------------------
\subsubsection{Projection Based Randomized Algorithm (PRA)}
Implementation: A projection-based randomized algorithm
Projection-based randomized algorithm (PRA) is to construct a smaller problem by performing a 'sparse projection' on the pre-processed problem.
$$
X = THD,
$$
where
\begin{itemize}
\item $H \in \mathbb{R}^{m \times m}$ is the Hadamard Transform and $D \in \mathbb{R}^{m \times m }$ is the randomized diagonal matrix as defined before. \\
\item $T \in \mathbb{R}^{k \times m}$ is a randomized matrix given by
$$
T_{ij} = \begin{cases} &+ \sqrt{1/kq} \quad \text{with probability} \quad q/2, \\
                                   &-  \sqrt{1/kq} \quad \text{with probability} \quad q/2, \\
                                   & 0 \quad \text{with probability} \quad 1-q.
             \end{cases}
$$
\end{itemize}
$$\tilde{x}_{opt} = (THDA)^{\dagger} THDb. $$

%------------------------------------------------

\begin{theorem}[Convergence and Complexity of PRA]
Suppose $A,b$ and $\epsilon$ satisfy the input requirement of FRA. Run FRA with
\begin{equation}
q \geq \frac{C_q n \ln(40mn)}{m} (2 \ln(m) + 16n + 16);
\end{equation}
\begin{equation}
k \geq max \left( C_k (118^2 n + 98^2), \frac{60n}{\epsilon} \right)
\end{equation}
and return $\tilde{x}_{opt}.$ Then, with probability at least $0.8$, the following claims hold:
\begin{enumerate}
\item $\tilde{x}_{opt}$ satisfies $\| A \tilde{x}_{opt} - b \|_2  \leq (1 + \epsilon) \mathcal{Z}; $ \\
\item If we assume that $\| U_A  U_A^T b \|_2 \geq \gamma \|b\|_2 $ for some $0< \gamma \leq 1$ then
$$
\| x_{opt} - \tilde{x}_{opt}\|_2 \leq \sqrt{\epsilon} \left( \kappa(A) \sqrt{\gamma^{-2} -1} \right) \| x_{opt}\|_2.
$$\\
\item $m(n+1) + 2m(n+1)log_2(mkq+1) + O(kn^2)$ time suffices to compute the solution $\tilde{x}_{opt}. $
\end{enumerate}
\end{theorem}

%------------------------------------------------
\subsection{A Fast Randomized Algorithm (FAR)}
The fast randomized algorithm (FAR) is from Rokhlin and Tygert's paper \cite{SRFT2008}. The algorithm is given as follows.
\begin{tcolorbox}
\begin{enumerate}
\item Compute $E = TA$, where $T$ is the $r \times m$ SRFT defined as follows, with $m \geq r \geq n. $ \\
\item Form a pivoted QR-decomposition of $E = Q R \Pi$,  where $Q_{r \times n}$ has orthonormal columns, $R_{n \times n}$ is upper triangular and $\Pi_{n \times n}$ is the permutation matrix.  \\
\item Solve a preconditioned Least Squares Problem
$$
\| AP^{-1} y - b\|
$$
using PCG where $P = R \Pi$ is the preconditioning matrix.
\end{enumerate}
\end{tcolorbox}

%------------------------------------------------
\subsubsection{Complexity for A Fast Randomized Algorithm (FAR)}
\begin{tcolorbox}
\begin{enumerate}
\item Applying $T$ to every column of $A$ : $O(mnlog(r))$.  \\
\item Computing the pivoted QR decomposition of $E$ : $O(n^2r) . $ \\
\item Applying $T$ to $b$: O(mlog(r)). Applying $Q^*$ to $Tb$: $O(nr)$.

 Applying $P^{-1} = \Pi^{-1} R^{-1}$ to $Q^* Tb$ : $O(n^2). $ \\
\item Applying $A,A^T$ a total of $O(\kappa(AP^{-1}) |log(\epsilon)|)$ times: {\color{red}$O(mn\kappa(AP^{-1}) |log(\epsilon)|).$} \\
\item Applying $P^{-1},(P^{-1})^*$ to a total of $O(\kappa(AP^{-1}) |log(\epsilon)|)$ times: {\color{red}$O(n^2 \kappa(AP^{-1}) |log(\epsilon)|).$} \\
\item Applying $P^{-1}$ to $y$ : $O(n^2). $
\end{enumerate}
\end{tcolorbox}
Thus
{\color{red}
$$
C_{theoretical} = O((log(r) + \kappa(AP^{-1}) |log(\epsilon)|) mn +n^2r).
$$}

%------------------------------------------------
\subsubsection{SRFT Matrix}
T is the sampled randomized Fourier transform (SRFT) Matrix defined by
$$T _{r \times m} = G_{r \times m} H_{m \times m}, \quad  r \leq m. $$
G is the random matrix given by
$$ G_{r \times m} = S_{r \times m} F_{m \times m} D_{m \times m}, $$

\begin{itemize}
\item $S$ is a random permutation matrix in each row j there is one column $s_j$ such that $S_{j,s_j} = 1$ and $S_{j,k} =0$ if $k \neq s_j$. $s_j$'s are i.i.d. random variables distributed uniformly over $\{ 1,\cdots,m\}. $ \\
\item $F$ is the $m \times m$ discrete Fourier transform. \\
\item $D = diag(d_1,d_2,\cdots,d_m)$
           where $d_1,\cdots,d_m$ are i.i.d. complex random variables distributed uniformly over the unit circle.
\end{itemize}

%------------------------------------------------
$$
H_{m \times m} =\Theta_{m \times m} \Pi_{m \times m} Z_{m \times m} \tilde{\Theta}_{m \times m}\tilde{\Pi}_{m \times m} \tilde{Z}_{m \times m},
$$
where

\begin{itemize}
\item $\Pi$ and $\tilde{\Pi}$ are permutation matrices chosen independently and uniformly at random. \\
\item $Z$ and $\tilde{Z}$ are diagonal matrices whose diagonal entries are i.i.d. complex random variables distributed uniformly over the unit circle. \\
\item
 $\Theta_{m \times m} = \begin{bmatrix} cos(\theta_1)& sin(\theta_1) & 0& \cdots &0\\
                                                                     - sin(\theta_1)  & cos(\theta_1) & 0 & \cdots & 0 \\
                                                                           0 & 0 & 1 & \cdots & 0 \\
                                                                               \cdots & \cdots & \cdots & \cdots & \cdots  \\
                                                                                0 & 0 &0 & \cdots & 1
                                                   \end{bmatrix}
                                                                                \cdots
                                                        \begin{bmatrix} 1& \cdots & 0& 0  &0\\
                                                                             \cdots  & \cdots  & \cdots & \cdots & \cdots \\
                                                                              0 & \cdots & 1 & 0 & 0 \\
                                                                              0 & \cdots & 0 & cos(\theta_{m-1})& sin(\theta_{m-1}) \\
                                                                             0 & \cdots &0 &  -sin(\theta_{m-1}) & cos(\theta_{m-1})
                                                         \end{bmatrix} ,$

 where $\theta_k, \quad k =1 ,\cdots m-1$ are i.i.d. real random variables drawn uniformly from $[0,2\pi]$.
 So is $\tilde{\Theta}$, but defined with different $\tilde{\theta}_k$.

\end{itemize}

%------------------------------------------------

\subsubsection{Why SRFT works?}
\begin{corollary}
Suppose that $\alpha$ and $\beta$ are real numbers greater than 1 and $r,m$ and $n$ are positive integers such that $m \geq r \geq (\frac{\alpha^2 +1}{\alpha^2-1})^2 \beta n^2. $ Suppose further that $T$ is the $r \times m$ SRFT matrix. Suppose in addition that $U$ is an $m \times n$ matrix whose columns are orthonormal.
Then, the condition number of $TU$ is at most $\alpha$ with probability at least $1 - \frac{1}{\beta}. $
\end{corollary}
$\kappa(TU)$ can be made arbitrarily close to 1 when $r$ is large enough.

%------------------------------------------------

\begin{theorem}[Rokhlin and Tygert (2008)]
Suppose that $r,m$ and $n$ are positive integers such that $m \geq r \geq n$. Suppose further that $A$ is a full rank $m \times n$ matrix and the SVD of $A$ is
$$
A_{m \times n} = U_{m \times n} \Sigma_{n \times n} V^*_{n \times n}.
$$
Suppose in addition that $T$ is an $r \times m$ matrix such that the $r \times n$ matrix $TU$ has full rank.
Then
$$
T_{r \times m} A_{m \times n} = Q_{r \times n} P_{n \times n},
$$
where $Q$ has orthonormal columns.
And
$$
\kappa(AP^{-1}) = \kappa(TU).
$$
\end{theorem}
\small{$r \geq 4n^2$ guarantees $\kappa(TU) \leq 3.$}

%------------------------------------------------
\subsection{Blendenpik}

%------------------------------------------------
The Blendenpik method is from the paper \cite{BLEN2010}. In this paper, they emphasis on the coherence number of a matrix.
\begin{definition}[Coherence]
Let $A$ be an $m \times n$ full rank matrix, and let $U$ be an $m \times n$ matrix whose columns form an orthonormal basis for the column space of $A$. The coherence of $A$ is defined as
$$
\mu(A) = max\| U_{i,*} \|_2^2.
$$
\end{definition}

Obviously,
$$
 \frac{n}{m} \leq \mu(A) \leq 1.
$$

%------------------------------------------------
\subsubsection{Matrices Classified by Coherence}
According to the coherence number, we can classify the matrices into 3 categories.
\begin{enumerate}
\item Incoherent matrix: $A = randn(m,n)$ since $\mu(A)$ is small. \\
\item Semicoherent matrix: $$
A_{m \times n} =
 \begin{bmatrix}
  B & \\
    \quad & I_{n/2}
 \end{bmatrix},
$$
where $B$ is an $(m - n/2) \times n/2$ rectangular random matrix and $I_{n/2}$ is a square identity of dimension $n/2$.

$\mu(A) =1$ but only $n/2$ rows have a large norm in the orthogonal factor.
\item Coherent matrix:
$$
A_{m \times n} =  \begin{bmatrix}
  D_{n \times n} & \\
   O_{(m-n) \times n}
 \end{bmatrix}
 + 10^{-8}
 \begin{bmatrix}
1 & \cdots & 1 \\
\cdots & \cdots & \cdots \\
1 & \cdots &1
 \end{bmatrix} ,
$$
where $D$ is a random diagonal matrix. The orthogonal factor has $n$ rows with a large norm.
\end{enumerate}

%------------------------------------------------
\subsubsection{Blendenpik: How coherence influence sampling?}
One preconditioned method given by Rokhlin and Tygert to solve the least squares problems using the R factor of the sampled rows as a preconditioner in a Krylov-subspace method like LSQR.
\begin{lstlisting}
[Q,R] = qr(SA,0);
x1 = lsqr(A,b,eps,100,R);
\end{lstlisting}
The method fails when the coherence number is large.
For the semicoherent matrix: $$
A_{m \times n} =
 \begin{bmatrix}
  B & \\
    \quad & I_{n/2}
 \end{bmatrix}.
$$\\
After uniform sampling, $SA$ is rank deficient with high probability. $R$ is ill conditioned. LSQR fails. One way to solve is to sample with the probability as leverage score $p_i = \|U_i\|_2^2/m$, where $A = UR$ is the reduced QR decomposition, which cost $O(n^3).$

%------------------------------------------------

\begin{theorem}[Avron, Maymounkov and Toledo (2010)]
Let $S$ be a random sampling operator that samples $r \geq n$ rows from $A$ uniformly. Let $\tau = C \sqrt{m \mu(A) log(r) /r},$
where $C$ is some constant defined in the proof. Assume that $\delta^{-1} \tau <1$. With probability of at least $1 - \delta$, the sampled matrix $SA$ is full rank, and if $SA = QR$ is a reduced QR factorization of $SA$, we have
$$
\kappa(A R^{-1}) \leq \frac{1 + \delta^{-1} \tau}{1 - \delta^{-1} \tau}.
$$
\end{theorem}
If $\mu(A)$ is large we need a large sample size $r$ to balance, otherwise we will have a larger bound for $\kappa(A R^{-1}).$

%------------------------------------------------
\subsubsection{Blendenpik: Row Mixing}
\begin{theorem}[Avron, Maymounkov and Toledo (2010)]
Let $A$ be an $m \times n$ full rank matrix, where $m \geq n$. Let $F$ be an $m \times m$ unitary matrix, let $D$ be a diagonal matrix whose diagonal entries are independent and identically distributed Rademacher random variables ($P(D_{ii} = \pm 1 ) = 1/2$), and let $\mathcal{F} = FD$. With a probability of at least $0.95$, we have
$$
\mu(\mathcal{F} A) \leq C n \eta log m,
$$
where $\eta = max | F_{ij} |^2$ and some constant $C$.
\end{theorem}

The seed matrix can be Walsh-Hadamard transform (WHT {\color{red}$\eta = 1/m$}), discrete cosine transform (DCT{\color{red}$\eta = 2/m$}) and discrete Hartley transform (DHT {\color{red} $\eta = 2/m$}). After row mixing, the coherence of $\mathcal{F} A$ becomes small. Then uniform sampling and LSQR can be applied. Complexity of calculting $\mathcal{F}A$: $O(mn log(m)).$

%------------------------------------------------
\subsubsection{Blendenpik Algorithm}
\begin{tcolorbox}
\begin{enumerate}
\item Row Mixing : $F_{\tilde{m}} DM$, where
\begin{itemize}
\item $\tilde{m} = \begin{cases}& 2^{[log_2 m]} , \quad \text{WHT} \\
                                         & [m/1000]*1000, \quad \text{DCT or DHT}.
                    \end{cases}$\quad  $M = \begin{bmatrix} A\\ 0 \end{bmatrix} \in \mathbb{R}^{\tilde{m} \times n}$. \\
\item $D$ is a diagonal matrix with $\pm 1$ on its diagonal with equal probability. \\
\item $F_{\tilde{m}}$ is the seed unitary transform (WHT/DCT/DHT) operations.
\end{itemize}
\item Sampling : $S \in \mathbb{R}^{\tilde{m} \times \tilde{m}}$ be a random diagonal matrix with
$S_{ii} = \begin{cases} & 1 \quad \text{with probability}\quad \gamma n /\tilde{m} \\
                                   & 0 \quad \text{with probability}\quad 1 - \gamma n /\tilde{m}.
             \end{cases}$\\
\item QR of Sampled Matrix : $S F_{\tilde{m}} D M = Q R,$  reduced QR factorization ($R\in \mathbb{R}^{n \times n} $). \\
\item Apply LSQR with preconditioner $R$. (If the iteration steps $>3$, solve using LAPACK and return).
\end{enumerate}
\end{tcolorbox}

%------------------------------------------------
\subsubsection{Complexity of Blendenpik Algorithm}
\begin{tcolorbox}
\begin{enumerate}
\item Row Mixing : $O(mn log m).$ \\
\item QR of Sampled Matrix : $O(n^3).$\\
\item Each LSQR Iteration: $O(mn).$ The number of iterations grows slowly.
\end{enumerate}
Total: ${\color{red} O(mn log m) + O(n^3) + O(mn k)}.$ $k$ is the number of LSQR iterations.
\end{tcolorbox}

%--------------------------------------------------------------------
\section{Randomized MultiGrid Methods}
%------------------------------------------------
Use AMG to solve the least squares problems we have the complexity ${\color{red} O(mn^2)}. $
\begin{tcolorbox}
\begin{enumerate}
\item \textbf{Matrix Multiplication:} Compute $A^TA$. \quad ${\color{red} O(mn^2)} $ \\
\item \textbf{Coarsening and Interpolation:} Use coarsenAMG to coarsen $A^TA$ and get the projection matrix $Pro$ and restriction Res. $n_c$ is the size of coarsened matrix.
\quad${\color{red} O(n n_c)}$
\\
\item \textbf{V-cycle Multigrid Precondition:} Dominant cost: several Gauss Seidel iterations.\quad${\color{red} O(n^2)}$ \\
\item \textbf{PCG:} Use PCG to solve $A^TA x = A^Tb$ with preconditioner constructed in Step 3.
\quad {\color{red} $O(kmn)$, where k is the steps of iterations.}
\end{enumerate}
{\color{red} Total Complexity: $O(mn^2).$}
\end{tcolorbox}
%------------------------------------------------
\subsection{Randomized MultiGrid Scheme}
We developed the Randomized Multigrid Methods by the following steps.
\begin{tcolorbox}
\begin{enumerate}
\item \textbf{Normalization:} Normalized $A$ with diagonal matrix $D$ where $D_{jj} = \|A^j\|_2, \quad j \in [n], $ \quad where
$A^j$ is the j-th column of $A$. \\
\item \textbf{Sampling:} Sample the row of $A$ to get $As_{sn \times n}, s_n \approx 4 n log(n)$ either by uniform sampling or row sampling. \\
\item \textbf{Coasening:} Use coarsenAMG to coarsen $A_s^T A_s$ and get the projection matrix  $Pro$ and restriction $Res$. \\
\item \textbf{Precondition:} Solve $A_s^T A_s e =r$ by TwoGrid/MultiGrid Method. \\
\scalebox{0.8}{
\begin{minipage}{\textwidth}
\begin{itemize}
\item Presmoothing: 3 times forward Gauss Seidel apply to $A_s^T A_s e =r$; \\
\item Restriction: Calculate residual and use $Res$ to do restriction $rc = Res*r$;\\
\item Direct solve on the coarsest grid $ec = (Ac'*Ac)/rc$ where $Ac = A*Pro$;\\
\item Prolongation: Use $Pro$ to get prolongation $e = e + Pro*ec$; \\
\item Postsmoothing: 3 times backward Gauss Seidel apply to $A_s^T A_s e =r$.
\end{itemize}
\end{minipage}} \\
\item \textbf{PCG:} Use PCG to solve $A^TA x = A^Tb$ with preconditioner constructed in Step 3.
\end{enumerate}
\end{tcolorbox}

%------------------------------------------------
\subsection{Uniform Sampling and Row Sampling}
\begin{enumerate}
\item \textbf{Uniform Sampling:}
$$
S_{ii} = \begin{cases} & \frac{1}{1/m} \quad \text{with probability} \quad \frac{1}{m}; \\
                             & 0 \quad \text{otherwise.}
                             \end{cases}
$$\\
\item \textbf{Row Sampling:}
$$
p_i = \frac{\| A_i \|_2}{\| A \|_F}, \quad \text{where $A_i$ is the i-th column of $A$}.
$$
$$
S_{ii} = \begin{cases} & \frac{1}{p_i} \quad \text{with probability} \quad p_i; \\
                             & 0 \quad \text{otherwise.}
                             \end{cases}
$$\\
\end{enumerate}
The sampled matrix is
$$
A_s = SA.
$$

%------------------------------------------------

\subsection{Complexity}
Complexity of Randomized MultiGrid Method is calculated as follows.
\begin{tcolorbox}
\begin{enumerate}
\item \textbf{Normalization:} Calculate $\|A^j\|_2 $ for $j \in [n]$ costs $O(mn)$. Multiplication $D A$ costs $O(mn)$.  \\
\item \textbf{Sampling: } Interpolation $O(s_n m ) = O(n log(n) m), $ where \quad $A_s \in \mathbb{R}^{s_n \times n}. $ \\
\item \textbf{Matrix Multiplication: } $A_s^T A_s.$ $O(s_n n^2) = O(n^3 log(n)). $ \\
\item \textbf{Precondition:} 6 times Gauss Seidel apply to $A_s^TA_s e =r $: $O(n^2). $\\
\item \textbf{PCG:} k times PCG until reaching threshold $\epsilon$ costs  $O(k mn). $
\end{enumerate}
\end{tcolorbox}

%------------------------------------------------
\subsection{Sampling Graph}
Graphs of $A^T A$ and $As^T As$
\tiny{
\begin{lstlisting}
Test matrix:  S = load('sparse_random_c_100.mat');   A = S.M{5};
\end{lstlisting}}
\begin{figure}[htbp]
    \includegraphics[width=1\textwidth]{sampling.png}
\end{figure}

%------------------------------------------------

\subsection{Performance}
%------------------------------------------------
\subsubsection{Well Conditioned Incoherent Matrix}
Performance: Well Conditioned Incoherent Matrix
The incoherent matrix (coherence $\mu(A)$ is small) is constructed by
$$
A = randn(m,n);
$$

\begin{table}\label{table_inc}
\centering
\caption{Incoherent Random Matrix}
\begin{tabular}{| l | l | l |l |l |l |l |l |l |l |}\hline
  m & n & Size  &   Conde    &    PCG.res    &  PCG.iter    &  TW.res    &  TW.iter  &   AMG.res    &  AMG.iter\\\hline
3000& 109 &  3.27e+05  & 8.0592  & 2.8265e-08  & 10  &  7.8964e-08  & 11  & 8.6322e-11 & 3   \\\hline
 5000 & 141 &7.05e+05  & 7.1294 &  6.3485e-08   & 9    & 5.0459e-08   & 11 & 6.8662e-12  & 3  \\\hline
  10000& 200 &   2e+06    & 7.6457   & 1.8065e-08   & 9    & 3.3853e-08   &11 &1.2271e-12   &3 \\\hline
   20000& 282 & 5.64e+06   &7.4708  &  3.6032e-08 &    8   &   3.4031e-08 &   11& 3.468e-13   &3  \\\hline
  40000& 400 &  1.6e+07  &  7.3094 &   8.8878e-08  &   7  &    2.8748e-08  &  11 &  3.6036e-14   & 3  \\\hline
\end{tabular}
\end{table}

%------------------------------------------------
\subsubsection{Well Conditioned Semicoherent Matrix}

Performance: Well Conditioned Semicoherent Matrix
\small
{Semicoherent matrix is given by
$$
A_{m \times n} =
 \begin{bmatrix}
  B & \\
    \quad & I_{n/2}
 \end{bmatrix},
$$
where $B$ us an $(m - n/2) \times n/2$ rectangular random matrix and $I_{n/2}$ is a square identity of dimension $n/2$. }

\begin{table}\label{table_semi}
\centering
\caption{Semicoherent Matrix with Row Sampling}
\begin{tabular}{| l | l | l |l |l |l |l |l |l |l |}\hline
  m & n & Size  &   Conde    &    PCG.res    &  PCG.iter    &  TW.res    &  TW.iter  &   AMG.res    &  AMG.iter\\\hline
3000& 109 &  3.24e+05 &  3.7979   & 4.4898e-08  &  8    &   6.2979e-08    &13    &  3.7059e-13   &  3   \\\hline
 5000 & 141 &    7e+05  & 3.6879   & 2.3175e-08 &   8     &  3.6585e-08    &12 &    5.6326e-14    & 3   \\\hline
  10000& 200 &     2e+06  &  3.7234   & 5.1245e-08  &  7   &    3.7377e-08 &   13   &   6.3799e-08   & 2    \\\hline
   20000& 282 &  5.64e+06  & 3.6137  &  2.4414e-08  &  7   &    4.0871e-08 &   12   &  3.5634e-08  &  2   \\\hline
   40000 & 400 & 1.6e+07   &  3.4741  &   8.1149e-08  &   6     &   4.6966e-08  & 12 &  1.094e-08 &   2  \\\hline
\end{tabular}
\end{table}

%------------------------------------------------
\subsubsection{Ill Conditioned Sparse Random Matrix}
Performance: Ill Conditioned Sparse Random Matrix
The sparse random matrix is generated by the MATLAB's function $sprand$
$$
 A = sprand(m,n,s,1/c);
$$
where $m$ is the number of rows, $n$ is the number of columns, $s$ is the sparsity and $c$ is the estimated condition number.
\\

%------------------------------------------------
Performance: Ill Conditioned Sparse Random Matrix

\begin{table}\label{table_spc}
\centering
\caption{Sparse Random Matrix Using Uniform Sampling $m = 40000, n = 200.$}
\begin{tabular}{|*{8}{c|}}\hline
     Nnz    &   Conde     &  PCG.res    &  PCG.iter   &   TW.res   &   TW.iter   &   AMG.res    &  AMG.iter \\\hline
    38942  &      2132.1  &  8.7014e-08   & 106    &     6.5273e-08   & 33     &    3.7773e-08  &  23   \\\hline
    38228   &      12321   & 9.6274e-08    & 167    &     7.8268e-08   & 44     &    9.9291e-08  &  36    \\\hline
    37718    &     73525    & 2.8338e-05    & 200    &     4.6732e-08   & 79     &    6.1329e-08  &  57    \\\hline
    38178     &    71728    & 2.7356e-07    & 200     &    8.5303e-08    &52      &   3.6577e-08   & 42     \\\hline
    37730    & 3.9579e+05  &    0.002827   & 121    &     6.8812e-08   & 76      &   3.2188e-08  &  58     \\\hline
\end{tabular}
\end{table}
\begin{table}
\centering
\caption{Sparse Random Matrix Using Row Sampling $m = 40000, n = 200.$}
\begin{tabular}{|*{8}{c|}}\hline
     Nnz    &   Conde    &    PCG.res    &  PCG.iter    &  TW.res    &  TW.iter  &   AMG.res    &  AMG.iter\\\hline
    38942  &      2132.1  &  8.7014e-08  &  106      &   5.3535e-08  &  30    &     3.6095e-08  &  23      \\\hline
    38228   &      12321   & 9.6274e-08   & 167       &  9.2237e-08   & 44      &   9.9291e-08   & 36      \\\hline
    37718    &     73525   & 2.8338e-05    & 200       &  8.9378e-08   & 75      &   6.1329e-08   & 57      \\\hline
    38178     &   71728    & 2.7356e-07    & 200        & 4.0376e-08    &58       &  3.8837e-08    &42      \\\hline
    37730    & 3.9579e+05  &    0.002827  &  121      &   4.7336e-08   & 81      &   6.2102e-08   & 57  \\\hline

\end{tabular}
\end{table}

\begin{itemize}
\item The iteration steps depend weakly on the condition number $\kappa(A). $
\end{itemize}

%------------------------------------------------
\subsubsection{UDV Matrix}
Performance: UDV Matrix
UDV matrix is a random matrix generated by
$$
A = U D V,
$$
where $U$ is an $m \times n$ orthonormal matrix, $V$ is an $n \times n$ orthonormal matrix and
$D = diag[1,1+(c-1)/n,\cdots,c]$.

\begin{table}\label{}
\caption{UDV Matrix Using CoarsenAMGa and Row Sampling $m = 10000, n = 200.$}
\begin{tabular}{|*{8}{c|}}\hline
  Nnz    &   Conde  &      PCG.res   &   PCG.iter     & TW.res  &    TW.iter     &   AMG.res &     AMG.iter \\\hline
      40000  &      3468.1 &   9.1949e-08  &  112     &    9.4608e-08  &  29       & 9.9643e-08  &  25      \\\hline
    40000   &      16679  &  8.3161e-08 &   192     &    5.4534e-08   & 50           & 5.7913e-08    &43      \\\hline
    40000  & 1.0667e+05  &  3.5074e-06 &   198    &     9.0271e-08 &   73      &   7.1027e-08   & 61     \\\hline
    40000  &  4.0013e+05  &   2.199e-05  &  190     &    8.3462e-08  &  81       &  6.5267e-08   & 66     \\\hline
    40000   & 1.0203e+06  &  1.8215e-05  &  197    &     9.5749e-08   & 83        & 1.1792e-08  &  68    \\\hline
\end{tabular}
\end{table}

%------------------------------------------------

\subsubsection{Graph Laplacian Matrix}
Performance: Graph Laplacian Matrix
Given edges and generate the graph laplacian matrix $G$ by
\begin{lstlisting}
        i = repmat(1:size(edge,1),1,2)';
        i = i(:);
        j = edge(:);
        s = repmat([-1 1],size(edge,1),1);
        s = s(:);
        B = sparse(i,j,s);
        G = B'*B;
\end{lstlisting}

\begin{table}\label{}
\caption{Graph Laplacian Matrix Using CoarsenAMGc $m = 9314,n=100$}
\begin{tabular}{|*{8}{c|}}\hline
  Nnz    &   Conde  &      PCG.res   &   PCG.iter     & TW.res  &    TW.iter    &   AMG.res &     AMG.iter \\\hline
9214     &   10.273  &   4.0992e-08   & 17       &   4.4332e-08  &  12         &5.7114e-10  &   4    \\\hline
    9214  &      112.71 &   6.6875e-08  &  37    &      7.0856e-08 &   12      &   6.4333e-08&     7   \\\hline
    9214  &      1657.1 &   8.3804e-08  &  47    &       8.406e-08 &   15    &   9.248e-08  &   9     \\\hline
    9214  &       26188 &    4.0137e-08  &  51   &       8.7113e-08  &  17     &   4.2357e-08  &  10   \\\hline
    9214  &  4.1839e+05 &   6.1113e-08   & 41   &       3.2572e-08  &  19     &    3.5725e-08 &   11  \\\hline
\end{tabular}
\end{table}

%------------------------------------------------

\subsection{Convergence Analysis}
\begin{theorem}[Sum of Rank-1 Matrix]
Let $y_1,y_2,\cdots,y_n$ be i.i.d. random column vectors in $\bbbc^d$ with
$$
\| y_i \| \leq M \quad \text{and} \quad \|\E[y_i y_i^*] \| \leq 1.
$$
Then for all $0 \leq \epsilon \leq 1$
$$
Pr \left( \| \frac{1}{n} \sum_{k=1}^n y_k y_k^* - \E [y_1 y_1^T] \| \geq \epsilon \right) \leq
2d exp(-\frac{3 n \epsilon^2}{8 (M^2+1)}).
$$
\end{theorem}

%------------------------------------------------
\subsubsection{Row Sampling}
Let
$$
y_k = \begin{cases} & A_i^T/ \sqrt{p_i}  \quad \text{with probability} \quad p_i, \quad 1 \leq i \leq m;  \\
                               & 0 \quad \quad \text{otherwise}.
        \end{cases}
$$

$$
\E [y_k y_k^T] = \sum_{i=1}^m A_i^T A_i = A^T A.
$$

Randomly choose r $y_i$'s. Put them together to get the sampled matrix $As$. Then
$$
As^T As = \frac{1}{r} \sum_{i=k}^r y_k y_k^T.
$$
By the Theorem (Sum of Rank-1 Matrix)
$$
\| As^T As - A^T A \| \leq \epsilon  \quad \text{with high probability}.
$$

\subsubsection{High Frequency and Low Frequency}

With high probability, $\forall x \neq 0,$
\begin{equation}\label{star}
|(A^TA x,x) - \epsilon^2(x,x) | \leq (As^T As x,x) \leq (A^TA x,x) + \epsilon^2(x,x).
\end{equation}

For the matrix $A^T A$ after normalization, $0 < \lambda_{min} \leq \lambda_{max} \leq c$. \\
For high frequency eigenvectors, i.e.
$$
(A^T Ax,x)  \approx \lambda_{max} \|x\|^2
$$
we have
$$
(A_s^T A_s x,x) \leq c (A^TAx,x).
$$

But for low frequency eigenvectors, we need the coarsening part.

%\begin{thebibliography}{99}

%\bibitem{CON2015} Peter Oswald, Weiqi Zhou
% Convergence analysis for Kaczmarz-type methods in a Hilbert space framework
%\emph{Linear algebra and its Applications} 478(2015),131-161.

%\bibitem{FLSA2011} P. Drineas, M. W. Mahoney, S. Muthukrishnan and T. Sarlos
%\newblock Faster least squares approximation
%\newblock \emph{Numerische Mathematik} (2011) 117: 219. %doi:10.1007/s00211-010-0331-6

%\bibitem{SRFT2008} V. Rokhlin and M. Tygert
%\newblock A Fast Randomized Algorithm for Overdetermined Linear Least-Squares %Regression,
%\newblock  \emph{Proceedings of the National Academy of Science of the United States of America,}  Vpl. 105, No.36(Sep.9,2008),pp. 13212-13217

%\bibitem{BLEN2010} H. Avron, P. Maymounkov and S. Toledo
%\newblock Blendenpik: Supercharging LAPACK's Least-Squares Solver
%\newblock \emph{ SIAM Journal on Scientific Computing}
%32.3 (2010): 1217. c2010 Society for Industrial and Applied Mathematics

%\bibitem{ARK01} Ji Liu and Stephen J. Wright, An Accelerated Randomized Kaczmarz Algorithm, \emph{Mathematics of Computation}, Volume 85, Number 297, January 2016, pp. 153--178.

%\bibitem{ITE01}
%Long Chen, Classical Iterative Methods, available at
%\url{http://www.math.uci.edu/~chenlong/226/Ch6IterativeMethod.pdf}.

%\bibitem{WikiRK} ] Randomized Kaczmarz Method, available at
%\url{https://en.wikipedia.org/wiki/Kaczmarz_method}.

%\end{thebibliography}


\section{Substitution Rule}\label{sec:SubRule}
Needless to say, most integration problems we will encounter will
not be so simply. That is to say we will require more than the basic
integration rules we have seen. Here's a slightly more complicated example: Find
$$\int 2x\cos(x^2)\,dx.$$
This is not a ``simple'' derivative, but a little thought reveals that
it must have come from an application of the chain rule. Multiplied
on the ``outside'' is $2x$, which is the derivative of the ``inside''
function $\ds x^2$. Checking:
$${d\over dx}\sin(x^2)  = \cos(x^2){d\over dx}x^2 = 2x\cos(x^2),$$
so
$$\int 2x\cos(x^2)\,dx=\sin(x^2)+C .$$

To summarize: If we suspect that a given function is the derivative of
another via the chain rule, we let $u$ denote a likely candidate for
the inner function, then translate the given function so that it is
written entirely in terms of $u$, with no $x$ remaining in the
expression. If we can integrate this new function of $u$, then the
antiderivative of the original function is obtained by replacing $u$
by the equivalent expression in $x$.

\begin{theorem}{Substitution Rule for Indefinite Integrals}{SubstitutionRule}
If $u=g(x)$ is a differentiable function whose range is an interval $I$ and $f$ is continuous on $I$, then
$$\int f(g(x))g'(x)\,dx=\int f(u)\,du.$$
\end{theorem}

Even in simple cases you may prefer to use this mechanical procedure,
since it often helps to avoid silly mistakes. For example, consider
again this simple problem:
$$\int 2x\cos(x^2)\,dx.$$
Let $\ds u=x^2$, then $du/dx = 2x$ or $du = 2x\,dx$. Since we have exactly
$2x\,dx$ in the original integral, we can replace it by $du$:
$$\int 2x\cos(x^2)\,dx=\int \cos u\,du=\sin u +C = \sin(x^2)+C.$$
This is not the only way to do the algebra, and typically there are
many paths to the correct answer. Another possibility, for example,
is: Since $du/dx = 2x$, $dx=du/2x$, and then the integral becomes
$$\int 2x\cos(x^2)\,dx=\int 2x\cos u\,{du\over 2x}=\int \cos u\,du.$$
The important thing to remember is that you must eliminate all
instances of the original variable $x$.

\begin{example}{Substitution Rule}{SubstitutionRuleex}
Evaluate $\ds\int(ax+b)^n\,dx$, assuming $a,b$ are
constants, $a\not=0$, and $n$ is a positive integer.
\end{example}

\begin{solution}
We let $u=ax+b$ so $du=a\,dx$ or $dx=du/a$. Then
$$
  \int(ax+b)^n\,dx=\int {1\over a} u^n\,du={1\over a(n+1)}u^{n+1}+C=
  {1\over a(n+1)}(ax+b)^{n+1}+C.
$$
\end{solution}

\begin{example}{Substitution Rule}{SubstitutionRulex}
Evaluate $\ds\int \sin(ax+b)\,dx$, assuming that $a$ and $b$ are
constants and $a\not=0$.
\end{example}

\begin{solution}
Again we let $u=ax+b$ so $du=a\,dx$ or $dx=du/a$. Then
$$
  \int\sin(ax+b)\,dx=\int {1\over a} \sin u\,du={1\over a}(-\cos u)+C=
-{1\over a}\cos(ax+b)+C.
$$
\end{solution}

\begin{formulabox}[Strategy for Substitution Rule]
A general strategy to follow is:
\begin{enumerate}\setlength{\itemsep}{0 in}
\item Choose a possible $u=u(x)$. \dfont{Tip:} Choose a substitution $u$ so that its derivate also appears in the integral (up to a constant).
\item Calculate $du=u'(x)~dx$.
\item Either replace $u'(x)~dx$ by $du$, or replace $dx$ by $\ds{\frac{du}{u'(x)}}$, and cancel.
\item Write the rest of the integrand in terms of $u$. If this is not possible, the substitution will not work: You must go back to step 1.
\item Find the indefinite integral. (Again, if this is not possible, try a different substitution, or a different method).
\item Rewrite the result in terms of $x$.
\end{enumerate}
\end{formulabox}

\begin{example}{Substitution}{Substitution}
Evaluate the following integral: $\ds\int \frac{2x}{\sqrt{1-4x^2}}\,dx.$
\end{example}

\begin{solution}
We try the substitution:
$$u=1-4x^2.$$
Then,
$$du=-8x~dx$$
In the numerator we have $2x~dx$, so rewriting the differential gives:
$$-\frac{1}{4}du=2x~dx.$$
Then the integral is:
\begin{eqnarray*}
\int \frac{2x}{\sqrt{1-4x^2}}\,dx&=&\int \left(1-4x^2\right)^{-1/2}(2x~dx)\\
\\
&=&\int u^{-1/2}\left(-\frac{1}{4}du\right)\\
\\
&=&\left(\frac{-1}{4}\right)\frac{u^{1/2}}{1/2}+C\\
\\
&=&-\frac{\sqrt{1-4x^2}}{2}+C
\end{eqnarray*}
\end{solution}

\begin{example}{Substitution}{Substitution}
Evaluate the following integral: $\ds\int \sech^2(7t-3)\ dt$
\end{example}

\begin{solution}
 We employ substitution, with $u = 7t-3$ and $du = 7dt$. We have:
$$ \int \sech^2 (7t-3)\ dt=  \frac17 \int \sech^2 (u)\ du= \frac17\tanh (u) + C = \frac17\tanh (7t-3) + C.$$
\end{solution}

\begin{example}{Substitution}{Substitution2}
Evaluate the following integral: $\ds\int \cos x(\sin x)^5\,dx.$
\end{example}

\begin{solution}
In this question we will let $u=\sin x$.
Then,
$$du=\cos x~dx.$$
Thus, the integral becomes:
\begin{eqnarray*}
\int \cos x(\sin x)^5\,dx&=&\int u^5\,du\\
\\
&=&\frac{u^6}{6}+C\\
\\
&=&\frac{(\sin x)^6}{6}+C
\end{eqnarray*}
\end{solution}

\begin{example}{Substitution}{Substitution3}
Evaluate the following integral:
$\ds\int \frac{\cos(\sqrt x)}{\sqrt x}\,dx.$
%\vspace{-0.5cm}
\end{example}

\begin{solution}
We use the substitution:
$$u=x^{1/2}.$$
Then,
$$du=\frac{1}{2}x^{-1/2}dx.$$
Rewriting the differential we get:
$$2~du=\frac{1}{\sqrt x}~dx.$$
The integral becomes:
\begin{eqnarray*}
\int \frac{\cos(\sqrt x)}{\sqrt x}\,dx&=&2\int \cos u\,du\\
\\
&=&2\sin u+C\\
\\
&=&2\sin(\sqrt x)+C
\end{eqnarray*}
\end{solution}

\begin{example}{Substitution}{Substitution4}
Evaluate the following integral:
$\ds\int 2x^3\sqrt{x^2+1}\,dx.$
%\vspace{-0.5cm}
\end{example}

\begin{solution}
This problem is a little bit different than the previous ones.
It makes sense to let:
$$u=x^2+1,$$
then
$$du=2x~dx.$$
Making this substitution gives:
\begin{eqnarray*}
\int 2x^3\sqrt{x^2+1}\,dx&=&\int x^2\sqrt{x^2+1}(2x)\,dx\\
\\
&=&\int x^2u^{1/2}\,du\\
\end{eqnarray*}
This is a problem because our integrals can't have a mixture of two variables in them.
Usually this means we chose our $u$ incorrectly.
However, in this case we can eliminate the remaining $x$'s from our integral by using:
$$u=x^2+1\quad\to\quad x^2=u-1.$$
We get:
\begin{eqnarray*}
\int x^2u^{1/2}\,du&=&\int (u-1)u^{1/2}\,du\\
\\
&=&\int u^{3/2}-u^{1/2}\,du\\
\\
&=&\frac{2}{5}u^{5/2}-\frac{2}{3}u^{3/2}+C\\
\\
&=&\frac{2}{5}(x^2+1)^{5/2}-\frac{2}{3}(x^2+1)^{3/2}+C
\end{eqnarray*}
\end{solution}

The next example shows how to use the Substitution Rule when dealing with definite integrals.

\begin{example}{Substitution Rule}{SubstitutionRuledefex}
Evaluate $\ds\int_2^4 x\sin(x^2)\,dx$.
\end{example}

\begin{solution}
First we compute the
antiderivative, then evaluate the integral.
Let $\ds u=x^2$, so $du=2x\,dx$ or $x\,dx=du/2$. Then
$$
  \int x\sin(x^2)\,dx=\int {1\over 2} \sin u\,du={1\over 2}(-\cos u)+C=
  -{1\over 2}\cos(x^2)+C.
$$
Now
$$
  \int_2^4 x\sin(x^2)\,dx=\left.-{1\over 2}\cos(x^2)\right|_2^4
  =-{1\over 2}\cos(16)+{1\over 2}\cos(4).
$$
A somewhat neater alternative to this method is to change the original
limits to match the variable $u$. Since $\ds u=x^2$, when $x=2$, $u=4$,
and when $x=4$, $u=16$. So we can do this:
$$
  \int_2^4 x\sin(x^2)\,dx=
  \int_4^{16} {1\over 2} \sin u\,du=\left.-{1\over 2}(\cos u)\right|_4^{16}
  =-{1\over 2}\cos(16)+{1\over 2}\cos(4).
$$
An incorrect, and dangerous, alternative is something like this:
$$
  \int_2^4 x\sin(x^2)\,dx=\int_2^4 {1\over 2} \sin u\,du=
  \left.-{1\over 2}\cos (u)\right|_2^4=
  \left.-{1\over 2}\cos(x^2)\right|_2^4=-{1\over 2}\cos(16)+{1\over
  2}\cos(4).
$$
This is incorrect because $\ds\int_2^4 {1\over 2} \sin u\,du$
means that $u$ takes on values between 2 and 4, which is wrong. It
is dangerous, because it is very easy to get to
the point $\ds\left.-{1\over 2}\cos (u)\right|_2^4$ and forget to substitute
$\ds x^2$ back in for $u$, thus getting the incorrect answer
$\ds -{1\over 2}\cos(4)+{1\over 2}\cos(2)$. An acceptable alternative is something like:
$$
  \int_2^4 x\sin(x^2)\,dx=\int_{x=2}^{x=4} {1\over 2} \sin u\,du=
  \left.-{1\over 2}\cos (u)\right|_{x=2}^{x=4}=
  \left.-{1\over 2}\cos(x^2)\right|_2^4=-{\cos(16)\over 2}+{\cos(4)\over2}.
$$
\end{solution}

To summarize, we have the following.

\begin{theorem}{Substitution Rule for Definite Integrals}{SubstitutionRuledef}
If $g'$ is continuous on $[a,b]$ and $f$ is continuous on the range of $u=g(x)$, then
$$\int_a^b f(g(x))g'(x)\,dx=\int_{g(a)}^{g(b)}f(u)\,du.$$
\end{theorem}

\begin{example}{Substitution Rule}{SubstitutionRuledef2}
Evaluate $\ds\int_{1/4}^{1/2}{\cos(\pi t)\over\sin^2(\pi t)}\,dt$.
\end{example}

\begin{solution}
Let $u=\sin(\pi t)$ so $du=\pi\cos(\pi t)\,dt$ or $du/\pi=\cos(\pi
t)\,dt$.
We change the limits to $\ds \sin(\pi/4)=\sqrt2/2$ and
$\sin(\pi/2)=1$.
Then
$$
  \int_{1/4}^{1/2}{\cos(\pi t)\over\sin^2(\pi t)}\,dt=
  \int_{\sqrt2/2}^{1}{1\over \pi}{1\over u^2}\,du=
  \int_{\sqrt2/2}^{1} {1\over \pi}u^{-2}\,du=
  \left.{1\over \pi}{u^{-1}\over -1}\right|_{\sqrt2/2}^{1}=
  -{1\over\pi}+{\sqrt2\over\pi}.
$$
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for Section \ref{sec:SubRule}}

\begin{enumialphparenastyle}

Find the following indefinite and definite integrals.

%%%%%%%%%%
\begin{ex}
 $\ds\int (1-t)^9\,dt$
\begin{sol}
 $\ds -(1-t)^{10}/10+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int (x^2+1)^2\,dx$
\begin{sol}
 $\ds x^5/5+2x^3/3+x+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int x(x^2+1)^{100}\,dx$
\begin{sol}
 $\ds (x^2+1)^{101}/202+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over\root 3 \of {1-5t}}\,dt$
\begin{sol}
 $\ds -3(1-5t)^{2/3}/10+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int \sin^3x\cos x\,dx$
\begin{sol}
 $\ds (\sin^4x)/4+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int x\sqrt{100-x^2}\,dx$
\begin{sol}
 $\ds -(100-x^2)^{3/2}/3+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int {x^2\over\sqrt{1-x^3}}\,dx$
\begin{sol}
 $\ds \ds -2\sqrt{1-x^3}/3+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int \cos(\pi t)\cos\bigl(\sin(\pi t)\bigr)\,dt$
\begin{sol}
 $\ds \sin(\sin\pi t)/\pi+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int {\sin x\over\cos^3 x}\,dx$
\begin{sol}
 $\ds \ds 1/(2\cos^2 x)=(1/2)\sec^2x+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int\tan x\,dx$
\begin{sol}
 $-\ln|\cos x|+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
  $\ds\int_0^\pi\sin^5(3x)\cos(3x)\,dx$
\begin{sol}
 $0$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int\sec^2x\tan x\,dx$
\begin{sol}
 $\ds \tan^2(x)/2+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int_0^{\sqrt{\pi}/2} x\sec^2(x^2)\tan(x^2)\,dx$
\begin{sol}
 $1/4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int {\sin(\tan x)\over\cos^2x}\,dx$
\begin{sol}
 $-\cos(\tan x)+C$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int_3^4 {1\over(3x-7)^2}\,dx$
\begin{sol}
 $1/10$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int_0^{\pi/6}(\cos^2x - \sin^2x)\,dx$
\begin{sol}
 $\ds \sqrt3/4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int {6x\over(x^2 - 7)^{1/9}}\,dx$
\begin{sol}
 $\ds (27/8)(x^2-7)^{8/9}$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int_{-1}^1 (2x^3-1)(x^4-2x)^6\,dx$
\begin{sol}
 $\ds -(3^7+1)/14$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int_{-1}^1 \sin^7 x\,dx$
\begin{sol}
 $0$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds\int f(x) f'(x)\,dx$
\begin{sol}
 $\ds f(x)^2/2$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


If you haven't downloaded and unzipped \href{https://libaoj.in/courses/2021f/MATH3341/zip/Math.3341.zip}{\texttt{Math.3341.zip}}. Download and unzip it under \verb|H:| (H Drive if you are working on the Remote Lab). Change the current working directory by typing \verb|cd H:\Math.3341\Math.3341.Lab.03| in the Command Window, and type \verb|edit lab_03_script| in the Command Window to edit \verb|lab_03_script.m|.

%---------------------------------------------
\section{Anonymous Functions}
%---------------------------------------------
\begin{enumerate}[(a)]
  \item Define an anonymous function \verb|rowSums| which calculates the row sums of a matrix of any dimension. Then define \verb|magicMat5| and \verb|magicMat7| to be a $5 \times 5$ and a $7 \times 7$ magic square matrix, respectively. Compute \verb|magicMat5RowSums| by calling \verb|rowSums(magicMat5)|, and compute \verb|magicMat7RowSums| by calling \verb|rowSums(magicMat7)|.
  \item Define anonymous functions \verb|f| and \verb|g|, where $f(x) = x \ln(x)$ and $g(y) = y e^{y}$. Create another anonymous function \verb|h| by composing \verb|f| and \verb|g|, i.e., $h(z) = g(f(z))$. Use \verb|linspace| to define a \emph{column vector} \verb|z|, of which the range is from $1$ to $5$ with $11$ entries. Evaluate function \verb|h| at \verb|z|, and assign the result to \verb|hz|.
  \item Define an anonymous function \verb|matProd| for calculating the product of two matrices, that is, $matProd(A, B) = ABB^{T} A^{T}$. Define \verb|A| and \verb|B| using \verb|colon|, \verb|reshape| and \verb|transpose| as follows,
    \begin{equation*}
      A = \begin{bmatrix}
        1 & 2 & 3 \\
        4 & 5 & 6
      \end{bmatrix},
      B = \begin{bmatrix}
        7 & 10 & 13 & 16 \\
        8 & 11 & 14 & 17 \\
        9 & 12 & 15 & 18
      \end{bmatrix}.
    \end{equation*}
    Store the result of \verb|matProd(A, B)| to \verb|matProdAB|.
  \item Define an anonymous function \verb|p|, where
    \begin{equation*}
      p(x) = \begin{cases}
        x^3 & \text{if $x < -1$}, \\
        x & \text{if $-1 \leq x \leq 1$}, \\
        x^2 & \text{if $x > 1$}.
      \end{cases}
    \end{equation*}
    Next visualize $p(x)$ using \verb|fplot| on [-2, 2] (Use \verb|help fplot| for more details about \verb|fplot|). Then run \verb|print(gcf, '-dpng', 'lab_03_1d.png')| to save the plot to a \verb|.png| file.
\end{enumerate}

%---------------------------------------------
\section{Function Files}
%---------------------------------------------
\begin{minipage}[t]{0.50\linewidth}
\begin{algorithm}[H]
    \label{alg:recursive}
    \caption{Recursive Factorial}
    \SetCommentSty{\color{gray}}
    \SetAlgoLined
    \LinesNumbered
    \SetKwProg{Fn}{Function}{:}{end}
    \Fn{$\operatorname{factorialRecursive}(n)$}{
        \KwIn{$n$: an nonnegative integer}
        \KwOut{$f$: $n!$}
        \uIf{$n = 0$} {
            $f \gets 1$\;
        }
        \Else {
            $f \gets n \times \operatorname{factorialRecursive}(n - 1)$\;
        }
    }
\end{algorithm}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\linewidth}
\begin{algorithm}[H]
    \label{alg:iterative}
    \caption{Iterative Factorial}
    \SetCommentSty{\color{gray}}
    \SetAlgoLined
    \LinesNumbered
    \SetKwProg{Fn}{Function}{:}{end}
    \Fn{$\operatorname{factorialIterative}(n)$}{
        \KwIn{$n$: an nonnegative integer}
        \KwOut{$f$: $n!$}
        $f \gets 1$\;
        \For{$i \gets 1$ \KwTo $n$} {
            $f \gets f \times i$;
        }
    }
\end{algorithm}
\end{minipage}

\begin{enumerate}[(a)]
    \item Create a function file \verb|factorialRecursive.m| to implement the pseudocode in Algorithm \ref{alg:recursive}.
    \item Create a function file \verb|factorialIterative.m| to implement the pseudocode in Algorithm \ref{alg:iterative}.
    \item In the script file \verb|lab_03_script.m|, use a for-loop to calculate $n!$ where $n = 1, \ldots, 20$ by calling the above two function files as follows
        \begin{lstlisting}[style=MATLAB]
fprintf('%2s %20s %20s\n', 'n', 'factorialRecursive', 'factorialIterative');
for n = 1:20
    f1 = factorialRecursive(n);
    f2 = factorialIterative(n);
    fprintf('%2d %20d %20d\n', n, f1, f2);
end
        \end{lstlisting}
\end{enumerate}

%---------------------------------------------
\section{Application: Real-Life Problems}
%---------------------------------------------

\begin{enumerate}[(a)]
  \item Create a function file \verb|dayOfWeek.m| to calculate day of week of a specific date. It is known that January 1st, 1970 is Thursday. In the script file \verb|lab_03_script|, calculate the day of week for 01-07-1970, 03-07-1970, 03-08-1971, 08-08-1988, 09-09-1999, 02-10-2021. Here is the suggested syntax for the function: \verb|d = dayOfWeek(year, month, day)|. For example, calling \verb|dayOfWeek(1970, 1, 1)| should return \verb|'Thursday'|. You may use the provided function file \verb|isLeapYear.m|, use \verb|help isLeapYear| for more information. You should use both \verb|if| and \verb|swtich| statements.

    Hint: Given that 01-01-1970 is a Thursday, your job is to determine the day of week of a specific date. You can compute the total number of days elapsed since 01-01-1970, disregard the number of weeks past between the dates and add the remainder to the day of week of 01-01-1970 to obtain the day of week of the given date. For example,
    \begin{itemize}
      \item 01-02-1970: 1 day elapsed since 01-01-1970, Thursday + 1 day = Friday. Therefore, 01-02-1970 is a Friday;
      \item 01-08-1970: 7 days elapsed since 01-01-1970, Thursday + 7 days = Thursday + 1 week = Thursday. Therefore, 01-08-1970 is a Thursday;
      \item 03-02-1970: 31 (number of days in Jan) + 28 (number of days in Feb) + 2 (number of days past in March) $- 1 = 60$ days elapsed since 01-01-1970, Thursday + 60 days = Thursday + 8 week + 4 days = Thursday + 4 days = Sunday + 1 day = Monday. Therefore, 03-02-1970 is a Monday;
    \end{itemize}
  \item Shop A is selling a beverage which costs \$2 per bottle and there are rules for promotional sales for the beverage:
    \begin{itemize}
      \item You can exchange 4 caps for 1 full bottle of the same beverage for free;
      \item You can exchange 2 empty bottles (without caps) for 1 full bottle of the same beverage for free.
    \end{itemize}
    You have $\$10$ in your pocket. What is the maximum number of bottles of the beverage you can get? Solve this question by writing a function \verb|maxBeverageBottles|.
    \begin{itemize}
      \item The function can be called as below:

        \verb|maxBeverageBottles(money, pricePerNewBottle, capsPerNewBottle, emptyBottlesPerNewBottle)|
        which returns the maximum number of bottles you can get with \verb|money|.
      \item \verb|money| is the amount of money available;
      \item \verb|pricePerNewBottle| is the unit price for buying a new bottle;
      \item \verb|capsPerNewBottle| is the number of caps needed for exchanging 1 free bottle;
      \item \verb|emptyBottlesPerNewBottle| is the number of empty bottles needed for exchanging 1 free bottle.
    \end{itemize}
    Then in the script file, store the result of calling \verb|maxBeverageBottles(10, 2, 4, 2)| to \verb|maxBottles1|. What if you have $\$1000$ in your pocket, the beverage still costs $\$2$/bottle, but $5$ caps/bottle or $3$ empty bottles/bottle for exchanging a new free bottle, how many bottles can you get? Store the result to \verb|maxBottle2|. You may find \verb|floor| and \verb|mod| useful.

    Hint: For each new bottle you get, you have a new pair of cap and empty bottle (after consumed). So you can get new bottles until no enough money/caps/empty bottles.
\end{enumerate}

Before proceeding, make sure you suppress the output in the function files and do \textsc{not} suppress the output in the script file. In the Command Window, enter the command \verb`diary('lab_03_output.txt')`, run the script file \verb|lab_03_script.m|, then type \verb`diary off` to store the output to \verb`lab_03_output.txt`. Then upload the script file \verb|lab_03_script.m|, plot file \verb|lab_03_1d.png|, output file \verb|lab_03_output.txt|, and function files \verb|factorialIterative.m|, \verb|factorialRecursive.m|, \verb|dayOfWeek.m|, \verb|maxBeverageBottles.m| to the folder \verb|src| on Overleaf.


\lab{Markov Chains}{Markov Chains}
\label{lab:Markov}
\objective{
A \emph{Markov chain} is a finite collection of states with specified probabilities for transitioning from one state to another.
They are characterized by the fact that the future behavior of the system depends only on its current state.
Markov chains have far ranging applications; in this lab, we create a Markov chain for generating random English sentences.}

% TODO: Changing column_sum = 1 to row_sum = 1 (transpose all transition matrices, ij^th entry --> probability of going from from state i to state j (instead of the other way around). Makes implementing everything easier.)

% TODO: Update test driver, solutions+spec on class implementation.

\section*{State Space Models} % ===============================================

Many systems can be described by a finite number of states.
% A Markov chain is a collection of states, together with the probabilities of moving from one state to another.
For example, a board game where players move around the board based on die rolls can be modeled by a Markov chain.
Each space represents a state, and a player is said to be in a state if their piece is currently on the corresponding space.
In this case, the probability of moving from one space to another only depends on the players current location: where the player was on a previous turn does not affect their current turn.

Finite Markov chains have an associated transition matrix that stores all the information about the chain.
The $(ij)^{th}$ entry of the matrix gives the probability of moving from state $j$ to state $i$.
Thus the columns of the transition matrix must sum to 1. %\footnote{This type of transition matrix is called \emph{column-stochastic}. Row stochastic matrices are also often used ti model Markov chains.}
% TODO: Note about column-stochastic v row-stochastic transition matrices.

Consider a very simple weather model where the probability of being hot or cold depends on the weather of the previous day.
If the probability that tomorrow is hot given that today is hot is 0.7, and the probability that tomorrow is cold given that today is cold is 0.4, then by assigning hot to the $0^{th}$ row and column, and cold to the $1^{st}$ row and column, this Markov chain has the following transition matrix:

\[W = \left[\begin{array}{cc}
0.7 & 0.6 \\
0.3 & 0.4 \end{array} \right]\]

% TODO: insert a PICTURE of a this Markov chain!

If it is hot today, we examine the $0^{th}$ column of $W$.
There is a $70\%$ chance that tomorrow will be hot ($0^{th}$ row), and a $30\%$ chance that tomorrow will be cold ($1^{st}$ row).
Conversely, if it is cold today, there is a $60\%$ chance that tomorrow will be hot, and a $40\%$ chance that tomorrow will be cold.

\begin{problem} % Problem: stochasticity.
Transition matrices for Markov chains are efficiently stored as NumPy arrays.
Write a function that accepts a dimension $n$ and returns the transition matrix for a random Markov chain with $n$ states.
\\
(Hint: use array broadcasting to avoid looping.)
\end{problem}

\subsection*{Simulating State Transitions} % ----------------------------------

In a general finite Markov chain, if we are in state $j$ then the $j^{th}$ column of the transition matrix gives the probabilities of moving to any other state $i$.
By definition, these probabilities sum to $1$.
Thus, the entries of each column partition the interval $[0, 1]$, and we can choose the next state to move to by generating a random number between $0$ and $1$.

Consider again the weather model example from the previous section.
Suppose that today is hot, and that we want to simulate tomorrow's weather.
The column that corresponds to ``hot'' in the transition matrix is $[0.7, 0.3]^T$.
If we generate a random number and it is smaller than $0.3$, then our simulation indicates that tomorrow will be cold.
Conversely, if the random number is between $0.3$ and $1$, then the simulation says that tomorrow will be hot.
The following code implements this idea.

\begin{lstlisting}
import numpy as np

def forecast():
	"""Forecast tomorrow's weather given that today is hot."""

	transition_matrix = np.array([[0.7, 0.6], [0.3, 0.4]])
	# Sample from the standard uniform distribution to choose a new state.
	if np.random.random() < transition_matrix[1,0]:
		print "Cold"
		return 1
	else:
		print "Hot"
		return 0
\end{lstlisting}

\begin{problem} % Problem: Forecasting over several days.
Modify \li{forecast()} so that it accepts a parameter \li{days} and runs a simulation of the weather for the number of days given.
Return a list containing the day-by-day weather predictions (0 for hot, 1 for cold).
Assume the first day is hot, but do not include the data from the first day in the list of predictions.
The resulting list should therefore have \li{days} entries.
\end{problem}

For Markov chains with very few states, the approach in \li{forecast()} is practical and the implementation is fairly simple.
However, small Markov chains are typically not very useful in applications.

\subsection*{Larger Chains} % -------------------------------------------------

The \li{forecast()} function makes one random draw from a \emph{uniform} distribution to simulate a state change.
For larger Markov chains, we draw from a \emph{multinomial} distribution.
A multinomial distribution is a multivariate generalization of the binomial distribution.
A single draw from a binomial distribution with parameter $p$ indicates successes or failure of a single experiment with probability $p$ of success.
The classic example is a coin flip, where the $p$ is the probability that the coin lands heads side up.
A single draw from a multinomial distribution with parameters $\left(p_1, p_2, ..., p_n \right)$ indicates which of $n$ outcomes occurs.
In this case the classic example is a dice roll, with $6$ possible outcomes instead of the $2$ in a coin toss.

\begin{lstlisting}
# To simulate a single dice roll, store the probabilities of each outcome.
>>> probabilities = np.array([1./6, 1./6, 1./6, 1./6, 1./6, 1./6])

# Make a single random draw (roll the die once).
>>> np.random.multinomial(1, probabilities)
array([0, 0, 0, 1, 0, 0])                       # The roll resulted in a 4.
\end{lstlisting}

\begin{problem} % Problem: 4 states instead of 2. Multinomial transitioning.
Let the following be the transition chain for a Markov chain modeling weather with four states: hot, mild, cold, and freezing.

\[ W^\prime = \left[\begin{array}{cccc}
0.5 & 0.3 & 0.1 & 0\\
0.3 & 0.3 & 0.3 & 0.3\\
0.2 & 0.3 & 0.4 & 0.5\\
  0 & 0.1 & 0.2 & 0.2\end{array} \right]\]
with hot, mild, cold, and freezing corresponding to columns (and rows) 0, 1, 2, and 3, respectively.

Write a new function that accepts a parameter \li{days} and runs the same kind of simulation as \li{forecast()}, but that uses the new four-state transition matrix.
This time, assume the first day is mild.
Return a list containing the day-to-day results (0 for hot, 1 for mild, 2 for cold, and 3 for freezing).
\label{problem:transition}
\end{problem}

\begin{problem} % Problem: Analysis of results.
Write a function that investigates and interprets the results of the simulations in the previous two problems.
Specifically, find the average percentage of days that are hot, mild, cold, and freezing in each simulation.
Does changing the starting day alter the results?
Print a report of your findings.
\end{problem}

\section*{Using Markov Chains to Simulate English} % ==========================
% TODO: is it okay to make this reference?
One of the original applications of Markov chains was to study natural languages.\footnote{In computer science, a \emph{natural language} is a spoken language, like English or Russian. See \url{http://langvillea.people.cofc.edu/MCapps7.pdf} for some details on the early applications of Markov chains, including the study of natural languages.}
In the early $20^{th}$ century, Markov used his chains to model how Russian switched from vowels to consonants.
By mid-century, they had been used as an attempt to model English.
It turns out that Markov chains are, by themselves, insufficient to model very good English.
However, they can approach a fairly good model of bad English, with sometimes amusing results.

By nature, a Markov chain is only concerned with its current state.
Thus a Markov chain simulating transitions between English words is completely unaware of context or even of previous words in a sentence.
For example, a Markov chain's current state may be the word ``continuous.''
Then the chain would say that the next word in the sentence is more likely to be ``function'' rather than ``raccoon.''
However, without the context of the rest of the sentence, even two likely words stringed together may result in gibberish.

We restrict ourselves to a subproblem of modeling the English of a specific file.
The transition probabilities of the resulting Markov chain will reflect the sort of English that the source authors speak.
Thus the Markov chain built from \emph{The Complete Works of William Shakespeare} will differ greatly from, say, the Markov chain built from a collection of academic journals.
We will call the source collection of works in the next problems the \emph{training set}.

\subsection*{Making the Chain} % ----------------------------------------------

In the weather models of the previous section, we chose a fixed number of states to simulate.
However, in English, sentences are of varying length.
To capture this feature in a Markov model, we include a \emph{start state} and an \emph{end state} in the model in addition to the words of the training set.
The start state should only transition to words that appear at the beginning of a sentence in the training set, and only words that appear at the end a sentence in the training set should transition to the end state.
Thus if a training set has $N$ unique words, the transition matrix will be $(N+2) \times (N+2)$.

After determining the states in the Markov chain, we need to determine the transition probabilities between the states.
As with the weather models, each state must be assigned a row and column in the transition matrix.
Then, while iterating through the sentences of the training set, we fill in the entries of the transition matrix.
When word $b$ follows word $a$, we add one to the $(b,a)^{th}$ entry of the matrix.
Once we have done this for every adjacent pair of words in the training set, we normalize the columns so that they each sum to one.

Consider the following small training set as an example.

\begin{lstlisting}
<<I am Sam Sam I am.
Do you like green eggs and ham?
I do not like them, Sam I am.
I do not like green eggs and ham.>>
\end{lstlisting}

If we include punctuation (so ``ham?'' and ``ham.'' are counted as distinct words) and do not alter the capitalization (so ``Do'' and ``do'' are also different), there are 15 unique words in this training set:
\begin{align*}
\text{``I'', ``am'', ``Sam'', ``am.'', ``Do'', ``you'', ``like'', ``green'',}
\\
\text{``eggs'', ``and'', ``ham?'', ``do'', ``not'', ``them,'', and ``ham.''.}
\end{align*}

With start and stop states, the transition matrix should therefore be $17 \times 17$.
The start state should transition to the words ``I'' and ``Do'', and the words ``am.'', ``ham?'', and ``ham?'' should transition to the end state.
Letting ``{\color[rgb]{.3,.6,.1}\$tart}'' represent the start state and ``{\color{red}\$top}'' represent the stop state, we first count the number of times that each state transitions to another state:

\begin{align*}
\begin{blockarray}{cccccccc}
& \text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & \text{I} & \text{am} & \text{Sam} & & \text{ham.} & \text{\textcolor[rgb]{1,0,0}{\$top}} \\
\begin{block}{c(ccccccc)} % TODO: turn these into hard brackets?
\text{\textcolor[rgb]{.3,.6,.1}{\$tart}} 	& 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{I} 		& 3 & 0 & 0 & 2 & \ldots & 0 & 0\\
\text{am} 		& 0 & 1 & 0 & 0 & \ldots & 0 & 0\\
\text{Sam} 		& 0 & 0 & 1 & 1 & \ldots & 0 & 0\\
& \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\
\text{ham.} 	& 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{\textcolor[rgb]{1,0,0}{\$top}} 		& 0 & 0 & 0 & 0 & \ldots & 1 & 1\\
\end{block}\end{blockarray}
\end{align*}
Now we divide each column by the column sum so that each column sums to one.

\begin{align*}
\begin{blockarray}{cccccccc}
& \text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & \text{I} & \text{am} & \text{Sam} & & \text{ham.} & \text{\textcolor[rgb]{1,0,0}{\$top}} \\
\begin{block}{c(ccccccc)} % TODO: turn these into hard brackets?
\text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{I} 		& 3/4 & 0 & 0 & 2/3 & \ldots & 0 & 0\\
\text{am} 		& 0 & 1/5 & 0 & 0 & \ldots & 0 & 0\\
\text{Sam} 		& 0 & 0 & 1 & 1/3 & \ldots & 0 & 0\\
& \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\
\text{ham.} 	& 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{\textcolor[rgb]{1,0,0}{\$top}} 	& 0 & 0 & 0 & 0 & \ldots & 1 & 1\\
\end{block}\end{blockarray}
\end{align*}

The $3/4$ correctly indicates that 3 out of 4 times, the sentences in the training set start with the word ``I''.
Similarly, the $2/3$ and $1/3$ tell us that ``Sam'' is followed by ``I'' twice and by ``Sam'' once in the training set.
Note that ``am'' (without a period) always transitions to ``Sam'' and that ``ham.'' (with a period) always transitions the stop state.
Finally, to avoid a column of zeros, we place a one in the bottom right hand corner of the matrix (the end state always transitions to itself).

The entire procedure of creating the transition matrix for the Markov chain with words from a file as states, with some hints for implementation, is summarized below in Algorithm \ref{alg:MarkovSentencesTransitionMatrix}.

\newpage % Might be unnecessary in later editions.

\begin{algorithm} % Read a file and convert it into a Markov chain.
\begin{algorithmic}[1]
\Procedure{MakeTransitionMatrix}{}
\State Count the number of unique words in the training set.
\State Initialize a square array of zeros of the appropriate size to be the transition matrix (remember to account for the start and stop states).
\State Initialize a list of states, beginning with ``\$tart''.
\For {each sentence in the training set}
    \State Split the sentence into a list of words.
    \State Add each \emph{new} word to the list of states.
    \State Add 1 to the entry of the transition matrix corresponding to the start state transitioning to the first word of the sentence.
   	\State For each consecutive pair $(a, b)$ of words in the list of words, add 1 to the entry of the transition matrix corresponding to moving from state $a$ to state $b$.
	\item Add 1 to the entry of the transition matrix corresponding to the last word of the sentence transitioning to the stop state.
\EndFor
\State Make sure the stop state transitions to itself.
\State Normalize each column by dividing by the column sums (hint: array broadcasting).
\EndProcedure
\end{algorithmic}
\caption{Convert a training set of sentences into a Markov chain.}
\label{alg:MarkovSentencesTransitionMatrix}
\end{algorithm}

\begin{problem} % Problem: Class that makes a Markov chain from a file.
Write a class called \li{SentenceGenerator}.
The constructor should accept a filename (for the training set).
Read the file and build a transition matrix from its contents.
You may assume that the file has one complete sentence written on each line.
\label{problem:MarkovClassPt1}
\end{problem}

% TODO: Put a picture of the actual Markov chain here and discuss a little.

\begin{problem} % Problem: Create random sentences
Add a method to the \li{SentenceGenerator} class called \li{babble()}.
Begin at the start state and use the strategy from Problem \ref{problem:transition} to transition through the object's Markov chain.
Keep track of the path through the chain and the corresponding path of words.
When the stop state is reached, stop transitioning and terminate the sentence.
Return the resulting sentence as a single string.

For example, your \li{SentenceGenerator} class should be able to create random sentences that sound somewhat like Yoda speaking.
\begin{lstlisting}
>>> yoda = SentenceGenerator("Yoda.txt")
>>> for i in xrange(5):
... 	print(yoda.babble())
...
<<
Impossible to my size, do not!
For eight hundred years old to enter the dark side of Congress there is.
But beware of the Wookiees, I have.
Fear leads to eat as well.
But agree on this, we must, and find your weapon!>>
\end{lstlisting}
\end{problem}

\section*{Additional Material} % ==============================================

\subsection*{Large Training Sets} % -------------------------------------------

The approach in the previous problems begins to fail as the training set grows larger.
For example, a single Shakespearean play may not be large enough to cause memory problems, but \emph{The Complete Works of William Shakespeare} certainly will.

To accommodate larger data sets, consider use a sparse matrix for the transition matrix in instead of a regular NumPy array. %(use the \li{lil_matrix} from the \li{scipy.sparse} library). % Why lil_matrix?
Ensure that the process still works on small training sets, then proceed to larger training sets.
How are the resulting sentences different if a very large training set is used instead of a small training set?

% \subsection*{Natural Language Processing Tools} % -----------------------------

% TODO: A brief overview of the \li{nltk} package.


\subsection{Test 1}
\begin{enumerate}
	\item Consider two intersecting lines $\vec{r_1}(t) = \langle 2, 3, 4t \rangle$ and $\vec{r_2}(t) = \langle 2+t, 3+2t, 0 \rangle$. Give the direction vector of each line. Find the equation of the plane which contains both lines. Draw a diagram of the lines, the plane, and the relevant vectors.\\
	\indent
	The direction vector of a line is the derivative of the position vector.
	\begin{itemize}
		\item Direction 1: $\langle 0, 0, 4 \rangle$
		\item Direction 2: $\langle 1, 2, 0 \rangle$
	\end{itemize}
	A the normal vector of the plane is the cross product of the direction vectors.
	\begin{itemize}
		\item $\vec{n} = \langle 0, 0, 4 \rangle \times \langle 1, 2, 0 \rangle = \langle -8, 4, 0 \rangle$
	\end{itemize}
	The lines intersect when $t = 0$ at $(2,3,0)$.
	So, the plane equation is: $\langle -8, 4, 0 \rangle \cdot \langle x-2, y-3, z \rangle = 0$

	\begin{figure}[h]
		\centering
		\includegraphics[scale=.5]{Images/additionalMaterials/test1_plane}
	\end{figure}

	\item Given the VVF $\vec{r}(t)\langle 10t, 7\cos{t}, 7\sin{t} \rangle$...
	\begin{enumerate}[label=\alph*.]
		\item Compute the unit tangent vector $\hat{T}(t)$ and the unit normal vector $\hat{N}(t)$.
		\begin{equation*}
			\hat{T} = \frac{\vec{r^\prime}(t)}{\norm{\vec{r^\prime}(t)}}
		\end{equation*}
		\begin{equation*}
			\vec{r^\prime}(t) = \langle 10, -7\sin{t}, 7\cos{t} \rangle
		\end{equation*}
		\begin{equation*}
			\norm{\vec{r^\prime}(t)} = \sqrt{10^2 + (-7\sin{t})^2 + (7\cos{t})^2} = \sqrt{149}
		\end{equation*}
		\begin{equation*}
			\hat{T}(t) = \frac{1}{\sqrt{149}}\langle 10, -7\sin{t}, 7\cos{t} \rangle
		\end{equation*}
		\begin{equation*}
			\hat{N}(t) = \frac{\mathrm{d}\hat{T}/\mathrm{d}t}{\norm{\mathrm{d}\hat{T}/\mathrm{d}t}}
		\end{equation*}
		\begin{equation*}
			\frac{\mathrm{d}\hat{T}}{\mathrm{d}t} = \frac{1}{\sqrt{149}} \langle 0, -7\cos{t}, -7\sin{t} \rangle
		\end{equation*}
		\begin{equation*}
			\norm{\frac{\mathrm{d}\hat{T}}{\mathrm{d}t}} = \frac{1}{\sqrt{149}}\sqrt{(-7\cos{t})^2 + (-7\sin{t})^2} = \frac{7}{\sqrt{149}}
		\end{equation*}
		\begin{equation*}
			\hat{N}(t) = \langle 0, -\cos{t},-\sin{t}\rangle
		\end{equation*}

		\item Show that $\hat{T}\perp\hat{N}$ for all $t$.\\
		If $\hat{T}\perp\hat{N}$, then $\hat{T}\cdot\hat{N} = 0$ for all $t$.\\
		\begin{align*}
			\hat{T} \cdot \hat{N} &= \frac{1}{\sqrt{149}}\langle 10, -7\sin{t}, 7\cos{t} \rangle \cdot \langle 0, -\cos{t}, -\sin{t}\rangle	\\
			&= \frac{1}{\sqrt{149}}(0 + 7\sin{t}\cos{t} - 7\sin{t}\cos{t}) = 0 \\
			&\implies \hat{T}\perp\hat{N}
		\end{align*}
	\end{enumerate}

	\item A cannon fires cannonballs with a speed of $20 \text{ m} / \text{s}$. Take acceleration due to gravity to be $g = 10 \text{m} / \text{s}^2$.
	\begin{enumerate}[label=\alph*.]
		\item Starting with a constant acceleration function $\vec{a} = \langle 0, -g \rangle$, find the velocity and position functions ($\vec{r^\prime}(t) \text{ and } \vec{r}(t)$ respectively) of the cannonball if the cannon is fired from an angle $\theta$ with respect to the horizontal. Assume the cannonball is initially positioned at the origin.\\
		We know that velocity is the integral of acceleration.
		\begin{equation*}
			\vec{v}(t) = \vec{r^\prime}(t) = \langle c_1, c_2-gt \rangle
		\end{equation*}
		We are given that the initial speed is $20 \text{ m} / \text{s}$ at an angle $\theta$.
		\begin{equation*}
			\vec{v_0} = 20\langle \cos{\theta}, \sin{\theta} \rangle
		\end{equation*}
		\begin{equation*}
			\vec{v}(t) = \langle 20\cos{\theta}, 20\sin{\theta}-gt \rangle
		\end{equation*}
		We know that position is the integral of velocity.
		\begin{equation*}
			\vec{r}(t) = \langle 20t\cos{\theta} + c_1, 20t\sin{\theta} - \frac{1}{2}gt^2 + c_2 \rangle
		\end{equation*}
		We are given that the cannonball starts at the origin.
		\begin{equation*}
			\vec{r}(t) = \langle 20t\cos{\theta}, 20t\sin{\theta} - \frac{1}{2}gt^2 \rangle
		\end{equation*}
		Taking $g = 10 \text{m} / \text{s}^2$,\\
		\begin{equation*}
			\vec{r}(t) = \langle 20t\cos{\theta}, 20t\sin{\theta}-5t^2 \rangle
		\end{equation*}

		\item What angle $\theta$ should the cannon be fired to hit a target on the ground at a distance $40\text{ m}$ away?\\
		We want to find a point on the trajectory where $y = 0$ and $x = 40$.
		$y = 0$ when $t = 0, 4\sin{\theta}$. We can reasonably eliminate $t = 0$ because this is when the cannon first fires and $x = 0$.\\
		Plugging in $t = 4\sin{\theta}$ to the x-component of position when $x = 40$,
		\begin{align*}
			20\cos{\theta} \cdot 4\sin{\theta} &= 40 \\
			2\sin{\theta}\cos{\theta} &= 1 \\
			\sin{(2\theta)}=1, 2\theta &= \pi/2 \\
			\implies \theta &= \pi/4
		\end{align*}
	\end{enumerate}

	\item Consider the following particle trajectory: $\vec{r}(t) = \langle R\cos{e^t}, R\sin{e^t}, \frac{h}{2\pi}e^t \rangle$ for $t \geq 0$. The shape of the trajectory is a helix with radius $R$ and vertical spacing $h$. Find the arc length function $s(t)$ of the trajectory starting with $s(0) = 0$. Give the arc length reparameterization of the helix.
	\begin{align*}
		s(t) &= \int_{0}^{t}{\norm{\vec{r^\prime}(\tau)}\mathrm{d}\tau}\\
		\vec{r^\prime}(t) &= \langle -Re^{t}\sin{e^t}, Re^{t}\cos{e^t}, \frac{h}{2\pi}e^{t} \rangle
	\end{align*}
	\begin{align*}
		\norm{\vec{r^\prime}(t)} &= \sqrt{(-Re^{t}\sin{e^t})^2 + (Re^{t}\cos{e^t})^2 + (\frac{h}{2\pi}e^t)^2} \\
		&= e^{t}\sqrt{R^2 + \frac{h^2}{4\pi^2}}
	\end{align*}
	\begin{equation*}
		s(t) = \int_{0}^{t}{e^{\tau}\sqrt{R^2 + \frac{h^2}{4\pi^2}}\mathrm{d}\tau} = \sqrt{R^2 + \frac{h^2}{4\pi^2}(e^{t} - 1)}
	\end{equation*}
	Solving for $t$,
	\begin{equation*}
		t = \ln{\left(\frac{s}{\sqrt{R^2 + \frac{h^2}{4\pi^2}}} + 1\right)}
	\end{equation*}
	\begin{equation*}
		\vec{r}(s) = \left< R\cos{\left(\frac{s}{\sqrt{R^2 + \frac{h^2}{4\pi^2}}} + 1\right)}, R\sin{\left(\frac{s}{\sqrt{R^2 + \frac{h^2}{4\pi^2}}} + 1\right)}, \frac{h}{2\pi}\left(\frac{s}{\sqrt{R^2 +\frac{h^2}{4\pi^2}}} + 1\right) \right>
	\end{equation*}

	\item Let $\vec{r}(t)$ be the position function of a particle trapped on the surface of a sphere centered at the origin. Show that $\vec{r}(t)\perp\frac{\mathrm{d}}{\mathrm{d}t}\vec{r}(t)$ for all $t$.\\
	Since $\vec{r}(t)$ is on a sphere, $\norm{\vec{r}(t)} = R$ and $\vec{r}(t) \cdot \vec{r}(t) = R^2$.
	\begin{equation*}
		\frac{\mathrm{d}}{\mathrm{d}t}(\vec{r}(t) \cdot \vec{r}(t)) = 2\vec{r}(t) \cdot \vec{r^\prime}(t)
	\end{equation*}
	\begin{equation*}
		\frac{\mathrm{d}}{\mathrm{d}t}(\vec{r}(t) \cdot \vec{r}(t)) = \frac{\mathrm{d}}{\mathrm{d}t}R^2 = 0
	\end{equation*}
	So,
	\begin{equation*}
		2\vec{r}(t) \cdot \vec{r^\prime}(t) = 0
	\end{equation*}
	and
	\begin{equation*}
		\vec{r}(t) \cdot \vec{r^\prime}(t) = 0
	\end{equation*}
	\begin{equation*}
		\implies \vec{r}(t)\perp\vec{r^\prime}(t)
	\end{equation*}
\end{enumerate}


\chapter{Orbital rotations}
\label{appendix:orbital-rotations}

\begin{dfn}
\label{dfn:normal-matrix}
\thmtitle{Normal matrix}
A square matrix satisfying $\bo{N}\dg\bo{N}=\bo{N}\bo{N}\dg$ is termed \textit{normal}.
Several important kinds of matrices meet this criterion:
\textit{Hermitian matrices}, $\bo{H}\dg=\bo{H}$;
\textit{anti-Hermitian matrices}, $\bo{A}\dg=-\bo{A}$;
and
\textit{unitary matrices}, $\bo{U}\dg=\bo{U}^{-1}$.
Note that Hermitian and anti-Hermitian matrices can always be written as $\bo{X}+\bo{X}\dg$ and $\bo{X}-\bo{X}\dg$.
\end{dfn}

\begin{rmk}
\label{rmk:spectral-theorem}
The spectral theorem\footnote{See \url{https://en.wikipedia.org/wiki/Spectral_theorem}} for normal matrices says that $\bo{N}=\bo{V}\widetilde{\bo{N}}\bo{V}\dg$ where $\bo{V}$ is unitary and $\widetilde{\bo{N}}$ is diagonal.
A direct corollary\footnote{Since there exists a basis in which $\bo{N}$ is diagonal, statements about $\bo{N}$ translate into statements about its eigenvalues.} is that the eigenvalues of Hermitian, anti-Hermitian, and unitary matrices can be written as follows.
\begin{align}
  h^*
=
  h
\implies
  h
=
  \f
&&
  a^*
=
-
  a
\implies
  a
=
  i\f
&&
  u^*
=
  u^{-1}
\implies
  u
=
  e^{i\f}
&&
  \f
\in
  \mb{R}
\end{align}
In words, Hermitian eigenvalues are real, anti-Hermitian eigenvalues are pure imaginary, and unitary eigenvalues lie on the unit circle.
Note that unitary eigenvalues have the form $u=\text{exp}(a)$ where $a$ is an anti-Hermitian eigenvalue.
This implies that any unitary matrix $\bo{U}$ can be written as $\text{exp}(\bo{A})$, where $\bo{A}$ is anti-Hermitian.
\end{rmk}

\begin{rmk}
\label{rmk:spin-orbital-transformation-law}
According to \cref{dfn:normal-matrix} and \cref{rmk:spectral-theorem}, unitary transformations of the spin-orbitals can be parametrized as
\begin{align}
\label{eq:spin-orbital-transformation}
  \y_p'
=
  \sum_q
  \y_q
  (\text{exp}(\bo{X} - \bo{X}\dg))_{qp}
\end{align}
in terms of a square matrix $\bo{X}$.
The form of this parametrization leads to redundancies.
In particular, notice that $(\bo{X})_{pq}\equiv z\,\delta_{pp'}\delta_{qq'}$ generates the same transformation as $(\bo{X})_{pq}\equiv -z^*\,\delta_{pq'}\delta_{qp'}$.
These redundancies are eliminated by setting the upper or lower triangle of $\bo{X}$ to zero.
The creation operators for these orbitals are given by
$
  a_p^{\prime\,\dagger}
=
  \sum_q
  a_q\dg
  (\text{exp}(\bo{X} - \bo{X}\dg))_{qp}
$.
\end{rmk}

\begin{prop}
\label{prop:creation-operator-similarity-transform}
\thmstatement{
The identity\ \
$\ds{
  \text{exp}(G)\,a_p\dg\,\text{exp}(-G)
=
  \sum_q
  a_q\dg\,
  (\text{exp}(\bo{G}))_{qp}
}$\
holds for any
$
  G
=
  \sum_{pq}
  (\bo{G})_{pq}\,
  a_p\dg a_q
$.
}\vspace{3pt}
\thmproof{
  This follows from
  $
    [G,\cdot\,]^m(a_p\dg)
  =
    \sum_q
    a_q\dg
    (\bo{G}^m)_{qp}
  $,
  which we will prove by induction.
  For $m=0$ the statement is trivially true.
  If we assume it holds for $m$, then the following shows that it also holds for $m+1$,\footnote{
  The second equality here follows from expanding $G$ and using
$
  [a_r\dg a_s, a_q\dg]
=
  \no{
    a_r\dg
    \ctr{}{a}{_s}{}
    a_s a_q\dg
  }
=
  a_r\dg\,
  \delta_{sq}
$.
}
\begin{align*}
\ts{
  [G,\cdot\,]^{m+1}(a_p\dg)
=
  \sum_q
  [G, a_q\dg]\,
  (\bo{G}^m)_{qp}
=
  \sum_{qr}
  a_r\dg
  (\bo{G})_{rq}
  (\bo{G}^m)_{qp}
=
  \sum_r
  a_r\dg
  (\bo{G}^{m+1})_{rp}
}
\end{align*}
  which completes the induction.
  Substituting this result into the Hausdorff expansion of
$
  \text{exp}(G)\,a_p\dg\,\text{exp}(-G)
$
and  recognizing the Taylor expansion of $\text{exp}(\bo{G})$ completes the proof.
}
\end{prop}

\begin{rmk}
Given \cref{rmk:spin-orbital-transformation-law} and \cref{prop:creation-operator-similarity-transform}, the transformation of particle-hole operators can be expressed as
\begin{align}
\label{eq:operator-transformation}
\begin{array}{r@{\ }l}
  a_p^{\prime\,\dagger}
&=
  \text{exp}(X - X\dg)
  a_p\dg\,
  \text{exp}(X\dg - X)
\\[4pt]
  a_p^{\prime}
&=
  \text{exp}(X - X\dg)
  a_p\,
  \text{exp}(X\dg - X)
\end{array}
&&
  X
=
  \sum_{p>q}
  (\bo{X})_{pq}
  a_p\dg a_q
\end{align}
where the annihilation operator transformation is simply the adjoint of the one for creation operators.
\end{rmk}

\begin{rmk}
If $\Th'$ is obtained by replacing all of the orbitals in the basis expansion of $\Th\in\mc{F}$ with primed orbitals, then
\begin{align}
  \Th'
=
  \text{exp}(X - X\dg)
  \Th
\end{align}
which follows from substituting equation~\ref{eq:operator-transformation} into
$
  a_{p_1}^{\prime\,\dagger}
  \cd
  a_{p_n}^{\prime\,\dagger}
  \kt{\vac}
$
to prove that
$
  \kt{\F'_{(p_1\cd p_n)}}
=
  \text{exp}(X - X\dg)
  \kt{\F_{(p_1\cd p_n)}}
$\footnote{
  Note that
$
  \text{exp}(X\dg - X)
=
  \pr{\text{exp}(X - X\dg)}^{-1}
$
and
$
  \text{exp}(X\dg - X)
  \kt{\vac}
=
  \kt{\vac}
$.
}
for any basis state.
\end{rmk}