\lab{HIV Treatment Using Optimal Control}{HIV Treatment Using Optimal Control}
\label{lab:hiv}

\section*{Introduction}
    Viruses are the cause of many common illnesses in society today, such as ebola, influenza, the common cold, and Human Immunodeficiency Virus (HIV). Viruses are not considered to be living organisms as they cannot reproduce on their own. Instead they inject their genes in the form of DNA or RNA into a host's genome. They then use the cell's ribosomes and proteins to make the protein coat and replicate their genes. At the end they lyse the cell (tear it apart) and release many virus particles to infect other cells.\par
    The body has an adaptive immune system which learns to recognize viruses and bacteria and their hosts, and how to destroy them. A major component of this system are T cells. These cells perform many necessary functions such as recognizing invaders, destroying infected cells, and remembering previous infections long after recovery. Of particular interest is the helper T cell, also known as the CD4+T cell, due to a protein found on its surface which regulates the immune responses. HIV is unique in that it specifically targets this particular type of T cell. This means that the system responsible for fighting infections is specifically targeted.\par
    This loss of CD4+T cells is what causes Acquired Immune Deficiency Syndrome (AIDS). Note that AIDS itself is not an infection, which is a common misconception among the population. Due to the lack of T cells to recognize viruses and bacteria, the body becomes susceptible to other forms of infection. Whereas most people are easily able to shake off a common cold, someone suffering from the advanced stages of AIDS will be at serious risk of dying. Since AIDS comes from a loss of T cells, it may be several years before the host notices the effects of the infection. This enables the HIV virus to spread more easily since the host might not realize they are infected and continue in whatever behavior made them susceptible to the infection initially.\par
   Currently there is no cure or vaccine for HIV. However, there are treatments that reduce the virus and bolster the immune system by increasing the CD4+T cell count. Since these treatments can be expensive and often have negative side effects, it is important to optimize the amount of drugs used. Sometimes combinations of these drugs are used to provide a better effect. In this lab we will use optimal control to find the optimal dosage of a two-drug combination\footnote{\textit{SHORT COURSES ON THE MATHEMATICS OF BIOLOGICAL COMPLEXITY}, Web. 15 Apr. 2015 http://www.math.utk.edu/~lenhart/smb2003.v2.html.} \footnote{`Immunotherapy of HIV-1 Infection', Kirschner, D. and Webb, G. F., Journal of Biological Systems, 6(1), 71-83 (1998)}.

   % \begin{thebibliography}{9}
%
% \bibitem{paper}
% "SHORT COURSES ON THE MATHEMATICS OF BIOLOGICAL COMPLEXITY." \textit{SHORT COURSES ON THE MATHEMATICS OF BIOLOGICAL COMPLEXITY}, Web. 15 Apr. 2015 http://www.math.utk.edu/~lenhart/smb2003.v2.html.

% \bibitem{constants}
% .
% \end{thebibliography}

\begin{problem}
Explain what makes the HIV virus unique. What are the consequences of this? What is AIDS?
\label{problem:hiv:virusunderstanding}
\end{problem}

\begin{problem}
How is AIDS treated and what are the considerations for treatment?
\label{problem:hiv:treatment}
\end{problem}

\section*{Derivation of Control}
First we begin by writing the state system, the equations that describe the changes in T cells and viruses:
\begin{align*}
\frac{\it{d}T(t)}{\it{dt}} &= s_1 - \frac{s_2V(t)}{B_1+V(t)} - \mu T(t)-kV(t)T(t)+u_1(t)T(t), \\
\frac{\it{d}V(t)}{\it{dt}} &= \frac{gV(t)}{B_2+V(t)}(1-u_2(t)) - cV(t)T(t).
\end{align*}
To these equations we add initial conditions $T(0)=T_0$ and $V(0)=V_0$, where $T$ represents the concentration of $CD4^+T$ cells and $V$ the concentration of HIV particles.
The term $s_1-\frac{s_2V(t)}{B_1+V(t)}$ is the source/proliferation of unaffected T cells,
$\mu T(t)$ the natural loss of T cells, $kV(t)T(t)$ the loss of T cells by infection.
$\frac{gV(t)}{B_2+V(t)}$ represents the viral contribution to plasma, and $cV(t)T(t)$ the viral loss.

We now seek to maximize the functional
\[
J(u_1,u_2) = \int_0^{t_f}[T-(A_1u_1^2+A_2u_2^2)]\it{dt}.
\]
This functional considers i) the benefit of T cells, and ii) the systematic costs of drug treatments.
The constants $A_1$ and $A_2$ represent scalars to adjust the size of terms coming from $u_1^2$ and $u_2^2$ respectively.
We seek an optimal control $u_1^*,u_2^*$ satisfying
\[
J(u_1^*,u_2^*)=\max\{J(u_1,u_2)|(u_1,u_2)\in U\},
\]
where $U=\{(u_1,u_2)|u_i $ is measurable $,\,a_i\le u_i \le b_i$, $t\in[0,t_f]$ for $i=1,2\}$.

\section*{Optimality System}
The Hamiltonian is defined as:
\begin{align*}
	H = [T-(A_1u_1^2+A_2u_2^2)]&+\lambda_1\Big[ s_1 - \frac{s_2V}{B_1+V}-\mu T-kVT+u_1T \Big] \\
	&+ \lambda_2 \Big[\frac{g(1-u_2)V}{B_2+V}-cVT\Big].
\end{align*}
Note that the costate is represented with $\lambda$ instead of $p$. Now based on what we know from class we have:
\begin{align*}
\lambda_1^{'} &=-\frac{\partial H}{\partial T} =  -1+\lambda_1[\mu+kV^*-u_1^*]+\lambda_2cV^*,\\
\lambda_2^{'} &= -\frac{\partial H}{\partial V} = \lambda_1\Big[\frac{B_1s_2}{(B_1+V^*)^2}+kT^*\Big] -\lambda_2\Big[\frac{B_2g(1-u_2^*)}{B_2+V^*)^2}-cT^*\Big].
\end{align*}

The transversality conditions are $\lambda_1(t_f)=\lambda_2(t_f)=0$, with $T(0)=T_0$ and $V(0)=V_0$.
%Also
%$$u_1^* = min\Bigg\{max\Bigg\{a_1,\frac{1}{2A_1}(\lambda_1T^*(t))\Bigg\},b_1\Bigg\}$$
%$$u_2^* = min\Bigg\{max\Bigg\{a_2,\frac{-\lambda_2}{2A_2}\frac{V*(t)}{(B_2+V^*(t))}\Bigg\},b_2\Bigg\}$$
%The optimality equations are:
%$$\frac{\partial H}{\partial u_1} = -2A_1u_1^*(t)+\lambda_1T^*(t)=0$$
%$$\frac{\partial H}{\partial u_2} = -2A_2u_2^*(t)+\lambda_2\big[\frac{-gV^*(t)}{B_2+V^*(t)}\big
From these conditions we obtain
\begin{align*}
u_1^*(t) &= \frac{1}{2A_1}\left[\lambda_1T^*(t)\right],\\
u_2^*(t) &= \frac{1}{2A_2}\left[-\lambda_2\frac{gV^*(t)}{B_2+V^*(t)}\right].
\end{align*}
From the bounds on the controls we have
\begin{align*}
	\begin{split}
		u_1^*(t)&=\min\left\{\max\{a_1,\frac{1}{2A_1}(\lambda_1T^*(t))\},b_1\right\},\\
		u_2^*(t)&=\min\left\{\max\{a_2,-\frac{\lambda_2}{2A_2}\frac{gV^*(t)}{B_2+V^*(t)}\},b_2\right\}.
	\end{split}
\end{align*}
This gives us the optimal system
\begin{align*}
T'&=s_1-\frac{s_2V}{B_1+V}-\mu T-kVT+\min\{\max\{a_1,\frac{1}{2A_1}(\lambda_1T)\},b_1\}T,\\
V'&=\frac{g(1-\min\{\max\{a_2,\frac{-\lambda_2}{2A_2}\frac{gV}{B_2+V}\},b_2\})V}{B_2+V}-cVT,\\
\lambda_1'&=-1+\lambda_1\left[\mu+kV-\min\{\max\{a_1,\frac{1}{2A_1}(\lambda_1T)\},b_1\}\right]+\lambda_2cV,\\
\lambda_2'&=\lambda_1\left[\frac{B_1s_2}{(B_1+V)^2}+kT\right]-\lambda_2\left[\frac{B_2g(1-\min\{\max\{a_2,\frac{-\lambda_2}{2A_2}\frac{V}{B_2+V}\},b_2\})}{(B_2+V)^2}-cT\right],
\end{align*}
with end conditions $\lambda_1(t_f)=\lambda_2(t_f)=0$, and $T(0)=T_0,V(0)=V_0$.

\section*{Creating a Numerical Solver}

We iteratively solve for our control $u$. In each iteration we solve our state equations and our costate equations numerically, then use those to find our new control. Lastly, we check to see if  our control has converged. To solve each set of differential equations, we will use the RK4 solver from a previous lab with one minor adjustment. Our state equations depend on $u$, and our costate equations depend on our state equations. Therefore, we will pass another parameter into the function that RK4 takes in that will index the arrays our equations depend on.

\begin{lstlisting}
# Dependencies for this lab's code:
import numpy as np
from matplotlib import pyplot as plt

#Code from RK4 Lab with minor edits
def initialize_all(y0, t0, t, n):
    """ An initialization routine for the different ODE solving
    methods in the lab. This initializes Y, T, and h. """
    if isinstance(y0, np.ndarray):
        Y = np.empty((n, y0.size)).squeeze()
    else:
        Y = np.empty(n)
    Y[0] = y0
    T = np.linspace(t0, t, n)
    h = float(t - t0) / (n - 1)
    return Y, T, h

def RK4(f, y0, t0, t, n):
    """ Use the RK4 method to compute an approximate solution
    to the ODE y' = f(t, y) at n equispaced parameter values from t0 to t
    with initial conditions y(t0) = y0.

    y0 is assumed to be either a constant or a one-dimensional numpy array.
    t and t0 are assumed to be constants.
    f is assumed to accept three arguments.
    The first is a constant giving the value of t.
    The second is a one-dimensional numpy array of the same size as y.
    The third is an index to the other arrays.

    This function returns an array Y of shape (n,) if
    y is a constant or an array of size 1.
    It returns an array of shape (n, y.size) otherwise.
    In either case, Y[i] is the approximate value of y at
    the i'th value of np.linspace(t0, t, n).
    """
    Y,T,h = initialize_all(y0,t0,t,n)
    for i in xrange(n-1):
        K1 = f(T[i],Y[i],i)
        K2 = f(T[i]+h/2.,Y[i]+h/2.*K1,i)
        K3 = f(T[i]+h/2.,Y[i]+h/2.*K2,i)
        K4 = f(T[i+1],Y[i]+h*K3,i)
        Y[i+1] = Y[i] + h/6.*(K1+2*K2 +2*K3+K4)
    return Y
\end{lstlisting}

\begin{problem}
Create a function that defines the state equations and returns both equations in a single array. The function should be able to be passed into the RK4 solver. This function can depend on the global variables defined below.

\begin{lstlisting}
# define constants
s_1 = 2.
s_2 = 1.5
mu = 0.002
k = 0.000025
g = 30.
c = 0.007
B_1 = 14
B_2 = 1

# initialize global variables, state, costate, and u.
state = np.zeros((n,2))
state0 = np.array([T0, V0])

costate = np.zeros((n,2))
costate0 = np.zeros(2)

u=np.zeros((n,2))
u[:,0] += .02
u[:,1] += .9

# define state equations
def state_equations(t,y,i):
	'''
	Parameters
	---------------
	t : float
		the time
	y : ndarray (2,)
		the T cell concentration and the Virus concentration at time t
	i : int
		index for the global variable u.
	Returns
	--------------
	y_dot : ndarray (2,)
		the derivative of the T cell concentration and the virus concentration at time t
	'''
	pass
\end{lstlisting}
\label{problem:hiv:state}
\end{problem}

The state equations work great in the RK4 solver; however, the costate equations have end conditions rather than initial conditions. Thus we want our RK4 solver to iterate backwards from the end to the beginning. An easy way to accomplish this is to define a function $ \hat{\lambda}_i(t)=\lambda_i(t_f - t).$ Then $\hat{\lambda}_i$ has the initial conditions $\hat{\lambda}_i(0) = \lambda_i(t_f)$. We get the new equations

\begin{align*}
\dot{\hat{\lambda}}_1(t) &=\lambda_1(t_f-t)\left(-\mu - kV(t_f-t) + u_{1}\right) - c\lambda_2(t_f-t)V(t_f-t) + 1, \\
\dot{\hat{\lambda}}_2(t) &= -\lambda_1(t_f-t)\left(\frac{s_2B_1}{(B_1+V(t_f-t))^2}+kT(t_f-t)\right) \\
&+ \lambda_2(t_f-t)\left(\frac{gB_2(1-u_2(t_f-t))}{(B_2 + V(t_f-t))^2} - cT(t_f-t)\right).
\end{align*}

These we can solve with our RK4 solver and recover the original costate equations by simply indexing the array backwards.

\begin{problem}
Create a function that defines the costate equations and returns both equations in a single array. The function should be able to be passed into the RK4 solver. Use the global variables as defined in Problem \ref{problem:hiv:state}.

\begin{lstlisting}
def lambda_hat(t,y,i):
	'''
	Parameters
	---------------
	t : float
		the time
	y : ndarray (2,)
		the lambda_hat values at time t
	i : int
		index for global variables, u and state.
	Returns
	--------------
	y_dot : ndarray (2,)
		the derivative of the lambda_hats at time t.
	'''
	pass
\end{lstlisting}

\label{problem:hiv:costateequations}
\end{problem}

\begin{figure}
\centering
\includegraphics[width=5in]{solutions.png}
\caption{The solution to Problem \ref{problem:hiv:numericalsolver}.}
\label{fig:hiv:solutions}
\end{figure}

Finally, we can put these together to create our solver.
\begin{problem}
Create a numerical solver for the HIV two drug model using the code below.

\begin{lstlisting}
epsilon = 0.001
test = epsilon + 1

while(test > epsilon):
	oldu = u.copy();

	#solve the state equations with forward iteration
	#state = RK4(...)

	#solve the costate equations with backwards iteration
	#costate = RK4(...)[::-1]

	#solve for u1 and u2

	#update control
	u[:,0] = 0.5*(u1 + oldu[:,0])
	u[:,1] = 0.5*(u2 + oldu[:,1])

	#test for convergence
	test = abs(oldu - u).sum()
\end{lstlisting}

Run with the following parameter values:
\begin{lstlisting}
a_1, a_2 = 0, 0
b_1, b_2 = 0.02, 0.9
s_1, s_2 = 2., 1.5
\mu = 0.002
k = 0.000025
g = 30.
c = 0.007
B_1, B_2 = 14, 1
A_1, A_2 = 250000, 75
T_0, V_0 = 400, 3
t_f = 50
n = 1000
\end{lstlisting}
These constants come from both references cited at the end of this lab. Your solutions should match Figure \ref{fig:hiv:solutions}.
\label{problem:hiv:solver}

\label{problem:hiv:numericalsolver}
\end{problem}

In modern medicine, patients generally take combinations of five or more medications with different functions. These include Nucleotide Reverse Transcriptase Inhibitors, which prevent HIV inserting its genes into host DNA, Non-Nucleoside Reverse Transcriptase Inhibitors, which do the same job as NRTIs in a different fashion, Protease Inhibitors, which cut up replicated HIV strands, Fusion Inhibitors, which block the virus from entering the cells to begin with, and Integrase Inhibitors, which prevents the virus' replicated DNA from being inserted into a cell's DNA. These drugs often can interact with each other and have different side effects on the body. Also, doctors rotate medications as the body and virus develop immunity.

% \begin{thebibliography}{9}
%
% \bibitem{paper}
% "SHORT COURSES ON THE MATHEMATICS OF BIOLOGICAL COMPLEXITY." \textit{SHORT COURSES ON THE MATHEMATICS OF BIOLOGICAL COMPLEXITY}. Web. 15 Apr. 2015. http://www.math.utk.edu/~lenhart/smb2003.v2.html.
%
% \bibitem{constants}
% Kirschner, D. and Webb, G. F., `Immunotherapy of HIV-1 Infection', Journal of Biological Systems, 6(1), 71-83 (1998).
% \end{thebibliography}


\section{Convergence rates in Sobolev norms}
\subsection{Compactly supported activation function}
Given a bounded domain $\Omega\subset\mathbb R^d$, we consider the function
$$
f: \Omega\mapsto \mathbb R
$$
Let
$$
f^e: \mathbb R^d\mapsto \mathbb R
$$
be any extension of $f$ so that
$$
f^e|_\Omega=f(x), \quad x\in \Omega.
$$
Most time, we will drop the superscript $``e"$ to still use $f$ to
denote an extension of $f$.

Consider the Fourier transform:
\begin{equation}
\label{Fourier}
\hat f(\omega)=\frac{1}{(2\pi)^d}\int_{\mathbb{R}^d}e^{-i\omega\cdot x}f(x)dx
\quad \forall \omega \in \mathbb R^d.
\end{equation}
Let $x_\Omega\in\Omega$ is such that
\begin{equation}
\label{xOmega}
x_\Omega\in \arg\min_{y\in\bar\Omega}(\max_{x\in\bar\Omega}|x-y|)
\end{equation}
We may call
\begin{equation}
\label{rOmega}
r_\Omega=\max_{x\in\bar\Omega}|x-x_\Omega|
\end{equation}
the radius of $\Omega$.

Using the Fourier inversion formula, we can have a Fourier
representation of $f(x)$ as follows
\begin{equation}
\label{eqn1}
f(x)=\int_{\mathbb{R}^d}e^{i\omega\cdot x}\hat{f}(\omega)d\omega
\quad \forall x \in \Omega,
\end{equation}
Let us write
\begin{equation}
\label{theta-omega}
\hat{f}(\omega)=e^{i\beta_1(\omega)}|\hat{f}(\omega)|.
\end{equation}

%Define $\mu_f$ as a complex measure on $\mathbb{R}^d$ such that
%\begin{equation}
%d\mu_f = e^{i\alpha(\omega)}|\hat{f}(\omega)|d\omega
%\end{equation}
%and
%\begin{equation}
%\label{norm}
%\|\mu_f\| = \int_{\mathbb{R}^d} (1+|\omega|)^2d|\mu_f|(\omega)
%\end{equation}
%Also we have:
%\begin{equation}
%f(x)=\int_{\mathbb{R}^d}e^{i\omega\cdot x}d\mu_f
%\end{equation}

Let $\sigma$ be the activation function with compact support and bounded derivatives up to order $m$, that is
\begin{equation}
\|\sigma\|_{m,\infty} := \max_{0\le\alpha\le m}\sup_{x\in\mathbb{R}}|\sigma^{(\alpha)}(x)|\le \infty
\end{equation}
Set
\begin{equation}
\label{eq:3}
\tilde x=x-x_\Omega.
\end{equation}
Choose $a\ne0$ such that $\hat{\sigma}(a)\ne 0$, by Fourier transform, we have:
\begin{equation*}
\hat{\sigma}(a)=\frac{1}{2\pi}\int_{\mathbb{R}}\sigma(y)e^{-ia
	y}dy=\frac{1}{2\pi}\int_{\mathbb{R}}\sigma(\omega\cdot \tilde
x+b)e^{-ia(\omega\cdot \tilde x+\theta)}db
\end{equation*}
and we can also write
\begin{equation*}
\hat{\sigma}(a)=e^{i\beta_2(a)}|\hat{\sigma}(a)|.
\end{equation*}

Thus
\begin{equation}
|\hat{\sigma}(a)|=\frac{1}{2\pi}\int_{\mathbb{R}}\sigma((\omega\cdot \tilde{x}+b ) )e^{-ia(\omega\cdot \tilde{x}+b )-i\beta_2(a)}db.
\end{equation}
%be the hat function defined as:
%\begin{equation*}
%\sigma(x)=\left\{
%\begin{aligned}
%&x,\qquad\ \  0\le x\le 1 \\
%&2-x,\quad 1\le x\le 2\\
%&0,\qquad\quad \mathrm{otherwise}
%\end{aligned}
%\right.
%\end{equation*}

Then we can write
\begin{equation}
\begin{aligned}
f(x)=& \int_{\mathbb{R}^d}e^{i\omega\cdot x}e^{i\beta_1(\omega)}|\hat{f}(\omega)|d\omega\\
=&\int_{\mathbb{R}^d}|a|^d e^{ia(\omega\cdot x)}e^{i\beta_1(a\omega)}|\hat{f}(a\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\frac{|a|^d }{ |\hat{\sigma}(a)|} |\hat{\sigma}(a)|e^{ia(\omega\cdot x)+i\beta_1(a\omega)}|\hat{f}(a\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\int_{\mathbb{R}}\frac{|a|^d}{2\pi|\hat{\sigma}(a)|}\sigma(\omega\cdot \tilde{x}+b  )e^{-i\beta_2(a)}
e^{ia\omega\cdot x_\Omega-iab}db e^{i\beta_1(a\omega)}|\hat{f}(a\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\int_{\mathbb{R}}\sigma(\omega\cdot x+b )\frac{|a|^d}{2\pi|\hat{\sigma}(a)|}e^{i\beta_3(a,\omega,b)}|\hat{f}(a\omega)|db d\omega
\end{aligned}
\end{equation}
where $\beta_3(a,\omega,b)=a\omega\cdot x_\Omega-ab-\beta_2(a)+\beta_1(a\omega)$.

Let
$$
\theta=(\omega, b).
$$
Since $f$ is real, we have
$$
f(x)={\rm Re}\; f(x)
%\int_{\mathbb{R}^d}\int_{\mathbb{R}}\frac{\cos(\beta_1(a\omega)-ab-\beta_2(a))}{2\pi|\hat{\sigma}(a)|}\sigma(\omega\cdot
%\tilde x+b )|a|^d|\hat{f}(a\omega)|db d\omega
=\int_{\mathbb{R}^{d+1}}\kappa(\theta)\sigma(\omega\cdot\tilde x+b )|a|^d|\hat{f}(a\omega)|d\theta.
$$
where
\begin{equation}
\kappa(\theta)\equiv
\kappa(\omega,b)=\frac{\cos\beta_3(a,\omega,b)}{2\pi|\hat{\sigma}(a)|},
\end{equation}

Notice that $\sigma$ is compactly supported, assume that $\mathrm{supp}(\sigma)\subset[-M_1,M_1]$. Then we have:
\begin{equation}
\begin{aligned}
f(x) =\int_{\mathbb{R}^{d+1}}\kappa(\theta)\mathbf{1}_{D_M}\sigma(\omega\cdot \tilde{x}+b)|a|^d|\hat{f}(a\omega)|d\theta
\end{aligned}
\end{equation}
where
\begin{equation}
M=\max(M_1,r_\Omega)
\end{equation}
\begin{equation}
D_M=\{\theta=(\omega,b):|b|\le(1+|\omega|)M  \}
\end{equation}

Define
\begin{equation}
\gamma(f)=\int_{\mathbb{R}^{d+1}}\mathbf{1}_{D_M}(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|d\theta
\end{equation}

Now that
\begin{equation}
\begin{aligned}
f(x) 	&=\int_{\mathbb{R}^{d+1}}\kappa(\theta)\mathbf{1}_{D_M}\sigma(\omega\cdot \tilde{x}+b)|a|^d|\hat{f}(a\omega)|d\theta \\
&= \int_{\mathbb{R}^{d+1}}\frac{\kappa(\theta)\gamma(f)}{(1+|\omega|)^m}\sigma(\omega\cdot \tilde{x}+b)\frac{(1+|\omega|)^m\mathbf{1}_{D_M}|a|^d|\hat{f}(a\omega)|}{\gamma(f)}d\theta\\
&:=\int_{\mathbb{R}^{d+1}}\frac{\kappa(\theta)\gamma(f)}{(1+|\omega|)^m}\sigma(\omega\cdot \tilde{x}+b) d\lambda
\end{aligned}
\end{equation}
where

$$
d\lambda= \frac{(1+|\omega|)^m\mathbf{1}_{D_M}|a|^d|\hat{f}(a\omega)|}{\gamma(f)}d\theta ,\quad \int_{\mathbb{R}^{d+1}}d\lambda=1,
$$

and we can write
\begin{equation}
\tilde f(x)\equiv\frac{f(x)}{\gamma(f)}
=\mathbb{E}(g(\theta; x))
\end{equation}
where
\begin{equation}
\label{gx}
g(\theta; x))
=\frac{\kappa(\theta)}{(1+|\omega|)^m}\sigma(\omega\cdot\tilde x+b))
\end{equation}
And
\begin{equation}
D^\alpha \tilde f(x)=\mathbb{E}(D^\alpha g).
\end{equation}
%\mathbb{E}(\kappa(\omega,b)\frac{D^\alpha\sigma(\omega\cdot\tilde x+b)}{(1+|\omega|)^m})
Let $\theta_i=(\omega_i,b_i)_{i=1}^n$ be independently drawn from the same distribution $\lambda$ and let
$$
\bar g(\theta_1,\ldots,\theta_n)
=\frac{1}{n} \sum_{i=1}^ng(\theta_i, x).
$$
Then
$$
\bar{\mathbb E} \left(\sum_{|\alpha|\le m}(D^\alpha \tilde f-D^\alpha \bar
g)^2\right)
=\sum_{|\alpha|\le m}\bar{\mathbb E} (D^\alpha \tilde f-D^\alpha \bar g)^2
=\sum_{|\alpha|\le m}\bar{\mathbb E} (\mathbb E(D^\alpha g)-D^\alpha \bar g)^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
Taking the $L^2$ norm (with a probability measure) on both side of the above inequality
and using Fubini's theorem,
\begin{equation}
\mathbb{\bar{\mathbb{E}}}(\|\tilde f(\cdot)-g(\theta,\cdot)\|^2_{H^m(\Omega)})
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
\end{equation}
Thus there exists $\theta_i=(\omega_i^*,b_i^*)_{i=1}^n$ and $\beta_i$ such that
$$
\|\tilde f-\frac{1}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)\|_m^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
$$
\|f-f_n\|_m^2
\le{\gamma(f)^2\over n}
\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
where
$$
f_n(x)=\frac{\gamma(f)}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)=\frac{1}{n} \sum_{i=1}^n a_i\sigma(\omega_i^*\cdot\tilde{x}+b_i^*).
$$
where
$$
a_i= \frac{\gamma(f)\kappa(\theta_i^*)}{(1+|\omega_i^*|)^m} \le \frac{\gamma(f)}{2\pi|\hat{\sigma}(a)|}
$$

Noting that $\frac{1}{1+|\omega|}\le1$, we have
$$
\|D^\alpha g\|_{\infty}=
\max_{0\le|\alpha|\le m}\max_{\in\mathbb{R}^d,\theta\in\mathbb{R}^{d+1}}|\kappa(\theta)|\frac{|D^\alpha\sigma(\omega\cdot\tilde x+b)|}{(1+|\omega|)^m}\le \frac{\|\sigma\|_{m,\infty}}{2\pi|\hat{\sigma}(a)|}
$$
Then we have:

\begin{equation}
\begin{aligned}
\gamma(f)&=|a|^d\int_{\mathbb{R}^d}(1+|\omega|)^m \int_{-(1+|\omega|)M}^{(1+|\omega|)M}db |\hat{f}(a\omega)|d\omega \\
&= |a|^d\int_{\mathbb{R}^d}2M(1+|\omega|)^{m+1} |\hat{f}(a\omega)|d\omega\\
&\le 2M \int_{\mathbb{R}^d}(1+|\omega/a|)^{m+1} |\hat{f}(\omega)|d(\omega)\\
\end{aligned}
\end{equation}

Denote
$$
\|f\|_{m+1} = \int_{\mathbb{R}^d}(1+|\omega/a|)^{m+1} |\hat{f}(\omega)|d(\omega)
$$
Then there exists $(\omega_i^*,b_i^*)_{i=1}^n$ and $\beta_i$ such that
$$
\|f-f_n\|_m
\le\frac{\|\sigma\|_{m,\infty}}{\sqrt{n}}\|f\|_{m+1}
$$
where $C=\frac{M\sqrt{C_{m+d}^m}}{\pi|\hat{\sigma}(a)|} $.

\subsection{Periodic activation function}

Now $\sigma$ is periodic with period $T$. Then we can write the Fourier series for $\sigma$:
$$
\sigma(x) =  \sum_{i=-\infty}^{\infty}c_n e^{i\frac{2\pi nx}{T}}
$$
choose $n_1$ such that $c_{n_1}\ne 0$, we also have:
$$
c_{n_1} = \frac{1}{T}\int_{0}^{T}\sigma(x) e^{-i\frac{2\pi n_1x}{T}}dx= |c_{n_1}|e^{i\beta_2(c_{n_1})}
$$
Then for $f$:
\begin{equation}
\begin{aligned}
f(x)=& \int_{\mathbb{R}^d}e^{i\omega\cdot x}e^{i\beta_1(\omega)}|\hat{f}(\omega)|d\omega\\
=&\int_{\mathbb{R}^d}\frac{1}{c_{n_1}}\frac{1}{T}\int_{x_0}^{x_0+T}\sigma(t) e^{-i\frac{2\pi n_1t}{T}}dte^{i(\omega\cdot x)+i\beta_1(\omega)}|\hat{f}(\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\frac{1}{|c_{n_1}|e^{i\beta_2(c_{n_1})}}\frac{1}{2\pi n_1}\int_{0}^{2\pi n_1}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b)) e^{-i(\omega\cdot x+b)}dbe^{i(\omega\cdot x)+i\beta_1(\omega)}|\hat{f}(\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\frac{1}{2|c_{n_1}|\pi n_1}\int_{0}^{2\pi n_1}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b)) e^{i(\beta_1(\omega)-b-\beta_2(c_{n_1})}db|\hat{f}(\omega)|d\omega \\
=&\int_{\mathbb{R}^d}\int_{0}^{2\pi n_1}\kappa(b,\omega)\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b)) db|\hat{f}(\omega)|d\omega \\
\end{aligned}
\end{equation}
here we do a substitution $t=\frac{T}{2\pi n_1}(\omega\cdot x+b)$ and
\begin{equation}
\kappa = \frac{\cos(\beta_1(\omega)-b-\beta_2(c_{n_1}))}{2|c_{n_1}|\pi n_1}
\end{equation}
which is then the same as what we did for compactly supported activation functions.
\begin{equation}
\begin{aligned}
f(x)=&\int_{\mathbb{R}^d}\int_{0}^{2\pi n_1}\kappa(b,\omega)\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b)) db|\hat{f}(\omega)|d\omega \\
=&\int_{\mathbb{R}^d\times[0,2\pi n_1]}\frac{\kappa(\theta)}{(1+|\omega|)^m}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b))(1+|\omega|)^m|\hat{f}(\omega)|d\theta \\
=&\int_{\mathbb{R}^d\times[0,2\pi n_1]}\frac{\kappa(\theta)\gamma(f)}{(1+|\omega|)^m}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b))\frac{(1+|\omega|)^m|\hat{f}(\omega)|}{\gamma(f)}d\theta \\
:=&\int_{\mathbb{R}^{d+1}}\frac{\kappa(\theta)\gamma(f)}{(1+|\omega|)^m}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b))d\lambda
\end{aligned}
\end{equation}
where
$$
\gamma(f)=\int_{\mathbb{R}^{d+1}}\mathbf{1}_{0\le b\le 2\pi n_1}(1+|\omega|)^m|\hat{f}(\omega)|d\theta =2\pi n_1\int_{\mathbb{R}^d}(1+|\omega|)^m|\hat{f}(\omega)|d\omega
$$
and
$$
d\lambda= \frac{(1+|\omega|)^m|\hat{f}(\omega)|\mathbf{1}_{0\le b\le 2\pi n_1}}{\gamma(f)}d\theta,\quad \int_{\mathbb{R}^{d+1}}d\lambda=1,
$$

We can write
\begin{equation}
\tilde f(x)\equiv\frac{f(x)}{\gamma(f)}
=\mathbb{E}(g(\theta; x))
\end{equation}
where
\begin{equation}
\label{gx}
g(\theta; x))
=\frac{\kappa(\theta)}{(1+|\omega|)^m}\sigma(\frac{T}{2\pi n_1}(\omega\cdot x+b))
\end{equation}
And
\begin{equation}
D^\alpha \tilde f(x)=\mathbb{E}(D^\alpha g).
\end{equation}
%\mathbb{E}(\kappa(\omega,b)\frac{D^\alpha\sigma(\omega\cdot\tilde x+b)}{(1+|\omega|)^m})
Let $\theta_i=(\omega_i,b_i)_{i=1}^n$ be independently drawn from the same distribution $\lambda$ and let
$$
\bar g(\theta_1,\ldots,\theta_n)
=\frac{1}{n} \sum_{i=1}^ng(\theta_i, x).
$$
Then
$$
\bar{\mathbb E} \left(\sum_{|\alpha|\le m}(D^\alpha \tilde f-D^\alpha \bar
g)^2\right)
=\sum_{|\alpha|\le m}\bar{\mathbb E} (D^\alpha \tilde f-D^\alpha \bar g)^2
=\sum_{|\alpha|\le m}\bar{\mathbb E} (\mathbb E(D^\alpha g)-D^\alpha \bar g)^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
Taking the $L^2$ norm (with a probability measure) on both side of the above inequality
and using Fubini's theorem,
\begin{equation}
\mathbb{\bar{\mathbb{E}}}(\|\tilde f(\cdot)-g(\theta,\cdot)\|^2_{H^m(\Omega)})
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
\end{equation}
Thus there exists $\theta_i=(\omega_i^*,b_i^*)_{i=1}^n$ and $\beta_i$ such that
$$
\|\tilde f-\frac{1}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)\|_m^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
$$
\|f-f_n\|_m^2
\le{\gamma(f)^2\over n}
\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
where
$$
f_n(x)=\frac{\gamma(f)}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)=\frac{1}{n} \sum_{i=1}^n a_i\sigma(\frac{T}{2\pi n_1}(\omega_i^*\cdot x+b_i^*)).
$$
where
$$
a_i= \frac{\gamma(f)\kappa(\theta_i^*)}{(1+|\omega_i^*|)^m} \le \frac{\gamma(f)}{2|c_{n_1}|\pi n_1}
$$

Noting that $\frac{1}{1+|\omega|}\le1$, we have
$$
\|D^\alpha g\|_{\infty}=
\max_{0\le|\alpha|\le m}\max_{\in\mathbb{R}^d,\theta\in\mathbb{R}^{d+1}}|\kappa(\theta)|\frac{|D^\alpha\sigma( \frac{T}{2\pi n_1}(\omega\cdot x+b) )|}{(1+|\omega|)^m}\le \frac{k\|\sigma\|_{m,\infty}}{2|c_{n_1}|\pi n_1}
$$
where
$$
k = \max(\frac{T}{2\pi n_1},(\frac{T}{2\pi n_1})^\alpha)
$$

Denote
$$
\||f\||_{m} = \int_{\mathbb{R}^d}(1+|\omega|)^{m} |\hat{f}(\omega)|d(\omega)
$$
Then there exists $(\omega_i^*,b_i^*)_{i=1}^n$ and $\beta_i$ such that
$$
\|f-f_n\|_m
\le\frac{\|\sigma\|_{m,\infty}}{\sqrt{n}}|\|f\||_{m}
$$
where $C=\frac{k\sqrt{C_{m+d}^m}}{2|c_{n_1}|\pi n_1} $.

\subsubsection{Exponential Decay}
If there exists $\eta>0$ such that for all $0\le k \le m$
$$
|e^{\eta|t|}D^{k}\sigma(t)|_{L^1(t)}\le \zeta<\infty,
$$
which means that $|e^{\eta|t|}D^{k}\sigma(t)|$ is finite for all $t$ and all $0\le k \le m$, with abuse of notation here, we still use $\zeta$ to denote the bound.
$$
e^{\eta|t|}|D^{k}\sigma(t)|\le \zeta,
$$

We have
\begin{equation}
	\begin{aligned}
		&f(x)=\int_{\mathbb{R}^{d}}\int_{\mathbb{R}}\kappa(\omega,b)\sigma(\omega\cdot x+b)|a|^d|\hat{f}(a\omega)|dbd\omega.\\
		=&\int_{\mathbb{R}^{d}}\int_{\mathbb{R}}\frac{\kappa(\omega,b)}{(1+|\omega|)^m}\sigma(\omega\cdot x+b)(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|dbd\omega.\\
		=&\int_{\mathbb{R}^{d}}\int_{\mathbb{R}}\frac{\kappa(\omega,b)e^{\eta\max\{0,|b|-M|\omega|\}}}{(1+|\omega|)^m}\sigma(\omega\cdot x+b)\frac{(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|}{e^{\max\{0,\eta(|b|-M|\omega|)\}}}dbd\omega.\\
		=&\int_{\mathbb{R}^{d+1}}\frac{\kappa(\theta)e^{\max\{0,\eta(|b|-M|\omega|)\}}\gamma(f)}{(1+|\omega|)^m}\sigma(\omega\cdot x+b) d\lambda
	\end{aligned}
\end{equation}

where
$$
\gamma(f)=\int_{\mathbb{R}^{d+1}}\frac{(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|}{e^{\max\{0,\eta(|b|-M|\omega|)\}}}d\theta
$$
and
$$
d\lambda= \frac{(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|}{e^{\max\{0,\eta(|b|-M|\omega|)\}}\gamma(f)}d\theta,\quad \int_{\mathbb{R}^{d+1}}d\lambda=1,
$$

We can write
\begin{equation}
	\tilde f(x)\equiv\frac{f(x)}{\gamma(f)}
	=\mathbb{E}(g(\theta; x))
\end{equation}
where
\begin{equation}
	\label{gx}
	g(\theta; x))
	=\frac{\kappa(\theta)e^{\max\{0,\eta(|b|-M|\omega|)\}}}{(1+|\omega|)^m}\sigma(\omega\cdot x+b)
\end{equation}
And
\begin{equation}
	D^\alpha \tilde f(x)=\mathbb{E}(D^\alpha g).
\end{equation}
%\mathbb{E}(\kappa(\omega,b)\frac{D^\alpha\sigma(\omega\cdot\tilde x+b)}{(1+|\omega|)^m})

Also we should have:
\begin{equation*}
	\begin{aligned}
		\|D^\alpha g\|_{\infty}&\le \frac{1}{2\pi|\hat{\sigma}(a)|}e^{\max\{0,\eta(|b|-M|\omega|)\}}|D^\alpha\sigma(\omega\cdot x+b)|\\
		&\le\frac{1}{2\pi|\hat{\sigma}(a)|}e^{\eta|b+\omega\cdot x|}|D^\alpha\sigma(\omega\cdot x+b)|\\
		&\le\frac{\zeta}{2\pi|\hat{\sigma}(a)|}
	\end{aligned}
\end{equation*}

Let $\theta_i=(\omega_i,b_i)_{i=1}^n$ be independently drawn from the same distribution $\lambda$ and let
$$
\bar g(\theta_1,\ldots,\theta_n)
=\frac{1}{n} \sum_{i=1}^ng(\theta_i, x).
$$
Then
$$
\bar{\mathbb E} \left(\sum_{|\alpha|\le m}(D^\alpha \tilde f-D^\alpha \bar
g)^2\right)
=\sum_{|\alpha|\le m}\bar{\mathbb E} (D^\alpha \tilde f-D^\alpha \bar g)^2
=\sum_{|\alpha|\le m}\bar{\mathbb E} (\mathbb E(D^\alpha g)-D^\alpha \bar g)^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
Taking the $L^2$ norm (with a probability measure) on both side of the above inequality
and using Fubini's theorem,
\begin{equation}
	\mathbb{\bar{\mathbb{E}}}(\|\tilde f(\cdot)-g(\theta,\cdot)\|^2_{H^m(\Omega)})
	\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
\end{equation}
Thus there exists $\theta_i=(\omega_i^*,b_i^*)_{i=1}^n$ and $\beta_i$ such that
$$
\|\tilde f-\frac{1}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)\|_m^2
\le{1\over n}\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
$$
\|f-f_n\|_m^2
\le{\gamma(f)^2\over n}
\sum_{|\alpha|\le m}\|D^\alpha g\|_\infty^2.
$$
where
$$
f_n(x)=\frac{\gamma(f)}{n}\sum_{i=1}^{n} g(\theta_i; \cdot)=\frac{1}{n} \sum_{i=1}^n a_i\sigma(\omega_i^*\cdot\tilde{x}+b_i^*).
$$
where
$$
a_i= \frac{\gamma(f)\kappa(\theta_i^*)e^{\max\{0,\eta(|b_i^*|-M|\omega_i^*|)\}}}{(1+|\omega_i^*|)^m}
$$

Now with
$$
\||f\||_{m+1} = \int_{\mathbb{R}^d}(1+|\omega/a|)^{m+1} |\hat{f}(\omega)|d(\omega)
$$
we need to give an estimation on $\gamma(f)$:
\begin{equation}
	\begin{aligned}
		&\gamma(f)=\int_{\mathbb{R}^{d+1}}\frac{(1+|\omega|)^m|a|^d|\hat{f}(a\omega)|}{e^{\max\{0,\eta(|b|-M|\omega|)\}}}dbd\omega\\
		=&\int_{\mathbb{R}^d}|a|^d(1+|\omega|)^m \int_{-|\omega|M}^{|\omega|M}db |\hat{f}(a\omega)|d\omega + \int_{\mathbb{R}^d}|a|^d(1+|\omega|)^m \int_{|b|\ge M|\omega|}\frac{1}{e^{\eta(|b|-M|\omega|)}}db |\hat{f}(a\omega)|d\omega  \\
		\le&\||f\||_{m+1}+C\||f\||_{m}\\
		\le&C\||f\||_{m+1}
	\end{aligned}
\end{equation}

%\subsection{Noncompactly supported activation function}
%
%Now $\sigma$ is not compactly supported, however, we can still write $f$ as:
%$$
%f(x)=\int_{\mathbb{R}^{d+1}}\kappa(\theta)\sigma(\omega\cdot\tilde x+b )|a|^d|\hat{f}(a\omega)|d\theta.
%$$
%
%denote
%$$
%f_M(x)=\int_{\mathbb{R}^{d+1}}\kappa(\theta)\mathbf{1}_{D_M}\sigma(\omega\cdot \tilde{x}+b)|a|^d|\hat{f}(a\omega)|d\theta
%$$
%where for $M >0$,
%\begin{equation}
%D_M=\{\theta=(\omega,b):|b|\le(1+|\omega|)M  \}
%\end{equation}
%by the conclusion in last subsection, we have :
%$$
%\|f_M-f_n\|_m
%\le C\frac{\|\sigma\|_{m,\infty}}{\sqrt{n}}\|f\|_{m+1}
%$$
%where $C=\frac{M\sqrt{C_{m+d}^m}}{\pi|\hat{\sigma}(a)|} $.
%
%Now it suffices to estimate $\|f-f_M\|_m$:
%\begin{equation}
%f_r:=f-f_M=\int_{\mathbb{R}^{d}}\int_{|b|>(1+|\omega|)M}\kappa(\omega,b)\sigma(\omega\cdot \tilde{x}+b)|a|^d|\hat{f}(a\omega)|d\omega db
%\end{equation}
%
%Since we are in a bounded domain $\Omega$, we can choose $M>0$ such that $x\in\Omega\subset B_M$ where $B_M$ is the ball of radius $M$ in $\mathbb{R}^d$.
%If $x\in\Omega$ and $|b|>(1+|\omega|)M$, then:
%$$
%|\omega\cdot x+b|\ge |b|-|\omega\cdot x| \ge M
%$$
%
%Hence for all $x\in\Omega$ and $0\le|\alpha|\le m$, since $(1+|\omega|)^k$ is non decreasing w.r.t k, we have:
%\begin{equation}
%\begin{aligned}
%|D^{\alpha}f_r|&\le\frac{1}{2\pi|\hat{\sigma}(a)|}\|D^{|\alpha|}\sigma(t)\|_{L^1(|t|\ge M)}\int_{\mathbb{R}^d}(1+|\omega/a|)^{m} |\hat{f}(\omega)|d(\omega)\\
%&= \frac{1}{2\pi|\hat{\sigma}(a)|}\|D^{|\alpha|}\sigma(t)\|_{L^1(|t|\ge M)}\|f\|_{m}\\
%&\le\frac{1}{2\pi|\hat{\sigma}(a)|}\|D^{|\alpha|}\sigma(t)\|_{L^1(|t|\ge M)}\|f\|_{m+1}
%\end{aligned}
%\end{equation}
%Thus take the $L^2$ norm w.r.t a probability measure on $\Omega$, we can have:
%\begin{equation}
%\int_{\mathbb{R}^{d}}|D^{\alpha}f_r|^2\le \frac{1}{(2\pi|\hat{\sigma}(a)|)^2}\|D^{|\alpha|}\sigma(t)\|^2_{L^1(|t|\ge M)}\|f\|_{m+1}^2
%\end{equation}
%
%Summing over $0\le|\alpha|\le m$, and let
%$$
%R_{M}(\sigma)=\max_{0\le k\le m}\|D^{k}\sigma(t)\|^2_{L^1(|t|\ge M)}
%$$
%we have:
%\begin{equation}
%\|f_r\|_{H^m}^2\le \frac{C_{m+d}^m}{(2\pi|\hat{\sigma}(a)|)^2}R_{M}^2(\sigma)\|f\|_{m+1}^2
%\end{equation}
%
%Thus in the end we have:
%\begin{equation}
%\begin{aligned}
%\|f-f_n\|_{H^m}&\le\|f_r\|_{H^m}^2+\|f_M-f_n\|_m\\
%&\le \frac{\sqrt{C_{m+d}^m}}{2\pi|\hat{\sigma}(a)|}R_{M}(\sigma)\|f\|_{m+1}+\frac{M\sqrt{C_{m+d}^m}}{\pi|\hat{\sigma}(a)|} \frac{\|\sigma\|_{m,\infty}}{\sqrt{n}}\|f\|_{m+1} \\
%&\le  \frac{\sqrt{C_{m+d}^m}}{\pi|\hat{\sigma}(a)|}\|f\|_{m+1}(\frac{R_{M}(\sigma)}{2}+\frac{M\|\sigma\|_{m,\infty}}{\sqrt{n}})
%\end{aligned}
%\end{equation}


\documentclass[10pt,letterpaper]{article}

\usepackage{amsmath, amsfonts}
\usepackage{amsthm}
\usepackage[usenames,dvipsnames]{color}
\usepackage{paralist}   % For better control of itemizations
\usepackage[margin=1in]{geometry}
\usepackage{parallel}

\newtheorem{thm}{Theorem}

% New commands
\newcommand{\QR}{\ensuremath{QR} }

\title{\QR Algorithm}
\author{Jeremy Lloyd Conlin}

\begin{document}
\maketitle
\begin{abstract}
    This document is used to help formulate some of the important proofs resulting from the QR algorithm.  Many of them are exercises from David S. Watkins' \emph{Fundamentals of Matrix Computations}.
\end{abstract}

\section{General Infomation}
Arnoldi factorization
\begin{equation}
    A = \QR
    \label{eq:QRFactorization}
\end{equation}
Recombining:
\begin{equation}
    \hat{A} = RQ.
\end{equation}

\subsection{QR Iteration}
\begin{subequations}\begin{align}
    A_{m-1} &= Q_mR_m \\
    A_{m} &= R_mQ_m,
\end{align}
    \label{eq:QRIteration}
\end{subequations}
Alternatively
\begin{subequations}\begin{align}
    A_{m} &= Q_m^*A_{m-1}Q_m \\
    A_{m} &= R_mA_{m-1}R_m^{-1}.
\end{align}
    \label{eq:QRIterationAlternate}
\end{subequations}
Also
\begin{equation}
    Q_mA_m = A_{m-1}Q_m
\end{equation}

\subsection{Shifted QR Iteration}
\begin{subequations}\begin{gather}
    A_{m-1} - \nu_{m-1} I = Q_mR_m \label{eq:ShiftedQRFactorization} \\
    A_m = R_mQ_m = \nu_{m-1}I
\end{gather}
    \label{eq:ShiftedQRIteration}
\end{subequations}
Alternatively
\begin{subequations}\begin{align}
    A_m &= Q_m^*A_{m-1}Q_m \\
    A_m &= R_mA_{m-1}R_{m-1}^{-1}.
\end{align}
    \label{eq:ShifteDQRIterationAlternative}
\end{subequations}

\section{$p(A) = \hat{Q}_j \hat{R}_j$ (Exercise 6.2.36)} \label{sec:QRPolynomial}
This exercise shows several things.  First it shows that the shifted \QR algorithm creates a polynomials of $A$ with roots at the chosen shifts.
Let $A_m$ be generated by \QR algorithm:
\begin{subequations}\begin{align}
    A_{m-1} - \nu_{m-1}I = Q_mR_m \\
    A_m = R_mQ_m + \nu_{m-1}
\end{align}\end{subequations}
where $A_0 = A$.  Let $\hat{Q}_m = Q_1\cdots Q_m$ and $\hat{R}_m = R_m \cdots R_1$.

\begin{enumerate}[(a)]
    \item Show that
        \begin{equation}
            A_m = \hat{Q}_m^*A\hat{Q}_m
        \end{equation}
        for $m=1,2,3,\ldots$.

        \begin{enumerate}[$m=1$: ]
            \item $\hat{Q}_1 = Q_1$,
                \begin{equation}
                    \boxed{A_1 = Q_1^*AQ_1} \text{ by definition, see Eq. \ref{eq:ShifteDQRIterationAlternative} }.
                \end{equation}

            \item $\hat{Q}_2 = Q_1Q_2, \hspace{2ex} \hat{Q}_2^* = Q_2^*Q_1^*$
                \begin{subequations}\begin{align}
                    \text{we know: } A_2 &= Q_2^*A_1Q_2 \\
                     &= Q_2^*\left(Q_1^*AQ_1\right)Q_2 \\
                    A_2 &= \hat{Q}_2^*A\hat{Q}_2
                \end{align}\end{subequations}

            \item $\hat{Q}_3 = Q_1Q_2Q_3, \hspace{2ex} \hat{Q}_3^* = Q_3^*Q_2^*Q_1^*$
                \begin{subequations}\begin{align}
                    \text{we know: } A_3 &= Q_3^*A_2Q_3 \\
                     &= Q_3^*\left[Q_2^*A_1Q_2\right]Q_3 \\
                     &= Q_3^*\left[Q_2^*\left(Q_1^*AQ_1\right)Q_2\right]Q_3 \\
                    A_3 &= \hat{Q}_3^*A\hat{Q}_3
                \end{align}\end{subequations}
                Clearly this can continue for all $m$.
        \end{enumerate}

    \item Deduce that
        \begin{equation}
            \left(A-\nu_mI\right)\hat{Q}_m = \hat{Q}_m\left(A_m - \nu_mI\right)
        \end{equation}
        for $m=1,2,3,\ldots$.

        \begin{enumerate}[$m=1$: ]
            \item $\hat{Q}_1 = Q_1$
                \begin{equation}
                    \left(A-\nu I\right)Q_1 = AQ_1 - \nu_1Q_1
                    \label{eq:m1Results}
                \end{equation}

                First we must make note:
                \begin{subequations}\begin{align}
                    A_1 &= Q_1^*AQ_1 \\[2mm]
                    Q_1A_1 &= Q_1Q_1^*AQ_1 \\
                    Q_1A_1 &= AQ_1
                \end{align}\end{subequations}
                Plug this result into Eq. (\ref{eq:m1Results}) to get our answer
                \begin{subequations}\begin{align}
                    \left(A-\nu I\right)Q_1 &= Q_1A_1 - \nu_1Q_1 \\
                    \left(A-\nu I\right)\hat{Q}_1 &= \hat{Q}_1\left( A_1 - \nu_1 I \right)
                \end{align}\end{subequations}

            \item $\hat{Q}_2 = Q_1Q_2, \hspace{2ex} \hat{Q}_2^* = Q_2^*Q_1^*$
                \begin{equation}
                    \left(A-\nu_2 I\right)\hat{Q}_2 = \left(A-\nu_2 I\right)Q_1Q_2
                    \label{eq:m1Results}
                \end{equation}

                Again:
                \begin{subequations}\begin{align}
                    A_2 &= Q_2^*A_1Q_2 \\[2mm]
                    Q_2A_2 &= Q_2Q_2^*A_1Q_2 \\
                    Q_2A_2 &= A_1Q_2
                \end{align}\end{subequations}
                We will use this result shortly
                \begin{subequations}\begin{align}
                    \left(A-\nu_2 I\right)\hat{Q}_2 &= \left(A-\nu_2 I\right)Q_1Q_2 \\
                    &= \left(AQ_1-\nu_2 Q_1\right)Q_2 \\
                    &= \left(Q_1A_1-\nu_2 Q_1\right)Q_2 \\
                    &= Q_1\left(A_1-\nu_2 I\right)Q_2 \\
                    &= Q_1\left(A_1Q_2-\nu_2 Q_2\right) \\
                    &= Q_1\left(Q_2A_2-\nu_2 Q_2\right) \\[2mm]
                    \left(A-\nu_2 I\right)\hat{Q}_2 &= Q_1Q_2\left(A_2-\nu_2 I\right)
                \end{align}\end{subequations}
        \end{enumerate}
        I think it's clear that the rest follows by induction.

    \item Now prove by induction on $m$ that
        \begin{equation}
            \left(A - \nu_{m-1}I\right)\cdots\left(A-\nu_0I\right) = \hat{Q}_m\hat{R}_m, \hspace{5ex}
        \end{equation}
        for $m=1,2,3,\ldots$.

        \begin{enumerate}[$m=1$: ]
            \item $\left(A-\nu_0I\right) = \hat{Q}_1\hat{R}_1$
                \begin{equation}
                    \left(A-\nu_0I\right) = Q_1R_1
                \end{equation}
                This is true by construction of shifted \QR algorithm.

            \item $\left(A-\nu_1I\right)\left(A-\nu_0I\right) = \hat{Q}_2\hat{R}_2$
                \begin{subequations}\begin{align}
                    \left(A-\nu_1I\right)\left(A-\nu_0I\right) &= \left(A-\nu_1I\right)\left(Q_1R_1\right) \\
                     &= \left[ \left( A - \nu_1 \right)Q_1 \right]R_1 \\
                     &= \left[ \hat{Q}_1\left(A_1 - \nu_1I\right) \right]R_1 \\
                     &= \hat{Q}_1\left(Q_2R_2\right)R_1 \\
                    \left(A-\nu_1I\right)\left(A-\nu_0I\right) &= \hat{Q}_2\hat{R}_2
                \end{align}\end{subequations}

            \item $\left(A-\nu_2I\right)\left(A-\nu_1I\right)\left(A-\nu_0I\right) = \hat{Q}_3\hat{R}_3$
                \begin{subequations}\begin{align}
                    \left(A-\nu_2I\right)\left(A-\nu_1I\right)\left(A-\nu_0I\right) &= \left(A-\nu_2I\right)\hat{Q}_2\hat{R}_2 \\
                    &= \left(A-\nu_2I\right)Q_1Q_2\hat{R}_2 \\
                    &= \left(AQ_1-\nu_2Q_1\right)Q_2\hat{R}_2 \\
                    &= \left(Q_1A_1-\nu_2Q_1\right)Q_2\hat{R}_2 \\
                    &= Q_1\left(A_1-\nu_2I\right)Q_2\hat{R}_2 \\
                    &= Q_1\left(A_1Q_2-\nu_2Q_2\right)\hat{R}_2 \\
                    &= Q_1\left(Q_2A_2-\nu_2Q_2\right)\hat{R}_2 \\
                    &= Q_1Q_2\left(A_2-\nu_2I\right)\hat{R}_2 \\
                    &= Q_1Q_2\left(Q_3R_3\right)R_2R_1 \\
                    \left(A-\nu_2I\right)\left(A-\nu_1I\right)\left(A-\nu_0I\right) &= \hat{Q}_3\hat{R}_3
                \end{align}\end{subequations}
        \end{enumerate}
        The rest follows by induction.

\end{enumerate}

\section{Proving Theorem 6.4.11, (Exercise 6.4.22)}
\begin{thm}
    Suppose
    \begin{equation*}
        AV_m = V_mH_m + v_{m+1}h_{m+1,m}e_m^T
    \end{equation*}
    and let $p$ be a polynomial of degree $j < m$.  Then
    \begin{equation}
        p(A)V_m = V_mp(H_m) + E_j,
        \label{eq:ArnoldiPolynomial}
    \end{equation}
    where $E_j \in \mathbb{C}^{n \times m}$ is identically zero, except in the last $j$ columns.
    \label{thm:First}
\end{thm}

We will prove Theorem \ref{thm:First} by induction
\begin{enumerate}[(a)]
    \item Show that the theorem holds when $j=1$.  In this case $p(z) = \alpha_1\left(z-\nu_1\right)$, where $\alpha_1$ is some nonzero constant.
        \begin{subequations}
            \begin{align}
                \left(A-\nu_1I\right)V_m &= V_m\left(H_m-\nu_1I\right) + v_{m+1}h_{m+1,m}e_m^T \\
                p(A)V_m &= V_mp(H_m) + E_1 \\
            \end{align}
            \label{eq:j=1}
        \end{subequations}

        This one is trivial.
    \item Show that the theorem holds when $j=2$.  In this case $p(z) = k\alpha_2\left(z-\nu_1\right)\left(z-\nu_2\right)$.  This step is just for practice; it is not crucial to the proof of the Theorem.

        We begin with the Arnoldi factorization after applying one shift
        \begin{equation}
            \left(A-\nu_1I\right)V_m = V_m\left(H_m-\nu_1I\right) + E_1.
            \label{eq:OneShift}
        \end{equation}
        Now we can apply a second shift (this isn't applying a shift rather just multiplying by $\left(A-\nu_2I\right)$):
        \begin{equation}
            \left(A-\nu_2I\right)\left(A-\nu_1I\right)V_m = \left(A-\nu_2I\right)V_m\left(H_m-\nu_1I\right) + \left(A-\nu_2I\right)E_1.
            \label{eq:SecondShift}
        \end{equation}
        We can substitute $\nu_2$ for $\nu_1$ in Equation \ref{eq:OneShift} to obtain the identity
        \begin{equation}
            \left(A-\nu_2I\right)V_m = V_m\left(H_m-\nu_2I\right) + E_1.
        \end{equation}
        and insert this into Equation \ref{eq:SecondShift}
        \begin{subequations}
            \begin{align}
                \begin{split}
                    \left(A-\nu_2I\right)\left(A-\nu_1I\right)V_m &= \left(A-\nu_2I\right)V_m\left(H_m-\nu_1I\right) \\
                     &+ \left(A-\nu_2I\right)E_1.
                \end{split} \\
                \begin{split}
                    &= \left[V_m\left(H_m-\nu_2I\right) + E_1\right]\left(H_m-\nu_1I\right) \\
                    &+ \left(A-\nu_2I\right)E_1.
                \end{split} \\
                \begin{split}
                    &= V_m\left(H_m-\nu_2I\right)\left(H_m-\nu_1I\right) + E_1\left(H_m-\nu_1I\right) \\
                    &+ \left(A-\nu_2I\right)E_1.
                \end{split} \\
                    \left(A-\nu_2I\right)\left(A-\nu_1I\right)V_m &= V_m\left(H_m-\nu_2I\right)\left(H_m-\nu_1I\right) + E_2
            \end{align}
        \end{subequations}
        where $E_2 = E_1\left(H_m-\nu_1I\right) + \left(A-\nu_2I\right)E_1$.which is identically zero, except in the last 2 columns.

    \item Show that if the theorem holds for polynomials of degree $j-1$, then it holds for polynomials of degree $j$.

        We know:
        \begin{equation}
            \left(A-\nu_{j-1}I\right)\cdots\left(A-\nu_1I\right)V_m = V_m\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) + E_{j-1}
        \end{equation}
        Multiply this by $\left(A-\nu_jI\right)$,
        \begin{equation}\begin{split}
            \left(A-\nu_jI\right)\left(A-\nu_{j-1}I\right)\cdots\left(A-\nu_1I\right)V_m &= \\
            & \left(A-\nu_jI\right)V_m\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) \\
            &+ \left(A-\nu_jI\right)E_{j-1}
        \end{split}\end{equation}
        Of course:
        \begin{equation}
            \left(A-\nu_jI\right)V_m = V_m\left(H_m-\nu_jI\right) + E_1.
        \end{equation}
        which can be inserted into the previous equation  as before
        \begin{subequations}\begin{align}
            \begin{split}
                \left(A-\nu_jI\right)\cdots\left(A-\nu_1I\right)V_m &= \\
                &\left[ \left(A-\nu_jI\right)V_m \right]\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) \\
                &+ \left(A-\nu_jI\right)E_{j-1}
            \end{split} \\
            \begin{split}
                &= \left[ V_m\left(H_m-\nu_jI\right) + E_1 \right]\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) \\
                &+ \left(A-\nu_jI\right)E_{j-1}
            \end{split} \\
            \begin{split}
                &= V_m\left(H_m-\nu_jI\right)\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) \\
                &+ E_1\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) + \left(A-\nu_jI\right)E_{j-1}
            \end{split} \\
               \left(A-\nu_jI\right)\cdots\left(A-\nu_1I\right) &= V_m\left(H_m-\nu_jI\right)\left(H_m-\nu_{j-1}I\right)\cdots\left(H_m-\nu_1I\right) + E_j \\
       \intertext{Finally}
       p(A)V_m &= V_mp(H_m) + E_j.
        \end{align}\end{subequations}
\end{enumerate}

This isn't how IRAM actually proceeds.  However we can create an equation as in eq. \ref{eq:ArnoldiPolynomial} by simply operating on the left by some polynomial of $A$; in this case we have chosen a polynomial with root $\nu_j$.  It just so happens (not coincidentally) that using the same $\nu_j$'s as shifts for the shifted \QR algorithm on $H_m$.  Now we have proven in section \ref{sec:QRPolynomial} that
\begin{equation}
    p\left(H_m\right) = \hat{Q}_m\hat{R}_m
\end{equation}
where $\hat{Q}_m$ and $\hat{R}_m$ come from the shifted \QR algorithm.  Continue on from equation 6.4.12 in Watkins.

\section{Exercise (6.4.21) pg. 460}
Let $j$ be a non-negative integer.  A matrix $B$ is called $j$-Hessenberg if $b_{ij} = 0$ whenever $i-k>j$.  A $j$-Hessenberg matrix is said to be \emph{properly} $j$-Hessenberg if $b_{ij} \neq 0$ whenever $i-k=j$.

\begin{enumerate}[(a)]
    \item \emph{What are the common names for 0-Hessenberg and 1-Hessenberg matrices?}

    A 0-Hessenberg matrix is \emph{upper triangular}.  A 1-Hessenberg is commonly called \emph{upper Hessenberg}.

    \item Show that the product of a properly $j$-Hessenberg and a properly $k$-Hessenberg matrix is properly $\left(j+k\right)$-Hessenberg.\label{itm:j+k}

    The $r$th row of a properly $j$-Hessenberg matrix has
    \begin{equation*}
        \max\left(r-j-1,0\right)
    \end{equation*}
    leading zeros.  The $c$th column of a properly $k$-Hessenberg matrix has
    \begin{equation*}
        \max\left[s-\left(c+k)\right),0\right]
    \end{equation*}
    trailing zeros where $s$ is the size of the matrix.

    Suppose $\mathcal{J}$ is properly $j$-Hessenberg and $\mathcal{K}$ is properly $k$-Hessenberg and $\mathcal{J},\mathcal{K} \in \mathbb{R}^{s\times s}$.  The resulting product $\mathcal{A} = \mathcal{JK}$.  The elements of $\mathcal{A}$, $a_{mn}$, are the inner product of the $m$-th row of $\mathcal{J}$ and the $n$-th column of $\mathcal{K}$.  This element will be zero if the sum of the number of leading zeros in the $m$-th row of $\mathcal{J}$ and the number of trailing zeros in the $n$-th column of $\mathcal{K}$ are greater than the size of the matrix, $s$:
    \begin{equation}
        a_{mn} = \begin{cases}
            0 & \left( \max\left[(r-j-1),0\right] + \max\left[s-\left(c+k\right),0\right] \right) \geq s \\
            \mathcal{J}_m\cdot\mathcal{K}_n & \mathrm{otherwise}
        \end{cases}
    \end{equation}
    where $\mathcal{J}_m$ is the $m$-th row of $\mathcal{J}$ and $\mathcal{K}_n$ is the $n$-th column of $\mathcal{K}$.  For $\mathcal{A}$ to be properly $\left(j+k\right)$-Hessenberg $a_{mn} = 0$ whenever $m-n>\left(j+k\right)$ and $a_{mn}\neq 0$ whenever $m-n = \left(j+k\right)$.

    \item  Show that if $B \in \mathbb{C}^{m\times m}$ is properly$j$-Hessenberg ($j<m$>, then the first $m-j$ columns of $B$ are linearly independent.

    If $B$ is properly $j$-Hessenberg then column $k$ will have one more non-zero element than column \mbox{$k-1$} for \mbox{$k =2,\ldots j$}.  Thus they must be linearly independent.

    \item Show that if $H_m$ is properly upper Hessenberg and $p$ is a polynomial of degree $j$, say $p(z) = \left(z-\nu_1\right)\cdots\left(z-\nu_j\right)$, then $p(H_m)$ is properly $j$-Hessenberg.

    In $p(H_m)$ there will be a term $H_m^j$, which by item \ref{itm:j+k} will be $j$-Hessenberg.  Adding in the other lower order terms will not affect this.

\end{enumerate}

\section{Manual Application of Shifted Algorithm}
I write this section to demonstrate manually, and laboriously what happens when you perform shifted \QR iterations.  The traditional way to show the iterations is given in the left column.  The right column is just some algebra to get the shift in the form
\begin{equation}
    \left( A_0-\nu_nI\right)\cdots\left( A_0 - \nu_1I \right) = \hat{Q}_n\hat{R}_n.
\end{equation}
This is necessary for some proofs I am doing.

\clearpage
\begin{Parallel}[v]{0.49\textwidth}{0.49\textwidth}
\textbf{First Shift}
\ParallelLText{
    \begin{equation}
        \left(A_0 - \nu_1I\right) = Q_1R_1
    \end{equation}

    \begin{subequations}
        \begin{align}
            A_1 &= R_1Q_1  + \nu_1I \\
            A_1 &= Q_1^*A_0Q_1 \\
            A_1 &= \hat{Q}_1^*A_0\hat{Q}_1
        \end{align}
    \end{subequations}
}
\ParallelRText{
    \begin{align}
        \left(A_0 - \nu_1I\right) &= Q_1R_1 \\[2mm]
        Q_1A_1 &= A_0Q_1
    \end{align}
}
\ParallelPar

\vspace{\baselineskip}
\textbf{Second Shift}
\ParallelLText{
    \begin{equation}
        \left(A_1 - \nu_2I\right) = Q_2R_2
    \end{equation}

    \begin{subequations}
        \begin{align}
            A_2 &= R_2Q_2  + \nu_2I \\
            A_2 &= Q_2^*A_1Q_2 \\
            A_2 &= Q_2^*\left(Q_1^*A_0Q_1\right)Q_2 \\
            A_2 &= \hat{Q}_2^*A_0\hat{Q}_2
        \end{align}
    \end{subequations}
}
\ParallelRText{
    \begin{align}
        \left(A_1 - \nu_2I\right) &= Q_2R_2 \\[2mm]
        Q_2A_2 &= A_0Q_2    \label{eq:2Identity}
    \end{align}

    \begin{subequations}
        \begin{align}
            \hat{Q}_1\left(A_1-\nu_2I\right)\hat{R}_1 &= \hat{Q}_1Q_2R_2\hat{R}_1 \\
            \hat{Q}_1A_1\hat{R}_1-\nu_2\hat{Q}_1\hat{R}_1 &= \hat{Q}_2\hat{R}_2 \\
            A_0\hat{Q}_1\hat{R}_1-\nu_2\hat{Q}_1\hat{R}_1 &= \hat{Q}_2\hat{R}_2 \\
            \left( A_0-\nu_2I\right) \hat{Q}_1\hat{R}_1 &= \hat{Q}_2\hat{R}_2 \\
            \left( A_0-\nu_2I\right)\left( A_0 - \nu_1I \right) &= \hat{Q}_2\hat{R}_2
        \end{align}
    \end{subequations}

}
\ParallelPar

\vspace{\baselineskip}
\textbf{$n$'th Shift}
\ParallelLText{
    \begin{equation}
        \left(A_{n-1} - \nu_n\right) = Q_nR_n
    \end{equation}

    \begin{subequations}
        \begin{align}
            A_n &= R_nQ_n  + \nu_nI \\
            A_n &= Q_n^*A_{n-1}Q_n \\
            A_n &= Q_n^*\left(Q_{n-1}^*A_{n-2}Q_{n-1}\right)Q_n \\
            \vdots \\
            A_n &= \hat{Q}_n^*A_0\hat{Q}_n
        \end{align}
    \end{subequations}
}
\ParallelRText{
    \begin{align}
        \left(A_{n-1} - \nu_nI\right) &= Q_nR_n \\[2mm]
        \hat{Q}_{n-1}A_{n-1} &= A_0\hat{Q}_{n-1}
    \end{align}

    \begin{subequations}
        \begin{align}
            \hat{Q}_{n-1}\left(A_{n-1}-\nu_nI\right)\hat{R}_{n-1} &= \hat{Q}_{n-1}Q_nR_n\hat{R}_{n-1} \\
            \hat{Q}_{n-1}A_{n-1}\hat{R}_{n-1}-\nu_n\hat{Q}_{n-1}\hat{R}_{n-1} &= \hat{Q}_n\hat{R}_n \\
            A_0\hat{Q}_{n-1}\hat{R}_{n-1}-\nu_n\hat{Q}_{n-1}\hat{R}_{n-1} &= \hat{Q}_n\hat{R}_n \\
            \left( A_0-\nu_nI\right)\left( \hat{Q}_{n-1}\hat{R}_{n-1} \right) &= \hat{Q}_n\hat{R}_n \\
            \left( A_0-\nu_nI\right)\cdots\left( A_0 - \nu_1I \right) &= \hat{Q}_n\hat{R}_n
        \end{align}
    \end{subequations}

}
\end{Parallel}

\end{document}


%% SECTION HEADER /////////////////////////////////////////////////////////////////////////////////////
\section{3D Model of the PZT transducers}
\label{sec:3Dpzt}

%% SECTION CONTENT ////////////////////////////////////////////////////////////////////////////////////

The displacement vector of the PZT transducer is composed of three translational displacements and  is defined as:
\begin{eqnarray}
	\left \{ \begin{array}{c}
		\textbf{u}^e(\xi,\eta,\zeta) \\
		\textbf{v}^e(\xi,\eta,\zeta) \\
		\textbf{w}^e(\xi,\eta,\zeta)
	\end{array} \right\}
	= \textbf{N}^e(\xi,\eta, \zeta)\widehat{\textbf{d}}^e
	= \sum_{l=1}^r\sum_{n=1}^q\sum_{m=1}^p\textbf{N}_m^e(\xi)\textbf{N}_n^e(\eta)\textbf{N}_l^e(\zeta)
	\left \{ \begin{array}{c}
		\widehat{\textbf{u}}^e(\xi_m,\eta_n,\zeta_l) \\
		\widehat{\textbf{v}}^e(\xi_m,\eta_n,\zeta_l) \\
		\widehat{\textbf{w}}^e(\xi_m,\eta_n,\zeta_l)
	\end{array} \right\},
	\label{eq:3D_displ}
\end{eqnarray}
where \(\widehat{\textbf{u}}^e\), \(\widehat{\textbf{v}}^e\) and
\(\widehat{\textbf{w}}^e\) are displacements of the element nodes in \(\xi,\eta\) and \(\zeta\) direction.

The nodal strain--displacement relations are given as \cite{kudela20093d}:
\begin{eqnarray}
	\boldsymbol{\epsilon}^e=\textbf{B}_{d}^e\widehat{\textbf{d}}^e=
	\left [
	\begin{array}{ccc}
		\frac{\partial N^e}{\partial x} & 0 & 0\\
		0 & \frac{\partial N^e}{\partial y} & 0\\
		0 & 0 & \frac{\partial N^e}{\partial z}\\
		0 & \frac{\partial N^e}{\partial z} & \frac{\partial N^e}{\partial y}\\
		\frac{\partial N^e}{\partial z} & 0 & \frac{\partial N^e}{\partial x}\\
		\frac{\partial N^e}{\partial y} & \frac{\partial N^e}{\partial x} & 0
	\end{array} \right]
	\left \{ \begin{array}{c}
		\widehat{\textbf{u}}^e \\
		\widehat{\textbf{v}}^e \\
		\widehat{\textbf{w}}^e
	\end{array} \right\}.
\end{eqnarray}

The electromechanical coupling is governed by the linear constitutive equation of piezoelectric material according to~\cite{giurgiutiu2009micromechatronics, rekatsinas2017cubic}, and this is defined as:
\begin{eqnarray}
	\left [
	\begin {array}{c}
	\boldsymbol{\sigma}\\
	\textbf{D}
\end{array}\right ]=
\left [
\begin{array}{cc}
	\textbf{c}^E & -\textbf{e}^T \\
	\textbf{e} & \epsilon^S
\end{array} \right ]
\left[
\begin{array}{c}
	\textbf{S}\\
	\textbf{E}
\end{array} \right ],
\end{eqnarray}
where \(\boldsymbol{\sigma}\) and \(\textbf{S}\) are the stress and strain components, respectively, \(\textbf{c}^E\) is the stiffness coefficient matrix measured at zero electric field, \textbf{e} is the piezoelectric coupling tensor,  \(\boldsymbol{\epsilon}^S\) is the electric permittivity, and \textbf{E} and \textbf{D} are the electric field and electric displacement measured at zero strain.
The superscript T denotes a transpose matrix.
The electric field is defined as:
\begin{eqnarray}
\textbf{E}^e=-\textbf{B}_\phi^e \widehat{\boldsymbol{\phi}}^e = \left[ \begin{array}{c}
	\frac{\partial N^e}{\partial \xi}\\
	\frac{\partial N^e}{\partial \eta}\\
	\frac{\partial N^e}{\partial \zeta}
\end{array} \right] \widehat{\boldsymbol{\phi}}^e.
\end{eqnarray}
where \(\widehat{\boldsymbol{\phi}}^e\) is a nodal voltage of the transducer.


\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[osf]{libertine}
\usepackage[scaled=0.8]{beramono}
\usepackage[margin=1.5in]{geometry}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{bm}

\usepackage{sectsty}
\sectionfont{\large}
\subsectionfont{\normalsize}

\usepackage{titlesec}
\titlespacing{\section}{0pt}{10pt plus 2pt minus 2pt}{0pt plus 2pt minus 0pt}
\titlespacing{\subsection}{0pt}{5pt plus 2pt minus 2pt}{0pt plus 2pt minus 0pt}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex}

\newcommand{\acro}[1]{\textsc{\MakeLowercase{#1}}}
\newcommand{\given}{\mid}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\data}{\mc{D}}
\newcommand{\intd}[1]{\,\mathrm{d}{#1}}
\newcommand{\mat}[1]{\bm{\mathrm{#1}}}
\renewcommand{\vec}[1]{\bm{\mathrm{#1}}}
\newcommand{\trans}{^\top}
\newcommand{\inv}{^{-1}}

\DeclareMathOperator{\var}{var}
\DeclareMathOperator{\tr}{tr}

\usepackage{pgfplots}
\pgfplotsset{
  compat=newest,
  plot coordinates/math parser=false,
  tick label style={font=\footnotesize, /pgf/number format/fixed},
  label style={font=\small},
  legend style={font=\small},
  every axis/.append style={
    tick align=outside,
    clip mode=individual,
    scaled ticks=false,
    thick,
    tick style={semithick, black}
  }
}

\pgfkeys{/pgf/number format/.cd, set thousands separator={\,}}

\usepgfplotslibrary{external}
\tikzexternalize[prefix=tikz/]

\newlength\figurewidth
\newlength\figureheight

\setlength{\figurewidth}{8cm}
\setlength{\figureheight}{6cm}

\begin{document}

{\large \textbf{CSE 515T (Spring 2015) Assignment 2 solutions}} \\

\begin{enumerate}

\item
  (Curse of dimensionality.)
  Consider a $d$-dimensional, zero-mean, spherical multivariate
  Gaussian distribution:
  \begin{equation*}
    p(\vec{x}) = \mc{N}(\vec{x}; \vec{0}, \mat{I}_d).
  \end{equation*}
  Equivalently, each entry of $\vec{x}$ is drawn iid from a univariate
  standard normal distribution.

  In familiar small dimensions ($d \leq 3$), ``most'' of the vectors
  drawn from a multivariate Gaussian distribution will lie near the
  mean.  For example, the famous 68--95--99.7 rule for $d = 1$
  indicates that large deviations from the mean are unusual.  Here we
  will consider the behavior in larger dimensions.
  \begin{itemize}
  \item Draw 10\,000 samples from $p(\vec{x})$ for each dimension in
    $d \in \{1, 5, 10, 50, 100\},$ and compute the length of each
    vector drawn: $y_d = \sqrt{\vec{x}\trans \vec{x}} = (\sum_i^d
    x_i^2)^{\nicefrac{1}{2}}$.  Estimate the distribution of each
    $y_d$ using either a histogram or a kernel density estimate (in
    \acro{MATLAB}, \texttt{hist} and \texttt{ksdensity},
    respectively).  Plot your estimates.  (Please do not hand in your
    raw samples!)  Summarize the behavior of this distribution as $d$
    increases.
  \item
    The true distribution of $y_d^2$ is a chi-square distribution with
    $d$ degrees of freedom (the distribution of $y_d$ itself is the
    less-commonly seen chi distribution).  Use this fact to compute
    the probability that $y_d < 5$ for each of the dimensions in the
    last part.
  \item
    For $d = 1\,000$, compute the 5th and 95th percentiles of $y_d$.
    Is the mean $\vec{x} = \vec{0}$ a representative summary of the
    distribution in high dimensions?  This behavior has been called
    ``the curse of dimensionality.''
  \end{itemize}

\end{enumerate}

\subsection*{Solution}

Kernel density estimates of the empirical distributions of $y$ (using
10\,000 samples each) are shown in Figure \ref{problem_1}. As the
dimension increases, we see that the bulk of the probability mass
actually lives in a thin ``shell'' centered around the origin: all
samples are approximately the same length, with no vectors near the
mean.  This is somewhat unintuitive.

The compute the probability that $y_d < 5$, we can evaluate the
corresponding $\chi^2$ \acro{CDF} at $y_d^2 = 25$:\footnote{Computed
  with \texttt{chi2cdf(25, [1, 5, 10, 50, 100])} in \acro{MATLAB}.}
\begin{align*}
  \Pr(y < 5 \given d = \phantom{00}1) &\approx 1.0000 \\
  \Pr(y < 5 \given d = \phantom{00}5) &\approx 0.9999 \\
  \Pr(y < 5 \given d = \phantom{0}10) &\approx 0.9947 \\
  \Pr(y < 5 \given d = \phantom{0}50) &\approx 0.0012 \\
  \Pr(y < 5 \given d = 100) &\approx 0.0000,
\end{align*}
so the probability of being within a distance of five standard
deviations form he mean decreases from near certainty to near
impossibility, another surprising result.

For the last part, we invert the $\chi^2$ \acro{CDF} and take the
square root.\footnote{Computed with \texttt{sqrt(chi2inv([0.05, 0.95],
    1000))} in \acro{MATLAB}.}  The 5th percentile is 30.46 and the
95th percentile is 32.78.  Again, we see that most of the mass lies in
a narrow shell centered around the mean.

Whether the mean is a representative summary is a much more
complicated question with no definitive answer.  In some sense, it's a
very odd summary: in dimensions higher than 10 or so, we can't imagine
seeing a vector anywhere near the mean!  On the other hand, if we want
to choose another single point to summarize the distribution, there's
no clear better alternative.  By definition, the mean minimizes the
average squared distance from the chosen point to the vector
$\vec{x}$.  It just so happens that the \emph{minimum} squared
distance to the mean is relatively high in large dimension.  This is
the ``curse of dimensionality:'' all points are ``far away from the
mean'' (and also each other!).

\begin{figure}
  \centering
  \input{figures/problem_1.tex}
  \caption{Empirical distributions of lengths $y$ as a function of the
    dimension $d$.}
  \label{problem_1}
\end{figure}

\clearpage
\begin{enumerate}
\setcounter{enumi}{1}
\item
  (Bayesian linear regression.)
  Consider the following data:
  \begin{align*}
    \vec{x}
    &=
    [-2.26, -1.31, -0.43, 0.32, 0.34, 0.54, 0.86, 1.83, 2.77, 3.58]\trans; \\
    \vec{y}
    &=
    [1.03, 0.70, -0.68, -1.36, -1.74, -1.01, 0.24, 1.55, 1.68, 1.53]\trans.
  \end{align*}
  Fix the noise variance at $\sigma^2 = 0.5^2$.
  \begin{itemize}
  \item
    Perform Bayesian linear regression for these data using the
    polynomial basis functions $\phi_k(x) = [1, x, x^2, \dotsc
      x^k]\trans$ for $k \in \{1, 2, 3\}$, in each case using the
    parameter prior $p(\vec{w}) = \mc{N}(\vec{w}; \vec{0}, \mat{I})$.
    Evaluate and plot the posterior means $\mathbb{E}[\vec{y}_\ast
      \given \mat{X}_\ast, \data, \sigma^2]$ on the interval $x_\ast
    \in [-4, 4]$ for each model.  Also plot the posterior mean
    plus-or-minus two times the posterior standard deviation:
    \begin{equation*}
      \mathbb{E}[\vec{y}_\ast \given \mat{X}_\ast, \data, \sigma^2] \pm
      2 \sqrt{\var[\vec{y}_\ast \given \mat{X}_\ast, \data, \sigma^2]}.
    \end{equation*}
    This is a pointwise 95\% credible interval for the regression
    function.  Where is the pointwise uncertainty the largest?
  \item
    Compute the marginal likelihood of the data for each of the basis
    expansions above: $p(\vec{y} \given \mat{X}, k, \sigma^2)$.  Which
    model explains the data the best?
  \end{itemize}
\end{enumerate}

\subsection*{Solution}

Given the feature expansions $\mat{\Phi} = \phi(\mat{X})$ and
$\mat{\Phi}_\ast = \phi(\mat{X}_\ast)$, we may compute the posterior
distribution of $\vec{y}_\ast$:
\begin{equation*}
  p(\vec{y}_\ast \given \vec{X}_\ast, \data)
  =
  \mc{N}(\vec{y}_\ast;
  \mat{\Phi}_\ast \vec{\mu}_{\vec{w}\given\data},
  \mat{X}_\ast \mat{\Sigma}_{\vec{w}\given\data} \mat{X}_\ast\trans + \sigma^2 \mat{I}),
\end{equation*}
where
\begin{align*}
  \vec{\mu}_{\vec{w}\given\data}
  &=
  \vec{\mu}
  +
  \mat{\Sigma}
  \mat{\Phi}\trans
  (\mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans + \sigma^2 \mat{I})\inv
  (\vec{y} - \mat{\Phi}\vec{\mu});
  \\
  \mat{\Sigma}_{\vec{w}\given\data}
  &=
  \mat{\Sigma}
  -
  \mat{\Sigma}
  \mat{\Phi}\trans
  (\mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans + \sigma^2 \mat{I})\inv
  \mat{\Phi}
  \mat{\Sigma}.
\end{align*}
The diagonal of the posterior covariance for $\vec{y}\ast$,
$\mat{\Phi}_\ast\mat{\Sigma}\mat{\Phi}_\ast + \sigma^2\mat{I}$,
gives the desired variance for plotting the credible interval.

Plugging in the prior $p(\vec{w}) = \mc{N}(\vec{w}; \vec{0}, \mat{I})$
gives the result, plotted below.  For all three models, the pointwise
uncertainty is maximized on the extreme ranges of the domain at $x =
-4$ and $x = 4$; we have few observations near either of these
locations.  The pointwise uncertainty tends to be especially large at
$x = -4$.

\begin{figure}
  \centering
  \input{figures/order_1_expansion}
  \caption{Posterior for $k = 1$.}
  \label{order_1_expansion}
\end{figure}

\begin{figure}
  \centering
  \input{figures/order_2_expansion}
  \caption{Posterior for $k = 2$.}
  \label{order_2_expansion}
\end{figure}

\begin{figure}
  \centering
  \input{figures/order_3_expansion}
  \caption{Posterior for $k = 3$.}
  \label{order_3_expansion}
\end{figure}

To compute the marginal likelihood, we must compute
\begin{equation*}
  p(\vec{y} \given \mat{X}, \sigma^2)
  =
  \mc{N}(\vec{y};
  \mat{\Phi}\vec{\mu},
  \mat{\Phi}\mat{\Sigma}\mat{\Phi}\trans + \sigma^2 \mat{I}).
\end{equation*}
Computing the marginal likelihood comes down to plugging in our
observations $\vec{y}$ into this Gaussian \acro{PDF}.  In practice it
is more convenient to compute the logarithm of the marginal
likelihood, due to the large dynamic range of this function.  For our
data, we can compute:
\begin{align*}
  \log p(\vec{y} \given \mat{X}, \sigma^2, k = 1)
  &= -32.9; \\
  \log p(\vec{y} \given \mat{X}, \sigma^2, k = 2)
  &= -22.3; \\
  \log p(\vec{y} \given \mat{X}, \sigma^2, k = 3)
  &= -22.2.
\end{align*}
There is a clear preference for either the quadratic or the cubic
model over the linear model, but there is no clear-cut winner between
those two.

\clearpage
\begin{enumerate}
\setcounter{enumi}{2}
\item
  (Optimal design for Bayesian linear regression.)
  Consider the data from the last problem, and suppose we have
  selected the quadratic model corresponding to $k = 2$ (do not assume
  that this is the answer to the last part of the last question).
  Imagine we are allowed to evaluate the function at a point $x'$ of
  our choosing, giving a new dataset $\data' = \data \cup \bigl\{ (x',
  y') \bigr\}$ and a new posterior for the parameters $p(\vec{w}
  \given \data', \sigma^2) = \mc{N}(\vec{w};
  \vec{\mu}_{\vec{w}\given\data'},
  \mat{\Sigma}_{\vec{w}\given\data'})$.  We hope to select the
  location $x'$ to best improve our current model, under some quality
  measure.

  Assume that we ultimately wish to predict the function at a grid of
  points
  \begin{equation*}
    \vec{x}_\ast = [-4, -3.5, -3, \dotsc, 3.5, 4]\trans.
  \end{equation*}
  We select the squared loss for a set of predictions
  $\hat{\vec{y}}_\ast$ at these points:
  \begin{equation*}
    \ell(\vec{y}_\ast, \hat{\vec{y}}_\ast)
    =
    \sum_i \bigl((y_\ast)_i - (\hat{y}_\ast)_i\bigr)^2;
  \end{equation*}
  therefore, we will predict using the new posterior mean
  $\hat{\vec{y}}_\ast = \mat{X}_\ast \vec{\mu}_{\vec{w}\given\data'}$.
  \begin{itemize}
  \item
    Given a potential observation location $x'$, derive a closed-form
    expression for the expected loss
    $\mathbb{E}\bigl[\ell(\vec{y}_\ast, \hat{\vec{y}}_\ast) \given x',
      \data \bigr]$.  Note: this does not require integration over
    $y'$!  (What is the expected squared deviation from the mean?)
  \item
    Plot the expected loss over the interval $x' \in [-4, 4]$.  Where
    is the optimal location to sample the function?
  \end{itemize}

  Note: this approach of actively selecting where to sample a function
  to maximize some utility function is known as \emph{active learning}
  in machine learning and \emph{optimal experimental design} in
  statistics.  Bayesian decision theory provides a convenient and
  consistent framework for performing active learning with a variety
  of objectives.
\end{enumerate}

\subsection*{Solution}

Imagine for the sake of argument that we have been given a new
observation $(x', y')$ to our dataset $\data$, forming the augmented
dataset $\data'$.  Imagine further that we have computed the updated
posterior $p(\vec{w} \given \data', \sigma^2)$ with this new dataset.

We are compelled to predict the value of the function at the given
test inputs $\vec{x}_\ast$.  Under the given squared loss function
$\ell(\vec{y}_\ast, \hat{\vec{y}}_\ast)$, we will predict the
posterior mean $\hat{\vec{y}}_\ast \vec{\mu}_{\vec{w}\given\data'}$.
(Recall the Bayes estimator under squared loss is the posterior mean).

Let us explicitly compute the expected loss given $\data$':
\begin{align*}
  \mathbb{E}\bigl[
    \ell(\vec{y}_\ast, \hat{\vec{y}}_\ast)
    \given
    \vec{x}_\ast,
    \data'
  \bigr]
  &=
  \mathbb{E}\Bigl[
    \sum_i
    \bigl(
    (y_\ast)_i - (\hat{y}_\ast)_i
    \bigr)^2
    \given \vec{x}_\ast, \data'
  \Bigr]
  \\
  &=
  \sum_i
  \mathbb{E}\Bigl[
    \bigl(
    (y_\ast)_i
    -
    \mathbb{E}\bigl[(y_\ast)_i \given (x_\ast)_i, \data'\bigr]
    \bigr)^2
    \given (x_\ast)_i, \data'
    \Bigr]
  \\
  &=
  \sum_i
  \var\bigl[
    (y_\ast)_i
    \given
    (x_\ast)_i, \data'
  \bigr]
  \\
  &=
  \tr
  \bigl(
  \vec{\Phi}_\ast
  \Sigma_{\vec{w}\given\data'}
  \vec{\Phi}_\ast\trans
  +
  \sigma^2 \mat{I}
  \bigr),
\end{align*}
where, in successive lines, we have: applied the linearity of
expectation, substituted the posterior mean predictions for
$\hat{\vec{y}}_\ast$, applied the definition of variance, and
rewritten the sum of the variances in terms of the trace of the
posterior covariance matrix over $\vec{y}_\ast$ given $\data'$.

The key observation here is that the posterior covariance matrix, and
therefore the expected loss given $\data'$, \emph{does not depend} on
the value of $y'$, only the location of the new input $x'$.  So we may
actually compute the future expected loss as a function of the next
observation location $x'$.  The Bayes action will then be to sample
the function at the point minimizing the trace of the updated
posterior covariance over $\vec{y}_\ast$.

In Figure \ref{problem_3}, we plot the expected loss as a function of
$x'$.  The expected loss is minimized at $x' = -4$, which is the Bayes
action.

Of course, we could have iteratively performed this procedure to
select every observation location!  The result would fall under the
general framework of so-called \emph{active learning.}

\begin{figure}
  \centering
  \input{figures/problem_3.tex}
  \caption{Expected loss for predicting $\vec{y}_\ast$ given $\data'$
    as a function of $x'$.}
  \label{problem_3}
\end{figure}

\clearpage
\begin{enumerate}
\setcounter{enumi}{3}
\item
  (Woodbury matrix identity.)
  The \emph{Woodbury matrix identity} is a very useful result.  Let
  $\mat{A}$ be an $(n \times n)$ matrix, let $\mat{U}$ and $\mat{V}$
  be $(n \times k)$ matrices, and let $\mat{C}$ be a $(k \times k)$
  matrix.  Then:
  \begin{equation*}
    (\mat{A} + \mat{U}\mat{C}\mat{V}\trans)\inv
    =
    \mat{A}\inv
    -
    \mat{A}\inv
    \mat{U}
    (\mat{C}\inv + \mat{V}\trans \mat{A}\inv \mat{U})\inv
    \mat{V}\trans
    \mat{A}\inv.
  \end{equation*}
  This result is useful when you already have the inverse of a matrix
  $\mat{A}$ and want to know the inverse after a rank-$k$ adjustment.
  When $k \ll n$, the Woodbry matrix identity can be considerably
  faster than direct inversion!
  \begin{itemize}
  \item
    Prove this result.
  \item
    Use this result to rewrite the posterior covariance of the weight
    vector $\vec{w}$ in Bayesian linear regression (as written in the
    notes to lecture 5) in a simpler form.
  \end{itemize}
\end{enumerate}

\subsection*{Solution}

The first part of the problem can be completed by multiplying the
right hand side by $(\mat{A} + \mat{U}\mat{C}\mat{V}\trans)$ and
checking you get the identity.

The posterior covariance for $\vec{w}$ was previously given as
\begin{equation*}
  \mat{\Sigma}_{\vec{w}\given\data}
  =
  \mat{\Sigma}
  -
  \mat{\Sigma}
  \mat{X}\trans
  (\mat{X}\mat{\Sigma}\mat{X}\trans + \sigma^2 \mat{I})\inv
  \mat{X}
  \mat{\Sigma}.
\end{equation*}
Taking
\begin{equation*}
  \mat{A} = \mat{\Sigma}\inv
  \qquad
  \mat{U} = \mat{V} = \mat{X}\trans
  \qquad
  \mat{C} = \sigma^2 \mat{I},
\end{equation*}
we may rewrite this as
\begin{equation*}
  \mat{\Sigma}_{\vec{w}\given\data}
  =
  (\mat{\Sigma}\inv
  +
  \sigma^{-2}
  \mat{X}\trans\mat{X})\inv.
\end{equation*}

\clearpage
\begin{enumerate}
\setcounter{enumi}{4}
\item
  (Laplace approximation.)
  Find a Laplace approximation to the Gamma distribution:
  \begin{equation*}
    p(\theta \given \alpha, \beta)
    =
    \frac{1}{Z}
    \theta^{\alpha - 1}
    \exp(-\beta\theta).
  \end{equation*}
  Plot the approximation against the true density for $(\alpha, \beta)
  = (2, \nicefrac{1}{2})$.

  The true value of the normalizing constant is
  \begin{equation*}
    Z = \frac{\Gamma(\alpha)}{\beta^\alpha}.
  \end{equation*}
  If we fix $\beta = 1$, then $Z = \Gamma(\alpha)$, so we may use the
  Laplace approximation to estimate the Gamma function.  Analyze the
  quality of this approximation as a function of $\alpha$.
\end{enumerate}

\subsection*{Solution}

We first define the unnormalized log density $\Psi(\theta)$:
\begin{equation*}
  \Psi(\theta)
  =
  \log p(\theta \given \alpha, \beta)
  =
  (\alpha - 1) \log \theta - \beta\theta.
\end{equation*}
Next we find the maximal value of the distribution, $\hat{\theta}$, by
computing the derivative and setting to zero:
\begin{equation*}
  0
  =
  \frac{d}{d\theta}
  \Psi(\theta)
  =
  \frac{\alpha - 1}{\theta}
  - \beta
  \quad
  \Rightarrow
  \quad
  \hat{\theta} = \frac{\alpha - 1}{\beta}.
\end{equation*}
Next we compute the negative Hessian of $\Psi$ at $\hat{\theta}$.
Note that here we have a one-dimensional density, so the Hessian is
simply equal to the second derivative:
\begin{equation*}
  H
  =
  -\frac{d^2}{d\theta^2}
  \Psi(\theta)
  \biggr\rvert_{\theta = \hat{\theta}}
  =
  \frac{\alpha - 1}{\theta^2}
  \biggr\rvert_{\theta = \hat{\theta}}
  =
  \frac{\beta^2}{\alpha - 1}.
\end{equation*}
Now the Laplace approximation to the gamma distribution is
\begin{equation*}
  p(\theta \given \alpha, \beta)
  \approx
  \mc{N}(\theta; \hat{\theta}, H\inv)
  =
  \mc{N}\biggl(\theta;
  \frac{\alpha - 1}{\beta},
  \frac{\alpha - 1}{\beta^2}
  \biggr).
\end{equation*}
The corresponding estimate for the normalizing constant $Z$
is
\begin{equation*}
  Z
  \approx
  \exp\bigl(\Psi(\hat{\vec{\theta}})\bigr)
  \sqrt{
    \frac{(2\pi)^d}
         {\det \mat{H}}
  }
  =
  p(\hat{\theta} \given \alpha, \beta)
  \sqrt{\frac{2\pi}{H}}
  =
  \sqrt{\frac{2\pi(\alpha - 1)}{\beta^2}}
  \biggl(\frac{\alpha - 1}{\beta}\biggr)^{\alpha - 1}
  \exp\bigl(-(\alpha - 1)\bigr).
\end{equation*}
Plugging in $\beta = 1$ and using the true normalizing constant of the
gamma distribution, we have the approximation
\begin{equation*}
  \Gamma(\alpha)
  \approx
  \sqrt{2\pi}
  (\alpha - 1)^{\alpha - \nicefrac{1}{2}}
  \exp\bigl(-(\alpha - 1)\bigr).
\end{equation*}
Note we also have an approximation to the logarithm
of $\Gamma$:
\begin{equation*}
  \log \Gamma(\alpha)
  \approx
  \textstyle
  \frac{1}{2}\log 2 \pi
  +
  (\alpha - \nicefrac{1}{2})
  \log(\alpha - 1)
  -
  (\alpha - 1).
\end{equation*}

Figure \ref{problem_5} shows the resulting approximation to $Z =
\Gamma(\alpha)$ as a function of $\alpha$.  The approximation
appears to be quite good for $\alpha \geq 2$.

To those who are mathematically inclined, note that $n! = \Gamma(n +
1)$.  With a bit of manipulation, we have actually rediscovered a very
famous result known as \emph{Stirling's approximation:}
\begin{equation*}
  n!
  \approx
  \sqrt{2\pi n}
  \biggl(\frac{n}{e}\biggr)^n.
\end{equation*}

\begin{figure}
  \centering
  \input{figures/problem_5.tex}
  \caption{Laplace approximation to $Z = \Gamma(\alpha)$ as a function
    of $\alpha$.}
  \label{problem_5}
\end{figure}

\end{document}


\chapter{Spans, linear independence, and bases in \texorpdfstring{$\R^n$}{Rn}}


\documentclass[journal,11pt]{IEEEtran}
\usepackage[utf8]{inputenc}
\usepackage{parskip}
\usepackage[sorting=none]{biblatex}
\addbibresource{biblio}
\usepackage{datetime}
\usepackage{float}
\usepackage{neuralnetwork}
\usepackage{tikz}
\usepackage{siunitx}
\usepackage{pgfplots}
\pgfplotsset{compat=1.16}
\usepackage{tabularx}
\usepgfplotslibrary{units}
\usepackage{graphicx}
\graphicspath{{img}}

\title{LELEC2870 - Project in Machine learning\\\vspace*{10pt} \Large Prediction of air-quality in Beijing}
\author{Group AM: Martin Braquet (06641500), Amaury Gouverneur (69331500)\\\vspace*{10pt} \normalsize \today{}}
\date{December 2019}

\begin{document}

\maketitle

\section{Introduction}

This report aims in predicting the concentration of PM2.5 in the air of Beijing. This is done using regression models on a dataset of 7684 records of meteorological and weather data from March $1^{\text{st}}$ 2013 to February $28^{\text{th}}$ 2017. The 15 recorded input features are the following:
\begin{itemize}
    \item time [year, month, day, hour],
    \item SO2, NO2, CO, O3, concentrations [\si{\micro g/m^3}],
    \item temperature and dew point temperature [\si{C^\degree}],
    \item pressure [\si{h \pascal}],
    \item rain precipitation [\si{m m}],
    \item wind direction [cardinal] and speed [\si{m/s}],
    \item id of the air-quality. monitoring site.
\end{itemize}
The recorded output variable is the corresponding PM2.5 concentration [\si{\micro g/m^3}].

This paper is organized as follows: features processing, selection and extraction will be discussed in sections \ref{Features_processing}, \ref{Features_selection} and \ref{Features_extraction}, the error estimation and the models implementations are presented in section \ref{Error_estimation} and \ref{Models_implementations} before concluding in section \ref{Conclusion}.

\section{Features processing}
\label{Features_processing}

The time feature recorded in the year, month, day, hour format is converted in seconds in the variable \texttt{time} with $\texttt{time}=0$ corresponding to the first record. This format is more usable but however does not express the cyclic relations in time, namely the rotation of the earth around the sun and the rotation of the earth around itself (see Figure \ref{fig:earth_revolution}). To do so 4 extra variables are created: \texttt{syear} and \texttt{cyear}, encoding the progress of the earth rotation around the sun, and \texttt{sday} and \texttt{cday}, encoding the progress of the earth rotation around itself. The values are computed as follows:
\begin{align*}
    \texttt{syear} &= \sin\left(2 \pi \dfrac{\texttt{time}}{365\cdot24 \cdot 60 \cdot 60}\right),\\
    \texttt{cyear} &= \cos\left(2 \pi \dfrac{\texttt{time}}{365\cdot24 \cdot 60 \cdot 60}\right),\\
    \texttt{sday} &= \sin\left(2 \pi \dfrac{\texttt{time}}{24 \cdot 60 \cdot 60}\right),\\
    \texttt{sday} &= \cos\left(2 \pi \dfrac{\texttt{time}}{24 \cdot 60 \cdot 60}\right).
\end{align*}

\begin{figure}[H]
    \centering
    \input{img/earth_revolution.tex}
    \caption{Earth revolution around the sun}
    \label{fig:earth_revolution}
\end{figure}

The wind direction recorded as cardinals directions are also translated in a format expressing the cyclic relation, \texttt{swd} and \texttt{cwd}, respectively the sine and cosine of the angle of the cardinal direction on a wind rose (see Figure \ref{fig:wind_rose}).

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.3]{img/compass.pdf}
    \caption{Wind rose}
    \label{fig:wind_rose}
\end{figure}

The complete set of inputs is thus composed of 17 features.
The set of inputs is then normalized using a standard normalization method.

\section{Features selection}
\label{Features_selection}

Features selection is a technique to drop some less useful inputs. The intrinsic principle aims to maximize the relevance (relation between input and output) and minimize the redundancy (relation between input and input).

It is important to consider the newly created features (cos/sin) in the previous paragraph as a pair of inputs resulting from one unique feature, it is thus not advised to remove one of them  (the sine or cosine) even if they present a high dependency.

\subsection{Correlation}

Figure~\ref{fig:corr} presents the correlation between the inputs and the output (absolute value). Since the correlation only detects linear relations between variables, a zero correlation between an input and the output is not sufficient to drop this input.

\begin{figure}[H]
    \centering
    \input{img/correlation.tex}
    \caption{Correlation matrix between the inputs and the output}
    \label{fig:corr}
\end{figure}

\subsection{Mutual information}

Figure~\ref{fig:MI} shows the mutual information between the inputs and the output (absolute value). This method is able to detect any dependency between variables and is thus particularly purposeful to select the right features.

\begin{figure}[H]
    \centering
    \input{img/MI.tex}
    \caption{Mutual Information matrix between the inputs and the output}
    \label{fig:MI}
\end{figure}

\subsection{Redundancy and relevance}

Based on the correlation and MI obtained above, one can drop the features in Table~\ref{tab:relevance} because they have few relevance with the output.

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{cc}
\hline
Feature &  MI with output  \\ \hline
\texttt{station} &  0.074  \\
\texttt{rain}    &  0.079  \\
\texttt{swd}  &  0.098   \\
\texttt{cwd}  &  0.1   \\
\texttt{sday}  &  0.11   \\
\texttt{cday}  &  0.11   \\
\texttt{wspm}  &  0.18   \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Non relevant features}
\label{tab:relevance}
\end{table}

Additionally, it is possible to remove the temperature (see Table~\ref{tab:temp}), the pressure (see Table~\ref{tab:pres}) and the time (see Table~\ref{tab:time}) since they are redundant with other inputs.

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{ccc}
\hline
Feature &  MI & Correlation \\ \hline
 \texttt{dewp}  & 0.57  & 0.81  \\
 \texttt{cyear}  & 0.79  &  0.9 \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Relations between some features and a redundant feature: Temperature}
\label{tab:temp}
\end{table}

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{ccc}
\hline
Feature &  MI & Correlation \\ \hline
 \texttt{dewp}  & 0.54  & 0.74  \\
 \texttt{cyear}  & 0.79  &  0.82 \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Relations between some features and a redundant feature: Pressure}
\label{tab:pres}
\end{table}

It is worth noting that the dependencies between the time and the other features is non linear, such that a small correlation is computed.

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{ccc}
\hline
Feature &  MI & Correlation \\ \hline
  \texttt{syear}  & 0.99  & 0.1  \\
 \texttt{cyear}  & 0.99  &  0.17 \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Relations between some features and a redundant feature: Time}
\label{tab:time}
\end{table}

Finally, the set resulting of features selection contains 7 features.

\section{Features extraction}
\label{Features_extraction}

Features extraction consists to transform the input into other features such that this new smaller set of features almost fully describes the content of the inputs.

\subsection{Principal Component Analysis}

The Principal Component Analysis method uses an orthogonal transformation to convert a set of inputs into a set of linearly uncorrelated variables called principal components~\cite{pca}.

Figure~\ref{fig:error_PCA} depicts the error for several numbers of principal components, ranging from 1 to 17 (the full number of features). One can conclude that the most important correlation of the data are combined in 3 principal components, lowering the error to \SI{55.5}{\micro g/m^3}. Adding more components does not lead to better results considering that having only 3 input features brings very efficient computations.

However, it is important to note that this method is linear and thus drops important variables if they are non-linearly dependent. This problem depends upon the linear characteristics of the input features. It is consequent for the data analysed here and hence leads to bad results as discussed thereafter.

\begin{figure}[H]
    \centering
    \input{img/error_PCA.tex}
    \caption{Bootstrap 632 error for different numbers of principal components}
    \label{fig:error_PCA}
\end{figure}

\section{Error estimation}
\label{Error_estimation}

The errors of the following models are estimated thanks to the Bootstrap 632 method gently provided in the MLxtend package \cite{raschkas_2018_mlxtend}. This highly efficient method is characterized by a low bias and a low variance, which makes it particularly purposeful for model validation (compared to the simpler K-fold validation for instance). If not mentioned in the following sections, the number of splits in the Bootstrap method is set to 10 due to the limited available computation power in classical computers. This small number only influences the randomness of the bootstrap error but not the mean value (most important part).

Figure~\ref{fig:error_analysis_bootstrap} shows the error distribution of the linear regression model with the selected features, for 1 and 10 splits. The error is very narrow for 10 splits ($\sigma = \SI{0.255}{\micro g/m^3}$) compared to 1 split ($\sigma = \SI{0.762}{\micro g/m^3}$), computing the error with 10 splits is thus cogent.

\begin{figure}[H]
    \centering
    \input{img/error_analysis_bootstrap.tex}
    \caption{Bootstrap 632 error distribution for different numbers of splits}
    \label{fig:error_analysis_bootstrap}
\end{figure}

\section{Models implementations}
\label{Models_implementations}

Seven models have been trained and compared in order to bring the most powerful one for the estimation of the secret dataset. These models are tested with the bootstrap 632 error on the three previously detailed input sets: the full features, the selected features and the PCA features.

Let the dataset be $D = { (x_1 , y_1 ),(x_2 , y_2 ),\ldots,(x_N , y_N ) }$ where  $x_i$ is the input value and $y_i$ is the target value, the goal of a regression model is to estimate the function $y = f(x)$.

\subsection{Linear regression}

The linear regression estimates the function $y = f(x)$ by $\hat{f}(x) = w^Tx$. The weight vector $w$ is computed by minimizing the mean squared error on the training data: $$\displaystyle \min_{\mathbf{w}} ||\mathbf{y}_{\text{train}} - \mathbf{w}^T\mathbf{X}_{\text{train}}||_2^2.$$

The error for the three implemented sets is given in Table~\ref{tab:linreg}. One can see that the 7 selected features give almost the same error as for the full input set, which validates the features selection (at least for linear models). However, the PCA method, although very fast, is not convincing because it has few features.

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{cc}
\hline
 Features &  Error [\si{\micro g/m^3}] \\ \hline
 Full  & 44.35 \\
 Selected  & 44.50 \\
 PCA  & 54.10 \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Bootstrap 632 error for the Linear regression model}
\label{tab:linreg}
\end{table}

\subsection{Ridge regression}

The ridge regression is similar to the linear regression with a shrinkage penalty on $\mathbf{w}$ ($L_2$ regularization): $$\displaystyle \min_{\mathbf{w}} ||\mathbf{y}_{\text{train}} - \mathbf{w}^T\mathbf{X}_{\text{train}}||_2^2 + \lambda ||\mathbf{w}||_2^2.$$
The parameter $\lambda$ controls the relative impact of the two terms. Increasing $\lambda$ decreases the variance while increasing the bias.

As shown in Figure~\ref{fig:RR}, the ridge regression performs better with the \textit{selected features} and \textit{full features} than with the \textit{PCA features}. The performance is almost constant for $\lambda \in [10^{-2},10^2]$ and decreases after when the model starts to suffer from underfitting. The best result is achieved with the \textit{full features} and $\lambda = 0.61$, the error is \SI{44.15}{\micro g/m^3}.

\begin{figure}[H]
    \centering
    \input{img/RR.tex}
    \caption{Bootstrap 632 error for the Ridge regression model}
    \label{fig:RR}
\end{figure}

\subsection{Lasso}

The Lasso regression is similar to the ridge regression with a shrinkage penalty on $\mathbf{w}$ ($L_1$ regularization): $$\displaystyle \min_{\mathbf{w}} ||\mathbf{y}_{\text{train}} - \mathbf{w}^T\mathbf{X}_{\text{train}}||_2^2 + \lambda \sum_{i=1}^N |\mathbf{w_i}|.$$

As shown in Figure~\ref{fig:Lasso}, the lasso regression performances are almost constant until $\lambda = 1$ and start to decrease after because of underfitting. Similarly to the ridge regression the Lasso performs better with the \textit{full} and \textit{selected features}. The best result is achieved with the \textit{full features} and $\lambda = 0.01$, the error is \SI{44.06}{\micro g/m^3}.

\begin{figure}[H]
    \centering
    \input{img/Lasso.tex}
    \caption{Bootstrap 632 error for the Lasso model}
    \label{fig:Lasso}
\end{figure}

\begin{figure}[H]
    \centering
    \input{img/Lasso_weights.tex}
    \caption{Weights of the Lasso model ($\lambda = 0.1$)}
    \label{fig:Lasso_weights}
\end{figure}

The weights for $\lambda = 0.1$ are given in Figure~\ref{fig:Lasso_weights}. As found in the features selection, the output is very sensitive to three features, and more or less 7 features completely describe the input space. Some coefficients end up being set to almost zero, making the model easier to interpret.

As both the Ridge regression and Lasso models are better with a small regularization coefficient, they are redundant with the linear regression. This can be explained by the high number of samples used for the training, which are sufficient to prevent overfitting in the linear regression model. These two additional models cannot exploit their advantage of reducing the linear regression overfitting.

\subsection{K-Nearest Neighbour}

Figure~\ref{fig:KNN} presents the error for the KNN model with respect to the number of neighbours $K$ (hyperparameter). For this model, using the full input set should dramatically increase the error since in a high dimensional space, the $K$ nearest neighbours (with the Euclidian norm) are greatly influenced by useless dimensions and thus lead to select wrong neighbours. As expected, the model with features selection outperforms the others, reaching an error of \SI{38.51}{\micro g/m^3} for $K = 4$.

\begin{figure}[H]
    \centering
    \input{img/KNN.tex}
    \caption{Bootstrap 632 error for the KNN model}
    \label{fig:KNN}
\end{figure}

\subsection{Regression tree}

A regression tree partitions the input space in smaller regions recursively until reaching a leaf where a simple model is fitted to. Each node represents a binary questions about one or several feature(s) that will divide the space in two.  For a classic regression tree, the model at each leaf is a constant estimate: the average of the target value of the training data that belongs to this leaf. The binary questions on the nodes are chosen such that the information gain is maximized. Mechanisms such as pruning are necessary to avoid overfitting.

As shown in Figure~\ref{fig:tree}, the performance obtained by the \textit{full} and \textit{selected features} are very close and outperform the ones obtained with the \textit{PCA features}. The performances increase as the maximal depth of the tree increases until reaching a maximum at 7 for the \textit{PCA features}, then it starts to overfit since there are too few features in PCA. The error obtained with the \textit{full} and \textit{selected features} stops decreasing and starts to level out at a depth of around 10. It has to be noted than the trees do not grow further than a depth of 33, explaining the absence of overfitting as the allowed maximal depth continues to increase. The best result is achieved with the \textit{selected features} and a depth of 11, the error is \SI{40.23}{\micro g/m^3}.

\begin{figure}[H]
    \centering
    \input{img/tree.tex}
    \caption{Bootstrap 632 error for the regression tree model}
    \label{fig:tree}
\end{figure}

\subsection{Bootstrap aggregating trees}

The core idea of an ensemble method is to train weak regressors (or classifiers) and combine them to construct a regressor (or classifier) more powerful than any of the individual ones.

A simple ensemble method is the so-called Bootstrap aggregating or Bagging. Let $D = { (x_1 , y_1 ),(x_2 , y_2 ),\ldots,(x_N , y_N ) }$ be the training data, the principle of the Bagging method is detailed below.

Iteratively, for $b = 1,\ldots,B$, do the following:
\begin{itemize}
    \item sample training examples, with replacement, $N'$ times from $D$ to create $D_b (N' \leq N)$,
    \item use this bootstrap sample $D_b$ to estimate the regression (or classification) function $f_b$.
\end{itemize}
The bagging estimate is $$f_{\text{bag}}(x) = \frac{1}{B} \sum_{b=1}^B f_b(x).$$
As shown in Figure~\ref{fig:boost_tree}, the \textit{full} and \textit{selected features} are very close and outperform the ones obtained with the \textit{PCA features}. The error decreases as the maximum depth increases until starting to plateau at a depth of 12 with no overfitting. As for the regression tree, this is due to the fact that the trees do not grow deeper than 33. The best result is achieved with the \textit{selected features} and a depth of 19, the error obtained in this case is \SI{31.42}{\micro g/m^3}.

\begin{figure}[H]
    \centering
    \input{img/boost_tree.tex}
    \caption{Bootstrap 632 error for the Bagging trees model}
    \label{fig:boost_tree}
\end{figure}

\subsection{Multilayer Perceptron}

A multilayer perceptron takes as inputs the features (17 for the full set and 7 for the selected/PCA set) and propagates these information though a deep neural network to output the final PM2.5 estimation. At each layer, the data are subject to a batch normalisation followed by a ReLU activation function.

Considering the high number of parameters which are subject to optimization (number of hidden layers, neurons per hidden layer, epochs, learning rate, batch size), it might seem interesting to use an optimization algorithm such as a greedy search or a genetic search. However, it appears after simulation that finding this optimum is difficult since the performances are better when the network is very large, in such a way that overfitting is only noticed for big neural networks which are impossible to train in a limited time. The following paragraphs will thus be dedicated to analysing the error for several numbers of neurons per layer, epochs per training period, and hidden layers.

The results presented in Figure~\ref{fig:MLP} depict the error variation for different numbers of neurons (hyperparameter) in each of the 8 hidden layers, ranging from 1 to 50. The learning step is done by feeding the network with the inputs/output pairs, more precisely with 50 epochs and a batch size of 128. It can be seen that the neural network performs the best with the full features as inputs since the weights of the network are precisely adapted to select the information in each features, even the smallest one. On the other hand, deep neural networks do not aim at dropping some features, which is proven by the poor results from features selection and features extraction.

\begin{figure}[H]
    \centering
    \input{img/MLP.tex}
    \caption{Bootstrap 632 error for the MLP model with different numbers of neurons per hidden layer}
    \label{fig:MLP}
\end{figure}

One can also analyse the error for different training periods, characterised by a varying number of epochs. Figure~\ref{fig:MLP_epochs} shows this error with a number of epochs ranging from 1 to 80. The error is stabilized after around 50 epochs for all inputs subsets since all the neural networks weights do not vary anymore. The error does not increase for additional epochs, leading to the conclusion that the network configuration is too small to bring overfitting due to long training periods\footnote{In case of possible overfitting, the early stop approach consists to stop the training when the validation error reaches a minimum.}.

\begin{figure}[H]
    \centering
    \input{img/MLP_epochs.tex}
    \caption{Bootstrap 632 error for the MLP model with different numbers of epochs per training period}
    \label{fig:MLP_epochs}
\end{figure}

The next step thus consists to increase the number of hidden layers and reduce the error as far as possible. Figure~\ref{fig:MLP_layers} presents this error for the full set, 50 neurons per hidden layer and a number of epochs proportional to the number of layers since larger networks require a longer training period. One can see a minimum error for 18 hidden layers but no overfitting appears as it should be expected for such large networks. Hence, overfitting should arise for larger networks.

\begin{figure}[H]
    \centering
    \input{img/MLP_layers.tex}
    \caption{Bootstrap 632 error for the MLP model with different numbers of hidden layers}
    \label{fig:MLP_layers}
\end{figure}

Finally, a huge neural network of 20 hidden layers and 100 neurons has been trained with 300 epochs. It gives a bootstrap 632 error of \SI{34.12}{\micro g/m^3}, suggesting the start of a small overfitting.

It can be noted that the PyTorch library has been used, leading the authors to implement an estimator in order to match the specifications of the bootstrap 632 method \cite{NEURIPS2019_9015}.

\section{Conclusion}
\label{Conclusion}

To conclude, the lowest error for the analysed models are summarized in Table~\ref{tab:sum} (the errors are expressed in \si{\micro g/m^3}).

\begin{table}[H]
\setlength\abovecaptionskip{-2\baselineskip}
\centering
\begin{tabular}{lccc}
\hline
 Model &  Error (Full) &  Error (Selected) &  Error (PCA)\\ \hline
 Linear regression & 44.35 & 44.50 & 54.10 \\
 Ridge regression & 44.15 & 44.38 & 53.91  \\
 Lasso & 44.06 & 44.07 & 53.92 \\
 KNN & 44.12 & 38.51 & 48.58 \\
 Regression tree & 40.39 & 40.23 & 51.48 \\
 Boot. agg. trees & 31.60 & 31.42 & 41.78 \\
 MLP & 31.94 & 37.52 & 46.16 \\ \hline
\end{tabular}
\vspace*{3mm}
\caption{Summarized error for all the models}
\label{tab:sum}
\end{table}

\subsection{Final model selection}

The bootstrap aggregating trees method with the features selection is chosen as the final method since it has the lowest error. This model is used to predict the output of the secret set, \texttt{Y2}. The bootstrap 632 method aims to estimate the error on a set belonging to the entire possible space. Hence, the expected RMS error on \texttt{Y2} is the bootstrap 632 error on \texttt{Y1}: \SI{31.42}{\micro g/m^3}.

Finally, combining the previous model predictions by means of ensemble methods or a voting classifier are good perspectives for a future work in the prediction improvements and the quest for the lowest error.

\printbibliography

\end{document}