\section {Semilattice Representations, Size, Product and Height}
\setlength{\parindent}{0pt}
(Prepared by Shivam Goyal)

\vspace{0.3cm}

\subsection{Semilattice Representation}
\begin{itemize}
    \item \textbf{Set of values $V$ and a meet operator \^{}}
    \item \textbf{Set of values $V$ and a partial order $\leq$}
    \item \textbf{A semilattice diagram} : which is a directed acyclic graph with nodes and edges. It can have potentially multiple Top and Bottom nodes, but they may not necessarily exist for eg in a semilattice with $V$ as set of integers and $max$ as the meet operator
\end{itemize}

\subsection{Semilattice Size}
For some DFA problems, size of the semilattice i.e. the number of elements in the set of values $V$ can become very large. For eg. it could be exponential in number of definitions for a program, if we consider the set of definitions as a value for the semilattice. As the set of values $V$ would be the power set of the set of definitions in the program for some DFA like available expressions.
\subsection{Product of Semilattices}
\begin{itemize}
    \item One idea considering large semilattices is to represent a larger semilattice using product of smaller semilattices.
    \item Consider 2 semilattices with sets of values as $V1$ and $V2$ then we can define a new semilattice as their product.This would contain values in the set $V1$ x $V2$.
    \item The partial order for the product semilattice would be obtained by applying the respective partial order relations on both the elements of the tuple, i.e. for $\leq$ (the new partial order):
    \par
    ($x_{1}$,$x_{2}$) $\leq$ ($y_{1}$,$y_{2}$) iff $x_{1}$ $\leq_{1}$ $y_{1}$ and $x_{2}$ $\leq_{2}$ $y_{2}$ where $\leq_{1}$ and $\leq_{2}$ are the respective partial orders of the smaller semilattices.
    \item Similar thing can be done in case of the meet operator with the respective meet operators being applied on each element of the tuple, i.e. for \^{} (the new meet operator):
    \par
    ($x_{1}$,$x_{2}$) \^{} ($y_{1}$,$y_{2}$) $=$ ($x_{1}$ \^{}$_{1}$ $y_{1}$ , $x_{2}$ \^{}$_{2}$ $y_{2}$), where \^{}$_{1}$ and \^{}$_{2}$ are the respective meet operators of the smaller semilattices.
\end{itemize}


\documentclass{discussion}

\usepackage{framed}
\usepackage[position=b]{subcaption}
\DeclareMathOperator{\Parents}{Parents}
\DeclareMathOperator{\NonDesc}{NonDesc}
\newcommand{\G}{\mathcal{G}}
\newcommand{\I}{\mathcal{I}}
\linespread{1.0}
\begin{document}

% Lecture Info
\lecture{8}{EM}{Chansoo Lee}
%\begin{exercise}
%\begin{example}

%%%% INTRODUCTION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand{\D}{\mathcal{D}}
\section{Supervised Naive Bayes}
In supervised learning of Naive Bayes, we get \emph{labeled} training data $\D = \{(\x^{(i)}, y^{(i)})\}_{i=1}^{N}$, where $\x^{(i)} \in \{1,\ldots,M\}^{D}$ and $y^{(i)} \in \{ 1,\ldots, C\}$.
 The estimation (training) problem is to find the maximum log likelihood estimate parameters. The log-likelihood function breaks down into a sum of simple terms, which we can easily differentiate:
% \begin{align}
% 	\sum_{i=1}^{N} \log P(y^{(i)}, \x^{(i)}; \pi, \theta)
% 	&= \sum_{i=1}^{N} \log P({y^{(i)}})P(y^{(i)} | \x^{(i)}) \label{eq:mle} \\
% 	&= \sum_{i=1}^{N} \left( \log \pi_{y^{(i)}} + \sum_{i=1}^{D}\log \theta_{y^{(i)}d\x^{(i)}_d} \right)
% \end{align}
\begin{align}
	\sum_{\x,y \in \mathcal{D}} \log P(y, \x; \pi, \theta)
	&= \sum_{\x,y \in \mathcal{D}} \big( \log P({y}) + \log P(y | \x)\big) \label{eq:mle} \\
	&= \sum_{\x,y \in \mathcal{D}} \left( \log \pi_{y} + \sum_{i=1}^{D}\log \theta_{ydx_d} \right). \label{eq:mle-param}
\end{align}
Because each log term involves only one parameter, when we differentiate the above expression with respect to any model parameter $\pi_c$ or $\theta_{cdm}$, we have a very simple expression, which we can set it to zero and easily solve for.

The closed-form solution shows that \textbf{sufficient statistics} required to solve the above problem are counts: $N_c$, number of examples in category $c$ and $N_{cdm}$, number of examples in category $c$ with $d$-th feature taking value $M$.

\section{Unsupervised Naive Bayes}
In \textbf{unsupervised Naive Bayes}, we get unlabeled training data $\{\x^{(i)}\}_{i=1}^{n}$ where each data is a binary feature vector: $\x^{(i)} \in \{0,1\}^D$. The estimation problem is to find the maximum log likelihood estimate parameters for our \emph{partially observed data}:
\begin{align}
\label{eq:em-mle}
\sum_{\x \in \mathcal{D}} \log P(\x; \pi, \theta)
&= \sum_{\x \in \mathcal{D}} \log \left(\sum_{c=1}^{C} P(\x,y = c)\right) \\
&= \sum_{\x \in \mathcal{D}} \log\left(\sum_{c=1}^{C} \Big(\pi_{c} \prod_{d=1}^{D} \theta_{cdx_d} \Big) \right) \label{eq:em-mle-param}.
\end{align}
Since our probabilstic model does not directly define the marginal distribution over $\x$, we have to marginalize $y$ out from the joint distribution \eqref{eq:em-mle}. As a result, there is now a summation inside the logarithm, which makes derivatives messy. If you want to check, take derivatives with respect to $\pi_1$ of \eqref{eq:mle-param} and \eqref{eq:em-mle-param}.\footnote{An alternative explanation for the hardness of unsupervised setting is that the sufficient statistics to compute the MLE estimates are not provided.
}

The important observation is that we can break this hard problem into two easy problems: (1) If we fix the model parameters, then it is a simple prediction step to compute the conditional distribution of the missing values: $P(y_d = c | \x; \theta, \pi) \propto \pi_{c} \prod_{d=1}^{D} \theta_{c d x_d}$. (2) If we fill in the missing values, it is the supervised learning problem that we know how to solve.

This leads to the EM algorithm, an alternating optimization technique. EM solves the generic question of: \emph{how do we find the MLE parameters given partially observed data in a principled manner that properly accounts for the dependence among random variables in our probabilstic model?}

\section{Soft assignment EM}

\begin{itemize}
    \item Initiliaze the parameters to the values that will break the symmetry, and then repeat:
    \item In the E-step, we compute the distribution of the missing data. We often say that we compute the \emph{expected sufficient statistics}, because we fill in the missing data only probabilistically.
    \item In the M-step, update the model parameters to be the MLE based on new expected sufficient statistics.
\end{itemize}

\subsection{Case Study: Binary Unsupervised Naive Bayes}

Suppose the training set has two data points  $\x^{(1)} = (0, 1)$ and $\x^{(2)} = (1,1)$; for clarity, we note that the dimensions are: $N=2, C=2$, $D = 2$, and $M = 2$. In this case study, our binary features and labels take values $\{0,1\}$ instead of $\{1,2\}$.

\paragraph{Initilization} Suppose our random initialization gives $\pi_1 = 0.7, \theta_{111} = 0.9, \theta_{011} = 0.3, \theta_{121} = 0.6, \theta_{021} = 0.2$.

\paragraph{Iteration 1: E-step} Compute the probability for each assignment of missing values\footnote{Unnoramlized probability is $P(\x,y)$ and the normalized probability is $P(y | \x)$.}.
\[
\begin{tabu}{c|c|c|c}
y^{(1)} & y^{(2)} & \text{unnormalized probability} & \text{noramlized probability} \\
\hline
1 & 1 & (0.7)(0.1)(0.6)(0.7)(0.9)(0.6) = 0.015876 & 0.4773\\
1 & 0 & (0.7)(0.1)(0.6)(0.3)(0.3)(0.2) = 0.000756 & 0.0227\\
0 & 1 & (0.3)(0.7)(0.2)(0.7)(0.9)(0.6) = 0.015876 & 0.4773\\
0 & 0 & (0.3)(0.7)(0.2)(0.3)(0.3)(0.2) = 0.000756 & 0.0227\\
\end{tabu}
\]
The table reads as: the missing $y$ value of the $i$-th example is assigned the value specified in column $i$, with probability in column 4.

\begin{exercise}
	Compute the expected sufficient statistics using the table.
\end{exercise}
\begin{proof}
 Consider $\overline{N_1}$, the \emph{expected} number of examples with class label $y = 1$:% = E_{y^{(1)},y^{(2)}}$
\begin{itemize}
		\item row 1 shows with probability 0.4773 there are two instances with $y = 1$.
		\item row 2 shows with probability 0.0227 there is one instance with $y = 1$.
		\item row 3 shows with probability 0.4773 there is one instance with $y = 1$.
		\item row 4 shows with probability 0.0227 there is zero instance with $y = 1$.
		\item Therefore, \[\overline{N_1} = (0.4773)(2) + (0.0227)(1) + (0.4773)(1) + (0.0227)(0) = 1.4546.\]
		\item Because class labels are binary, \[\overline{N_0} = \overline{N} - \overline{N_1} = 2 - 1.4546 = 0.5454\]
\end{itemize}

 Note that given fully observed data, we would do a simple counting where each example adds either 0 or 1. Each example adds 1 to $N_c$ for a single $c$. But for the EM, each example adds an a \emph{fraction} between 0 and 1, such that it \emph{distributes} the count 1 across all $N_c$'s.

 Consider $\overline{N_{111}}$, the \emph{expected} number of examples with class label $y=1$ and first feature taking value 1.
\begin{itemize}
		\item row 1 shows with probability 0.4773, our train set is $((0,1), 1)$ and $((1,1), 1)$. So there is one example with $y=1$ and $x_1 = 1$.
		\item row 2 shows with probability 0.0227, our train set is $((0,1), 1)$ and $((1,1), 0)$. So there is zero example with $y=1$ and $x_1 = 1$.
		\item row 3 shows with probability 0.4773, our train set is $((0,1), 0)$ and $((1,1), 1)$. So there is one example with $y=1$ and $x_1 = 1$.
		\item row 4 shows with probability 0.0227, our train set is $((0,1), 0)$ and $((1,1), 0)$. So there is zero example with $y=1$ and $x_1 = 1$.
		\item Therefore, \[\overline{N_{111}} = (0.4773)(1) + (0.0227)(0)+ (0.4773)(1) + (0.0227)(0) = 0.9546.\]
		\item Because features are binary, $\overline{N_{110}} = \overline{N_{1}} - \overline{N_{111}} = 1.4546 - 0.9546 = 0.5$
\end{itemize}
Similarly, we can compute the rest of the parameters. For example,
\[\overline{N_{121}} = (0.4773)2 + (0.0227) + (0.4773) = 1.4546 \qedhere\]
\end{proof}

\paragraph{Iteration 1: M-step} It's the same MLE parameter computation as in supervised Naive Bayes, except we use \emph{expected} counts instead of counts.

\[\pi_1 =\frac{\overline{N_1}}{N} = 1.4546 / 2 = 0.7273.\]

\[\theta_{111} = \frac{\overline{N_{111}}}{\overline{N_{1}}} = 0.6563.\]

\[\theta_{121} = \frac{\overline{N_{121}}}{\overline{N_{1}}} = 1.\]

\paragraph{Efficient E-step}

The presented implementation of E-step takes exponential time $\Omega(N^C)$, because it computes the full joint distribution over $N$ labels. Indeed, you might have noticed a bit of redundancy when we computed the expected counts.

The trick is that we can compute the distribution of the unobserved variables indepnedently, making it $O(NC)$ computation. Mathematically,
\begin{align*}
\overline{N_1} &= P(y^{(1)} = 1, y^{(2)} = 1) + P(y^{(1)} = 1, y^{(2)} = 1) + P(y^{(1)} = 1, y^{(2)} = 0) + P(y^{(1)} = 0, y^{(2)} = 1) \\
&= (P(y^{(1)} = 1, y^{(2)} = 1) + P(y^{(1)} = 1, y^{(2)} = 0)) + (P(y^{(1)} = 1, y^{(2)} = 1) + P(y^{(1)} = 0, y^{(2)} = 1)) \\
&= P(y^{(1)} = 1) + P(y^{(2)} = 1)
\end{align*}
Then we compute the probability distributions for missing values, independently for each example:
\[P(y^{(1)} = 1) = \frac{(0.7)(0.1)(0.6)}{(0.7)(0.1)(0.6) + (0.3)(0.7)(0.2)} = 0.5 \]
\[P(y^{(2)} = 1) = \frac{(0.7)(0.9)(0.6)}{(0.7)(0.9)(0.6) + (0.3)(0.3)(0.2)} = 0.9546 \]

It is also easy to note that
\[\overline{N_0} = 2 -  \overline{N_1} = P(y^{(1)} = 0) + P(y^{(2)} = 0).
\]

This gives a very intuitive explanation of the EM algorithm on unsupervised Naive Bayes. Given labeled (fully observed) data, we replace the probabilities with indicator functions; each example contributes one count to either $N_{1}$ or $N_0$. Given unlabeled (partially observed) data, each instance \emph{distributes} a total of one count over $N_1$ and $N_0$.

\begin{exercise} Numerically work out the second iteration of EM, using the efficient E-step.
\end{exercise}

\subsection{Hard assignment EM}
Sometimes it is difficult (due to mathematical complexity or limited computation time) to handle fractional contribution of each data point. In such applications, we can make E-step assign the missing values with the highest probability, breaking ties arbitrarily. M-step becomes the MLE given fully observed data.

\begin{exercise}
	Show that $k$-means is equivalent to the hard-assignment EM for GMM clustering with fixed identiyy covariance matrices.
\end{exercise}

\begin{exercise}
What are the differences between soft and hard assignment EM?
\end{exercise}
	The hard-assignment EM explores the combinatorial space of missing variable assignments. The soft-assignment EM, on the other hand, explores the continuous space.

	In clustering, the hard assignment EM tends to amplify the contrast among classes, while soft assignment EM attempts to model mixed-class memberships.

\subsection{Random assignment EM}

In E-step, we compute the distribution over missing values and sample accordingly, instead of computing the expected statistics. Use the random samples to fill in the missing data. M-step becomes the MLE given fully observed data.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Coding Concepts}
\label{coding}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Sorting Algorithms}
\label{coding:sorts}

\begin{table}[H]
\centering
\begingroup
\renewcommand*{\arraystretch}{1}
\input{tables/sorting_algos.tex}
\endgroup
\caption{
A collection of sorting algorithms with time complexities.
}
\label{tab:sorting_table}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Search Algorithms}
\label{coding:searchs}

\begin{table}[H]
\centering
\begingroup
\renewcommand*{\arraystretch}{1}
\input{tables/search_algos.tex}
\endgroup
\caption{
A collection of search algorithms with time complexities.
}
\label{tab:search_table}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Breadth First Search (BFS)}
\label{coding:other:bfs}
% TODO

\subsubsection{Dijkstra's Algorithm}
\label{coding:other:bfs:dijkstra}
% TODO
% https://youtu.be/GazC3A4OQTE

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Depth First Search (DFS)}
\label{coding:other:dfs}
% TODO

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Other}
\label{coding:other}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Linked Lists}
\label{coding:other:lls}
% TODO

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Inverting a Binary Tree}
\label{coding:other:binary_tree_inversion}
% TODO


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 6.7 Killing vectors of the Schwarzschild spacetime}

\begin{cadabra}
   {t, r, \theta, \varphi}::Coordinate.
   {a,b,c,d,e,f,g,h#}::Indices(values={t, r, \theta, \varphi}, position=independent).

   ;::Symbol.

   \partial{#}::PartialDerivative.

   g_{a b}::Metric.
   g^{a b}::InverseMetric.  # essential when using complete (gab, $g^{a b}$)

   Gamma := \Gamma^{a}_{f g} -> 1/2 g^{a b} (   \partial_{g}{g_{b f}}
                                              + \partial_{f}{g_{b g}}
                                              - \partial_{b}{g_{f g}} ).

   deriv := \xi_{a ; b} -> \partial_{b}{\xi_{a}} - \Gamma^{c}_{a b} \xi_{c}.
   lower := \xi_{a} -> g_{a b} \xi^{b}.

   expr  := \xi_{a ; b} + \xi_{b ; a}.                  # cdb(ex-0607.100,expr)

   substitute   (expr, deriv)                           # cdb(ex-0607.101,expr)
   substitute   (expr, lower)                           # cdb(ex-0607.102,expr)
   substitute   (expr, Gamma)                           # cdb(ex-0607.103,expr)
   distribute   (expr)                                  # cdb(ex-0607.104,expr)
   product_rule (expr)                                  # cdb(ex-0607.105,expr)
   canonicalise (expr)                                  # cdb(ex-0607.106,expr)

   # choose a vector

   # Kvect := {\xi^{t} = 1}.
   # Kvect := {\xi^{\varphi} = 1}.
   Kvect := {\xi^{\theta} = \sin(\varphi), \xi^{\varphi} = \cos(\theta)/\sin(\theta) \cos(\varphi)}.
   # Kvect := {\xi^{\theta} = \cos(\varphi), \xi^{\varphi} = - \cos(\theta)/\sin(\theta) \sin(\varphi)}.
                                                         # cdb(ex-0607.107,Kvect)

   gab := { g_{t t}            = -(1-2*m/r),
            g_{r r}            = 1/(1-(2*m/r)),
            g_{\theta\theta}   = r**2,
            g_{\varphi\varphi} = r**2 \sin(\theta)**2}.  # cdb(ex-0607.108,gab)

   complete   (gab, $g^{a b}$)                           # cdb(ex-0607.109,gab)

   evaluate   (expr, gab+Kvect)                          # cdb(ex-0607.110,expr)
\end{cadabra}

\begin{dgroup*}
   \Dmath*{[\xi^{a}] \hiderel{=} \Cdb*{ex-0607.107}}
   \Dmath*{[g_{ab}] \hiderel{=} \Cdb*{ex-0607.108}}
   \Dmath*{[g_{ab},g^{ab}] \hiderel{=} \Cdb*[\hfill\hskip2.5cm]{ex-0607.109}}
   \Dmath*{\cdb{ex-0607.100} = \Cdb*{ex-0607.101}
                             = \Cdb*{ex-0607.102}
                             = \Cdb*{ex-0607.103}
                             = \Cdb*{ex-0607.104}
                             = \Cdb*{ex-0607.105}
                             = \Cdb*{ex-0607.106}
                             = \Cdb*{ex-0607.110}}
\end{dgroup*}

\end{document}


\section{Comparing the Non Relativistic and Relativistic Results}
    \begin{figure}[h]
    \begin{center}
    \begin{tabular}{cc}
        \includegraphics[width=7.0cm]{f0_hydrogen.eps}
        &
        \includegraphics[width=7.0cm]{f0_uranium.eps}
    \end{tabular}
    \caption{Normal Form Factor for Single Electron Hydrogen and Uranium}
    \label{fig:hydrogen-uranium}
    \end{center}
    \end{figure}
    The relativistic normal form factor in equation~\ref{eq:nff-relativistic}
    can be compared to the non relativistic case by considering the small values
    of $Z$ in which the parameter $\gamma_1 = \sqrt{1-\alpha^2Z^2} \approx 1$.
    In this case equation~\ref{eq:nff-relativistic} becomes:
    \begin{equation} \label{eq:reduction}
        f_0(q) \approx
        2 \frac{\Gamma(2)}{\Gamma(3)}
        \left( \frac{2Z}{a_0} \right)^4
        \left[
            \left( \frac{2Z}{a_0} \right)^2 + q^2
        \right]^{-2}
        =
        \left( \frac{2Z}{a_0} \right)^4
        \left[
            \left( \frac{2Z}{a_0} \right)^2 + q^2
        \right]^{-2}
    \end{equation}
    which is in agreement with equation~\ref{eq:nff-nonrelativistic}.
    In figure~\ref{fig:hydrogen-uranium} we see plots for $f_0(q)$ comparing the non relativistic
    and relativistic results for standard atomic hydrogen and hydrogenic
    uranium.
    We can see that in the case of atomic hydrogen ($Z=1$), there is
    excellent agreement between the two results.
    In the case of hydrogenic uranium ($Z=92$) we can see that there is a
    significant difference over a range of momentum transfers between the
    relativistic and non relativistic results.
    This shows the need for using relativistic Dirac wave functions as opposed to
    the Schr\"odinger wave functions when computing atomic form factors
    especially for heavier atoms.
    In figure~\ref{fig:hydrogen-uranium} it is difficult to make out difference
    between relativistic and non relativistic values for atomic hydrogen, since
    there seems to be quite good agreement between the two. However, if we plot
    the difference between the two results as is done in
    figure~\ref{fig:delta-theory}, we can clear identify the regions of momentum
    transfer where the relativistic theory is better even though the difference
    is of the order of $10^{-6}$ or less. We can see that the maximum
    difference between the two theories is at approximately $q = 0.3 \InvAngstrom$ and
    and it gets smaller at higher values of $q$.
    \begin{figure}[h]
    \begin{center}
        \includegraphics[width=7.0cm]{delta_theory.eps}
        \caption{Difference ($\Delta f_0(q)$) between relativistic and non relativistic theory}
        \label{fig:delta-theory}
    \end{center}
    \end{figure}

\section{Comparison with other theoretical results}
    \begin{figure}[H]
    \begin{center}
    \begin{tabular}{cc}
        \includegraphics[width=7.0cm]{hubbel_papa.eps}
        &
        \includegraphics[width=7.0cm]{delta_hubbell.eps}
    \end{tabular}
        \caption{Normal Form Factor for Hydrogen -- Comparison With Hubbell's Theoretical Results}
        \label{fig:hubbell-comparison}
    \end{center}
    \end{figure}
    We can make a comparison between the results obtained using equation~\ref{eq:nff-relativistic}
    and other theoretical results which have used relativistic wave functions in
    computing $f_0(q)$.
    Tabulations of atomic form factors, incoherent scattering function and
    photon scattering cross sections were published by Hubbel et.  al.~\cite{Hubbell-1975}.
    The tabulations for the normal form factor for hydrogen used non
    relativistic wave functions. Figure~\ref{fig:hubbell-comparison} shows two
    plots comparing the analytic result of equation~\ref{eq:nff-relativistic} with
    Hubbel's tabulated results. The plot on the left shows good agreement, but
    looking at the difference between the two theories on the plot on the right
    shows the differences more clearly. The shape is similar to that shown
    in figure~\ref{fig:delta-theory} but the maximum difference is of the order
    of $10^{-5}$.


\section{Ideal Rigid Body (IRB) Bound}

\begin{frame}{Ideal Rigid Body (IRB) Bound}
\begin{itemize}
    \item Post-hoc model which selects an impulse pair (Normal and Tangential Impulses) from the impulse space
    \item Impulse space is defined by energy allowable impulses given pre-impact energy (no increases in energy due to impact)
    \item Impulse pair is chosen to minimize $\ell_2$ norm of post-impact velocity
\end{itemize}
     \begin{figure}
        \centering
        \includegraphics[scale=0.6]{figures/energyEllipse.jpg}
        \caption{Energy Ellipse\cite{nima2}}
        \label{fig:energyEllipse}
\end{figure}

\end{frame}

\subsection{Implementing the IRB Bound}
\begin{frame}{Implementing the IRB Bound}
    \begin{itemize}
        \item Initial energy, KE $= \frac{1}{2}v_{pre}^T*\textbf{M}*v_{pre}$
        \item Post-impact velocity, $v_{predicted} = v_{pre} + \textbf{M}^{-1} \textbf{J}^T \textbf{P}^T$
            \begin{itemize}
                \item $\textbf{M}$ -- generalized mass matrix
                \item $\textbf{J}$ -- contact Jacobian
                \item $\textbf{P}$ -- impulse vector
            \end{itemize}
        \item Run \textit{fmincon} to find the best pair of impulses using normalized error as cost with the energy ellipse as a primary constraint
        \item Even though this is a flexible post-hoc model, we were seeing large errors from the IRB predicted impulses (similar to Nima's paper)

    \end{itemize}
\end{frame}

%surprisingly high error from normal IRB

%IRB with width/Torque
% found that width could be as high as roughly 20 mm, which is highly      unlikely given the actual dimensions of the ellipse and the material properties.

%angle correlations/other correlations
\subsection{IRB with Torque}
\begin{frame}{IRB with Torque}
One idea Prof. Posa suggested to explain the errors we were seeing (discussed further in a later section) is to remove the single point contact assumption and treat the impact as if it occurs over a region.
    \begin{figure}
        \centering
        \includegraphics[scale=0.6]{figures/IRBTorque.png}
        \caption{IRB Torque Diagram}
        \label{fig:IRBTorque}
\end{figure}
\end{frame}

\begin{frame}{IRB with Torque}
\begin{itemize}
    \item Adds an additional torque ($|\tau| = w_{patch}*\textbf{P}_n$)
    \item Additional constraint ($|w_{patch}|\leq w_{max}$)
    \item The calculation is still $v_{predicted} = v_{pre} + \textbf{M}^{-1} \textbf{J}^T \textbf{P}^T$ however \textbf{J} and \textbf{P} now have a third row:
\end{itemize}

\centering
\[
    \textbf{J} =
        \left [
        \begin{array}{ccc}
             & \textbf{d} & \\
             & \textbf{n} & \\
             0 & 0 & 1 \\
        \end{array}
        \right ],\
            \textbf{P} =
       \left [
        \begin{array}{c}
            \textbf{P}_n \\
            \textbf{P}_t \\
            \tau   \\
        \end{array}
        \right ]
\]

\begin{itemize}
    \item This additional decision variable ($\tau$) should reduce error and lead to more accurate predictions from the IRB Bound

\end{itemize}

\end{frame}

\begin{frame}{IRB with Torque's Effect on Error}
    \begin{figure}
        \centering
        \includegraphics[scale=0.15]{figures/IRBErrorVsMaxWidth.jpg}
        \caption{Average Error Across 1000 Random Trials Vs. Maximum Allowable Width}
        \label{fig:IRBErrorVSMaxWidth}
    \end{figure}
\end{frame}

%TAKEAWAYS FROM PLOT ^^^^^
%Small Patch size has significant effect in lowering error (1mm cuts average error in half)
%Patch size isn't a sole solution however, since to get really low errors, you need unrealistically large widths
%As a good sanity check however, as the max width increases and becomes nearly unconstrained, the error approaches 0

\begin{frame}{IRB Trends}
One of the things we did to try to understand how the IRB Bound worked was plotting different properties against each other.

%\begin{figure}
    %\centering
    %\includegraphics[scale=0.18]{figures/Trends.png}
    %\caption{Torque against IRB moment}
    %\label{fig:MomentsError}
%\end{figure}

\begin{figure}
    \centering
    \begin{minipage}{.5\textwidth}
      \centering
      \includegraphics[width=1\linewidth]{figures/MomentProp2.jpg}
    \captionof{figure}{IRB Moment vs Torque}
      \label{fig:contourWM2}
    \end{minipage}%
    \begin{minipage}{.5\textwidth}
      \centering
      \includegraphics[width=1\linewidth]{figures/ellipseAngleWidth.jpg}
    \captionof{figure}{Optimal Width vs Angle}
      \label{fig:histWM2}
    \end{minipage}
\end{figure}

\end{frame}


\documentclass{article}
\usepackage{graphicx}
\usepackage{color}
\usepackage{amsmath}
\begin{document}
	\title{GST108-Introduction to Quantitative Reasoning: Managing Money}
	\author{Oghenemaro Egbodo}
	\maketitle
	\newpage
	\tableofcontents
	\newpage
	\section{Interest Rate}
	\textcolor{red}{Interest(I)} is the price of money. It is the fee paid for borrowing or investing money.\\
	\textcolor{red}{Principal(P)}
is the amount of money being borrowed or deposited.\\
	\textcolor{red}{Interest rate (R)} is the percentage of the principal that uis paid as a fee over a period of time.\\
	\textcolor{red}{Accumulated Balance(A)} is the total amount to be repaid or the total value of money invested.\\
	\textcolor{red}{Simple Interest Formula I=PRT\\
		Accumulated Balance A= P + I\\
		A = P(1+RT)}
	\newpage
	\section{Compound Interest}
	\textcolor{red}{Compound Interest} is interest paid on the original principal of a deposit or loan and also on the accumulated interest.
	The addition of interest to the principal is called \textit{compounding}.
	Interest can be compounded periodically.
	\textcolor{red}{\begin{equation}
			A=P(1+\frac{R}{n})^{nt}
	\end{equation}}
    \textcolor{red}{\begin{equation}
    		I= A+P
    		\end{equation}}
	A = Accumulated Balance or Amount\\
	P = Principal \\ R = Annual Rate (in decimal)\\ n = Number of compounding periods per year\\
	T = Time\\
	I = Interest
	\newpage
	\section{Continuous Compounding}
	Interest accrued on investment of \$1000 at 8\% for 5 years with various compounding periods.\\
	\includegraphics[width=1.0\linewidth]{bar chart.png}
	\newpage
	\subsection{Continuous Compounding}
	Continuous Compounding is when the principal is constantly earning interest and the number of compounding periods increases without bound.
	\textcolor{red}{\begin{equation}
			A=Pe^{RT}
	\end{equation}}\\
	A  =  Accumulated Balance or Amount\\
	P = Principal
\\
	R = Annual Rate (in decimal)
\\
	T = Time
	\newpage
	\section{Effective Rate}
	When interest is compounded, the annual rate of interest (R) is called the nominal rate.\\
	The effective rate, R$_{e}$ is the simple interest rate that would yield the same amount of interest after 1 year.\\
	When a bank advertises a “7\% annual interest rate compounded daily and yielding 7.25\%,” the nominal interest rate is 7\% and the effective rate is 7.25\%. \\
	\textcolor{red}{\begin{equation}
			R_{e} = [(1+\frac{R}{n})^{n-1}]*100
	\end{equation}}\\
	The effective rate is useful for comparing rates with different compounding periods. \\
	\newpage
    \section{Annuity Plan}
	An annuity plan is an investment plan consisting of equal periodic payments.\\
	\textcolor{red}{\begin{equation}
			A = PMT * \frac{[(1+ \frac{R}{n})^{nt} - 1]}{\frac{R}{n}}
		\end{equation}}\\
		A = Accumulated Balance or Amount\\
		PMT  = regular payment  or deposit
\\
		R = Annual Rate (in decimal)
\\
		n = Number of compounding periods per year
\\
		T = Time
\\
			\newpage
		\section{Loan Payment Formula}
		The \textcolor{red}{principal} is the amount of money owed at any particular time. Interest is charged on the loan principal. To pay off a loan, you must gradually pay down the principal\\
		An \textcolor{red}{installment loan (or amortized loan)}is a loan that is paid off with equal regular payments.
\\
		The \textcolor{red}{loan term} is the time you have to pay back the loan in full.
\\
		\textcolor{red}{\begin{equation}
				PMT = \frac{P*\frac{R}{n}}{[1-(1+\frac{R}{n})^{-nt}]}
		\end{equation}}\\
		P = starting loan principal\\
		PMT  = regular payment  or deposit
\\
		R = Annual Rate (in decimal)
\\
		n = Number of compounding periods per year
\\
		T = Time
		\newpage
		\section{Exercises}
		1. Find the simple interest and Accumulated balance for the following:
		\begin{itemize}
			\item Principal = 10000 naira; APR = 8\%; Time = 4 months
			\item Principal = 20000; APR = 6\%; Time = 3 months
			\item Principal = 60,000 naira; APR = 4\%; Time = 2 years
			\item Principal = 50,000 naira; APR = 7\%; Time = 3 years
		\end{itemize}

		2. A principal of 20,000 naira earns 6\% per year simple interest. How long will it take for the accumulated balance to become 23000 naira?\\
		3.An account with 10000 naira earns interest at annual rate of  8\%. Find the amount in this account after 10 years if the compounding is  a) monthly    b) daily     c) Quarterly   d) weekly\\
		4.How much money should be deposited in a bank account earning an annual interest rate of 8\% compounded quarterly, in order to have 10 million naira at the end of 10 years?
\\
		5.Every six months, Segun puts 10000 naira into an account earning an APR of 10\% compounded bi-annually. How much will be in the account at the end of 15 years\\
		6.Kunle wants to start a small business in 5 years. He will need 20 million naira to start the business. How much should he deposit every month into an account with an APR of 9\% compounded monthly in order to meet this goal?
		\newpage
		\subsection{Exercises contd.}
		7.Imagine you want to retire in 30 years with a pension of 1,000,000 naira from an investment plan. \\
		a)How much money would you have to invest today at an APR of 9\% , compounded daily, in order to have 1,000,000 naira in 30 years? \\
		b)Calculate the total return on investment after 30 years and comment on what will happen to the original investment after 30 years.\\
		c)Calculate the Annual Percentage Yield (APY) on the investment over the 30 year period and comment on what will happen to the original investment every year.\\
		d)How much will the N1, 000, 000 generate in interest each year, if it is invested at an APR of 9\%, compounded daily?
		\newpage
		\subsection{Exercises contd.}
		8.  You want to buy a motorcycle costing 2 million naira. You have two options:
		Option 1 - You can borrow 2 million naira at an APR of 8\% for 1 year and pay it back in monthly payments over the year.\\
		Option 2-   You can save the money you would have made in loan payments during 1 year and purchase the motorcycle.\\
		If you decide to save your money for 1 year:
\\
		a) You will have to deposit the equivalent of 1 month’s loan payment in your savings account at the end of each month, and you will earn 5\% interest on the account, compounded monthly. \\
		b) You will pay 28\% of the interest earned on the savings account in taxes. \\
		c) An annual inflation rate of 7\% will have increased the price of the motorcycle by 7\%.\\
		If you choose the option of saving your money for 1 year, how much money will you have left after you pay the tax and purchase the motorcycle?
		\newpage
		\includegraphics[width=1.0\linewidth]{bruh.png}
	\end{document}


\documentclass[11pt]{article}
\usepackage{geometry}
\geometry{letterpaper}
\usepackage[]{graphicx}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{enumitem}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{multicol}
\usepackage{graphics}

\newcommand{\braket}[2]{\langle #1 | #2 \rangle}

\input{/home/prall/.vim/tex/Qcircuit.tex}

\begin{document}

\section*{Probability calculation and the Hidden Shift Algorithm}

\subsection*{Introduction}

This note concerns the remark on page 7, which concerns classically controlled gates in response to measurements of qubits. The trick goes as follows: say a qubit has a 50-50 chance of being 0 or 1. Then we can postselect a measurement result $x = $ 0 or 1 with a 50-50 chance, select a circuit $V_x$ depending on the measurement output $x$, and project the measured qubit onto $\ket{x}\bra{x}$.

This trick works for the sampling algorithm, but not for the probability algorithm. This can be seen with the following circuit:
\[
\Qcircuit @C=.7em @R=.4em @! {
    \lstick{\ket{0}} & \gate{H} & \meter\\
\lstick{\ket{0}} &  \qw & \gate{X} \cwx &\qw
}
\]

Say we desire to calculate the probability of the second qubit being zero: $P(0) = 1/2$. We postselect a value $x$ giving either $V_0 = H \otimes I$ or $V_1 = H \otimes X$, or compactly $V_x = H \otimes X^x$. Then we calculate:

$$P_x(0) = \frac{ \bra{0^{\otimes 2}} V^\dagger_x (\ket{x}\bra{x} \otimes \ket{0}\bra{0}) V_x \ket{0^{\otimes 2}} }{\bra{0^{\otimes 2}} V^\dagger_x (\ket{x}\bra{x} \otimes I) V_x \ket{0^{\otimes 2}} } = \frac{2^{-1/2} |\bra{0} X^x \ket{0}|^2 }{2^{-1/2}} = 1-x$$

Neither $P_1(0) = 0$ nor $P_0(0) = 1$ are the correct probability, therefore this technique does not work for calculating probabilities. However, if a random $x$ is sampled, and we then sample a result from $P_x(0)$, it will be as if we had sampled from the correct distribution.

\textbf{Why does it always work for the T gate gadget, but not for other situations?}

I believe this subtlety was neglected in the implementation of the Hidden Shift Algorithm (HSA) as detailed in the final appendix. In the following I will demonstrate that this is indeed an issue for HSA in particular, and that it can be resolved without compromising the algorithm's runtime.

In the following I will use the shorthand:
$$f(x, \Pi, y, V_x) = (\bra{0^{\otimes n}} \otimes \bra{A^{\otimes t}}) V^\dagger_x ( \ket{x}\bra{x} \otimes \Pi \otimes \ket{y}\bra{y}) V_x (\ket{0^{\otimes n}} \otimes \ket{A^{\otimes t}}) $$

Here $\Pi$ is a projector into the qubit(s) whose output probability is being measured. $x$ is a postselection measurements appearing in the circuit, for example in the Toffoli gate gadget. $y$ is a postselection on measurements in the $T$-gate gadget only. The paper's discussion lumps $x,y$ into a single variable $y$, but here it is handy to keep them separate. $V_x$ is a gadgetized circuit, where the gadgetization depends on both $x$ and $y$ (keeping the $y$ dependence implicit).

\textbf{Proposition 1.} There exists an HSA $V_x$ and postselections $x$ and $y$ such that:
$$P(q=0) \neq \frac{f(x, \ket{0}\bra{0}_q, y, V_x)}{f(x, I_q, y, V_x)}$$
In other words, the procedure detailed in appendix F can fail to calculate the correct probability.

\textit{Proof} ...

\textbf{Proposition 2.} For any algorithm $V_x$, any postselections $x$ and $y$, and potentially several output qubits $q$ and output string $\tilde q$:
$$P(q=\tilde q) = \frac{\left\langle  f(x, \ket{\tilde q}\bra{\tilde q}_q, y, V_x) \right\rangle_x}{\Big\langle  f(x, I_q, y, V_x) \Big\rangle_x} = \frac{ \sum_x  f(x, \ket{\tilde q}\bra{\tilde q}_q, y, V_x) }{\sum_{x}  f(x, I_q, y, V_x) }$$
In other words, if we average over all postselections $x$ in the numerator and denominator, the result is always correct.

\textit{Proof} ...

\textbf{Proposition 3.} For any algorithm $V_x$, $T$-gate postselection $y$ and output projector $\Pi$, let $\Big\langle  f(x, \Pi, y, V_x) \Big\rangle_x = \mu$. Then the random variable $\alpha = f(\xi, \Pi, y, V_\xi)$, where $\xi$ is a uniformly random distribution of bit strings, obeys $\langle\alpha\rangle = \mu$ and:
$$P( (\alpha - \mu)^2 < \epsilon ) \leq n/(4\epsilon) $$.
In other words, I can efficiently approximate the average value of $f$ via a polynomial-size random subset of postselections $x$.

\textit{Proof} ...

\end{document}


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{ambifunb}
\section*{\hspace*{-1.6cm} ambifunb}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Narrow-band ambiguity function.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[naf,tau,xi] = ambifunb(x)
[naf,tau,xi] = ambifunb(x,tau)
[naf,tau,xi] = ambifunb(x,tau,N)
[naf,tau,xi] = ambifunb(x,tau,N,trace)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty ambifunb} computes the narrow-band ambiguity function of a
        signal, or the cross-ambiguity function between two signals. Its
        definition is given by
\[A_x(\xi,\tau)=\int_{-\infty}^{+\infty} x(s+\tau/2)\ x^*(s-\tau/2)\
e^{-j2\pi \xi s}\ ds.\]

\hspace*{-.5cm}
\begin{tabular*}{14cm}{p{1.5cm} p{8.5cm} c}
Name & Description & Default value\\
\hline
{\ty x} & signal if auto-AF, or {\ty [x1,x2]} if cross-AF ({\ty length(x)=Nx})&\\
{\ty tau} & vector of lag values &{\ty (-Nx/2:Nx/2)}\\
{\ty N} & number of frequency bins &{\ty Nx}\\
{\ty trace} & if non-zero, the progression of the algorithm is shown&{\ty 0}\\
\hline
{\ty naf} & doppler-lag representation, with the doppler bins stored in the rows
and the time-lags stored in the columns&\\
{\ty xi} & vector of doppler values\\\hline
\end{tabular*}
\vspace*{.5cm}

This representation is computed such as its 2D Fourier transform equals the
Wigner-Ville distribution.  When called without output arguments, {\ty
ambifunb} displays the squared modulus of the ambiguity function by means
of {\ty contour}.

The ambiguity function is a measure of the time-frequency correlation of a
signal $x$, i.e. the degree of similarity between $x$ and its translated
versions in the time-frequency plane.
\end{minipage}
\vspace*{1cm}

\newpage

{\bf \large \sf Examples}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Consider a BPSK signal (see {\ty anabpsk}) of 256 points, with a keying
period of 8 points, and analyze it with the narrow-band ambiguity
function\,:
\begin{verbatim}
         sig=anabpsk(256,8);
         ambifunb(sig);
\end{verbatim}
The resulting function presents a high thin peak at the origin of the
ambiguity plane, with small sidelobes around. This means that the
inter-correlation between this signal and a time/frequency-shifted version
of it is nearly zero (the ambiguity in the estimation of its arrival time
and mean-frequency is very small).\\

Here is an other example that checks the correspondance between the WVD and
the narrow-band ambiguity function by means of a 2D Fourier transform\,:
\begin{verbatim}
         N=128; sig=fmlin(N); amb=ambifunb(sig);
         amb=amb([N/2+1:N 1:N/2],:);
         ambi=ifft(amb).';
         tdr=zeros(N); 		% Time-delay representation
         tdr(1:N/2,:)=ambi(N/2:N-1,:);
         tdr(N:-1:N/2+2,:)=ambi(N/2-1:-1:1,:);
         wvd1=real(fft(tdr));

         wvd2=tfrwv(sig);
         diff=max(max(abs(wvd1-wvd2)))
         diff =
                1.5632e-13
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
ambifuwb.
\end{verbatim}
\end{minipage}


\documentclass{article}

\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{marvosym}
\usepackage{dingbat}

\title{The Conjugate Gradient Method}
\date{}

\begin{document}
\maketitle

\section{The Conjugate Gradient}

The Conjugate Gradient (CG) is an iterative method for the solution
for sparse, symmetric and positive-definite linear systems. Remember,
sparse matrices are matrices where most of the coefficients are equal
to zero. The zero coefficients are not explicitly stored in memory and
are not considered in all the operations involving the matrix; this
allows to reduce the memory consumption as well as the complexity of
matrix operations.

The CG method receives in input a starting solution $x=x_0$ and
refines it until the desired accuracy is reached (for example, until
the norm of the residual is smaller than a certain threshold $\|r\|_2
= \|b-Ax\|_2 < \varepsilon$ and/or when the number of iterations
exceeds a maximum value). In its basic form, the CG is described by
the algorithm below:

\begin{algorithmic}[1]
  \REQUIRE a matrix $A$, a right-hand side $b$ and a starting solution $x_0$
  \STATE $d_0 = r_0 = b-Ax_0$
  \WHILE {$\|r_i\|_2 > \varepsilon$ and $i < itmax$}
  \STATE $\alpha_i = \frac{r^T_i r_i}{d^T_iAd_i}$
  \STATE $x_{i+1} = x_i + \alpha_id_i$
  \STATE $r_{i+1} = r_i - \alpha_iAd_i$
  \STATE $\beta_{i+1} = \frac{r^T_{i+1} r_{i+1}}{r^T_i r_i}$
  \STATE $d_{i+1} = r_{i+1} + \beta_{i+1}d_i$
  \ENDWHILE
\end{algorithmic}

The method only requires four basic operations:
\begin{enumerate}
\item matrix-vector product $y = \alpha Ax + \beta y$
\item dot-product $v = x^Ty$
\item vector 2-norm $v = \|x\|_2$
\item vector sum $y = \alpha x + \beta y$
\end{enumerate}
where $y$ and $x$ are dense vectors, $A$ is a sparse matrix, $\alpha$,
$\beta$ and $v$ are scalars.

The objective of this exercise is to parallelize each of these four
operations independently using OpenMP in order to accelerate the
Conjugate Gradient. The mathematical details of the CG method are, in
this context, not important and can be neglected.

\section{Sparse Matrix Format}

Because the zero coefficients of the matrix must not be stored, a
standard 2D array cannot be used for storing a sparse matrix.
In this exercise, sparse matrices are represented in Compressed Sparse
Row (CSR) format that consists of three arrays:

\begin{itemize}
\item \texttt{val}: this array of size \texttt{nz} contains the
  nonzero coefficients of the matrix sorted by rows (all the
  coefficients in the first row, then all those in the second row
  etc.)
\item \texttt{colind}: this array of size \texttt{nz} contains the
  column indices for the coefficients in the \texttt{val} array
\item \texttt{rowptr}: this array of size \texttt{n+1} contains
  pointer to the beginning of each row inside the \texttt{val} and
  \texttt{colind} arrays. For example \texttt{rowptr[3]=4} means that
  the first coefficient of row 3 is the 4th element of array
  \texttt{val} and its column index is equal to
  \texttt{colind[4]}. Therefore, all the coefficients of row
  \texttt{i} are between positions \texttt{rowptr[i]} and
  \texttt{rowptr[i+1]-1}.
\end{itemize}
where \texttt{n} is the size of the matrix and \texttt{nz} is the number
of nonzero coefficients in the matrix.

Here is an example:

\vspace{0.5cm}

\begin{minipage}{0.4\linewidth}
\begin{displaymath}
   A=\left[\begin{array}{ccccc}
   1 & 0 & 3 & 0 & 0\\
   4 & 2 & 0 & 1 & 0\\
   0 & 0 & 7 & 0 & 2\\
   8 & 2 & 5 & 0 & 0\\
   0 & 3 & 0 & 4 & 0\\
  \end{array}\right]
\end{displaymath}
\end{minipage}%
\begin{minipage}{0.6\linewidth}
  \centering
  \includegraphics[width=\textwidth]{csr}
\end{minipage}

\newpage

The sparse matrix-vector product $y = \alpha Ax + \beta y$ is computed
with the following routine: % in Figure~\ref{fig:spmv}.
% \begin{figure}[!h]
  % \centering
\small
\begin{verbatim}
void spmv(int n, int *rowptr, int *colind, double *val,
          double alpha, double *x, double beta, double *y){

  int i, j;

  for(i=0; i<n; i++){
    /* for each row... */
    y[i] = beta*y[i];
    for(j=rowptr[i]; j<rowptr[i+1]; j++){
      /* for each coefficient in the row... */
      y[i] += alpha*val[j]*x[colind[j]];
    }
  }
  return;
}
\end{verbatim}
% \caption{\label{fig:spmv} The sparse matrix-vector product.}
% \end{figure}
\normalsize

Note that each iteration of the outer loop handles one row of the
matrix and each iteration of the inner loop handles one coefficient of
a row.

\section{Package content}
In the \texttt{ConjugateGradient} directory you will find the
following files:
\begin{itemize}
\item \texttt{conjugategradient.c}: this file contains the main
  program that reads a matrix $A$ from a file, generates a right-hand
  side $b$ and computes the solution $x$ of the system $Ax=b$ using
  the Conjugate Gradient method. {\bf This file should not be
    modified}.
\item \texttt{kernels.c}: this file contains the subroutines that
  perform the four basic operations described above. {\bf This is the
    only file that must be modified} as described below.
\item the remaining files contain auxiliary routines and can be safely
  ignored.
\end{itemize}

The code can be compiled with the \texttt{make} command: just type
\texttt{make} inside the \texttt{ConjugateGradient} directory; this
will generate a \texttt{main} program that can be run like this:

\begin{verbatim}
$ ./main matrix_file
\end{verbatim}

where \texttt{matrix\_file} can be \texttt{matrix1.rb},
\texttt{matrix2.rb} or \texttt{matrix3.rb}. When executed, the main
program will read the matrix $A$ from the file, generate the
right-hand side $b$ and find the solution $x$ of the linear system
$Ax=b$ using the CG method. It will print the norm of the residual
$\|r\|_2 = \|b-Ax_i\|_2$ every 10 iterations and, at the end, will
print the total number of iterations done, the residual of the final
solution and the CG execution time.

% The matrices can be found in the
% \texttt{/mnt/n7fs/ens/tp\_abuttari/TP\_SysCo/Matrices\_BE/}
% directory. Before running the program for the first time, just make a
% copy of these files into the \texttt{ConjugateGradient} directory:

% \begin{verbatim}
% $ cp /mnt/n7fs/ens/tp_abuttari/TP_SysCo/Matrices_BE/matrix*.rb .
% \end{verbatim}

\section{Assignment}
\begin{itemize}
\item {\huge \Keyboard} Use OpenMP to parallelize the four subroutines in the
\texttt{kernels.c} file. In case you used the \texttt{omp parallel
  for} construct, consider using different scheduling types.
\item \smallpencil Analyze and compare the performance of the parallel
  code using one or two threads. Report in the \texttt{responses.txt}
  file the number of iterations and the execution time for the three
  example matrices \texttt{matrix1.rb}, \texttt{matrix2.rb} and
  \texttt{matrix3.rb}. Comment on the observed results: did you
  observe any speedup (reduction of the execution time) using 2 or 4
  threads instead of 1? did you observe any difference in the number
  of iterations using 2 or 4 threads instead of 1?  Is the method
  still converging when using 2 or 4 threads? Did you observe any
  difference when using a dynamic scheduling instead of a static one?
  can you explain this difference?
\end{itemize}

\paragraph{Advice.}
\begin{itemize}
\item Note that some operations are easier to parallelize than
  others. The vector sum routine \texttt{axpby} is the easiest, the
  \texttt{dot} and \texttt{norm2} are a bit more difficult and,
  finally, the matrix-vector product \texttt{spmv} is the hardest. It
  is recommended to begin with the easier routines first and to
  validate the correctness of the result each time one routine is
  parallelized.
\item It is reasonable to expect that the number of iterations to
  convergence is slightly different when using 2 (or more) threads
  instead of 1. The difference, however, should be very small (not
  more than 10 iterations for the given test matrices). Therefore, if
  you observe a big difference in the number of iterations or if the
  method does not converge anymore it means that the code was not
  correctly parallelized.
\end{itemize}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\section{sparse grid for $x^2$}
Consider multilevel grid points
$$
T_k:\quad x_i^k = ih_k,\quad h_k = 2^{-k},\quad k=0,1,2,...,J,\quad 0\le i\le 2^k.
$$
Consider linear interpolation on $T_k$:
$$
(I_k v)(x) = \sum_{i=0}^{2^k}v(x_i^k)\varphi_i^k(x).
$$
Let $v(x)=x^2$, it is easy to see that
$$
(I_0v)(x) = x.
$$
For $k\ge 1$,
\[
\begin{split}
(I_k-I_{k-1})v &= \sum_{j=1}^{2^{k-1}}[(I_k-I_{k-1})v]\varphi_{2j-1}^k(x)\\
&=\sum_{j=1}^{2^{k-1}}[v(x_{2j-1}^k)-\frac{1}{2}(v(x_{2j-2}^k)+v(x_{2j}^k))]\varphi_{2j-1}^k(x)\\
&=\sum_{j=1}^{2^{k-1}}\frac{1}{2}h_k^2[2(2j-1)^2-(2j-2)^2-(2j)^2]\varphi_{2j-1}^k(x)\\
&=-\sum_{j=1}^{2^{k-1}}4^{-k}\varphi_{2j-1}^k(x)\\
&=-4^{-k}\underbrace{\varphi_1\circ\cdots\circ\varphi_1(x)}\limits_{\varphi_k}.
\end{split}
\]

\begin{lemma}
	$$
	I_J(x^2) = x-\sum_{k=1}^J4^{-k}\varphi_k(x).
	$$
\end{lemma}
Consequently,
\[
\begin{split}
x^2 &= I_Jv + O(h_J^2) = I_0v+\sum_{k=1}^J(I_k-I_{k-1})v+O(h_J^2)\\
&=x - \sum_{k=1}^J4^{-k}\varphi_k + O(h_J^2)
\end{split}
\]

\section{sparse grid for $x^3$}

Given $(x_i,y_i),i=1,2,3,$ the quadratic interpolation function is
\[
I(x) = y_1\frac{(x-x_2)(x-x_3)}{(x_1-x_2)(x_1-x_3)}+y_2\frac{(x-x_1)(x-x_3)}{(x_2-x_1)(x_2-x_3)}+y_3\frac{(x-x_1)(x-x_2)}{(x_3-x_1)(x_3-x_2)}.
\]
On grid
\[T_k: x_i = i2^{-k},\quad i = 0,...,2^k,
\]
we have the piece quadratic interpolation on each $[x_i,x_{i+1}]$. Define $x_{i+\frac{1}{2}} = \frac{1}{2}(x_i+x_{i+1})$, then we have the basis function $\varphi_i^k,\varphi_{i+\frac{1}{2}}^k,\varphi_{i+1}^k$.

On grid
\[T_{k+1}: x_i = i2^{-(k+1)},\quad i = 0,...,2^{k+1},
\]
we have the basis function $\varphi_i^{k+1},\varphi_{i+\frac{1}{4}}^{k+1},\varphi_{i+\frac{1}{2}}^{k+1}$ on $[x_i,x_{i+\frac{1}{2}}]$ and $\varphi_{i+\frac{1}{2}}^{k+1},\varphi_{i+\frac{3}{4}}^{k+1},\varphi_{i+1}^{k+1}$ on $[x_{i+\frac{1}{2}},x_{i+1}]$. The basis function $\varphi^{k+1}_i$ satisfies
\[
\varphi^{k+1}_i(x_i) = 1,\quad \varphi^{k+1}_i(x_{i-\frac{1}{4}}) = 0,\quad \varphi^{k+1}_i(x_{i+\frac{1}{4}}) = 0,
\]
and $\varphi^{k+1}_i$ is a quadratic function on $[x_{i-\frac{1}{4}},x_{i+\frac{1}{4}}],$and equal to $0$ out of $[x_{i-\frac{1}{4}},x_{i+\frac{1}{4}}].$
Then we have
\[
\begin{split}
I_k(x^3) &= x_i^3\varphi_i^k + x_{i+{\frac{1}{2}}}^3\varphi_{i+\frac{1}{2}}^k + x_{i+1}^3\varphi_{i+1}^k\\
&=x_i^3(\varphi^{k+1}_i + \frac{3}{8}\varphi^{k+1}_{i+\frac{1}{4}}-\frac{1}{8}\varphi^{k+1}_{i+\frac{3}{4}})\\
&+x_{i+\frac{1}{2}}^3(\frac{3}{4}\varphi^{k+1}_{i+\frac{1}{4}}+\varphi^{k+1}_{i+\frac{1}{2}}+\frac{3}{4}\varphi^{k+1}_{i+\frac{3}{4}})\\
&+ x_{i+1}^3(-\frac{1}{8}\varphi_{i+\frac{1}{4}}^{k+1}+\frac{3}{8}\varphi_{i+\frac{3}{4}}^{k+1}+\varphi_{i+1}^{k+1})
\end{split}
\]
and
\[
I_{k+1}(x^3) = x_i^3\varphi_i^{k+1} + x_{i+\frac{1}{4}}^3\varphi_{i+\frac{1}{4}}^{k+1} + x_{i+\frac{1}{2}}^3\varphi_{i+\frac{1}{2}}^{k+1} + x_{i+\frac{3}{4}}^3\varphi_{i+\frac{3}{4}}^{k+1} + x_{i+1}^3\varphi_{i+1}^{k+1}
\]
Then on $[x_i,x_{i+1}]$
\[
\begin{split}
(I_{k+1}-I_k)(x^3) &=(x^{3}_{i+\frac{1}{4}}-\frac{3}{8}x_i^3-\frac{3}{4}x_{i+\frac{1}{2}}^3+\frac{1}{8}x^3_{i+1})\varphi_{i+\frac{1}{4}}^{k+1} +(x_{i+\frac{3}{4}}^3+\frac{1}{8}x_i^3-\frac{3}{4}x_{i+\frac{1}{2}}^3-\frac{3}{8}x^3_{i+1})\varphi_{i+\frac{3}{4}}^{k+1}\\
&=3\times2^{-3k-6}(\varphi_{i+\frac{1}{4}}^{k+1}-\varphi_{i+\frac{3}{4}}^{k+1})
\end{split}
\]
where $\varphi_{i+\frac{1}{4}}^{k+1}-\varphi_{i+\frac{3}{4}}^{k+1}$ is
\begin{center}
	\begin{tikzpicture}[scale = 1]
	\draw[gray, step = 1cm] (0, -1) grid (4, 1);
	\draw[->] (-.5, 0) -- (4.5, 0);
	\draw[->] (0, -2) -- (0, 2);

	\node[anchor = north east] at (0, 0) {$ x_i $};
	\node[anchor = north east] at (1, 0) {$ x_{i+\frac{1}{4}} $};
	\node[anchor = north east] at (2, 0) {$ x_{i+\frac{1}{2}} $};
	\node[anchor = north east] at (3, 0) {$ x_{i+\frac{3}{4}} $};
	\node[anchor = north east] at (4, 0) {$ x_{i+1} $};
	\node[anchor = north east] at (0, 1) {$ 1 $};
	\node[anchor = north east] at (0, -1) {$ -1 $};
	\node[anchor = north] at (4.5, 0) {$ x $};
	\node[anchor = west] at (0, 2) {$ y $};

	\draw[domain = 0:2, smooth, variable=\x, black]
	plot ({\x}, {\x * (2-\x)})
	node[anchor = west] {};
	\draw[domain = 2:4, smooth, variable=\x, black]
	plot ({\x}, {-(4-\x) * (\x-2)})
	node[anchor = west] {};
	\end{tikzpicture}
\end{center}

We sum all the function on $[0,1]$,
\[
(I_{k+1}-I_k)(x^3) = 3\times2^{-3k-6}\sum_{i=0}^{2^k-1}(\varphi_{i+\frac{1}{4}}^{k+1} - \varphi_{i+\frac{3}{4}}^{k+1})
\]

\section{2D sparse grid for $xy$}
Consider the uniform gird $[0,1]^2$, and we define the grid
\[
T_k:(x_i,y_j),\quad x_i = ih,y_j = jh,\quad h = 2^{-k}
\]
$I_k$ is the piece linear interpolation operator which satisfies
\[
(I_k f)(x_i,y_j) = f(x_i,y_j)
\]
We consider the grid $T_k$ and $T_{k-1}$.
\begin{itemize}
	\item The basis functions on $T_{k-1}$ are $\varphi_1^{k-1},\varphi_2^{k-1},\varphi_3^{k-1}$.
	\item  The basis functions on refine grid $T_k$ are $\varphi_i^k,i=1,..,6$.
\end{itemize}
\begin{center}
	\begin{tikzpicture}[scale = 3 ]
	\draw[gray, step = 1cm] (0, 0) grid (2, 2);
	\draw[->] (-.5, 0) -- (2.5, 0);
	\draw[->] (0, -.5) -- (0, 2.5);
	\node[anchor = north west] at (0, 0) {$1, (x_i,y_j) $};
	\node[anchor = north west] at (2.5, 0) {$ x $};
	\node[anchor = north west] at (0, 2.5) {$ y $};
	\node[anchor = north west] at (1,0) {$6,(x_{i+\frac{1}{2}},y_j)$};
	\node[anchor = north west] at (2,0) {$2,(x_{i+1},y_j)$};
	\node[anchor = north west] at (0,1) {$(x_{i},y_{j+\frac{1}{2}})$};
	\node[anchor = north west] at (1,1) {$5,(x_{i+\frac{1}{2}},y_{j+\frac{1}{2}})$};
	\node[anchor = north west] at (2,1) {$4,(x_{i+1},y_{j+\frac{1}{2}})$};
	\node[anchor = north west] at (0,2) {$(x_{i},y_{j+1})$};
	\node[anchor = north west] at (1,2) {$(x_{i+\frac{1}{2}},y_{j+1})$};
	\node[anchor = north west] at (2,2) {$3,(x_{i+1},y_{j+1})$};

	\draw (0,0) -- (0,2);
	\draw (0,2) -- (2,2);
	\draw (2,2) -- (2,0);
	\draw (2,0) -- (0,0);
	\draw (0,0) -- (2,2);
	\draw (0,1) -- (1,2);
	\draw (1,0) -- (2,1);
	\draw (1,0) -- (1,2);
	\draw (0,1) -- (2,1);
	\end{tikzpicture}
\end{center}
\[
\begin{split}
I_{k-1}(xy) &= x_iy_j\varphi^{k-1}_1 + x_{i+1}y_j\varphi^{k-1}_2 + x_{i+1}y_{j+1}\varphi^{k-1}_3\\
&=x_iy_j(\varphi^k_1 + \frac{1}{2}(\varphi^k_5+\varphi^k_6)) + x_{i+1}y_j(\varphi^k_2 + \frac{1}{2}(\varphi^k_4+\varphi^k_6))\\
&+x_{i+1}y_{j+1}(\varphi^k_3+\frac{1}{2}(\varphi^k_4+\varphi^k_5))\\
&=x_iy_j\varphi^k_1 + x_{i+1}y_j\varphi^k_2+x_{i+1}y_{j+1}\varphi^k_3+\frac{1}{2}(x_{i+1}y_j+x_{i+1}y_{j+1})\varphi^k_4\\
&+\frac{1}{2}(x_{i}y_j+x_{i+1}y_{j+1})\varphi^k_5+\frac{1}{2}(x_{i}y_j+x_{i+1}y_{i})\varphi^k_6\\
&=x_iy_j\varphi^k_1 + x_{i+1}y_j\varphi^k_2+x_{i+1}y_{j+1}\varphi^k_3+x_{i+1}y_{j+\frac{1}{2}}\varphi^k_4\\
&+\frac{1}{2}(x_{i}y_j+x_{i+1}y_{j+1})\varphi^k_5+x_{i+\frac{1}{2}}y_j\varphi^k_6\\
I_k(xy) & = x_iy_j\varphi^k_1+x_{i+1}y_j\varphi^k_2+x_{i+1}y_{j+1}\varphi^k_3+x_{i+1}y_{j+\frac{1}{2}}\varphi^k_4 \\
&+x_{i+\frac{1}{2}}y_{j+\frac{1}{2}}\varphi^k_5 + x_{i+\frac{1}{2}}y_j\varphi^k_6
\end{split}
\]

Then,
\[
(I_k - I_{k-1})(xy) = (x_{i+\frac{1}{2}}y_{j+\frac{1}{2}}-\frac{1}{2}(x_{i}y_j+x_{i+1}y_{j+1}))\varphi^k_5 = -\frac{h^2}{4}\varphi_5^k
\]
The error is uniform! which is the basis function on each rectangle $[x_i,x_{i+1}]\times [y_j,y_{j+1}]$ and
\[
\varphi_5^k(x_{i+\frac{1}{2}},y_{j+\frac{1}{2}}) = 1
\]

\section{Spectral approximation properties of ReLU DNN}
Here we will talk about how to use ReLU DNN to approximate polynomials
and then achieve the ``spectral" accuracy for analytical functions.
The main results can be found in \cite{yarotsky2017error},
\cite{wang2018exponential}. Some related works are
\cite{liang2016why,lu2017expressive}

First, we will introduce the following notation
\begin{itemize}
\item $x=(x_1,\ldots,x_d)$.
\item Colon notation for subscript: let $\{x_{m:n}\} = \{x_i:i = m,m+1,...,n\}$ and $\{x_{m_1:n_1,m_2:n_2}\}= \{x_{i,j}:i = m_1,...,n_1,j = m_2,...,n_2\}.$
\item Linear combination: denote $y\in \mathcal L(x_1,...,x_d)$ if
  there exist $\beta_i\in \mathbb{R},i=1,...,d,$ such that $y =
  \beta_0+\beta_1x_1+\cdots+\beta_d x_d$.
\item Linear combination with ReLU activation: denote $\tilde{y}\in
  \tilde{\mathcal L}(x_1,...,x_d)$ if there exists $y\in
  \mathcal{L}(x_1,...,x_d)$ and $\tilde{y} = \mbox{ReLU}(y) =
  \max(y,0).$
\item $\tilde {\mathcal L} =\sigma\circ\mathcal L$
\end{itemize}
\begin{definition}
Given a function $f(x)$, if there exist variables $\{y_{1:L,1:M}\}$ such that
\begin{equation}
y_{1,m}\in \tilde{\mathcal L}(x),\quad y_{l+1,m} \in \tilde{\mathcal L}(x,y_{l,1:M}),\quad f\in\mathcal{L}(x,y_{1:L,1:M}),
\end{equation}\label{def:netclass}
where $m=1,...,M,l=1,...,L-1$, then $f$ is said to be in the neural nets class $\mathcal{F}_{L,M}(\mathbb{R}^d)$, and $\{y_{1:L,1:M}\}$ is called a set of hidden variables of $f$.
\end{definition}
\newpage
%\begin{properties}
\begin{proposition}
A function $f\in \mathcal F_{L,M}(\mathbb R^d)$ can be represented by a ReLU network with depth $L+1$ and width $M+d+1$.
\end{proposition}
%\end{properties}
\begin{proof}
Let $\{y_{1:L}\}$ be the hidden variables of $f$ that satisfies (\ref{def:netclass}), where
$$
f = \alpha_0 + \sum_{i=1}^d \alpha_ix_i+\sum_{l=1}^L\sum_{m=1}^M \beta_{l,m}y_{l,m}.
$$
$$
h\leftarrow (y,ex^T), e=(1,\ldots 1)^T.
$$
Consider the following variables $\{h_{1:L,1:M}\}$:
$$
h_{l,1:M} = y_{l,1:M}, \quad h_{l,M+1:M+d} = x_{1:d}
$$
for $l = 1,...,L,$ and
$$
h_{1,M+d+1} = \alpha_0 + \sum_{i=1}^d\alpha_ix_i,\quad h_{l+1,M+d+1} =h_{l,M+d+1} + \sum_{m=1}^M\beta_{l,m}h_{l,m}
$$
for $l=1,...,L-1$. One can see that $h_{1,m}\in \tilde{\mathcal L}(x),h_{l+1,m}\in \tilde{L}(h_{l,1:M+d+1}),m = 1,...,M+d+1,l = 1,...,L-1,$ and $f\in \mathcal{L}(h_{L,1:M+d+1}),$ which is a representation of a standard neural net.
\end{proof}

%\begin{properties}
\begin{proposition}\label{prop:net class}
(Addition and composition of neural net class $\mathcal F_{L,M}$)
\begin{itemize}
\item[1]
$$
\mathcal F_{L_1,M} + \mathcal{F}_{L_2,M} \subseteq \mathcal{F}_{L_1+L_2,M},
$$
i.e. if $f_1\in \mathcal F_{L_1,M}(\mathbb{R}^d)$ and $f_2\in \mathcal{F}_{L_2,M}(\mathbb{R}^d),$ then $f_1+f_2\in\mathcal{F}_{L_1+L_2,M}(\mathbb{R}^d)$.
\item[2]
$$
\mathcal F_{L_2,M}\circ \mathcal F_{L_1,M+1} \subseteq \mathcal F_{L_1+L_2,M+1}.
$$
i.e. if $f_1(x)\in \mathcal{F}_{L_1,M+1}(\mathbb{R}^d)$ and $f_2(x_0,x)\in \mathcal{F}_{L_2,M}(\mathbb{R}^{d+1}),$ then
$$
f_2(f_1(x),x)\in \mathcal{F}_{L_1+L_2,M+1}(\mathbb{R}^d).
$$
\end{itemize}
\end{proposition}
%\end{properties}

\newpage
\begin{proof}
For the addition property, denote the hidden variables of $f_1$ and $f_2$ as $\{y_{1:L_1,1:M}^{(1)}\}$ and $\{y_{1:L_2,1:M}^{(2)}\}$.
So we have
\[
f_1 = \alpha_0^1 + \sum_{i=1}^d \alpha^1_ix_i+\sum_{l=1}^L\sum_{m=1}^M \beta^1_{l,m}y^{(1)}_{l,m}
\]
\[
f_2 = \alpha_0^2 + \sum_{i=1}^d \alpha^2_ix_i+\sum_{l=1}^L\sum_{m=1}^M \beta^2_{l,m}y^{(2)}_{l,m}
\]
\[
f_1+f_2 = \alpha_0^1 + \alpha_0^2 + \sum_{i=1}^d (\alpha_i^1+\alpha_i^2)x_i + \sum_{l=1}^L\sum_{m=1}^M \beta^1_{l,m}y^{(1)}_{l,m} +\sum_{l=1}^L\sum_{m=1}^M \beta^2_{l,m}y^{(2)}_{l,m}
\]
$$
y^{(1)}_{1,1:M} = \sigma\circ \mathcal{L}(x),y^{(1)}_{2,1:M} = \sigma\circ \mathcal L(x,y^{(1)}_{1,1:M}),\cdots,y^{(1)}_{L_1,1:M}=\sigma\circ \mathcal L(x,y^{(1)}_{L_1-1,1:M}),
$$
$$
y^{(2)}_{1,1:M} = \sigma\circ \mathcal{L}(x) =\sigma(\mathcal L(x)+\bm 0\cdot y^{(1)}_{L_1,1:M}) = \sigma\circ \mathcal{L}(x,y^{(1)}_{L_1,1:M})
$$
$$
y^{(2)}_{2,1:M} = \sigma\circ \mathcal L(x,y^{(2)}_{1,1:M}),\cdots,y^{(2)}_{L_2,1:M}=\sigma\circ \mathcal L(x,y^{(2)}_{L_2-1,1:M}),
$$
so $f_1 + f_2 \in \mathcal{F}_{L_1+L_2,M}$
Let
$$
y_{1:L_1,1:M} = y_{1:L_1,1:M}^{(1)},\quad y_{L_1+1:L_1+L_2,1:M} = y_{1:L_2,1:M}^{(2)}.
$$
By definition, $\{y_{1:L_1+L_2,1:M}\}$ is a set of hidden variables of $f_1+f_2$. Thus $f_1+f_2\in \mathcal F_{L_1+L_2,M}.$

For the composition property, let the hidden variables of $f_1$ and $f_1$ as $\{y_{1:L_1,1:M+1}^{(1)}\}$ and $\{y_{1:L_2,1:M}^{(2)}\}$. Let
$$
y_{1:L_1,1:M+1} = y_{1:L_1,1:M+1}^{(1)},\quad y_{L_1+1:L_1+L_2,1:M} = y_{1:L_2,1:M}^{(2)},
$$
$$
y_{L_1+1,M+1} =\cdots= y_{L_1+L_2,M+1} = f_1(x).
$$
One can see that $\{y_{1:L_1+L_2,1:M+1}\}$ is a set of hidden variables of $f_2(f_1(\bm x),\bm x)$, thus the composition property holds.
\end{proof}

\begin{definition}
Given a continuous function $\varphi(\bm{x}),\bm{x} \in [-1,1]^d$ and a continuous function class $\mathcal F([-1,1]^d),$ define the $L^\infty$ distance
$$
\mbox{dist} (\varphi,\mathcal F) = \inf_{f\in \mathcal F} \max_{\bm x \in [-1,1]^d} |\varphi(\bm{x}) - f(\bm{x})|.
$$
\end{definition}

%\begin{properties}
\begin{proposition}\label{prop:dis}
(Addition and composition properties for distance function)
\begin{itemize}
\item[1] Let $\varphi_1$ and $\varphi_2$ be continuous functions. Let $\mathcal F_{1}$ and $\mathcal F_2$ be two continuous function classes, then
$$
\mbox{dist}(\alpha_{1}\varphi_1+\alpha_{2}\varphi_2,\mathcal{F}_1+\mathcal F_2)\le |\alpha_1|\mbox{dist}(\varphi_1,\mathcal F_1) + |\alpha_2|\mbox{dist}(\varphi_2,\mathcal F_2),
$$
where $\alpha_1$ and $\alpha_2$ are two real numbers.
\item[2] Assume that $\varphi_1(\bm x) = \varphi_1(x_1,...,x_d),\varphi_2(y,\bm x) = \varphi_2(y,x_1,...,x_d)$ satisfy $\varphi_1([-1,1]^d)\subseteq[-1,1]$. Let $\mathcal F_1([-1,1]^d),\mathcal F_2([-1,1]^{d+1})$ be two continuous function classes, then
$$
\mbox{dist}(\varphi_2(\varphi_1(\bm x),\bm x),\mathcal F_2\circ \mathcal F_1)\le L_{\varphi_2}\mbox{dist}(\varphi_1,\mathcal F_1) +\mbox{dist}(\varphi_2,\mathcal F_2)
$$
where $L_{\varphi_2}$ is the Lipschitz norm of $\varphi_2$ with respect to $y$.
\end{itemize}
\end{proposition}
%\end{properties}
\begin{proof}
The additional property obviously holds. Now we prove the composition property. For any $f_1\in\mathcal F_1,f_2\in \mathcal F_2$, one has
\[
\begin{split}
|\varphi_2(\varphi_1(\bm x),\bm x) - f_2(f_1(\bm x),\bm x)|&\le |\varphi_2(\varphi_1(\bm x),\bm x) - \varphi_2(f_1(\bm x),\bm x)|+|\varphi_2(f_1(\bm x),\bm x)-f_2(f_1(\bm x),\bm x)|\\
& \le L_{\varphi_2} ||\varphi_1(\bm x) -f_1(\bm x)||_\infty +||\varphi_2(y,\bm x) - f_2(y,\bm x)||_\infty
\end{split}
\]
Take $f_1^* = \argmin_f ||\varphi_1(\bm x) - f(\bm x)||_\infty$ and $f_2^* = \argmin_f ||\varphi_2(y,\bm x) - f(y,\bm x)||_\infty$, then it is proved.
\end{proof}

\begin{lemma}\label{lem:xsquare}
The function $\varphi(x) = x^2,x\in [-1,1]$ can be approximated by deep neural nets with an exponential convergence rate:
$$
\mbox{dist}(x^2,\mathcal{F}_{L,2}([-1,1]))\le 2^{-2L}.
$$
\end{lemma}
\begin{proof}
Consider the function
$$
g(y) = \left\{\begin{split}
	&2y,&\quad 0\le y<1/2,\\
	&2(1-y),&\quad 1/2\le y\le 1,
	\end{split}\right.
$$
then
\begin{equation}\label{func:g}
g(y) = 2y -4\mbox{ReLU}(y-1/2)
\end{equation}
in $[0,1]$. Define the hidden variables $\{y_{1:L,1:2}\}$ as follows:
\[
\begin{split}
 y_{1,1} = \mbox{ReLU}(x),&\quad y_{1,2}=\mbox{ReLU}(-x),\\
 y_{2,1} = \mbox{ReLU}(y_{1,1}+y_{1,2}),&\quad y_{2,2} = \mbox{ReLU}(y_{1,1}+y_{1,2}-1/2),\\
 y_{2,1} = \mbox{ReLU}(|x|),&\quad y_{2,2} = \mbox{ReLU}(|x|-1/2),\\
 y_{l+1,1} = \mbox{ReLU}(2y_{l,1}-4y_{l,2}),&\quad y_{l+1,2} = \mbox{ReLU}(2y_{l,1}-4y_{l,2}-1/2)\\
\end{split}
\]
for $l = 2,3,...,L-1$. Using induction, one can see that $|x| = y_{1,1}+y_{1,2}$ and
\begin{equation}\label{func:glinduction}
g_l(|x|)=\underbrace{g\circ g\circ \cdots \circ g}_{l}(|x|) = 2y_{l+1,1}-4y_{l+1,2},\quad l=1,...,L-1,
\end{equation}
for $x\in [-1,1]$. i.e.
\[
\begin{split}
g_l(|x|) &= g\left(g_{l-1}(|x|)\right)\\
         &= g(2y_{l,1}-4y_{l,2})\qquad (\mbox{Eq~(\ref{func:g})})\\
         &= 2(2y_{l,1}-4y_{l,2}) - 4\mbox{ReLU}(2y_{l,1}-4y_{l,2}-1/2)\qquad (g_{l-1}(|x|)\ge 0~\mbox{if}~x\in[0,1])\\
         &= 2\mbox{ReLU}(2y_{l,1}-4y_{l,2})- 4\mbox{ReLU}(2y_{l,1}-4y_{l,2}-1/2)\\
         & = 2y_{l+1,1} - 4y_{l+1,2}\\
\end{split}
\]
by induction, Eq (\ref{func:glinduction}) holds.
\begin{figure}[h]
	\centering
	\includegraphics[width=0.8\textwidth]{figures/dl_approx_analytic_gl}
	\caption{The figure of $g_l(x)$}
	\label{fig:gl}
\end{figure}

Let $f_m$ be the piecewise linear interpolation of $f$ with $2^m+1$ uniformly distributed breakpoints $\frac{k}{2^m},k = 0,...,2^m:$
$$
f_{m}(\frac{k}{2^m}) = (\frac{k}{2^m})^2,\quad k = 0,...,2^m.
$$
Note that refining the interpolation from $f_{m-1}$ to $f_m$ amounts to adjusting it by a function proportional to a sawtooth function:
$$
f_{m-1}(x) - f_m(x) = \frac{g_m(x)}{2^{2m}}.
$$
Hence
$$
f_{L-1}(|x|) = |x| - \sum_{l=1}^{L-1}\frac{g_l(|x|)}{2^{2l}} .
$$
then $f_{L-1}\in \mathcal{F}_{L,2}$, and
\[
\begin{split}
||x^2-f_{L-1}(x)||_{\infty} & = \max_{k} \max_{x\in [\frac{k}{2^{L-1}},\frac{k+1}{2^{L-1}}]} |x^2 - f_{L-1}(x)|\\
  & = \max_{k} |(\frac{1}{2}(\frac{k}{2^{L-1}}+\frac{k+1}{2^{L-1}}))^2 - f_{L-1}(\frac{1}{2}(\frac{k}{2^{L-1}}+\frac{k+1}{2^{L-1}}))|\\
  & = \max_k |(\frac{1}{2}(\frac{k}{2^{L-1}}+\frac{k+1}{2^{L-1}}))^2 - \frac{1}{2}((\frac{k}{2^{L-1}})^2+(\frac{k+1}{2^{L-1}})^2))|\\
  &=\frac{1}{4^L}.
\end{split}
\]
$|x^2-f_{L-1}(x)|\le 2^{-2L}$ for $x\in[-1,1].$
\end{proof}

\begin{lemma}\label{lem:xy}
For multiplication function $\varphi(x,y) = xy$, we have
$$
\mbox{dist}(xy,\mathcal{F}_{3L,2}([-1,1]^2))\le 3\cdot 2^{-2L}.
$$
\begin{proof}
Notice that
$$
\varphi = xy = 2\left(\frac{x+y}{2} \right)^2-\frac{1}{2}x^2-\frac{1}{2}y^2.
$$
\[
\begin{split}
\mbox{dist}(xy,\mathcal{F}_{3L,2}([-1,1]^d))&\le2\mbox{dist}(\left(\frac{x+y}{2} \right)^2,\mathcal{F}_{L,2}([-1,1]^2)) + \frac{1}{2} \mbox{dist}(x^2,\mathcal{F}_{L,2}([-1,1]^2)) \\
 &+\frac{1}{2} \mbox{dist}(y^2,\mathcal{F}_{L,2}([-1,1]^2))\\
 &\le 3\mbox{dist}(x^2,\mathcal{F}_{L,2}([-1,1]^2))\\
 &\le 3\mbox{dist}(x^2,\mathcal{F}_{L,2}([-1,1])) + \mbox{dist}(I_x,\mathcal{F}_{L,2}([-1,1]^2))\\
 &= 3\cdot 2^{-2L}
\end{split}
\]
\end{proof}
\end{lemma}

\begin{lemma}
For a monomial $M_p(\bm x)$ of $d$ variables with degree $p$, we have
$$
\mbox{dist}(M_p,\mathcal F_{3(p-1)L,3}(\mathbb R^d))\le 3(p-1)\cdot 2^{-2L}.
$$
\end{lemma}

\begin{proof}
Let $M_p(\bm x)=x_{i_1}x_{i_2}\cdots x_{i_p},i_1,...,i_p\in\{1,...,d\}.$ Using induction, assume that the lemma holds for the degree-$p$ monomial $M_p$, consider a degree-$(p+1)$ monomial $M_{p+1}(\bm{x}) = M_p(\bm{x})\cdot x_{i_{p+1}}$. Let $\varphi(y,x) = yx,$ then $M_{p+1}(\bm x) = \varphi(M_p(\bm{x}),x_{i_{p+1}})$. We have
\[
\begin{split}
\mbox{dist}(M_{p+1},\mathcal{F}_{3pL,3})&\le \mbox{dist}(\varphi(M_p(\bm x),x_{i_{p+1}}),\mathcal F_{3L,2}\circ \mathcal F_{3(p-1)L,3})\\
& \le L_{\varphi}\mbox{dist}(M_p,\mathcal{F}_{3(p-1)L,3})+\mbox{dist}(\varphi,\mathcal F_{3L,2})\le 3p\cdot 2^{-2L}.
\end{split}
\]
Note that the Lipschitz norm $L_{\varphi}=1$ since $x_{i_{p+1}}\in [-1,1].$
\end{proof}

\begin{lemma}
For a degree-$p$ polynomial $P_p(\bm{x}) = \sum_{|\bm{k}|\le p}a_{\bm k}\bm x^{\bm k},\bm x\in [-1,1]^d,\bm k = (k_1,...,k_d)\in\mathbb{N}^d,$ we have
\[
\mbox{dist}\left(P_p,\mathcal F_{\binom{p+d}{d}(p-1)L,3}\right)<3(p-1)\cdot 2^{-2L}\sum_{|\bm k|\le p}|a_{\bm k}|
\]
\end{lemma}\label{lem:poly}

\begin{proof}
This lemma can be proved by the properties \ref{prop:net class}, \ref{prop:dis} and lemma \ref{lem:poly}.
Note that the number of monomials of $d$ variables with degree less or equal to $p$ is $\binom{p+d}{d}$.
$$
k_1 + k_2 + \cdots + k_d \le p,
$$
Add a variable $k_{d+1}$, we have
$$
k_1 + k_2 + \cdots + k_d + k_{d+1} = p.
$$
the number of the non-negative solution is $\binom{p+d}{d}$.
\end{proof}

\begin{theorem}
Let $f$ be an analytic function over $(-1,1)^d$. Assume that the power series $f(\bm x) = \sum_{\bm k\in \mathbb N^d}a_{\bm k}\bm x^{\bm k}$ is absolutely convergent in $[-1,1]^d.$ Then for any $\delta>0,$ there exists a function $\hat f$ that can be represented by a deep ReLU network with depth $L$ and width $d+4$, such that
\[
|f(\bm x) - \hat f(\bm x)|<2\sum_{\bm k\in \mathbb N^d}|a_{\bm k}|\cdot \exp\left(-d\delta\left(e^{-1}L^{1/2d}-1\right)\right)
\]
for all $\bm x\in [-1+\delta,1-\delta]^d.$
\end{theorem}
\begin{proof}
Let $\epsilon = \exp(-d\delta(e^{-1}L^{1/2d}-1)),$ then $L = [e(\frac{1}{d\delta}\log\frac{1}{\epsilon}+1)]^{2d}$. Without loss of generality, assume $\sum_{\bm k}|a_{\bm k}|=1.$ We will show that there exists $\hat f\in \mathcal F_{L,3}$ such that $||f-\hat f||_{\infty}<2\epsilon.$
Denote
$$
f(\bm x) = P_p(\bm x) + R(\bm x): = \sum_{|\bm k|\le p}a_{\bm k}\bm x^{\bm k} +  \sum_{|\bm k|> p}a_{\bm k}\bm x^{\bm k}.
$$
For $\bm x\in [-1+\delta,1-\delta]^d$, we have $|R(\bm x)|<(1-\delta)^p$, thus truncation to $p = \frac{1}{\delta}\log\frac{1}{\epsilon}$ with ensure $|R(\bm x)|<2\epsilon$. From lemma \ref{lem:poly}, we have dist$(P_p,\mathcal F_{L,3})<3(p-1)\cdot 2^{-2L'}$, where
\[
\begin{split}
L' &= L\binom{p+d}{d}^{-1}(p-1)^{-1}\\
   &\ge L(\frac{(p+d)!}{p!d!})^{-1}p^{-1}\quad (d! \thicksim \sqrt{2\pi d}(d/e)^d)\\
   &\ge L(\frac{(p+d)^d}{(d/e)^d})^{-1}p^{-1}\\
   &= L[e(\frac{1}{d\delta}\log \frac{1}{\epsilon}+1)]^{-d}(\frac{1}{\delta}\log\frac{1}{\epsilon})^{-1}\\
   & = [e(\frac{1}{d\delta}\log \frac{1}{\epsilon}+1)]^{d}(\frac{1}{\delta}\log\frac{1}{\epsilon})^{-1}\\
   & \ge e^d (\frac{1}{\delta}\log\frac{1}{\epsilon}) \\
   &\gg\log\frac{1}{\delta}+ \log\frac{1}{\epsilon}
\end{split}
\]
for $d\ge 2$ and $\epsilon\ll 1$, then dist$(P_p,\mathcal F_{L,3})<3(p-1)\cdot 2^{-2L'} = 3(p-1)(\epsilon^2+\delta^2)\ll\epsilon$. Thus there exists $\hat f\in \mathcal F_{L,3}$ such that $||P_p - \hat f||_{\infty}<\epsilon$, and $||f-\hat f||_{\infty}\le ||f-P_p||_\infty + ||P_p - \hat f||_\infty<3\epsilon$.
\end{proof}

\subsection{Cosine Function}

Let $f(\bm x) = \cos(|\bm x|^2)$. Assume that the power series
\[
f(\bm x) = \sum_{k=0}^{+\infty} \frac{(-1)^k|\bm x|^{4k}}{(2k)!},\quad \bm x\in [-1,1]^d.
\]
Denote
\[
\begin{split}
f(\bm x) = P_p(\bm x) + R(\bm x): &= \sum_{|\bm k|\le p}a_{\bm k} \bm x^{\bm k} + \sum_{|\bm k|>p} a_{\bm k} \bm x^{\bm k}\\
&=\sum_{k\le p} \frac{(-1)^k}{(2k)!}|\bm x|^{4 k} + \frac{(-1)^{p+1}}{(2(p+1))!}|\bm t|^{4 (p+1)},\quad \bm t\in[0,1]^d\\
\end{split}
\]

Let $p(\varepsilon,d)=p$ be large, s.t.
\[
\big|\frac{(-1)^{p+1}}{(2(p+1))!}|\bm t|^{4(p+1)}\big| < \varepsilon.
\]
\[
\begin{split}
\big|\frac{(-1)^{p+1}}{(2(p+1))!}|\bm t|^{4 (p+1)}\big| &\le \frac{d^{2(p+1)}}{(2(p+1))!}\\
& \le C(\frac{ed}{2(p+1)})^{2(p+1)}\\
\end{split}
\]
Let $p = \log \frac{1}{\varepsilon}-1$, then
\[
(\frac{ed}{2(p+1)})^{2(p+1)}\le (\frac{ed}{2\log \frac{1}{\varepsilon}})^{2\log\frac{1}{\varepsilon}}\le \varepsilon^{-2\log \frac{ed}{2\log \frac{1}{\varepsilon}}}\le \varepsilon
\]

We want to show that
\[
\mbox{dist}(P_{p(\varepsilon,d)},\mathcal{F}_{L(\varepsilon,d),3})<\varepsilon.
\]

By Lemma~\ref{lem:poly}, we have
\[
\begin{split}
\mbox{dist}(P_{p},\mathcal{F}_{L,3})&<3(p-1)2^{-\frac{2L}{\binom{p+d}{d}(p-1)}}\sum_{\bm k\le p}|a_{\bm k}|\\
&\le3p2^{-\frac{2Ld^d}{p(p+d)^de^d}}\sum_{|\bm k|\le p}|a_{\bm k}|.
\end{split}
\]

Let $L = \frac{1}{2} ((p+d)e/d)^{2d}$, we can see that
\[
\begin{split}
\mbox{dist}(P_{p},\mathcal{F}_{L,3})
&\le3p2^{-\frac{2Ld^d}{p(p+d)^de^d}}\sum_{|\bm k|\le p}|a_{\bm k}|\\
&= 3p2^{-\frac{(p+d)^de^d}{d^dp}}\sum_{|\bm k|\le p}|a_{\bm k}|\\
&\le 3(\log \frac{1}{\varepsilon}-1)2^{-\frac{e^d}{4}(d-1)d(\log \frac{1}{\varepsilon}-1)} \sum_{|\bm k|\le p}|a_{\bm k}|\\
&\le C(d)\varepsilon^{\frac{e^d}{4}d(d-1)\log 2}\log\frac{1}{\varepsilon}\le \varepsilon
\end{split}
\]

Take
\[
\varepsilon = \exp\left(-d\left(e^{-1}(2L)^{1/2d}-1\right)-1\right)
\]
then, we have
\[
\mbox{dist}(f,\mathcal{F}_{L,3})\le\mbox{dist}(P_{p},\mathcal{F}_{L,3})+ R(\bm x) \le 2\varepsilon \le 2e^{-d\left(e^{-1}(2L)^{1/2d}-1\right)-1}
\]

\section{A modified ResNet structure and its properties}
Using the notation in Lin Li's notes, we have the next two properties for this modified ResNet structure in $\mathbb{R}^d$ as
\begin{itemize}
	\item Added with depth
	$$
	\mathcal F_{L_1,M}(\mathbb{R}^d) + \mathcal{F}_{L_2,M}(\mathbb{R}^d)  \subseteq \mathcal{F}_{L_1+L_2,M}(\mathbb{R}^d) .
	$$
	\item Added with width
	$$
	\mathcal F_{L,M_1}(\mathbb{R}^d)  + \mathcal{F}_{L,M_2}(\mathbb{R}^d)  \subseteq \mathcal{F}_{L,M_1 + M_2}(\mathbb{R}^d) .
	$$
\end{itemize}
This second property works for all DNN structures, but the first structure only works for this special ResNet structure with any activation functions.

\section{h-method by partition of unit}
First, for any positive integer $M$, let us consider the next partition of unit function of $[0, 1]^d$
$$
\phi_{\bm m}(x) = \prod_{k=1}^d \Phi\left(3M(x_k - \frac{m_k}{M})\right),
$$
with
$$
{\bm m} = (m_1, m_2, \cdots, m_d) \in \{0,1, \cdots,M\}^d = \mathcal{M},
$$
and
$$
\Phi(x) = \begin{cases}
1, \quad  &|x| \le 1, \\
0, \quad &|x| \ge 2, \\
2 - |x|, \quad &1 < |x| <2.
\end{cases}
$$
Thus, we have
$$
\sum_{{\bm m} \in \mathcal{M}} \phi_{\bm m}({\bm x}) = 1, \quad \forall {\bm x} \in [0,1]^d.
$$

\begin{itemize}
	\item How to choose M?
\end{itemize}
\section{p-method by local Taylor expansion}
Taylor expansion with $p-$th polynomials at most at ${\bm x} = \frac{{\bm m}}{M}$:
$$
P_{{\bm m}, p}({\bm x}) = \sum_{|{\bm n}| < p}\frac{\partial^{\bm n}f}{ \bm n !}\left(\frac{{\bm m}}{M}\right) ({\bm x} - \frac{{\bm m}}{M})^{\bm n}.
$$
Here we just consider about the local Taylor expansion, so the global approximation function is
$$
f_p({\bm x}) = \sum_{{\bm m} \in \mathcal{M}} \phi_{\bm m} P_{{\bm m}, p}({\bm x}), \quad \forall {\bm x} \in [0,1]^d.
$$

\begin{itemize}
	\item How to choose $p$? Balance with the residual term?
\end{itemize}

\section{Error estimate for $|f - f_p|_{0,\infty}$}
\begin{align*}\label{fperoor}
|f - f_p|_{0,\infty} &= |\sum_{\bm m} \phi_{\bm m}(f - P_{{\bm m}, p})|_{0, \infty},  \\
&\le \max_{{\bm m} \in \mathcal{M}}  |\sum_{\bm m} \phi_{\bm m}(f - P_{{\bm m}, p})|_{0, \infty,  |{\bm x} - \frac{\bm m}{M}| \le {\frac{1}{M}}}, \\
&\le \max_{{\bm m} \in \mathcal{M}} \sum_{ {\bm m} \in \mathcal{N}({\bm m})} |f - P_{{\bm m}, p}|_{0,\infty, |{\bm x} - \frac{\bm m}{M}| \le {\frac{1}{M}}}, \\
&\le \max_{{\bm m} \in \mathcal{M}}  2^d \max_{{\bm m} \in \mathcal{N}({\bm m})} |f - P_{{\bm m}, p}|_{0,\infty, |{\bm x} - \frac{\bm m}{M}| \le {\frac{1}{M}}}, \\
&\le \left( 2^{d} \left( \frac{1}{M} \right)^p \frac{d^p}{ p!} \right)|f|_{p, \infty}.
\end{align*}

\section{Approximate $f_p$ by ReLU DNN with error estimate}
\subsection{Approximate $\phi_m P_{m, p}$}
\begin{itemize}
	\item Approximate $\phi_{\bm m}$

	First we know that
	$$
	\Phi(x) = ReLU(x+2) - ReLU(x+1) - ReLU(x-1) + ReLU(x-2),
	$$
	so
	$$
	\inf_{v \in DNN_{J}} |\phi_{\bm m} - v| \le 2^{-\frac{2J}{3d}}.
	$$
	\item Approximate $P_{{\bm m}, p}$ (Original version in Yarotsky2017 )

	Assume that
	$$
	P_{{\bm m}, p} = \sum_{|{\bm n}| \le p}a_{\bm m, \bm n} {(\bm x - \frac{\bm m}{M})}^{\bm n},
	$$
	then
	$$
	\inf_{v \in DNN^{T}_{J}} |P_{{\bm m}, p} - v| \le (\max_{ \bm n} a_{\bm m,\bm n})2^{-\frac{2J}{3(p-1)}},
	$$
	with
	$$
	T = \tbinom{p+d}{d}(d+4).
	$$

	\item Approximate $\phi_{\bm m} P_{\bm m, p}$
	$$
	\inf_{v \in DNN^{T}_{J}} |\phi_{\bm m} P_{\bm m, p} - v| \le (\max_{ \bm n} a_{\bm m,\bm n}) \max\{2^{-\frac{2J}{9(p-1)}},  2^{-\frac{2J}{9d}}\},
	$$
\end{itemize}

\begin{remark}
In fact, in Yarotsky2017 they estimate as:
$$
\inf_{v \in DNN^{T}_{J}} |\phi_{\bm m} P_{\bm m, p} - v| \le (\max_{ \bm n} a_{\bm m,\bm n}) 2^{-\frac{2J}{9(p + d -1)}},
$$
and
$$
\max_{ \bm n} a_{\bm m,\bm n} \le 1,
$$
for
$$
\|f\|_{p+1,\infty} \le 1.
$$
\end{remark}

\subsection{Approximate $f_p$}
In Yarotsky2017,
$$
\inf_{v \in DNN^{\hat T}_{J}} |f_p - v| \le (\max_{ \bm n} a_{\bm m,\bm n}) 2^d d^p2^{-\frac{2J}{9(p + d -1)}},
$$
with
$$
\hat T = d^p (M+1)^d.
$$

\begin{remark}
	Here is a difference in Yarotsky2017 and Prof. E's paper:
	\begin{itemize}
		\item In Prof. E's paper
		$$
		\sum_{k} a_k e_k \le \max_k e_k (\sum_k a_k).
		$$
		\item In Yarotsky2017
		$$
		\sum_{k} a_k e_k \le \max_k a_k (\sum_k e_k).
		$$
	\end{itemize}
\end{remark}
More exactly, we can get
\begin{align}
\inf_{v \in DNN^{T}_{J}} |f_p - v| &\le  2^d (p+d) \sum_{k=0}^{p-1}\sum_{|\bm n|=k} a_{\bm m, \bm n}2^{-\frac{2J}{3(p + d )}}, \\
&\le  2^d (p+d) 2^{-\frac{2J}{3(p + d )}} \left(\sum_{k=0}^{p-1}\sum_{|\bm n|=k} \frac{C(\bm n)}{k!} |f|_{k,\infty}\right), \\
&= 2^d (p+d) 2^{-\frac{2J}{3(p + d )}} \left(\sum_{k=0}^{p-1}\sum_{|\bm n|=k} \frac{d^k}{k!} \right)\|f\|_{p-1,\infty}, \\
&\le  2^d (p+d) e^d 2^{-\frac{2J}{3(p + d )}}\|f\|_{p-1,\infty}.
\end{align}

\section{Balance $m, p$ with error and depth(d.o.f)}
In the end, we have
\begin{align}
\|f- \tilde f_p\|_{0,\infty} &\le \|f - f_p\|_{0,\infty} + \|f_p - \tilde f_p\|_{0,\infty}, \\
&\le \left[ \frac{2^dd^p}{p!}(\frac{1}{M})^p + 2^d e^d (d+p) 2^{-\frac{2J}{3(d+p)}}\right] \|f\|_{p, \infty}
\end{align}

Question: How to balance $m,p$?

The d.o.f of above model is about
$$
N = (M+1)^d \tbinom{p+d}{d} d^2 J,
$$
so the result for the approximation w.r.t the d.o.f is
\begin{align}
\|f- \tilde f_p\|_{0,\infty} &\le \|f - f_p\|_{0,\infty} + \|f_p - \tilde f_p\|_{0,\infty}, \\
&\le \left[ \frac{2^dd^p}{p!}(\frac{1}{M})^p + 2^d e^d (d+p) 2^{-\frac{2N}{3(d+p)(M+1)^d \tbinom{p+d}{d}d^2}}\right] \|f\|_{p, \infty}.
\end{align}

There are some problems to balance this two terms.

\subsection{Exponential convergence in this case}
Take $M = 2$, and
$$
p = \log_2\frac{1}{\epsilon}
$$
and $\epsilon << 1$ such that
$$
\frac{2^dd^p}{p!} \le 1,
$$
then we can take
$$
N = \left( 3e(1 + \frac{p}{d})\right)^{2d},
$$
then it is easy to see
$$
\|f- \tilde f_p\|_{0,\infty} \le 2\epsilon \lesssim \exp({ -d((3e)^{-1}N^{\frac{1}{2d}} - 1)}).
$$
\begin{proof}\label{proof:1}
\begin{align}
-\frac{2N}{3(d+p)(M+1)^d \tbinom{p+d}{d}d^2} &\le\frac{-2(3e(1+\frac{p}{d}))^d}{3(p+d)d^2}, \\
&\le -\frac{2e}{3}\times \frac{3^d e^{d-1}(\tbinom{d}{2}\frac{p^2}{d^2} + p)}{pd^3}, \quad (d \ge 2) \\
&= -\left( \frac{3^de^{d-1}(d-1)}{2d^2}p + \frac{3^de^{d-1}}{d^3} \right), \\
&\le - \left((\log_2{e}) p + (1+\log_2e)d\right). \quad (d \ge 2)
\end{align}
\end{proof}

\begin{remark}
	In Prof. E's result, it is:
	\begin{align}
	\|f- \tilde f_p\|_{0,\infty, I_\delta^d} &\le \|f - f_p\|_{0,\infty, I_\delta^d} + \|f_p - \tilde f_p\|_{0,\infty, I_\delta^d}, \\
	&\le \min_{p} \left[(1-\delta)^p + 3p2^{-\frac{2J}{3pd^2\tbinom{p+d}{d}}}\right] (\sum_{\bm n}a_{\bm n}).
	\end{align}
	seems easier to balance.
	\end{remark}

\section{Just $p$ version: expansion at $ (\frac{1}{2}, \cdots, \frac{1}{2})$}

For $\Omega = [0,1]^d$, and $f_p$ is like
$$
f_p = \sum_{k=0}^{p-1} \frac{1}{k!} \left(\sum_{i=1}^d (x_i - \frac{1}{2})\frac{\partial}{\partial x^i}\right)^k f |_{x = \frac{\bm 1}{\bm 2}},
$$
then we have:
\begin{align}
\|f - f_p\|_{0, \infty} &= \sup_{\bm x \in \Omega} \left|  \frac{1}{p!} \left(\sum_{i=1}^d (x_i - \frac{1}{2})\frac{\partial}{\partial x^i}\right)^p f |_{\xi = \frac{\bm 1}{\bm 2} + \theta_{\bm x}(\bm x - \frac{\bm 1}{\bm 2})} \right|, \\
&\le \frac{d^p}{p!} (\frac{1}{2})^p |f|_{p, \infty}.
\end{align}

Then using $\tilde f_p$ to approximate $f_p$ with ``spectral" accuracy and finally get the ``spectral" accuracy fo analytical functions with $W^{p, \infty}$ norm with $p = C(\epsilon, d)$. This seems can remove the $\delta$ and $\sum_{\bm k} a_{\bm k}$ in Prof. E's results.

\subsection{$\|f\|_{p, \infty, I^d} \le C$ for any $p \in \mathbb{N}$}
Here $f_p$ can also be write as:
$$
f_p = \sum_{k=0}^{p-1} \sum_{|\bm n| = k} \frac{\partial^{\bm n}}{\bm n !} f|_{x = \frac{\bm 1}{\bm 2}} (\bm x - \frac{\bm 1}{\bm 2})^{\bm n},
$$
with approximation as:
$$
\tilde f_p = \sum_{k=0}^{p-1} \sum_{|\bm n| = k} \frac{\partial^{\bm n}}{\bm n !} f|_{x = \frac{\bm 1}{\bm 2}}  f^J_{\bm n}.
$$

We have the next error estimate
$$
| f^J_{\bm n} - (\bm x - \frac{\bm 1}{\bm 2})^{\bm n} |_{0,\infty, I^d} \le 2^{\frac{-2J}{3(|\bm n|-1)}}
$$

So we have the next error estimate
\begin{align}
\|f_p - \tilde f_p\| &= \| f^J_{\bm n} - (\bm x - \frac{\bm 1}{\bm 2})^{\bm n} \|_{0,\infty, I^d}, \\
&= \|  \sum_{k=0}^{p-1} \sum_{|\bm n| = k} \frac{\partial^{\bm n}}{\bm n !} f|_{x = \frac{\bm 1}{\bm 2}} (f^J_{\bm n} - (\bm x - \frac{\bm 1}{\bm 2})^{\bm n})\|_{0, \infty,I^d} \\
&\le \sum_{k=0}^{p-1} \sum_{|\bm n| = k} | \frac{\partial^{\bm n}}{\bm n !} |f|_{x = \frac{\bm 1}{\bm 2}}| 3k2^{\frac{-2J}{3(|\bm n|-1)}}\\
&\le \sum_{k=0}^{p-1} \frac{d^k}{k!}|f|_{k,\infty,I^d} 3k2^{\frac{-2J}{3(|\bm n|-1)}} \\
&\le   3pe^d2^{\frac{-2J}{3p}} \|f\|_{p-1, \infty, I^d}.
\end{align}
Because $\tilde f_p$ has $\tbinom{p+d}{d}$ terms, so if only $J$-th layer, we will have
$$
\| f_p - \tilde f^J_p\| \le  3pe^d  2^{\frac{-2J}{3p\tbinom{p+d}{d}}}\|f\|_{p-1, \infty, I^d},
$$
at last we have:
\begin{align}
\|f- \tilde f^J_p\|_{0,\infty} &\le \|f - f^J_p\|_{0,\infty} + \|f_p - \tilde f^J_p\|_{0,\infty}, \\
&\le \left[  \frac{d^p}{p!} (\frac{1}{2})^p +3pe^d  2^{\frac{-2J}{3p\tbinom{p+d}{d}}}\right] \|f\|_{p, \infty}.
\end{align}

A simple case, consider $\epsilon << 1$, and $\frac{d^p}{p!} \le 1$, we can take
$$
p = \log_2 \frac{1}{\epsilon},
$$
and
$$
J = [e(1 + \frac{p}{d})]^{2d},
$$
it is easy to see that
$$
\frac{d^p}{p!} (\frac{1}{2})^p \le \epsilon,
$$
and
$$
e^d  2^{\frac{-2J}{3p\tbinom{p+d}{d}}} \le \epsilon,
$$
such that
$$
\|f- \tilde f^J_p\|_{0,\infty}  \le 2\epsilon \lesssim \exp(-d(e^{-1}J^{\frac{1}{2d}} - 1))\|f\|_{p, \infty}
$$
\begin{proof}\label{proof:2}
The similar process in \ref{proof:1}.
\begin{align}
-\frac{2J}{3p\tbinom{p+d}{d}} &\le \frac{-2(e(1+\frac{p}{d}))^d}{p}, \\
&\le -2 \times \frac{ e^{d}(\tbinom{d}{2}\frac{p^2}{d^2} + p)}{3p}, \quad (d \ge 2) \\
&= -\frac{2}{3}\left( \frac{e^{d}(d-1)}{2d}p + e^d \right), \\
&\le - 2(\log_2{e}) p - (\log_2e)d. \quad (d \ge 2)
\end{align}
\end{proof}

\subsection{Derivative scaling}
\subsubsection{Power scaling}
If there exist $M \ge 1$ and $C$ such that
$$
|f|^P_{p,\infty,I^d} := \frac{|f|_{p,\infty,I^d}}{M^p} \le C, \quad \forall p \ge 1.
$$
We will have the next error estimate:
	\begin{align}
\|f- \tilde f^J_p\|_{0,\infty} &\le \|f - f^J_p\|_{0,\infty} + \|f_p - \tilde f^J_p\|_{0,\infty}, \\
&\le \left[  \frac{(Md)^p}{p!} (\frac{1}{2})^p +3pe^{Md}  2^{\frac{-2J}{3p\tbinom{p+d}{d}}}\right] C.
\end{align}
Then we can still have the next result:
\begin{equation}
\|f- \tilde f^J_p\|_{0,\infty, I^d}  \le 2\epsilon \lesssim \exp(-d(e^{-1}J^{\frac{1}{2d}} - 1))C
\end{equation}
for $\epsilon \ll 1$ and $J \gg1$.
\begin{proof}
Considering that $\epsilon \ll 1$, and $p$ is big enough such that
$$
 \frac{(Md)^p}{p!} \le 1,
$$
then we will have
$$
p \ge Md.
$$
Then take
$$
p = \max\{\log_2\frac{1}{\epsilon}, Md\},
$$
and
$$
J = [e(1 + \frac{p}{d})]^{2d}.
$$
Then
\begin{align}
-\frac{2J}{3p\tbinom{p+d}{d}} &\le \frac{-2(e(1+\frac{p}{d}))^d}{p}, \\
&\le -2 \times \frac{ e^{d}(\tbinom{d}{3}\frac{p^3}{d^3}  + \tbinom{d}{2}\frac{p^2}{d^2} + p)}{3p}, \quad (d \ge 3) \\
&= -\frac{2}{3}\left(  \frac{e^{d}(d-1)(d-2)}{2d^2}p^2 + \frac{e^{d}(d-1)}{2d}p + e^d \right), \quad (p \ge Md)\\
&\le - 2(\log_2{e}) p - (\log_2e)Md. \quad (d \ge 2)
\end{align}
\end{proof}
\subsubsection{Factorial scaling}
Define
$$
|f|^F_{p,\infty,I^d} := \frac{d^p|f|_{p,\infty,I^d}}{p!} \le C, \quad \forall p \ge 1.
$$
We will have the next error estimate:
\begin{align}
\|f- \tilde f^J_p\|_{0,\infty} &\le \|f - f^J_p\|_{0,\infty} + \|f_p - \tilde f^J_p\|_{0,\infty}, \\
&\le \left[  (\frac{1}{2})^p +3p^2 2^{\frac{-2J}{3p\tbinom{p+d}{d}}}\right] C.
\end{align}
Then we can still have the next result:
\begin{equation}
\|f- \tilde f^J_p\|_{0,\infty, I^d}  \le 2\epsilon \lesssim \exp(-d(e^{-1}J^{\frac{1}{2d}} - 1))C
\end{equation}
for $\epsilon \ll 1$ and $J \gg1$.
\begin{proof}
	This is same but simpler than the above proof.
	\end{proof}
\newpage
\subsection{A more compressive structure: 1d case}
Here we consider a more compressive structure for $1$d case with just $\log_2(p)$ layers to recover all $p$-th polynomials.
We use $f_{sq,J}$ to denote $J$-th layers network to approximate $x^2$ with width $3$ (without the identity neuron for $x$).

Considering the composition of $f_{sq,J}$ as
$$
f^k_{sq,J}(x) = f^{k-1}_{sq, J}(f_{sq,J}(x)),
$$
with
$$
f^1_{sq,J} = f_{sq,J}.
$$

\begin{properties}
	We have the next approximation properties
	$$
	\|f^k_{sq,J}(x) - x^{2^k} \|_{o,\infty, I} \le k2^{-2J}
	$$
\end{properties}
\begin{proof}We prove it by induction,
	\begin{align}
	\|f^k_{sq,J}(x) - x^{2^k} \|_{o,\infty, I} &= \|f_{sq,J}(f^{k-1}_{sq,J}) - f_{sq,J}(x^{2^{k-1}}) +f_{sq,J}(x^{2^{k-1}}) - f(x^{2^{k-1}}) \|_{0,\infty,I}, \\
	&\le L_{f_{sq,J}} \|f^{k-1}_{sq,J}(x) - x^{2^{k-1}}\|_{0, \infty, I} + \|f_{sq,J}(x) - x^2\|_{0,\infty,I}, \\
	&\le (k-1)2^{-2J} + 2^{-2J}.
	\end{align}

	\end{proof}


\section{Deep Neural Net - RegularNet}
The regular neural network is composed exclusively of regular and strided convolutional layers. While this architecture works well for relatively shallow networks, it becomes increasingly more difficult to train as the network depth increases.

This section will elaborate on the implementaiton of a RegularNet used for this project.

The RegularNet implemented in the project consists of 34 layers, with an image as input layer. The first five layers in the neural network are normal convolutional layers with a zero padding of one, striding unit distance of one and feature mapping of 3x3. The convolutional layers are followed by a convolutional layer with a striding unit distance of two. The sixth layer functions as a pooling layer, since the output from the layer is an image of half the input size. The RegularNet consists of 25 convolutional layers, 5 pooling layers, 1 input layer, 1 flatten layer and a softmax classifier layer is appended to the end of the RegularNet. Adam is used as the activation function for the RegularNet, and normalization is used for regularization function. The creation of the RegularNet layers can be seen in listing \ref{lst:regularloop}.

\begin{lstlisting}[language=Python, label=lst:regularloop, caption=For loop that creates the layers in the RegularNet]
input_layer = tf.placeholder(shape=[None,32,32,3],dtype=tf.float32,
		name='input')
label_layer = tf.placeholder(shape=[None],dtype=tf.int32)
label_oh = slim.layers.one_hot_encoding(label_layer,10)

layer1 = slim.conv2d(input_layer,64,[3,3],
		normalizer_fn=slim.batch_norm, scope='conv_'+str(0))
for i in range(5):
	for j in range(units_between_stride):
		layer1 = slim.conv2d(layer1,64,[3,3],normalizer_fn=slim.batch_norm,
				scope='conv_'+str((j+1) + (i*units_between_stride)))
	layer1 = slim.conv2d(layer1,64,[3,3],stride=[2,2],
			normalizer_fn=slim.batch_norm,scope='conv_s_'+str(i))

top = slim.conv2d(layer1,10,[3,3],
		normalizer_fn=slim.batch_norm, activation_fn=None,scope='conv_top')

output = slim.layers.softmax(slim.layers.flatten(top))
\end{lstlisting}

As seen in listing \ref{lst:regularloop} the function conv2d is called to create a convolution neuron. This function creates the convolution block with the internal layers and returns the output from the block. One neuron consists of 1 convolutional layer and 1 batch normalization block. Between the convolutional layer and the batch layer a ReLU activation function is added. The convolutional layer consists of 64 feature maps which each has 9 weights, so the convolutional layer outputs 64 new images for the ReLU function. The code for training and updating the model can be seen in \ref{lst:ConvLoss}.

\begin{lstlisting}[language=Python, label=lst:ConvLoss, caption= Implementation of learning rate type ADAM]
loss = tf.reduce_mean(-tf.reduce_sum(label_oh * tf.log(output) + 1e-10
, axis=[1]))
trainer = tf.train.AdamOptimizer(learning_rate=0.001)
update = trainer.minimize(loss)
\end{lstlisting}


\subsection{Example: Slider-crank Mechanism}
\begin{frame}{Normal Approach}
	\begin{block}{Example 1: Slider-crank Mechanism}
		\begin{table}
			\begin{minipage}{0.5\linewidth}
				\begin{tabular}{l|l}
					& $l_{AB}=l_1=0.5m$ \\
					& $l_{BC}=l_2=1m$ \\
					Given& $\theta_1=60^\circ$ \\
					& $\vb{\omega}{1} = 1\kh $ $(rad/s)$\\
					& $\vb{\alpha}{1} = -1\kh $ $(rad/s^2)$ \\\hline
					Find & $\vb{v}{B}$, $\vb{v}{C}$, $\vb{a}{B}$, $\vb{a}{C}$,\\
					 &\boldmath{$\vb{\omega}{2}$}, \boldmath{$\vb{\alpha}{2}$}\\
				\end{tabular}
			\end{minipage}\hfill
			\begin{minipage}{0.5\linewidth}
				\includegraphics[width=50mm]{images/R-RRT.png}
			\end{minipage}
		\end{table}
	\end{block}
\emph{Solution}\vskip2.5mm
Using position analysis / MATLAB,\vskip1.25mm
$\Rightarrow \vb{r}{B} = 0.25\ih + 0.433\jh\text{ $(m)$}, \vb{r}{C} = 1.1514\ih \text{ $(m)$}$\\
$\vb{r}{CB} = \vb{r}{C}-\vb{r}{B}=0.9014\ih-0.433\jh \text{ $(m)$}$
\end{frame}

\begin{frame}
\begin{itemize}
	\item Find $\vb{v}{B}$\vskip1.25mm
	$\vb{v}{B} = \vb{\omega}{1} \times \vb{r}{B} = -0.433\ih + 0.25\jh \text{ $(m/s)$}$\vskip2.5mm
	\item Find $\vb{v}{C}$, $\vb{\omega}{2}$\vskip1.25mm
	$\vb{v}{C} = \vb{v}{B} + \vb{v}{CB} = \vb{v}{B} + \vb{\omega}{2}\times \vb{r}{CB}$\vskip2.5mm
%	$v_{Cx}\ih = (-0.433\ih + 0.25\jh) + \omega_{2z}\kh\times(0.9014\ih-0.433\jh)$\vskip2.5mm
	Projecting onto $x,y$-axes, we obtain the solution:\vskip1.25mm
	$\Rightarrow\begin{cases}
	v_{Cx} = -0.5531\text{ $(m/s)$}\\ \omega_{2z}=-0.2774\text{ $(rad/s)$}
	\end{cases}$\\$\Rightarrow\begin{cases}
	\vb{v}{C} = -0.5531\ih\text{ $(m/s)$}\\ \vb{\omega}{2}=-0.2774\kh\text{ $(rad/s)$}
	\end{cases}$
\end{itemize}
\end{frame}
\begin{frame}
	\begin{itemize}
		\item Find $\vb{a}{B}$, $\vb{a}{^n_{CB}}$
		\vskip1.25mm
		$\vb{a}{B} = \vb{\alpha}{1}\times \vb{r}{B} - \vb{\omega}{1}^2\vb{r}{B}=0.183\ih - 0.683\jh \text{ $(m/s^2)$}$\\
		$\vb{a}{^n_{CB}}=- \vb{\omega}{2}^2\vb{r}{CB}=-0.0693\ih+0.0333\jh\text{ }(m/s^2)$
		\vskip2.5mm
		\item Find $\vb{a}{C}$, $\vb{\alpha}{2}$
		\vskip1.25mm
		$\vb{a}{C} = \vb{a}{B} + \vb{a}{CB} = \vb{a}{B} + \vb{\alpha}{2}\times \vb{r}{CB} - \vb{\omega}{2}^2\vb{r}{CB}$
%		$a_{Cx}\ih = 0.1137\ih-0.6497\jh+ \alpha_{2z}\kh\times(0.9014\ih-0.433\jh)$
		\vskip2.5mm
		Projecting onto $x,y$-axes, we obtain the solution:
		\vskip1.25mm
		$\Rightarrow\begin{cases}
			a_{Cx} = 0.4258\text{ $(m/s^2)$}\\\alpha_{2z}=0.7208\text{ }(rad/s^2)
		\end{cases}$\\
		$\Rightarrow\begin{cases}
		\vb{a}{C} = 0.4258\ih\text{ $(m/s^2)$}\\\vb{\alpha}{2}=0.7208\kh\text{ }(rad/s^2)
		\end{cases}$\\
	\end{itemize}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_v.m}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_a.m}
\end{frame}

\begin{frame}{Derivative Method}
	Let $\theta_1=\theta(t)$ be a function of time $t$. We perform position analysis to find $\vb{r}{B}(\theta(t))$ and $\vb{r}{C}(\theta(t))$. Using MATLAB, we find out that:\vskip1.25mm
	\hskip20mm$\displaystyle \vb{r}{B}(\theta(t)) = \frac{\cos{\theta(t)}}{2}\ih + \frac{\sin{\theta(t)}}{2}\jh$\vskip1.25mm
	\hskip20mm$\displaystyle \vb{r}{C}(\theta(t)) = \left(\frac{cos{\theta(t)}}{2} + \frac{\sqrt{2 - \sin{\theta(t)}}\sqrt{\sin{\theta(\theta(t))} + 2)}}{2}\right)\ih$\vskip2.5mm
	Using the results above to obtain the angle  $\theta_2(\theta(t))$ of link 2:
	\[\displaystyle\theta_2(\theta(t))=\arctan{\left(-\frac{\sin{\theta(t)}}{\sqrt{2 - \sin{\theta(t))}}\sqrt{\sin{\theta(t)} + 2}}\right)}\]
\end{frame}
\begin{frame}
Then, find velocities and accelerations by taking first and second derivative of $\vb{r}{B}$, $\vb{r}{C}$, $\theta_2$, respectively:
\[\begin{cases}
\displaystyle \vb{v}{B}(\theta(t))=\xvec[.]{\bm{r}}_{\bm{B}}\\\displaystyle \vb{v}{C}(\theta(t))=\xvec[.]{\bm{r}}_{\bm{C}}\\\displaystyle \vb{\omega}{2}(\theta(t))=\dot{\theta_2}\kh
\end{cases}, \begin{cases}
\displaystyle \vb{a}{B}(\theta(t))=\xvec[:]{\bm{r}}_{\bm{B}}\\\displaystyle \vb{a}{C}(\theta(t))=\xvec[:]{\bm{r}}_{\bm{B}}\\\displaystyle \vb{\alpha}{2}(\theta(t))=\ddot{\theta_2}\kh
\end{cases}\]
Let $\theta(t)=60^\circ$, we obtain the final results.
\end{frame}
\begin{frame}{MATLAB R2019a code}
	\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_p_d.m}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_v_a_d.m}
\end{frame}
\begin{frame}{Independent Contour Method}
	Using the solution above,\vskip1.25mm
	$\vb{r}{B} = 0.25\ih + 0.433\jh\text{ $(m)$}, \vb{r}{C} = 1.1514\ih \text{ $(m)$}$ \\
	$\vb{r}{CB}=0.9014\ih-0.433\jh \text{ $(m)$}$, $ \vb{a}{B}=0.183\ih - 0.683\jh \text{ $(m/s^2)$}$\vskip2.5mm
	Applying velocity formulas to find $\vb{\omega}{12} = \omega_{12z}\kh$, $\vb{\omega}{23} = \omega_{23z}\kh$,
	$\vb{v}{23} = v_{23x}\ih\kh$:
	\vskip1.25mm% and notice that there exists $\vb{v}{23}$ corresponding to slider 2:\vskip1.25mm
	$\begin{cases}
	\vb{\omega}{01} + \vb{\omega}{12} + \vb{\omega}{23} = \vb{0}{}\\
	\vb{r}{B}\times\vb{\omega}{12} + \vb{r}{C}\times\vb{\omega}{23} + \vb{v}{23} = \vb{0}{}
	\end{cases}$\vskip2.5mm%\$\Rightarrow\begin{cases}
%	1\kh+\omega_{12z}\kh + \omega_{23z}\kh=\vb{0}{}\\
%	(0.25\ih + 0.433\jh)\times\omega_{12z}\kh+(1.1514\ih)\times\omega_{23z}\kh+v_{23x}\ih=\vb{0}{}
%	\end{cases}$\vskip2.5mm
	Solving the system of equations by projecting onto $x,y$-axes, we obtain:\vskip1.25mm $\text{\boldmath$\vb{\omega}{12}$}=-1.2774\kh\text{ }(rad/s)$, $\text{\boldmath$\vb{\omega}{23}$}=0.2774\kh\text{ }(rad/s)$,
	$\vb{v}{23}=0.5531\ih\text{ }(m/s)$\vskip2.5mm
	The results is consistent with the 2 previous methods.
\end{frame}

\begin{frame}
	Again, using acceleration formulas and notice that there exists $\vb{a}{23}$ corresponding to slider 2:\vskip1.25mm
	$\begin{cases}
	\text{\boldmath$\vb{\alpha}{01}$}+\text{\boldmath$\vb{\alpha}{12}$}+\text{\boldmath$\vb{\alpha}{23}$}=\vb{0}{}\\
	\vb{r}{B}\times\text{\boldmath$\vb{\alpha}{12}$}+\vb{r}{C}\times\text{\boldmath$\vb{\alpha}{23}$}+\vb{a}{23}-\vb{\omega}{1}^2\vb{r}{B}-\vb{\omega}{2}^2\vb{r}{CB}=\vb{0}{}
	\end{cases}$\vskip2.5mm%\\$\Rightarrow\begin{cases}
%	-1\kh+\alpha_{12z}\kh + \alpha_{23z}\kh=\vb{0}{}\\
%	(0.25\ih + 0.433\jh)\times\alpha_{12z}\kh+(1.1514\ih)\times\alpha_{23z}\kh+a_{23x}\ih\\\hskip40mm-0.3193\ih-0.3997\jh=\vb{0}{}
%	\end{cases}$\vskip2.5mm
	Solving the system of equations by projecting onto $x,y$-axes, we obtain:\vskip1.25mm $\text{\boldmath$\vb{\alpha}{12}$}=1.7208\kh\text{ }(rad/s^2)$, $\text{\boldmath$\vb{\alpha}{23}$}=-0.7208\kh\text{ }(rad/s^2)$,
	$\vb{a}{23}=-0.4258\ih\text{ }(m/s^2)$\vskip2.5mm
	The results is consistent with the 2 previous methods.
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_v_i.m}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_v_i_2.m}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_a_i.m}
\end{frame}
\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_a_i_2.m}
\end{frame}
\begin{frame}{Velocity - Acceleration plotting}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/s-c_v_a_plotting.m}
\end{frame}
\begin{frame}{Output figures}
\begin{table}
\begin{minipage}{0.5\linewidth}
	\begin{figure}
		\centering
		\includegraphics[width=60mm]{images/v_RRRT.png}
	\end{figure}
\end{minipage}\hfill
\begin{minipage}{0.5\linewidth}
	\begin{figure}
		\centering
		\includegraphics[width=60mm]{images/a_RRRT.png}
	\end{figure}
\end{minipage}
\end{table}
\end{frame}