\documentclass[../../main.tex]{subfiles}

\begin{document}

\subsection{Motivation}

It is considered a time series classification problem. This task is solved using ODE-RNN model. The ODE-RNN architecture was decribed in \cite{Samokhina}.

\subsection{Problem statement}

Given a multivariate time series:

$$\mathfrak{D} = \{(x_i, y_i)\}, \quad i = \overline{1, n}, \quad x_i \in \mathbb{R}^m.$$

\noindent
The problem of time series intervals classification is solved. We have to define if there is a P300 potential on the EEG interval.

Optimization task:

$$\hat{\theta} = \arg\max\limits_{\theta}L(\theta, \mathbf{X}).$$

\subsection{Problem solution}

ODE-LSTM is based on standard LSTM architecture but the continuity of hidden state is added. Hidden state of LSTM is a pair $(\mathbf{c}_t, \mathbf{h}_t)$, where $\mathbf{c}_t$ is a long term memory state, $\mathbf{h}_t$ is a hiden state. The function $f_\theta(\mathbf{x}_{t+1}, (\mathbf{c}_t, \mathbf{h}_t), \mathbf{1}) \rightarrow (\mathbf{c}_{t+1}, \mathbf{h}_{t+1})$ that updates these states can be described using the following equations:

$$\mathbf{z}_{t+1} = \tanh(\mathbf{W}_z\mathbf{x}_{t+1} + \mathbf{R}_z\mathbf{h}_t + \mathbf{b}_z)$$
$$\mathbf{i}_{t+1} = \sigma(\mathbf{W}_i\mathbf{x}_{t+1} + \mathbf{R}_i\mathbf{h}_t + \mathbf{b}_i)$$
$$\mathbf{f}_{t+1} = \sigma(\mathbf{W}_f\mathbf{x}_{t+1} + \mathbf{R}_f\mathbf{h}_t + \mathbf{b}_f + \mathbf{1})$$
$$\mathbf{o}_{t+1} = \sigma(\mathbf{W}_o\mathbf{x}_{t+1} + \mathbf{R}_o\mathbf{h}_t + \mathbf{b}_o)$$
$$\mathbf{c}_{t+1} = \mathbf{z}_{t+1} \odot \mathbf{i}_{t+1} + \mathbf{c}_{t} \odot \mathbf{f}_{t+1}$$
$$\mathbf{h}_{t+1} = \tanh(\mathbf{c}_{t+1}) \odot \mathbf{o}_{t+1}$$

When shifting from LSTM to ODE-LSTM the above equations are changed as follows:

$$(\mathbf{c}_{i}, \mathbf{h}_{i}^\prime) = \text{LSTM}(\theta_l, (\mathbf{c}_{i}, \mathbf{h}_{i}), x_i)$$
$$\mathbf{h}_i = \text{ODESolve}(f_\theta, \mathbf{h}_{i-1}, \mathbf{h}_i^\prime, t_i - t_{i-1})$$
$$\mathbf{o}_i = \mathbf{h}_i\mathbf{W}_{\text{output}} + b_{\text{output}}$$

The differential equation is solved numerically using fourth order Runge-Kutta methods.

\subsection{Data}

The data was collected on 60 healthy people.  They were participating in a virtual reality game with a neural interface that is based on the classification of P300 potentials. It is considered a binary problem statement. In this dataset both the train and test samples are unbalanced. The class ratio is 1 to $(s-1)$, where $s$ is a number of stimuli. The number of timestamps in the dataset is 56540.

\subsection{Code analysis}

Code for ODE-LSTM was taken from GitHub repository\footnote{https://github.com/Alina-Samokhina/MasterThesis}.

\subsection{Experiment}

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{sections/gorpinich/data.png}
\caption{Time series visualization}
\label{fig:data}
\end{figure}

Fig. \ref{fig:data} plots P300 time series.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{sections/gorpinich/res.png}
\caption{Result after applying ODE-RNN}
\label{fig:res}
\end{figure}

Fig. \ref{fig:res} plots the result of applying ODE-RNN.

\end{document}


\section{Conclusion}\label{sec:stringmatcher:conclusion}
We made the first non-trivial use of (Liquid) Haskell as a proof
assistant.
We proved the parallelization of chunkable monoid
morphisms to be correct
and applied our parallelization technique to string matching,
resulting in a formally verified parallel string matcher.
%
Our proof uses refinement types to specify
equivalence theorems,
Haskell terms to express proofs,
and Liquid Haskell to check that the terms prove the theorems.
%
Based on our 1839LoC sophisticated proof we conclude that
Haskell can be successfully used as a theorem prover
to prove arbitrary theorems about real Haskell code
using SMT solvers to automate proofs
over key theories like linear arithmetic and equality.
%
However, Coq-like tactics or Dafny-like heurestics are required
to ease the user from manual proof term generation.

\begin{comment}
  - lines of code
  - interaction of proofs with code
        no interaction: the main proof
        invariant good indices are requied to prove that
           if target it bigger than input then indices is empty
           proof oof good indexing requires casting
   - proof reuse
   - trust library factions with assume annotations
\end{comment}


\section{Returns time series simulations}
\label{sec:simulations}

In Sect. \ref{subsec:gauss_alg_sim} we simulate returns time series following
a multivariate Gaussian and algebraic distribution. We test the influence of
the normalization within the epochs in the ergodicity defect in Sect.
\ref{subsec:norm_epochs_sim} and propose a solution for this defect in Sect.
\ref{subsec:norm_full_sim}. Finally we use empirical data to support our
simulation findings in Sect. \ref{subsec:emp_results}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Simulation of multivariate Gaussian and algebraic distributions}
\label{subsec:gauss_alg_sim}

In Sect. \ref{subsubsec:gauss_sim} we describe the methodology to simulate
multivariate Gaussian distributed returns time series and in Sect.
\ref{subsubsec:alg_sim} we describe the methodology to simulate multivariate
algebraic distributed returns time series.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Multivariate Gaussian distributions}\label{subsubsec:gauss_sim}

To simulate the returns time series, we use a method \cite{drawing_dist} for
drawing a random vector $x$ from the $N$-dimensional multivariate Gaussian
distribution with mean vector $\mu$ and covariance matrix $\Sigma$. First, we
create a correlation matrix $C$ with $c = 1$ on its diagonal and $c = 0.3$ on
its non-diagonal entries. Then, we compute the eigenvalues and eigenvectors of
the correlation matrix, such that $C = U \Lambda U^{-1}$. We get a $z$ vector
whose components are drawn from an independent standard Gaussian distribution.
Finally we obtain the returns with the desired distribution as
\begin{equation}
    r = \mu + U \Lambda^{1/2} z
\end{equation}

In our case, the $r$ vector components are drawn from a normal distribution
with $\mu$ vector zero. With this method we want to obtain time series
simulating the data matrix G with dimensions $2 \times T$, where $T$ is the
window length of the epochs. These returns can be later normalized, rotated and
aggregated to compare with the behavior of the results in Sect.
\ref{subsec:epochs}.  The goal of this approach is that all simulations should
show standard normal distributions.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_gauss_agg_ret_pairs_no_norm.png}
    \caption{Simulated aggregated rotated and scaled Gaussian distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ without
             normalization, neither within the epochs nor for all the time
             series. $\Delta t = 1$ unit and epochs window  lengths
             $T=10, 25, 40, 55, 100$ units.}
    \label{fig:gauss_epochs_agg_ret_pairs_no_norm}
\end{figure}

In Fig. \ref{fig:gauss_epochs_agg_ret_pairs_no_norm} we simulate time series
for $K = 200$. Each time series is made of $200$ epochs to make them comparable
to the empirical data. We use epochs window lengths $T = 10, 25, 40, 55$ units
to rotate, scale and aggregate without normalizing neither within the epochs
nor the full time series. As expected, as we draw the returns from a
multivariate Gaussian distribution, all the simulations show standard Gaussian
distribution behavior. Thus, this is our reference to check what is introducing
the ergodicity defect in the original method for the multivariate Gaussian
case.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Multivariate algebraic distributions}\label{subsubsec:alg_sim}

To simulate returns time series drawn from multivariate algebraic
distributions, we use a similar approach as in Sect. \ref{subsubsec:gauss_sim}.
First, we create a correlation matrix $C$ with $c = 1$ on its diagonal and
$c = 0.3$ on its non-diagonal entries. From \cite{t_student_dist} we know that
\begin{equation}
    T = \left( S^{-1/2} \right)^{\dagger} X + M,
\end{equation}
where $T$ is a vector of length $K$. Then it is needed to repeat the following
steps to generate a data matrix where the columns are the $T$ vectors. $X$ is
drawn from a matrix variate normal distribution. Matrix $S$ is a Wishart
distributed covariance matrix without normalization and M is a parameter that
for this case is zero. With these in mind, we first generate $S$ as a
positive semi-definite matrix. To do this, we first create time series of
simulated data matrix $G$ with dimension $K \times \left(n + K - 1 \right)$
where $n$ is the a parameter of the degree of freedom and is connected to the
shape parameter $l$ as
\begin{equation}
    l = \frac{n + K}{2},
\end{equation}
where $K$ is the number of companies. These time series are generated by
calculating
\begin{equation}
    y = U \Lambda^{1/2} z
\end{equation}
with a fixed covariance matrix $\Sigma$. Vector $y$ is a column vector of $G$,
$z$ is a univariate standard normal distribution vector, $U$ has the
eigenvectors of $\Sigma$ as columns and the diagonal matrix $\Lambda$ contains
the eigenvalues of $\Sigma$. Thus, we compute
\begin{equation}
    S = G G^{\dagger}
\end{equation}
Then, we obtain $S^{1/2}$ as
\begin{equation}
    S^{1/2} = U_{S} \Lambda_{S}^{1/2} U_{S}^{\dagger},
\end{equation}
since
\begin{align}
    S &= S^{1/2} S^{1/2} \\
    &= U_{S} \Lambda_{S}^{1/2} U_{S}^{\dagger}
    U_{S} \Lambda_{S}^{1/2} U_{S}^{\dagger}\\
    &= U_{S} \Lambda_{S} U_{S}^{\dagger}
\end{align}
We generate $X$ as
\begin{equation}
    X = \sqrt{m} z
\end{equation}
where $z$ is a univariate standard normal distribution vector of length $K$ and
$m$ is the variance.

These returns can be later normalized, rotated and aggregated to compare with
the behavior of the results in Sect. \ref{subsec:epochs}.  The goal of this
approach is that all simulations should show standard algebraic distributions.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_alg_agg_ret_pairs_no_norm.png}
    \caption{Simulated aggregated rotated and scaled algebraic distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ without
             normalization, neither within the epochs nor for all the time
             series. $\Delta t = 1$ unit and epochs window  lengths
             $T=10, 25, 40, 55, 100$ units.}
    \label{fig:alg_epochs_agg_ret_pairs_no_norm}
\end{figure}

In Fig. \ref{fig:alg_epochs_agg_ret_pairs_no_norm} we simulate time series for
$K = 200$. Each time series is made of $200$ epochs to make them comparable
to the empirical data. We use epochs window lengths $T = 10, 25, 40, 55$ units
to rotate, scale and aggregate without normalizing neither within the epochs
nor the full time series. We can see an interesting behavior in the algebraic
case, where for small epochs window lengths, the tails are similar to the
Gaussian distribution, and as the epochs window lengths grow, the simulations
reveal good agreement with the algebraic distribution. Thus, this is our
reference to check what is introducing the ergodicity effect in the original
method for the multivariate algebraic case.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Normalization within the epochs}
\label{subsec:norm_epochs_sim}

Now, to check the normalization within the epochs, we first simulate the
returns and then normalize each epoch to mean $\mu = 0$ and variance
$\sigma^{2} = 1$. Finally we repeat the procedure of rotate, scale and
aggregate.

For the Gaussian case with the simulated pair returns time series, we proceed
to normalize the epoch, compute the $2 \times 2$ sample covariance matrix and
diagonalize it. We rotate the two-component returns vectors into the eigenbasis
of the covariance matrix and normalize the axis with the eigenvalues. Finally
we aggregate all the components into a single univariate distribution.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_gauss_agg_ret_pairs_norm.png}
    \caption{Simulated aggregated rotated and scaled Gaussian distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ with
             normalization within the epochs. $\Delta t = 1$ unit and epochs
             window lengths $T=10, 25, 40, 55, 100$ units.}
    \label{fig:epochs_gauss_agg_ret_pairs_norm}
\end{figure}

As we can see in Fig. \ref{fig:epochs_gauss_agg_ret_pairs_norm}, the ergodicity
defect clearly appears for an epoch window length $T = 10$. As the epoch window
length grows, the ergodicity defect starts to disappear. We could even argue
that with an epoch window length greater or equal to $T = 25$, we already are
close enough to the Gaussian distribution. Thus, this ergodicity defect has a
large impact in small epochs window length, and its effects tend to disappear
as the epochs window lengths grow. This can be seen with $T = 100$.

Something similar happens with the algebraic case. We again simulate the
returns, normalize each epoch to mean $\mu = 0$ and variance $\sigma^{2} = 1$
and finally repeat the procedure of rotate, scale and aggregate. In Fig.
\ref{fig:epochs_alg_agg_ret_pairs_norm} can be seen how the ergodicity effect
appears again. It seems to have a large effect in epochs window lengths with
small values. Thus, we need to find an alternative to solve this problem.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_alg_agg_ret_pairs_norm.png}
    \caption{Simulated aggregated rotated and scaled Gaussian distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ with
             normalization within the epochs. $\Delta t = 1$ unit and epochs
             window lengths $T=10, 25, 40, 55, 100$ units.}
    \label{fig:epochs_alg_agg_ret_pairs_norm}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Normalization complete return time series}
\label{subsec:norm_full_sim}

We already showed that the ergodicity defect is directly related with the
normalization of the time series. To try to solve this issue, instead of
normalizing the time series within each epoch, we normalize the complete time
series and then proceed to rotate, scale and aggregate.

In Fig. \ref{fig:epochs_gauss_agg_ret_pairs_norm_full_ts} and Fig.
\ref{fig:epochs_alg_agg_ret_pairs_norm_full_ts} are shown the Gaussian and
algebraic cases with the corresponding simulations. We can notice how in both
cases the ergodicity defect disappears. In each epochs window length, the
probability density function perfectly fits the Gaussian distribution and the
algebraic distribution. It can be noted that the larger the epochs window
length, the better the simulated data fit the distributions. Furthermore, to
accomplish our assumption of stationarity on short time scales, the results
for short epochs window lengths have a good agreement with the theoretical
distribution.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_gauss_ts_norm.png}
    \caption{Simulated aggregated rotated and scaled Gaussian distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ with
             normalization for the complete time series. $\Delta t = 1$ unit
             and epochs window lengths $T=10, 25, 40, 55, 100$ units.}
    \label{fig:epochs_gauss_agg_ret_pairs_norm_full_ts}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\columnwidth]
    {figures/06_epochs_sim_alg_ts_norm.png}
    \caption{Simulated aggregated rotated and scaled algebraic distributed
             returns ($\tilde{r}$) for fixed covariance and $K=200$ with
             normalization for the complete time series. $\Delta t = 1$ unit
             and epochs window lengths $T=10, 25, 40, 55, 100$ units.}
    \label{fig:epochs_alg_agg_ret_pairs_norm_full_ts}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Empirical results}
\label{subsec:emp_results}

After we found the problem in the method and solved for the simulated time
series, it is time to check the solution in the empirical data. We first
normalize the complete time series. Then we rotate and scale the returns and
finally we aggregate them. With this method, we can see how the ergodicity
defect disappears, as we expected from the simulations. Furthermore, we can now
see that with an epoch window length of $T = 25$ the returns show small fat
tails. Then, we confirm that the aggregated returns have internal structures
that we will use to define the exact multivariate amplitude distributions in
Sect. \ref{sec:exact_distributions}.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\columnwidth]
    {figures/06_window_comparison.png}
    \caption{Aggregated distribution of returns ($\tilde{r}$) for fixed
             covariance of different number of companies selected from the S\&P
             500 dataset. $\Delta t = 1d$ and different epochs window lengths
             $T=10d$ (top left), $T=25d$ (top right), $T=40d$ (bottom left) and
             $T=55d$ (bottom right).}
    \label{fig:window_comparison_long_norm}
\end{figure}


\input{../../style/preamble}
\input{../../latex-math/basic-math}
\input{../../latex-math/basic-ml}

\newcommand{\titlefigure}{figure_man/hypercube.png}
\newcommand{\learninggoals}{
  \item Understand that our intuition about geometry fails in high-dimensional spaces
  \item Understand the effects of the curse of dimensionality
}

\title{Introduction to Machine Learning}
\date{}

\begin{document}

\lecturechapter{Curse of Dimensionality}
\lecture{Introduction to Machine Learning}

\begin{vbframe}{Curse of dimensionality}

\begin{itemize}
\item The phenomenon of data becoming sparse in high-dimensional spaces is one effect of the \textbf{curse of dimensionality}.
\item The \textbf{curse of dimensionality} refers to various phenomena that arise when analyzing data in high-dimensional spaces that do not occur in low-dimensional spaces.
\item Our intuition about the geometry of a space is formed in two and three dimensions.
\item We will see: This intuition is often misleading in high-dimensional spaces.
\end{itemize}

\end{vbframe}

\begin{vbframe}{Curse of Dimensionality: Example}
To illustrate one of the problematic phenomena of data in high dimensional data, we look at an introductory example: \\ \lz
We are given $20$ emails, $10$ of them are spam and $10$ are not. \\
Our goal is to predict if a new incoming mail is spam or not.

\medskip

For each email, we extract the following features:

\begin{itemize}
\item frequency of exclamation marks (in \%)
\item the length of the longest sequence of capital letters
\item the frequency of certain words, e.g., \enquote{free} (in \%)
\item ...
\end{itemize}

... and we could extract many more features!

\framebreak

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 11cm ]{figure_man/exclamation_marks.png}
\end{center}

Based on the frequency of exclamation marks, we train a very simple classifier (a decision stump with split point $\xv = 0.25$):

\begin{itemize}
\item We divide the input space into $2$ equally sized regions.
\item In the second region $[0.25, 0.5]$, $7$ out of $10$ are spam.
\item Given that at least $0.25\%$ of all letters are exclamation marks, an email is spam with a probability of $\frac{7}{10} = 0.7$.
\end{itemize}
\framebreak

Let us feed more information into our classifier. We include a feature that contains the length of the longest sequence of capital letters.
\medskip

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 10cm ]{figure_man/capital_letters.png}
\end{center}

\begin{itemize}
\item In the 1D case we had $20$ observations across $2$ regions.
\item The same number is now spread across $4$ regions.
\end{itemize}
\framebreak

Let us further increase the dimensionality to 3 by using the frequency of the word \enquote{your} in an email.

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 10cm]{figure_man/capital_letters2.png}
\end{center}

\vspace*{-.3cm}

\framebreak

\begin{itemize}
\item When adding a third dimension, the same number of observations is spread across $8$ regions.
\item In $4$ dimensions the data points are spread across $16$ cells, in $5$ dimensions across $32$ cells and so on ...
\item As dimensionality increases, the data become \textbf{sparse}; some of the cells become empty.
\item There might be too few data in each of the blocks to understand the distribution of the data and to model it.
\end{itemize}

\vspace*{-.2cm}

\begin{center}
\includegraphics[width = 0.5\textwidth]{figure_man/exponentialcubes.png}\\
\scriptsize{Bishop, Pattern Recognition and Machine Learning, 2006}
\end{center}

\end{vbframe}

\section{Geometry of High-Dimensional Spaces}

\begin{vbframe}{The high-dimensional cube}

\begin{itemize}
  \item We embed a small cube with edge length $a$ inside a unit cube.
  \item How long does the edge length $a$ of this small hypercube have to be so that the hypercube covers $10\%, 20\%, ...$ of the volume of the unit cube (volume 1)?

  \medskip
  \begin{center}
    \includegraphics[height = 4cm, width = 4cm]{figure_man/hypercube.png}
  \end{center}

\framebreak

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 10cm ]{figure_man/high-dim-cube.png}
\end{center}

\medskip

  \begin{footnotesize}
  \begin{eqnarray*}
    a^p &=& \frac{1}{10} \Leftrightarrow a = \frac{1}{\sqrt[p]{10}}
  \end{eqnarray*}
  \end{footnotesize}
  \vspace*{-0.5cm}
  \item  So: covering $10\%$ of total volume in a cell requires cells with almost $50\%$ of the entire range in $3$ dimensions, $80\%$ in $10$ dimensions.
\end{itemize}

\end{vbframe}

\begin{vbframe}{The high-dimensional sphere}

Another manifestation of the \textbf{curse of dimensionality} is that the majority of data points are close to the outer edges of the sample.

Consider a hypersphere of radius $1$. The fraction of volume that lies in the $\epsilon$-\enquote{edge}, $\epsilon := R - r$, of this hypersphere can be calculated by the formula

\vspace*{-0.7cm}

$$
1-\left(1-\frac{\epsilon}{R}\right)^p.
$$

\vspace*{-0.5cm}

\begin{center}
\includegraphics[width=0.9\textwidth]{figure_man/orange.png}
\end{center}

\vspace*{-0.5cm}

If we peel a high-dimensional orange, there is almost nothing left.

\flushleft

\framebreak

Consider a $20$-dimensional sphere. Nearly all of the volume lies in its outer shell of thickness $0.2$:
\medskip

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 10cm ]{figure_man/cursedim-fractionedge-plot-1.pdf}
\end{center}

\end{vbframe}

\begin{vbframe}{Hyphersphere within hypercube}
Consider a p-dimensional hypersphere of radius $r$ and volume $S_p(r)$ inscribed in a p-dimensional hypercube with sides of length $2r$ and volume $C_p(r)$. Then it holds that
\begin{footnotesize}
$$\lim_{p\rightarrow \infty} \frac{S_p(r)}{C_p(r)} = \lim_{p\rightarrow \infty}
\frac{\left( \frac{\pi^{\frac{p}{2}}}{\Gamma(\frac{p}{2}+1)} \right)r^p}{(2r)^p} =
 \lim_{p\rightarrow \infty} \frac{\pi^{\frac{p}{2}}}{2^p\Gamma(\frac{p}{2}+1)} = 0,$$
\end{footnotesize}
i.e., as the dimensionality increases, most of the volume of the hypercube can be found in its corners.

\begin{center}
\includegraphics[height = 3cm, keepaspectratio]{figure_man/sphere_in_cube.png}\\
\scriptsize{Mohammed J. Zaki, Wagner Meira, Jr., Data Mining and Analysis: Fundamental Concepts and Algorithms, 2014}
\end{center}

\framebreak

Consider a $10$-dimensional sphere inscribed in a $10$-dimensional cube. Nearly all of the volume lies in the corners of the cube:
\medskip

\vspace*{0.1cm}
\begin{center}
\includegraphics[width = 10cm ]{figure_man/vol_dim.png}
\end{center}

\begin{footnotesize}
Note: For $r > 0$, the volume fraction $\frac{S_p(r)}{C_p(r)}$ is independent of
$r$.
\end{footnotesize}

\end{vbframe}

\begin{vbframe}{Uniformly distributed data}
The consequences of the previous results for uniformly distributed data in the high-dimensional hypercube are:

\medskip

\begin{itemize}
\item Most of the data points will lie on the boundary of the space.
\item The points will be mainly scattered on the large number of corners of the hypercube, which themselves will become very long spikes.
\item Hence the higher the dimensionality, the more similar the minimum and maximum distances between points will become.
\item This degrades the effectiveness of most distance functions.
\item Neighborhoods of points will not be local anymore.
\end{itemize}

\end{vbframe}

\begin{vbframe}{Gaussians in high dimensions}

A further manifestation of the \textbf{curse of dimensionality} appears if we consider a standard Gaussian $N_p(\bm{0}, \id_p)$ in $p$ dimensions.

\begin{itemize}
    \item After transforming from Cartesian to polar coordinates and integrating out the directional variables, we obtain an expression for the density $p(r)$ as a function of the radius $r$ (i.e., the point's distance from the origin), s.t.
    $$ p(r) = \frac{S_p(r)r^{p-1}}{(2 \pi \sigma^2)^{p/2}} \exp \left( -\frac{r^2}{2\sigma^2}\right),$$
    where $S_p(r)$ is the surface of the $p$-dimensional hypersphere of radius $r$.
    \item Thus $p(r) \delta r$ is the approximate probability mass inside a thin shell of thickness $\delta r$ located at radius $r$.

\framebreak
\item To verify this functional relationship empirically, we draw $10^4$ points from the p-dimensional standard normal distribution and plot $p(r)$ over the histogram of the points' distances to the origin:

\begin{center}
\includegraphics[width = 11cm ]{figure_man/histograms.png}
\end{center}

\item We can see that for large $p$ the probability mass of the Gaussian is concentrated in a fairly thin \enquote{shell} rather far away from the origin. \\
This may seem counterintuitive, but:

\framebreak

  \item For the probability mass of a hyperspherical shell it follows that
    $$\int^{r + \frac{\delta r}{2}}_{r - \frac{\delta r}{2}}p(\tilde{r})d\tilde{r} = \int_{r - \frac{\delta r}{2} \; \leq \; ||\xv||_2 \; \leq \; r + \frac{\delta r}{2}} f_p(\tilde{\xv}) d\tilde{\xv},$$
    where $f_p(\xv)$ is the density of the $p$-dimensional standard normal distribution and $p(r)$ the associated radial density.

\begin{center}
\includegraphics[width = 11cm ]{figure_man/2d_normal.png}
\end{center}

\item While $f_p$ becomes smaller with increasing $r$, the region of the integral -the hyperspherical shell- becomes bigger.
\end{itemize}

\normalsize

\end{vbframe}

\begin{vbframe}{Intermediate Remarks}
However, we can find effective techniques applicable to high-dimensional spaces if we exploit these properties of real data:
\begin{itemize}
\item Often the data is restricted to a manifold of a lower dimension. \\ (Or at least the directions in the feature space over which significant changes in the target variables occur may be confined.)
\item At least locally small changes in the input variables usually will result in small changes in the target variables.
\end{itemize}

\begin{center}
\includegraphics[width = 11cm ]{figure_man/manifold.png}
\end{center}

\end{vbframe}

\endlecture
\end{document}


% DOCUMENT HEADER
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{pgfplots}
\usepackage{amsfonts}
\usepackage{fancyhdr}
\usepackage{tabularx}
\usepackage{gensymb}
\usepackage{amsmath}
\usepackage{hyperref}

% Set up hyperlinks
\hypersetup{colorlinks=true,
            linkcolor=blue,
            urlcolor=blue}

% TITLE
\title{\textbf{Understanding the Tangent Function}}
\author{\textit{Michal Špano}}
\date{}

% Set up default path for pictures
\graphicspath{{assets/}}

% Page header
\pagestyle{fancy}
\fancyhf{}
\rhead{MAT, @michalspano}
\lhead{Tangent function}
\rfoot{Page \thepage}

\begin{document}

\maketitle

% SECTION 1
\section{Definition}
To define the $tangent$ function, one must understand the basic principles of deriving $sine$ and $cosine$ using a so-called \textbf{unit circle}.

A \textbf{unit circle} is the circle of radius 1 centred at the origin $O(0, 0)$ in the \textbf{Cartesian coordinate system}.

% Include a picture of a unit circle
\begin{figure}[htp]
    \centering
    \includegraphics[width=9cm]{unit_circle.png}
    \caption{Unit circle}
\end{figure}

Then for a point $A = [x, y]$ moved along the circumference $k$ by an angle $\theta$ is valid:\\
\[
A = \left[ cos(\theta), sin(\theta) \right]
\]

\textit{Note: an applet is available (\href{https://www.geogebra.org/m/gvtvjtpf}{link}) or scan its \textbf{QR code} under \textbf{Resources}}.

A right-angled triangle consists of the \textbf{hypotenuse} (the side opposing the right angle) and 2 other adjacent sides called \textbf{legs}. The definition of tangent states that it is the quotient of the opposite side and the adjacent side to a certain angle $\theta$, symbolically:
\[
tan(\theta) = \frac{opposite}{adjacent}
\]

Which can be further transformed into:
\[
tan(\theta) = \frac{y}{x} ... tan(\theta) = \frac{sin(\theta)}{cos(\theta)}
\]

% SECTION 2
\section{Properties}

Now, let's address the properties of $f:y=tan(x)$

\textbf{Graph: }

% GENERAL GRAPH
\begin{figure}[htp]
    \centering
    \includegraphics[width=8cm]{tan.png}
    \caption{$y=tan(x)$}
\end{figure}

% ENUMERATED PROPERTIES
% http://www.nabla.hr/TF-TrigFunctionsD3.htm
\textbf{Properties: }
\begin{enumerate}

    % DOMAIN
    \item \textbf{Domain}
    \[
    f:y=tan(x) ... y=\frac{sin(x)}{cos(x)} \Leftrightarrow cos(x) \neq 0
    \]

    \begin{figure}[htp]
        \centering
        \includegraphics[width=10cm]{cosine.png}
        \caption{$y=cos(x)$}
    \end{figure}

    \textit{Let $f:y=cos(x)$}, solve for $x\in\mathbb{R}$.
    \[
    x \in \Big\{ \frac{\pi}{2}; \frac{3\pi}{2} + 2k\pi, k \in\mathbb{Z} \Big\} \Leftrightarrow
    x \in \Big\{ \frac{\pi}{2} + k\pi, k \in\mathbb{Z} \Big\}
    \]
    Through additional steps, we obtain the following result:
    \[
    x\in\{(2k+1)\frac{\pi}{2}, k\in\mathbb{Z} \}
    \]

    In other words, the \textbf{Domain} of $y=tan(x)$ is all real numbers except every $x$ for which is $cos(x) = 0$ defined. Symbolically:
    \[
    D_f= \mathbb{R}-\{ (2k+1)\frac{\pi}{2}, k\in\mathbb{Z} \}
    \]

    % Range of values
    \item \textbf{Range of values}

    As shown in the figure above, the \textbf{Range} of $f: y=tan(x)$:
    \[
    H_f=\mathbb{R}
    \]

    % Values in quadrants
    \item \textbf{Values in quadrants}

    \begin{center}
    \begin{tabular}{||c c c c||}
    \hline
    \textit{I.} & \textit{II.} & \textit{III} & \textit{IV} \\ [0.5ex]
    \hline\hline
    $\Big(0;\frac{\pi}{2}\Big)$ & $\Big(\frac{\pi}{2};\pi\Big)$ & $\Big(\pi;\frac{3\pi}{2}\Big)$ &
    $\Big(\frac{3\pi}{2};2\pi\Big)$ \\
    \hline
    + & - & + & - \\ [1ex]
    \hline
    \end{tabular}
    \end{center}

    % Periodicity of tangent
    \item \textbf{Periodicity}

    Suppose that the $period$ of $y=tan(x)$ is $2\pi$:
    \[
    tan(x + 2k\pi) = \frac{sin(x + 2k\pi)}{cos(x + 2k\pi)} = \frac{sin(x)}{cos(x)} = tan(x)
    \]
    However, $2\pi$ is not the smallest possible period. Let us consider a case where $period$ is $\pi$.
    \[
    tan(x + \pi) = \frac{sin(x +\pi)}{cos(x + \pi)} = \frac{-sin(x)}{-cos(x)} = tan(x)
    \]

    % Include graphical depiction
    \begin{figure}[ht!]
        \centering
        \vspace{-10pt}
        \centering\includegraphics[scale=2.5]{unit_circle_tangent_proof.png}
        \vspace{-10pt}
        \caption{Periodicity of tangent}
        \vspace{-10pt}
    \end{figure}

    \textbf{Explanation}: The $period$ of $f:y=sin(x)$ and $f:y=cos(x)$ is $2\pi$, any $x \in D_f$ shifted by the period of $\pi$ will produce a \textbf{function value} for which is valid: $\forall x \in D_f \subset \mathbb{R}: f(x + \pi)=-f(x)$.

    Let $A'$ be a point shifted along the circumference $k$ by an angle $\theta$, $A'_1$ be a point shifted along the same circumference by an angle $\theta + \pi$. Their coordinates are depicted using a \textbf{unit circle} and are symmetrical through $O[0,0]$ (point symmetry). Notice:
    \[
    A'_x = -A'_{x_1} \land A'_y = -A'_{y_1}
    \]

    \textit{Note: an applet is available (\href{https://www.geogebra.org/m/qr7hjqs6}{link}) or scan its \textbf{QR code} under \textbf{Resources}}.

    \textbf{Conclusion}
    Function \textbf{tangent} is \textbf{periodic} with the smallest possible
    period $\pi$.

    % Parity of tangent
    \item \textbf{Parity}
    \[
    f^+(x):y=tan(x)
    \]
    \[
    f^-(-x):y=tan(-x) = \frac{sin(-x)}{cos(-x)} = \frac{-sin(x)}{cos(x)} = -\frac{sin(x)}{cos(x)} = -tan(x),
    \]
    Based on the parity of \textbf{sine} (odd), \textbf{cosine} (even) $\Rightarrow sin(-x)=-sin(x) \land cos(-x) = cos(x)$.

    $f(-x) = -f(x) \Rightarrow$ Tangent is an \textbf{odd function}.

    % Monotonicity of tangent
    \item \textbf{Monotonicity}

    The tangent function is \textbf{increasing} on $\mathbb{R}$ between 2 consecutive vertical \textbf{asymptotes}, where every asymptote is defined as: $x=(2k + 1)\frac{\pi}{2}, k \in\mathbb{Z}$,\\that is:
    \[
    \forall x_{1,2}\in D_f:tan(x_1)<tan(x_2) \land x_1<x_2
    \]

    \begin{figure}[ht!]
        \centering
        \vspace{-10pt}
        \centering\includegraphics[scale=0.92]{monotony.png}
        \vspace{-10pt}
        \caption{Monotonicity of tangent}
        \vspace{-10pt}
    \end{figure}

    \item \textbf{Graph}

    \textbf{Tangentoid} is the plot of the tangent function.

    \item \textbf{Important function values of tangent}

    \begin{center}
    \begin{tabular}{||c c c c c c||}
    \hline
     $x: $ & $0\degree$ & $30\degree$ & $45\degree$ & $60\degree$ & $90\degree$ \\ [0.5ex]
     \hline
     $tan(x): $ & 0 & $\frac{\sqrt{3}}{3}$ & 1 & $\sqrt{3}$ & $\text{\O}$ \\ [1ex]
     \hline
    \end{tabular}

    \end{center}

\end{enumerate}

% Include scanable resources
\section{\textbf{Resources}}

\centering\includegraphics[scale=0.05]{applet1.png}
\includegraphics[scale=0.05]{applet2.png}

\centering\textit{Applet 1, Applet 2}

% Plot list of figures
\listoffigures

\end{document}


\chapter{Appendix}\label{section-appendix}
\section{MATLAB Thrust Stabilization-Single tether implementation}\label{appendix-thruststab}
\begin{lstlisting}
% This code is an implementation of Thrust Stabilization
% for single tether
% Hovell, K, and Ulrich, S. (2017)
% "Experimental Validation for Tethered Capture of the Spinning Space Debris"
% AIAA SciTech Forum, AIAA Guidance, Navigation, and Control Conference, Grapevine, Texas. 9 ?13 January
% Paper no AIAA 2017-1049.

% ======
% Nomenclature:
% Jzzt = Moment of inertia, target about z-axis
% Jzzc = Moment of inertia, chaser about z-axis

% mt = Mass of the target
% mc = Mass of the chaser
% mj = Mass of the junction

% thetac0 = Initial angle of the chaser
% thetat0 = Initial angle of the target

% TS = Time step

% Lm0= unstretched length of the main tether
% Ls10 = unstretched length of the sub tether 1
% Ls20 = unstretched length of the sub tether 2
% Ls0 = unstretched length of the single tether

% ac = tether attachment point to body chaser with respect to chaser's
% center of mass
% as1 = tether attachment point to body sub tether 1 with respect to
% sub-tether1's center of mass
% as2 = tether attachment point to body sub tether 2 with respect to
% sub-tether2's center of mass
% at = tether attachment point to body target with respect to
% target's center of mass

% csingle = damping coefficient for single tether
% csub = damping coefficient for sub tether

% KPtheta = proportional gain matrix for the theta
% KDtheta = derivative gain matrix for the theta
% KP = proportional gain matrix for the position
% KD = derivative gain matrix for the postion

% ======
% Define initial conditions and TSS parameters for all simulations and
% experiment：

%in Kg·m^2
Jzzt=0.22;
Jzzc=0.30;

%in Kilogram
mt=12.19;
mc=17.24;
mj=0.01;

%in Degree
thetac0=180;
thetat0=0;

% in Second
TS=0.004;

%in Meter
Lm0=0.28;
Ls10=0.28;
Ls20=0.28;
Ls0=0.54;
ac=[0.135;-0.009];
as1=[0.110;0.127];
as2=[0.113;-0.094];
at=[0.110;0.016];

% in Newton·second/meter
csingle=3.0;
csub=0.9;

KPtheta=0.33;
KDtheta=0.56;
KP=[19.1 0;0 19.1];
KD=[32.7 0;0 32.7];

% ======
% Define initial conditions for Thurst Stabilization simulations and experiment
rt0=[0.35;1.19];
rc0=[1.10;1.20];
vt0=[0;0];
vc0=[0;0];
wt0=10; %degree/second
rf=[3.10;1.20];

% ======
% Initialization
% Simulation at time t=2s
% Nothing change in the first two seconds
rt=rt0;
rc=rc0;
vt=vt0;
vc=vc0;
thetat=thetat0;
thetac=thetac0;
Ls=rt+A(thetat)*at-rc-A(thetac)*ac;

% fai is the target tether angle
fai=norm(atand(Ls(2)/Ls(1))-thetat);

wt=wt0;
wc=0;
t=0; % It started to simulate at T=2, which seem it as t=0;
Fs=[0;0];

% AngularMomentum
% In radius

AngularMomentum = Jzzt * wt/57.3 + Jzzc * wc/57.3 ;

accelerationc=[0;0];
accelerationt=[0;0];
angularaccelerationt=0;
taot=0;
Fcontrol=[0;0];

for i=1:25000 % i* TS = the time , i is the number of the current time step
%record the data
recordt(i)=t;
recordrc(i,1)=rc(1);
recordrc(i,2)=rc(2);
recordrt(i,1)=rt(1);
recordrt(i,2)=rt(2);
recordthetat(i)=thetat;

recordwt(i)=wt;
recordFs(i,1)=Fs(1);
recordFs(i,2)=Fs(2);

recordLs(i,1)=Ls(1);
recordLs(i,2)=Ls(2);

recordvc(i,1)=vc(1);
recordvc(i,2)=vc(2);

recordvt(i,1)=vt(1);
recordvt(i,2)=vt(2);

recordaccelerationc(i,1)=accelerationc(1);
recordaccelerationc(i,2)=accelerationc(2);
recordaccelerationt(i,1)=accelerationt(1);
recordaccelerationt(i,2)=accelerationt(2);

recordangularaccelerationt(i)=angularaccelerationt;

recordtaot(i)=taot;

recordFcontrol(i,1)=Fcontrol(1);
recordFcontrol(i,2)=Fcontrol(2);
recordfai(i)=fai;

recordAngularMomentum(i) = AngularMomentum ;

%recordstiffness(i) = stiffness(norm(Ls)-Ls0);

%Ls is the vector which is from chaser to target, pointing at the target
Ls=rt+A(thetat)*at-rc-A(thetac)*ac;

% fai is target tether angle
fai=norm(-atand(Ls(2)/Ls(1))+thetat);

% Calculate the state of the tether
% Hovell and Ulrich (2017) eqn 19
if (norm(Ls)-Ls0)>0 % tether is tensioned
    Fs=(stiffness(norm(Ls)-Ls0)*(norm(Ls)-Ls0)+...
        dot(csingle*(vt+A(thetat)*[-wt/57.3*at(2);wt/57.3*at(1)]-vc-A(thetac)*[-wc/57.3*ac(2);wc/57.3*ac(1)]),...
        Ls/norm(Ls))).*Ls/norm(Ls);
else % tether is slackness
    Fs=[0;0];
end

% Force on target due to tether tension:
% through center of mass
% referenced to the target's body frame
Fst=(A(thetat))'*Fs;
Fsc=(A(thetac))'*Fs;

% Torque on target due to tether tension:
% about the z-axis of body frame (which is parallel to the z-axis of inertial frame)
% through center of mass target
% referenced to target body frame (the torque referenced to the body is the same as to the inertial frame)
taot=at(2)*Fst(1)-at(1)*Fst(2); % Nm
taoc=ac(1)*Fsc(2)-ac(2)*Fsc(1); % Nm

% The control thrust of chaser vehicle
Fcontrol=KP*(rdes(t)-rc)+KD*(vdes(t)-vc);

% The control torque of chaser vehicle
taocontrol=KPtheta* (180 - thetac) + KDtheta*(0-wc)  ;
accelerationt=(-Fs/mt);
accelerationc=(Fs+Fcontrol)/mc;
angularaccelerationt=taot/Jzzt;
angularaccelerationc=(taoc+taocontrol)/Jzzc;

% position and angle update
rt=rt+vt*TS;
rc=rc+vc*TS;
thetat=thetat+wt*TS;
thetac=thetac+wc*TS;

vt=vt+accelerationt*TS;
wt=wt+angularaccelerationt*TS*(180/pi);%/degree t-1
vc=vc+accelerationc*TS;
wc=wc+angularaccelerationc*TS*(180/pi); %/degree t-1
AngularMomentum = Jzzt * wt/57.3 + Jzzc * wc/57.3;
t=t+TS;
end
recordt=recordt';
recordthetat=recordthetat';
recordwt=recordwt';

recordrc=recordrc';
recordrt=recordrt';
recordvc=recordvc';
recordvt=recordvt';
recordaccelerationc=recordaccelerationc';
recordaccelerationt=recordaccelerationt';
recordangularaccelerationt=recordangularaccelerationt';
recordangularaccelerationt=recordangularaccelerationt';
recordaccelerationc=recordaccelerationc';
recordaccelerationt=recordaccelerationt';
recordtaot=recordtaot';
recordFcontrol=recordFcontrol';

n=1;
figure(n)
plot(recordt,recordwt);
title('Target angular rate')
xlabel('Time,s')
ylabel('Angular rate, deg/s')
n=n+1;

figure(n)
plot(recordt,recordthetat)
title('angle of target')
n=n+1;

figure(n)
plot(recordt,recordrc(1,:))
title(' x position of the chaser')
n=n+1;

figure(n)
plot(recordt,recordrc(2,:))
title('y position of the chaser')
n=n+1;

figure(n)
plot(recordt,recordrt(1,:))
title('x position of the target')
n=n+1;

figure(n)
plot(recordt,recordrt(2,:))
title('y position of the target')
n=n+1;

figure(n)
plot(recordt,mod(recordfai,360))
title('Target tether angle')
xlabel('Time,s')
ylabel('Thether Angle,deg')
n=n+1;

figure(n)
plot(recordt,recordAngularMomentum)
title('Total Angular Momentum')
xlabel('Time,s')
ylabel('Angular Momentum')

n = n + 1;
figure(n)
plot(recordrc(1,:),recordrc(2,:))
title('Track of Chaser')

n = n+1;
figure(n)
plot(recordrt(1,:),recordrt(2,:))
title('Track of Target')

% The function gives the attitude matrix in terms of theta
% cos -sin; sin cos;

function attitude=A(sita)
attitude=[cosd(sita) -sind(sita);sind(sita) cosd(sita)];
end

% calculate the design radius vector of time t
% the goal is to reach the destination in 20s after t=2
function [rdes]=rdes(t)
rc0=[1.10;1.20];
rf=[3.10;1.20];
d=2.*(rf-rc0)./(20*20); %in 20s
[rdes]=(rc0+d.*(t*t/2));
end

% calculate the single tether stiffness(N/m) in terms of
% stretch(m)-independent variable
% x=(||Ls||-Ls,0)
function stiff=stiffness(x)
originalstiffness=[163.500000000000,114.450000000000,80.2636363636363,
51.3857142857143,29.9096000000000,18.8842500000000,17.0724030612245,
10.1577253521127,8.06715957446808,7.09840970654628,6.99056826923077,
7.22393661971831,8.06497910662824,8.78250400534045,9.58060266159696];
originalstretch=[0.00300000000000000,0.00600000000000001,0.0110000000000000,
0.0210000000000000,0.0450000000000000,0.0860000000000001,
0.0980000000000001,0.213000000000000,0.329000000000000,0.443000000000000,
0.520000000000000,0.639000000000000,0.694000000000000,0.749000000000000,
0.789000000000000];
stiff=spline(originalstretch,originalstiffness,x);
end

% calculate the design velocity of time t
% the goal is to reach the destination in 20s after t=2s
function vdes=vdes(t)
rc0=[1.10;1.20];
rf=[3.10;1.20];
d=2.*(rf-rc0)./(20*20); %in 20s
vdes=d*t;
end
\end{lstlisting}

\section{Arduino CSV Streaming Example}\label{appendix-CSV}
\begin{lstlisting}
#include <ArdusatSDK.h>
#include <Wire.h>
#include <Arduino.h>

/*--------------------------------------------------------------
 *  Setup Software Serial to allow for both RF communication and USB communication
 *    RX is digital pin 8 (connect to TX/DOUT of RF Device)
 *    TX is digital pin 9 (connect to RX/DIN of RF Device)
 *-------------------------------------------------------------*/
ArdusatSerial serialConnection(SERIAL_MODE_HARDWARE_AND_SOFTWARE, 8, 9);

/*--------------------------------------------------------------
 *  Constant Definitions
 *------------------------------------------------------------*/

Acceleration accel;
Gyro gyro;
Magnetic mag;
Orientation orient(accel, mag);

void setup(void) {
  serialConnection.begin(57600);

  accel.begin();
  gyro.begin();
  mag.begin();
  orient.begin();
  serialConnection.println("Start of the loop");
  serialConnection.println("The acceleration x y z unit in m/s^2");
  serialConnection.println("The angular rate x y z unit in radian per second");
  serialConnection.println("The orientation roll pitch heading unit in degree");
}

void loop(void) {
  serialConnection.println(accel.readToCSV("Acceleration"));
  serialConnection.println(gyro.readToCSV("Gyro"));
  serialConnection.println(orient.readToCSV("Orientation"));
}

\end{lstlisting}

\section{MATLAB Serial REALTIME PLOTING}\label{appendix-realtime}
\begin{lstlisting}
%% Pre-process
close all;  % close all figures
clear all;  % clear all workspace variables
clc;        % clear the command line
fclose('all'); % close all open files
delete(instrfindall); % reset com port

%% Constatns
BAUDRATE = 57600; % Need to configure Xbee parts BD i.e. Interface Data Rate
INPUTBUFFER = 51200; %Unit in bytes. Definition:A location that holds all
% incoming information before it continues to the CPU for processing;
% Buffers used to store information before it processed

%% Initialize the stripchart
figure_acceleration = figure('Name','Acceleration');
axes_accelerationx = subplot(3,1,1); % Axes Object
xlabel(axes_accelerationx,'Time/ second')
ylabel(axes_accelerationx,'Accelerationx/ m/s^2')

axes_accelerationy = subplot(3,1,2);
xlabel(axes_accelerationy,'Time/ second')
ylabel(axes_accelerationy,'Accelerationy/ m/s^2')

axes_accelerationz = subplot(3,1,3);
xlabel(axes_accelerationz,'Time/ second')
ylabel(axes_accelerationz,'Accelerationz/ m/s^2')

h_ax = animatedline(axes_accelerationx,'MaximumNumPoints',Inf,'Marker','o','Color','red'); % animated line object
h_ay = animatedline(axes_accelerationy,'MaximumNumPoints',Inf,'Marker','+','Color','blue');
h_az = animatedline(axes_accelerationz,'MaximumNumPoints',Inf,'Marker','*','Color','green');

figure_gyro = figure('Name','Gyro');
axes_gyrox = subplot(3,1,1);
xlabel(axes_gyrox,'Time/ second')
ylabel(axes_gyrox,'Gyrox/ rad/s')

axes_gyroy = subplot(3,1,2);
xlabel(axes_gyroy,'Time/ second')
ylabel(axes_gyroy,'Gyroy/ rad/s')

axes_gyroz = subplot(3,1,3);
xlabel(axes_gyroz,'Time/ second')
ylabel(axes_gyroz,'Gyroz/ rad/s')

h_gx = animatedline(axes_gyrox,'MaximumNumPoints',Inf,'Marker','o','Color','red');
h_gy = animatedline(axes_gyroy,'MaximumNumPoints',Inf,'Marker','+','Color','blue');
h_gz = animatedline(axes_gyroz,'MaximumNumPoints',Inf,'Marker','*','Color','green');

figure_orientation = figure('Name','Orientation');
axes_roll = subplot(3,1,1);
xlabel(axes_roll,'Time/ second')
ylabel(axes_roll,'Roll/ deg')

axes_pitch = subplot(3,1,2);
xlabel(axes_pitch,'Time/ second')
ylabel(axes_pitch,'Pitch/ deg')

axes_heading = subplot(3,1,3);
xlabel(axes_heading,'Time/ second')
ylabel(axes_heading,'Heading/ deg')

h_roll = animatedline(axes_roll,'MaximumNumPoints',Inf,'Marker','o','Color','red');
h_pitch = animatedline(axes_pitch,'MaximumNumPoints',Inf,'Marker','+','Color','blue');
h_heading =animatedline(axes_heading,'MaximumNumPoints',Inf,'Marker','*','Color','green');

% Index
i = 1;
j = 1;
k = 1;

% Record the data
% 10000 Row data
record_acceleration = zeros(10000,4);
record_gyro = zeros(10000,4);
record_orientation = zeros(10000,4);

%% Create a serial object
board = serial('COM4','BaudRate',BAUDRATE,'DataBits',8);
% Related with the used COM, can be different
% COM4 for wireless, COM3 for wire
% BAUDRATE unit: bits per second, e.g. 9600 means 9600 bits per second,
% i.e. 1200 bytes per second i.e. 1.2 KB/s or 1.17KB/s

% Set serial port buffer
set(board,'InputBufferSize',INPUTBUFFER);
% InputBufferSize: total number of bytes that can be stored in the input
% buffer during a read operation. The read operation is terminated if the
% amount of data stored in the input buffer equals the InputBufferSize

fopen(board);
while(1)
    a = fgetl(board);% a:character vector
    b = textscan(a,'%u32 %s %f %f %f %u32','Delimiter',',');

    if strcmp(char(b{2}),'Acceleration')

        time = double(b{1})/1000; % in second
        ax = b{3};
            addpoints(h_ax,time,ax);
            drawnow limitrate

        ay = b{4};
            addpoints(h_ay,time,ay);
            drawnow limitrate

        az = b{5};
            addpoints(h_az,time,az);
            drawnow limitrate

        record_acceleration(i,:)=[time ax ay az];
        i=i+1;

    elseif strcmp(char(b{2}),'Gyro')
        time = double(b{1})/1000; % in second
        gx = b{3};
            addpoints(h_gx,time,gx);
            drawnow limitrate

        gy = b{4};
            addpoints(h_gy,time,gy);
            drawnow limitrate
        gz = b{5};
            addpoints(h_gz,time,gz);
            drawnow limitrate

        record_gyro(j,:)=[time gx gy gz];
        j=j+1;

    elseif strcmp(char(b{2}),'Orientation')
        time = double(b{1})/1000; % in second
        roll = b{3};
            addpoints(h_roll,time,roll);
            drawnow limitrate

        pitch = b{4};
            addpoints(h_pitch,time,pitch);
            drawnow limitrate

        heading = b{5};
            addpoints(h_heading,time,heading);
            drawnow limitrate

        record_orientation(k,:) = [time roll pitch heading];
        k = k+1;
    end

end
\end{lstlisting}
\newpage


\chapter{First Chapter}
\label{Chapter:1}

\section{A Section}
\label{Section:1-1}

Example of some math:
\begin{equation}
	\begin{alignedat}{2}
	\mathcal{H} = &-\mu_\mathrm{S}\, H \sum\limits_{i=1}^{N_\mathrm{S}} \hat{H}\cdot\vec{n}_i &&- \sum\limits_{\braket{ij}}\, J_{ij} \vec{n}_i\cdot\vec{n}_j\\
	&-K \sum\limits_{i=1}^{N_\mathrm{S}} (\hat{K}\cdot\vec{n}_i)^2 &&- \sum\limits_{\braket{ij}} \vec{D}_{ij} \cdot (\vec{n}_i\times\vec{n}_j),
	\label{Eq:Hamiltonian_Model}
	\end{alignedat}
\end{equation}
This is an atomistic spin model which can be used in \acf{ASD}


%classic
\section{Picard Vessiot Theory}

\begin{frame}
\frametitle{}
\bi
\item $(k, \partial)$ differential field over $\qzcl$,
\item $L  \in k[\partial] \subset \mathrm{End}_{\qzcl}(k)$ our differential operator
$$L = \partial^2 - a,\ a \in k^\times = k\backslash \{0\}$$
\item differential module
$$\left<\partial^i(x) = e_{1-i} : i = 0, 1\right>_{\qzcl} \subset k \oplus k,$$
the solution space(!)
\item representation matrix
$$A_\partial = \left(\bao{cc}0&1\\a&0\\\ea\right),$$
\ei
\end{frame}

\begin{frame}
\frametitle{}
\bi
\item $A_\partial$ has eigenvalues in $\qzcl$ if $a \in \qzcl^\times$,
\item eigenspaces of $A_\partial$ are non-trivial diff submodules of solution space,
\ei
\end{frame}


\documentclass{article}
    % General document formatting
    \usepackage[margin=0.7in]{geometry}
    \usepackage[parfill]{parskip}
    \usepackage[utf8]{inputenc}
    \usepackage[mathscr]{euscript}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{tikz}
    \usepackage{fancyhdr}

\pagestyle{fancy}
\fancyhf{}
\rhead{Edgar Jacob Rivera Rios - A01184125}

\renewcommand{\labelenumi}{\alph{enumi})}
\renewcommand{\labelenumii}{\roman{enumii})}

\begin{document}
\section*{1.3.1 Functions: Image, Closure}
\begin{enumerate}
    \item The floor function from $\mathbb{R}_{+}$ into $\mathbb{N}$ is defined by putting $\lfloor{x}\rfloor$ to be the largest integer less than or equal to $x$. What are the images under the floor function of the sets
    \begin{enumerate}
        \item $[0, 1] = \{x \in R: 0 \leq x \leq 1 \}$
        \begin{equation*}
            Image(Floor([0, 1])) = \{0, 1\}
        \end{equation*}
        \item $[0, 1) = \{x \in R: 0 \leq x < 1 \}$
        \begin{equation*}
            Image(Floor([0, 1))) = \{0\}
        \end{equation*}
        \item $(0, 1] = \{x \in R: 0 < x \leq 1 \}$
        \begin{equation*}
            Image(Floor((0, 1])) = \{0, 1\}
        \end{equation*}
        \item $(0, 1) = \{x \in R: 0 < x < 1 \}$
        \begin{equation*}
            Image(Floor((0, 1))) = \{0\}
        \end{equation*}
    \end{enumerate}
    \item Let $f : A\rightarrow A$ be a function from set $A$ into itself. Show that for all $X \subseteq A$, $f(X) \subseteq f[X]$, and give a simple example of the failure of the converse inclusion.
    \begin{align*}
        f[X] &= \{a \in A: f(x) = a, x \in X\}\\
        \forall x\in X, x \in A &\implies f[X] = \{a \in A: f(x) = a, x \in A\}\\
        &\implies f(x) \rightarrow A
    \end{align*}
    \item Show that when $f(A) \subseteq A$ then $f[A] = A$
    \begin{align*}
        f(A) \subseteq A &\implies  f[A] = \{a \in A: f(x) = a\}\\
        &\implies \forall a \in f[A], a \in A
    \end{align*}
    \item Show that for any partition of $A$, the function $f$ taking each element $a \in A$ to its cell is a function on $A$ into the power set $\mathscr{P}(A)$ of $A$ with the partition as its range.
    \begin{align*}
        f: A &\rightarrow \mathscr{P}(A)\\
        range(f) &= \mathcal{P}(A)\\
        \forall X \in \mathcal{P}(A)\\
    \end{align*}
    \item Let $f: A \rightarrow B$ be a function from set $A$ into set $B$. Recall the ‘abstract inverse’ function$f^{-1}: B \rightarrow \mathcal{P}(A)$ defined at the end of Slide 52 by putting $f^{-1}(b) = \{a \in A: f(a) = b\}$ for each $b \in B$.
    \begin{enumerate}
        \item Show that the collection of all sets for $b \in f(A) \subseteq B$ is a partition of $A$ in the sense defined in Chapter 2 of the David Makinson’s book.
        \begin{align*}
            \mathcal{P}(A) &= \{ A_{i}: i \in I, \forall a_{i} \in A_{i}, A_{i} \neq \emptyset, A_{i} \cap A_{i'} = \emptyset \}\\
            \{a \in A: f(a) = b, b \in B\} &= \mathcal{P}(A)\\
            \forall a \in A, \exists b \in B : f(a) = b &\implies f^{-1}(b_{i}) \cap f^{-1}(b_{i'}) = \emptyset\\
            &\implies f^{-1}(B) = \mathcal{P}(A)
        \end{align*}
        \item Is this still the case if we include in the collection the sets $f^{-1}(b)$ for $b \in B \setminus f(A)$?\\
        Depends in if $f$ is bijective, if it is, B then would just add an empty set, which would not affect the partition, but if B is not srujective, then $f^{-1}$ would be greater that $\mathcal{P}(A)$
    \end{enumerate}
\end{enumerate}
\section*{1.3.2 Injections, surjections, bijections}
\begin{enumerate}
    \item Is the floor function from $\mathbb{R}_{+}$ into $\mathbb{N}$ injective?\\
    No, because several values from $\mathbb{R}_{+}$ map to the same values in $\mathbb{N}$, An exaple is that 1.01, 1.1, 1.9 all map to 1
    \item Show that the composition of two bijections is a bijection. You may make use of results of exercises in the previous slides on injectivity and surjectivity.
    \begin{align*}
        f: A &\rightarrow B & g: B &\rightarrow C\\
        \forall a \in A,&\ \exists b \in B: f(a) = b & \forall b \in B,&\ \exists c \in C: g(b) = c\\
        f(a) = f(a') &\iff a = a' & g(b) = g(b') &\iff b = b'
    \end{align*}
    \begin{align*}
        g \circ f &\implies \forall a \in A,\ \exists c \in C: g(f(a)) = c\\
        &\implies g(f(a)) = g(f(a')) \iff a = a'\\
        &\implies g \circ f\ is\ bijective
    \end{align*}
    \item Use the equinumerosity principle to show that there is never any bijection between a finite set and any of its proper subsets.
    \begin{align*}
        \#(A) = \#(B) &\iff f: A \rightarrow B\ is\ bijective\\
        B \subset A &\implies B = \{\forall b \in B, b \in A \},\ A \setminus B \neq \emptyset\\
        B \subset A &\implies \exists a \in A: f(a) \notin B\\
        &\implies f\ is\ not\ bijective\\
        &\implies \#(A) \neq \#(B)
    \end{align*}
    \item Give an example to show that there can be a bijection between an infinite set and certain of its proper subsets.
    \begin{align*}
        A = \mathbb{N}, &B = \mathbb{N}_{+}\\
        f: A \rightarrow B &= f(a) = a + 1\\
        f(0) = 1, f(1) = 2 &... f(n) = n + 1
    \end{align*}
    \item Use the principle of comparison to show that for finite sets $A$, $B$, if there are injective functions $f : A \rightarrow B$ and $g: B \rightarrow A$, then there is a bijection from $A$ to $B$. (Hint: Consider the superposition $g \circ f$ and establish that it provides for a desired bijection).
    \begin{align*}
        f: A &\rightarrow B & g: B &\rightarrow A\\
        \forall a \in A,&\ \exists b \in B: f(a) = b & \forall b \in B,&\ \exists a \in A: g(b) = a\\
        f(a) = f(a') &\iff a = a' & g(b) = g(b') &\iff b = b'
    \end{align*}
    \begin{align*}
        g \circ f &: A \rightarrow A\\
        f \circ g &: B \rightarrow B\\
        &\implies \forall a \in A,\ \exists b \in B\\
        &\implies A\ and\ B\ are\ bijective
    \end{align*}
\end{enumerate}
\newpage

\section*{1.3.3 Pigeonhole principle}
\begin{enumerate}
    \item If a set $A$ is partitioned into n cells, how many distinct elements of $A$ need to be selected to guarantee that at least two of them are in the same cell?\\
    $n + 1$
    \item Let $K =\{1,2,3,4,5,6,7,8\}$. How many distinct numbers must be selected from $K$ to guarantee that there are two of them that sum to 9? (Hint: Let $A$ be the set of all unordered pairs $(x,y)$ with $x,y \in K$ and $x+y = 9$. Check that this set forms a partition of K and apply the preceding part of the exercise).
    \begin{align*}
        A &= \{{x, y}: x, y \in K, x + y = 9 \}\\
        A &= \{\{1, 8\}, \{2, 7\}, \{3, 6\}, \{4, 5\}\}\\
        A &= \mathcal{P}(K)\\
        \#(A) = 4 &\implies You\ need\ 5\ numbers
    \end{align*}
\end{enumerate}

\section*{1.3.4 Handy functions}
\begin{enumerate}
    \item Let $f: A \rightarrow B$ and $g: B \rightarrow C$.
    \begin{enumerate}
        \item Show that if at least one of $f$, $g$ is a constant function, then $g \circ f : A \rightarrow C$ is a constant function.
        \begin{align*}
            f\ is\ constant &\implies \exists b \in B \forall a \in A: f(a) = b\\
            &\implies f(A) = b\\
            g \circ f &\implies \exists c \in C \forall a \in A: g(f(a)) = c
        \end{align*}
        \item If $g \circ f : A \rightarrow C$ is a constant function, does it follow that at least one of $f$,$g$ is a constant function (give a verification or a counterexample).
        \begin{align*}
            g \circ f &\implies \exists c \in C \forall a \in A: g(f(a)) = c\\
            &\implies f(A) = b \vee g(B) = c\\
            &\implies\exists b \in B \forall a \in A: f(a) = b \vee \exists c \in C \forall b \in B: g(b) = c\\
            &\implies f\ is\ constant\ or\ g\ is\ constant
        \end{align*}
    \end{enumerate}
\end{enumerate}
\end{document}


\subsection{States, hashes, messages, representatives}

Light node protocol states are also sets of messages,
each message being a triple $(c, v, j)$, where:
\begin{itemize}
    \item $c$ is a (proposed) consensus value;
    \item $v$ identifies the message sender;
    \item $j$, the justification, is the {\em set of hashes} of all messages
        of the protocol state seen by the sender at the time of message
        sending.
\end{itemize}

The fact that justifications are sets of hashes allows for a simpler,
non-recursive definition of states. To begin, the type of hashes can be any
totally ordered set. In turn, a justification is a list of hashes, and
a message is a triple:

\begin{coq}
Variable hash : Type.
Variable (about_H : `{StrictlyComparable hash}).
Definition justification_type : Type := list hash.
Definition message : Type := C * V * justification_type.
\end{coq}

The total order on hashes induces a total lexicographic order on
justifications, which can be extended to messages.

We can therefore work with sorted lists of hashes as representatives
for sets of hashes, reducing equality between justifications to
syntactic equality.

We define state equality set equality, that is,
double inclusion between the sets of messages representing states.

We assume an injective function from messages to hashes:
\begin{coq}
Parameters (hash_message : message -> hash)
           (hash_message_injective : Injective hash_message).
\end{coq}
This allows us to recursively define a function \verb"hash_state" taking
states and returning sorted lists of hashes, i.e., justifications, with the
property that two justifications are equal iff they belong to
states which are equal as sets:

\begin{coq}
Lemma hash_state_injective : forall sigma1 sigma2,
  hash_state sigma1 = hash_state sigma2
  <->
  set_eq sigma1 sigma2.
\end{coq}

This allows for the following inductive definition of protocol states:

\begin{coq}
Inductive protocol_state : state -> Prop :=
| protocol_state_nil : protocol_state state0
| protocol_state_cons : forall (j : state),
    protocol_state j ->
    forall (c : C),
      valid_estimate c j ->
      forall (v : V) (s : state),
        In (c, v, hash_state j) s ->
        protocol_state (set_remove compare_eq_dec
        													(c, v, hash_state j) s) ->
        NoDup s ->
        not_heavy s ->
        protocol_state s.
\end{coq}

The above definition reads as:
\begin{itemize}
    \item a protocol state is either empty; or
    \item it is a duplicate-free state \verb|s| that does not exceed the fault tolerance threshold for which
        there exists a consensus value \verb|c|, a sender \verb|v|, and a state \verb|j| such that:
        \begin{itemize}
            \item \verb|j| is a protocol state;
            \item \verb|c| is a consensus value which the estimator agrees on for \verb|j|;
            \item \verb|(c,v, hash_state j)| belongs to \verb|s|;
            \item The state obtained from \verb|s| by removing
                \verb|(c,v, hash_state j)| is a protocol state.
        \end{itemize}
\end{itemize}


\section{Examples of Linearly Separable Sets}
In this section, we shall give several examples of linearly separable sets.  We will give a detailed description of these sets and list the web pages that contain these sets.

\subsection{L-MNIST sets}
The original MNIST data set contains the pictures of single digits. The size of each digit picture is 28*28 and the label is the digit from 0 to 9. There are 60000 training data and 10000 test data in all. The accuracy of the state-of-art CNN model applied on the MNIST is higher than 99\%. If we apply standard least square to the original MNIST data, we only get 86\% accuracy. If we apply logistic regression to the original MNIST data, we get 92\% accuracy. This means that the original MNIST is linearly separable.

This collection of linearly separable L-MNIST data set are derived from the MNIST data basis after applying some appropriate DNN model. The fully connected neural network contains an input layer, a hidden layer and an output layer. The corresponding widths are 784, 500 and 10. We use this DNN to train all the data (70000) in MNIST for 50 epochs with cross entropy as the loss function and Adam as the optimization method. Then the training accuracy reaches 100\%. We save this model and define the output of the hidden layer as the linearly separable L-MNIST data set.

To verify this data set is indeed linearly separable, we can use a fully connected neural network with input layer (500) and output layer (10) only without activation. So this is a straightforward linear combination instead of  logistic regression. Due to the weight initialization, the immediate classification accuracy may be low. But after training with this DNN for only 3 epochs, the classification accuracy for all the data in MNIST is already 100\%.  As a comparison, if we apply standard least square to the linearly separable MNIST data, we get 98.38\% accuracy.

We have also uploaded the linearly separable MNIST data set to \\
http://multigrid.org/wiki/index.php/deep-neural-network/
\subsection{CIFAR sets}


% When using TeXShop on the Mac, let it know the root document. The following must be one of the first 20 lines.
% !TEX root = ../design.tex

\chapter[Cox Proportional-Hazards]{Cox Proportional-Hazards}

\section{Introduction}
% Abstract. What is the problem we want to solve?
Proportional-Hazard models enable comparison of \textit{survival models}.
Survival models are functions describing the probability of an one-item event
(prototypically, this event is death) with respect to time.  The interval of
time before death occurs is the \textit{survival time}.   Let $T$ be a random
variable representing the survival time, with a cumulative probability function
$P(t)$.  Informally, $P(t)$ represents the probability that death has happened
before time $t$.

An equivalent formation is the \textit{survival function} $S(t)$, defined as
$S(t) \equiv 1 - P(t)$.  Informally, this is the probability that death hasn't
happened by time $t$.  The \textit{hazard function} $h(t)$ which assesses the
instantaneous risk of demise at time $t$, conditional on survival upto time $t$.
\begin{align}
    h(t) &= \lim_{\Delta t \rightarrow 0} \frac{p\left(t \le T < t + \Delta t | T \ge t \right)}
                                              {\Delta t} \\
         &= \lim_{\Delta t \rightarrow 0}  \frac{1}{\Delta t}\frac{p(T < t + \Delta t) - p(T < t)}{P(T \ge t)}
\end{align}

The relationship between $h(t)$ and $S(t)$, using $S(t) = 1 - p(T < t)$ is
\begin{align}
h(t) & = \lim_{\Delta t \rightarrow 0}  \frac{1}{\Delta t} \frac{\left(S(t + \Delta_t) - S(t)\right)}{-S(t)}\\
     & = \frac{-S'(t)}{S(t)}
\end{align}
where
$$S'(t) = \lim_{\Delta t \rightarrow 0}  \frac{S(t + \Delta t) - S(t)}{\Delta t}$$
denotes the derivative of $S(t)$.

In the simplest case, the Cox model assumes that $h(t)$ is
\begin{equation}
h(t) = e^{\alpha(t)}
\end{equation}

where exp$(\alpha(t))$ is the \textit{baseline function}, which depends only on
$t$.  However, in many applications, the probability of death may depend on more
than just $t$.  Other covariates, such as age or weight, may be important.  Let
$x_i$ denote the observed value of the  $i$th covariate, then the Cox model is
written as

\begin{equation}
h(t) = e^{\alpha(t)} e^{\beta_1 x_1 + \beta_2 x_2 + \dots + \beta_m x_m} = e^{\alpha(t)} e^{\bold{\beta^T x}}
\end{equation}
where $\beta_i$ is the coefficient associated with the $i$th covariate.

Many applications take values from multiple observations, measuring the values
of $x_i$ for each observation.  %The $j$th observation has the hazard function

%\begin{equation}
%h_j(t) = e^{\alpha(t)} e^{\beta_1 x_{j1} + \beta_2 x_{j2}+ \dots + \beta_k x_{jk}}= e^{\alpha(t)} e^{\bold{\beta^T x_j}}.
%\end{equation}

In the \textit{proportional-hazard model}, the hazard functions of two
observations $j$ and $k$ are compared. The ratio of the two is
\begin{equation}
\frac{h_j(t)}{h_k(t)} = \frac{e^{\alpha(t)} e^{\bold{\beta^T x_j}} }{e^{\alpha(t)} e^{\bold{\beta^T x_k}} } = \frac{e^{\bold{\beta^T x_j} }}{ e^{\bold{\beta^T x_k}}}
\end{equation}

The critical idea here is that the ratio of the two hazard functions is
completely independent of the baseline function.  This allows meaningful
comparisons between samples without knowing the baseline function, which may be
difficult to measure or specify.

% SECTION:  Applications
\section{Applications}\label{cox:Application}
Generally, applications start with a list of $n$ observations, each with $m$
covariates and a time of death.  From this $n \times (m+1)$ matrix, we would
like to derive the correlation between the covariates and the hazard function.
This amounts to finding the values of  $\beta$.

The values of $\beta$ can be estimated with the method of \textit{partial
likelihood}.  This method begins by sorting the observations by time of death
into a list $[t_1, t_2, \dots, t_n]$ such that $t_i \le t_j : i < j\ \forall
i,j$.  For any time $t_i$, let $R(t_i)$ be the set of observations still alive
at time $t_i$.

Given that there was a death at time $t_i$ and observation $k$ was alive prior
to $t_i$, the probability that the death happened to observation $k$  is
\begin{equation}\label{cox:equ:death-prob}
\Pr(T_k = t_i | R(t_i)) =  \frac{e^{\bold{\beta^T x_k} }}{ \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j}}}.
\end{equation}

The \textit{partial likelihood function} can now be generated as the product of conditional probabilities.
More formally,
\begin{equation}\label{cox:equ:likelihood}
\mathcal{L} = \prod_{i = 1}^n \left(  \frac{e^{\bold{\beta^T x_i} }}{ \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j}}} \right).
\end{equation}

 The log-likelihood form of this equation is
\begin{equation}\label{cox:equ:LLH}
L = \sum_{i = 1}^n \left[  \bold{\beta^T x_i} - \log\left(\sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } \right) \right].
\end{equation}

An estimation of $\beta$ can be found by simply maximizing this log-likelihood.
To maximize the likelihood, it helps to have the derivative of equation
\ref{cox:equ:LLH}, which is
\begin{equation}\label{cox:equ:LLH derivative}
\frac{\partial L}{\partial \beta_k} = \sum_{i = 1}^n \left( x_{ik} - \frac{\sum_{j \in R(t_i)} x_{jk} e^{\bold{\beta^T x_j} } }{\sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } } \right).
\end{equation}
It follows that the second derivative is
\begin{equation}\label{cox:equ:LLH second derivative}
\frac{\partial^2 L}{\partial \beta_k \beta_l} = \sum_{i = 1}^n
            \left(
                \frac{\left(  \sum_{j \in R(t_i)} x_{jk} e^{\bold{\beta^T x_j} } \right)
                            \left(  \sum_{j \in R(t_i)} x_{jl} e^{\bold{\beta^T x_j} } \right)}
                     {\left( \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } \right)^2 } -
                \frac{  \sum_{j \in R(t_i)} x_{jk}x_{jl} e^{\bold{\beta^T x_j} } }
                     {\sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } }
            \right).
\end{equation}

% Incomplete Data
\subsection{Incomplete Data}
Frequently, not every observation will have an associated time of death.
Typically, this arises when the period of observation terminates before the
entire population being studied has died.  This is known as \textit{censoring}
the data.  To account for this, an additional indicator variable is introduced
$\delta_i$, which is set to 1 if the $i$th observation has an associated time of
death, and 0 otherwise.

Incorporating this indicator variable into equation \ref{cox:equ:likelihood} gives
\begin{equation}\label{cox:equ:likelihood-censoring}
\mathcal{L} = \prod_{i = 1}^n \left(  \frac{e^{\bold{\beta^T x_i} }}{ \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j}}} \right)^{\delta_i}.
\end{equation}
The appropriate changes to the LLH function and its derivatives are trivial.

% SECTION: Implementation
\subsection{Partition and aggregation of the data to speed up}

In order to speed up the computation, we first partition the data and
aggregate each piece of the data into one big row. During the
computation, the whole big row is loaded into the memory for
processing, which speeds up the computation.

When we partition the data, we want to (1) keep the sorted descending
order of the time column, (2) make sure that each piece has
approximately the same amount of data so that the work load is even,
and (3) do this as fast as possible.

Our solution to to first sample a certain amount of the data, and then
compute the approximate break points using the sampled data. The
sampled data should be small enough to load into the memory, abd also
large enough so that the break points can be computed relatively
accurate.

After we have partitioned the data, we aggregate each partition into
one big row. The order of the data should be kept during the
aggregation.

Then we use the sequential algorithm described in the next to process
the new data row by row in the order of the time column. Since each
big row contains lots of original rows, and we deal with them all in
the memory, this can speed up the computation.

\subsection{Implementation of Newton's Method}
Newton's method is the most common choice for estimating $\beta$ by minimizing
\ref{cox:equ:likelihood} using the following update rule:
\begin{equation}
\beta_{k} = \beta_{k} - \alpha_k \left( {\nabla^2 L}^{-1} \nabla L \right)
\end{equation}
where $\alpha_k$ is a positive scalar denoting the step length in the newton
direction ${\nabla^2 L}^{-1} \nabla L$ determined using the first and second
derivative information. We would like to emphasize that the problems we are
designing this system for are those with many records and few features i.e. $n
\gg m$, thereby keeping the inverse operation on the hessian matrix relatively
cheap.

The gradient and Hessian matrix may be hard to parallelize therefore reducing an
advantage for large number of observations. To elaborate, consider equations
\ref{cox:equ:LLH derivative} and \ref{cox:equ:LLH second derivative} which are
sums with independent terms. One might think it is natural to reduce the
computational by parallelization. Efficient parallelization may be achieved if
each term could be computed independently in parallel by a set of worker tasks
and a master task could collect the output from each worker node sum them
together. However, this might not work well in this setting. To see why,
consider parallelizing equation \ref{cox:equ:LLH second derivative}.  Each
worker task is given one term in the sum, which looks like
\begin{equation}
 \frac{\left(  \sum_{j \in R(t_i)} x_{jk} e^{\bold{\beta^T x _j} } \right) \left(  \sum_{j \in R(t_i)} x_{jl} e^{\bold{\beta^T x _j} } \right)}{\left( \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } \right)^2 }   -  \frac{  \sum_{j \in R(t_i)} x_{jk}x_{jl} e^{\bold{\beta^T x_j} } }{\sum_{j \in R(t_i)} e^{\bold{\beta^T x_j} } }.
\end{equation}

Note that the sums in the numerator are summing over all the data points in the
data matrix. A similar such issue is encountered while computing the first
derivative terms as defined in \ref{cox:equ:LLH derivative}. However, we note
that this sum has a structure that allows it to be computed in linear time (with
respect to the number of data points) using the following quantities.
\begin{align}
H_{i} &=   \sum_{j \in R(t_i)} x_{j} e^{\bold{\beta^T x_j}}\\
S_{i}&=   \sum_{j \in R(t_i)} e^{\bold{\beta^T x_j}} \\
V_{i}&=   \sum_{j \in R(t_i)} x_{j}x_{j}^T e^{\bold{\beta^T x_j} }
\end{align}

Note that $H_{i}$ is a column vector with $m$ elements ( $H_{i}\in
\mathbb{R}^m$), $S_{i}$ is a scalar and and $V_{i}$ is an $m \times m$ matrix.
We can now write the first derivative of the maximum likelihood estimator,
defined in Equation \ref{cox:equ:LLH derivative} as
\begin{align}
\frac{\partial L}{\partial \beta_k} = \sum_{i = 1}^n \left( x_{i} - \frac{H_{i} }{ S_{i}}  \right)
\end{align}
while the second derivative, defined in Equation \ref{cox:equ:LLH second
derivative}, can be reduced to

\begin{align}
\frac{\partial^2 L}{\partial \beta_k \beta_l} = \sum_{i = 1}^n \left( \frac{H_{i}H_{i}^T }{ S_{i}^2 } -  \frac{V_{i}}{ S_{i} } \right)
\end{align}
Since we assume that the data points are sorted in increasing order i.e
$R(t_i) = \{i, i+1 \ldots n \}$, we can calculate the above summation as
\begin{align}
H_{i}& =   H_{i+1} +  x_{i} e^{\bold{\beta^T x_i}} \label{eq:numerator_avg}\\
S_i & = S_{i+1} + e^{\bold{\beta^T x_i}} \label{eq:denominator_avg}\\
V_i & = V_{i+1} +  \frac{H_{i}H_{i}^T }{ S_{i}^2 } -  \frac{V_{i}}{
  S_{i}} \label{eq:hessian}.
\end{align}
With this recurrence relationship, the hessian matrix and the gradient direction
can be computed in linear time.

%In addition to the computational expense of computing the Hessian, the inverse
%Hessian must be computed as well.  Unfortunately, matrix inversion is an
%expensive operation, and  cannot be parallelized.

% SECTION: Stratification Support
\section{Stratification Support}\label{cox:stratified}
A crucial component of the Cox Proportional Hazards model is the proportional hazards assumption:
The hazard for a given individual is a fixed proportion of the hazard for any
other individual in the same stratum, and the ratio of the hazards is constant
across time.

In actual use cases, the proportional hazard assumption may not be satisfied if
we use all independent variables as covariates. A stratified Cox regression
model may then be useful. It offers a way such that we can choose a subset of
independent variables as covariates while are still taking the remaining
independent variables into account. The stratified Cox regression is available
in both R~\cite{r-cox} and Stata~\cite{stata-cox}.

Stratification is used as shorthand for building a Cox model that allows for
more than one stratum, and hence, allows for more than one baseline hazard
function.  Stratification provides two pieces of key, flexible functionality for
the end user of Cox models:
\begin{enumerate}
    \item Allows a categorical variable Z to be appropriately accounted for in
    the model without estimating its predictive/associated impact on the
    response variable (i.e. without estimating Z's ``coefficient'').
    \item Categorical variable Z is predictive/associated with the response
    variable, but Z may not satisfy the proportional hazards assumption.
\end{enumerate}

To explicitly clarify how stratification differentiates from grouping support:
\begin{itemize}
    \item Grouping by a categorical column would build a completely separate
    Cox model for each value of the categorical column, where the baseline
    hazards for each value of the categorical column would be different and
    the estimated coefficient values for each explanatory variable would be
    \textbf{different} for each value of the categorical column.
    \item Stratifying by a categorical column would build a single
    common Cox model, where the baseline hazards for each value of the
    categorical column would be different, but the estimated coefficient values
    for each explanatory variable would be the \textbf{same} for each value of the stratum.
\end{itemize}

It is valuable to emphasize that all strata share all coefficients, and that the
only difference between strata is the baseline hazard.  In other words,
coefficients for all non-strata explanatory variables are identical across the
different strata.

\subsection{Estimating A Stratified Cox Model}\label{cox:estimate-stratified}
The parameter estimation is done by maximizing the product of likelihoods, each
from a stratum~\cite{stratified-ethz-slides}.

Given $n$ observations, each with $m$ covariates and each in one of $K$
strata~\footnote{Note that this does not mean that we have $K$ variables other
than the $m$ covariates, but $K$ groups classified by strata ID variables.}, let
$\mathit{ST}_{k}$ denote the set of observations in the $k$-th stratum.

Because, as an objective function, the sum of log-likelihoods is equivalent to
the product of likelihoods, according to Equation \ref{cox:equ:LLH}, we have the
log-likelihood associated with the $k$-th stratum,
\begin{equation}\label{cox:equ:obj-each-stratum}L_{k} = \sum_{i \in \mathit{ST}_{k}} \left[  \bold{\beta^T x_i} - \log\left(\sum_{j \in R_k(t_i)} e^{\bold{\beta^T x_j} } \right) \right],
\end{equation}
where $R_k(t_i)$ the set of observations in $\mathit{ST}_{k}$ and still alive at time $t_i$.

Therefore, the objective function of stratified cox regression can be expressed as
\begin{equation}\label{cox:equ:stratified-obj}
L^{\mathit{stratified}} = \sum_{k=1}^{K} L_{k} = \sum_{k=1}^{K} \sum_{i \in \mathit{ST}_{k}} \left[  \bold{\beta^T x_i} - \log\left(\sum_{j \in R_k(t_i)} e^{\bold{\beta^T x_j} } \right) \right].
\end{equation}
The appropriate changes to gradient, Hessian and censoring are trivial.

\subsection{When We Need A Stratified Cox Model?}\label{cox:diagnostic}
General practice in standard statistical packages (i.e. R, SAS) is to
make use of the Schoenfeld residuals to gauge whether or not the
proportional hazards assumption has been violated.

% To this end, we need to investigate the implementation of R's
% \texttt{cox.zph()} function for MADlib.
% There are several parametric options available in the
% \texttt{cox.zph()} function, but we prioritize implementation of
% \texttt{cox.zph()} with its default parameter settings.
% The output of \texttt{cox.zph()} contains 3 numbers for each
% covariate, scaled Schoenfeld residual, chi-square, and p-value.
% The actual formulas of computing these are still to be investigated.

The Schoenfeld residuals are centered on zero and should be
independent of time if PHA (proportional hazard assumption) is
true. Deviations from this, i.e. residuals that exhibit some trend in
time, indicate that the PHA is violated.

The Schoenfeld residuals, at the time when a failure or death were to occur, are
defined by the difference between the observed and expected covariate values at
that time. Also note that the Schoenfeld residuals are zeroes for censored
observations.
\begin{equation}
\hat{\vec{r}}_i = \vec{x}_i - E[\vec{x}_i]\ ,\mbox{ only for
}\delta_i=1\ ,
\end{equation}

To compute the expected values at time $t_{i}$, we use the probability distribution given by Equation \ref{cox:equ:death-prob}.
\begin{eqnarray}
E[\vec{x}_i] &=& \sum_{k\in \mathcal{R}(t_i)}\vec{x}_k p(k\mbox{ dies at
}t_i) \nonumber\\
&=& \frac{\sum_{k\in \mathcal{R}(t_i)}\vec{x}_k
  e^{\vec{\beta}\vec{x}_k}}{\sum_{j\in
    \mathcal{R}(t_i)}e^{\vec{\beta}\vec{x}_j} },
\end{eqnarray}
where the values of $\vec{\beta}$ are the fitted coefficients, and $\mathcal{R}(t)$ is the set of individuals that are still alive at time $t$.

\subsubsection{Scaled Schoenfeld Residuals}\label{cox:scaled-residual}
Suggested by Grambsch and Therneau \cite{grambsch1994proportional} and also
followed by statistical softwares \cite{testph}, scaling the Schoenfeld
residuals by an estimator of its variance yields greater diagnostic power. The
scaled Schoenfeld residuals is
\begin{eqnarray}
\hat{\vec{r}}^{*}_i &= [\mathit{Var}(\hat{\vec{r}}_i)]^{-1} \hat{\vec{r}}_i\\
&\approx m \mathit{Var}(\hat{\vec{\beta}}) \hat{\vec{r}}_i  ,
\end{eqnarray}
where $\mathit{Var}$ denotes a covariance matrix, and $m$ is the
number of uncensored survival times. $\hat{\vec{r}}_i$ is a length-$n$
vector, and $\mathit{Var}(\hat{\vec{\beta}})$ is a $n\times n$
matrix, where $m$ is the number of uncensored data points and $n$ is
the number of features.

\subsubsection{Transformation of Time}\label{cox:transform}
Transformation of the time values often helps the analysis of the correlation
between the scaled Schoenfeld residuals and time. Common transformation methods
include ranking, computing log, and Kaplan-Meier's method. (We don't fully
understand the last one yet.)

\footnote{The \texttt{cox.zph()} function in R allows different options for
transformation of times: \texttt{``km''}, \texttt{``rank''},
\texttt{``log''} and \texttt{``identity''}.
\texttt{``km''} stands for Kaplan-Meier's method.
\texttt{``rank''} takes the ranks of the times instead of times.
\texttt{``log''} uses the logarithm of times.
And \texttt{``identity''} does nothing and directly uses times.\cite{cox-zph}}

%To quantify whether there is a trend in the Schoenfeld residuals over
%time, we can compute the correlation between the scaled residuals and the
%ranks of the times, where the scaled residuals are given by
%\begin{equation}
%\hat{\vec{r}}_i = m \widehat{\mbox{Var}}(\vec{\beta}) \vec{r}_i\ ,
%\end{equation}
%where $m$ is the number of uncensored survival times, and
%$\widehat{\mbox{Var}}(\vec{\beta})$ is the estimated variance matrix for
%the fitted coefficients.

\subsubsection{$p$-values}
The process for computing the $p$-values of the correlation in R's
\texttt{survival} package is given
in the following.

Let $m$ be the number of uncensored data points, $n$ be the number of
features, $\hat{\vec{t}}$ be the transformed survival time, which is a
length-$m$ vector,
$\mathit{Var}(\hat{\vec{\beta}})$ be the variance matrix for the
fitted coefficients which is a $n\times n$ matrix, $\hat{\vec{R}}$
is the unscaled Schoenfeld residuals, which is a $m\times n$ matrix.
\begin{eqnarray}
\vec{w} &=& \hat{\vec{t}} - \bar{\hat{t}}\ , \mbox{ a length-}m \mbox{
  vector}\ ,\\
\vec{v} &=& m\,\vec{w}^T\hat{\vec{R}}\mathit{Var}(\hat{\vec{\beta}})\ , \mbox{
a length-}n\mbox{ vector}\\
z_i &=& \frac{1}{m\,\vec{w}^T\vec{w}}
\cdot\frac{v^2_i}{\left[\mathit{Var}(\hat{\vec{\beta}})\right]_{ii}}\
, \mbox{ for }i=1,\dots,n\ ,\\
p_i &=& 1 - \chi_1^2(z_i), \mbox{ for }i=1,\dots,n\ .
\end{eqnarray}
Here $\bar{\hat{t}}$ is the average of the transformed survival
times. $z_i$ is the z-stats for $i$-th coefficient. $p_i$ is the
p-value for $i$-th coefficient. We need a separate function to compute
$\vec{w}$, but both $\vec{v}$ and $\vec{w}^T\vec{w}$ can be computed
in an aggregate function. $z_i$ and $p_i$ can be computed in the final
function of that aggregate function. $\chi^2_1(\cdot)$ is the
chi-square function with the degree of freedom being $1$.

\subsection{Implementation}
We can use the iteration equations Eqs. (\ref{eq:numerator_avg},
 \ref{eq:denominator_avg}, \ref{eq:hessian}) to compute the residuals and the
hessian, which is needed by the variance matrix.

The current implementation uses an ordered aggregate on the data to
compute these quantities, which is not in parallel. To enable a distributed
solution we can use the ``GROUP BY'' functionality provided by SQL to enable the
independent computation of the log-likelihood function in each distributed segment
corresponding to each strata (`transition' function). These values can then be added
across the segments (`merge' function), with the gradient for the parameter computed
on the final sum (`final' function).

\subsection{Resolving Ties}
In ``coxph\_train'', Breslow method is used to resolve ties, which uses
as the partial likelihood~\cite{hosmer2011applied}
\begin{eqnarray}
  L &=& \sum_{i=1}^{m}\left[ \vec{\beta}^{T} \vec{x}_{i+} -
    d_i \log\left( \sum_{j \in R(t_i)} e^{\vec{\beta}^T
        \vec{x}_j}\right) \right]\\
  &=& \sum_{i=1}^{m}\left[ \vec{\beta}^{T} \vec{x}_i - \log\left( \sum_{j \in R(t_i)} e^{\vec{\beta}^T
        \vec{x}_j}\right)\right]_{+}
\end{eqnarray}
where $d_i$ is the number of observations that have the same $t_i$,
and $x_{i+}$ is the sum of the covariants over all observations that
have the same $t_i$. $m$ is the number of distinct survival
times. Here $[\cdot]_{+}$ means sum over all observations with the
same survival time.

\subsection{Robust Variance Estimators}
In MADlib, we implement robust variance estimator devised by Lin and Wei~\cite{lin1989robust}.
With our notations above, let
\begin{eqnarray*}
    \vec{W}_{i}
    &=& \delta_i \cdot \left[ \vec{x_i} - \frac{\left( \sum_{l: t_l \ge t_i} e^{\vec{\beta}^T \vec{x_l}} \vec{x_l} \right)}{\sum_{l: t_l \ge t_i} e^{\vec{\beta}^T \vec{x_l}}} \right]
    - \sum_{j: t_j \le t_i} \left\{ \delta_j \cdot \frac{e^{\vec{\beta}^T \vec{x_i}}}{\sum_{k: t_k \ge t_j} e^{\vec{\beta}^T \vec{x_k}}}
    \cdot \left[ \vec{x_i} - \frac{\left( \sum_{k: t_k \ge t_j} e^{\vec{\beta}^T \vec{x_k}} \vec{x_k} \right)}{\sum_{k: t_k \ge t_j} e^{\vec{\beta}^T \vec{x_k}}} \right] \right\}.\\
    &=& \delta_i \cdot \left[ \vec{x_i} - \frac{\vec{H}_i}{S_i} \right]
    - \sum_{j: t_j \le t_i} \left\{ \delta_j \cdot \frac{e^{\vec{\beta}^T \vec{x_i}}}{S_j}
    \cdot \left[ \vec{x_i} - \frac{\vec{H}_j}{S_j} \right] \right\}.\\
    &=& \delta_i \cdot \left[ \vec{x_i} - \frac{\vec{H}_i}{S_i} \right]
    - \sum_{j: t_j \le t_i} \delta_j \cdot \frac{e^{\vec{\beta}^T \vec{x_i}}}{S_j} \vec{x_i}
    + \sum_{j: t_j \le t_i} \delta_j \cdot \frac{e^{\vec{\beta}^T \vec{x_i}}}{S_j} \frac{\vec{H}_j}{S_j}.\\
\end{eqnarray*}
where
\begin{eqnarray*}
H_i &=&  \sum_{l: t_l \ge t_i} e^{\vec{\beta}^T \vec{x_l}} \vec{x_l}\\
S_i &=& \sum_{l: t_l \ge t_i} e^{\vec{\beta}^T \vec{x_l}}
\end{eqnarray*}

Let
\begin{eqnarray*}
    A_i &=& \sum_{j: t_j \le t_i} \frac{\delta_j}{S_j},\\
    \vec{B}_i &=& \sum_{j: t_j \le t_i} \frac{\delta_j \vec{H}_j}{S_j^2},\\
\end{eqnarray*}
we have
\[
\vec{W}_i = \delta_i \cdot \left[ \vec{x_i} - \frac{\vec{H}_i}{S_i} \right] - e^{\vec{\beta}^T \vec{x_i}} A_i \vec{x_i} + e^{\vec{\beta}^T \vec{x_i}} \vec{B}_i.
\]
And the recursions of $A_i$ and $\vec{B}_i$ are given if we assume time $t_i$ is
sorted ascending order
\begin{eqnarray*}
    A_i &=& A_{i-1} + \frac{\delta_i}{S_i}\\
    \vec{B}_i &=& \vec{B}_{i-1} + \frac{\delta_i \vec{H}_i}{S_i^2}
\end{eqnarray*}

The meat part of the sandwich estimator is
\begin{equation}
  \mathbb{M} = \sum_{i=1}^n\vec{W}_i\vec{W}_i^T\ .
\end{equation}

\subsection{Clustered Variance Estimation}

The data has multiple clusters $m=1,\dots,K$, and the meat part is
\begin{equation}
  \mathbb{M} = \mathbb{A}^T\mathbb{A}\ ,
\end{equation}
where the matrix $\mathbb{A}$'s $m$-th row is given by
\begin{equation}
  \mathbb{A}_m = \sum_{i\in G_m} \vec{W}_i\ .
\end{equation}
Here $G_m$ is the set of rows of that belong to the $m$-th cluster.

With stratification, we take into account of the strata only when we
compute $\vec{H}_i$, $S_i$, $A_i$, $\vec{B}_i$ and $\vec{W}_i$. The
calculations of $\mathbb{A}_m$ and $\mathbb{M}$ only need to group by
the clustered variables and the strata variables are irrelavent.

\section{How to prevent under/over -flow errors ?}

A problem that is not mentioned above but appears in the real
applications of the CoxPH training model is the under-flow or
over-flow errors. We have exponential functions in the computation,
and it is very easy to get under-flow or over-flow errors if the
coefficients become too small or large at certain steps.

We use the same method as R's "survival" package to deal with the
possible under-flow/over-flow errors. This methods contains three
parts, which makes the algorithm described above even more
complicated:

(1) center and scale the independent variables.
\begin{equation}
    x_i \rightarrow \frac{x_i - E[x]}{E[\vert x_i - E[x] \vert]}
\end{equation}

(2) Estimate the maximum possible value of all the coefficients using
the coefficients computed from the first iteration.
\begin{equation}
    \beta_k^{(max)} = 20 \, \sqrt{\frac{h_{kk}}{\sum_i \delta_i}}\ ,
\end{equation}
where $\beta_k^{(max)}$ is the estimate of the possible maximum value
of the coefficient $\beta_k$, $h_{kk} = \partial^2 L / \partial
\beta_k^2$ is the diagonal elements of the Hessian matrix, and
$\delta_i=0,1$ is the censoring status of the records.

During the computation, whenever the coefficient $\vert \beta_k \vert
> \beta_k^{(max)}$, we set $\beta_k \rightarrow \mbox{sign}(\beta_k)
\beta_k^{(max)}$.

The authors of "survival" package explains in
http://stat.ethz.ch/R-manual/R-devel/RHOME/library/survival/doc/sourcecode.pdf
why they use such an estimate for the coefficients:

"We use a cutpoint of $\beta * \mbox{std}(x) < 23$
where the standard deviation is the average standard deviation of $x$ within a risk
set. The rationale is that $e^{23}$ is greater than the current world
population, so such a coefficient corresponds to a between subject
relative risk that is larger than any imaginable."

In their implementation, they also use $1/h_{kk}$ as an approximation
to $\mbox{std}(x)$. And besides $23$, they also use $20$ in the
estimate.

(3) Although (1) and (2) stabalize the computation, it is still not
enough. Step-halving method is used. Whenever the current iteration's
log-likelihood is smaller than that of previous iteration, we accept
the coefficients as
\begin{equation}
    \beta_k = \frac{1}{2}(\beta_k^{new} + \beta_k^{old})
\end{equation}

(4) The stopping threshold is
\begin{equation}
    1 - \frac{L_{new}}{L_{old}} < \mbox{threshold}\ .
\end{equation}

\section{Marginal Effects}

See \ref{sub:marginal_effects} for an introduction to marginal effects
(all notations below are the same as those defined in \ref{sub:marginal_effects}).
We implement the default algorithm used by Stata 13.1 for computing the
marginal effects. Note that older versions of Stata may use a
different default algorithm to compute values different from MADlib.

\subsection{Basic Formulation}

The relative hazard ratio for $i$-th record is given by
\begin{equation}
    h_i = h_0 \exp \left( \vec{\beta}\cdot\vec{f} \right)\ ,
\end{equation}
where $h_0$ is the baseline and both $\vec{\beta}$ and
$\vec{f}$ are vectors.  Here we use the indices
$i$ or $j$ for the data records, and $a$ or $b$ for the indices of
covariate terms in $\vec{\beta}\cdot \vec{f}(\vec{x}_i)$. And we will use
$u$ or $v$ to denote the indices of $\vec{x}$.

The value of the baseline $h_0$ is arbitrary and difficult to compute. Strata
ignores the baseline hazard value for the computation of marginal effect. For
MADlib, we follow the same principle and ignore the baseline (i.e. set baseline
as 1) to compute hazard value.

Thus the marginal effect corresponding to variable $x_k$ is computed as,
\begin{align*} \label{eq:me1}
    \mathit{ME}_{k} & = \pder[h_i]{x_k} \\
           & =  \pder[e^{\vec{\beta} \vec{f}}]{x_k} \\
           & = e^{\vec{\beta} \vec{f}} \vec{\beta} \pder[f]{x_k}.
\end{align*}
Vectorizing the above equation (similar to \ref{ssub:logistic_regression}) gives
\begin{equation*}
  \mathit{ME} = e^{\vec{\beta} \vec{f}} J^T \vec{\beta},
 \end{equation*}
where $J$ is defined in \ref{sub:marginal_effects_for_regression_methods}.

Censoring status and stratification are irrelavant to the marginal effect
calculation and can be ignored.

\subsection{Categorical variables}

For categorical variables, we compute the discrete difference as described in
\ref{sub:categorical_variables}.
The discrete difference with respect to $x_k$ is given as
\begin{align*}
    \mathit{ME_k} &= h^{set} - h^{unset} \\
                  &= e^{\vec{\beta}\vec{f^{set}}} - e^{\vec{\beta}\vec{f^{unset}}}
\end{align*}

\subsection{Standard Error}

As has already been described in \ref{sub:standard_errors}, the method to compute
the standard errors is
\begin{equation}
    \mbox{Var}(ME) = \mathbb{S}\, \mbox{Var}(\vec{\beta})\,
    \mathbb{S}^T,
\end{equation}
where the matrix $\mathbb{S}$, computed as the partial derivative of
the marginal effect over all the coefficients, is a $M\times N$ matrix $S_{mn}
= \pder[\mathit{ME}_m]{\beta_n}$

\begin{align*}
    S_{mn} & = \pder[\mathit{ME}_m]{\beta_n} \\
           & = \pder{\beta_n} \left( e^{\vec{\beta} \vec{f}} \vec{\beta} \pder[\vec{f}]{x_m}  \right)\\
           & = e^{\vec{\beta} \vec{f}} \pder{\beta_n}\left(\vec{\beta} \pder[\vec{f}]{x_m} \right) +
                \pder[e^{\vec{\beta} \vec{f}}]{\beta_n} \vec{\beta} \pder[\vec{f}]{x_m} \\
           & = e^{\vec{\beta} \vec{f}} \pder[f_n]{x_m} +
                e^{\vec{\beta} \vec{f}} \cdot f_n \cdot \vec{\beta} \pder[\vec{f}]{x_m} \\
           & = e^{\vec{\beta} \vec{f}} \left( \pder[f_n]{x_m} +
                  f_n \cdot \vec{\beta} \pder[\vec{f}]{x_m} \right).
\end{align*}

Vectorizing this equation to express for the complete matrix,
\begin{equation*}
  \vec{S} = e^{\vec{\beta} \vec{f}} \left(J^T +
                (J^T \vec{\beta}) \vec{f}^T \right).
\end{equation*}


\section{Deep Water Wave Propagation}
This simulates the free propagation of a sinusoidal wave in deep water. The initial condition is still water in a ``large'' box with uniform depth. The wave is generated from the left boundary and propagates to the right. The depth on the right boundary is set as time dependent function
\begin{equation}
h(t) = A\sin{\frac{2\pi t}{\lambda}}
\end{equation}
with $u=v=0$. Here $A$ is the amplitude of the generating wave, $t$ is time variable, and $\lambda$ is the wave length as well as the period of the generating wave.

Analytically, the wave should travel through the domain without deformation. Lower-order-accuracy algorithms may result in undue wave dampening with the mesh size used in the current problem. This can have practical implications e.g. for tsunami propagation problems, and is usually dealt with by using second-order accurate methods. Alternatively you can refine the mesh until the dampening becomes insignificant, but this may be computationally expensive in realistic problems.

This example can also illustrate difficulties with radiation-type boundary conditions where the wave exits the domain (of course, for this problem, we could do that by exploiting the analytical solution - but this is not possible for general wave propagation problems). This will most obviously affect the right edge of the domain, but its effects will ultimately be felt throughout.

\subsection{Results}
In this test, we consider $A=1$ and $\lambda=300$.
Figure~\ref{fig:stagewave} shows the time-evolution of the water elevation at three points in the domain. Ideally these time series should show the wave propagating without deformation or attenuation (i.e. the wave has the same shape, amplitude, period, mean water level etc. at each point).
\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{wave_atten.png}
\caption{Stage over time at 3 points in space}
\label{fig:stagewave}
\end{center}
\end{figure}

The corresponding momentums of Figure~\ref{fig:stagewave} are shown in Figures~\ref{fig:xmom} and~\ref{fig:ymom}.
\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{xmom.png}
\caption{Xmomentum over time at 3 points in space}
\label{fig:xmom}
\end{center}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{ymom.png}
\caption{Ymomentum over time at 3 points in space}
\label{fig:ymom}
\end{center}
\end{figure}

\endinput