% Created 2016-02-27 Sat 15:28
% -----------------------------*- LaTeX -*------------------------------
\documentclass[12pt]{report}
\usepackage{scribe_hgen486}
\begin{document}

\scribe{Nicholas Knoblauch}		% required
\lecturenumber{11}			% required, must be a number
\lecturedate{February 9}		% required, omit year
\lecturer{John Novembre}

\maketitle

% please leave this comment
\framebox[.95\textwidth]{\parbox{.93\textwidth}{ {{\bf Note:}} These
lecture notes are still rough, and have only have been mildly
proofread.  }}
\vspace*{.1in}

% feel free to delete content below this line
% ----------------------------------------------------------------------

% please leave this comment
\tolerance=1000
\providecommand{\alert}[1]{\textbf{#1}}

\title{HMMLec}
\author{nwknoblauch}
\date{\today}

\section{Continuous Time Markov Chains}
\label{sec-1}
\subsection{Differential equations that lead to the poisson distribution}
\label{sec-1-1}

We can describe the probability that the number of events that have occured by time $t$ as $N_t$, and can write the probability as:
$P(N_t=i)=P_i(t)$

With the rate parameter $\lambda$, the probability that $N_t=i$ at $t+h$ is given by:

$P_i(t+h)=P_{\mathrm{i-1}}(t)(\lambda h + o(h)) + P_i(t)(1-\lambda h + o(h)) $

Here we are summing over the probability of two scenarios.  The first represents that $N_t=i-1$ and that there was another event in time $h$.  The second is that $N_t=i$ and that there was no event in time $h$. For small vaues of $h$ we can ignore the probabilty of two steps in time $h$.

The limit as $h$ goes to 0 is: $\frac{P_i(t+h)-P_i(t)}{h}=\frac{d}{dt}P_i(t)=P_{i-1}(t)(\lambda h + o(h)) + P_i(t)(1-\lambda h + o(h))-P_i(t)$

Cancel the one and divide by $h$ to get:

$(\lambda )P_{i-1}(t) -P_i(t) P_0(t+h)=P_0(t)(1-\lambda h + o(h))$

 $\frac{d P_0(t)}{dt}= -\lambda P_0(t)$

The solution to this differential equation is:
$P_0(t)= e^{\mathrm{-\lambda t}}$ plus some constant

$\frac{P_i(t)}{dt}=\lambda P_0(t) -\lambda P_i(t)=\lambda e^{-\lambda t} -\lambda P_1(t)$

To get the solution for $P_1$ we use an integrating factor

$\frac{d P_1(t)}{dt} + \lambda P_1(t) = \lambda e^{-\lambda t}$

$\int_0^T e^{-\lambda t} \frac{d P_1(t)}{dt} + \lambda e^{-\lambda t} P_1(t) = \int_0^T \lambda$

Remembering the product rule: $(fg)'=f'g+g'f$

$$ \int_0^T \frac{d}{dt} = \int_0^T \frac{d}{dt} ( e^{\lambda t} P_1(t)) = \int_0^T \lambda dt = P_1(t)=\lambda t e^{-\lambda t}$$

We can keep iterating:
$P_2(t)= \frac{(\lambda t)^2}{2} e^{-\lambda t}$
And eventually we have
$P_n(t) = \frac{(\lambda t)^n}{n!} e^{-\lambda t}$

This is the poisson process
\subsection{A More General Process, the Pure Birth Process}
\label{sec-1-2}

More general than the poisson process is a ``birth process'', or ``pure birth process''

In a pure birth process there is a state dependent rate of arrival:

$\frac{d}{dt} P_i(t) = \lambda_{i-1} P_{i-1}(t) -\lambda_i P_i(t)$

The poisson process is a special case where $\lambda_i$ is constant.

Another example is the \textbf{linear birth process}, where the $\lambda_i=i \lambda$ (This is also known as a Yule process)

$P_j(t) = {{j-1}\choose{k-1}} e^{-\lambda t} (1-e^{\lambda t})^{k-i}$ where $k$ is the starting size.  This is a negative binomial distribution (and is a transient solution)

Another major class is a birth-death process.  We have a set of birth rates and a set of death rates. ($\lambda$ from 0 to infinity, and $\mu$ from 1 to infinity)

$\frac{d}{dt} P_i(t) = \lambda_{i-1} P_{i-1}(t) + \mu_{i-1}P_{i+1}(t) - (\lambda_i + \mu_i)P(i)(t)$

There is a linear birth-death process $\lambda_i=i \lambda$ and $\mu_i = i\mu$

There is also linear birth-death with immigration

$\lambda_i = i \lambda + \theta$
$\mu_i = i \mu$
\subsection{Continuous version of the Markov property}
\label{sec-1-3}

Independence of the past before the immediate past
$P(X(t+s)=j|X(s)=i,X_u=x(u),0 \leq u < s ) = P(X(t+s)=j|X(s)=i)$
\subsection{Rate matrix $Q$}
\label{sec-1-4}

We have been thinking about going from 0 to 1, but we can think in general about going from $i$ to $j$
Defining the generator matrix or Rate Matrix $Q$
$P_{i,j}(h) = q_{ij} h + o(h)$
$P_{i,i}(h)= 1- v_i h + o(h)$ Where $v_i = \sum_j q_{ij}$

If we define $q_{ii}=-v_i$ Then $Q = {q_{ij}}$

Inter-event times depend on $i$ and they are exponential with rate $v_i$
$P_{ij}=\frac{q_{ij}}{v_i}$
This looks like a discrete markov chain. There is an idea that we have a discrete markov chain embedded in a continuous markov chain
\subsubsection{$Q$ matrix for poisson}
\label{sec-1-4-1}

For the poisson process, $Q$ is infinite in both directions.
The diagonal has $-\lambda$ along the diagonal. One to the right of the diagonal is $\lambda$, the rate at which we arrive at the next state. (There is no return to previous states in the poisson process)

\subsubsection{$Q$ matrix for the birth-death process}
\label{sec-1-4-2}

Matrix goes on infinitely in both directions.  Along the diagonal we have $\lambda0$ for 0,0 followed by $-(\mu_k+\lambda_k)$ at $k,k$  at $k,k-1$ we have $\mu_k$ and at $k,k+1$ we have $\lambda_k$
For $P$, we have $\frac{\mu_k}{\lambda_k+\mu_k}$, at $k,k-1$.  At $k,k+1$ we have $\frac{\lambda_k}{\lambda_k+\mu_k}$, and at the diagonal we have 1 minus the two off diagonals

$\frac{P_{ij}(t)}{dt} = v_j P{ij}(t) + \sum_{k \neq j} P_{ik}(t) q_{kj}$ This is the probability of going from $i$ to $j$, plus the probability of going form $i$ to $k$ and then going to $j$

$\frac{d P_t}{dt} = P_t Q$
$P_t = e^{Qt}$

If $Q$ is diagonalizable, then we can write $Q t =U D U^{-1}$ where $D$ is a diagonal matrix, then $e^{Qt} = Ue^{D}U^{-1}$

There are a branch of methods called Krylov methods for exponentiating matrices.
\subsection{What about stationary distributions?}
\label{sec-1-5}

\subsubsection{The Global Balance Equations}
\label{sec-1-5-1}

Define $P_i$ as the stationary probability of being in state $i$ which is the limit as $t$ goes to infinity of $P_{ij}(t)$
$v_j P_j = \sum_k q_{kj} P_k$
This equation describes a situation where the rate out of state $j$ (weighted by the probability of being in state $j$, or the flux) is  equal to the flux into state $j$

If a Continuous Time Markov Chain is time reversible, then $P_i q_{ij} = P_j q_{ji}$ and it satisfies the local balance equations.  The flux from $i$ to $j$ is equal to the flux from $j$ to $i$

What is the flux out for state 0?

\begin{center}
\begin{tabular}{rll}
 State  &  Flux out                   &  Flux in                                    \\
\hline
     0  &  $\lambda_0 P_0$            &  $\mu_1 P_1$                                \\
     1  &  $(\lambda_1 + \mu_1) P_1$  &  $\lambda_0 P_0 + \mu_2 P_2$                \\
     2  &  $(\lambda_2 + \mu_2) P_2$  &  $\lambda_1 P_1 + \mu_3 P_3$                \\
     n  &  $(\lambda_n + \mu_n) P_n$  &  $\lambda_{n-1} + \mu_{n+1} P_{n+1}$  \\
\end{tabular}
\end{center}

These all have to be equal

$\lambda_0 P_0 = \mu_1 P_1$
$\lambda_1 P_1 = \mu_2 P_2$
$\lambda_n P_n= \mu_{n+1} P_{n+1}$

We can start solving everything in terms of $P_0$

$P_1 = \frac{\lambda_0 P_0}{\mu_1}$
$P_3 = \frac{\lambda_2}{\mu_3} P_2$
$P_n = \frac{\lambda_{n-1} ... \lambda_1}{\mu_n ... \mu_1} P_0$

We know that $1= P_0 + \sum_{n=1}^\infty P_n P_0$

in the linear birth-death model
$P_n = \frac{\lambda}{\mu}^n (\frac{1}{1+\sum_{i=1}{\infty} \frac{\lambda}{\mu}^n}$

Even though this is an infinte sum, it turns out that:
$P_n= \frac{\lambda}{\mu}^n (1-\frac{\lambda}{\mu})$

We know this because $\sum_{n=1}^\infty p(1-p)^{n-1} = 1$

\end{document}


\section{Test Description and Success Criteria}
This test is located in \texttt{simulation/dynamics/sphericalPendulum/UnitTest/\newline
test\_sphericalPendulum.py}. In this integrated test there are two spherical pendulums connected to the spacecraft hub.  There are two scenarios being tested: one with the time step being 0.01 and one with the time step 0.001. Both tests should show energy and momentum conservation but the 0.001 should show less integration error.

\section{Test Parameters}

To validate and test the code a simulation with two spherical pendulums has been run using the following input parameters:
\begin{align}
\begin{split}
l_1=0.3 m, \qquad m_1=20 kg, \qquad \dot{\varphi}_{1,0}=0.01 rad/s, \qquad \dot{\vartheta}_{1,0}= 0.05 rad/s \\
\bm{d}_1=
\begin{bmatrix}
0.1 \\
0.1 \\
0.1\\
\end{bmatrix} m
\qquad
\qquad
\bm{\hat{p}}_{0_1,1}=
\begin{bmatrix}
\sqrt{2}/2 \\
0 \\
\sqrt{2}/2\\
\end{bmatrix}
\qquad
\bm{\hat{p}}_{0_1,2}=
\begin{bmatrix}
0 \\
1 \\
0\\
\end{bmatrix}
\qquad
\bm{\hat{p}}_{0_1,3}=
\begin{bmatrix}
-\sqrt{2}/2 \\
0 \\
\sqrt{2}/2\\
\end{bmatrix}
\end{split}
\end{align}

\begin{align}
\begin{split}
l_2=0.4 m, \qquad m_2=40 kg, \qquad \dot{\varphi}_{2,0}=0.1 rad/s, \qquad \dot{\vartheta}_{2,0}= 0.5 rad/s \\
\bm{d}_1=
\begin{bmatrix}
0.1 \\
0.1 \\
0.1\\
\end{bmatrix} m
\qquad
\bm{\hat{p}}_{0_2,1}=
\begin{bmatrix}
1 \\
0 \\
0\\
\end{bmatrix}
\qquad
\bm{\hat{p}}_{0_2,2}=
\begin{bmatrix}
0 \\
1 \\
0\\
\end{bmatrix}
\qquad
\bm{\hat{p}}_{0_2,3}=
\begin{bmatrix}
0 \\
0 \\
1 \\
\end{bmatrix}
\end{split}
\end{align}

\begin{table}[htbp]
	\caption{Error Tolerance - Note: Relative Tolerance is $\textnormal{abs}(\frac{\textnormal{truth} - \textnormal{value}}{\textnormal{truth}}$)}
	\label{tab:errortol}
	\centering \fontsize{10}{10}\selectfont
	\begin{tabular}{| c | c |} % Column formatting,
		\hline
		Test   & Relative Tolerance \\
		\hline
		Energy and Momentum Conservation & 1e-8 \\
		\hline
	\end{tabular}
\end{table}

\section{Test Results}

The following figures show the conservation of the quantities described in the success criteria for each scenario. The conservation plots are all relative difference plots. All conservation plots show integration error which is the desired result. In the python test these values are automatically checked therefore when the tests pass, these values have all been confirmed to be conserved. An additional note: the angular momentum plots are plotting the change in the components of the angular momentum vector in the inertial frame. The individual components are not labeled because the goal is for each component to show conservation therefore the individual components do not have separate information needing to be specified.

\subsection{Time Step = 0.01}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInOrbitalAngularMomentumOneHundredth.pdf}}
	\caption{Change in Orbital Angular Momentum Time Step = 0.01}
	\label{fig:ChangeInOrbitalAngularMomentumTimeStep01}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInOrbitalEnergyOneHundredth.pdf}}
	\caption{Change in Orbital Energy Time Step = 0.01}
	\label{fig:ChangeInOrbitalEnergyTimeStep01}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInRotationalAngularMomentumOneHundredth.pdf}}
	\caption{Change In Rotational Angular Momentum Time Step = 0.01}
	\label{fig:ChangeInRotationalAngularMomentumTimeStep01}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInRotationalAngularMomentumOneHundredth.pdf}}
	\caption{Change In Rotational Energy Time Step = 0.01}
	\label{fig:ChangeInRotationalEnergyTimeStep01}
\end{figure}
\clearpage

\subsection{Time Step = 0.001}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInOrbitalAngularMomentumOneThousandth.pdf}}
	\caption{Change in Orbital Angular Momentum Time Step = 0.001}
	\label{fig:ChangeInOrbitalAngularMomentumTimeStep001}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInOrbitalEnergyOneThousandth.pdf}}
	\caption{Change in Orbital Energy Time Step = 0.001}
	\label{fig:ChangeInOrbitalEnergyTimeStep001}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInRotationalAngularMomentumOneThousandth.pdf}}
	\caption{Change In Rotational Angular Momentum Time Step = 0.001}
	\label{fig:ChangeInRotationalAngularMomentumTimeStep001}
\end{figure}
\begin{figure}[htbp]
	\centerline{
		\includegraphics[width=0.8\textwidth]{AutoTeX/ChangeInRotationalAngularMomentumOneThousandth.pdf}}
	\caption{Change In Rotational Energy Time Step = 0.001}
	\label{fig:ChangeInRotationalEnergyTimeStep001}
\end{figure}
\clearpage


%!TEX root = ../notes.tex
\section{February 2, 2022}
\subsection{Inverses mod \texorpdfstring{$m$}{m}}
\recall Last time, we showed in \cref{existence-of-inverse} that there exists an integer $b$ with with $a\cdot b\equiv 1\mod m$ iff $\gcd(a, m) = 1$.

\begin{claim}
    We further claim that if such a $b$ exists, then it is unique mod $m$.

    That is, if we have
    \begin{align*}
        a\cdot b_1\equiv 1\pmod{m} \\
        a\cdot b_2\equiv 1\pmod{m}
    \end{align*}
    then we have that $b_1\equiv b_2\pmod{m}$.
\end{claim}
\begin{proof}
    We consider $b_1 ab_2$. We have
    \[b_2\equiv (b_1 a)b_2 = b_2(a b_2) \equiv b_2\]
    all taking mod $m$.
\end{proof}

How, then, could we compute this inverse $b$ efficiently?

Recall that last class, we used the extended Euclidean algorithm to compute the linear combination of $a$ and $m$ efficiently,
\begin{align*}
    1 & = a\cdot u + m\cdot v          \\
      & \equiv a\cdot \boxed{u}\mod{m}
\end{align*}
where $u$ is $b$.

\subsection{Modular Arithmetic \emph{continued}}
\begin{definition}[Ring of Integers mod $m$]
    $\ZZ/m\ZZ = \{0, 1, 2, \dots, m-1\}$ with operations $+, -, \times \pmod{m}$.
\end{definition}
\begin{example}
    $\ZZ/4\ZZ = \{0, 1, 2, 3\}$. We have the following operation tables for $\ZZ/4\ZZ$:
    \[\begin{array}[]{c|cccc}
            + & 0 & 1 & 2 & 3 \\ \hline
            0 & 0 & 1 & 2 & 3 \\
            1 & 1 & 2 & 3 & 0 \\
            2 & 2 & 3 & 0 & 1 \\
            3 & 3 & 0 & 1 & 2 \\
        \end{array}\qquad
        \begin{array}[]{c|cccc}
            \times & 0 & 1 & 2 & 3 \\ \hline
            0      & 0 & 0 & 0 & 0 \\
            1      & 0 & 1 & 2 & 3 \\
            2      & 0 & 2 & 0 & 2 \\
            3      & 0 & 3 & 2 & 1 \\
        \end{array}\]
\end{example}

\begin{definition}[Group of Units mod $m$]
    We have the set of units in $\ZZ/m\ZZ$ as
    \begin{align*}
        (\ZZ/m\ZZ)^\times & = \{a\in \ZZ/m\ZZ\mid \exists b \text{s.t. }a\cdot b \equiv 1\} \\
                          & = \{a\in \ZZ/m\ZZ\mid \gcd(a, m) = 1\}
    \end{align*}
\end{definition}
\begin{example}
    $(\ZZ/4\ZZ)^\times = \{1, 3\}$.
\end{example}

\begin{definition}[Euler Totient Function]
    We have
    \begin{equation*}
        \varphi(m) = \# (\ZZ/m\ZZ)^\times
    \end{equation*}
    which counts the number of units modulo $m$.
\end{definition}
\begin{example}
    $\varphi(4) = 2$.
\end{example}

Let's investigate the properties of units. Let's say $a_1, a_2$ are units. Which of the following have to be units?

\begin{table}[h]
    \centering
    \renewcommand\arraystretch{1.7}
    \begin{tabular}{c|l}
                       & Does this have to be a unit?                                                \\ \hline
        $a_1\cdot a_2$ & \begin{minipage}[t]{0.7\textwidth}
                             \ul{Yes}! \vspace{0.5em}

                             Since $\gcd(a_1, m) = 1$ and $\gcd(a_2, m) = 2$
                             so we have $\gcd(a_1a_2, m) = 1$. We also have $a_1b_1\equiv 1\mod m$
                             and $a_2b_2\equiv 1\mod m$,
                             we have $(a_1a_2)(b_2b_1)\equiv 1\mod m$.
                             \vspace{0.5em}
                         \end{minipage} \\ \hline
        $a_1 + a_2$    & \ul{No}. We have counterexample $m = 4$: $1 + 1$ is not a unit.             \\ \hline
        $a_1 - a_2$    & \ul{Also no}. For any $a$, $a - a = 0$ which is never a unit.
    \end{tabular}
    \renewcommand\arraystretch{1.05}
\end{table}

\begin{definition}[Prime Number]
    An integer $n\geq 2$ is \ul{prime} if its only (positive) divisors are $1$ and $n$.
\end{definition}
\begin{example}
    Numbers like $2, 3, 5, 7, 11, 12, \dots$.
\end{example}

What if $m$ is a prime number? Then we have
\[(\ZZ/m\ZZ)^\times = \{1, 2, \dots, m-1\}\]
so we can divide by elements of $\ZZ/m\ZZ$, just like in $\QQ, \RR, \CC$. We can divide by any nonzero element of $\ZZ/m\ZZ$. We call these \ul{fields}!

\subsection{Fast\emph{ish} Powering}
\begin{problem*}
    How might we compute $g^a\mod m$?
\end{problem*}
A na\"ive solution might be
\begin{lstlisting}[language=Python]
def pow_mod(g, a, m):
    return g ** a % m
\end{lstlisting}
What if we tried to compute \textsf{pow\_mod(239418762304, 12349876234, 12394876123482783641)} or something of the like? Something like this\dots

\begin{center}
    \includegraphics[height=5cm]{images/broken_laptop.png}
\end{center}

We could do something a bit more clever, like taking a mod every time we multiply:
\begin{lstlisting}[language=Python]
def pow_mod(g, a, m):
    p = 1
    for i in range(a):
        p = (p * q) % m
    return p
\end{lstlisting}

Yet we \emph{still} couldn't do \textsf{pow\_mod(239418762304, 12349876234, 12394876123482783641)} since that takes the amount of time proportional to \textsf{a}\footnote{Which can become big\dots}.

\begin{example}
    Let's try to compute $3^{37}$ by hand.
    \begin{align*}
         & 3^1                                   & \equiv & \ 3  \mod{100} \\
         & 3^2                                   & \equiv & \ 9  \mod{100} \\
         & 3^4 = (3^2)^2 =                       & \equiv & \ 81 \mod{100} \\
         & 3^8 = (3^4)^2 = 81^2 = 6561           & \equiv & \ 61 \mod{100} \\
         & 3^{16} = (3^8)^2 \equiv 61^2 = 3721   & \equiv & \ 21 \mod{100} \\
         & 3^{32} = (3^{16})^2 \equiv 21^2 = 441 & \equiv & \ 41 \mod{100}
    \end{align*}
    Since $37 = 32 + 4 + 1$, we can simply do
    \[3^{37} = 3^{32} \cdot 3^{4} \cdot 3^{1} = 41\cdot 81\cdot 3=1863\equiv 63 \mod{100}\]
\end{example}


% !TEX root = thesis.tex

\chapter{Incremental construction}
\label{ch:incremental-construction}

One more method to the extrusion of \refch{ch:extrusion}, \emph{incremental construction} exploits the property that an $n$-cell can be defined based on a set of $(n-1)$-cells known to form its complete (closed) boundary.
In practice, defining an $n$-cell in this manner is significantly more complex than doing so based on extrusion.
However, unlike extrusion it permits the creation of cells of an \emph{arbitrary shape}.
It can also be applied cell by cell in increasing dimension, starting with the construction of isolated 0-cells (embedded in $\mathbb{R}^n$) first, and then constructing 2-cells, 3-cells and further based on their boundary, allowing for the \emph{incremental construction} of objects of any dimension---hence the name given here.

This chapter describes an operation that permits the aforementioned process to be easily applied in practice using a topological data structure.
Based on combinatorial maps (\refse{ss:ordered-topological-models}) and their fundamental operations (\refse{ss:operations-maps}), it connects the separate boundary $(n-1)$-cells together by computing the appropriate adjacency relationships between them, encapsulating low-level details such as the creation of new individual combinatorial elements and setting the correct orientation for the existing ones.

The chapter begins with some background on the operations and a description of the overall approach in \refse{se:incremental-approach}.
It then explains the steps needed to create the cells of each dimension in \refse{se:primitives}.
\refse{se:incremental-implementation} describes how the approach was implemented based on the Computational Geometry Algorithms Library (CGAL).
\refse{se:incremental-experiments} summarises some experiments based on this implementation, generating relatively large objects in up to 4D.
Finally, \refse{se:incremental-conclusions} concludes the chapter with the possibilities to use incremental construction to build higher-dimensional datasets.

Most of this chapter is based on the paper:
\begin{itemize}
\papericaaincrementalconstruction%
\end{itemize}

\section{Background and overall approach}
\label{se:incremental-approach}

Based on the Jordan-Brouwer separation theorem \citep{Lebesgue11,Brouwer11}, it is known that a subset of space homeomorphic to an $(n-1)$-dimensional sphere\footnote{A generalisation of the concept of a sphere in arbitrary dimensions.
A 0-sphere is the pair of points, a 1-sphere is a circle, and a 2-sphere a sphere. As opposed to disks and balls, circles and spheres are hollow.} $S^{n-1}$ in the $n$-dimensional Euclidean space $\mathbb{R}^n$ divides the space into two connected components: the \emph{interior}, which is the region bounded by the sphere, and the \emph{exterior}, which is the unbounded region in which the sphere is a hole.
This means that the principles of boundary representation as described in \refse{ss:data-models} extend to higher dimensions, and so an $n$-cell in a cell complex can be described (and therefore constructed) based a set of $(n-1)$-cells that are known to form its complete (closed) boundary\footnote{This is true in practice.
However, there are fractal-like pathological objects where this is not the case because their interior or exterior components are not homeomorphic to disks \citep{Alexander24}.
While they do exist, this type of objects are certainly out of scope here.}.

The \emph{incremental construction} method proposed in this chapter exploits this property by applying this process cell by cell in increasing dimension.
Isolated vertices are first constructed, which are embedded in $\mathbb{R}^n$ and are uniquely defined based on their coordinates.
These vertices can then be connected to form individual edges, or instead to directly form cycles of implicit edges representing faces, as in most of the typical GIS approaches described in \refse{ss:formats}.
Sets of faces can be connected to form volumes, sets of volumes to form 4-cells and so on.

In a similar manner as extrusion, presented in \refch{ch:extrusion}, this \emph{incremental construction} process significantly reduces the conceptual complexity of defining and creating higher-dimensional objects.
The $(n-1)$-dimensional boundary of an $n$-cell is much easier to conceive than the original $n$-cell because it can be subdivided into multiple simpler $(n-1)$-cells, which can be individually described and constructed using the same method.

However, applying this incremental construction process using a topological data structure is not that simple, as it involves many intricate small problems: finding the topological relationships between the cells, appropriately connecting them, keeping track of multiple connected components, avoiding the creation of duplicate cells (as part of the boundary of independently-described higher-dimensional cells), changing the orientation of cells on the fly (since those that have been created separately will often have incompatible orientations), and detecting non-manifold shapes (which would result in invalid structures), among others.

The incremental construction operator presented in this chapter solves all of the aforementioned issues efficiently.
It is based on $n$-dimensional combinatorial maps with linear geometries and it is fully dimension-independent.
As it generates all the incidence and adjacency relationships between $(n-1)$-cells in an $n$-dimensional cell complex, it can also be used to obtain these relationships when needed, such as when multiple datasets are combined into one or when a non-topological representation is instead used (\eg\ the Simple Features-like approach shown in \refse{ss:nd-topology}).

In order to be efficient, the algorithm uses two basic techniques:
(i) indexes on the lexicographically smallest vertex (\refse{se:spatial-indexing}) of certain cells, which are used in order to keep track of individual cells (which might be disconnected) and to access cells efficiently; and
(ii) signature-generating traversals for specific cells \citep{Gosselin11}, which are used to efficiently compare whether two cells are equivalent by checking if it is possible to find an isomorphism $f$ that maps corresponding darts with equivalent ($\beta$) or reversed ($\beta^{-1}$) relations and which are embedded at the same location in $\mathbb{R}^n$, as was shown in \refse{se:ndqueries}.

The incremental construction operation as described in this chapter is applicable only to perfect data.
The $(n-1)$-cells bounding an $n$-dimensional cell should thus form a manifold and perfectly match each other at their common $(n-2)$-dimensional boundaries.
However, the method can be easily extended to handle some more complex configurations, such as by cleaning the input data using the methods described in \refch{ch:cleaning}.

\section{Incremental construction of primitives per dimension}
\label{se:primitives}

The idea of the incremental construction algorithm is to construct cells individually, using lower-dimensional cells that have already been constructed in order to describe part of the boundary of a higher-dimensional cell.
In terms of the darts of a combinatorial map, this sometimes implies the reuse of existing darts, and sometimes the creation of new ones which are connected to the existing ones.
There is however no strict requirement that the cells are created in strictly increasing dimension, and so new cells can be easily added to an existing cell complex.

Due to the need to embed 0-cells into a point in space, as well as the oriented nature of a combinatorial map, the incremental construction method is different for 0-, 1- and 2-cells.
3-cells and those of higher dimensions follow a unified procedure.
These cases are therefore described separately below, using as an example the creation of the pair of adjacent tetrahedra shown in \reffig{fig:2tetra}.
\marginpar{
\captionsetup{type=figure}
\centering
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/2tetra}} \\
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/2tetra-map}}
\caption[Two adjacent tetrahedra as a combinatorial map]{(a) Two adjacent tetrahedra and (b) their representation as a combinatorial map.}
\label{fig:2tetra}
}

\subsection{Vertices (0-cells)}
\label{ss:incremental-vertices}

The aim of the process of 0-cell construction is to create an isolated dart and an associated point embedding structure for every 0-cell, while avoiding having duplicate embedding structures having the same point coordinates.
By making point embeddings unique---as defined by their coordinates---, they can be used to compare 0-cells without checking an entire tuple of coordinates.
Moreover, by maintaining an index of all point embeddings and links from every point embedding to a dart there, it is possible to use this index to access the combinatorial structure at that point.
In this manner, it is possible to find either an already free dart that can be reused, or a non-free dart that is part of a larger combinatorial structure, which can be then copied and its copy used instead.

Thus, calling a function $\texttt{make\_0\_cell(}x_0, \ldots, x_n\texttt{)}$ with the coordinates $x_0, \ldots, x_n$ describing an $n$-dimensional point, should use the index of 0-cells to return an existing dart embedded at that location (using an existing point embedding) if one exists, or a new dart embedded at that location (using a new point embedding) otherwise.
In the latter case, the new point embedding and its associated dart should be added to the index.
The result of calling this function for all the point coordinates of the vertices in \reffig{fig:2tetra} is shown in \reffig{fig:reconstruction-0}.
Note that the result is identical whether the function is called once for every unique vertex, or multiple times (\eg\ once for every vertex in every face or volume).
\marginpar{
\captionsetup{type=figure}
\centering
\includegraphics[width=\marginparwidth]{figs/reconstruction-0}
\caption[Constructing the 0-cells of \reffig{fig:2tetra}]{Constructing the 0-cells of \reffig{fig:2tetra} consists of creating exactly one dart embedded at each point.
These darts are isolated (\ie\ not linked by any $\beta$ relations to the other darts).
The direction of the arrows shown here is arbitrary.}
\label{fig:reconstruction-0}
}

\subsection{Edges (1-cells)}
\label{ss:incremental-edges}

Generally, it is best to skip the generation of 1-cells and proceed directly to the creation of 2-cells from sequences of points.
In order to represent an isolated edge in a combinatorial map not one but \emph{two} darts are required: one embedded at each of the two vertices bounding the edge.
More precisely, taking into account the (arbitrary) orientation defined for the edge, the dart embedded at the \emph{origin} of the edge is connected to the \emph{destination} by $\beta_1$, and the destination is connected to the origin by $\beta_1^{-1}$.
Having two darts per edge is not a major problem, but it is wasteful as it often creates unnecessary darts and unnecessary connections between them that would later be deleted.
For instance, if a single face that uses all the edges is created from its bounding edges, half of the darts lose their original purpose (\ie\ to store a connection to their otherwise unlinked point embeddings) and will be eliminated\footnote{These could be those embedded at the edges' origin, destination or a mixture of the two.}, which is accompanied with having to reset the $\beta$ relationships pointing to them.

In addition, as shown in \refse{ss:formats}, polygons can be easily described as an ordered sequence of vertices connected by implicit line segments between each consecutive pair.
Therefore, incrementally constructing $1$-cells often brings no efficiency gains or practical benefits, and so it is normally best to skip dimension one, constructing 2D facets from a sequence of 0D points, 3D volumes from their 2D faces, 4-cells from their 3-cell faces, and so on.

However, there is an exception to this rule, as isolated edges and polygonal lines sometimes do need to be explicitly represented.
In these cases, it is possible to simply follow the process described for 2-cells below, omitting sewing the last dart of the line segment or polygonal line to the first dart (and vice versa), and appropriately using the 1-cells index to find if a given 1-cell already exists, or otherwise to index the newly created 1-cell or 1-cells (in the case of a polygonal line).

\subsection{Faces (2-cells)}
\label{ss:incremental-faces}

Starting from the unique 0-cells obtained from the procedure presented in \refse{ss:incremental-vertices}, in order to create a 2-cell from a sequence of 0-cells, three general steps are needed:
\begin{figure*}[tb]
\centering
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2}} \quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-a}\label{subfig:reconstruction-2-a}} \quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-b}\label{subfig:reconstruction-2-b}} \quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-c}}\\
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-d}} \quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-e}}\quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-f}} \quad
\subfloat[]{\includegraphics[width=0.22\linewidth]{figs/reconstruction-2-g}\label{subfig:reconstruction-2-g}}\\
\caption[Constructing the 2-cells of \reffig{fig:2tetra}]{The construction of the 2-cells of \reffig{fig:2tetra}. (a)~Initial configuration: one dart per vertex. (b)~After $b=\texttt{make\_2\_cell(}1,2,3\texttt{)}$. (c)~After $c=\texttt{make\_2\_cell(}2,4,3\texttt{)}$. (d)~After $d=\texttt{make\_2\_cell(}1,4,3\texttt{)}$. (e)~After $e=\texttt{make\_2\_cell(}1,4,2\texttt{)}$. (f)~After $f=\texttt{make\_2\_cell(}1,3,5\texttt{)}$. (g)~After $g=\texttt{make\_2\_cell(}5,3,4\texttt{)}$. (h)~Final result, after $h=\texttt{make\_2\_cell(}4,5,1\texttt{)}$.}
\label{fig:reconstruction-2}
\end{figure*}

\begin{enumerate}

\item
The procedure first checks if the 2-cell that is being requested already exists\footnote{This test can also be done at the end of the process, which makes it possible to use the easy cell comparison method described in \refse{se:ndqueries}.
In that case, the newly created cells (in the form of darts) that are found to be redundant should be deleted and removed from (or not added to) the indices.}.
Just as in the creation of 0-cells, a 2-cell being requested might have already been created, either independently or as part of a 3-cell.
This can be easily checked using the index of 2-cells with the lexicographically smallest vertex of the 0-cells being passed, and finding if from a dart starting at the lexicographically smallest vertex and following the $\beta_1$ or $\beta_1^{-1}$ relations, one passes through the same point embeddings as those passed to this construction method.

\item
Each of these 0-cells might be a $1$-free dart $d$ (\ie\ $\beta_1(d) = \emptyset$) and thus have no dart linked to it by $\beta_1$, \ie\ $\beta_1^{-1}(d) = \emptyset$ or $\forall d^\prime \nexists \beta_1(d^\prime) = d$, or if $\beta_1^{-1}$ is not stored, $\nexists d^\prime \mid \beta_1(d^\prime) = d$, \emph{in which case it can be used directly}, such as in \reffig{subfig:reconstruction-2-a}.
It can also be a non $1$-free dart or one that has a dart linked by $\beta_1$ to it, which means that it is used as part of a $1$-cell (and possibly other higher dimensional cells).

The darts that are already used as part of $1$-cells are more difficult to handle, as they \emph{can be reused only when the 1-cell they are part of is also part of the boundary of a geometrically identical $2$-cell as the one that will be constructed using this method}.
This needs to be tested in both possible orientations for the sequence of 0-cells, possibly resulting in the reversal of the orientation of some 1-cells.
If a given dart \emph{cannot be reused} (because it is part of a 1-cell that is not part of the 2-cell to be created), a copy of it has to be created.
The result of this step is thus a list that contains for every 0-cell a reusable existing dart or a newly created dart.

\item
The darts obtained in the previous step are 1-sewn sequentially (using $\beta_1$ and $\beta_1^{-1}$), and the last is 1-sewn to the first, forming a closed cycle.
During these sewing operations, the function checks whether every newly created edge (consisting of a pair of linked darts) is present in the index of 1-cells.
If a given 1-cell is already there, the new edge's dart is linked to its edge embedding structure.
If a 1-cell is not there, the edge is added to the index\footnote{Here, depending on the kind of expected input data, it might be more convenient to index edges at their origin vertex rather than at their lexicographically smallest.}, a new edge embedding is created, and the edge's dart is linked to it.
Finally, the newly created 2-cell is then added to the index of 2-cells.

In order to verify that the 2-cell being generated is a quasi-manifold, it is useful to assert that all darts that are $1$-free and have their $\beta_1^{-1}$ relation set to $\emptyset$ at the beginning of this third step, \ie\ those that were not copied, should continue to be $1$-free and have a $\beta_1^{-1}$ set to $\emptyset$ until they are sewn.
This condition, which is applied differently to the 1-cell, ensures that a vertex is not used twice within the same face, and as such, that the 2-cell is a quasi-manifold.

\end{enumerate}

\reffig{fig:reconstruction-2} shows the incremental construction procedure for all the 2-cells of \reffig{fig:2tetra}, which consists of 7 2-cells and is thus obtained by 7 calls to the \texttt{make\_2\_cell} method.

\subsection{Volumes (3-cells) and higher}
\label{ss:incremental-volumes}

The method to create $i$-cells from their $(i-1)$-cell boundaries is identical for all $i \geq 3$, allowing for a fully dimension-independent function to be created.
This function consists of four general steps:

\begin{enumerate}

\item
First of all, the procedure checks whether each $(i-1)$-cell that is passed is $(i-1)$-free or not.
If it is $(i-1)$-free, it is reused as part of the $i$-cell, such as in \reffig{subfig:reconstruction-3-a}.
If it is not $(i-1)$-free, then it is already part of a different $i$-cell, so a copy of it is made \emph{with a reversed orientation}, as shown in \reffig{subfig:reconstruction-3-b}.
This copy can then be used for the construction of the $i$-cell.

For a given dart $d$ that is known to be part of the $(i-1)$-cell, this copy can be made by taking all the darts in the orbit $C = \langle \beta_1, \ldots, \beta_{i-2} \rangle (d)$, creating a new set of darts $C^\prime$, and inserting a new corresponding dart in $C^\prime$ for every dart in $C$.
Using a function $f : C \rightarrow C^\prime$ that maps a dart in $C$ to its corresponding dart in $C^\prime$, for every dart $c \in C$ the $\beta_1$ relations are set as $\beta_1^{-1}\left(f\left(c\right)\right) = f\left(\beta_1\left(c\right)\right)$ and $\beta_1\left(f\left(c\right)\right) = f\left(\beta_1^{-1}\left(c\right)\right)$. For the ones of higher dimensions, they are set such that $\forall 2 \leq j \leq i-2$, $\beta_j\left(f\left(c\right)\right) = f\left(\beta_j\left(c\right)\right)$.

Note that the combinatorial structures are copied with reversed orientation, as this ensures that the two (old and new) can be directly $i$-sewn together, if necessary.

\item
A temporary \emph{ridge index} of the $i$-cell is built, containing all the $(i-2)$-cells on the boundary of all $(i-1)$-cells that have been passed to the construction method---not all the $(i-1)$-cells in the cell complex---using their lexicographically smallest vertices, such that their index entries link to a usable dart in the $(i-2)$-cell with a point embedding at the lexicographically smallest vertex.
This dart should be one that is $(i-1)$-free, either because it was already free as part of the combinatorial structure of the passed $(i-1)$-cells, or because it is a copy (with reversed orientation) of one that was not $(i-1)$-free.

\item
Using the ridge index, $(i-1)$-cells (\ie\ facets) are $(i-1)$-sewn along their common $(i-2)$-cell boundaries (\ie\ ridges).
For this, for every $(i-2)$-cell on the boundary of an $(i-1)$-cell passed to the function, exactly one matching $(i-2)$-cell should be found in the index, which should be equivalent as compared using the method described in \refse{se:ndqueries} and might be of the same or opposite orientation (but should not be part of the same $(i-1)$-cell or matched to itself).
This criterion ensures that a quasi-$(i-1)$-manifold will be constructed.

If the two $(i-1)$-cells are sewable along their common $(i-2)$-cell boundaries, \ie\ they have compatible orientations, they are directly sewn together.
Otherwise, the orientation of the entire connected component of either of the two $(i-1)$-cells should be reversed.
If the two $(i-1)$-cells have incompatible orientations but are part of the same connected component, the cell that is being requested is unorientable, and thus cannot be represented using combinatorial maps.
Although not discussed further here, note that as long as it does forms a quasi-manifold it can however be represented using \emph{generalised maps}.

\item
Using the index of $i$-cells, the newly constructed $i$-cell is compared to others to check if it had already been created, which is also done using the lexicographically smallest vertex of the cell.
If an equivalent $i$-cell (with the same or opposite orientation) is found, the function deletes the newly created darts of the cell and their corresponding index entries, and instead finally returns the existing $i$-cell.
If an equivalent $i$-cell is not found, the newly created cell is added to the index and returned.

\end{enumerate}

\reffig{fig:reconstruction-3} shows the incremental construction procedure for the two 3-cells of \reffig{fig:2tetra}.
\marginpar{
\captionsetup{type=figure}
\centering
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/reconstruction-3-a}\label{subfig:reconstruction-3-a}} \\
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/reconstruction-3-b}\label{subfig:reconstruction-3-b}}
\caption[Constructing the 3-cells of \reffig{fig:2tetra}]{The construction of the 3-cells of \reffig{fig:2tetra}. (a)~After $\texttt{make\_3\_cell(}b,c,d,e\texttt{)}$. (b)~Final result, after $\texttt{make\_3\_cell(}e,f,g,h\texttt{)}$.}
\label{fig:reconstruction-3}
}

\section{Implementation and complexity analysis}
\label{se:incremental-implementation}

The incremental construction algorithm has been implemented in C++11 and made available under the open source MIT licence at \url{https://github.com/kenohori/lcc-tools}.
As the extrusion related code (\refse{se:extrusion-implementation}), it requires and builds upon the CGAL packages Combinatorial Maps and Linear Cell Complex, among others.
% The first package provides data structures and algorithms to store and to efficiently iterate over the darts of a combinatorial map, and the second links the 0-embeddings to other CGAL types in order to store the geometry of a model.

For this prototype implementation, the lexicographically smallest vertex indices were implemented as C++ Standard Library \texttt{maps}\footnote{The exact implementation depends on the library that is used, but normally they are red-black trees.} for every dimension, using point embeddings as keys and \texttt{lists} of darts as values.
Each dart in a \texttt{list} represents a separate cell (of the dimension of the index) that has that point as its lexicographically smallest.
A custom comparison function is passed as a template so that the points are internally sorted in lexicographical order.
As a \texttt{map} has $O(\log n)$ search, insertion and deletion times and $O(n)$ space \citep[\S{}23.4]{ISO14882:2015}, all of these operations can be performed efficiently.

If a strictly incremental approach is followed, creating all $i$-cells before proceeding to the $(i+1)$-cells, it is not necessary to maintain indices for all the cells of all dimensions at the same time.
The only ones that are needed are those for: all $i$- and $(i-1)$-cells, as well as a temporary index for the $(i-2)$-cells on the boundary of the $(i-1)$-cells for the $i$-cell that is currently being constructed.
This means that only three indexes---one of which likely covers only a small part of the dataset---need to be kept at a given time.

Most of these indices can be easily built incrementally, adding new cells as they are created in $O(\log c)$, with $c$ the number of cells of that dimension, assuming that the smallest vertex and a dart embedded there are kept during its construction.
The complexity of building any index of cells of any dimension is thus $O(c \log c)$ and it uses $O(c)$ space.

Checking whether a given cell already exists in the cell complex is more complex.
Finding a list of cells that have a certain smallest vertex is done in $O(\log c)$.
In an unrealistically pathological case, all existing cells in the complex could have the same smallest vertex, leading to up to $c$ quadratic time cell-to-cell comparisons just to find whether one cell exists.
However, every dart is only part of \emph{a single} cell of any given dimension\footnote{This is true for the type of combinatorial maps with linear geometries that are handled here, but not necessarily so in the general case.}, so while every dart could conceivably be a starting point for the comparison, a single dart cannot be used as a starting point in more than one comparison, and thus a maximum of $d_{\mathrm{complex}}$ identity comparisons will be made for \emph{all} cells, with $d_{\mathrm{complex}}$ being the total number of darts in the cell complex.
From these $d_{\mathrm{complex}}$ darts, two identity comparisons are started, one assuming that the two cells (new and existing) have the same orientation, and one assuming opposite orientations.
Each of these involves a number of dart-to-dart comparisons in the canonical representations that \emph{cannot} be higher than the number of darts in the smallest of the two cells.
The number of darts in the existing cell is unknown, but starting from the number of darts in the newly created cell ($d_{\mathrm{cell}}$), it is safe to say that no more than $d_{\mathrm{cell}}$ dart-to-dart comparisons will be made in each identity test, leading to a worst-case time complexity of $O(d_{\mathrm{complex}} d_{\mathrm{cell}})$.
Note that this is similar to an isomorphism test starting at every dart of the complex.

Finally, creating an $i$-cell from a set of $(i-1)$-cells on its boundary is more expensive, since the $(i-2)$-cell (ridge) index needs to be computed for every $(i-1)$-cell.
Following the same reasoning as above, it can be created in $O(r \log r)$ with $r$ the number of ridges in the $i$-cell, and uses $O(r)$ space.
Checking whether a single ridge has a corresponding match in the index is done in $O(d_{\mathrm{cell}} d_{\mathrm{ridge}})$, with $d_{\mathrm{cell}}$ the number of darts in the $i$-cell and $d_{\mathrm{ridge}}$ the number of darts in the ridge to be tested.
Since this is done for all the ridges in an $n$-cell, the total complexity of this step, which dominates the running time of the algorithm, is

\begin{equation*}
\displaystyle\sum_{\mathrm{ridges}} O(d_{\mathrm{cell}} d_{\mathrm{ridge}}) = O(d_{\mathrm{cell}}^{2}).
\end{equation*}

The analyses given above give an indication of the computational and space complexity of the incremental algorithm as a whole.
However, it is worth noting that in realistic cases the algorithm fares far better than in these worst-case scenarios: the number of cells that have a certain smallest vertex is normally far lower than the total number of cells in the complex, most of their darts are not embedded at the smallest vertex, and from these darts most identity comparisons will fail long before reaching the end of their canonical representations.

Finally, one more nuance can affect the performance of this approach.
When two connected components of darts with incompatible orientations have to be joined, the orientation of one of these has to be reversed.
This is easily done by obtaining all the connected darts of one of the connected components, preferably the one that is expected to be smaller, and reversing their orientation 2-cell by 2-cell.
Every dart $d$ in a 2-cell is then $1$-sewn to the previous dart in the polygonal curve of the 2-cell (\ie\ $\beta_{1}^{-1}(d)$).
A group of $n$ darts can thus have its orientation reversed in $O(n)$ time.
This is not a problem in practice since GIS datasets generally store nearby objects close together, but if a cell complex is incrementally constructed in the worst possible way, \ie\ creating as many disconnected groups as possible, this could have to be repeated for \emph{every} cell of every dimension, creating a very inefficient process.

\section{Experiments}
\label{se:incremental-experiments}

\subsection*{Simple comparisons with valid primitives}

The CGAL Linear Cell Complex package provides functions to generate a series of primitives (line segments, triangles, quadrangles, tetrahedra and hexahedra) which are known to be created with correct geometry and topology, and can then be sewn together to generate more complex models.
Models constructed in this manner were thus created independently and compared to those incrementally constructed using the method presented in this chapter.
By using the approach shown in \refse{se:ndqueries}, it was possible to validate that they were equivalent.

\subsection*{A tesseract}

In order to present an example in more than three dimensions, a tesseract was also incrementally constructed using the approach presented in this chapter, which is shown in \reffig{fig:tesseract-darts}.
A tesseract is a 4-cell bounded by 8 cubical 3-cells, each of which is bounded by 6 square 2-cells.
It thus consists of one 4-cell, 8 3-cells, 24 2-cells, 32 1-cells and 16 0-cells.
\marginpar{
\captionsetup{type=figure}
\centering
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/tesseract2}} \\
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/tesseract3}} \\
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/tesseract}}
\caption[A tesseract as a combinatorial map]{A tesseract as a combinatorial map: (a) the darts of its 7 inner cubes, (b) the darts of its 8 cubes, and (c) all of its darts shown together.}
\label{fig:tesseract-darts}
}

Using the approach presented here, an empty 0-cell index is first created.
Then, the 16 vertices of the tesseract, each vertex $p_{i}$ being described by a tuple of coordinates $(x_{i},y_{i},z_{i},w_{i})$, were created as $p_{i}=\texttt{make\_0\_cell(}x_{i},y_{i},z_{i},w_{i}\texttt{)}$, which returns a unique dart embedded at each location that is also added to the 0-cell index.
At this point, the algorithm has built an unconnected cell complex consisting solely of 16 completely free darts.

An empty index of 2-cells is then created.
Each of the tesseract's 24 square facets are then built based on their vertices as $f_{i}=\texttt{make\_2\_cell(}p_{j},p_{k},p_{l},p_{m}\texttt{)}$, which 1-sews (copies of) these darts in a loop and returns the dart embedded at the smallest vertex of the facet.
These are added to the index of 2-cells.
Since every vertex is used in 6 different 2-cells, each dart would be copied 5 times.
The cell complex at this point thus consists of 24 disconnected groups of 4 darts each.

Next, an empty index of 3-cells is created and the index of 0-cells can be deleted.
For each of the 8 cubical 3-cells, a function call of the form $v_{i}=\texttt{make\_3\_cell(}f_{j},f_{k},f_{l},f_{m},f_{n},f_{o}\texttt{)}$ is made.
At this point, an index of the 1D ridges of each face is built, which is used to find the 12 pairs of corresponding ridges that are then be 2-sewn together.
When a 3-cell is created, it is added to the index.
Since every face bounds two 3-cells, each dart is duplicated once again, resulting in a cell complex of 8 disconnected groups of 24 darts each.

Finally, the tesseract is created with the function $t=\texttt{make\_4\_cell(}v_{1},v_{2},\dots,v_{8}\texttt{)}$.
This can use the index of 2-cells to find the 24 corresponding pairs of facets that are then 3-sewn to generate the final cell complex.

The validity of this object was tested by checking whether it formed a valid combinatorial map, testing whether each cube was identical to a cube created with the Linear Cell Complex package, and manually verified the $\beta$ links of its 192 darts.

\subsection*{2D+scale data}

In order to test the incremental construction algorithm and its applicability to data incorporating non-spatial characteristics, a few 2D+scale datasets from \citet{Meijers11} using ATKIS data\footnote{\url{http://www.bkg.bund.de/nn_147094/SharedDocs/Download/Barrierefreie-Textversionen/EN-InfoMaterial/EN-Text-Vector-Data.html}}, were also incrementally constructed.
As shown in \reffig{fig:utm}, these datasets model the generalisation of a planar partition as a set of stacked prisms.
Both of the simple datasets of this figure were created successfully in under a second.
\marginpar{
\captionsetup{type=figure}
\centering
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/utm}} \\
\subfloat[]{\includegraphics[width=\marginparwidth]{figs/utm2}}
\caption[Simple 2D+scale datasets]{Simple 2D+scale datasets from \citet{Meijers11}, which represent the generalisation of a planar partition by merging areas.
The dataset shown in (a) has realistic scale intervals such that the generalisation operations are performed at scales that depend on the area of a polygon, while the dataset in (b) uses equally spaced generalisation operations.}
\label{fig:utm}
}

A larger dataset, shown in \reffig{fig:atkis} and consisting of 698 polyhedra with a total of 457 185 faces, was used as a benchmark to test the performance of the algorithm.
This dataset was processed without errors in roughly 30 minutes, including validation tests to ensure that every face created was a valid combinatorial map and that the faces of a volume formed a closed quasi-manifold.
\begin{figure}[bp]
\includegraphics[width=\linewidth]{figs/atkis}
\caption[A large 2D+scale dataset]{A large 2D+scale dataset from \citet{Meijers11}, which uses equally spaced generalisation operations that merge two adjacent polygons.}
\label{fig:atkis}
\end{figure}

In this latter dataset, an additional test was made using its first 250 polyhedra, which lie on the top of \reffig{fig:atkis}, comparing the time used for the construction of vertices, faces and volumes with and without the use of indices.
The results of both methods were equivalent using the approach shown in \refse{se:ndqueries}.
On average, using indices resulted in a faster vertex construction time by a factor of 2 200, faster face construction by a factor of 56 and faster volume construction by a factor of 38.
However, as \reffig{fig:construction-times} shows, these speed gains are not uniform throughout the 250 polyhedra.
The speed gained from using an index on the vertices increases roughly linearly on the number of constructed vertices, while that gained on from the faces index remains roughly constant between a factor of 50 and 60, and that gained from the volumes index tends to slowly increase as well.
\begin{figure*}[tbp]
\centering
\subfloat[]{\includegraphics[width=0.33\linewidth]{figs/construction-vertices}\label{subfig:construction-vertices}}
\subfloat[]{\includegraphics[width=0.33\linewidth]{figs/construction-faces}\label{subfig:construction-faces}}
\subfloat[]{\includegraphics[width=0.33\linewidth]{figs/construction-volumes}\label{subfig:construction-volumes}}
\caption[Construction time speed-up from the use of indices]{Construction time speed-up from the use of indices on the (a) vertices, (b) faces and (c) volumes.}
\label{fig:construction-times}
\end{figure*}

\section{Conclusions}
\label{se:incremental-conclusions}

Creating computer representations of higher-dimensional objects can be complex.
Common construction methods used in 3D, such as directly manipulating combinatorial primitives, or using primitive-level construction operations (\eg\ Euler operators \citep{Mantyla88}), rely on our intuition of 3D geometry, and thus do not work well in higher dimensions.
It is therefore all too easy to create invalid objects, which then cannot be easily interpreted or fixed.

The incremental construction method proposed in this section follows a completely different approach, which has a solid underpinning in the Jordan-Brouwer separation theorem \citep{Lebesgue11,Brouwer11}.
By exploiting the principles of boundary representation, it constructs an $i$-cell based on a set of its bounding $(i-1)$-cells.
Since individual $(i-1)$-cells are easier to describe than the $i$-cell, it thus subdivides a complex representation problem into a set of simpler, more intuitive ones.
The method can moreover be incrementally applied to construct cell complexes of any dimension, starting from a set of vertices in $\mathbb{R}^n$ as defined by a $n$-tuple of their coordinates, and continuing with cells of increasing dimension---optionally creating edges from vertices, then faces from vertices or edges, volumes from faces and so on.

While a set of $(i-1)$-cells bounding an $i$-cell can be said to already form a complete representation of an $i$-cell, this is not sufficient for its representation in a topological data structure, which requires the topological relationships between the $(i-1)$-cells to be computed.
The incremental construction algorithm proposed in this section thus computes the relationships that are required for two data structures: generalised or combinatorial maps.
However, these relationships are also applicable to most other data structures.

The algorithm is efficient due to its use of indices using the lexicographically smallest vertex of every cell per dimension, as well as an added index using the lexicographically smallest vertex of the bounding ridges of the cell that is being built.
It generates an $i$-cell in $O(d^{2})$ in the worst case, with $d$ the total number of darts in the cell.
However, it fares markedly better in real-world datasets, as cells do not generally share the same lexicographically smallest vertex.
By checking all matching ridges within a cell's facets, the algorithm can optionally verify that the cell being constructed forms a combinatorially valid quasi-manifold, avoiding the construction of invalid configurations.

A publicly available implementation of the algorithm has been made based on CGAL Combinatorial Maps, and its source code has been released under a permissive licence.
It is worth noting that it is one of very few general-purpose object construction methods that has been described and implemented for four- or higher-dimensional cell complexes.

The implementation has been tested with simple 2D--4D objects, as well as with large 2D+scale datasets from \citet{Meijers11}.
The constructed objects were tested to verify that they form valid combinatorial maps.
The small objects were also manually inspected, visualised, and where possible, they were compared with equivalent objects known to be valid using the method discussed in \refse{se:ndqueries}.

While the incremental construction operation as described in this chapter works only with perfect data, small modifications could make it applicable to additional configurations.
For instance, merging adjacent cells that can be perfectly described by a single cell with linear geometry (\eg\ collinear edges and coplanar faces) as a preprocessing step can be used to handle cases where cells are subdivided only on one side of a boundary.
Snapping points together can solve several common invalid configurations, such as those described in \refch{ch:cleaning} and in \citet{Diakite14}.

Finally, the logical step forward at this point would be the use of the incremental construction algorithm for the creation of large higher-dimensional datasets.
However, since to the best of my knowledge there are no large space-filling higher-dimensional datasets currently available---without even considering any validity criteria---, such tests could not be conducted within this thesis.
The generation of such a higher-dimensional dataset is thus a necessary first step in this direction and is considered as a related piece of future work.


\documentclass{article}
\usepackage{amsmath}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage[margin=1.0in]{geometry}
\title{SPR-based Error Estimation}
\author{Dan Ibanez and Brian Granzow}
\date{Jun 21, 2014}
\begin{document}
\maketitle

\section{Overview}
An SPR-based error estimation procedure is defined in detail
in Jie Wan's thesis.
We have reimplemented this system using the latest APF
libraries and it supports the Albany adaptive cycle.
This document details the inner workings of this estimator.

From a high level, the SPR-based error estimator works by
recovering a $p$-order field from what is essentially
a $(p-1)$-order input field, then it computes the
error at each element as a function of the difference
between the input (less accurate) and output
(hopefully more accurate) fields.

The input to SPR is actually a ``field" $\epsilon$ that contains values
at integration points rather than nodes.
This is typically the result of some kind of gradient computation
$\epsilon = \nabla f$ from a $p$-order field $f$.
In mechanics applications, $\epsilon$ is a strain or stress quantity
evaluated at integration points.
Note that the choice of integration points is independent of
anything else, although typically they should be chosen such
that they can integrate a field of order $p$.

\section{Field Recovery}
The SPR recovery process generates a $p$-order field $\epsilon^*$ which
has values at the nodes for the same quantity that the
input $\epsilon$ provides at integration points.

This recovery is done node-by-node, and the value at each
node is recovered by fitting an ordinary $p$-order polynomial
$q (x, y, z) = v$ using input data $v_i = \epsilon(x_i, y_i, z_i)$
at integration points $(x_i,y_i,z_i)$ around the node.
The collection of integration points is done by building
a ``patch" of mesh elements around the node.

Fitting a polynomial from data is done by solving a least-squares linear
system $Ac \approx b$.
The solution $c$ should minimize $\|Ac - b\|$, and represents
the coefficients of the polynomial $q$.
For example, if $p=1$:
\[q(x,y,z) = c_0 + c_1 x + c_2 y + c_3 z \]

Each data point becomes a row of the linear system:

\[q(x_i,y_i,z_i) = c_0 + c_1 x_i + c_2 y_i + c_3 z_i \approx v_i \]

Which can be expressed as a dot product:

\[q(x_i,y_i,z_i) = (1,x_i,y_i,z_i)(c_0,c_1,c_2,c_3)^T \approx v_i \]

Which yields the matrix equation:

\[\begin{bmatrix}
1 & x_0 & y_0 & z_0 \\
1 & x_1 & y_1 & z_1 \\
1 & x_2 & y_2 & z_2 \\
\text{...}
\end{bmatrix}
\begin{bmatrix} c_0 \\ c_1 \\ c_2 \\ c_3 \end{bmatrix}
\approx
\begin{bmatrix} v_0 \\ v_1 \\ v_2 \\ ... \end{bmatrix}
\]

The matrix $A$ on the left is then $m\times n$ where $m$
is the number of sample points and $n$ is the number of
polynomial coefficients.
We would rather have an over-determined answer than
an under-determined one, so we require $m \geq n$.
Typically, we get $m > n$, in which case the solution
is not exact and we must choose coefficients to minimize

\[ \left( \sum_{i=0}^{m-1} (q(x_i,y_i,z_i) - v_i)^2 \right)^\frac12 \]

To solve the least squares problem $Ac \approx v$, we
use a QR factorization approach.
We use Householder reflectors to obtain a decomposition

\[QR = A, QQ^T = I, QRc \approx v, Rc = Q^Tv\]

Where R is upper triangular and Q is stored implicitly
as a set of Householder vectors.
This allows us to compute $y = Q^Tv$ implicitly
(which is more accurate)
and then solve $Rc = y$, which is an $n\times n$ problem,
with back substitution.

However, the solution $c$ can only be accurately found
if the matrix $A$ is well-conditioned, i.e. rank$(A) = n$.
We must be concerned with this because least-squares polynomial
fitting is well-known for producing ill-conditioned matrices.
The QR solver provides a single point of inspection where
a division by zero will occur if $A$ is ill-conditioned,
so it will signal a failure in this case.

During recovery, the sample points consist of all
integration points in the elements of the patch.
Therefore, we must increase the number of elements
in the patch so long as the resulting $A$ does
not satisfy the conditions $m \geq n$ and rank$(A) = n$.

The elements $S$ in the patch are collected as follows
until those conditions are met:

\begin{algorithm}
\caption{Patch building algorithm}
\begin{algorithmic}
\State let $o$ be the mesh entity to which the node is associated.
\State set $S$ to be the elements adjacent to $o$ $(S \gets o\{M^3\})$.
\Loop
\For{$d$ from 2 down to 0}
\State add the elements adjacent by $d$-dimension entities to $S$
$(S \gets S\{M^d\}\{M^3\})$.
\If{$S$ has enough elements}
\State \Return $S$
\EndIf
\EndFor
\EndLoop
\end{algorithmic}
\end{algorithm}

Finally, once the patch is obtained, the integration points
$(x_i,y_i,z_i)$ and the field values $\epsilon(x_i,y_i,z_i) = v_i$
from the input are used to build $A$ and $v$.
Note that the field values $v_i$ may be tensors instead of scalars,
in which case the analysis is solved one component at a time.
Each component $j$ forms a set of scalar values $v^{(j)} = (v_0^{(j)},...,v_m^{(j)})^T$
and results in a set of polynomial coefficients $c^{(j)}$
that minimize $\|Ac^{(j)} - v^{(j)}\|$ and define a polynomial $q^{(j)}(x,y,z)$.
We then evaluate this polynomial at the location $(x_a,y_a,z_a)$
of the node to obtain the field component value at
that node: $\epsilon^*(x_a,y_a,z_a)^{(j)} = \rho^{(j)}(x_a,y_a,z_a)$.
One can also think of the coefficients $c$ being
tensors.
Note that the matrix $A$ remains the same for each component,
only the right hand side $v^{(j)}$ changes to give a different
$c^{(j)}$. This allows us to only compute the QR decomposition
of A once and only repeat the computations of $y=Q^Tv^{(j)}$
and $Rc^{(j)}=y$.

\section{Error Estimation}

A per-element scalar error estimate
$\|e_\epsilon\|_e$ is computed using the
norm of the integrated difference between
direct and recovered gradient fields.
\[e_\epsilon=\epsilon - \epsilon^*\]
The current implementation uses entry-wise L2 norms on
3x3 matrices, which seems to agree with Wan's thesis.
The following notations describe the L2 norm integration
over an element and over the whole mesh:

\[\|A\|_e=
\left(
\int_{\Omega^e} A : A d\Omega
\right)^\frac12,
\|A\|=
\left(
\sum_{e=1}^{n_e}
\int_{\Omega^e} A : A d\Omega
\right)^\frac12\]

Where $A:A$ is the Frobenius inner product.
If $\|A\|$ or $\|A\|_e$ will be computed for
some matrix field $A$, it may be advantageous
to store the field $A:A$.

\section{Size Field Computation}

A per-element scalar size factor is computed based on
the per-element error following this formula:

\[h^\text{new}_e = h^\text{current}_e
\|e_\epsilon\|^{-\frac{2}{2p+d}}_e
\left(
\frac
{\hat{\eta}^2\|\epsilon^*\|^2}
{\sum_{i=1}^n\|e_\epsilon\|^\frac{2d}{2p+d}_i}
\right)^\frac{1}{2p}
\]

To which the following definitions apply:

\begin{center}
\begin{tabular}{ll}
$d$ & element dimension \\
$p$ & polynomial order of $f$ \\
$\eta$ & $\frac{\|e_\epsilon\|}{\|\epsilon\|}$ \\
$\hat{\eta}$ & threshold on $\eta$ for adaptivity \\
$n$ & number of elements in the mesh \\
$h_e^\text{current}$ & the current element size (longest edge) \\
$h_e^\text{new}$ & the desired element size for adaptivity \\
\end{tabular}
\end{center}

Finally, a linear isotropic size field at
the vertices is recovered from the per-element
size factors $h_e^\text{new}$ by a simple
process of averaging the values from elements
adjacent to a vertex.

\section{Target Size Field Computation}

In practical applications, it is desirable to target a specified
number of elements $N$ during mesh adaption to avoid over-coarsening
or over-refinement, and to avoid exhausting the memory on the available
machine.

Boussetta et al. ("Adaptive remeshing based on a posteriori error
estimation for forging simulation") define a size field specification
by the following:
\[
h^{new}_e = \left(
\frac{\theta_{uni}}{\theta_e} \right)
^{( \frac{2}{2p +d} )} h_e
\]
where $\theta_e$ is the element-wise contribution to the error,
$\theta_{uni}$ is the value of a uniformly distributed error
across the mesh, $h_e^{new}$ is the new element mesh size,
$h_e$ is the old element mesh size, $p$ is the polynomial order
of accuracy of the finite element method, and $d$ is the number
of spatial dimensions of the mesh.

Let $G$ be a globally computed error metric given by:
\[
G = \sum_e^n \left( \theta_e \right) ^{(\frac{2d}{2p+d})}
\]
where $n$ is the total number of elements in the mesh.
Then value of the uniformly distributed error is given by
\[
\theta_{uni} = \left( \theta_{imp} \right) ^{( \frac{2p+d}{2p} )}
\left( G \right) ^{-(\frac{2p+d}{4p} )}
\]
where $\theta_{imp}$ is a prescribed accuracy parameter.
To generate a size field that will yield an adapted
mesh with approximately $N$ elements, the accuracy parameter
$\theta_{imp}$ is given by
\[
\theta_{imp} = \left( N \right) ^{-(\frac{p}{d})}
\left( G \right)^{(\frac{2p+d}{2d})}
\]

Algebraic manipulation of these quantities yields a simpler
equivalent expression for the new element size field:
\[
h_e^{new} = \left(\frac{G}{N}\right)^{\frac{1}{d}}
\left( \theta_e \right)^{-(\frac{2}{2p+d})} h_e
\]
The element-level error contribution $\theta_e$ is chosen to be
\[
\theta_e = \left(
\int_{\Omega^e}(\epsilon - \epsilon^*) : (\epsilon - \epsilon^*)
\text{d} \Omega
\right) ^{\frac{1}{2}}
\]
as is done in the previous size field computation.

In addition to targeting a specified number of elements in the
output mesh, Boussetta et al. define user-input parameters
$\alpha$ and $\beta$ to control the gradients of the mesh size
field such that
\[
\alpha < \frac{h_e^{new}}{h_e} < \beta
\]

\end{document}


\section{Semidecidability of the CF Regularity Problem}
\label{sec:semidecidability}

We know that the context-free regularity problem is undecidable
\cite{Pettorossi13}, due to its reduction to the undecidable Post Correspondence
Problem \cite{Hopcroft06}.

We prove the semidecidability and the undecidability of the context-free
regularity problem.

\begin{theorem}
	\label{thm:semidecidability-mnf}
	Given a context-free grammar, the problem of saying whether or not there
	exists an equivalent grammar in Marciani Normal Form is semidecidable and
	undecidable.

	\begin{proof}
		Let us consider a context-free grammar in Chomsky Normal Form.
		Let us derive an equivalent grammar by unfolding every production,
		until getting the productions for the axiom only.
		Now, it' easy to check if the derived grammar is in Marciani Normal Form.
		So, given any context-free grammar in Chomsky Normal Form, it is always
		possible to check if there exists an equivalent context-free grammar in
		Marciani Normal Form.

		As this possibility holds for grammars in Chomsky Normal Form, then it
		holds for every context-free grammar \cite{Pettorossi13}.
	\end{proof}
\end{theorem}

\begin{theorem}
	\label{thm:semidecidability}
	The context-free regularity problem is semidecidable and undecidable.

	\begin{proof}
		Follows from the semidecidability and undecidability of the problem of
		saying whether or not, given a context-free grammar, there exists an
		equivalent grammar in Marciani Normal Form, and from the regularity of
		the language generated by a context-free grammar in Marciani Normal Form.
	\end{proof}
\end{theorem}


\documentclass[12pt]{article}
\bibliographystyle{apsrev4-1}%amsplain}
%%a4paper compact
\usepackage[utf8x]{inputenc}
\usepackage{amsfonts, amssymb, amsmath, amscd, eucal, amsthm}
\usepackage{array}
\usepackage{graphicx}
\usepackage{subfig}
\usepackage{fancyhdr}
\usepackage{color}
\usepackage{extarrows}
\usepackage{enumerate}
\usepackage{multirow}
\usepackage[numbers]{natbib}
\usepackage{geometry}
\usepackage{CJK}
\geometry{left=3cm,right=3cm,top=2.5cm,bottom=2.5cm}

%\theoremstyle{plain}
%\newtheorem{thm}{Theorem}[chapter] % reset theorem numbering for each chapter
%\theoremstyle{definition}
%\newtheorem{defn}[thm]{Definition} % definition numbers are dependent on theorem numbers
%\newtheorem{exmp}[thm]{Example}

\pagestyle{fancy}
%\chea
\rhead{}

%\setlength{\textwidth}{13.72cm}
%\setlength{\oddsidemargin}{1cm}
%\setlength{\evensidemargin}{1cm}

\graphicspath{{pics/}}
%opening
\title{MNIST Experiment in Python}
%\author{Jianhong Chen}

\begin{document}
\maketitle
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\begin{CJK*}{UTF8}{gbsn}

\section{Least square}
\subsection{Model}
For each data $x_i\in \mathbb{R}^{784}, i=1,...,60000$, the corresponding label is $y_i\in \mathbb{R}^{10}$ where $y_i=e_j$ if $x_i \in A_j$ or the label for $x_i$ is j.

If we denote
A=
$\begin{pmatrix}
    x_{1}       & 1 \\
    x_{2}       & 1 \\
    ... &... \\
    x_{60000}       & 1\\
\end{pmatrix} \in \mathbb{R}^{60000\times 785}$ , Y=
$\begin{pmatrix}
    y_{1}        \\
    y_{2}       \\
    ... \\
    y_{60000}     \\
\end{pmatrix}\in \mathbb{R}^{60000\times 10}$.

We want to find $\theta=\left[ {\begin{array}{c}
   W^T \\
   b^T \\
  \end{array} } \right]\in \mathbb{R}^{785\times 10}$ to minimize $\|Y-A\theta\|^2_2$.

The empirical formulae gives
\begin{align}
\hat{\theta}=(A^TA)^{-1}A^TY.
\end{align}

\subsection{Numerical result}
For the MNIST dataset, after $\hat{\theta}$ is obtained, we can classify $x_i$ based on
\begin{align}
\hat{y_i}=Wx_i+b.
\end{align}
We choose the digit with the maximum value as the predicted label for $x_i$. The training accuracy is 85.77$\%$.

\section{Logistic regression }
\subsection{Model}
\begin{itemize}
\item X: the input image of a handwritten digit
\item Y: the true value of the digit
\item W,b: weight and bias
\item Y$\_$pred=softmax(Wx+b)
\item Loss=cross$\_$entropy(Y, Y$\_$pred)+ regularization term
\item Optimization method: Adam
\end{itemize}

\subsection{Implementation }
\begin{itemize}
\item Parameter
\begin{enumerate}
\item learning rate = 0.001
\item training epochs=50
\item batch size =100
\item regularization coefficient =0.0001
\end{enumerate}
\item Result
\begin{enumerate}
\item no regularization: training accuracy=93$\%$; test accuracy =92$\%$
\item regularization coefficient =0.0001: training accuracy=93$\%$; test accuracy =92$\%$
\item regularization coefficient =0.001: training accuracy=92$\%$; test accuracy =92$\%$
\item no regularization+SGD: training accuracy= 88$\%$; test accuracy =89$\%$
\end{enumerate}

\end{itemize}

\section{One hidden layer }
\subsection{Model}
\begin{itemize}
\item X: the input image of a handwritten digit
\item Y: the true value of the digit
\item Hidden layer size=500
\item Loss=cross$\_$entropy(Y, Y$\_$pred)+ regularization term
\item Optimization method: Adam
\end{itemize}

\subsection{Implementation }
\begin{itemize}
\item Parameter
\begin{enumerate}
\item learning rate = 0.001
\item training epochs=50
\item batch size =100
\item regularization coefficient =0.0001
\end{enumerate}
\item Result
\begin{enumerate}
\item no regularization: training accuracy=100$\%$; test accuracy =98.36$\%$
\item regularization coefficient =0.0001: training accuracy=99.00$\%$; test accuracy =98.04$\%$
\item regularization coefficient =0.001: training accuracy=98.00$\%$; test accuracy =97.67$\%$
\item no regularization+SGD: training accuracy= 90$\%$; test accuracy =90.54$\%$
\end{enumerate}

\end{itemize}

\end{CJK*}
\end{document}


\section{Method}
\label{sec:method}

In the following, we first introduce the mathematical formulation of the weakly-supervised 3D shape completion problem. Subsequently, we briefly discuss the concept of variational auto-encoders (\VAEs) \cite{Kingma2013ARXIV} which we use to learn a shape prior.
% -- which are used to learn a shape prior based on the reference shapes.
Finally, we formally derive our proposed amortized maximum likelihood (\AML) approach. The overall framework is also illustrated in Figure \ref{fig:method}.

\subsection{Problem Formulation}

In a supervised setting, our task can be described as follows: given a set of partial observations $\mathcal{X} = \{x_n\}_{n = 1}^N \subseteq \mathbb{R}^R$ and corresponding ground truth shapes $\mathcal{Y}^* = \{y_n^*\}_{n = 1}^N \subseteq \mathbb{R}^R$, learn a mapping $x_n \mapsto y_n^*$ that is able to generalize to previously unseen observations. Here, we assume $\mathbb{R}^R$ to be a suitable vector representation of observations and shapes; in practice, we resort to occupancy grids or signed distance functions (SDFs) defined on regular grids, \ie, $x_n, y_n^* \in \mathbb{R}^{H \times W \times D} \simeq \mathbb{R}^R$.
SDFs represent the distance of each voxel's center to the closest point on the surface; we use negative signs for interior voxels.
For the (partial) observations, we write $x_n \in \{0, 1, \uk\}^R$ to make missing information explicit; in particular, $x_{n,i} = \uk$ corresponds to unobserved voxels, while $x_{n,i} = 1$ and $x_{n,i} = 0$ correspond to occupied and unoccupied voxels, respectively.

On real data, \eg, KITTI \cite{Geiger2012CVPR}, supervised learning is often not possible as obtaining ground truth annotations is labor intensive (\eg, \cite{Menze2015CVPR,Xie2016CVPR}). Therefore, we target a weakly-supervised variant of the problem instead. Given observations $\mathcal{X}$ and a set of reference shapes $\mathcal{Y} = \{y_m\}_{m = 1}^M \subseteq \mathbb{R}^R$ both of the same, known object category, learn a mapping $x_n \mapsto \tilde{y}(x_n)$ such that the predicted shape $\tilde{y}(x_n)$ matches the unknown ground truth shape $y_n^*$ as close as possible.
Here, supervision is provided in the form of the known object category, allowing to derive the reference shapes from (watertight) triangular meshes; on real data, we also assume the object locations to be given in the form of 3D bounding boxes in order to extract the observations $\mathcal{X}$.
%\green{In practice, the reference shapes $\mathcal{Y}$ are derived from watertight, triangular meshes in order to obtain well-defined occupancy grids and SDFs.}
%In the following, we assume that the shapes $\mathcal{Y}$ are available as meshes from which we derive occupancy grids and SDFs; the observations are voxelized as described above.

\subsection{Shape Prior}
\label{subsec:method-prior}

We propose to use the provided reference shapes $\mathcal{Y}$ to learn a model of possible 3D shapes over the latent space $\mathcal{Z} = \mathbb{R}^Q$ with $Q \ll R$. The prior model is learned using a \VAE where the joint distribution $p(y, z)$ decomposes into $p(y, z) = p(y | z)p(z)$ with $p(z)$ being a unit Gaussian, \ie, $p(z) = \mathcal{N}(z;0, I_Q)$ with $I_Q \in \mathbb{R}^{R \times R}$ being the identity matrix. Sampling from the model is then performed by choosing $z \sim p(z)$ and subsequently sampling $y \sim p(y | z)$. For training the generative model, we also need to approximate the posterior $q(z | y) \approx p(z | y)$, \ie, the inference model. In the framework of the variational auto-encoder, both the so-called recognition model $q(z | y)$ and the generative model $p(y | z)$ -- corresponding to encoder and decoder -- are represented by neural networks. In particular,
\begin{align}
%\text{\bf Encoder: }
q(z | y) = \mathcal{N}(z_i; \mu_i(y), \text{diag}(\sigma_i^2(y)))
\end{align}
where $\mu(y), \sigma^2(y) \in \mathbb{R}^Q$ are predicted using the encoder neural network and $p(y_i | z)$ is assumed to be a Bernoulli distribution when working with occupancy grids, \ie, $p(y_i | z) = \text{Ber}(y_i ; \theta_i(z))$ while a Gaussian distribution is used when predicting SDFs, \ie, $p(y_i | z) = \mathcal{N}(y_i ; \mu_i(z), \sigma^2)$. In both cases, the parameters, \ie, $\theta_i(z)$ or $\mu_i(z)$, are predicted using the decoder neural network. For SDFs, we neglect the variance ($\sigma^2 = 1$) as it merely scales the training objective.

In the framework of variational inference, the parameters of the encoder and the decoder are found by maximizing the likelihood $p(y)$. In practice, the likelihood is often intractable. Instead, the evidence lower bound is maximized, resulting in the following loss to be minimized:
\begin{align}
\mathcal{L}_{\text{VAE}}(w) = - \mathbb{E}_{q(z |y)}[\ln p(y|z)] + \text{KL}(q(z | y)| p(z)).
\end{align}
where $w$ are the weights of the encoder and decoder. The Kullback-Leibler divergence $\text{KL}$ can be computed analytically; the expectation corresponds to a binary cross-entropy error for occupancy grids or a scaled sum-of-squared error for SDFs. The loss $\mathcal{L}_{\text{VAE}}$ is minimized using stochastic gradient descent (SGD). We refer to \cite{Kingma2013ARXIV} for details.

\subsection{Shape Inference}
\label{subsec:method-inference}

After learning the shape prior $p(y, z) = p(y| z) p(z)$, shape completion can be formulated as a maximum likelihood (\ML) problem over the lower-dimensional latent space $\mathcal{Z} = \mathbb{R}^Q$. The corresponding negative log-likelihood, \ie, $-\ln p(y, z)$, can be written as
%
\begin{align}
\mathcal{L}_{\text{ML}}(z) &= - \sum_{x_i \neq \uk} \ln p(y_i = x_i | z) - \ln p(z).\label{eq:ml}
\end{align}
%
where $x_i \neq \uk$ expresses that the summation ranges only over observed voxels.
As the prior $p(z)$ is Gaussian, the corresponding negative log-probability $- \ln p(z) \propto \|z\|_2^2$ results in a quadratic regularizer. As before, the generative model $p(y | z)$ decomposes over voxels.
% here, we can only consider actually observed voxels, \ie, $x_i \neq \uk$.
Instead of solving Equation \eqref{eq:ml} for each observation $x \in \mathcal{X}$ individually, we follow the idea of amortized inference \cite{Gersham2014COGSCI} and train an encoder $z(x;w)$ to \emph{learn} \ML. To this end, we keep the generative model $p(y|z)$ fixed and train the weights $w$ of the encoder $z(x;w)$ using the \ML objective as loss:
%
\begin{align}
\mathcal{L}_{\text{AML}}(w) = - \sum _{x_i \neq \uk} \ln p(y_i = x_i | z) - \lambda \ln p(z).\label{eq:aml}
\end{align}
%
where we added an additional parameter $\lambda$ controlling the importance of the shape prior. The exact form of the probabilities $p(y_i = x_i | z)$ depends on the used shape representation. In the case of occupancy grids, this term results in a cross-entropy error (as both $y_i$ and $x_i$ are, for $x_i \neq \uk$, binary). However, when using SDFs, the term is not well-defined as $p(y_i | z)$ is modeled with a continuous Gaussian distribution, while the observations $x_i$ are binary, \ie, it is unclear how to define $p(y_i = x_i | z)$. Alternatively, we could derive distance values along the rays corresponding to observed points (\eg, following \cite{Steinbrucker2013ICCV}). However, as illustrated in Figure \ref{fig:method-sdf}, noisy rays lead to invalid observations along the whole ray. This problem can partly be avoided when relying on occupancy to represent the observations.

\begin{figure}
    \centering
    \vspace*{-0.25cm}
    \begin{subfigure}[t]{0.425\linewidth}
        \vspace{0px}
        \centering
        \includegraphics[width=1\linewidth]{fig/method_sdf_2}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.5\linewidth}
        \vspace{3px}
        \centering
        \hspace*{-12px}
        \includegraphics[width=1\linewidth]{fig/method_sdf_1}
    \end{subfigure}
    \vspace*{-10px}
    \caption{{\bf Left: Problem when Predicting SDFs.} Illustration of a ray ({\color{red}red}) correctly hitting a surface ({\color{blue}blue}) causing the SDF values and occupancy values in the underlying voxel grid to be correct (\cf (a)). A noisy ray, however, causes all voxels along the ray to get invalid distances assigned (marked {\colorbox{red!25}{red}}; \cf (b)). When using occupancy, in contrast, only the voxels behind the surface are assigned invalid occupancy states (marked {\colorbox{red!25}{red}}); the remaining voxels are labeled correctly (marked {\colorbox{green!25}{green}}; \cf (c)).
    {\bf Right: Proposed Gaussian-to-Bernoulli Transformation.} For $p(y_i) := p(y_i | z) = \mathcal{N}(y_i;\mu_i(z), \sigma^2)$ ({\color{blue}blue}), we illustrate the transformation discussed in Section \ref{subsec:method-inference}, allowing to use the binary observations $x_i$ (for $x_i \neq \uk$) to supervise the SDF predictions. This is achieved, by transforming the predicted Gaussian distribution to a Bernoulli distribution with occupancy probability $\theta_i(\mu_i(z)) = p(y_i \leq 0)$ ({\color{blue}blue area}).}
    \label{fig:method-sdf}
    \vspace*{-0.25cm}
\end{figure}

In order to still work with SDFs (to achieve sub-voxel accuracy) we propose to define $p(y_i = x_i | z)$ through a simple transformation. In particular, as $p(y_i | z)$ is modeled as Gaussian distribution $p(y_i | z) = \mathcal{N}(y_i ; \mu_i(z), \sigma^2)$ where $\mu_i(z)$ is predicted using the fixed decoder (and $\sigma^2 = 1$) and $x_i$ is binary (for $x_i \neq \uk$), we introduce a mapping $\theta_i(\mu_i(z))$ transforming the predicted Gaussian distribution to a Bernoulli distribution with occupancy probability $\theta_i(\mu_i(z))$, \ie, $p(y_i = x_i|z)$ becomes $\text{Ber}(y_i = x_i; \theta_i(\mu_i(z)))$.
%
%ag: no need to discuss this I think
%This is necessary, as the observation $x_i \neq \uk$ is inherently binary. Although we could derive distance functions from the observations, \eg, along the rays corresponding to observed points \ag{You had a paper here, right?}, even slight noise can result in large changes in the derived distances (along the whole ray).
%
%\begin{align}
%p(y_i = x_i | z) = \text{Ber}(y_i = x_i; \theta_i(\mu_i(z)))
%\end{align}
As we defined occupied voxels to have negative sign in the SDF, we can derive the occupancy probability $\theta_i(\mu_i(z))$ as the probability of a negative distance:
\begin{align}
\theta_i(\mu_i(z)) &= \mathcal{N}(y_i \leq 0; \mu_i(z), \sigma^2)\label{eq:sdf}\\
%& = \int_{y_i'}^0 \mathcal{N}(y_i' \leq 0; \mu_i(z), \sigma^2) dy_i'\\
&= \frac{1}{2} \left(1 + \text{erf}\left(\frac{- \mu_i(z)}{\sigma \sqrt{\pi}}\right)\right).\label{eq:sdf-erf}
\end{align}
%
Here, $\text{erf}$ is the error function which, in practice, is approximated following \cite{Abramowitz1974}. Equation \eqref{eq:sdf} is illustrated in Figure~\ref{fig:method-sdf} where the occupancy probability $\theta_i(\mu_i(z))$ is computed as the area under the Gaussian bell curve for $y_i \leq 0$. This per-voxel transformation can easily be implemented as non-linearity layer and its derivative \wrt $\mu_i(z)$ is -- by construction -- a Gaussian distribution. Overall, this transformation allows to predict SDFs while using binary observations.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
% \usepackage{graphicx}
%     \DeclareGraphicsExtensions{.png, .jpeg}
% \usepackage{caption}
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\title{STAT 775: Machine Learning \\ HW 08}
\author{Terence Henriod}
\date{\today}

\begin{document}

\clearpage            % All of
\maketitle            % this,
\thispagestyle{empty} % removes the page number from the title page

\begin{abstract}
In this assignment, we use Support Vector Machines (SVM) to classify hand-written digits that are typically hard to discern from one another for classifiers.
\end{abstract}

\newpage
\section{Exercise 01}
\subsection{Problem Statement}
Using the zip.data data set from the ESL website and perform classification using Support Vector Machines. Perform binary classification on $7$s vs. $9$s, $3$s vs. $5$s, and $0$s vs. $8$s. You may use a library since implementing the SVM training algorithm is complex.
%
%
\subsection{Results}
The SVM method performed pretty well. Without fiddling with which kernel type was used too much or any parameter values, classification performance was comparable to the better methods we have explored this semester. It appeared that the linear kernel was the best kernel.
%
\subsubsection{The Tables}
The confusion matrices for classification are shown below.\\
\newline
$7$s vs. $9$s:\\
\newline
\begin{tabular}{| c || c | c || c |}
  \hline
  Actual/Prediction &   7 &   9 &\% Correct \\
  \hline
  \hline
  7                 & 136 &  11 & 92.5 \\
  \hline
  9                 &   2 & 175 & 98.9 \\
  \hline
  \hline
  Overall           &     &     & 96.0 \\
  \hline
\end{tabular}\\
\newline
\newline
$3$s vs. $5$s:\\
\newline
\begin{tabular}{| c || c | c || c |}
  \hline
  Actual/Prediction &   3 &   5 &\% Correct \\
  \hline
  \hline
  3                 & 152 &  14 & 91.6 \\
  \hline
  5                 &  13 & 147 & 91.9 \\
  \hline
  \hline
  Overall           &     &     & 91.7 \\
  \hline
\end{tabular}\\
\newline
\newline
$0$s vs. $8$s:\\
\newline
\begin{tabular}{| c || c | c || c |}
  \hline
  Actual/Prediction &   0 &   8 &\% Correct \\
  \hline
  \hline
  0                 & 351 &   8 & 97.8 \\
  \hline
  8                 &  15 & 151 & 91.0 \\
  \hline
  \hline
  Overall           &     &     & 95.6 \\
  \hline
\end{tabular}\\
\newline
%
\subsubsection{The Code}
The following R code was used to perform classification:
\begin{verbatim}
##############
# STAT775: Machine Learning
#
# HW08
# Exercise 01
#
# Using the zip.data data set from the ESL website, use Support Vector Machines
# (SVM) to perfrom classification on traditionally hard to differentiate digits:
# 7s vs. 9s, 3s vs. 5s, and 0s vs. 8s.
###############

#
# Initial Setup
#
setwd("C:/Users/Terence/Documents/GitHub/STAT775/HW07")

#
# Data Cleaning
#
DATA.PATH <- "../DataSets/zip.data/"
ZIP.TRAIN.FILE.NAME <- paste0(DATA.PATH, "zip.train")
ZIP.TEST.FILE.NAME <- paste0(DATA.PATH, "zip.test")

construct.classification.frame <- function(data, targets) {
  return(
    data.frame(
      'target' = targets,
      'prediction' = rep(0, length(targets)),
      data
    )
  )
}

get.data <- function(
  train.file, test.file, class.1, class.2, num.principle.components = 50) {
  # Args:
  #   train.file:
  #   test.file:
  #   class.1:
  #   class.2
  #   num.principle.components:
  #

  train <- read.table(train.file)
  train <- subset(train, V1 == class.1 | V1 == class.2)
  for (i in 1:nrow(train)) {
    train[i, 1] <- if (train[i, 1] == class.1) {1} else {2}
  }

  test <- read.table(test.file)
  test <- subset(test, V1 == class.1 | V1 == class.2)
  for (i in 1:nrow(test)) {
    test[i, 1] <- if (test[i, 1] == class.1) {1} else {2}
  }

  pca.model <- prcomp(train[, -1])

  train[, -1] <- predict(pca.model, train[, -1])
  train <- train[, 1:(num.principle.components + 1)]
  train <- construct.classification.frame(
    targets = train[, 1],
    data = train[, -1]
  )

  test[, -1] <- predict(pca.model, test[, -1])
  test <- test[, 1:(num.principle.components + 1)]
  test <- construct.classification.frame(
    targets = test[, 1],
    data = test[, -1]
  )

  return(list(
    train = train,
    test = test
  ))
}

#
# Data Presentation and Evaluation
#

compute.confusion.matrix <- function(predictions, targets, num.classes = 2) {
  confusion.matrix <- matrix(0, nrow = num.classes + 1, ncol = num.classes + 1)
  for (i in 1:length(predictions)) {
    confusion.matrix[targets[[i]], predictions[[i]]] <-
      confusion.matrix[targets[[i]], predictions[[i]]] + 1
  }
  confusion.matrix[num.classes + 1, num.classes + 1] <-
    sum(diag(confusion.matrix)) / sum(confusion.matrix)
  for (i in 1:num.classes) {
    confusion.matrix[i, num.classes + 1] <-
      confusion.matrix[i, i] / sum(confusion.matrix[i, 1:num.classes])
    confusion.matrix[num.classes + 1, i] <-
      confusion.matrix[i, i] / sum(confusion.matrix[1:num.classes, i])
  }
  return(confusion.matrix)
}

#
# Main
#
library('e1071')

tests <- list(
  list(7, 9),
  list(3, 5),
  list(0, 8)
)

test <- tests[[1]]

for (test in tests) {
  data <- get.data(
    train.file = ZIP.TRAIN.FILE.NAME,
    test.file = ZIP.TEST.FILE.NAME,
    class.1 = test[[1]],
    class.2 = test[[2]],
    num.principle.components = 50
  )

  svm.model <- svm(
    x = data$train[, -(1:2)],
    y = data$train[, 1],
    type = 'nu-classification',
    kernel = 'linear'
  )
  data$test$prediction <- (predict(svm.model, data$test[, -(1:2)]))

  print(compute.confusion.matrix(
    predictions = data$test$prediction,
    targets = data$test$target,
    num.classes = 2
  ))
}
\end{verbatim}
%
\end{document}


\section{Power analysis}
\label{sec:power}

In this section, we'll calculate the variance of our different annotation schemes.

\newcommand{\xh}{\hat{x}}
\newcommand{\xb}{\bar{x}}
\newcommand{\yh}{\hat{y}}
\newcommand{\yb}{\bar{y}}
\newcommand{\rh}{\hat{r}}
\newcommand{\ph}{\hat{p}}

Let $\sC$ be a corpus (our ``population'') that we wish to compute statistics over
and $\sP$ be the population of pooled contexts,
  with $r \eqdef \Pr(c \in \sP \given c \in \sC)$ (the recall of the pool).

Let $E$ be a randomly drawn subset of $\sC$ with $n_E$ samples for exhaustive annotation and
    $P$ be a randomly drawn subset of $\sP$ with $n_P$ samples for pooling annotation.

\begin{figure}
  \begin{subfigure}{0.49\textwidth}
    \includegraphics[width=\textwidth]{figures/variance-ratio-pool}
    \caption{\label{fig:pool-for-corpus} The relative efficiency of using pooled annotations versus exhaustive annotation, assuming that there are twice as many pooled annotations as there are exhaustive annotations.}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.49\textwidth}
    \includegraphics[width=\textwidth]{figures/variance-ratio-unassessed}
    \caption{\label{fig:pool-for-unassessed} The relative efficiency of using unassessed annotations versus exhaustive annotation, assuming that there are twice as many pooled annotations and ten times as many unassessed contexts as there are exhaustive annotations.}
  \end{subfigure}
\end{figure}

\begin{theorem}[Relative efficiency of using pooled annotations to compute corpus statistics]
  \label{thm:variance-pooled-recall}
  % Define all the variables
  Let $x$ be a statistic that we wish to compute on $\sC$ (e.g.\ recall),
    $\rh$ be an unbiased estimator of $r$ on $E$ with variance $\sigma_r^2$,
    $\yh_p$ be an unbiased estimator of $x$ on $P$ with variance $\sigma_y^2$,
  then $\xh_p = \rh \yh_p$ is an unbiased estimator of $x$ on $P$ with variance
  $$
  \sigma_p^2 = \sigma_r^2 \sigma_y^2 + y^2 \sigma_r^2 + r^2 \sigma_r^2.
  $$

  Furthermore, if $\xh_e$ be an unbiased estimator of $x$ on $E$ with variance $\sigma_e^2$,
  the relative efficiency of $\xh_p$ with respect to $\xh_e$ is,
  \begin{align*}
  \frac{\sigma_e^2}{\sigma_p^2}
  &\approx \frac{1-x}
      {
      y (1-r) + \frac{n_E}{n_P} r (1-y)
      }.
  \end{align*}
\end{theorem}
\begin{proof}
  \begin{align*}
    \sigma_e^2 &= \frac{x (1-x)}{n_E + 1} \\
    \sigma_r^2 &= \frac{r (1-r)}{n_E + 1} \\
    \sigma_y^2 &= \frac{y (1-y)}{n_P + 1} \\
    \sigma_p^2
    &\approx y^2 \frac{r (1-r)}{n_E + 1} + r^2 \frac{y (1-y)}{n_P + 1} \\
    &= x^2 \left( \frac{(1-r)/r}{n_E + 1} + \frac{(1-y)}{n_P + 1} \right).
  \end{align*}
\end{proof}

\begin{lemma}[Benefit of combining estimators]
  Let $x_1, \dots, x_n$ be independent and unbiased estimators for a statistic $x$ with variances $\sigma_1^2, \dots \sigma_n^2$.
  Let $z = \sum_{i=1}^n \alpha_i x_i$ where $\sum_{i=1}^n \alpha_i = 1$ be a combined estimator for $x$ with minimal variance.
  Then, the relative efficiency, i.e ratio of variances, of $z$ and $x_1$ is,
  $\frac{\sigma_z^2}{\sigma_1^2} = \sum_{i=1}^n \frac{\sigma_i^2}{\sigma_1^2}$.

  In other words, relative efficiency of a combined estimator with respect
  to an original estimator is the sum of relative efficiencies of each component
  estimator with respect to the original estimator.
\end{lemma}