\documentclass[twocolumn]{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsthm}

\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]

% Recap the problem and motivation. This part should be brief, since we've already
% heard about this.

% Briefly describe your overall approach. A small concrete example snippet or two
% would be very helpful.

% Results. Describe what you have achieved. What remains to be done to make this a
% full-fledged research result?

% Lessons Learned. Reflect on the work you did, so you and I can learn from it.
% How did your approach differ from what you originally proposed, and why? What
% was more (or less) challenging than you expected? What would you do differently
% next time? What do you now see as the strengths/weaknesses of a tool like Coq,
% and what does that imply about possible research directions to pursue?

% Code. Please submit all of your Coq code along with some brief documentation so
% I can relatively easily play with it and understand what's going on.
\title{Verified Sentential Decision Diagrams}
\author{Steven Holtzen\\Saketh Ram Kasibatla}
\date{\today}
\begin{document}
\maketitle

\section{Introduction and Motivation}
The goal of this project is to construct and reason about sentential decision
diagrams (SDDs) in Coq. A sentential decision diagram is a data structure which encodes
an arbitrary Boolean formula; see Fig.\ref{fig:sdd} for an example SDD.

\begin{figure}
  \includegraphics[width=\linewidth]{sdd.png}
  \caption{An SDD for the Boolean formula $f = (A \land B) \lor (B \land C) \lor
    (C \land D)$. Reproduced from \cite{Darwiche2011}.}
  \label{fig:sdd}
\end{figure}

The SDD data structure has several desirable properties:
\begin{enumerate}
\item Canonicity. Any two equivalent Boolean formulas will always compile to
  the same SDD.
\item Determinism. Any given Or-node in an SDD will have at most one true
  child.
\item Decomposability. The children of any given And-node in an SDD share
  no variables.
\end{enumerate}
These properties allow one to perform many queries in linear time the SDD data
structure, for example: model enumeration, weighted model counting,
satisfiability and validity checking. In our work, we focused on non-canonical
(i.e. uncompressed) SDDs, which are easier to represent and manipulate.

The key operations that we focused on implementing were (1) SDD compilation;
(2) SDD application; (3) SDD evaluation.

\subsection{Proof Goals}
Our original goal was to prove the following properties:
  \begin{enumerate}
  \item Correctness: For all Boolean formalae $e$ and inputs $i$,
    \texttt{eval\_boolexpr}($e$, $i$) =
    \texttt{eval\_sdd}(\texttt{compile}($e$),  $i$).
  \item Prove that the resulting SDD is decomposable.
  \item Prove that the resulting SDD is deterministic.
  \item Prove that \texttt{init\_var} generates SDDs with respect to the same
    $v$-tree (will most likely be a necessary lemma for proving the correctness
    of \texttt{apply}).
  \end{enumerate}
During the course of implementation, these goals proved to be too ambitious. We
were able to prove, subject to certain assumptions, that SDDs remain
decomposable after application; See Fig. \ref{fig:decomposable}. We relaxed goal (1) to be to prove a particular
property of the SDD post-application; see Fig.\ref{fig:unsatthm}.

\begin{figure}
\begin{verbatim}
Theorem sdd_decomposable :
  forall sdd v,
    sdd_vtree sdd v ->
    vtree_correct v ->
    decomposable sdd.
\end{verbatim}
  \caption{Theorem around decomposability}
  \label{fig:decomposable}
\end{figure}

\begin{figure}
\begin{verbatim}
Theorem apply_unsat :
  forall sdd1 sdd2 sddres,
    sdd_unsat sdd1 ->
    sdd_apply OAnd sdd1 sdd2 sddres ->
    sdd_unsat sddres.
\end{verbatim}
  \caption{The apply unsat theorem.}
  \label{fig:unsatthm}
\end{figure}
The proof of this fact required a custom inductive hypothesis on the derivation
of \texttt{sdd\_apply}, which we will elaborate on in later sections. The reason
that this approach can not be extended to prove stronger claims and possbile
future work will also be discussed.

\section{Approach}

\subsection{OCaml Prototype and Fixpoints vs. Inductive Constructions}
We created an initial OCaml implementation of SDD compilation and application to
guide our Coq implementation. This implementation can be found in the
\texttt{ml/} directory of the project.

Some key observations of this OCaml code is that it is reasonably complex; a
direct translation of this code to Gallina was attempted, but ultimately we
decided against it for the following reasons.

\begin{itemize}
\item It was difficult to establish that all the recursive calls (especially
  \texttt{apply}) provably terminated.
\item It is difficult to reason about Gallina code, as we would have had to prove many assumptions from the code, before being able to reason about its correctness.
\end{itemize}

Instead, we opted to specify the SDD data structure as an inductive in Gallina, and to specify operations on this data structure using inductive relations. This approach allowed us immediate access to assumptions that were integral to proving the theorems we were interested in, and made it easy to strengthen these assumptions as necessary.

\subsection{Inductive SDD Data Structure}
We used the following compact definition for our SDD datastructure:

\begin{verbatim}
Inductive atom : Type :=
| AFalse : atom
| ATrue : atom
| AVar :  nat -> bool -> atom.

Inductive sdd : Type :=
| Or: list (sdd * sdd) -> sdd
| Atom : atom -> sdd.
\end{verbatim}
An \texttt{or} node is simply a list of pairs $(p_i, s_i)$ (primes and subs).

\subsection{SDD V-Tree}\label{sec:vtree}
We experimented with (1) representing the V-Tree as a dependent type of the SDD
and (2) creating a seperate \texttt{sdd\_vtree : sdd -> vtree} inductive relation which
generates a V-Tree from a particular SDD:

\begin{verbatim}
Inductive sdd_vtree : sdd -> vtree -> Prop :=
| AtomTrue : forall n,
 sdd_vtree (Atom ATrue) (VAtom n)
| AtomFalse : forall n,
 sdd_vtree (Atom AFalse) (VAtom n)
| AtomVar : forall n b,
 sdd_vtree (Atom (AVar n b)) (VAtom n)
| OrEmpty : forall v, sdd_vtree (Or []) v
| OrSingle: forall prime sub lvtree rvtree tail,
 sdd_vtree prime lvtree ->
 sdd_vtree sub rvtree ->
 sdd_vtree (Or (tail)) (VNode lvtree rvtree) ->
 sdd_vtree (Or ((prime, sub) :: tail))
  (VNode lvtree rvtree)

\end{verbatim}

This relation is used throughout our proofs as a correctness assumption. A valid SDD must obey some V-Tree.

\subsection{SDD Application}
SDD application is the process of combining two SDDs $\alpha$ and $\beta$ into a
third SDD $\gamma$ according to some operation $\circ$. We defined application
as a 4-argument inductive relation:
\begin{verbatim}
Inductive sdd_apply :
  op -> sdd -> sdd -> sdd -> Prop
\end{verbatim}

where \texttt{op} is either $\land$ or $\lor$. We decomposed this relation in 3
mutually recursive sub-relations:
\begin{enumerate}
\item \texttt{atom\_apply}, which handles applying together atoms
\item \texttt{apply\_or\_list}, which takes two SDD \textt{or}-nodes as arguments and
  produces a new SDD \texttt{or}-node.
\item \texttt{apply\_single\_list}, which takes a prime $p$, a sub $s$, and an
  \texttt{or}-node $[(p_i, s_i)]$ as an argument and produces the SDD of the
  form $[(p_i \land p, s_i \circ s)]$ for all $i$, where $\circ$ is applying
  \texttt{op}.

  For this operation, if $p_i \land p$ is not satisfiable, it is not to be
  included in the final \texttt{or}-node which is produced. This necessitates
  an \texttt{sdd\_sat} and \texttt{sdd\_unsat} relations.
\end{enumerate}

\subsection{Custom Inductive Hypothesis}
Ultimately in order to prove theorems of the form in Fig\ref{fig:unsatthm}, we
need to be able to decompose along the primes and subs of the SDD. Because we implemented \texttt{or}-nodes' children as lists of pairs of nodes, the default inductive hypothesis that Coq inferred for the SDD datastructure, and for the \texttt{sdd\_apply} inductive construction proved too weak. These default inductive hypotheses failed to break down the lists of primes and subs embedded in our SDD data structure.

To resolve this issue, we created custom inductive hypotheses for the SDD data structure, and for the \texttt{sdd\_apply} inductive construction (\texttt{sdd\_ind'} and \texttt{sdd\_apply\_ind'} respectively). Refer to the attached source code for implementations of these hypotheses. Each of these hypotheses contains a fixpoint which operates on the proof goal. When the user calls the tactic \texttt{induction sdd using sdd\_ind'}, she is asked to prove the goal, in relation to each prime and sub. The machinery of the custom inductive hypothesis then builds a proof of the general goal, using proofs of each piece. In such a manner, \texttt{sdd\_apply\_ind'} gave us inductive hypotheses strong enough to prove \texttt{apply\_unsat}.

\subsection{SDD Compilation}
SDD compilation is the process of transforming a Boolean expression into an SDD.
This procedure requires the following components:
\begin{enumerate}
\item An implementation of Boolean expressions. (\texttt{BoolExpr.v})
\item A method which generates an sdd from an atom (\texttt{sdd\_of\_atom})
\end{enumerate}

We compile SDDs from boolean expressions in a bottom-up fashion. First, each atom is turned into an SDD atom. Then, we use \texttt{sdd\_apply} to combine these atoms according to the boolean expression (and them together for each and node, or them together for each or node).

\subsection{Decomposability}

Our proof of decomposability relies heavily on varSets (implemented in \texttt{VarSet.v}). For any SDD or vtree, we can obtain a varSet, which is the set of all variables mentioned in the SDD/vtree. We prove decomposability by showing that the varSet of an SDD is a subset of the varSet of any vtree it obeys (see \texttt{sdd\_vtree\_vars}). We then use this result, along with induction on the \texttt{sdd\_vtree} relation to prove the theorem shown in Fig. \ref{fig:decomposable}. Interestingly, we did not need a custom inductive hypothesis to prove this property, as the \texttt{sdd\_vtree} relation encoded enough information about how to decompose the SDD and vtree in question to push the proof through.

\section{Results}

Our final codebase contains the following inductive constructions:

\begin{itemize}
\item \texttt{sdd\_apply}
\item \texttt{sdd\_vtree}
\item \texttt{sdd\_sat}
\item \texttt{sdd\_unsat}
\item \texttt{sdd\_of\_var}
\item \texttt{sdd\_eval}
\item Judgements around varSets (see \texttt{VarSet.v} and judgements containing \texttt{\_varSet} in \texttt{IndSdd.v})
\item Judgements around boolean expressions (see \texttt{BoolExp.v})
\end{itemize}

We successfully proved \texttt{sdd\_unsat} and \texttt{sdd\_decomposable}, using custom inductive hypotheses and inductive relations on sdds and vtrees.

In order to make our codebase a full research result, we would need to prove:

\begin{itemize}
\item apply preserves vtree - that the operands and result of the \texttt{sdd\_apply} operation all obey the same vtree. This result, combined with our proof of decomposability would be enough to show that any compiled SDD is decomposable
\item determinism (as formulated in the original goals)
\item correctness (also as formulated in the original goals).
\end{itemize}

Together, these properties would prove that our implementation of SDDs is valid, and would be enough to make our implementation a point of comparsion for other researchers.

\section{Lessons Learned}

\subsection{Challenges}

Working with Coq this quarter has given us a good feel for the differences between writing software in a traditional context, and doing so for the purpose of verification.

Because the type system and language are so powerful, it is easy to formulate the same solution in many ways. A great deal of what we learned to do in this course was to pick the approach that would be easiest to prove correct. For example, we chose to use inductive judgements to prove our algorithms correct, instead of working directly with Gallina code.

We also had trouble working with the \texttt{apply} operation, as it performs complex manipulations that Coq couldn't work with automatically. We produced custom inductive hypotheses to try and get around this, but working with the proofs for apply was much more difficult without Coq doing a lot of the decomposition automatically.

\subsection{Things we'd do Differently}

If we were to start from scratch, we would formulate our SDD operations as inductive judgements from the start. Each inductive judgement would include invariant information (the vtree) directly. While these judgements would not be formulated in the simplest way possible, having invariants on hand would greatly ease the proof process. We would use a similar SDD implementation, as it guarantees a degree of structural correctness that it is useful to have. Overall, using inductive judgements and encoding information directly would help us produce the results shown here more quickly, and would give us a better jumping off point to begin to reason about our original goals.

\subsection{Strengths and Weaknesses of Coq}

Overall, Coq is a powerful tool for proving software correct. By helping the user keep track of proof obligations and their knowledge, it helps the user formulate and tweak strategies for their proofs.

But, it relies heavily on a complex dependent type system and (at times) a strong user understanding of the Curry-Howard isomorphism. Both of these difficult concepts make  Coq in its current form fairly inaccessible to the average user. If this system is to be adopted in industry (perhaps only for mission critical systems), the system should be something that an industry professional can use confidently, without requiring a deep understanding of these esoteric concepts.

\bibliographystyle{plain}
\bibliography{bib}
\end{document}


\chapter{Computational aspects}

\abstract{In this chapter
we discuss some of the classical methods for integrating a function. The methods we discuss are  the trapezoidal, rectangular and Simpson's rule for equally spaced
abscissas  and integration approaches
based on Gaussian quadrature. The latter are more suitable
for the case where the abscissas are not equally spaced.
The emphasis is on
methods for evaluating few-dimensional (typically up to four dimensions) integrals. In
chapter \ref{chap:mcint}
we show how Monte Carlo methods can be used to compute multi-dimensional
integrals.
We discuss also how to compute
singular integrals.
We end this chapter with an extensive discussion on MPI and parallel computing.
The examples focus on parallelization of algorithms for computing integrals. }


\newpage
%HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

\section{Calculator with memory\label{calc}}

%HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Back to the calculator from Section~\ref{arith}.
We suggest one more improvement:
the ability to store computed values for subsequent use.
The idea is that
a session may look like this:

\small
\begin{Verbatim}[samepage=true,xleftmargin=15mm,baselinestretch=0.8]
 java Calc
 > n = 3/2
 > n = (3/n + n)/2
 > (3/n + n)/2
   1.7321428571428572
\end{Verbatim}
\normalsize

In the first line we store the number \tx{3/2} under name "\tx{n}".
In the second, we replace it by \tx{(3/n + n)/2}, computed using the stored value.
Finally, we use the newest \tx{n} to compute one more value and print it.
(Is it just an accident that the result resembles $\sqrt{3}$\,?)

To achieve this kind of functionality, you can modify the grammar as follows:

\small
\begin{Verbatim}[frame=single,framesep=2mm,samepage=true,xleftmargin=15mm,xrightmargin=15mm,baselinestretch=0.8]
   Input   = Space (Store / Print) !_ ;
   Store   = Name Space Equ Sum {store} ;
   Print   = Sum {print} ;
   Sum     = Sign Product (AddOp Product)* {sum} ;
   Product = Factor (MultOp Factor)* {product} ;
   Factor  = Digits? "." Digits Space {fraction}
           / Digits Space {integer}
           / Lparen Sum Rparen {unwrap}
           / Name Space {retrieve} ;
   Sign    = ("-" Space)? ;
   AddOp   = [-+] Space ;
   MultOp  = [*/]? Space ;
   Lparen  = "(" Space ;
   Rparen  = ")" Space ;
   Equ     = "=" Space ;
   Digits  = [0-9]+ ;
   Name    = [a-z]+ ;
   Space   = " "* ;
\end{Verbatim}
\normalsize

We have here two alternative forms of \tx{Input}
to perform the two tasks,
and one more way to obtain a \tx{Factor}:
by retrieving a stored result.
The results will be stored in a hash table \tx{dictionary}
that resides in the semantics object.

The semantic actions for \tx{Store} and \tx{Print} are:

\small
\begin{Verbatim}[frame=single,framesep=2mm,samepage=true,xleftmargin=15mm,xrightmargin=15mm,baselinestretch=0.8]
   //-------------------------------------------------------------------
   //  Store = Name Space Equ Sum
   //            0    1    2   3
   //-------------------------------------------------------------------
   void store()
     { dictionary.put(rhs(0).text(),(Double)rhs(3).get()); }
\end{Verbatim}
\normalsize

\small
\begin{Verbatim}[frame=single,framesep=2mm,samepage=true,xleftmargin=15mm,xrightmargin=15mm,baselinestretch=0.8]
   //-------------------------------------------------------------------
   //  Print = Sum
   //           0
   //-------------------------------------------------------------------
   void print()
     { System.out.println((Double)rhs(0).get()); }
\end{Verbatim}
\normalsize

\newpage
The stored result is retrieved by semantic action
for the fourth alternative of \tx{Factor}:

\small
\begin{Verbatim}[frame=single,framesep=2mm,samepage=true,xleftmargin=15mm,xrightmargin=15mm,baselinestretch=0.8]
   //-------------------------------------------------------------------
   //  Factor = Name Space
   //             0    1
   //-------------------------------------------------------------------
   void retrieve()
     {
       String name = rhs(0).text();
       Double d = dictionary.get(name);
       if (d==null)
       {
         d = (Double)Double.NaN;
         System.out.println
           (rhs(0).where(0) + ": '" + name + "' is not defined.");
       }
       lhs().put(d);
     }
\end{Verbatim}
\normalsize

You have to handle the case where the name is not in the dictionary.
The example shown here returns in such a case a \tx{NaN} (Not a Number) object.
Using \tx{NaN} in subsequent calculations produces inevitably a \tx{NaN} result.
A message printed to \tx{System.out} explains where this \tx{NaN} comes from.
Note the call to \tx{rhs(0).where(0)} that returns a text telling where to find the name
in the input.

You find the modified grammar and semantics in \tx{example7}.
You find there also a file \tx{Calc.java} containing invocation of the calculator.
Copy these files to \tx{work} directory, generate a new parser, and compile
everything.
You can now invoke the calculator as shown at the beginning of this section.


\documentclass{article}
\usepackage{bm}
\usepackage{amssymb,amsmath}
\usepackage{color}
\usepackage{graphics}
\input{symbols}

\title{Minimization Algorithm in the GSI}
\author{Jing Guo}

\begin{document}
\maketitle

\section{Conventions}
This note describes the
optimization algorithm used by the current GSI system.

The notation used in this document
is selected to reflect a balanced consideration of
\begin{enumerate}
\item variable names used in the GSI software,
\item symbols used in GEOS DAS literature,
\item symbols used in related literature of optimization algorithms, and
\item notation consistency throughout this note.
\end{enumerate}

In this document, a superscript often categorizes the symbol, such
$()^b$ for background,
$()^a$ for analysis, and
$()^g$ for initial guess.
Exceptions
include $()^T$ for matrix-transpose and $()^{-1}$ for matrix-inverse,
numbers for exponents, or otherwise being specified in the text.

On the other hand, a subscript often denotes the instantiation of the
symbol, {\it i.e.} symbols evaluated at specific locations or with
respect to specific variables.

\section{Cost Function $J(x)$}
A variational data assimilation problem tries to minimize the
cost function in the following form
\begin{align}	\label{eq:cost-in-x}
J(x) &= \onehalf
  \Bigl\{
	(x-x^b)^T
	\bP^{-1}
	(x-x^b)
	+
	\bigl[ \cH(x) - o \bigr]^T
	\bR^{-1}
	\bigl[ \cH(x) - o \bigr]
  \Bigr\}
\end{align}
with respect to a gridded field $x$,
where
\begin{align*}
  x^b &: \text{a given background field of $x$}		\\
  o &: \text{observations}				\\
  \bR &: \text{observation error covariance}		\\
  \bP &: \text{background error covariance}		\\
  \cH &: \text{observation operator}
\end{align*}

Define
\begin{gather}
\label{eq:xhatsave}
\xhatsave \equiv x - x^b
\end{gather}
Then in term of $x$-increment $\xhatsave$,
the cost function is
\begin{subequations}
\begin{align}
\label{eq:cost-in-xhatsave}
J(x) &= \onehalf
  \Bigl\{
	(\xhatsave)^T
	\bP^{-1}
	\xhatsave
	+
	\bigl[ \cH(x) - o \bigr]^T
	\bR^{-1}
	\bigl[ \cH(x) - o \bigr]
  \Bigr\}		\\
\label{eq:x-in-xhatsave}
x &= x^b + \xhatsave
\end{align}
\end{subequations}

Define a preconditioned increment vector
\begin{align}
  \yhatsave &\equiv \bP^{-1}\xhatsave
\end{align}
Then in term of $y$-increment $\yhatsave$, the cost function is
\begin{subequations}
\begin{align}
\label{eq:cost-in-yhatsave}
J(x) &= \onehalf
  \Bigl\{
	(\yhatsave)^T
	\bP
	\yhatsave
	+
	\bigl[ \cH(x) - o \bigr]^T
	\bR^{-1}
	\bigl[ \cH(x) - o \bigr]
  \Bigr\}
  \\
\label{eq:x-in-yhatsave}
x &= x^b + \bP \yhatsave
\end{align}
\end{subequations}

The minimization of Eq. (\ref{eq:cost-in-x}) with respect to
$x$ is equivalent to
the minimization of Eq. (\ref{eq:cost-in-xhatsave}) with respect to
$\xhatsave$, and in particular, is equivalent to
the minimization of Eq. (\ref{eq:cost-in-yhatsave}) with respect to
$\yhatsave$.

\section{Gradient of Cost Function, $\nabla_\yhatsave J$}

The first thing of an iterative optimization algorithm is
to evaluate the gradient of the cost function.
Denoted by $\nabla_\yhatsave J$, the
gradient of the cost function $J(x)$ in Eq. (\ref{eq:cost-in-yhatsave})
with respect to $\yhatsave$ at $x=x^b+\xhatsave$ is
\begin{align}
\label{eq:grad-in-yhatsave}
  \nabla_\yhatsave J
    &=	\xhatsave +
	\bP\bH_x^T \bR^{-1}
	\bigl[ \cH(x) - o \bigr]
\end{align}
where
\begin{align*}
  \bH_x &\equiv \left. \frac{\partial \cH(x)}{\partial x}
  		\right|_{x}
\end{align*}
based on identities
\begin{gather*}
  \frac{\partial \cH(x(\yhatsave))}{\partial \yhatsave}
    =	\frac{\partial \cH\bigl(x(\yhatsave)\bigr)}{\partial x}	\cdot
	\frac{\partial          x(\yhatsave)      }{\partial \yhatsave}
    =	\bH_x \bP	\\
  \bP^T = \bP
\end{gather*}

For a cost function minimization process started with a given initial
``guess'' field $x^g$, let
\begin{gather}
\label{eq:xhat}
  \xhat \equiv x - x^g
\end{gather}
then
\begin{align*}
  x = x^b + \xhatsave = x^g + \xhat.
\end{align*}
Eq.~(\ref{eq:grad-in-yhatsave}) becomes
\begin{align}
\label{eq:grad-wrt-yhatsave-in-xhat}
  \nabla_\yhatsave J
    &=	\xhatsave +
	\bP\bH_x^T \bR^{-1}
	\bigl[ \cH(x^g+\xhat) - o \bigr]
\end{align}
Or correspondingly
\begin{align}
\label{eq:grad-wrt-xhatsave-in-xhat}
  \nabla_\xhatsave J
    &=	\yhatsave +
	\bH_x^T \bR^{-1}
	\bigl[ \cH(x^g+\xhat) - o \bigr]
\end{align}

\section{Minimization Algorithm}

The objective of an iterative step in the cost function minimization
process utilizing the gradient defined by
Eq.~(\ref{eq:grad-wrt-yhatsave-in-xhat})
is to find the next correction step $\delta x$ which reduces
the cost function in $x$
\begin{align*}
  J(x+\delta x) < J(x);
\end{align*}
Or in $\xhat$
\begin{align*}
  J(\xhat+\delta\xhat) < J(\xhat).
\end{align*}
Note that $\delta\xhat \equiv \delta x$ by the definition of $\xhat$.

This minimization objective can be achieved through the
algorithm in the GSI system, shown in Fig.~\ref{alg:algo} in
mathematical terms, and in Fig.~\ref{alg:code} in
program terms.

%\input{algo-pcgsoi-math}
\begin{figure}[hb]
\resizebox{!}{!}{
\includegraphics{algo-pcgsoi-math.eps}}

\caption{Routine {\tt PCGSOI()} in mathematical terms.}
\label{alg:algo}
\end{figure}

In Fig.~\ref{alg:algo},
the objective of code segment line 6-9 is to compute the gradient
with respect to $\xhat$ ({\em i.e.} $\xhatsave$) and $\yhatsave$,
respectively.  Line 10-13 is to compute the correction directions
in $\xhat$ ($\xhatsave$) and $\yhatsave$, where line 11 for
scalar $\beta$
is in agreement with Eqs.~(106) and (107) in Navon and Legler (1987),
in the ``one-step limited-memory BFGS quasi-Newton update'' form.
Line 14 is to compute the step-size
that minimize the cost function along the given correction direction
$\dirx_i$ or $\diry_i$, which
can also be noted as ``solve
\begin{align*}
0 &=(\dirx_i)^T\!\nabla_\xhat J	\\
  &=(\dirx_i)^T\!\yhatsave +
    (\dirx_i)^T\!\bH^T\Rinv
	\bigl(\cH(x_j^g+\xhat_{i-1}+\alpha_i\dirx_i)-o\bigr)	\\
  &=(\diry_i)^T\!\xhatsave +
    (\dirx_i)^T\!\bH^T\Rinv
	\bigl(\cH(x_j^g+\xhat_{i-1}+\alpha_i\dirx_i)-o\bigr)	\\
  &=(\diry_i)^T\!\nabla_\yhatsave J
\end{align*}
for $\alpha_i$''.
Line 15-17, is to compute the next values of $\xhat$ ($\xhatsave$)
and $\yhatsave$.

%\input{algo-pcgsoi-code}
\begin{figure}[hb]
\resizebox{!}{!}{
\includegraphics{algo-pcgsoi-code.eps}}

\caption{Routine {\tt PCGSOI()} in program terms.
Note two new operators ${\bm L}$ and ${\bm L}^{-1}$.
} \label{alg:code}
\end{figure}

The algorithm shown in Fig.~\ref{alg:algo} and Fig.~\ref{alg:code} is
known as the ``inner-loop'' of the minimization process in the GSI, or
\verb"SUBROUTINE PCGSOI()".  Note that beside that Fig.~\ref{alg:algo}
is in mathematical terms and Fig.~\ref{alg:code} is in program terms,
Fig.~\ref{alg:code} also identifies the difference between the source
grid-space defining background state $q^b$ and analysis $q^a$, and the
intermediate grid-space defining the guess state $x^g$ and the
increment state $\xhat$, and two linear operators
($\bL$ and $\bL^{-1}$) converting between
the two spaces back-and-forth.

The algorithm descriptions given in these two figures
include neither some configuration steps for the ``inner-loop'', nor
the so-called ``restart'' of the search directions.  Both are parts of
the algorithm known as the ``outer-loop'' of the process in the GSI,
or in \verb"SUBROUTINE GLBSOI()".

Figure~\ref{alg:twoloops} describes
the complete minimization process in its two level nested-loop form,
for the algorithm in routine \verb"GLBSOI()" as well as
its major lower level routines.
For convenience, some of these lower level
routines are marked as Fortran comments in the figure.
Note that the description of line 4-6 is highly simplified,
not an accurate representation of the complex details in this
part of the whole algorithm.

%\input{algo-glbsoi-code}
\begin{figure}[hb]
\resizebox{!}{!}{
\includegraphics{algo-glbsoi-code.eps}}
\caption{Minimization in two level nested loops
from routine {\tt GLBSOI()} down.
} \label{alg:twoloops}
\end{figure}

\end{document}


%[TODO: FROM VIDEO]
\section{Motion Analysis and Object Tracking}

\ifCPy

\cvCPyFunc{Acc}
Adds a frame to an accumulator.

\cvdefC{
void cvAcc( \par const CvArr* image,\par CvArr* sum,\par const CvArr* mask=NULL );
}
\cvdefPy{Acc(image,sum,mask=NULL)-> None}

\begin{description}
\cvarg{image}{Input image, 1- or 3-channel, 8-bit or 32-bit floating point. (each channel of multi-channel image is processed independently)}
\cvarg{sum}{Accumulator with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds the whole image \texttt{image} or its selected region to the accumulator \texttt{sum}:

\[ \texttt{sum}(x,y) \leftarrow \texttt{sum}(x,y) + \texttt{image}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

\cvCPyFunc{MultiplyAcc}
Adds the product of two input images to the accumulator.

\cvdefC{
void cvMultiplyAcc( \par const CvArr* image1,\par const CvArr* image2,\par CvArr* acc,\par const CvArr* mask=NULL );
}
\cvdefPy{MultiplyAcc(image1,image2,acc,mask=NULL)-> None}

\begin{description}
\cvarg{image1}{First input image, 1- or 3-channel, 8-bit or 32-bit floating point (each channel of multi-channel image is processed independently)}
\cvarg{image2}{Second input image, the same format as the first one}
\cvarg{acc}{Accumulator with the same number of channels as input images, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds the product of 2 images or their selected regions to the accumulator \texttt{acc}:

\[ \texttt{acc}(x,y) \leftarrow \texttt{acc}(x,y) + \texttt{image1}(x,y) \cdot \texttt{image2}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

\cvCPyFunc{RunningAvg}
Updates the running average.

\cvdefC{
void cvRunningAvg( \par const CvArr* image,\par CvArr* acc,\par double alpha,\par const CvArr* mask=NULL );
}
\cvdefPy{RunningAvg(image,acc,alpha,mask=NULL)-> None}

\begin{description}
\cvarg{image}{Input image, 1- or 3-channel, 8-bit or 32-bit floating point (each channel of multi-channel image is processed independently)}
\cvarg{acc}{Accumulator with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{alpha}{Weight of input image}
\cvarg{mask}{Optional operation mask}
\end{description}

The function calculates the weighted sum of the input image
\texttt{image} and the accumulator \texttt{acc} so that \texttt{acc}
becomes a running average of frame sequence:

\[ \texttt{acc}(x,y) \leftarrow (1-\alpha) \cdot \texttt{acc}(x,y) + \alpha \cdot \texttt{image}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

where $\alpha$ regulates the update speed (how fast the accumulator forgets about previous frames).

\cvCPyFunc{SquareAcc}
Adds the square of the source image to the accumulator.

\cvdefC{
void cvSquareAcc( \par const CvArr* image,\par CvArr* sqsum,\par const CvArr* mask=NULL );
}\cvdefPy{SquareAcc(image,sqsum,mask=NULL)-> None}

\begin{description}
\cvarg{image}{Input image, 1- or 3-channel, 8-bit or 32-bit floating point (each channel of multi-channel image is processed independently)}
\cvarg{sqsum}{Accumulator with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds the input image \texttt{image} or its selected region, raised to power 2, to the accumulator \texttt{sqsum}:

\[ \texttt{sqsum}(x,y) \leftarrow \texttt{sqsum}(x,y) + \texttt{image}(x,y)^2 \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

\fi

\ifCpp

\cvCppFunc{accumulate}
Adds image to the accumulator.

\cvdefCpp{void accumulate( const Mat\& src, Mat\& dst, const Mat\& mask=Mat() );}
\begin{description}
\cvarg{src}{The input image, 1- or 3-channel, 8-bit or 32-bit floating point}
\cvarg{dst}{The accumulator image with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds \texttt{src}, or some of its elements, to \texttt{dst}:

\[ \texttt{dst}(x,y) \leftarrow \texttt{dst}(x,y) + \texttt{src}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

The function supports multi-channel images; each channel is processed independently.

The functions \texttt{accumulate*} can be used, for example, to collect statistic of background of a scene, viewed by a still camera, for the further foreground-background segmentation.

See also: \cvCppCross{accumulateSquare}, \cvCppCross{accumulateProduct}, \cvCppCross{accumulateWeighted}

\cvCppFunc{accumulateSquare}
Adds the square of the source image to the accumulator.

\cvdefCpp{void accumulateSquare( const Mat\& src, Mat\& dst, \par const Mat\& mask=Mat() );}
\begin{description}
\cvarg{src}{The input image, 1- or 3-channel, 8-bit or 32-bit floating point}
\cvarg{dst}{The accumulator image with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds the input image \texttt{src} or its selected region, raised to power 2, to the accumulator \texttt{dst}:

\[ \texttt{dst}(x,y) \leftarrow \texttt{dst}(x,y) + \texttt{src}(x,y)^2 \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

The function supports multi-channel images; each channel is processed independently.

See also: \cvCppCross{accumulateSquare}, \cvCppCross{accumulateProduct}, \cvCppCross{accumulateWeighted}

\cvCppFunc{accumulateProduct}
Adds the per-element product of two input images to the accumulator.

\cvdefCpp{void accumulateProduct( const Mat\& src1, const Mat\& src2,\par
                        Mat\& dst, const Mat\& mask=Mat() );}
\begin{description}
\cvarg{src1}{The first input image, 1- or 3-channel, 8-bit or 32-bit floating point}
\cvarg{src2}{The second input image of the same type and the same size as \texttt{src1}}
\cvarg{dst}{Accumulator with the same number of channels as input images, 32-bit or 64-bit floating-point}
\cvarg{mask}{Optional operation mask}
\end{description}

The function adds the product of 2 images or their selected regions to the accumulator \texttt{dst}:

\[ \texttt{dst}(x,y) \leftarrow \texttt{dst}(x,y) + \texttt{src1}(x,y) \cdot \texttt{src2}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

The function supports multi-channel images; each channel is processed independently.

See also: \cvCppCross{accumulate}, \cvCppCross{accumulateSquare}, \cvCppCross{accumulateWeighted}

\cvCppFunc{accumulateWeighted}
Updates the running average.

\cvdefCpp{void accumulateWeighted( const Mat\& src, Mat\& dst,\par
                         double alpha, const Mat\& mask=Mat() );}
\begin{description}
\cvarg{src}{The input image, 1- or 3-channel, 8-bit or 32-bit floating point}
\cvarg{dst}{The accumulator image with the same number of channels as input image, 32-bit or 64-bit floating-point}
\cvarg{alpha}{Weight of the input image}
\cvarg{mask}{Optional operation mask}
\end{description}

The function calculates the weighted sum of the input image
\texttt{src} and the accumulator \texttt{dst} so that \texttt{dst}
becomes a running average of frame sequence:

\[ \texttt{dst}(x,y) \leftarrow (1-\texttt{alpha}) \cdot \texttt{dst}(x,y) + \texttt{alpha} \cdot \texttt{src}(x,y) \quad \text{if} \quad \texttt{mask}(x,y) \ne 0 \]

that is, \texttt{alpha} regulates the update speed (how fast the accumulator "forgets" about earlier images).
The function supports multi-channel images; each channel is processed independently.

See also: \cvCppCross{accumulate}, \cvCppCross{accumulateSquare}, \cvCppCross{accumulateProduct}

\fi


% Preamble.
\documentclass[12pt]{article}
\usepackage[margin=1.25in]{geometry}
\usepackage[fleqn]{amsmath}
\usepackage{textcomp}
\usepackage{gensymb}
\usepackage{amsfonts}
\usepackage{enumitem}
%\usepackage{tikz}  % Include for figures.
%\usepackage{subfiles}  % Include for subfiles.

%% Title macros.
\newcommand{\EXAMNUM}{2}
\newcommand{\NAME}{D. Choi}
\newcommand{\DATE}{2020-07-08}

\title{\vspace{-2\baselineskip}MATH 225 - Exam \#\EXAMNUM}
\author{\NAME}
\date{\DATE}

%% Formatting options.
%\pagenumbering{gobble}  % Include for single-page document.
\allowdisplaybreaks

% Macros.
%% Norm.
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
%% Rotation and reflection matrices.
\newcommand{\rota}[1]{\text{Rot}(#1 \degree)}
\newcommand{\refl}[1]{\text{Ref}(#1 \degree)}
%% Augmented matrix.
% https://tex.stackexchange.com/a/2238
\newenvironment{amatrix}[1]{%
  \left(\begin{array}{@{}*{#1}{c}|c@{}}
}{%
  \end{array}\right)
}
%% Contextualized by input/output bases.
\newcommand{\based}[3]{{\{#1\}}_{#2}^{#3}}

% Document.
\begin{document}
\maketitle

\section*{1.}
\textit{For each each of the ordinary differential equations listed below,
answer the following:
\begin{itemize}
	\item What is its order?
	\item Is it linear or non-linear?
	\item If non-linear, why?
	\item If linear, is it constant coefficient? Is it homogenous?
\end{itemize}
}
\begin{enumerate}[label=(\alph*)]
	\item $x^{\prime\prime\prime} + x \cdot x^{\prime\prime} + t x^\prime
	+ x = t^2$.
	\\[\baselineskip]
	Third-order, non-linear. \\
	It is non-linear because of the term $x \cdot x^{\prime\prime}$.

	\item $x^{\prime\prime\prime} + t x^{\prime\prime} + x = \text{sin}(t).$
	\\[\baselineskip]
	Third-order, linear. \\
	It is not constant coefficient because of the term $t x^{\prime\prime}$.
	\\
	It is not homogenous because of the term $\text{sin}(t)$.

	\item $x^2 + t x^\prime + x = \text{sin}(t)$.
	\\[\baselineskip]
	First-order, non-linear. \\
	It is non-linear because of the term $x^2$.

	\item $2x^{\prime\prime} + 4 x^\prime = -x$.
	\\[\baselineskip]
	Second-order, linear. \\
	It is constant coefficient because all of the coefficients are constant. \\
	It is homogenous because there are no constant terms.
\end{enumerate}
\newpage

\section*{2.}
\begin{enumerate}[label=(\alph*)]
	\item \textit{Solve
	$x^{\prime\prime} + 3x^\prime + 2x = 0$ with the restrictions that
	$x(0) = 2$ and $x^\prime(0) = 1$.}\\[\baselineskip]
	The roots of the characteristic equation are
	\begin{equation*}
		r^2 + 3r + 2 = 0 \quad \Rightarrow \quad r_1 = -1, \quad r_2 = -2.
	\end{equation*}
	With $2$ distinct real roots, the general solution is
	\begin{equation*}
		x(t) = c_1 e^{-t} + c_2 e^{-2 t}.
	\end{equation*}
	Using the system of equations given by the initial conditions,
	\begin{alignat*}{2}
		x(0) &=& \quad 2 &= c_1 + c_2 \\
		x^\prime(0) &=& \quad 1 &= -c_1 - 2 c_2,
	\end{alignat*}
	the coefficients can be solved for as
	\begin{equation*}
		c_1 = 5, \quad
		c_2 = -3.
	\end{equation*}
	Thus,
	\begin{equation*}
		\boxed{
			x(t) = 5 e^{-t} - 3 e^{-2 t}
		}.
	\end{equation*}

	\item \textit{Consider a similar, but non-homogenous o.d.e.:
	$x^{\prime\prime} + 3x^\prime + 2x = 2t + 3$. Show that
	$x(t) = 7e^{-2t} + 2e^{-t} + t$ solves this non-homogenous o.d.e.}
	\\[\baselineskip]
	Let's ``brute forge plug and chug'':
	\begin{align*}
		x &= 7e^{-2t} + 2e^{-t} + t \\
		x^\prime &= -14e^{-2t} - 2e^{-t} + 1 \\
		x^{\prime\prime} &= 28e^{-2t} + 2e^{-t}
	\end{align*}
	Thus,
	\begin{align*}
		2t + 3 &\overset{?}{=} x^{\prime\prime} + 3x^\prime + 2x
		\\ &\overset{?}{=}
		(28e^{-2t} + 2e^{-t})
		+ 3(-14e^{-2t} - 2e^{-t} + 1)
		+ 2(7e^{-2t} + 2e^{-t} + t)
		\\ &\overset{?}{=}
		(28 - 42 + 14)(e^{-2t})
		+ (2 - 6 + 4)(e^{-t})
		+ (2)(t) + (3)(1)
		\\
		2t + 3 &= 2t + 3.
	\end{align*}
\end{enumerate}
\newpage

\section*{3.}
\textit{Suppose that}
\begin{equation*}
	A =
	\begin{pmatrix}
		-1 & 1 & 1 \\
		1 & -1 & 1 \\
		1 & 1 & -1
	\end{pmatrix}
	\begin{pmatrix}
		1 & 0 & 0 \\
		0 & 2 & 0 \\
		0 & 0 & -1
	\end{pmatrix}
	\begin{pmatrix}
		-1 & 1 & 1 \\
		1 & -1 & 1 \\
		1 & 1 & -1
	\end{pmatrix}^{-1}
	.
\end{equation*}
\begin{enumerate}[label=(\alph*)]
	\item \textit{What is
		$A^{1001} \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}$
	?}
	\\[\baselineskip]
	Note that
	\begin{equation*}
		\begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}
		=
		\begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
		+
		\begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		.
	\end{equation*}
	Thus,
	\begin{align*}
		A^{1001} \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}
		&=
		A^{1001} \left(
			\begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
			+
			\begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		\right)
		\\
		&=
		A^{1001} \begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
		+
		A^{1001} \begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		\\
		&=
		\lambda_{1}^{1001} \begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
		+
		\lambda_{3}^{1001} \begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		\\
		&=
		1^{1001} \begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
		+
		{(-1)}^{1001} \begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		\\
		&=
		\begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}
		-
		\begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
		\\
		&=
		\boxed{
			\begin{pmatrix} -2 \\ 0 \\ 2 \end{pmatrix}
		}
	\end{align*}

	\item \textit{What is $\text{det}(A)$? Why?}
	\begin{align*}
		\text{det}(A)
		&= \text{det}(S) \text{det}(D) \text{det}(S^{-1}) \\
		&= \text{det}(S) \text{det}(D) \frac{1}{\text{det}(S)} \\
		&= \text{det}(D) \\
		&=
		\begin{vmatrix}
			1 & 0 & 0 \\
			0 & 2 & 0 \\
			0 & 0 & -1
		\end{vmatrix} \\
		&= 1 \cdot 2 \cdot -1 \\
		&= \boxed{-2}.
	\end{align*}

	\item \textit{What is an eigenbasis for $A$?}
	\begin{equation*}
		\boxed{
			\left\{
				\begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix},
				\begin{pmatrix} 1 \\ -1 \\ 1 \end{pmatrix},
				\begin{pmatrix} 1 \\ 1 \\ -1 \end{pmatrix}
			\right\}.
		}
	\end{equation*}
\end{enumerate}
\newpage

\section*{4.}
\textit{Let $A$ be the $3 \times 3$ matrix associated with the transformation
that orthogonally projects vectors onto the plane $2x + 3y - z = 0$. \\
Let $B$ be the $3 \times 3$ matrix associated with the transformation that
reflects vectors across the plane $x + y + z = 0$.}
\begin{enumerate}[label=(\alph*)]
	\item \textit{Pick one of the following and find a basis for it:
	\begin{itemize}
		\item $\text{nullspace}(AB)$
		\item $\text{range}(AB)$
	\end{itemize}
	}
	The range of $AB$ is the plane that $A$ projects onto. \\
	A basis for the range of $AB$ can be found using the normal vector of the
	plane,
	\begin{equation*}
		\vec{n} = \begin{pmatrix} 2 \\ 3 \\ -1 \end{pmatrix},
	\end{equation*}
	to find two linearly independent vectors orthogonal to $\vec{n}$ which lie
	on the plane:
	\begin{equation*}
		\boxed{
			\left\{
				\begin{pmatrix} 1 \\ 0 \\ 2 \end{pmatrix},
				\begin{pmatrix} 0 \\ 1 \\ 3 \end{pmatrix}
			\right\}.
		}
	\end{equation*}

	\item \textit{Repeat the above instructions for:
	\begin{itemize}
		\item $\text{nullspace}(BA)$
		\item $\text{range}(BA)$
	\end{itemize}
	}
	The nullspace of $BA$ consists of all the vectors that $A$ will
	orthographically project to $0$. \\
	Thus, a basis for the nullspace of $BA$ is
	\begin{equation*}
		\left\{ \vec{n} \right\}
		=
		\boxed{
			\left\{
				\begin{pmatrix} 2 \\ 3 \\ -1 \end{pmatrix}
			\right\}.
		}
	\end{equation*}

	\item \textit{Find all of the following and briefly explain your
	reasoning.}
	\begin{itemize}
		\item $\text{rank}(AB) = 2$
		\item $\text{nullity}(AB) = 1$
		\item $\text{rank}(BA) = 2$
		\item $\text{nullity}(BA) 1$
	\end{itemize}
	The image of both $AB$ and $BA$ are planes, thus the rank of both is $2$. \\
	By the rank-nullity theorem, the nullities of $AB$ and $BA$ must be
	$3 - 2 = 1$.
\end{enumerate}
\newpage

\section*{5.}
\textit{Let $T$ be the reflection in $\mathbb{R}^2$ across $y=x$.}
\begin{gather*}
	\mathcal{A}
	=
	\left\{
		\begin{pmatrix} \cos(30 \degree) \\ \sin(30 \degree) \end{pmatrix},
		\begin{pmatrix} -\sin(30 \degree) \\ \cos(30 \degree) \end{pmatrix}
	\right\}
	\\
	\mathcal{B}
	=
	\left\{
		\begin{pmatrix} -\sin(45 \degree) \\ \cos(45 \degree) \end{pmatrix},
		\begin{pmatrix} \cos(45 \degree) \\ \sin(45 \degree) \end{pmatrix}
	\right\}
\end{gather*}
\textit{Find the following and show your work.}
\begin{enumerate}[label=(\alph*)]
	\item
	\begin{align*}
		\based{T}{\mathcal{B}}{\mathcal{A}}
		&=
		\based{T}{\mathcal{S}}{\mathcal{A}}
		\based{T}{\mathcal{S}}{\mathcal{S}}
		\based{T}{\mathcal{B}}{\mathcal{S}}
		\\ &=
		\begin{pmatrix}
			\cos(30 \degree) & -\sin(30 \degree) \\
			\sin(30 \degree) & \cos(30 \degree)
		\end{pmatrix}^{-1}
		\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}
		\begin{pmatrix}
			-\sin(45 \degree) & \cos(45 \degree) \\
			\cos(45 \degree) & \sin(45 \degree
		\end{pmatrix}
	\end{align*}

	\item
	\begin{align*}
		\based{T}{\mathcal{A}}{\mathcal{A}}
		&=
		\based{T}{\mathcal{S}}{\mathcal{A}}
		\based{T}{\mathcal{S}}{\mathcal{S}}
		\based{T}{\mathcal{A}}{\mathcal{S}}
		\\ &=
		\begin{pmatrix}
			\cos(30 \degree) & -\sin(30 \degree) \\
			\sin(30 \degree) & \cos(30 \degree)
		\end{pmatrix}^{-1}
		\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}
		\begin{pmatrix}
			\cos(30 \degree) & -\sin(30 \degree) \\
			\sin(30 \degree) & \cos(30 \degree)
		\end{pmatrix}
	\end{align*}

	\item
	\begin{align*}
		\based{T}{\mathcal{A}}{\mathcal{A}}
		&=
		\based{T}{\mathcal{S}}{\mathcal{A}}
		\based{T}{\mathcal{S}}{\mathcal{S}}
		\based{T}{\mathcal{A}}{\mathcal{S}}
		\\ &=
		\begin{pmatrix}
			-\sin(45 \degree) & \cos(45 \degree) \\
			\cos(45 \degree) & \sin(45 \degree
		\end{pmatrix}^{-1}
		\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}
		\begin{pmatrix}
			-\sin(45 \degree) & \cos(45 \degree) \\
			\cos(45 \degree) & \sin(45 \degree
		\end{pmatrix}
	\end{align*}
\end{enumerate}
\newpage

\section*{6.}
\textit{Let $E$ be the $25 \times 25$ matrix with ones on both diagonals and
zeros everywhere else.}
\begin{enumerate}[label=(\alph*)]
	\item \textit{For each vector, determine if it is an eigenvector of $E$.
	If it is, find the corresponding eigenvalue. If it is not, give a brief
	explanation why.}
	{
	\itemize{
		\item \textit{$\vec{e}_1$ = 1 on top, 24 0s.}
		\\[\baselineskip]
		No, $E \vec{e}_1$ will have 1 on the top and bottom with zeros
		everywhere else.

		\item \textit{$\vec{e}_2$ = 12 0s, 1 in the middle, 12 0s.}
		\\[\baselineskip]
		Yes, $E \vec{e}_2 = \vec{e}_2$, thus $\boxed{\lambda_2 = 1}$.

		\item \textit{$\vec{e}_3$ = 1 on top, 23 0s, 1 on bottom.}
		\\[\baselineskip]
		Yes, $E \vec{e}_3$ will have 2 on top, 23 0s, and 2 on the bottom.
		Thus, $\boxed{\lambda_3 = 2}$.

		\item \textit{$\vec{e}_4$ = 0, 1 on top, 21 0s, 1, 0 on bottom.}
		\\[\baselineskip]
		Uh, yes, $\boxed{\lambda_4 = 2}$.

		\item \textit{$\vec{e}_5$ = 1 on top, 23 0s, -1 on bottom.}
		\\[\baselineskip]
		Yes, $E \vec{e}_5 = \vec{0}$, thus $\boxed{\lambda_5 = 0}$.
	}
	}

	\item \textit{Create a complete algebraic and geometric multiplicity table
	for the eigenvalues of $E$. Explain your reasoning.}
	Ouch.

	\item \textit{Is $E$ diagonalizable? Justify.} \\[\baselineskip]
	Yes. This was a guess.

	\item \textit{Is $E$ invertible? Justify.} \\[\baselineskip]
	No, row-reducing the matrix will result in the bottom half being zeros.
\end{enumerate}

\end{document}


\section{Monte Carlo EM Procedure}

Let $X_{ij} = [x_i, x_j, x_{ij}]$ denote the features, $\Delta_{ij} = [\alpha_i, \beta_j, \gamma_i, u_i, v_j, s_i]$ denote the latent factors, and
$\Theta = [b, g_0, d_0,, c_0, G, D, H, \lambda, \eta, a_\alpha, a_\beta, a_\gamma, A_u, A_v, A_s]$ denote the model parameters. We use the convention that $y = \{y_{ij}\}$, $X = \{X_{ij}\}$, $z = \{z_{jn}\}$ and so on. The EM algorithm seeks to find the $\Theta$ that maximizes the incomplete data likelihood (marginalizing over latent factors):
$$
\arg\max_{\Theta} \Pr[y, w \,|\, \Theta, X]
$$
Let $LL(\Theta; \Delta, z, y, w, X) = \log(\Pr[\Delta, z, y, w \,|\, \Theta, X])$ denote the log of the complete data likelihood. Let $\hat{\Theta}^{(t)}$ denote our current estimate of $\Theta$ at the $t$th iteration. The EM algorithm iterate through the following two steps until convergence.

\begin{itemize}
\item {\bf E-step:} Compute the sufficient statistics of $E_{\Delta, z}[LL(\Theta; \Delta, z, y, w, X) \,|\, \hat{\Theta}^{(t)}]$ as a function of $\Theta$, where the expectation is taken over the posterior distribution of $(\Delta, z \,|\, \hat{\Theta}^{(t)}, y, w, X)$.

\item {\bf M-step:} Find the $\Theta$ that maximizes the expectation computed in the E-step.
$$
\hat{\Theta}^{(t+1)}= \arg\max_{\Theta}
 E_{\Delta, z}[LL(\Theta; \Delta, z, y, w, X) \,|\, \hat{\Theta}^{(t)}]
$$
\end{itemize}

\subsection{Monte-Carlo E-Step}

Because $E_{\Delta, z}[LL(\Theta; \Delta, z, y, w, X) \,|\, \hat{\Theta}^{(t)}]$ is not in closed form, we compute the Monte-Carlo mean based on $L$ samples generated by a Gibbs sampler. The Gibbs sampler repeats the following procedure for $L$ times.

\begin{enumerate}
\item Sample from $(\alpha_i \,|\, \textrm{Rest})$, which is Gaussian, for all $i$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & = y_{ij} - (x_{ij}^{\prime} b)\gamma_i - \beta_j -
	u_i^\prime v_j - s_{i}^{\prime} \bar{z}_j \\
\textit{Var}[\alpha_i|\mbox{Rest}] & =
	( \frac{1}{a_{\alpha}} +
		\sum_{j \in \mathcal{J}_i} \frac{1}{\sigma_{ij}^{2}} )^{-1} \\
E[\alpha_i|\mbox{Rest}] & =
	\textit{Var}[\alpha_i|\mbox{Rest}]
	( \frac{g_0^{\prime} x_i}{a_{\alpha}} +
		 \sum_{j \in \mathcal{J}_i} \frac{o_{ij}}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(\beta_j \,|\, \textrm{Rest})$, which is Gaussian, for all $j$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & = y_{ij} - (x_{ij}^{\prime} b) \gamma_i - \alpha_i -
	u_i^\prime v_j - s_{i}^{\prime} \bar{z}_j \\
\textit{Var}[\beta_j|\mbox{Rest}] & =
	( \frac{1}{a_{\beta}} +
		\sum_{i \in \mathcal{I}_j} \frac{1}{\sigma_{ij}^{2}} )^{-1} \\
E[\beta_j|\mbox{Rest}] & =
	\textit{Var}[\beta_j|\mbox{Rest}]
	( \frac{d_0^{\prime} x_j}{a_{\beta}} +
		 \sum_{i \in \mathcal{I}_j} \frac{o_{ij}}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(\gamma_i \,|\, \textrm{Rest})$, which is Gaussian, for all $i$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & =  y_{ij} - \alpha_i - \beta_j - u_i^\prime v_j -
	s_i^\prime \bar{z}_j \\
\textit{Var}[\gamma_i|\mbox{Rest}] & =
	( \frac{1}{a_{\gamma}} +
	  \sum_{j \in \mathcal{J}_i}
			\frac{(x_{ij}^{\prime} b)^2}{\sigma_{ij}^{2}}
	)^{-1} \\
E[\gamma_i|\mbox{Rest}] & =
	\textit{Var}[\gamma_i|\mbox{Rest}]
	( \frac{ c_0^\prime x_i }{a_{\gamma}} +
		 \sum_{j \in \mathcal{J}_i}
			\frac{o_{ij} (x_{ij}^{\prime} b)}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(u_i \,|\, \textrm{Rest})$, which is Gaussian, for all $i$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & =  y_{ij} - (x_{ij}^{\prime} b)\gamma_i -
						\alpha_i - \beta_j - s_i^\prime \bar{z}_j \\
\textit{Var}[u_i|\mbox{Rest}] & =
	( A_u^{-1} +
	  \sum_{j \in \mathcal{J}_i}
			\frac{v_j v_j^{\prime}}{\sigma_{ij}^{2}}
	)^{-1} \\
E[u_i|\mbox{Rest}] & =
	\textit{Var}[u_i|\mbox{Rest}]
	( A_u^{-1} G x_i +
		 \sum_{j \in \mathcal{J}_i} \frac{o_{ij} v_j}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(v_j \,|\, \textrm{Rest})$, which is Gaussian, for all $j$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & =  y_{ij} - (x_{ij}^{\prime} b)\gamma_i -
						\alpha_i - \beta_j - s_i^\prime \bar{z}_j \\
\textit{Var}[v_j|\mbox{Rest}] & =
	( A_v^{-1} +
	  \sum_{i \in \mathcal{I}_j}
			\frac{u_i u_i^{\prime}}{\sigma_{ij}^{2}}
	)^{-1} \\
E[v_j|\mbox{Rest}] & =
	\textit{Var}[v_j|\mbox{Rest}]
	( A_v^{-1} D x_j +
		 \sum_{i \in \mathcal{I}_j} \frac{o_{ij} u_i}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(s_i \,|\, \textrm{Rest})$, which is Gaussian, for all $i$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij} & =  y_{ij} - (x_{ij}^{\prime} b)\gamma_i -
						\alpha_i - \beta_j - u_i^\prime v_j \\
\textit{Var}[s_i|\mbox{Rest}] & =
	( A_s^{-1} +
	  \sum_{j \in \mathcal{J}_i}
			\frac{\bar{z}_j \bar{z}_j^{\prime}}{\sigma_{ij}^{2}}
	)^{-1} \\
E[s_i|\mbox{Rest}] & =
	\textit{Var}[s_i|\mbox{Rest}]
	( A_s^{-1} H x_i +
		 \sum_{j \in \mathcal{J}_i} \frac{o_{ij} \bar{z}_j}{\sigma_{ij}^{2}} )
\end{split}
\end{equation}

\item Sample from $(z_{jn} \,|\, \textrm{Rest})$, which is multinomial, for all $j$ and $n$. Let $z_{\neg jn}$ denote $z$ with $z_{jn}$ removed. The probability of $z_{jn}$ being topic $k$ is given by
\begin{equation*}
\begin{split}
&\Pr[z_{jn} = k \,|\, z_{\neg jn}, \Delta, \hat{\Theta}^{(t)}, y, w, X] \\
&\propto \Pr[z_{jn} = k, y \,|\, z_{\neg jn}, \Delta, \hat{\Theta}^{(t)}, w, X] \\
&\propto \Pr[z_{jn} = k \,|\, w, z_{\neg jn}, \hat{\Theta}^{(t)}] \cdot
  		  \prod_{i\in I_j} \Pr[y_{ij}\,|\,z_{jn} = k, z_{\neg jn},
								   \Delta, \hat{\Theta}^{(t)}, X]
\end{split}
\end{equation*}
Let
$$
Z_{jk\ell} = \sum_n \mathbf{1}\{z_{jn} = k \mbox{ and } w_{jn} = \ell\}
$$
denote the number of times word $\ell$ belongs to topic $k$ in item $j$. Let $Z_{k\ell} = \sum_j Z_{jk\ell}$, $Z_{k} = \sum_{j\ell} Z_{jk\ell}$, and so on. We use $Z_{\cdot}^{\neg jn}$ to denote the count $Z_{\cdot}$ with $z_{jn}$ removed from the summation. Assume $w_{jn} = \ell$. Then,
\begin{equation*}
\begin{split}
& \Pr[z_{jn} = k \,|\, w, z_{\neg jn}, \hat{\Theta}^{(t)}] \\
&	\propto \Pr[z_{jn} = k, w_{jn} = \ell \,|\,
					w_{\neg jn}, z_{\neg jn}, \hat{\Theta}^{(t)}] \\
&	= \Pr[w_{jn} = \ell \,|\,
			w_{\neg jn}, z_{jn} = k, z_{\neg jn}, \eta] ~
	  \Pr[z_{jn} = k \,|\, z_{\neg jn}, \lambda] \\
&	= E[ \Phi_{k\ell} \,|\, w_{\neg jn}, z_{\neg jn}, \eta] ~
	  E[ \theta_{jk} \,|\, z_{\neg jn}, \lambda] \\
& 	= \frac{Z_{k\ell}^{\neg jn} + \eta}
			 {Z_{k}^{\neg jn} + W \eta}~
	  \frac{Z_{jk}^{\neg jn} + \lambda_k}
			 {Z_{j}^{\neg jn} + \sum_k \lambda_k}
\end{split}
\end{equation*}
Note that the denominator of the second term $(Z_{j}^{\neg jn} + \sum_k \lambda_k)$ is independent of $k$. Thus, we obtain
\begin{equation}
\Pr[z_{jn} = k \,|\, \mbox{Rest}]
 \propto \frac{Z_{k\ell}^{\neg jn} + \eta}
			 {Z_{k}^{\neg jn} + W \eta} ~
			 (Z_{jk}^{\neg jn} + \lambda_k) ~
			 \prod_{i\in \mathcal{I}_j} f_{ij}(y_{ij})
\end{equation}
where $f_{ij}(y_{ij})$ is the probability density at $y_{ij}$ given the current values of $(x_{ij}^{\prime}\, b) \gamma_i  + \alpha_i + \beta_j + u_i^\prime v_j + s_{i}^{\prime} \, \bar{z}_{j}$ and $\sigma_{ij}^2$.
\begin{equation}
\begin{split}
\mbox{Let } o_{ij}
	&= y_{ij} - (x_{ij}^{\prime}\, b) \gamma_i  -
		\alpha_i - \beta_j - u_i^\prime v_j \\
\prod_{i\in \mathcal{I}_j} f_{ij}(y_{ij})
	&\propto \exp\left\{
			-\frac{1}{2} \sum_{i \in \mathcal{I}_j}
				\frac{(o_{ij} -  s_{i}^{\prime} \, \bar{z}_{j})^2}{\sigma_{ij}^2}
		\right\} \\
	&\propto \exp\left\{
			\bar{z}_{j}^\prime B_j -
			\frac{1}{2} \bar{z}_{j}^\prime C_j \bar{z}_{j}
		\right\} \\
\mbox{where } &
	B_j = \sum_{i \in \mathcal{I}_j} \frac{o_{ij} s_i}{\sigma_{ij}^{2}}
	\mbox{ and }
	C_j = \sum_{i \in \mathcal{I}_j} \frac{s_i s_i^{\prime}}{\sigma_{ij}^{2}}
\end{split}
\end{equation}
\end{enumerate}

\subsection{M-Step}

In the M-step, we want to find the $\Theta$ = $[b, g_0, c_0, d_0, G, D, H, \lambda, \eta, a_\alpha, a_\beta, a_\gamma, A_u, A_v]$ that maximizes the expected complete data likelihood computed in the E-step.
$$
\hat{\Theta}^{(t+1)}= \arg\max_{\Theta}
 E_{\Delta, z}[LL(\Theta; \Delta, z, y, w, X) \,|\, \hat{\Theta}^{(t)}]
$$
where $- LL(\Theta; \Delta, z, y, w, X) =$
{\small\begin{equation*}
\begin{split}
& ~~   \mbox{Constant} +
			\frac{1}{2} \sum_{ij} \left( \frac{1}{\sigma_{ij}^{2}}
			(y_{ij}-\alpha_i-\beta_j- (x_{ij}^{\prime}b) \gamma_i
				- u_i^\prime v_j - s_{i}^{\prime} \bar{z}_{j})^{2}
			+ \log \sigma_{ij}^{2} \right) \\
& ~~ + \frac{1}{2 a_{\alpha}} \sum_{i}
				(\alpha_i - g_{0}^{\prime}x_i)^{2}
				+ \frac{M}{2} \log a_{\alpha}
		+ \frac{1}{2 a_{\gamma}} \sum_{i}
				(\gamma_i - c_{0}^{\prime}x_i)^{2}
				+ \frac{M}{2} \log a_{\gamma}  \\
& ~~	+ \frac{1}{2} \sum_{i}
				(u_i - Gx_i)^{\prime} A_u^{-1} (u_i - Gx_i)
				+ \frac{M}{2} \log(\det A_u) \\
& ~~	+ \frac{1}{2} \sum_{j}
				(v_j - Dx_j)^{\prime} A_v^{-1} (v_j - Dx_j)
				+ \frac{N}{2} \log(\det A_v) \\
& ~~	+ \frac{1}{2} \sum_{i}
				(s_i - Hx_i)^{\prime} A_s^{-1} (s_i - Hx_i)
				+ \frac{M}{2} \log(\det A_s) \\
& ~~ 	+ \frac{1}{2 a_{\beta}} \sum_{j}
				 (\beta_j - d_{0}^{\prime}x_j)^{2}
				+ \frac{N}{2} \log a_{\beta}  \\
& ~~ 	+ N \left( \sum_k \log\Gamma(\lambda_k)
				- \log\Gamma(\textstyle\sum_k \lambda_k) \right)
				+ \sum_j \left(
						\log\Gamma\left(Z_j + \textstyle\sum_k \lambda_k\right) -
						\sum_k \log\Gamma(Z_{jk} + \lambda_k)
				  \right) \\
& ~~ 	+ K \left( W \log\Gamma(\eta)
				- \log\Gamma(W \eta) \right)
				+ \sum_k \left(
						\log\Gamma(Z_k + W \eta) -
						\sum_\ell \log\Gamma(Z_{k\ell} + \eta)
				  \right) \\
\end{split}
\end{equation*}}

\noindent The optimal $g_0$, $c_0$, $G$, $D$, $d_0$, $a_\alpha$, $a_\gamma$, $A_u$, $A_v$ and $a_\beta$ can be obtained using the same regression procedure as that in the original RLFM.
\\

{\bf Optimal $b$ and $\sigma^2$:} Consider the Gaussian case, where $\sigma_{ij}^2 = \sigma^2$. Define
$$
o_{ij} = y_{ij} - (\alpha_i + \beta_j + u_i^\prime v_j + s_i^\prime \bar{z}_j)
$$
We use $\tilde{E}[\cdot]$ to denote the Monte-Carlo expectation. Here, we want to find $b$ and $\sigma^2$ that minimize
\begin{equation*}
\begin{split}
&	\frac{1}{\sigma^2} \sum_{ij}
		\tilde{E}[(o_{ij} - \gamma_i(x_{ij}^{\prime}b))^2]
		+ P \log(\sigma^2) \\
&=	\frac{1}{\sigma^2} \sum_{ij} \left(
			\tilde{E}[o_{ij}^2]
			- 2\tilde{E}[o_{ij}\gamma_i] (x_{ij}^{\prime}b)
			+ \tilde{E}[\gamma_i^2] (x_{ij}^{\prime}b)^2
		\right)
		+ P \log(\sigma^2) \\
&=	\frac{1}{\sigma^2} \sum_{ij} \left(
			\tilde{E}[o_{ij}^2]
			- \frac{(\tilde{E}[o_{ij}\gamma_i])^2}{\tilde{E}[\gamma_i^2]}
			+ \tilde{E}[\gamma_i^2] \left(
					\frac{\tilde{E}[o_{ij}\gamma_i]}{\tilde{E}[\gamma_i^2]}
					- x_{ij}^{\prime} b
				\right)^2
		\right)
		+ P \log(\sigma^2) \\
\end{split}
\end{equation*}
where $P$ is the total number of observed user-item pairs. The optimal $b$ can be found by weighted least squares regression with feature $x_{ij}$, response $\frac{\tilde{E}[o_{ij}\gamma_i]}{\tilde{E}[\gamma_i^2]}$ and weight $\tilde{E}[\gamma_i^2]$. The optimal $\sigma^2$ is the above summation (over $ij$) divided by $P$ with the optimal $b$ value.
\\

{\bf Optimal $\eta$:} We want to find $\eta$ that minimizes
\begin{equation*}
\begin{split}
& K \left( W \log\Gamma(\eta)
				- \log\Gamma(W \eta) \right)
				+ \sum_k \left(
						\tilde{E}[\log\Gamma(Z_k + W \eta)] -
						\sum_\ell \tilde{E}[\log\Gamma(Z_{k\ell} + \eta)]
				  \right) \\
\end{split}
\end{equation*}
Since this optimization is just one dimensional and $\eta$ is a nuance parameter, we can simply try a number of fixed possible $\eta$ values.
\\

{\bf Optimal $\lambda$:} We want to find $\lambda_1$, ..., $\lambda_K$ that minimize
\begin{equation*}
\begin{split}
& N \left( \sum_k \log\Gamma(\lambda_k)
		- \log\Gamma(\textstyle\sum_k \lambda_k) \right) \\
&		+ \sum_j \left(
				\tilde{E}[\log\Gamma\left(Z_j + \textstyle\sum_k \lambda_k\right)]
				- \sum_k \tilde{E}[\log\Gamma(Z_{jk} + \lambda_k)]
	\right)\\
\end{split}
\end{equation*}
{\bf Question: How to optimize this efficiently?} What's the sufficient statistics? Storing Monte-Carlo samples in memory for $Z_{jk}$ may not be feasible. Any approximation? For now, I just assume $\lambda_k = \lambda_0$, a single scalar and search through a fixed set of points (which is also assumed in the first Gibbs-sampling-based LDA paper by Griffiths, 2004).

{\bf Comment:}: I think assuming the same $\lambda$ should work fine with a lot of document data, in fact we can select $\eta$ and $\lambda_0$ through cross-validation in the first  implementation to simplify things.

{\bf Comment:} The Gibbs sampler here looks trivial, I also feel it would be better behaved than the sampler we had in RLFM. However, one drawback is the conditional
distribution of $u_i$ that involves inverting a matrix whose dimension equals number of topics. In general, we may want to try to a few thousand topics with LDA; we may have to break $u_i$ into blocks of small chunks and sample each block one at a time conditional on the others. However, this does not seem to be a worry in the first implementation where we could try few 100 topics in the LDA.


%!TEX root = ../../report.tex

\subsubsection{L-Systems} % (fold)
\label{ssub:l_systems}

Lindenmayer Systems (L-Systems) are a class of string rewriting mechanisms, originally developed by Lindenmayer as a mathematical theory for plant development.

One L-System is a type of a formal grammar and a string rewriting system that is capable of describe the behaviour of plant cells and model the growth processes of plant development.

It consists of two different parts, one axiom and a set of production rules. The axiom is the starting point of the system, acting as a seed. Then is't applied in this seed the set of production rules, that change the initial string and producing other strings.
This is an iterative process, so after the production of a larger set of strings, the rules can be applied to each one of them wish grows the size of the set even larger.

This L-Systems are used to produce natural growth of vegetation (Figure~\ref{fig:trees}), and the generation of Fractals.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.95\textwidth]{img/Theory/L_Systems/Dragon_trees.jpg}
    \caption{Trees with L-Systems}
    \label{fig:trees}
\end{figure}

In this process, each symbol is associated with a production rule. For instance having $\{F, +, -\}$ for the alphabet and \emph{production} $\{F \rightarrow
 F+F--F+F\}$. From a starting axiom \emph{aba}, and the application of the rules we have:\\
\begin{equation} \label{eq:seed}
F\\
\end{equation}
\begin{equation} \label{eq:step1}
F+F--F+F\\
\end{equation}
\begin{equation} \label{eq:step2}
F+F--F+F \; + \; F+F--F+F \;- \;- \;F+F--F+F \;+ \;F+F--F+F\\
\end{equation}

%\begin{align}
%\begin{split}
%F\\
%F+F--F+F\\
%F+F--F+F \; + \; F+F--F+F \;- \;- \;F+F--F+F \;+ \;F+F--F+F\\
%\end{split}
%\end{align}
%\\
\\
\\
This is an example of the evolution of one system where the production is applied  in (\ref{eq:seed}) that turns into $F+F--F+F$. In Note that the space between the symbols are just for readability.

All the symbols are assigned with a geometric meaning. The notion of a turtle with a pen, as proposed in \cite{abelson1982aa}, with the symbols being interpreted as moving instructions to the turtle, is a simple way to understand. If ``F'' means forward and the symbols ``+'' and ``-'' are interpreted as rotations counter-clockwise and clockwise respectively by a predefined angle.

Using the given example, and setting the angle for the rotation to $60^{\circ}$ the result is Figure~\ref{fig:kockLS}.
%$\bigodot \; \bigodot$

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.75\textwidth]{img/Theory/L_Systems/koch.png}
   \caption{}
   \label{fig:kockLS}
\end{figure}

% This concept of the turtle can be considered also in 3D.

% subsubsection l_systems (end)


\begin{latin}
\textbf{Solution}: In merge sort, the $n$ element array is divided into two halves each with $\frac{n}{2}$ elements (for $n$ = even), or one with $\dfrac{n}{2}$ and the other with $\frac{n}{2}+1$ elements. The two halves are sorted and merged. The algorithm for performing merge sort is as follows:
\begin{figure}[h!]
\centerline{ \includegraphics[width=12cm,height=18cm]{2.jpg}}
%\caption{ }
%\label{1.jpg}
\end{figure}\\

The algorithm Merge$(\min, m, \max)$ is performed in $O(n)$. The algorithm mergeSort $(int
\min, int \max)$ of $n$ element becomes half when it is called recursively. Thus, the whole algorithm
takes
\begin{align*}
T(n) &\leq 2 T (\frac{n}{2})+cn \\
&\leq 2 \bigg[ 2 T (\frac{n}{2^2}) + c \frac{n}{2} \bigg] + cn\\
T(n) &\leq 2^2 T (\frac{n}{2^2})+2cn \\
\cdots \\
\cdots
\end{align*}
\begin{align*}
&T(n)\leq 2^i T (\frac{n}{2^i})+icn \\
& if ~~2^i=n, i=\log_2 n \\
&T(n) \leq  2^i T(1) + cn \log_2 n
\end{align}
$T(1) = 1$ (time required to sort a list of one element).\\
Thus, the complexity of merge sort is $O(n \log_2 n)$.\\
$3.$ Prove that the k clique problem is $NP$ complete.\\
\textbf{Solution:}
\begin{Definition}
Given a graph $G = {V, E}$ and an integer $k$, check whether there exists a sub-graph $C$
of $G$ which contains $k$ number of vertices.
\end{Definition}
(The max clique problem is fi nding a clique with largest number of vertices. It is an $NP$ hard
problem.)\\
To prove that clique is $NP$ complete, we need to prove:\\
\textbf{A. Clique is NP}\\
$\square$ Check whether $C$ contains $k$ number of vertices. This can be done in $O(| C |)$.\\
$\square$ Check whether $C$ is a complete sub-graph. This can be done in $kC_2$ steps.\\
Thus, checking whether $C$ is a clique of $G$ or not can be performed in polynomial time.
A graph with $n$ number of vertices can have $2n$ number of such vertex combination. For each
of the combinations, the same checking algorithm is performed. Among these, the clique with $k$
vertices is the $ k$-clique of $G$. So, a $k$-clique algorithm is carried out by a non-deterministic Turing
machine in polynomial time. Hence, it is in $NP$.\\
\textbf{B. Reduce $3-SAT$ to $k$-clique}\\
\textbf{Reduction:}\\
$\square$ For each clause of a Boolean assignment, create distinct vertices for each literal. If a
Boolean assignment contains $C$ clauses, then the number of vertices is $3C$. \\
$\square$ Join the vertices of the clauses in such a way that\\
 – Two vertices of the same clause cannot be joined.\\
– Two vertices whose literals are the negation of the other cannot be joined.
\begin{exa}
Consider $F=(X_1 \vee X_2 \vee Y_1) \wedge (X_3 \vee \neg Y_1 \vee \neg Y_2) \wedge (X_4 \vee X_5 \vee \neg Y_2) $ \\
The graph is \\
\begin{figure}[h!]
\centerline{ \includegraphics[width=13cm,height=7cm]{1.jpg}}
%\caption{ }
%\label{1.jpg}
\end{figure}\\
If it can be proved that $G$ has a $k$-clique if and only if $F$ is satisfi able, then the reduction is correct.
\end{exa}
$\square$ \textbf{If F is satisfi able, G has k-clique: }If the $3-SAT$ instance $F$ is TRUE, then each clause is TRUE, which means every clause has a TRUE literal. By selecting a corresponding vertex to a TRUE literal in each clause, a clique in $G$ of size $k$ is constructed, where $k $ is the number of clauses of the $3-SAT$ instance. Because, if there is a missing edge, that would mean that our truth assignment effectively set something to be true and false at the same time ($Y_1$ and $\neg Y_1$)$!$
\\
$\square$ \textbf{If G-has k-clique, F is satisfi able:} Assume that there is a clique of size $k$ in $G$ where k is the number of clauses in $F$. It is to be proved that there must be a truth assignment that satisfies the given Boolean formula. Clique is a complete graph. Let us assume that the truth assignment induced by the labels of the vertices satisfies $F.$ It signifies that every pair of vertices in the clique has an edge. But the vertices labelling the literals may be set to both true and false. Already it is mentioned that every trio of vertices corresponding to a clause of $F$ have no edges between those vertices, which signifies that there must be a vertex from every clause of $F$ in the clique. This shows that the clauses of $F$ are satisfied as well as $F$  is satisfied.\\
This reduction is possible in polynomial time. Hence, clique is $NP$ complete.
\section*{Multiple Choice Questions}
$1.$ Worst case time complexity is denoted by
the notation. \\
a) Big oh notation\\
b) Big omega notation\\
c) Theta notation\\
d) Little omega notation\\
$2.$ Best case time complexity is denoted by the
notation.\\
a) Big oh notation\\
b) Big omega notation\\
c) Theta notation\\
d) Little omega notation\\
$3.$ if $f(n)=O(g(n))$ and $f(n)=\Omega(g(n))$, then which is true?\\
a) $f(n)=\omega (g(n))$~~~~~b) $f(n)=o(g(n))$ \\
c) $f(n)=\Theta(g(n))$~~~~~~d) $f(n)=\theta(g(n))$\\
$4.$ The problem which results in ‘yes’ or ‘no’ is \\
a) Decision problem\\
b) Optimization problem\\
c) Search problem\\
d) Functional problem\\
$5.$ Which type of problem is the shortest path
algorithm?\\
a) Decision problem\\
b) Optimization problem\\
c) Search problem\\
d) Functional problem\\
$6.$ Which is true for the $P$ class problem?\\
a) The number of steps (or time) required
to complete the algorithm is a polynomial
function of $n$.\\
b) It is computable by a deterministic
Turing machine in polynomial time.\\
c) It contains all sets in which the membership
may be decided by an algorithm
whose running time is bounded by a
polynomial.\\
d) All of these\\
\textbf{Answers:} $1.$a   \quad   $2.$b   \quad   3.c  \quad  4.a   \quad  5.b      \quad 6.d
\section*{GATE Questions}
1. For problems $X$ and $Y$, $ Y$ is $NP$ complete and $X$  reduces to $Y$ in polynomial time. Which of the following is true? \\
a) If $X$ can be solved in polynomial time, then so can $ Y$.\\
b) $X$ is $NP$ complete\\
c) $X $ is $NP$ hard\\
d) $ X $ is in $NP$, but not necessarily $NP$ complete.\\
2. Which of the problems is not $NP$ hard?\\
a) Hamiltonian circuit problem \quad \quad \quad \quad \quad \quad \quad \quad b) The $0/1$ knapsack problem \\
c) Finding bi-connected components of a graph \quad  d) The graph colouring problem \\
3. Ram and Shyam have been asked to show a certain problem $\Pi$  is $NP$ complete. Ram shows
a polynomial time reduction from $3-SAT$ problem to $\Pi$, and Shyam shows a polynomial time
reduction from $\Pi$ to $3$ SAT. Which of the following can be inferred from these reductions?\\
a) $\Pi$ is $NP$ hard but not $NP$ complete     \quad \quad b)$\Pi$ is $NP$, but is not $NP$ complete\\
c) $\Pi$ is $NP$ complete    \quad \quad \quad \quad \quad \quad \quad \quad \quad  d) $\Pi$ is neither $NP$ hard nor $NP$\\
4. No body knows yet if $P = NP$. Consider the language $L $defined as follows.
\begin{center}
$L=\begin{cases}
(0+1)^* \quad  \quad   if ~~P=NP \\
\varphi \quad \quad \quad \quad \quad Otherwise
\end{cases}$
\end{center}
Which of the following statements is true?\\
a) $L$ is recursive\\
b) $L$ is recursively enumerable but not recursive \\
c) $L$ is not recursively enumerable \\
d) Whether $L$ is recursive or not will be known after we find out if $P = NP$

\end{latin}


%\documentclass[aps,showpacs,prb,floatfix,twocolumn]{revtex4}
\documentclass[aps,prb,floatfix,epsfig,twocolumn,showpacs,preprintnumbers]{revtex4}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{amsmath,amssymb,graphicx,bm,epsfig}
\usepackage{color}

\newcommand{\eps}{\epsilon}
\newcommand{\vR}{{\mathbf{R}}}
\renewcommand{\vr}{{\mathbf{r}}}
\newcommand{\hr}{{\hat{\textbf{r}}}}
\newcommand{\vk}{{\mathbf{k}}}
\newcommand{\vdelta}{{\mathbf{\delta}}}
\newcommand{\vK}{{\mathbf{K}}}
\newcommand{\vq}{{\mathbf{q}}}
\newcommand{\vQ}{{\mathbf{Q}}}
\newcommand{\vPhi}{{\mathbf{\Phi}}}
\newcommand{\vS}{{\mathbf{S}}}
\newcommand{\cG}{{\cal G}}
\newcommand{\cF}{{\cal F}}
\newcommand{\cT}{{\cal T}}
\newcommand{\cH}{{\cal H}}
\newcommand{\cJ}{{\cal J}}
\newcommand{\cD}{{\cal D}}
\newcommand{\cU}{{\cal U}}
\newcommand{\cL}{{\cal L}}
\newcommand{\Tr}{\mathrm{Tr}}
\renewcommand{\a}{\alpha}
\renewcommand{\b}{\beta}
\newcommand{\g}{\gamma}
\renewcommand{\d}{\delta}
\newcommand{\npsi}{\underline{\psi}}
\renewcommand{\Im}{\textrm{Im}}
\renewcommand{\Re}{\textrm{Re}}
\newcommand{\cA}{{\cal A}}

\begin{document}

\title{Some notes on LAPW}
\author{Kristjan Haule}
\affiliation{Department of Physics, Rutgers University, Piscataway, NJ 08854, USA}
\date{\today}

%\begin{abstract}
%\end{abstract}
\pacs{71.27.+a,71.30.+h}
\date{\today}
\maketitle

\begin{widetext}
The LAPW basis takes the form:
\begin{eqnarray}
&& \chi_{\vk+\vK}(\vr) = \frac{1}{\sqrt{V}} e^{i(\vk+\vK)\vr}
=\frac{4\pi i^l}{\sqrt{V}}e^{i(\vk+\vK)\vr_\alpha}Y_{lm}^*(R(\hat{\vk}+\hat{\vK}))j_l(|\vk+\vK||\vr-\vr_\alpha|)Y_{lm}(R(\vr-\vr_\alpha))
  \qquad interstitial\\
&& \chi_{\vk+\vK}(\vr) = \left( a_{lm} u_l(|\vr-\vr_\alpha|) + b_{lm}
  \dot{u}_l(|\vr-\vr_\alpha|)\right)
  Y_{lm}(R(\hat{\vr}-\hat{\vr_\alpha}))\qquad MT-sphere
\end{eqnarray}
The matching condition at the MT-sphere $S$ gives
\begin{eqnarray}
\left(
\begin{array}{cc}
u_l(S) & \dot{u}_l(S)\\
\frac{d}{dr} u_l(S) & \frac{d}{dr} \dot{u}_l(S)
\end{array}
\right)
\left(
\begin{array}{c}
a_{lm}\\
b_{lm}
\end{array}
\right)=
\frac{4\pi i^l}{\sqrt{V}}e^{i(\vk+\vK)\vr_\alpha}Y_{lm}^*(R(\hat{\vk}+\hat{\vK}))
\left(
\begin{array}{c}
j_l(|\vk+\vK|S)\\
\frac{d}{dr} j_l(|\vk+\vK|S)
\end{array}
\right)
\end{eqnarray}
with the solution
\begin{eqnarray}
\left(
\begin{array}{c}
a_{lm}\\
b_{lm}
\end{array}
\right)=
\frac{4\pi i^l}{\sqrt{V}}e^{i(\vk+\vK)\vr_\alpha}Y_{lm}^*(R(\hat{\vk}+\hat{\vK}))
%
\left(
\begin{array}{cc}
\frac{d}{dr} \dot{u}_l(S)  & -\dot{u}_l(S)\\
-\frac{d}{dr} u_l(S) & u_l(S)
\end{array}
\right)
\frac{1}{u_l(S) \frac{d}{dr} \dot{u}_l(S)-\dot{u}_l(S) \frac{d}{dr} u_l(S)}
%
\left(
\begin{array}{c}
j_l(|\vk+\vK|S)\\
\frac{d}{dr} j_l(|\vk+\vK|S)
\end{array}
\right)
\end{eqnarray}

The two solutions satisfy the following equations
\begin{eqnarray}
&&\left(-\frac{d^2}{dr^2}+\frac{l(l+1)}{r^2}+V_{KS}(r)-\varepsilon \right) r u_l(r)=0\\
&&\left(-\frac{d^2}{dr^2}+\frac{l(l+1)}{r^2}+V_{KS}(r)-\varepsilon \right)r \dot{u}_l(r)= r u_l(r)
\end{eqnarray}
We multiply the first equation by $r\dot{u}_l(r)$ and the second by $ru_l(r)$ to obtain
\begin{eqnarray}
\int_0^S dr\left\{ r\dot{u}_l(r) \left(-\frac{d^2}{dr^2}\right) r u_l(r)
-r u_l(r)
\left(-\frac{d^2}{dr^2}\right)r \dot{u}_l(r)\right\}=-\int_0^S dr r^2 u_l(r)u_l(r)
\end{eqnarray}
Integration by parts gives
\begin{eqnarray}
\left[
- r\dot{u}_l(r) \frac{d}{dr} \left(r u_l(r)\right)
+r u_l(r) \frac{d}{dr} \left(r \dot{u}_l(r)\right)\right]_0^S
=-1
\end{eqnarray}
which finally leads to
\begin{eqnarray}
\dot{u}_l(S) \frac{d}{dr} u_l(S)
-u_l(S) \frac{d}{dr} \dot{u}_l(S) =\frac{1}{S^2}
\end{eqnarray}

We can than simplify the solution for $a_{lm}$ and $b_{lm}$ to
\begin{eqnarray}
\left(
\begin{array}{c}
a_{lm}\\
b_{lm}
\end{array}
\right)=
\frac{4\pi i^l}{S^2 \sqrt{V}}e^{i(\vk+\vK)\vr_\alpha}Y_{lm}^*(R(\hat{\vk}+\hat{\vK}))
%
\left(
\begin{array}{c}
\dot{u}_l(S)\frac{d}{dr} j_l(|\vk+\vK|S)-\frac{d}{dr} \dot{u}_l(S) j_l(|\vk+\vK|S)\\
\frac{d}{dr} u_l(S) j_l(|\vk+\vK|S)- u_l(S) \frac{d}{dr} j_l(|\vk+\vK|S)
\end{array}
\right)
\end{eqnarray}
This equation is implemented in Wien2k, and also in both dmft1 and
dmft2 steps.

To compute the projector, we need the overlap between a localized function
$\phi(r) Y_L(\vr)$, and Kohn-Sham states
\begin{eqnarray}
\cU^{\vk,\vr_\alpha}_{i,m}=\langle \phi_l Y_{lm}|\psi_{\vk i}\rangle  =
\sum_\vK C^\vk_{i\vK} \langle \phi_l(|\vr-\vr_\alpha|) Y_{lm}(R(\hat{\vr}-\hat{\vr}_\alpha))|\chi_{\vk+\vK}(\vr)\rangle
\end{eqnarray}
If function $\phi(r)$ extends sufficiently outside its MT-sphere, the
overlap $\cU^{\vk,\vr_\alpha}_{i,m}$ will have non-zero contribution
from all other MT-spheres. However, we will use only the envelope
function outside its center sphere, because the increased charge in
the neighboring spheres really should not be counted here as charge
contribution to $\phi(r)$ function.

Therefore we have  only two contributions. Inside MT-sphere we have
\begin{eqnarray}
\langle \phi_l(|\vr-\vr_\alpha|) Y_{lm}(\hat{\vr}-\hat{\vr}_\alpha)|\chi_{\vk+\vK}(\vr)\rangle  =
\sum_\kappa a^\kappa_{lm} \int_0^S \phi(r)  u_l^\kappa(r) r^2 dr
\end{eqnarray}
and outside MT-sphere we get
\begin{eqnarray}
\langle \phi_l(|\vr-\vr_\alpha|) Y_{lm}(R(\hat{\vr}-\hat{\vr}_\alpha))|\chi_{\vk+\vK}(\vr)\rangle  =
\frac{4\pi  i^l}{\sqrt{V}}  e^{i(\vk+\vK)\vr_\alpha}Y_{lm}^*(R(\hat{\vk}+\hat{\vK}))
\int_S^{S_2} \phi_l(r) j_l(|\vk+\vK|r)r^2 dr
\end{eqnarray}

\section{Free energy and Total Energy}

The equation for the total energy is
\begin{eqnarray}
E = \Tr(H_0 G) + \frac{1}{2}\Tr(\Sigma G) - \Phi^{DC}[n_{loc}] + \Phi^H[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
where
$$H_0 = -\nabla^2 + \delta(\vr-\vr')V_{ext}(\vr)$$
We typically evaluate it in the following way
\begin{eqnarray}
E = \Tr((-\nabla^2+V_{ext}+V_{H}+V_{xc}) G) + \frac{1}{2}\Tr(\Sigma G)-\Phi^{DC}[\rho_{loc}]
-\Tr((V_H+V_{xc})\rho) + \Phi^H[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
Namely, we use the Green's function of the solid to evaluate:
\begin{eqnarray}
E_1 = \Tr((-\nabla^2+V_{ext}+V_{H}+V_{xc}) G) -\Tr((V_H+V_{xc})\rho) + \Phi^H[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
and the impurity to evaluate
\begin{eqnarray}
E_2 =  \frac{1}{2}\Tr(\Sigma_{imp} G_{imp})-\Phi^{DC}[\rho_{imp}]
\end{eqnarray}
Notice that $\frac{1}{2}\Tr(\Sigma_{imp} G_{imp})$ is not evaluated as
a Matsubara sum, but we rather compute it from probabilities of atomic
states, i.e.,
\begin{eqnarray}
 \frac{1}{2}\Tr(\Sigma_{imp} G_{imp}) = \sum_m P_m E_m - \sum_\alpha
 \varepsilon_{imp}^\alpha n_{imp}^\alpha
\end{eqnarray}

The free energy functional is
\begin{eqnarray}
\Gamma[G] = \Tr\log G-\Tr\log((G_0^{-1}-G^{-1})G) + \Phi^{H}[\rho]+\Phi^{xc}[\rho]+\Phi^{DMFT}[G_{loc}]-\Phi^{DC}[\rho_{loc}]
\end{eqnarray}
hence stationarity gives
\begin{eqnarray}
G^{-1}-G_0^{-1} + V_H + V_{xc}+\Sigma_{DMFT}-V_{dc}=0
\end{eqnarray}
and hence
\begin{eqnarray}
F = \Tr\log G
-\Tr(\Sigma G) +
\Tr(V_{dc}\rho_{loc}) + \Phi^{DMFT}[G_{loc}]-\Phi^{DC}[\rho_{loc}]
-\Tr((V_H + V_{xc})\rho)+
\Phi^{H}[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
Since $F_{imp}$ contains $\Phi^{DMFT}$, i.e,
\begin{eqnarray}
F_{imp} = \Tr\log G_{imp} - \Tr(\Sigma_{imp} G_{imp}) + \Phi^{DMFT}[G_{imp}]
\end{eqnarray}
we can write
\begin{eqnarray}
F = \Tr\log(G)-\Tr\log(G_{loc})+F_{imp}+\Tr(V_{dc} \rho_{loc}) -
\Phi^{DC}[\rho_{loc}]
-\Tr((V_H + V_{xc})\rho)+
\Phi^{H}[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
where
$$F_{imp} = E_{imp}-T S_{imp}$$
and
$$E_{imp}=
\Tr((\Delta+\varepsilon_{imp}-\omega_n\frac{\partial\Delta}{\partial\omega_n})
G_{imp}) + \frac{1}{2}\Tr(\Sigma_{imp} G_{imp}) -T S_{imp}
$$
Hence
\begin{eqnarray}
F+T S_{imp} = \Tr\log(G)-\Tr\log(G_{loc})+
%
\Tr((\Delta+\varepsilon_{imp}-\omega_n\frac{\partial\Delta}{\partial\omega_n})
G_{imp}) + \frac{1}{2}\Tr(\Sigma_{imp} G_{imp})
%
+\Tr(V_{dc} \rho_{loc}) -
\Phi^{DC}[\rho_{loc}]\nonumber\\
-\Tr((V_H + V_{xc})\rho)+
\Phi^{H}[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
which can also be cast into the form
\begin{eqnarray}
F+T S_{imp} = \Tr\log(G)-\Tr\log(G_{loc})+
\Tr((\Delta-\omega_n\frac{\partial\Delta}{\partial\omega_n}+\varepsilon_{imp}+V_{dc})G_{loc})
+ \frac{1}{2}\Tr(\Sigma_{imp} G_{imp})
-\Phi^{DC}[\rho_{imp}]\nonumber\\
-\Tr((V_H + V_{xc})\rho)+
\Phi^{H}[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
We thus compute the following quantities with the Green's function of
the solid:
\begin{eqnarray}
F_1 = \Tr\log(G)-\Tr\log(G_{loc})+
\Tr((\Delta-\omega_n\frac{\partial\Delta}{\partial\omega_n}+\varepsilon_{imp}+V_{dc})G_{loc})
-\Tr((V_H + V_{xc})\rho)+
\Phi^{H}[\rho]+\Phi^{xc}[\rho]
\end{eqnarray}
and the following with the impurity:
\begin{eqnarray}
F_2 =   \frac{1}{2}\Tr(\Sigma_{imp} G_{imp})-\Phi^{DC}[\rho_{imp}] -T S_{imp}
\end{eqnarray}
Notice that $F_2$ is similar to $E_2$ (except for the entropy term), hence $E_{solid}$ and
$F_{solid}$ contain exactly the same Monte Carlo noise.

\end{widetext}

\end{document}


% ----------------------------------------------------------------------
\section{Elasticity with Infinitesimal Strain and No Faults}

We begin with the elasticity equation including the inertial term,
\begin{gather}
  \label{eqn:elasticity:strong:form}
  \rho \frac{\partial^2\vec{u}}{\partial t^2} - \vec{f}(\vec{x},t) - \tensor{\nabla} \cdot
\tensor{\sigma}
(\vec{u}) = \vec{0} \text{ in }\Omega, \\
%
  \label{eqn:bc:Neumann}
  \tensor{\sigma} \cdot \vec{n} = \vec{\tau}(\vec{x},t) \text{ on }\Gamma_\tau, \\
%
  \label{eqn:bc:Dirichlet}
  \vec{u} = \vec{u}_0(\vec{x},t) \text{ on }\Gamma_u,
\end{gather}
where $\vec{u}$ is the displacement vector, $\rho$ is the mass
density, $\vec{f}$ is the body force vector, $\tensor{\sigma}$ is the
Cauchy stress tensor, $\vec{x}$ is the spatial coordinate, and $t$ is
time. We specify tractions $\vec{\tau}$ on boundary $\Gamma_\tau$, and
displacements $\vec{u}_0$ on boundary $\Gamma_u$. Because both $\vec{\tau}$
and $\vec{u}$ are vector quantities, there can be some spatial overlap
of boundaries $\Gamma_\tau$ and $\Gamma_u$; however, a degree of freedom at
any location cannot be associated with both prescribed displacements
(Dirichlet) and traction (Neumann) boundary conditions simultaneously.

\begin{table}[htbp]
  \caption{Mathematical notation for elasticity equation with
    infinitesimal strain.}
  \label{tab:notation:elasticity}
  \begin{tabular}{lcp{3in}}
    \toprule
    {\bf Category} & {\bf Symbol} & {\bf Description} \\
    \midrule
    Unknowns & $\vec{u}$ & Displacement field \\
    & $\vec{v}$ & Velocity field \\
    Derived quantities & $\tensor{\sigma}$ & Cauchy stress tensor \\
                   & $\tensor{\epsilon}$ & Cauchy strain tensor \\
    Common constitutive parameters & $\rho$ & Density \\
  & $\mu$ & Shear modulus \\
  & $K$ & Bulk modulus \\
Source terms & $\vec{f}$ & Body force per unit volume, for example $\rho \vec{g}$ \\
    \bottomrule
  \end{tabular}
\end{table}

\subsection{Quastistatic}

If we neglect the inertial term
($\rho \frac{\partial \vec{v}}{\partial t} \approx \vec{0}$), then
time dependence only arises from history-dependent constitutive
equations and boundary conditions. Our solution vector is the
displacement vector and the elasticity equation reduces to
\begin{gather}
  \label{eqn:elasticity:strong:form:quasistatic}
  \vec{f}(\vec{x},t) + \tensor{\nabla} \cdot \tensor{\sigma}(\vec{u}) = \vec{0} \text{ in }\Omega, \\
%
  \tensor{\sigma} \cdot \vec{n} = \vec{\tau}(\vec{x},t) \text{ on }\Gamma_\tau, \\
%
  \vec{u} = \vec{u}_0(\vec{x},t) \text{ on }\Gamma_u.
\end{gather}
Because we will use implicit time stepping, we place all of the terms
in the elasticity equation on the LHS. We create the weak form by
taking the dot product with the trial function $\trialvec[u]$ and
integrating over the domain:
\begin{equation}
    \int_\Omega \trialvec[u] \cdot \left( \vec{f}(t) + \tensor{\nabla}
      \cdot \tensor{\sigma} (\vec{u}) \right) \, d\Omega = 0.
\end{equation}
Using the divergence theorem and incorporating the Neumann boundary conditions, we have
\begin{equation}
%
  \int_\Omega \trialvec[u] \cdot \vec{f}(\vec{x},t) + \nabla \trialvec[v] : -\tensor{\sigma}(\vec{u}) \, d\Omega
  + \int_{\Gamma_\tau} \trialvec[v] \cdot \vec{\tau}(\vec{x},t) \, d\Gamma = 0
\end{equation}

\subsubsection{Residual Pointwise Functions}

Identifying $F(t,s,\dot{s})$ and $G(t,s)$, we have
\begin{align}
  % Fu
  F^u(t,s,\dot{s}) &=  \int_\Omega \trialvec[u] \cdot \eqnannotate{\vec{f}(\vec{x},t)}{\vec{f}^u_0} + \nabla \trialvec[u] : \eqnannotate{-\tensor{\sigma}(\vec{u})}{\tensor{f^u_1}} \, d\Omega
  + \int_{\Gamma_\tau} \trialvec[u] \cdot \eqnannotate{\vec{\tau}(\vec{x},t)}{\vec{f}^u_0} \, d\Gamma, \\
  % Gu
  G^u(t,s) &= 0
\end{align}
Note that we have multiple $\vec{f}_0$ functions, each associated with
a trial function and an integral over a different domain or
boundary. Each material and boundary condition (except Dirichlet)
contribute pointwise functions. The integral over the domain $\Omega$
is subdivided into integrals over the materials and the integral over
the boundary $\Gamma_\tau$ is subdivided into integrals over the
Neumann boundaries. Each bulk constitutive model provides a different
pointwise function for the stress tensor
$\tensor{\sigma}(\vec{u})$. With $G=0$ it is clear that we have a
formulation that will use implicit time stepping algorithms.

\subsubsection{Jacobian Pointwise Functions}

We only have a Jacobian for the LHS:
\begin{align}
  J_F^{uu} &= \frac{\partial F^u}{\partial u} = \int_\Omega \nabla \trialvec[u] :
\frac{\partial}{\partial u}(-\tensor{\sigma}) \, d\Omega
  = \int_\Omega \nabla \trialvec[u] : -\tensor{C} : \frac{1}{2}(\nabla + \nabla^T)\basisvec[u]
\, d\Omega
  = \int_\Omega \trialscalar[u]_{i,k} \, \eqnannotate{\left( -C_{ikjl} \right)}{J_{f3}^{uu}} \, \basisscalar[u]_{j,l}\, d\Omega.
\end{align}

\subsection{Dynamic}

For compatibility with PETSc TS algorithms, we want to turn
the second order equation~\vref{eqn:elasticity:strong:form} into two first order
equations. We introduce the velocity as a unknown,
$\vec{v}=\frac{\partial u}{\partial t}$, which leads to
\begin{align}
  % Displacement-velocity
  \frac{\partial \vec{u}}{\partial t} &= \vec{v} \text{ in } \Omega, \\
  % Elasticity
  \rho(\vec{x}) \frac{\partial\vec{v}}{\partial t} &= \vec{f}(\vec{x},t) + \tensor{\nabla} \cdot \tensor{\sigma}(\vec{u}) \text{ in } \Omega, \\
  % Neumann
  \tensor{\sigma} \cdot \vec{n} &= \vec{\tau}(\vec{x},t) \text{ on } \Gamma_\tau, \\
  % Dirichlet
  \vec{u} &= \vec{u}_0(\vec{x},t) \text{ on } \Gamma_u.
\end{align}
We create the weak form by taking the dot product with the trial
function $\trialvec[u]$ or $\trialvec[v]$ and
integrating over the domain:
\begin{gather}
  % Displacement-velocity
  \int_\Omega \trialvec[u] \cdot \frac{\partial \vec{u}}{\partial t} \, d\Omega =
  \int_\Omega \trialvec[u] \cdot \vec{v} \, d\Omega, \\
  % Elasticity
    \int_\Omega \trialvec[v] \cdot \rho(\vec{x}) \frac{\partial \vec{v}}{\partial t} \, d\Omega
 = \int_\Omega \trialvec[v] \cdot \left( \vec{f}(t) + \tensor{\nabla} \cdot \tensor{\sigma} (\vec{u}) \right) \, d\Omega.
\end{gather}
Using the divergence theorem and incorporating the Neumann boundaries, we can rewrite the second equation as
\begin{equation}
%
  \int_\Omega \trialvec[v] \cdot \rho(\vec{x}) \frac{\partial \vec{v}}{\partial t} \, d\Omega
  = \int_\Omega \trialvec[v] \cdot \vec{f}(\vec{x},t) + \nabla \trialvec[v] : -\tensor{\sigma}(\vec{u}) \, d\Omega
  + \int_{\Gamma_\tau} \trialvec[v] \cdot \vec{\tau}(\vec{x},t) \, d\Gamma.
\end{equation}

For explicit time stepping, we want $F(t,s,\dot{s})=\dot{s}$, so we
solve an augmented system in which we multiply the RHS residual
function by the inversion of the lumped LHS Jacobian,
\begin{gather}
  F^*(t,s,\dot{s}) = G^*(t,s) \text{, where} \\
  F^*(t,s,\dot{s}) = \dot{s} \text{ and} \\
  G^*(t,s) = J_F^{-1} G(t,s).
\end{gather}
With the augmented system, we have
\begin{gather}
  % Displacement-velocity
  \frac{\partial \vec{u}}{\partial t}  = M_u^{-1} \int_\Omega \trialvec[u] \cdot \vec{v} \, d\Omega, \\
  % Elasticity
  \frac{\partial \vec{v}}{\partial t} = M_v^{-1} \int_\Omega \trialvec[v] \cdot \left( \vec{f}(t) + \tensor{\nabla} \cdot \tensor{\sigma} (\vec{u}) \right) \, d\Omega, \\
  % Mu
  M_u = \mathit{Lump}\left( \int_\Omega \trialscalar[u]_i \delta_{ij} \basisscalar[u]_j \, d\Omega \right), \\
  % Mv
  M_v = \mathit{Lump}\left( \int_\Omega \trialscalar[v]_i \rho(\vec{x}) \delta_{ij} \basisscalar[v]_j \, d\Omega \right).
\end{gather}

\subsubsection{Residual Pointwise Functions}

With explicit time stepping the PETSc TS assumes the LHS is
$\dot{s}$, so we only need the RHS residual functions:
\begin{align}
  % Gu
  G^u(t,s) &= \int_\Omega \trialvec[u] \cdot \eqnannotate{\vec{v}}{\vec{g}^u_0} \, d\Omega, \\
  % Gv
  G^v(t,s) &=  \int_\Omega \trialvec[v] \cdot \eqnannotate{\vec{f}(\vec{x},t)}{\vec{g}^v_0} + \nabla \trialvec[v] : \eqnannotate{-\tensor{\sigma}(\vec{u})}{\tensor{g^v_1}} \, d\Omega
  + \int_{\Gamma_\tau} \trialvec[v] \cdot \eqnannotate{\vec{\tau}(\vec{x},t)}{\vec{g}^v_0} \, d\Gamma,
\end{align}
In the second equation these are the same pointwise functions as in the
quasistatic case, only now they are on the RHS instead of the LHS.

\subsubsection{Jacobian Pointwise Functions}

These are the pointwise functions associated with $M_u$ and $M_v$ for
computing the lumped LHS Jacobian. We premultiply the RHS residual
function by the inverse of the lumped LHS Jacobian while
$s_\mathit{tshift}$ remains on the LHS with $\dot{s}$. As a result, we
use LHS Jacobian pointwise functions, but set $s_\mathit{tshift}=1$.
The LHS Jacobians are:
\begin{align}
  % J_F uu
  M_u = J_F^{uu} &= \frac{\partial F^u}{\partial u} + s_\mathit{tshift} \frac{\partial F^u}{\partial \dot{u}} =
             \int_\Omega \trialscalar[u]_i \eqnannotate{s_\mathit{tshift} \delta_{ij}}{J^{uu}_{f0}} \basisscalar[u]_j  \, d\Omega, \\
  % J_F vv
  M_v = J_F^{vv} &= \frac{\partial F^v}{\partial v} + s_\mathit{tshift} \frac{\partial F^v}{\partial \dot{v}} =
             \int_\Omega \trialscalar[v]_i \eqnannotate{\rho(\vec{x}) s_\mathit{tshift} \delta_{ij}}{J ^{vv}_{f0}} \basisscalar[v]_j \, d\Omega
\end{align}

% End of file


\section{The Monodromy Group; The Space of Germs}
\subsection{The Monodromy Group}
Let $\pi:\tilde{X}\to X$ be a regular covering map.
Pick a pasepoint $x_0\in X$.
For any choice of loop $\gamma:[0,1]\to X$ based at $x_0$ (that is $\gamma(0)=\gamma(1)=x_0$), we want to defines a permutation $\sigma_\gamma:\pi^{-1}(\{x_0\})\to\pi^{-1}(\{x_0\})$.
If you took Algebraic Topology, you should already know about this construction, but we'll do it again.
\begin{definition}
    Let $\tilde{x}\in\pi^{-1}(\{x_0\})$ and let $\tilde{\gamma}_{\tilde{x}}$ be the unique lift of $\gamma$ starting at $\tilde{x}$.
    Then $\pi(\tilde{\gamma}_{\tilde{x}}(1))=\gamma(1)=x_0$, so $\tilde{\gamma}_{\tilde{x}}(1)\in\pi^{-1}(\{x_0\})$.
    Therefore we define $\sigma_\gamma(\tilde{x})=\tilde{\gamma}_{\tilde{x}}(1)$.
\end{definition}
\begin{remark}
    1. The constant loop corresponds to the identity permutation.\\
    2. Let $\bar\gamma(t)=\gamma(1-t)$, then obviously $\sigma_{\bar\gamma}=\pi_{\gamma}^{-1}$, which precisely means that $\sigma_\gamma$ is a permutation.\\
    3. The previous two remarks hints that the set of all $\sigma_\gamma$ makes a subgroup of $\operatorname{Sym}(\pi^{-1}(\{x_0\}))$.
    We want to realise this group operation in an intuitive way.
    For $\alpha,\beta$ loops based at $x_0$, define their concatenation to be
    $$\alpha\cdot\beta=\begin{cases}
        \alpha(2t)\text{, for $t\in[0,1/2]$}\\
        \beta(2t-1)\text{, for $t\in[1/2,1]$}
    \end{cases}$$
    which is easily seen to be a well-defined loop based at $x_0$.
    The uniqueness of lifts then implies that
    $$(\widetilde{\alpha\cdot\beta})_{\tilde{x}_1}=\tilde{\alpha}_{\tilde{x}_1}\cdot\tilde{\beta}_{\tilde{\alpha}_{\tilde{x}_1}(1)}$$
    Therefore $\sigma_{\alpha\cdot\beta}=\sigma_\beta\sigma_\alpha$.
\end{remark}
\begin{definition}
    The group
    $$\{\sigma_\gamma|\gamma\text{ loop based at $x_0$}\}\le \operatorname{Sym}(\pi^{-1}(\{x_0\}))$$
    which is called the monodromy group of $\pi$.
\end{definition}
\begin{remark}
    1. By Theorem \ref{monodromy}, $\alpha\simeq\beta$ implies $\sigma_\alpha=\sigma_\beta$.\\
    2. One can easily show that the monodromy group is independent of the choice of basepoint (with the path-connectedness assumption, of course).
\end{remark}
\begin{example}
    Recall that $p_k:\mathbb C_\star\to\mathbb C_\star$ sending $z$ to $z^k$ is a regular covering map.
    Take basepoint $1$, then $\pi^{-1}(\{1\})$ consists of the $k^{th}$ roots of unity $\zeta_k^n=e^{2\pi in/k}$.
    Let $\gamma(t)=e^{2\pi it}$, then for each $n$, $\tilde{\gamma}_{\zeta_k^n}=\zeta_k^{n+1}$.
    Therefore $\sigma_\gamma(\zeta_k^n)=\zeta_k^{n+1}$.
    But it turns out every loop in $\mathbb C_\star$ is homotopic to $\gamma^n$ for some $n\in\mathbb Z$ (Algebraic Topology again!), therefore the monodromy group of $p_k$ is indeed the cyclic group of order $k$.
\end{example}


\documentclass[./\jobname.tex]{subfiles}
\begin{document}
\chapter{Theoretical Notes}
This chapter deals with a few theoretical notes. At first, the universal approximation theorem for the \gls{gsk} is shown. Further, the multimodality of the fitness function is proven. Thereby, a symmetry is observed that could lead to the design of better variation operators.

\section{Universial Approximation Theorem}
\label{chap:gsin_approximation_theorem}

The Gauss kernel is able to approximate all functions that are part of the Lebesgue space $f(\mathbf{x}) \in L^1(\mathbb{R}^n)$ arbitrarily close. This has been proven by various works (\cite{park_universal_1991}, \cite{hangelbroek_nonlinear_2010}). In particular, \cite{park_universal_1991} extends the universal approximation theorem to other kernels. The following paragraphs show that the \gls{gsk} fulfils the posed conditions and thus can benefit from the approximation theorem.

The Gauss-Sine kernel, as defined in equation \eqref{eq:gsin_kernel_theoretical_notes}, is a multiplication of the Gauss kernel with a sine function. Its main purpose is described in the experiment \mbox{chapter \ref{chap:experimet_3}}.

\begin{equation}
\label{eq:gsin_kernel_theoretical_notes}
gsk(\mathbf{x}) = \omega e^{-\gamma ||\mathbf{x} - \mathbf{c}||^2} sin(f ||\mathbf{x} - \mathbf{c}||^2 - \varphi)
\end{equation}

To prove that the universal approximation theorem is also applicable to the \gls{gsk}, it must comply with the conditions placed by \cite{park_universal_1991}. At first, a kernel, in this case the $gsk(\mathbf{x})$, must be continuous and bounded. This already restricts $\gamma > 0$. However, \cite{chaquet_using_2019} found that not placing any limits on the parameters results in a better performance. Thus, this constraint is not implemented in the current version of the algorithm. Secondly, the integral over the whole domain of the kernel $K(\mathbf{x})$ must not be $0$. Thus, it needs to be shown that

\begin{equation}
\int_{\mathbf{x} = -\mathbf{\infty}}^{\infty} gsk(\mathbf{x}) \text{ } d\mathbf{x} \neq 0
\end{equation}

Intuitively, the next restriction on $\omega \neq 0$ is found.

To simplify the following calculations, the \gls{gsk} is rewritten into polar coordinates. Further, the offsets by $c_0$ and $c_1$ are accounted for by an appropriate coordinate transformation. This results in

\begin{equation}
\lim_{t \to \infty} \int_{r=0}^{t} \int_{\theta = 0}^{2 \pi} e^{-\gamma(r^2)} sin(f r^2 - \varphi) r \text{ } dr \text{ } d\theta
\end{equation}

Since the kernel is radial symmetric and thus has no dependency on $\theta$, the respective integral can be solved immediately which results in a multiplicative factor of $2 \pi$. The integral can be further simplified by substituting $r^2 = u$.
The resulting expression can be solved with ``integration by part'' $\int f(u) \frac{g(u)}{du} = f(u) g(u) - \int g(u) \frac{f(u)}{du}$. The formula has to be applied twice.

\begin{equation}
\begin{split}
& 2 \pi \lim_{t \to \infty} \int_{r=0}^{t} e^{-\gamma(r^2)} sin(f r^2 - \varphi) r \text{ } dr  \\
= & \pi \lim_{t \to \infty} \int_{u=0}^{t} \underbrace{e^{-\gamma u }}_{\frac{g(u)}{du}} \underbrace{sin(f u - \varphi)}_{f(u)} \text{ } du \\
= - \frac{sin(fu - \varphi) e^{-\gamma u}}{\gamma} +\frac{f}{\gamma} & \left[ -cos(fu -\varphi) \frac{e^{-\gamma u}}{\gamma} -\frac{f}{\gamma} \int e^{-\gamma u} sin(fu -\varphi) du \right] \\
\end{split}
\end{equation}

The same integral is retrieved. Thus, the equation can be rearranged and solved for the integral. Considering the constant factors and the integral limits gives

\begin{equation}
\lim_{t \to \infty} \frac{\pi e^{-\gamma r^2}(- \gamma sin(f r^2 - \varphi) - f cos(f r^2 - \varphi))}{ \gamma^2 + f^2} + C \text{ } \Bigg|_{r=0}^{t}.
\end{equation}

To resolve the limit of the function towards $\infty$, the function value must be bounded. Therefore, $\gamma$ must be positive, which is already required. Finally, this results in

\begin{equation}
	\frac{-\pi (\gamma sin(\varphi) + f cos(\varphi))}{\gamma^2 + f^2} \neq 0.
\end{equation}

This places more constraints on the parameter of the \gls{gsk} as seen in the equation \eqref{eq:gsk_param_restrictions} below.

\begin{equation}
\label{eq:gsk_param_restrictions}
	\begin{split}
	\gamma & > 0 \\
	\omega & \neq 0 \\
	\gamma & sin(\varphi) + f cos(\varphi) \neq 0 \\
	\end{split}
\end{equation}

These restrictions on the parameters could be enforced during the optimisation process. They could further be used to limit the search dimension. Thus, other optimisation algorithms that are good at handling constraints must be used.

\section{Multimodality and Symmetry}
\label{chap:multimodality_and_symmetry}

The optimisation algorithm tries to find the best approximation of $N$ kernels to the solution of a differential equation. Assume, that a kernel $K$ is fully defined by a vector of parameters $\mathbf{p}$. The best fit is defined as

\begin{equation}
\mathbf{\hat{p}_{apx}} = \left[\underbrace{\left[ \mathbf{\hat{p}}_{K_0} \right] }_{\text{kernel 0}}, \cdots \underbrace{\left[ \mathbf{\hat{p}}_{K_i} \right] }_{\text{kernel i}}, \cdots \underbrace{\left[ \mathbf{\hat{p}}_{K_N} \right]}_{\text{kernel N}} \right]^T
\end{equation}

where the parameters $\mathbf{\hat{p}}$ of every kernel are chosen optimally. The optimal kernel functions $K(\mathbf{\hat{p}}_{K_i}, \mathbf{x})$ are summed up to form the optimal approximation $\hat{u}_{apx}(\mathbf{x})$.

\begin{equation}
\label{eq:uapx_kernel_sum}
\hat{u}_{apx}(\mathbf{x}) = \sum_{i=0}^{N} K(\mathbf{\hat{p}}_{K_i}, \mathbf{x})
\end{equation}

Since the order of the summation is irrelevant, any kernel-wise permutation describes an optimal solution $\mathbf{\hat{p}_{apx}}$. Thus, the fitness function $F(u_{apx}(\mathbf{x}))$ has at least $N!$ number of local optima and all of them share the same function value.

Further, a symmetry in the location of the optima is observed. All optima lay on the surface of the hypersphere that is centred at the origin and has a radius of $r = || \mathbf{\hat{p}_{apx}} ||$.

The following 3D plot in figure \ref{fig:optima_distribution} shows an exemplary distribution of optima on the fitness function. For the sake of simplicity, a kernel now consists of only one parameter. As an example, the vector $\mathbf{\hat{p}_{apx}} = \left[ 2, 1, -1 \right]^T$ describes an optimal solution that consists of 3 kernels. Any permutation of these three coordinates is itself a perfect fit.

\begin{figure}[H]
	\centering
	\noindent\adjustbox{max width=0.7\linewidth}{
		\includegraphics[width=\textwidth]{../img/pdf/symmetry.pdf}
	}
	\unterschrift{Distribution of exemplary optima on the fitness function in 3D space.}{}{}
	\label{fig:optima_distribution}
\end{figure}

This symmetry is independent of the kernel type. Large parts of the fitness function, such as the weighting and penalty factors or the number of collocation points, have no influence on the actual radial arrangement of the optima. The symmetry is a fundamental property of the sum of \gls{rbf}. Thus, it could even be applied in other fields that use this kind of representation. One of these could be non-linear function approximation.

Obviously, the order of summation is not only true for the optimum, but also for every other point in between. Thus, the entire fitness function exhibits this radial symmetry. More work investigating the structure of the fitness function must be done. With more knowledge about the features of this function, it might be possible to design algorithms specifically to that problem.

\end{document}