\Lecture{Jayalal Sarma}{ 16 November, 2020}{32}{Incidence Algebra and Mobius Inversion Over Posets}{Kaushik Arcot}{$\alpha$}{JS}

\section{Recall}

\begin{theorem}[Stronger PIE]
  \label{stronger PIE}
  Let $f,g:2^{[n]}\longrightarrow \mathbb{R}$ are functions assigning real numbers to subsets of $[n]$ with the property that for any $A\subseteq [n]$

  \[ g(A)=\sum_{S\subseteq A} f(S)\]

  Then,
  \[ f(A) = \sum_{S \subseteq A} (-1)^{|A|-|S|}g(S)\]

\end{theorem}

\textbf{Mobius Inversion of Functions}

$f,g:\mathbb{N} \to \mathbb{R} ~satisfying,$ $\forall n ~g(n) = \sum_{d|n} f(d)  ~~then,$
$$\forall n ~f(n) = \sum_{d|n} \mu(d)g(\frac{n}{d})$$

	\[
	\mu(d) = \begin{cases}
	+1 , & ~if~ ~d~ ~is~ ~a~ ~\href{https://en.wikipedia.org/wiki/Square-free_integer}{square-free} ~positive~ ~integer~ ~with~ ~an~ ~even~ ~number~ ~of~ ~prime~ ~factors~~\\
 	-1 , & ~if~ ~d~ ~is~ ~a~ ~square-free~ ~positive~ ~integer~ ~with~ ~an~ ~odd~ ~number~ ~of~ ~prime~ ~factors~~\\
	0 , & otherwise
	\end{cases}
	\]

We can see in both cases that, there is an underlying poset for both the functions. For the first pair of functions, the poset is the Subset Poset, and for the second, it is the divisibilty poset. Our endeavour in this lecture is to further abstract this concept to any poset and acquire tools for algebra within poset functions, and then for Mobius Inversion of functions over posets.

\section{Incidence Algebra of Posets}

Let $P = (X,\le)$ be a poset. Let

$$ A(P) = \{ f: X \times X \to \mathbb{R} | f(x,y) = 0, \forall x,y \in X  s.t x || y \}$$

Here $x || y$, indicates that x is incomparable to y in the Poset P. \\

Example functions : \\

\textbf{Zero Function}

$$ O(x,y) = 0 \forall x,y \in X $$

\textbf{Kronecker Delta Function}

$$ \delta(x,y) =  \begin{cases}
	1 , &  ~if~ ~x~ ~=~ ~y~ \\
	0 , & otherwise
	\end{cases} $$

\textbf{Characteristic Function of Poset}

$$ \zeta(x,y) =  \begin{cases}
	1 , &  ~if~ ~x~ ~\le~ ~y~ \\
	0 , & otherwise
	\end{cases} $$

\medskip

\textbf{Addition Operator}

Given $f \in A(P)$ , $g \in A(P)$, the '+' operator

$$ (f+g) (x,y) = f(x,y) + g(x,y) $$

If $x || y$, then $f(x,y) = 0$, $g(x,y) = 0$, and hence $(f+g)(x,y) = 0$.
Thus $(f+g) \in A(P)$

\textbf{Remark} : A(P) forms a group with '+' operator as addition.

\medskip
\textbf{Scalar Multiplication}

Given $f \in A(P)$ , $c \in \mathbb{R}$,
$$ (cf)(x,y) = c*f(x,y)$$

If $x||y$, then $f(x,y) = 0$, and so $(cf)(x,y) = 0 $, Hence $(cf) \in A(P)$

\textbf{Remark} : $A(P)$ is closed under addition and scalar multiplication. Thus, $A(P)$ forms a vector space.

\medskip

\textbf{Convolution Operator}

We define the operator '*' over $A(P)$. Given two functions,
$f,g \in A(P)$, the convolution operator is defined as,

$$ (f*g)(x,y) = \begin{cases}
	\sum_{z:x\le z \le y} f(x,z)g(z,y), &  ~if~ ~x~ ~\le~ ~y~ \\
	0 , & otherwise
	\end{cases} $$

We find if  the following properties of the convolution operator hold or not.

\textbf{Commutative} \\

Convolution operator is not commutative. Assume $x,y \in X$ , such that $x \le y$ and there exists no such z such that $x \le z \le y$. Then,

$$ (f*g)(x,y) = f(x,x)g(x,y) + f(x,y)g(y,y)$$

$$ (g*f)(x,y) = g(x,x)f(x,y) + g(x,y)f(y,y)$$

These two expressions are not always equal. Assume $f(x,x) = 1$, $g(x,y) = 1$, and $f(x,y) = f(y,y) = 0$, $g(x,x) = g(y,y) = 0$. Then, we have

 $(f*g)(x,y) = 1$ and $(g*f)(x,y) = 0$. Therefore, convolution is not commutative.

\textbf{Associative}\\

The convolution operator is associative. To prove, assume $f,g,h \in A(P)$. Then,

$$((f*g)*h)(x, y) = \sum_{z: x \le z \le y}(f*g)(x, z) h(z, y)$$

$$ = \sum_{z: x \le z \le y}(\sum_{w: x \le w \le z}f(x, w)g(w, z))h(z, y)$$

By changing the order of the summations, we get,

$$ \sum_{w: x \le w \le z}f(x, w)(\sum_{z : w \le z \le y}g(w, z)h(z, y))$$

$$ = \sum_{w: x \le w \le z}f(x, w) (g*h)(w,y) $$

$$ = (f*(g*h)) (x,y)$$

Hence, convolution is associative.

\textbf{Identity}

\textbf{Claim} : The Kronecker Delta function is the identity function of A(P)

\begin{proof}
For $f \in A(P)$, if $x \le y$,
$$ (f*\delta)(x,y) = \sum_{z:x\le z \le y} f(x,z)\delta(z,y)$$
$$ =  \sum_{z:x\le z < y} f(x,z)\delta(z,y) + f(x,y)\delta(y,y)$$

Given that $z \ne y$, we have $\delta(z,y) = 0$. Thus, $(f*\delta)(x,y)$

$$ = f(x,y)\delta(y,y)$$

$\delta(y,y) = 1$, So

$$ = f(x,y) $$

Thus, Kronecker Delta function($\delta$) is the identity function of A(P).

\end{proof}

\section{Inverse of a Function}

The inverse of a function $f \in A(P)$, is a function $g$, such that

$$ (f*g)(x,y) = \delta(x,y)$$

Does the inverse of any $f \in A(P)$ exist ? No. In fact, we check for O(x,y). Let us assume there exists an inverse, Q(x,y) for the zero function. Then,
$$ (O*Q)(x,x) = \sum_{z:x\le z \le x} O(x,z)Q(z,x)$$

$$ = O(x,x)Q(x,x) = 0$$

But $\delta(x,x)$ = 1. Thus, O(x) (zero function) does not have an inverse. Can inverse exist for any other functions ? Yes, we see from this argument that for inverse to exist for $f \in A(P)$, there exists no $x \in X$, such that $f(x,x) = 0$. Infact, we prove next that for $f \in A(P)$, such that $\forall x \in X, f(x,x) \ne 0$, then there exists a unique inverse of f.

\section{Mobius Inversion over Posets}

\begin{lemma}
For any $f \in \mathbb{A}(P)$ such that $\forall x \in X, f(x,x) \ne 0$, there exists $g \in \mathbb{A}(P)$ (which we will call $f^{-1}$) such that $\forall x,y \in X, g \star f = \delta$ where $\delta$ is the Kronecker delta function.
\end{lemma}
\begin{proof}
We will directly write down the function $g$ based on $f$. For incomparable pairs $(x,y)$ we can define $g(x,y) = 0$. This ensures that $g \in \mathbb{A}(P)$.

For comparable pairs, the function $g$ on input $(x,y)$ is defined based on an induction on a parameter of $\ell(x,y) \in X$. The distance between $x$ and $y$, denoted by $\ell(x,y)$ is said to be $k$, if $\exists z_1, z_2, \ldots z_{k-1}$ such that $x < z_1 < z_2 < \ldots < z_{k-1} < y$ and $\not\exists w_1, w_2, \ldots w_k$ such that $x < w_1 < w_2 < \ldots < w_{k-1} < w_k < y$. In other words, this is the length of the longest directed path from $x$ to $y$ (or vice versa).

Now we are ready to describe the definition of $g$ on comparable pairs $(x,y)$.

As the base case, when $\ell(x,y) = 0$, we know that, $x=y$, hence we define:

$$g(x,x) = \frac{1}{f(x,x)}$$
Let us quickly check that this indeed satisfies the property that we wanted for inputs of the kind $(x,x)$. That is,
$$(g \star f) (x,x) = \sum_{x \le z \le y} g(x,z)f(z,y) = g(x,x)f(x,x) = 1 = \delta(x,x)$$

To do the definition inductively: let us assume that $g(x,y)$ is defined when $\ell(x,y) \le k$, and we consider a pair $(x,y)$ with distance $k+1$. We define :
$$g(x,y) = \frac{-1}{f(y,y)}\sum_{x \le z < y} g(x,z)f(z,y)$$
Note that this is well-defined since $g(x,z)$ is used in the RHS only for pairs $(x,z)$ with $\ell(x,z) \le k$ and that $f(y,y) \ne 0$ for every $y \in X$.
With this definition :
\begin{eqnarray*}
(g \star f) (x,y) & = & \sum_{x \le z \le y} g(x,z)f(z,y)\\
& = & \sum_{x \le z < y} g(x,z)f(z,y) + g(x,y)f(y,y)\\
& = & \sum_{x \le z < y} g(x,z)f(z,y) + \left(\frac{-1}{f(y,y)}\sum_{x \le z < y} g(x,z)f(z,y)\right) f(y,y) \\
& = & 0 = \delta(x,y)
\end{eqnarray*}
Since $g \in \mathbb{A}(P)$ we have completed the proof.
\end{proof}

\Lecture{Jayalal Sarma}{ 18 November, 2020}{33}{Mobius Inversion Theorem for Posets and Corollaries}{Abhishek Aladahalli}{$\alpha$}{JS}

\noindent In particular the \textbf{zeta function} ($\zeta$)
$$ \zeta (x, y) =
\begin{cases}
&1 ~~~~\textrm{if} ~~x \le y \\
&0 ~~~~\textrm{otherwise} \\
\end{cases}
$$
has an inverse called the \textbf{Mobius Function} ($\mu$) of the poset.\\

\noindent From the lemma proved above we define Mobius Function as,

\noindent \underline{For $x = y$},
$$g(x,x) = \mu(x,x) = \frac{1}{\zeta (x,x)} = 1$$
$$=> \boxed{\mu(x,x) = 1 ~~ (\textrm{Since} ~\zeta (x,x) = 1)}$$

\noindent \underline{For $x ~ || ~ y$}, (i.e. when $x$ and $y$ are incomparable)
$$g(x,y) = \mu(x,y) = \frac{-1}{\zeta(y,y)}\sum_{x \le z < y} \mu(x,z) \zeta(z,y)$$
$$=> \mu(x,y) = -1\sum_{x \le z < y} \mu(x,z) \zeta(z,y) $$
$$=> \mu(x,y) = -1\sum_{x \le z < y} \mu(x,z) (0) $$
(Since, zeta function ($\zeta$) is $0$ for $x$ and $y$ which are incomparable)
$$=> \boxed{\mu(x,y) = 0}$$

\noindent \underline{For $x \le y$},
$$g(x,y) = \mu(x,y) = \frac{-1}{\zeta(y,y)}\sum_{x \le z < y} \mu(x,z) \zeta(z,y)$$
$$=> \mu(x,y) = -1\sum_{x \le z < y} \mu(x,z) \zeta(z,y) $$
$$=> \mu(x,y) = -1\sum_{x \le z < y} \mu(x,z) (1) ~~~~(\textrm{Since}~~ \zeta(z,y) = 1 ~ \textrm{for} ~z < y)$$
$$=> \boxed{\mu(x,y) = -\sum_{x \le z < y} \mu(x,z)}$$

Hence, Mobius Function is defined as

\begin{equation}
\mu (x, y) =
    \begin{cases}
    &-\sum\limits_{x \le z < y} \mu(x,z) ~~~~~~~~~~~~~~~~~~~~~\textrm{if} ~~x < y \\
    &1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\textrm{if} ~~ x=y \\
    &0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\textrm{if x and y are incomparable i.e.} ~~ x ~||~ y\\
    \end{cases}
\end{equation}

\noindent Now, Let's consider an example of Divisibility Poset and apply the mobius function on it,

\begin{center}

\begin{tikzpicture}[auto, node distance=2cm, every loop/.style={}, thick,main node/.style={circle,draw,font=\sffamily\Large\bfseries}]

\node[main node] (1) {90};
\node[main node] (2) [below left of=1] {30};
\node[main node] (3) [right of=2] {45};
\node[main node] (4) [right of=3] {18};
\node[main node] (5) [below left of=2] {10};
\node[main node] (6) [right of=5] {15};
\node[main node] (7) [right of=6] {6};
\node[main node] (8) [right of=7] {9};
\node[main node] (9) [below right of=5] {5};
\node[main node] (10) [right of=9] {2};
\node[main node] (11) [right of=10] {3};
\node[main node] (12) [below left of=11] {1};

\path[every node/.style={font=\sffamily\small}]
    (1) edge node [] {} (4)
        edge node [] {} (3)
        edge node [] {} (2)
    (2) edge node [] {} (5)
        edge node [] {} (6)
        edge node [] {} (7)
    (3) edge node [] {} (6)
        edge node [] {} (8)
    (4) edge node [] {} (7)
        edge node [] {} (8)
    (5) edge node [] {} (9)
        edge node [] {} (10)
    (6) edge node [] {} (9)
        edge node [] {} (11)
    (7) edge node [] {} (10)
        edge node [] {} (11)
    (8) edge node [] {} (11)
    (9) edge node [] {} (12)
    (10) edge node [] {} (12)
    (11) edge node [] {} (12);

\end{tikzpicture}

\end{center}

\begin{align*}
\textrm{For} ~~(x,y) = (1,1),\\
    \mu (x,y) &= \mu (1,1)\\
=>  \mu (1,1) &= 1~~~~~~~~~~~(\because x = y)\\\\
\textrm{For} ~~(x,y) = (1,2),\\
    \mu (x,y) &= \mu (1,2)\\
=>  \mu (1,2) &= -\sum\limits_{1 \le z < 2} \mu(1,z)    ~~~~~~~\textrm{(By definition of mobius function)}\\
    &= - \mu(1,1)\\
=>  \mu(1,2) &= -1\\\\
\textrm{Similarly for} ~~(x,y) = (1,3),\\
=> \mu(1,3) &= -1\\\\
\textrm{Similarly for} ~~(x,y) = (1,5),\\
=> \mu(1,5) &= -1\\
\end{align*}

Thus, In general for a prime number $p$,

$$\mu(1,p) = -1$$

Now,
\begin{align*}
\textrm{For} ~~(x,y) = (1,10),\\
    \mu (x,y) &= \mu (1,10)\\
=>  \mu (1,10) &= -\sum\limits_{1 \le z < 10} \mu(1,z)    ~~~~~~~\textrm{(By definition of mobius function)}\\
    &= - (\mu(1,1) + \mu(1,2) + \mu(1,5)) \\
=>  \mu(1,10) &= 1\\\\
\textrm{Similarly for} ~~(x,y) = (1,15),\\
=> \mu(1,15) &= 1\\\\
\textrm{Similarly for} ~~(x,y) = (1,6),\\
=> \mu(1,6) &= 1\\
\end{align*}

Thus, In general for prime numbers $p$ and $q$ ,
\begin{eqnarray*}
\mu(1,p q) & = & -(\mu(1,1) + \mu(1,p) + \mu(1,q)) \\
\implies \mu(1,p q) & = & 1
\end{eqnarray*}

\noindent Now,
\begin{align*}
\textrm{For} ~~(x,y) = (1,9),\\
    \mu (x,y) &= \mu (1,9)\\
=>  \mu (1,9) &= -\sum\limits_{1 \le z < 9} \mu(1,z)    ~~~~~~~\textrm{(By definition of mobius function)}\\
    &= - (\mu(1,1) + \mu(1,3)) \\
=>  \mu(1,9) &= 0\\
\end{align*}

Thus, In general for a prime number $p$,
$$\mu(1,p^2) = 0$$

\noindent Now,
\begin{align*}
\textrm{For} ~~(x,y) = (1,45),\\
    \mu (x,y) &= \mu (1,45)\\
=>  \mu (1,45) &= -\sum\limits_{1 \le z < 45} \mu(1,z)    ~~~~~~~\textrm{(By definition of mobius function)}\\
    &= - (\mu(1,1) + \mu(1,3) + \mu(1,5) + \mu(1,9) + \mu(1,15)) \\
=>  \mu(1,45) &= 0\\\\
\textrm{Similarly for} ~~(x,y) = (1,18),\\
=> \mu(1,18) &= 0\\
\end{align*}

Thus, In general for prime numbers $p$ and $q$,

\begin{eqnarray*}
\mu(1,p^2 q) & = & -(\mu(1,1) + \mu(1,p) + \mu(1,q) + \mu(1,p q) + \mu(1,p^2) ) \\
\implies \mu(1,p^2 q) & = & 0
\end{eqnarray*}

\noindent Now,
\begin{align*}
\textrm{For} ~~(x,y) = (1,30),\\
    \mu (x,y) &= \mu (1,30)\\
=>  \mu (1,30) &= -\sum\limits_{1 \le z < 30} \mu(1,z)    ~~~~~~~\textrm{(By definition of mobius function)}\\
    &= - (\mu(1,1) + \mu(1,3) + \mu(1,5) + \mu(1,2) + \mu(1,6) + \mu(1,10) + \mu(1,15)) \\
=>  \mu(1,30) &= -1\\
\end{align*}

Thus, In general for prime numbers $p$, $q$ and $r$,

\begin{eqnarray*}
\mu(1,p q r) & = & -(\mu(1,1) + \mu(1,p) + \mu(1,q) + \mu(1,p q) + \mu(1,q r) + \mu (1, p r) ) \\
\implies \mu(1,p q r) & = & 1
\end{eqnarray*}

Therefore, in general, The mobius function for the divisibility poset turns out to be,

\begin{eqnarray*}
\mu (d)
    \begin{cases}
        &= +1 ~~~~~~~~~~~~\textrm{If $d$ is a product of even number of primes}\\
        &= -1 ~~~~~~~~~~~~\textrm{If $d$ is a product of odd number of primes}\\
        &= 0 ~~~~~~~~~~~~~\textrm{Otherwise}\\
    \end{cases}
\end{eqnarray*}

\section{Mobius Inversion theorem over Posets}

\textbf{Assumption :}

\noindent In the poset $X$, there is a unique $m$ such that $\forall x \in X$, $m \le x$

\begin{theorem}

For a poset $X$, if $f$ is a function $f: X \rightarrow \mathbb{R}$ and $g$ is a function $g: X \rightarrow \mathbb{R}$ such that,
$$\forall a \in X, ~~~~~~~~g(a) = \sum_{x \le a} f(x)$$
then
$$\boxed{\forall a \in X, ~~~~~~~~f(a) = \sum_{x \le a} \mu(x,a) g(x)}$$
where $\mu$ is the mobius function of the poset.
\end{theorem}

\begin{proof}

As from the assumption, let $m$ be the unique minimal element of the poset $X$.

\noindent And let $f$ and $g$ be the functions defined as above such that

$$\forall a \in X, ~~g(a) = \sum\limits_{x \le a} f(x)  ~~~~~~~~~(1)$$

\noindent Now, we define two functions $F(x,y)$ and $G(x,y)$ such that,

\begin{minipage}{0.48\linewidth}
\begin{align*}
   & \forall a \in X ,\\
   & F(m,a) = f(a)\\
   & G(m,a) = g(a)\\
\end{align*}
\end{minipage}
\begin{minipage}{0.48\linewidth}
\begin{align*}
   & \textrm{For all other}~ x ~\textrm{and} ~ y ~ \textrm{where} ~ x \ne m,\\
   & F(x,y) = 0\\
   & G(x,y) = 0\\
\end{align*}
\end{minipage}

\noindent Thus, By Incidence Algebra $F,G \in A(P)$ for $P(X,\le)$.\\

\noindent \textbf{Claim :}
$$\boxed{G = F \star \zeta ~~\textrm{where $\zeta$ is the zeta function.}}$$

\noindent \underline{Note:} Through this claim we can show that $F = G \star \mu$ (where $\mu$ is the mobius function) and thus proving the theorem.\\

\noindent \textbf{Proof for the claim :}

\noindent Consider $G(x,y)$,

\textbf{Case 1 : $\forall a \in X$}
\begin{align*}
    G(m,a) &= g(a)\\
    &= \sum_{x \le a} f(x) ~~~~~~~~~~~~~~~~ \textrm{( $\because~$ from (1))}\\
    &= \sum_{m \le x \le a} F(m,x) ~~~~~~~~ \textrm{($\because~$from definition of $F(x,y)$)}\\
    &= \sum_{m \le x \le a} F(m,x) \zeta(x,a) ~~~~~~\textrm{($\because \zeta (x,a) = 1, \forall x \le a$ )}
\end{align*}
$$=> \boxed{G(m,a) = F \star \zeta (m,a)} ~~~~~~~~~~~~ \textrm{( $\because~$ By defintion of convolution ($\star$))}$$

\textbf{Case 2 :} For all other $x$ and $y$ where $x \ne m$
\begin{align*}
    G(x,y) &= 0\\
    &= \sum_{x \le z \le y} F(x,z) \zeta(z,y) ~~~~~~\textrm{($\because F(x,z) = 0, \forall x \ne m$ )}
\end{align*}
$$=> \boxed{G(x,y) = F \star \zeta (x,y)} ~~~~~~~~~~~~ \textrm{($\because~$By defintion of convolution ($\star$))}$$

\noindent Hence, The claim $G = F \star \zeta $ is true.\\

\noindent Thus, By definition
$$\boxed{F = G \star \mu}$$

\noindent Therefore, $\forall a \in X$,
\begin{align*}
    F(m,a) &= \sum_{m \le x \le a} G(m,x) \mu(x,a) ~~~~~~~~ \textrm{($\because~$By defintion of convolution ($\star$))}\\
 => f(a)  &=\sum_{x \le a} g(x) \mu(x,a) ~~~~~~~~ \textrm{($\because~$From defintion of $F(m,a) = f(a), G(m,x) = g(x)$)}
\end{align*}

\noindent Thus,
$$\forall a \in X, ~~~~~~~~f(a)  = \sum\limits_{x \le a} \mu(x,a) g(x)$$

Hence Proved.

\end{proof}

\subsection{Corollary}

\noindent \underline{\textbf{Application of Mobius Inversion theorem on Subset Poset}}\\

\noindent For the subset poset we will show, if $f$ and $g$ are functions defined from subsets to real numbers (i.e.
$f: 2^{[n]} \rightarrow \R$ , $g: 2^{[n]} \rightarrow \R $ ) such that
$$\forall A \subseteq [n], ~~~~~ g(A) = \sum_{S \subseteq A} f(S) $$
then
$$\boxed{\forall A \subseteq [n], ~~~~~ f(A) = \sum_{S \subseteq A} (-1)^{|A|- |S|}g(S) } $$

\noindent \textbf{Claim :} For $X \subseteq [n]$ and $Y \subseteq [n]$,
$$ \mu (X,Y) =
\begin{cases}
    &0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~\textrm{if $~X \nsubseteq Y$}\\
    &(-1)^{|Y| - |X|} ~~~~~~~~~~\textrm{if $~X \subseteq Y$}\\
\end{cases}
$$
\underline{\textsf{Note :}} If we prove the claim to be true then the above subset poset can directly be shown as a corollary of the mobius inversion theorem by substituting the value of $\mu$.\\

\noindent \textbf{Proof for the claim :}

\noindent We will apply Induction on $|Y|-|X|$ for $X \subseteq Y$.\\

\noindent \underline{Base step}, For $|Y| - |X| = 0$ i.e. $X = Y$
\begin{align*}
    \mu (X,Y) &= \mu(X,X)\\
    => \mu (X,Y) &= 1  ~~~~~~~~~~\textrm{($\because X \subseteq X ~$ and $~\mu (X,X) = 1$ )}\\
    => \mu (X,Y) &= (-1)^{0} = (-1)^{|Y| - |X|}
\end{align*}

\noindent \underline{Induction step}, Assume it's true for $|Y| - |X| \le k$. Now, we need to prove it's also true for $|Y| - |X| = k + 1$.\\

\noindent Thus, Consider $\mu (X,Y)$ for $|Y| - |X| = k + 1$,
\begin{align*}
    \mu (X,Y) &= - \sum_{X \subseteq Z \subset Y} \mu (X,Z) ~~~~~~~~~~~~~\textrm{($\because X < Y~$ and by definition of mobius function )}\\
 => \mu (X,Y) &= - \sum_{X \subseteq Z \subset Y} (-1)^{|Z|-|X|} ~~~~~~~\textrm{($\because |Z| - |X| \le k~$ and it's assumed true for $|Y| - |X| \le k$)}\\
 => \mu (X,Y) &= - \sum_{X \subseteq Z \subseteq Y} (-1)^{|Z|-|X|} + (-1)^{|Y| - |X|} ~~~~~~~\textrm{($\because~$Adding and subtracting the subset $Y$)}
\end{align*}
$$=> \boxed{\mu (X,Y) = - \sum_{X \subseteq Z \subseteq Y} (-1)^{|Z|-|X|} + (-1)^{|Y| - |X|}} ~~~~~~\textrm{(1)}$$

\noindent Now, we will show that 1st term in the R.H.S goes to 0 by using a combinatorial argument.\\

\noindent So, For given $X, Y \subseteq [n]$ and $X \subseteq Y$,
\begin{align*}
    \sum_{X \subseteq Z \subseteq Y} (-1)^{|Z|-|X|} &= \sum_{k} \sum_{\substack{X \subseteq Z \subseteq Y\\ |Z| - |X| = k}} (-1)^k ~~~~~\textrm{(By rewriting the summation based on the size of $|Z| - |X|$)}\\
    &= \sum_{k} (-1)^k {n \choose k} ~~~~~~~~~\textrm{($\because~$ Number of ways of choosing such a subset $|Z|$ is ${n \choose k}$) }\\
    &= 0 ~~~~~~~~~~~~~~~~~~~~~~~~~~\textrm{($\because ~$ Signed summation of binomial coefficients is $0$)}\\
=> \sum_{X \subseteq Z \subseteq Y} (-1)^{|Z|-|X|} &= 0\\
\end{align*}
Therefore, Equation (1) becomes,\\

$$\mu (X,Y) = (-1)^{|Y|-|X|}$$

Hence Proved.\\

\noindent Thus, from this definition of $\mu$ and the theorem of Mobius Inversion, the corollary on the subset poset follows that ,\\

\noindent \textbf{Corollary :} If $f$ and $g$ are functions defined from subsets to real numbers (i.e.
$f: 2^{[n]} \rightarrow \R$ , $g: 2^{[n]} \rightarrow \R $ ) such that
$$\forall A \subseteq [n], ~~~~~ g(A) = \sum_{S \subseteq A} f(S) $$
then
$$\forall A \subseteq [n], ~~~~~ f(A) = \sum\limits_{S \subseteq A} (-1)^{|A|- |S|}g(S)$$

\Lecture{Jayalal Sarma}{ 19 November, 2020}{34}{More Applications of Structure of Partial Orders, Fixed Point Theorems}{Kaushik Arcot, Abhishek Aladahalli}{$\alpha$}{JS}

\section{Equinumerous Sets and Bijections}

We define Cardinality using the concept of Bijections and Equinumerous sets.
\begin{theorem}
Two sets A, B are equinumerous or equal in cardinality iff there
exists a bijection $f: A \to B$. Or,\\

$ |A| = |B| \Leftrightarrow \exists$ bijection $f:A \to B $ \\

\end{theorem}

\textbf{Example of Equinumerous sets and Bijections}\\

\textbf{Even Numbers and $\mathbb{N}$}\\

$f:\mathbb{E} \to \mathbb{N}$,  $f(x) = x/2$

Is a bijection from even numbers to Natural Numbers.\\

The Noton of Cardinality arises from the fact that these Equinumerous sets divide the Subset Partial Order into different equivalence classes and each of those classes correspond to different cardinalities. \\

So to prove that two sets are equinumerous sets, we need to prove there exists a bijection between the two sets. To prove that there exists a bijection between two sets, we can just prove that there exists an injection in both directons.

\begin{theorem}[Cantor Schroeder Bernstein Theorem]
$A,B \subseteq U$ ,$\exists f:A \to B, g:B \to A$ are injections $\implies \exists h:A \to B$ which is a bijection
\end{theorem}

This theorem is intuitive in finite sets, but to prove for infinite sets, we need some more tools that concern posets and cardinality.

\section{Knaster-Tarski Fixed point theorem}

To prove this theorem, we need to understand some properties.

\textbf{Fixed Point of a Function}\\
Let $f: X \to X$, be a function. Then, a point $x \in X$, is said to be a fixed point of function $f$ if $f(x) = x$.

\textbf{Preserving Order of Posets}

Let $P(X,\le)$ be a poset. Then any function $f: X \to X$ is said to preserve order if
$$ \forall x,y \in X, x \le y \implies f(x)  \le f(y) $$

\textbf{Upper Bound of a Set}

Let $P(X,\le)$ be a poset, and $A \subseteq X$, then the upper bound of set A is defined as

$$ UB(A) = \{x \in X | \forall y \in A, y \le x\} $$

\textbf{Least Upper Bound of a Set}

Let $P(X,\le)$ be a poset, and $A \subseteq X$, then the least upper bound of set A is defined from the upper bound set, as the least element in the upper bound set of A.

$$ LUB(A) = \{x  | \forall y \in UB(A), x \le y\} $$

\textbf{Complete Partial Order}

A partial order $P(X,\le)$ is set to be a complete partial order, if $\forall A \subseteq X$ , $\exists LUB(A)$

\medskip

We can understand the notion of a complete partial order using the subsets partial order. Let $P(X,\le)$, be the subsets partial order, where $X = 2^{[n]}$. Then, let $A \subseteq X$, be a subset of X, which is a set of subsets of $[n]$. Assume, $A = \{A_1,A_2,...A_k\}$, and S =  $\bigcup_{i=1}^{k} A_{i}$ forms the LUB of the set A. We know that $\forall i \in [k], A_i \le S$, (because $A_i$ is a subset of S, S is the union of all $A_i$'s). And it is the lowest upper bound, as any element missing from $S$, will be present in one of the $A_i$'s.

Now, we're ready to prove the Knaster-Tarski Theorem.

\begin{theorem}[Knaster-Tarski Fixed point theorem]
Let $P(X,\le)$ be a complete partial order, and let $f: X \to X$, be a function that preserves the order on the partial order P. Then $f$ has a fixed point.
\end{theorem}

\begin{proof}
Consider the set $A = \{x | x \le f(x)\}$. Assume that A is non-empty. This is possible when X has a least element in P. Then, for that least element $x \in X$, $\forall y \in X, x \le y$. And $f(x) \in X$, thus, $x \le f(x)$, and therefore $x \in A$. \\

\textbf{Observation} If $x \in A$, then $f(x) \in A$. \\
$x \in A$. $\implies x \le f(x)$. As f preserves order,\\
$ \implies f(x) \le f(f(x) $. Let f(x) = z, then \\
$z \le f(z)$. Thus, $z \in A$. $\implies f(x) \in A$.\\

Since the partial order is complete, and $A \subseteq X$, there exists an LUB(A). Let

$$ y = LUB(A)$$

Consider, $\forall x \in A$. As y is LUB,\\
$x \le y$ , As f preserves order\\
$ f(x) \le f(y) $. Becasue $x \in A$, \\

$x \le f(x) \le f(y), \forall x \in A$ \\

This implies that $f(y)$ is an upper bound of A. Since $y$ is the LUB of A,
$$y \le f(y) $$

By the property of A, it is clear that $y \in A$. And because $y \in A$, and from the observation proven above, we have that $f(y) \in A$. \\

Since $f(y) \le A$, and y is LUB(A) of A, we have that
$$ f(y) \le y $$

So, we have,
$$ f(y) \le y \And y \le f(y) $$.

By the anti-symmetry of the relation $\le$ in the partial order, we have
$$f(y) = y $$

So, we have $y$ as the fixed point.\\

Hence proved.

\end{proof}

\section{Cantor-Schroeder-Bernstein Theorem}

\begin{theorem}[Cantor Schroeder Bernstein Theorem]
$A,B \subseteq U$ ,$\exists f:A \to B, g:B \to A$ that are injections $\implies \exists h:A \to B$ which is a bijection.
\end{theorem}

\begin{proof}

We find a $H \subseteq A$, and we build a function h, such that
$$ h |_H = f \And h|_{\bar H} = g $$

Then, we prove that H is a bijection.

\textbf{Claim} For $H \subseteq A$, if $g( \bar{f(H)} ) = \bar H$ then by  the above definition, h is a bijection. \\

Or, we have to find an H such that $H = \bar{g(\bar{f(H)})}$ \\

First, we find an H such that this claim is satisfied, and then we prove the claim. \\

We define the following function on the subset partial order of A.\\

$\tau : 2^{[A]} \to 2^{[A]}$, such that for any $T \subset A$,

$$ \tau(T) = \bar{g(\bar{f(T)})}$$

We will prove the properties of preserving order and completeness of this Partial Order. \\

We know that the partial order is complete, as it is a subset poset. \\

For preserving order, assume $S \subset T$. . By applying f on both sides, we have $f(S) \subset f(T)$. \\

Taking complement, we have $\bar{f(T)} \subset \bar{f(S)}$.\\

As g is a function from B to A, and $f(x) \in B \forall x \in A$, applying g on both sides, we get \\

$$ g(\bar{f(T)}) \subset g(\bar{f(S)})$$

Again taking complement, we get

$$ \bar{g(\bar{f(S)})} \subset \bar{g(\bar{f(T)})} $$

So, $\tau(S) \subset \tau(T)$. Hence $\tau$ preserves order.\\

Because $\tau$ preserves order and and $P = ([A],\le)$ is a complete poset, we have that there exists a fixed point for $\tau$. Therefore

$$ \exists H \subset A, \bar{g(\bar{f(H)})} = H $$

Now that we have arrived at an H, we now need to prove that $h$ is a bijection. We need to prove three parts. \\

\textbf{Well-defined} \\
h is well defined. $\forall x \in A$, if $x \in H$, the output of $h$ is f(x). Otherwise, as $g$ is injective, and $\bar H = g(\bar{f(H)})$, and so $\bar H \subseteq g(B)$, and $g^{-1}(x)$ is defined $\forall x \in g(B)$, and subsequently in $\bar H$. So $\forall x \in A$, h(x) is well defined.

\textbf{Injective}\\
We need to prove $\forall x,y \in A$, $x \ne y \implies h(x) \ne h(y)$.

\textsf{Case 1} : $x \in H \And y \in H$\\

$\forall x \in H$, $h(x) = f(x)$. Therefore, $x \ne y \implies f(x) \ne f(y) \implies h(x) \ne h(y) $

\textsf{Case 2}: $x \in \bar H \And y \in \bar H$

$\forall x \in \bar H$, $h(x) = g^{-1}(x)$.  Since g is injective, $g^{-1}$  is well-defined, and thus $g^{-1}(x) \ne g^{-1}(y) \implies h(x) \ne h(y)$

\textsf{Case 3} : $x \in H \And y \in \bar H$
For $x \in H$, $h(x) = f(x)$\\
For $y \in \bar H$, $h(y) = g^{-1}(y)$ \\

We need to prove $h(x) \ne g^{-1}(y)$ \\

We know, $g(\bar{f(H)}) = \bar H $. As $g^{-1}$ is well defined, $\implies g^{-1}(\bar H) \subseteq \bar{f(H)}$ \\. Therefore, $g^{-1}(H) \nsubseteq f(H)$. Moreover, as $g^{-1}(\bar H) \subseteq \bar{f(H)}$, $ \nexists u \in \bar H$, such that $g^{-1}(u) \in f(H)$.
Thus, for $x \in H \And y \in \bar H$, $g^{-1}(y) \ne f(x)$. Hence proved.

\textbf{Surjective} \\
To prove, $range(h) = B$. That is, $\forall y \in B, \exists x \in A$, such that $h(x) = y$. Assume, $y \in f(H)$. Given that $\forall x in H, h(x) = f(x)$, and because $y \in f(H)$, $\exists x, f(x) = y$.

Now, for $x \in \bar{f(H)}$. We know,

\begin{equation} \label{eqn:1} \bar H = g(\bar{f(H)}) \end{equation}

And that $g^{-1}(x)$ is defined $\forall x \in \bar H$ (already proved). Since $g^{-1}$ is obviously injective ( if $x \ne y$ and  $g^{-1}(x) = g^{-1}(y) = a$, then $g(a)$ is not defined), we can apply $g^{-1}$ on both sides of \ref{eqn:1}, to get

$$ g^{-1}(\bar{H}) \subseteq \bar{f(H)}$$

As $g{-1}$ is injective, and we know that $|\bar H| = g(\bar{f(H)})$, both the sets are of the same size, thus,
$$ g^{-1}(\bar H) = \bar{f(H)}$$.

$\forall x \in \bar{f(H)}$, we have a pre-image in $g^{-1}(\bar H)$. Hence, h is surjective.

From the three parts, we have proven that $h : A \to B$, is a bijective function, constructed from $f : A \to B$ and $g : B \to A$, two injective functions.

\end{proof}


\subsection{T-Hat $\left(\hat{T}\right)$}
Arc length parameterization gives us another way to find $\hat{T}$.\\
\begin{equation*}
	\hat{T} = \frac{\vec{r^\prime}(t)}{\norm{\vec{r^\prime}(t)}} = \frac{\mathrm{d}\vec{r}/\mathrm{d}t}{\mathrm{d}s/\mathrm{d}t} = \frac{\mathrm{d}\vec{r}}{\mathrm{d}s}
\end{equation*}


\section{DT LTI -- transfer function and block diagrams}

\subsection{}

\begin{frame}
\frametitleTC{Preliminary}
\framesubtitleTC{for an alternative system representation, particularly useful for control}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Let us introduce the \TC{one-step advance operator} $z$:
       \begin{displaymath}
        z\nu(k) := \nu(k+1) \qquad\qquad \text{whatever $\nu$ is}.
       \end{displaymath}
 \item This allows for a compact way to write LTI systems without evidencing the state (but it is there!).
 \item Suppose for example that we have a system with input $u$ and output $y$, ruled by
       \begin{displaymath}
        y(k) = ay(k-1)+bu(k-1),
       \end{displaymath}
       where clearly the state is the previous value of $y$;
 \item we can re-write $y(k+1)=ay(k)+bu(k)$ in the form
       \begin{displaymath}
        (z-a)y(k) = bu(k).
       \end{displaymath}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Preliminary}
\framesubtitleTC{introducing the transfer function}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Rearranging -- remember the \emph{operatorial} meaning of $z$ -- we get
       \begin{displaymath}
        y(k) = \frac{b}{z-a}u(k)
       \end{displaymath}
\item[] and we can define the \TC{Transfer Function (TF)} notation, that expresses the system\\
        as a compound operator built upon the elementary one $z$:
       \begin{itemize}[<+-| alert@+>]
       \item we write the system's transfer function $G(z)$ as
             \begin{displaymath}
               G(z) = \frac{b}{z-a},
             \end{displaymath}
       \item[] and in force of this
             \begin{displaymath}
              y(k) = G(z)u(k) \quad \text{means} \quad y(k) = ay(k-1)+bu(k-1).
             \end{displaymath}
       \end{itemize}
 \item \vfill May not look so useful at the moment, but you will see later on\\
       how handy transfer functions are to determine controllers.
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{State space representation and transfer function}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Given the SS representation we saw, shift all times by one (which is legit as the system is TI):
       \begin{displaymath}
        \left\{
         \begin{array}{rlll}
          x(k+1) &= A x(k)   + b u(k)\\
          y(k+1) &= c x(k+1) + d u(k+1)
         \end{array}
        \right.
       \end{displaymath}
 \item Apply the advance operator:
       \begin{displaymath}
        \left\{
         \begin{array}{rl}
          \textcolor{red}{z}x(k) &= A x(k) + b u(k)\\
          \textcolor{red}{z}y(k) &= c \textcolor{red}{z}x(k) + d\textcolor{red}{z} u(k)
         \end{array}
        \right.
       \end{displaymath}
 \item Drop useless $z$'s and rearrange ($I$ is identity matrix of dimension $n$):
       \begin{displaymath}
        \left\{
         \begin{array}{rl}
          (zI-A)x(k) &= b u(k)\\
          y(k)       &= c x(k) + d u(k)
         \end{array}
        \right.
       \end{displaymath}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{State space representation and transfer function}
\framesubtitleTC{once again, sticking to the SISO case}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Solve state equation for $x(k)$:
       \begin{displaymath}
        x(k) = (zI-A)^{-1} b u(k)
       \end{displaymath}
 \item Substitute into output equation and rearrange:
       \begin{displaymath}
        y(k) = \textcolor{red}
                         {\underbrace{\left[c(zI-A)^{-1} b +d \right]}
                                    _{\text{Transfer function } G(z)}
                         } \; u(k)
       \end{displaymath}
 \item We then define the transfer function of the generic SISO DT LTI\\
       system $(A,b,c,d)$ as
       \begin{displaymath}
        G(z) = c(zI-A)^{-1} b +d.
       \end{displaymath}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Block diagrams}
\framesubtitleTC{Preliminaries}
\myPause
\begin{itemize}[<+-| alert@+>]
\item Block diagrams (BDs) are a graphical formalism to represent dynamic systems, that we see here limited to the
      DT LTI class.
\item They are useful to study \TC{interconnected} systems, i.e., compounds of \TC{subsystems} (e.g., a controller
      and the controlled object).
\item Important \emph{caveat}, that we state right from the beginning:
      \begin{itemize}[<+-| alert@+>]
      \item    a BD MUST NOT BE CONFUSED with a flow diagram;
      \item [] although subsystems have inputs and outputs,
      \item [] their compound comes from assembling \TC{equations}.
      \end{itemize}
\end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Block diagram components}
\framesubtitleTC{Block and summation node}
\myPause
\begin{center}
 \input{./Unit-02/img/BlockTF.tex}
\end{center} \myPause
\begin{itemize}[<+-| alert@+>]
\item \TC{Block} (left):\\
      an LTI SISO system with the indicated transfer function -- in the shown\\
      example, the \TC{equation} $G(z)u(k)-y(k)=0$.
\item \TC{Summation node} (right):\\
      a summation expression -- in the shown example, the \TC{equation}
      $u_1(k)-u_2(k)+u_3(k)-y(k)=0$.
\end{itemize}
\end{frame}


\documentclass{article}

% include some useful things
\usepackage{verbatim}  % for printing unformatted text
\usepackage{float}         % for controlling the location of figure and graphics on the page
\usepackage{blindtext}
\usepackage{graphicx}\usepackage{amsmath}
\usepackage{amsmath}
\usepackage{amssymb}
%  Begin writing content below this line
\begin{document}

%  Print your name and the assignment number
\begin{center}{\huge  Deepayan Bhadra - hmwk2 Solutions}\end{center}

\section*{Solution to Q1(a) to create a classification}
% The verbatim environment is good for reproducing text without having latex try to format it for you.
\begin{verbatim}

def create_classification_problem(Nd,Nf,kappa):
    D = buildmat(Nd,Nf,kappa)
    w = np.random.randn(Nf,1);
    c = np.sign(D.dot(w));
    flipper = np.ones((Nd,1)) - 2*(np.random.rand(Nd,1)>.9);
    c = c*flipper;
    return D,c

[ D,c ] = create_classification_problem(100, 2, 1)
valid1 = D[np.where(c==1)[0]]
valid2 = D[np.where(c==-1)[0]]
plt.scatter(valid1[:, 0], valid1[:, 1])
plt.scatter(valid2[:, 0], valid2[:, 1])
plt.title("Scatter plot with condition no = 1")
plt.show()

[ D,c ] = create_classification_problem(100, 2, 10)
valid1 = D[np.where(c==1)[0]]
valid2 = D[np.where(c==-1)[0]]
plt.scatter(valid1[:, 0], valid1[:, 1])
plt.scatter(valid2[:, 0], valid2[:, 1])
plt.title("Scatter plot with condition no = 10")
plt.show()

\end{verbatim}

\section*{Solution to Q1(b) to evaluate logistic regression}
\begin{verbatim}

def logreg_objective(x,D,c):
    z = np.diagflat(c).dot(D).dot(x)
    idxN, idxP = z<0, z>=0
    y1 = np.sum(-z[idxN]) + np.sum([math.log(np.exp(x)+1) for x in z[idxN]])
    y2 = np.sum([math.log(1+np.exp(-x)) for x in z[idxP]])
    y = y1+y2
    return y

Nd, Nf, kappa = 1000,100,10
[D,c] = create_classification_problem(Nd,Nf,kappa)
x = np.random.randn(Nf,1);

print("The logistic regression objective is")
y = logreg_objective(x,D,c)
print(y)

input("Press Enter to continue...")

print("The logistic regression objective for a large x is");
x = 10000*np.random.randn(Nf,1)
#Calling on a large input to verify that we don't get any NaNs
y = logreg_objective(x,D,c)
print(y)

input("Press Enter to continue...")

\end{verbatim}

\section*{Solution to Q2(a): to load the dataset}
\begin{verbatim}

def load_mnist():
    import h5py
    f = h5py.File("mnist.h5")
    x_test = f["x_test"]
    x_train = f["x_train"]
    y_test = f["y_test"]
    y_train = f["y_train"]

    with x_test.astype("float"):
        x_test = x_test[:]
    with y_test.astype("float"):
        y_test = y_test[:]
    with x_train.astype("float"):
        x_train = x_train[:]
    with y_train.astype("float"):
        y_train = y_train[:]
    return x_test,x_train,y_test,y_train

x_test,x_train,y_test,y_train = load_mnist()

\end{verbatim}

\section*{Solution to Q2(b):to implement SoftPlus}
\begin{verbatim}

def softplus(X):
    y = np.zeros(X.shape)
    idxN, idxP = X<0, X>=0
    xn,xp = X[idxN],X[idxP]
    y[X<0] = [math.log(1+np.exp(x)) for x in xn]
    y[X>=0] = xp+[math.log(np.exp(-x)+1) for x in xp];
    return y

\end{verbatim}
\section*{Solution to Q2(c):to implement cross-entropy}
\begin{verbatim}

def cross_entropy(X,Y):
    Xm = X - np.max(X,axis = 1, keepdims = True)
    softmax = np.divide(np.exp(Xm),np.sum(np.exp(Xm),axis = 1,keepdims = True))
    y = -math.log(np.mean(np.sum(softmax*Y,axis=1,keepdims = True)))
    return y

\end{verbatim}

\section*{Solution to Q2(d): to implement neural  loss}
\begin{verbatim}

def net_objective(W,D,L):
    temp = D
    for i in range(0,len(W)):
        temp = softplus(np.dot(temp,W[i]))
    y = cross_entropy(temp,L);
    return y

\end{verbatim}

\section*{Solution to Q2(e): to evaluate objective function on MNIST data}

\begin{verbatim}

D = x_train
W = [None] * 2
W[0] = np.random.randn(784,100)
W[1] = np.random.randn(100,10)
y_train += np.ones((60000,))

# One HOT Encoding
s = pd.Series(y_train)
L = pd.get_dummies(s).values

print("""The objective value using the random weights and MNIST
      training data is {r:5.3f}""".format(r=net_objective(W,D,L)))

input("Press Enter to continue...")

\end{verbatim}

\section*{Solution to Q3(a): to compute image gradient}

\begin{verbatim}
def grad2d(X):

    # Computing the x-differences
    kernel = np.zeros(X.shape)
    kernel[0,0] = 1
    kernel[0,1] = -1
    Dx = np.fft.fft2(kernel)
    Gx = np.fft.ifft2(Dx*np.fft.fft2(X))

    #Computing the y-differences
    kernel = np.zeros(X.shape)
    kernel[0,0] = 1
    kernel[1,0] = -1
    Dy = np.fft.fft2(kernel)
    Gy = np.fft.ifft2(Dy*np.fft.fft2(X))

    return np.stack([Gx, Gy], axis=2)
\end{verbatim}

\section*{Solution to Q3(b): to compute (-ve) divergence}

\begin{verbatim}
def div2d(G):
    # Computing the x-differences
    #  extract the x- and y- derivatives
    Gx = G[:,:,0]
    Gy = G[:,:,1]

# We must convolve with the FLIPPED stencil to get the linear filtering we want
    kernel = np.zeros(Gx.shape)
    kernel[0,0]=1
    kernel[0][-1] =-1

#  create diagonal matrix in decomposition K=F'DF
    Dx = np.fft.fft2(kernel)

#  Use the eigen-decomposition to convolve the stencil with X, and get the
#  differences in the horizontal direction.
    Divx = np.fft.ifft2(Dx*np.fft.fft2(Gx))

    kernel = np.zeros(Gy.shape)
    kernel[0,0]=1
    kernel[-1][0]=-1

    Dy = np.fft.fft2(kernel)

    Divy = np.fft.ifft2(Dy*np.fft.fft2(Gy))

    return Divx+Divy
\end{verbatim}
\section*{Solution to Q3(c): to verify adjointness using $<x,Ay> \quad = \quad <A'x,y>$ where A = G or d}

\begin{verbatim}

print('\nTesting adjoints\n')
# random inputs
x = np.random.randn(100,100);
y = np.random.randn(100,100,2)
Ay = grad2d(y)
x_Ay = np.sum(np.conj(x)*Ay)
#  Compute Ax, where A is the gradient operator
Ax = grad2d(x)
#  Compute Aty, where A is the gradient operator
Aty = div2d(y)

left = np.inner(Ax.flatten(),y.flatten(1))
right = np.inner(x.flatten(),Aty.flatten(1))
print('first inner product = \t',left,'\n')
print('second inner product = \t',right)

\end{verbatim}

\end{document}


%!TEX root = kPtx_paper.tex
\section*{Theory}

\subsection*{k-Space-Domain Subproblems}
We propose to compute a sparse
design matrix $\bm{W}$ that directly maps the Fourier transform of a desired spatial
excitation pattern to a vector of RF pulses that together produce the pattern, as:
\begin{equation}
\bm{b} = \bm{W} \mathcal{F}(\bm{d}),
\end{equation}
where $\bm{b}$ is a vector of concatenated multi-channel RF pulses with length $N_t  N_c$,
where $N_t$ and $N_c$ are the numbers of time points and channels, respectively,
and $\mathcal{F}(\bm{d})$ is a length-$N_s$ vector containing the
Fourier transform of the complex-valued target k-space excitation pattern $\bm{d}$.
$\bm{W}$ has dimensions $N_t  N_c \times N_s$.
Once the entries of $\bm{W}$ are computed,
the RF design problem can be instantaneously solved by a sparse matrix multiplication.
%without a full matrix inversion or an iterative algorithm.
Here we describe how to break down the problem of constructing $\bm{W}$
into a set of smaller independent subproblems that can be solved in parallel.

%Furthermore, in design problems involved iterative procedures, such as the phase updates in magnitude-least-squares pulse designs \cite{setsompop2008magnitude} and the regularization factor updates in power/roughness constrained pulse design, the sparsity of the $\bm{W}$ matrix makes the iterative process much more efficient in terms of memory and speed. In the work of Katscher et al \cite{katscher2003transmit}, parallel transmit pulse design was also formulated in the k-space domain. However, in Ref \cite{katscher2003transmit}, the matrix $\bm{W}$ was solved as a whole, which required constructing and inverting a large $\bm{S}^H\bm{S}$ matrix, which will be explained in the following text. Our proposed method finely divides the process of finding $\bm{W}$ into many independent instances, which allows the use of parallel computing to largely accelerate the computation. Furthermore, the $\bm{S}^H\bm{S}$ matrices of different instances are each small, and our proposed method efficiently constructs these matrices, which further accelerates the computation.

\par The columns of $\bm{W}$ can be solved independently,
based on the observation that if the desired k-space pattern is a delta function at a k-space location
$\vec{k}_j$ ($1\leq j \leq N_s$),
then only the $j$-th column $\bm{w}_j$ of $\bm{W}$ is required to relate
the desired k-space pattern to the desired RF pulse.
In this case, the vector $\bm{w}_j$ is identical to the desired RF.
Therefore, the problem of finding the $j$-th column $\bm{w}_j$ of $\bm{W}$ can be stated as:
what should the RF pulse samples be to generate a unit delta function at $\vec{k}_j$,
and zeros elsewhere?
Mathematically, the relationship that should be satisfied by the weights in column $\bm{w}_j$  can be expressed as:
\begin{equation}\label{eq:ScalerForm}
\delta(\vec{k}-\vec{k}_j)=\sum_{i=1}^{N_t}\sum_{c=1}^{N_c} w_c(\vec{k}_i) s_c(\vec{k}-\vec{k}_i),
\end{equation}
where $i$ indexes time points in the pulses' excitation k-space trajectory,
and $c$ indexes channels.
$w_c(\vec{k}_i)$ is the $((c-1)N_t+i)$-th element of the $\bm{w}_j$ vector,
and $s_c(\vec{k}-\vec{k}_i)$ is the Fourier transform of channel-$c$'s $B_1^+$ map,
shifted to be centered at excitation k-space location $\vec{k}_i$.
Equation \ref{eq:ScalerForm} can be restated in matrix-vector form as:
\begin{equation}\label{eq:MatrixForm}
\bm{Sw}_j=\bm{e}_j,
\end{equation}
where $\bm{S}$ is a $N_s \times N_c  N_t$
matrix containing values of $s_c(\vec{k}-\vec{k}_i)$ for all $c$ and $i$,
and $\bm{e}_j$ is a vector containing a one in the entry corresponding to excitation k-space location
$\vec{k}_j$, and zeros in all other entries.
The regularized pseudoinverse solution for $\bm{w}_j$ is:
\begin{equation}\label{eq:Solution}
\bm{w}_j=\left( \bm{S}^{H} \bm{S} + \lambda \bm{I} \right) ^{-1} \bm{s}_j^{H}
\end{equation}
where $\bm{s}_j^{H}$ is the conjugate transpose of the $j$-th row of $\bm{S}$.

\par If all points on the excitation k-space trajectory are considered in Equation \ref{eq:ScalerForm},
then there is no reduction in computation compared to a conventional pseudoinverse-based pulse design.
However, because $B_1^+$ maps are localized near $\vec{k} = 0$ in excitation k-space,
only a small number ($\ll N_t$) of trajectory points near the target location $\vec{k}_j$ can contribute significantly
to the excitation pattern at $\vec{k}_j$ or its neighbors,
and therefore need to be considered in the subproblem;
all other trajectory points can be ignored.
This means Equation \ref{eq:ScalerForm} can be modified to:
\begin{equation}
\delta(\vec{k}-\vec{k}_j) = \sum_{\vec{k}_i \in \mathbb{N}}\sum_{c=1}^{N_c} w_c(\vec{k}_i) s_c(\vec{k}-\vec{k}_i),
\end{equation}
where $\mathbb{N}$ is the set of trajectory points in the vicinity of $\vec{k}_j$.
In this work `inclusion width' is defined as the distance,
in phase cycles over the excitation field-of-view (cycles/FOV),
from each target location $\vec{k}_j$ within which all trajectory points are accrued into $\mathbb{N}$.
This reduced set of trajectory points reduces the column dimension of $\bm{S}$
and both dimensions of $\bm{S}^H\bm{S}$,
and implies that the design matrix $\bm{W}$ is sparse.
Figure \ref{fig:Patch}a illustrates a target point $\vec{k}_j$ for a two-dimensional pulse design,
and the nearby points on the trajectory that need to be included in the solution for $\bm{w}_j$
for an inclusion width of 4 cycles/FOV.

%\par A key advantage of the proposed pulse design formalism is that it can be finely parallelized
%across the columns of the $\bm{W}$ matrix.
\par The $\bm{W}$ matrix columns $\bm{w}_j$ can be solved in parallel one-at-a-time or patch-wise,
wherein all target points in a patch share the same $\bm{S}^{H}\bm{S}$ matrix,
and their $\bm{s}_j^H$ vectors are concatenated in the column dimension
to form a matrix in Equation \ref{eq:Solution}.
An illustration of a $4 \times 4$ patch and its included trajectory points is shown in Figure \ref{fig:Patch}b,
for a two-dimensional spiral pulse design.
The inclusion width is also 4 cycles/FOV from any target point in the patch,
%we truncate the $B_1^+$ map Fourier transforms to zero outside a 4 by 4 cycle/FOV square centered at DC.
so all the excitation trajectory points within a $12 \times 12$ cycles/FOV square are included in the calculation.

%Since target locations that are distant to $\vec{k}_j$ can also be ignored,
%the row dimension of $\bm{S}$ is also reduced to the size of a smaller neighborhood immediately surrounding $\vec{k}_j$.
%These reduced problem dimensions further imply that the design matrix $\bm{W}$ is sparse.
%Figure \ref{fig:Patch}a illustrates a target point $\vec{k}_j$,
%the nearby points on a spiral trajectory that need to be included in the solution for $\bm{w}_j$,
%and the neighboring target location points where a zero excitation must be enforced.
%In the following, we define target and trajectory neighborhood sizes,
%and describe an efficient method to construct the $\bm{S}^{H}\bm{S}$ matrices.

\subsection*{Efficient $\bm{S}^H\bm{S}$ matrix construction}
%%which is 4 cycle/FOV extended outside of the patch width 4 square, since they may contribute energy to the target locations corresponding to the 16 $\bm{W}$ matrix columns.
%For the patch illustrated in Figure \ref{fig:Patch},
%the 46 green points on the excitation trajectory may contribute significant energy and are considered in the weight design.
%Furthermore, we must constrain the total excitation to zero at all k-space locations within a 20 by 20 cycle/FOV red patch in Figure \ref{fig:Patch}.
%This is because any energy deposited at each considered excitation trajectory points will affect not only the target points in the yellow square, but also the whole 2$\times$4 wide patch around that trajectory point. And the 20 by 20 cycle/FOV red patch is the total of the points that can be affected by all the excitation trajectory points considered in this instance. Therefore, in the solution for this instance, $\bm{w}$ is a 46$N_c\times$16 matrix, $\bm{S}$ is a 400$N_c\times$46$N_c$ matrix, $\bm{S}^{H}\bm{S}$ is a 46$N_c\times$46$N_c$ matrix, and $\bm{s}_i^{H}$ is a 46$N_c\times$16 matrix.
%%Since $\bm{s}_i^{H}$ is no longer related to the $i$th row of $\bm{S}$ and $i$th column of $\bm{W}$ only, it will be denoted as $\bm{S_{targ}}$ in later discussions.
%In the next instance, another 16 columns centered around another 4 points by 4 points patch will be solved. It should be noted that in different instances, the number of neighboring excitation trajectory points (the 46 green points in the previous instance) can vary. We will denote this number of points by $N_n$. $N_n$ can be zero when a patch is at the corners where no excitation trajectory point is adjacent.
%%For patches near edges of the excitation k-space FOV, the solution neighborhood and area may wrap back in a circulant shift fashion, depending on the resolution of the final spatial grid on which the user will evaluate the pulse. In Figure \ref{fig:Patch}), denoted by the red area to the right of Figure \ref{fig:Patch}).

%\par The patch width determines the total number of instances to be solved. The inclusion width determines the accuracy of $B_1^+$ information utilized in the design. Together, the patch width and the inclusion width determine how many excitation trajectory points are included in each instance, which further determines the size of $\bm{S}^{H}\bm{S}$ whose construction is most computationally burdensome within an instance. Therefore, increasing patch width reduces the number of $\bm{S}^{H}\bm{S}$ matrices to be constructed but increases the burden of constructing each of them. An optimal patch width should be decided case by case for different excitation trajectories.

%\subsection*{Efficient construction of $\bm{S}^{H}\bm{S}$ matrix}
\par Constructing the $\bm{S}^{H}\bm{S}$ matrix by first constructing $\bm{S}$ is computationally expensive
because it requires each channel's $B_1^+$ map to be shifted to each trajectory location $\vec{k}_i$
by phase modulation followed by Fourier transform, or by Fourier transform followed by interpolation.
Furthermore,
the matrix multiplication between $\bm{S}^{H}$ and $\bm{S}$ is also computationally intensive due to the potentially
large row dimension of $\bm{S}$, if the zero excitation condition is enforced on a large grid of points around each $\vec{k}_j$.
The computational costs can be reduced by truncating the $B_1^+$ maps in k-space,
but this introduces large errors.
Instead, inspired by the non-Cartesian GRAPPA method of Ref \cite{luo2019grappa},
we propose to construct the Hermitian $\bm{S}^{H}\bm{S}$ matrix directly via interpolation of the Fourier transforms
of inner products of pairs of $B_1^+$ maps.
The $\bm{S}^H\bm{S}$ matrix can be written in block form as:
\begin{equation}\label{eq:SHS_blocks}
\bm{S}^{H}\bm{S} =
\begin{pmatrix}
\bm{S}_1^{H}\bm{S}_1 & \cdots & \bm{S}_1^{H}\bm{S}_{N_c} \\
\vdots  &  \ddots & \vdots  \\
\bm{S}_{N_c}^{H}\bm{S}_1 & \cdots & \bm{S}_{N_c}^{H}\bm{S}_{N_c}
\end{pmatrix},
\end{equation}
where each sub-matrix $\bm{S}_i^H\bm{S}_j$ contains inner products of the Fourier transforms of channel $i$ and channel $j$'s
$B_1^+$ maps, after they are shifted to be centered different points on the excitation trajectory corresponding to the row
and column indices of the sub-matrix.
These inner products can be calculated by interpolating the Fourier transform of the product of
channel $i$ and channel $j$'s $B_1^+$ maps.
In other words, each element in the Hermitian matrix $\bm{S}_i^{H}\bm{S}_j$
is the vector sum of channel $i$ and channel $j$'s shifted k-space $B_1^+$ maps,
which is also a single point of their k-space convolution.
Via the multiplication property of the Fourier transform,
this convolution in k-space can be calculated from the Fourier transform of the product of the two spatial domain $B_1^+$ maps.
For further details, please refer to Ref. \cite{luo2019grappa}.
%In practice, $\bm{S}^{H}\bm{S}$ is constructed in the following way.
%We first represent the $\bm{S}^{H}\bm{S}$ matrix with block matrices. Each block matrix $\bm{S}_i^{H}\bm{S}_j$ is of size $N_n$ by $N_n$, and $\bm{S}_i$ and $\bm{S}_j$ are the columns of $\bm{S}$ related to the k-space sensitivity maps of the $i$-th and $j$-th coils respectively.

\par Since $\bm{S}^{H}\bm{S}$ is Hermitian,
we only need to find the values within the lower triangle of the matrix,
which are the sub-matrices $\bm{S}_i^{H}\bm{S}_j$ with $i>j$. % that are themselves lower triangular.
The elements of a given sub-matrix $\bm{S}_i^{H}\bm{S}_j$
are interpolated from a densely sampled Fourier transform of the product of the spatial domain $B_1^+$ maps of the $i$-th and $j$-th channels.
The $(m,n)$-th element is the value interpolated at a position $\vec{k}_m-\vec{k}_n$ away from DC,
where $\vec{k}_m$ and $\vec{k}_n$ are the excitation k-space trajectory locations corresponding to row $m$ and column $n$.
Compared to phase-modulating and Fourier transforming each channel's $B_1^+$ map for each point
on the trajectory which would require $N_t  N_c$ Fourier transforms,
to solve all the columns of $\bm{W}$ the total number of Fourier transforms needed in this efficient method is $N_c(N_c+1)/2$.
The storage size is also much smaller,
and the matrix multiplication between $\bm{S}^H$ and $\bm{S}$ is avoided.
%An alternative strategy would be to construct $\bm{S}$ by Fourier transforming each channel's $B_1^+$ map
%and then interpolating it to be centered at each trajectory location,
%but this again requires multiplication of the $\bm{S}^{H}$ and $\bm{S}$ matrices,
%which further involves the multiplication of interpolated values.
%We have found empirically that this strategy introduces higher error
%than interpolating the inner product values.
%Furthermore, to calculate each element of $\bm{S}^{H}\bm{S}$, there are $2N_s$ number of interpolations involved with this sub-optimal strategy. While with the efficient algorithm there is only one interpolation from the densely sampled Fourier Transform of the dot product of two spatial domain $B_1^+$ maps needed. The extra interpolations involved in the sub-optimal strategy would introduce a larger error to $\bm{S}^{H}\bm{S}$.

\subsection*{Off-Resonance Compensation}
% In order to model the off-resonance into our k-space domain design, we adapted a fast implementation of the off-resonance compensation with time segmentation approximation and weighted least-squares interpolators \cite{fessler2005toeplitz}.
Off-resonance compensation is common in spatial-domain parallel pulse designs \cite{Grissom:2006:MRM},
but has not been previously implemented in a k-space-domain design.
Spatial domain designs based on NUFFTs make use of an approximate time-segmented
model for the off-resonance term \cite{fessler2005toeplitz},
which decouples the space- and time-dependence of off-resonant phase accrual into the product of
space- and time-dependent terms, which are applied on either side of the NUFFT operator.
Here we show how a time-segmented off-resonance model can be incorporated in the proposed k-space-domain algorithm.
Mathematically, the time-segmented model is given by:
\begin{equation}
e^{\imath \Delta\omega(\vec{x}_j) (t_i-T) }\approx\sum_{l=1}^{L} b_l(t_i) h_{l}(\vec{x}_j) ,
\end{equation}
where $\Delta\omega(\vec{x}_j)$ is the off-resonance frequency at $\vec{x}_j$,
$i$ indexes time points in the pulse,
$T$ is the pulse duration,
and $l$ indexes the $L$ time segments, where $L$ is typically between 4 and 8.
The $b_{l}(t_i)$ functions are temporal interpolators that multiply into the RF samples and are determined by
numerical optimization as described in Ref. \cite{fessler2005toeplitz},
and $h_{l}(\vec{x}_j) = e^{\imath \Delta\omega(\vec{x}_j)\breve{t}_l}$ are the off-resonance phase shifts
applied to the excitation patterns produced by the RF pulse time segments centered at time points $\{\breve{t}_l: l = 1, \dots, L\}$.
The time-segmented off-resonance model can be incorporated into Equation \ref{eq:ScalerForm} as:
\begin{equation}
	\delta(\vec{k}-\vec{k}_j)=\sum_{l = 1}^L \sum_{i=1}^{N_t}\sum_{c=1}^{N_c} w_c(\vec{k}_i) b_l(t_i) \tilde{s}_{cl} (\vec{k}-\vec{k}_i),
\end{equation}
where $\tilde{s}_{cl}(\vec{k})$ is the Fourier transform of the product of channel $c$'s $B_1^+$ map and time segment $l$'s
off-resonance phase shift $h_l(\vec{x}_j)$.
We can then write the equivalent of Equation \ref{eq:MatrixForm} with the off-resonance model:
\begin{equation}
	\sum_{l=1}^L \bm{S}_l \textrm{diag}(b_{l}(t_i)) \bm{w}_j = \bm{e}_j,
\end{equation}
where the matrix $\bm{S}_l$ is now formed from the $N_c  L$ Fourier transforms of the $B_1^+$ map and $h_l(\vec{x}_j)$ products.
Efficient construction of the $\bm{S}^{H}\bm{S}$ matrix follows much the same procedure as without
off-resonance except that with off-resonance,
the elements of each sub-matrix $\bm{S}_i^{H}\bm{S}_j$ in Equation \ref{eq:SHS_blocks}
are interpolated from $L(L+1)/2$ Fourier transformed maps,
and the interpolated values are weighted by the temporal interpolators for each pair of multiplied time segments before they
are summed.
Compared to constructing $\bm{S}^H\bm{S}$ without off-resonance,
taking off-resonance into account increases the total number of FFT operations form $N_c(N_c+1)/2$ to $N_cL(N_cL+1)/2$,
and increases the number of interpolations needed for each sub-matrix element from 1 to $L(L+1)/2$.
However, we note that only adjacent time segments with overlapping temporal interpolators at a given time point
need to be considered when calculating the elements of $\bm{S}^H\bm{S}$ corresponding to that time point.


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Sheet 3}

\subsection{Changes of coordinate system}

We denote by \(x^{\mu } = (x, y)\) and \(x^{\prime \mu } = (r, \theta)\). Then, we have the following Jacobian matrices:
%
\begin{subequations}
\begin{align}
  \pdv{x^{\mu }}{x^{\prime \nu }}  &= \left[\begin{array}{cc}
  \cos(\theta )  & -r \sin(\theta )  \\
  \sin(\theta )  & r \cos(\theta )
  \end{array}\right]  \\
  \pdv{x^{\prime \nu }}{x^{\rho }} &= \left[\begin{array}{cc}
  \displaystyle\frac{x}{\sqrt{x^2+y^2} } &\displaystyle \frac{y}{\sqrt{x^2+y^2} } \\
  \displaystyle -\frac{y}{x^2+y^2} &\displaystyle \frac{x}{x^2+y^2}
  \end{array}\right]
  =
  \left[\begin{array}{cc}
  \cos(\theta )  & \sin(\theta )  \\
  \displaystyle- \frac{\sin( \theta ) }{r} & \displaystyle \frac{\cos(\theta ) }{r}
  \end{array}\right]
\,,
\end{align}
\end{subequations}
%
which can be found by plain differentiation of the change of coordinates, recalling \((\arctan x)' = 1/ (1+x^2)\). Then, we can compute the product of these two matrices: it comes out to be
%
\begin{equation} \label{eq:inverse-jacobian-matrix-relation}
  \pdv{x^{\mu }}{x^{\prime \nu }} \pdv{x^{\prime \nu }}{x^{\rho }} = \delta^{\mu }_{\nu }
\,,
\end{equation}
%
since on the diagonal we get \(\cos^2(\theta ) + \sin^2(\theta ) = 1\), while on the off-diagonal terms we get a multiple of \(\sin(\theta ) \cos(\theta ) - \sin(\theta ) \cos(\theta ) =0\).

Note that relation \eqref{eq:inverse-jacobian-matrix-relation} is just the chain rule written in more generality: substituting the explicit coordinates for \(x^{\mu }\) and \(x^{\prime \mu }\) we get the desired expression.

\subsection{Properties of covariant differentiation}

\subsubsection{Metric compatibility of the connection}

We wish to show that \(\nabla_{\alpha } g_{\mu \nu } = 0\).
A very simple way to prove this is by going in the LIF: there, the equation reads \(\partial_{\alpha }\eta_{\mu \nu } = 0\), which is immediately satisfied since the components of the Minkowski metric are constants.
Then, since the equation is tensorial, the result extends to any frame.

This was not the spirit of the exercise, however: let us prove it in a different generic frame.
To this end, we define the \emph{Christoffel symbols of the first kind} (while the regular ones are of the second kind):
%
\begin{equation}
  \Gamma_{\mu \nu \rho } = g_{\mu \sigma }\Gamma^{\sigma }_{\nu \rho } = \frac{1}{2} \qty(g_{\mu \nu, \rho } + g_{\mu \rho , \nu } - g_{\nu \rho, \mu })
\,.
\end{equation}

These are useful since, as all their indices are down, it is easier to study their symmetry properties.

Note that if we symmetrize the first and last index, we get \(\Gamma_{(\mu | \nu | \rho)} = \frac[i]{1}{2} g_{\mu \rho, \nu }\) since the first and last terms in the sum cancel (in the latter we must invert the indices \(\nu \) and \(\rho \) in order to see this, but this can always be done by the symmetry of the metric).

Then, we write the expression for the covariant derivative of the metric:
%
\begin{equation}
  \nabla_{\alpha }g_{\mu \nu }= \partial_\alpha g_{\mu \nu }
  - \Gamma^{\rho }_{\mu \alpha }g_{\rho \nu }
  - \Gamma^{\rho }_{\nu \alpha  }g_{\mu \rho  }
  = g_{\mu \nu , \alpha }
  - \Gamma_{\nu \alpha \mu }
  - \Gamma_{\mu \alpha \nu }
\,,
\end{equation}
%
which is just \(g_{\mu \nu , \alpha } - 2 \Gamma_{(\nu | \alpha | \mu )} = g_{\mu \nu , \alpha } - g_{\nu \mu , \alpha } = 0\), again by the symmetry of the metric.

One could argue that this is the opposite way round: we should \emph{assume} \(\nabla_{\mu } g_{\rho \sigma } = 0   \) and derive from it the formula that was given for the Christoffel symbols in terms of the partial derivatives of the metric.

\subsubsection{Leibniz rule}

As before, this can be proved in the LIF from the Leibniz rule of regular partial derivatives.

As before, we like to calculate therefore we show this explicitly in any frame.

The derivative of the tensor product looks like:
%
\begin{equation}
    \nabla_{\mu } \qty(A_{\nu \lambda } B_{ \rho }) =
    \partial_{\mu }\qty(A_{\nu \lambda }B_{\rho })
    -\Gamma_{\mu \nu }^{\sigma }A_{\sigma \lambda }B_{\rho }
    -\Gamma_{\mu \lambda  }^{\sigma }A_{\nu \sigma }B_{\rho }
    -\Gamma_{\mu \rho }^{\sigma }A_{\nu \lambda }B_{\sigma }
\,,
\end{equation}
%
while the sum of derivatives looks like:
%
\begin{equation}
  \begin{split}
  B_{\rho }\nabla_{\mu }A_{\nu \lambda }
  + A_{\nu \lambda }\nabla_{\mu }B_{\rho }=
  &B_{\rho }\partial_{\mu }A_{\nu \lambda }
  -\Gamma_{\mu \nu }^{\sigma }A_{\sigma \lambda }B_{\rho }
  -\Gamma_{\mu \lambda  }^{\sigma }A_{\nu \sigma }B_{\rho } \\
  + & A_{\nu \lambda }\partial_{\mu }B_{\rho }
  -\Gamma_{\mu \rho }^{\sigma }A_{\nu \lambda }B_{\sigma }
  \end{split}
  \,,
\end{equation}
%
so we can see that the Christoffel terms are equal, and the partial derivative terms also are since we have the Leibniz rule for partial derivatives.

\subsection{2D Christoffel symbols}

\subsubsection{Polar coordinates}

The metric and inverse metric are respectively given by \(g_{\mu \nu } = \diag{1, r^2}\) and \(g^{\mu \nu }= \diag{1, r^{-2}}\). We only care about the partial derivatives of the lower-indices one, and the only nonvanishing derivative is \(g_{11,0} = 2r\), where we mean \((x^{0}, x^{1}) = (r, \theta )\).

Then it is tedious but straightforward to perform the direct computation.
Things that make it faster are discarding immediately terms which cannot contribute (such as \(g_{\alpha \beta ,\gamma }\) where at least one of \(\alpha \) and \(\beta \) is not 1 or \(\gamma \) is not \(0\), and only looking at the six independent symbols instead of the eight total ones (since \(\Gamma^{\alpha }_{01} = \Gamma^{\alpha }_{10}\) for any \(\alpha \)).

Then one can see that the nonvanishing symbols are
%
\begin{subequations}
\begin{align}
  \Gamma^{0}_{11} &= \frac{1}{2} g^{0\alpha}\qty(g_{\alpha 1, 1}  + g_{\alpha 1, 1} - g_{11, \alpha })
  = \frac{1}{2} g^{00}(-g_{11, 0}) = - \frac{2r}{2} = -r  \\
  \Gamma^{1}_{01} &= \frac{1}{2} g^{1 \alpha }\qty(g_{\alpha 0, 1} + g_{\alpha 1, 0} - g_{01, \alpha })
  = \frac{1}{2} g^{11} g_{11, 0} = \frac{1}{2} \frac{1}{r^2} 2r = \frac{1}{r}
\,.
\end{align}
\end{subequations}
%

\subsubsection{Spherical surface} \label{sec:spherical-surface-curvature}

Now we have the following metric and inverse metric:
%
\begin{equation} \label{eq:spherical-metric}
  g_{\mu \nu } = \left[\begin{array}{cc}
  R^2 & 0 \\
  0 & R^2 \sin^2(\theta )
  \end{array}\right]\,,
  \qquad
  g^{\mu \nu } = \left[\begin{array}{cc}
  R^{-2} & 0 \\
  0 & R^{-2}\sin^{-2}(\theta )
  \end{array}\right]
\,,
\end{equation}
%
but do note that \(R\) is a constant: given \((x^{0}, x^{1}) = (\theta, \varphi)\), we have as before that the only nontrivial derivative is \(g_{11,0} = 2 R^2  \sin(\theta ) \cos(\theta) \).

Then this case is exactly analogous to the previous one: the same symbols are zero, so we can skip almost all of the computation and jump straight to:
%
\begin{subequations} \label{eq:spherical-christoffel}
\begin{align}
  \Gamma^{0}_{11} &= -\frac{1}{2} g^{00}g_{11,0} = - \sin(\theta ) \cos(\theta ) \\
  \Gamma^{1}_{01} &= \frac{1}{2} g^{11}g_{11,0} = \frac{\cos(\theta ) } {\sin(\theta ) }
\,.
\end{align}
\end{subequations}

\subsection{Parallel transport}

We know from the last exercise the metric and Christoffel symbols of 2D space and of the surface of a sphere.

The equations of parallel transport are in general \(u^{\mu }\nabla_{\mu }V^{\nu } = 0\).

\subsubsection{Flat space}

We want to determine the behaviour of a vector field \(V^{\mu }(\theta )\) defined on a curve \(x^{\mu }(\theta ) = (R, \theta)\) with fixed \(R\), such that \(V^{ \mu } (\theta = 0 ) = (0,1/R)\) (a unit vector: \(V^{\mu } (0) V^{\nu }(0) g_{\mu \nu }(0) = 1\)).

In our case the tangent vector of the curve is \(u^{\mu }= (0,1)\). Therefore the equations simplify to:
%
\begin{equation}
  \nabla_{1}V^{\mu }= \partial_{1}V^{\mu }+\Gamma^{\mu }_{1 \alpha }V^{\alpha } = 0
\,,
\end{equation}
%
which are two coupled differential equations; we can make them explicit and substitute the Christoffel symbols found earlier.
%
\begin{subequations}
\begin{align}
  \nabla_{1}V^{0} &= \partial_1 V^{0} + \cancelto{}{\Gamma^{0}_{10}V^{0}} + \Gamma^{0}_{11}V^{1}
  = \partial_1 V^{0} - r V^{1}  \\
  \nabla_{1}V^{1} &= \partial_1 V^{1} + \Gamma^{1}_{10}V^{0} + \cancelto{}{\Gamma^{1}_{11}V^{1}}
  = \partial_{1}V^{1} + \frac{1}{r} V^{0}
\,,
\end{align}
\end{subequations}
%
so we can write this linear system like:
%
\begin{equation}\label{diffeqes3}
  \partial_{1} \left[\begin{array}{c}
  V^{0} \\
  V^{1}
  \end{array}\right]
  =
  \left[\begin{array}{cc}
  0 & r \\
  -r^{-1} & 0
  \end{array}\right]
  \left[\begin{array}{c}
  V^{0} \\
  V^{1}
  \end{array}\right]
\,.
\end{equation}
%

The eigenvalues of this matrix are \(\pm i\), so its exponential is a pure rotation matrix:
%
\begin{equation}
  \exp(\theta \left[\begin{array}{cc}
  0 & r \\
  -r^{-1} & 0
  \end{array}\right]) =
  \left[\begin{array}{cc}
  \cos(\theta )  & \sin(\theta )  \\
  -\sin(\theta )  & \cos(\theta )
  \end{array}\right] = R(-\theta )
\,,
\end{equation}
%
so the solution is
%
\begin{equation}
  V^{\mu }(\theta ) = R(-\theta )V^{\mu }(0)
\,.
\end{equation}

This means that our vector is rotating clockwise with unit angular velocity in the \((r, \theta )\) plane, just as the point it is defined at moves counterclockwise with unit angular velocity: therefore, if we look at the vector in Cartesian coordinates, we will see it always aligned with its initial direction and the same modulus.

Specifically, when \(\theta = \pi /2\) we get \(V^{\mu }=(1, 0)\): this has the same modulus as \((0,1/R)\) since we compute lengths with respect to the metric of our space, as \(\norm{V}^2 = V^{\mu } V^{\nu } g_{\mu \nu }\).

Another way to solve \eqref{diffeqes3} is to explicit from the second equation $V^0=-r\partial_1V^1$ and plug it into the second, so that it becomes
\begin{equation}
\frac{1}{r} \partial_1(-r\partial_1V^1)=V^1,
\end{equation}
which leads to the same result.
\subsubsection{Curved space: spherical surface}

Now our curve is \(x^{\mu } (\theta ) = (\theta , 0)\) in the spherical coordinates \(x^{\mu } = (\theta, \varphi )\). The metric is the one given in \eqref{eq:spherical-metric}, the Christoffel symbols are the ones given in \eqref{eq:spherical-christoffel}.

The parallel transport equations are now:
%
\begin{subequations}
\begin{align}
  \nabla_{0}V^{0} &= \partial_0 V^{0} + \cancelto{}{\Gamma^{0}_{00}V^{0}} + \cancelto{}{\Gamma^{0}_{01}V^{1} }  \\
  \nabla_{0}V^{1} &= \partial_0 V^{1} + \cancelto{}{\Gamma^{1}_{00}V^{0}} + \Gamma^{1}_{01}V^{1}
  = \partial_0 V^{1} + \frac{V^{1}}{\tan(\theta )}
\,.
\end{align}
\end{subequations}

The first equation gives us \(V^{1} = \const\), while we do not need to actually solve the second one: our initial condition is \(V^{\mu}(\theta = 0) = (R^{-1}, 0)\), and \(V^{1} \equiv 0\) is a solution to that first-order equation, so by the uniqueness we have found the whole solution.

Therefore the vector is constant in these coordinates.

Specifically, at \(\theta = \pi /2\) we get \(V^{\mu } = (1/R, 0)\).

\end{document}


$k$-nearest Neighbor was put forward by Cover and Hart in 1968\cite{knn}, is a classification algorithm, which is supervised and non-parametric. The problem, which K-NN solves, can be described as follows.

	  Given a labeled set $T=\{(x_i,y_i)|x_i\in\mathbb{R}^n, y_i\in C, i=1,2,\dots,N\}$, where $C$ is the set of categories, $C=\{c_1,c_2,\dots,c_s\}$. We need to find out the category of the new instance, $x$. We will denote $y$ as the category of $x$.

%	We will define ``neighbor" i.

\section{How to Define ``Neighbor"}
	To define ``neighbor", we introduce the definition of distance metrics in $\mathbb{R}^n$ firstly.
%	Because the labeled data instance $x_i\in\mathbb{R}^n$, here we only introduce the distance metrics in European space, $\mathbb{R}^n$.

	\begin{definition}
	Assume that $x_i, x_j\in\mathbb{R}^n$, and note that
	$$x_i=(x_i^{(1)},x_i^{(2)},\dots,x_i^{(n)})$$
		$$x_j=(x_j^{(1)},x_j^{(2)},\dots,x_j^{(n)})$$
		Then we define $L_p$ distance
		\begin{equation}
		L_p(x_i, x_j):=(\sum_{l=1}^{n} |x_i^{(l)}-x_j^{(l)}|^p)^{\cfrac{1}{p}}, p\geq 1
		\end{equation}

		i.e. when $p=2$, this distance is Euclidean distance,
		\begin{equation}
		L_2(x_i, x_j):=(\sum_{l=1}^{n} |x_i^{(l)}-x_j^{(l)}|^2)^{\cfrac{1}{2}}
		\end{equation}

	%	If $p=1$, it is Manhattan distance,
	%	\begin{equation}
	%	L_1(x_i, x_j):=\sum_{l=1}^{n} |x_i^{(l)}-x_j^{(l)}|
	%	\end{equation}

	%	If $p=\infty$, it is The maximum value of each coordinate distance,
	%	\begin{equation}
	%	L_{\infty}(x_i,x_j):=\max_{l} |x_i^{(l)}-x_j^{(l)}|
	%	\end{equation}
         \end{definition}

	    The figure \ref{alo:distance} shows that the points whose distance from the origin is 1 in $\mathbb{R}^n$.
	    \begin{figure}[htbp]
	    	\centering{\includegraphics[width=10cm]{distance.png}}
	    	\caption{Relationship between the different $L_p$ distance}
	    	\label{alo:distance}
	    \end{figure}

    Now we can define ``neighbor", i.e., given $p$, the nearest neighbor of $x_i$ is the point that has the minimal $L_p$ distance to $x_i$. It is worthy to note that the nearest neighbor determined under different distance metrics are generally different.

\section{$k$-NN Algorithm}

       $k$-NN predicts category of $x$ via the following Algorithm.

	\begin{algorithm}
    	\caption{$k$-NN}
    	\label{alo:K-NN}
    	\textbf{Input:}  Set $T=\{(x_i, y_i)|x_i\in\mathbb{R}^n, y_i\in C, i=1,2,\dots,N\}$, $x$ and $k$, where $C = \{c_{1}, c_{2}, \dots, c_{s}\}$. \\
    	\textbf{Output:} $y$ (the category of $x$)

    	\begin{algorithmic}[1]
         \State Find $k$ neighbors $x_j\in T, j=i_1,\dots,i_k$, i.e.
    	 \begin{align*}
    	 x_{i_1}&={\arg\min}_{x_i\in T} \|x-x_i\|_2,\\
    	 x_{i_m}&={\arg\min}_{x_i\in T\setminus \{\cup_{n<m}x_{i_n}\}} \|x-x_i\|_2, 1<m\leq k
    	 \end{align*}
    	 \State Define $y$ is the category of $x$,and find out it. $$y={\arg\max}_{c_l}\sum_{j=i_1,\cdots,i_k}I(y_j=c_l), l=1,2,\dots,s$$ where I is instruction function, that is, if $y_j=c_l$, then $I=1$, or $I=0$.
    	\end{algorithmic}
    \end{algorithm}

	\textbf{Shortcomings of $k$-NN}:
	  \begin{enumerate}
	  	\item It is sensitive to the local structure of the data, easy to over-fit the data.
	  	\item It is expensive to calculate the distance each time.
	  \end{enumerate}

	  To cut down the expense of computation, one classical method is the $\textbf{$k$-d tree}$(short for $k$-dimensional tree), where the $k$ is different with the $k$ used by above $k$-NN. In computer science, a $k$-d tree is a space-partitioning data structure for organizing points in a $k$-dimensional space. Next we introduce how to construct one $k$-d tree, which can represent and store the distance relation between points.

	 \begin{algorithm}
    	\caption{Construct Balanced $k$-d Tree}
    	\label{alo:kdtree}
    	\textbf{Input:} Set $T=\{x_i | x_i\in\mathbb{R}^k, i=1,2,\dots,N\}$, where $x_{i} = (x_{i}^{(1)}, x_{i}^{(2)}, \dots, x_{i}^{(k)})$. \\
    	\textbf{Output:} $k$-d Tree

    	\begin{algorithmic}[1]
	\State Find one rectangle region $\mathscr{R}_{0}$ to cover set $T$, i.e.
	\[\mathscr{R}_{0} = \{x| \min_{i} x_{i}^{(j)} < x^{(j)} < \max_{i} x_{i}^{(j)}\}\]
         Define $\mathscr{R}_{0}$ as the root node with $0$-depth.

    	 \State \textbf{Repeat}: for $m$-th $j$-depth node and $l = (j \ mod \ k) + 1$,

	 \begin{enumerate}[a]
	 \item If $\# (T \cap  \mathscr{R}_{j}^{m}) \nmid 2$,
	 $$ i_{0} = \arg median_{i}\{ x_{i}^{(l)}\}$$

	 use the hyperplane $x^{(l)} = x_{i_{0}}^{(l)}$ to divide $\mathscr{R}_{j}^{m}$ into two subregions as $\mathscr{R}_{j + 1}^{2m-1}$ and $\mathscr{R}_{j + 1}^{2m}$. Then store $x_{i_{0}}$ in the node $\mathscr{R}_{j}^{m}$.

	 \item If $\# (T \cap  \mathscr{R}_{j}^{m}) \mid 2$ and $\# (T \cap  \mathscr{R}_{j}^{m}) \neq 0$,
	 $$ \hat{x}^{(l)} = median_{i}\{ x_{i}^{(l)}\}$$

	 similarly, the hyperplane $x^{(l)} = \hat{x}^{(l)}$ to divide $\mathscr{R}_{j}^{m}$ into two subregions as $\mathscr{R}_{j + 1}^{2m-1}$ and $\mathscr{R}_{j + 1}^{2m}$.

	 \item If $\# (T \cap  \mathscr{R}_{j}^{m}) \mid 2$ and $\# (T \cap  \mathscr{R}_{j}^{m}) = 0$, no partition will be exceeded.

	 Collect all $\mathscr{R}_{j + 1}^{l}$ to get (j + 1)-depth nodes.
  	 \end{enumerate}
	 \textbf{Stop} until
	 \[ \sum_{m} \# (T \cap  \mathscr{R}_{j + 1}^{m}) = 0 \]
    	\end{algorithmic}
    \end{algorithm}

    Now we import an example using $k$-d tree on 2-d set to show the specific details of $\textbf{Algorithm} \  \ref{alo:kdtree}$.

    Given the set $T = \{x_{i}| i = 1: 7\} = \{(1, 1), (1, 5), (2, 9), (3, 3), (5, 2), (6, 7), (8, 4)\}$(Fig $\ref{alo:kdtree}$) as following. At the very beginning, we choose the 1-th coordinate to get the first partition plane $x_{1} = 3$ for the reason that the median of $\{1, 1, 2, 3, 5, 6, 8\}$ is 3. Here we left the following details to our reader to complete the whole algorithm. The $k$-d tree can be drown as one visual tree%(Fig $\ref {alo:kdtree}$).

    %%%

	\begin{figure}[htbp]
	\centering{\includegraphics[width=10cm]{kdtree.png}}
	\caption{Result of $k$-d Tree}
	\label{alo:kdtree}
	\end{figure}
    %%%%%%%%%%%%

%%%%%%%%%%%
\begin{tikzpicture}[line width = 1pt,
                    solid/.style = {circle, draw, fill = black, minimum size = 0.3cm},
                    empty/.style = {circle, draw, fill = white, minimum size = 0.3cm}]
%%%%%

\node [empty, label = right:$x_4$] (A) at (4,2) {};
\node [empty, label = right:$x_{2}$] (B) at (2,1) {};
\node [solid, label = right:$x_{7}$] (C) at (6,1) {};
\node [empty, label = right:$x_{1}$] (D) at (1,0) {};
\node [solid, label = right:$x_{3}$] (E) at (3,0) {};
\node [empty, label = right:$x_{4}$] (F) at (5,0) {};
\node [solid, label = right:$x_{6}$] (G) at (7,0) {};

\draw (A) -- (B);
\draw (A) -- (C);
\draw (B) -- (D);
\draw (B) -- (E);
\draw (C) -- (F);
\draw (C) -- (G);

\end{tikzpicture}

    $k$-d tree can save lots of computation in $k$-NN. We briefly give the recipe of $k$-d tree method of finding the nearest neighbor of one new instance, $x$.

    	 \begin{algorithm}
    	\caption{Using $k$-d Tree to Find the Nearest Neighbor}
    	\label{alo:kdtree}
    	\textbf{Input:} $k$-d tree of data set and new instance, $x$ \\
    	\textbf{Output:} Nearest neighbor

    	\begin{algorithmic}[1]
	\State Find the leaf node that contains $x$.

	\textbf{Downward retrieval}: from the root node, according to information stored in the nodes, such as the corresponding partition hyperplane, to move our searching road to next depth right or left child node. Repeat the above process until arriving one leaf node, say $x^{*}$, which named as the current nearest neighbor.

	\State{\textbf{Upward check and Repeat}}: for one current nearest neighbor, $x^{*}$, whose depth is j,

	\begin{enumerate}[(1)]
	\item Compute
	$$d(x^{*}, x) = ||x^{*} - x||$$
	\item Backward to $x^{*}$'s parent node, say $x_{j - 1}$.
	\item Check the other child node of $x_{j - 1}$, say $x^{**}$, if $d(x^{**}, x) \leq d(x^{*}, x) $, let $x^{*} = x^{**}$.
	\end{enumerate}

         \textbf{Stop}: j = 0.
    	\end{algorithmic}
    \end{algorithm}

\section{The Choice of Parameter $k$}
    The choice of $k$ has significant impact on the results of $k$-NN.

    If we choose a smaller $k$, the forecast results are usually very sensitive to the neighboring instance points. In particular, the neighboring instance points may be noise. For one exact example, we have one labeled set in $\mathbb{R}^{2}$, which has two categories drawn in colors, `green' and `blue', and one red instance waiting its category. It's obvious that the red instance should better be grouped into the green class. When choosing a small $k$, we will get the bad result, `blue'.  By contrast, choosing a larger $k$, the labeled instances, which are distant from the target, will make unnecessary contribution. Then $k$-NN will be fail again. As the figure $\ref{alo: large k}$ shows, if we select the whole labeled points, then we will get category `blue' instead of `green'.

	\begin{figure}[htbp]
	    \centering{\includegraphics[width=10cm]{small_k.png}}
	    \caption{One example of choosing a smaller $k$}
	    \label{alo:small k}
        \end{figure}

       \begin{figure}[htbp]
	\centering{\includegraphics[width=10cm]{large_k.png}}
	\caption{One example of choosing a larger $k$}
	\label{alo: large k}
       \end{figure}

    In general, the choice of $k$ depends on the data. One can use Cross-Validation method to find an optimal $k$.

    \begin{algorithm}
    	\caption{Find the Optimal $k$}
    	\label{alo:find the optimal k}
    	\textbf{Input:}  labeled set $D=\{(x, y)|x \in\mathbb{R}^n, y \in C, i=1,2,\dots, N\}$, where $C = \{c_{1}, c_{2}, \dots, c_{s}\}$.\\
    	\textbf{Output:} $k$

    	\begin{algorithmic}[1]
         \State Divide data from $D$ into two parts: labeled set $T$ and validation set $Y$, i.e., which are noted as follows,   	 $$T = \{(x_{i}, y_{i})|x_{i} \in\mathbb{R}^{n}, y_{i} \in C, i = 1, 2, \dots, N_{1}\}$$
         $$Y = \{(\hat{x}_{j}, \hat{y}_{j})|\hat{x}_{j} \in\mathbb{R}^{n}, \hat{y}_{j} \in C, j = 1, 2, \dots, N_{2}\}$$
    	 \State Initialize $k = 2$. For $j = 1, 2, \dots, N_{2}$, use the Algorithm $k$-NN on labeled set $T$, then get category label $\tilde{y}_{j}$.\\
	 Define $O^{k}$,
	 $$O^{k} = \sum_{j = 1}^{N_{2}} I(\tilde{y}_{j} = \hat{y}_{j})$$
	 Update $k = k + 1$
    	 \State Stop until $k = N$,
	 $$k = {\arg\max}_{k} O^{k}$$
    	\end{algorithmic}
    \end{algorithm}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Orders of Growth}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Various exercises relating to orders of growth.
\end{abstract}
\maketitle

\begin{exercise}
Use orders of growth to identify dominant terms and compute limits.
\begin{itemize}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{e^x + x^2}{3 e^x + x^3} = \answer{\frac{1}{3}} \]
The dominant terms are
\begin{multipleChoice}
\choice[correct]{$e^x$ in the numerator and $3 e^x$ in the denominator}
\choice{$e^x$ in the numerator and $x^3$ in the denominator}
\choice{$x^2$ in the numerator and $3 e^x$ in the denominator}
\choice{$x^2$ in the numerator and $x^3$ in the denominator}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow 0} \frac{e^x + x^2}{3 e^x + x^3} = \answer{\frac{1}{3}} \]
The dominant terms are
\begin{multipleChoice}
\choice[correct]{$e^x$ in the numerator and $3 e^x$ in the denominator}
\choice{$e^x$ in the numerator and $x^3$ in the denominator}
\choice{$x^2$ in the numerator and $3 e^x$ in the denominator}
\choice{$x^2$ in the numerator and $x^3$ in the denominator}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{x + x^2}{3 x^4 + x^3} = \answer{0} \]
The dominant terms are
\begin{multipleChoice}
\choice{$x$ in the numerator and $3 x^4$ in the denominator}
\choice{$x$ in the numerator and $x^3$ in the denominator}
\choice[correct]{$x^2$ in the numerator and $3 x^4$ in the denominator}
\choice{$x^2$ in the numerator and $x^3$ in the denominator}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow 0} \frac{x + x^2}{3 x^4 + x^3} = \answer{\infty} \]
The dominant terms are
\begin{multipleChoice}
\choice{$x$ in the numerator and $3 x^4$ in the denominator}
\choice[correct]{$x$ in the numerator and $x^3$ in the denominator}
\choice{$x^2$ in the numerator and $3 x^4$ in the denominator}
\choice{$x^2$ in the numerator and $x^3$ in the denominator}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{\sqrt{x^3 + 9} - x}{\sqrt{x^3 + 9} + x} = \answer{1} \]
\begin{multipleChoice}
\choice[correct]{$\sqrt{x^3+9}$ dominates $x$}
\choice{$x$ dominates $\sqrt{x^3+9}$}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{\sqrt{x^3 + 9x^2} - x}{\sqrt{x^3 - 9x^2} + x} = \answer{1} \]
(First identify whether the square-root term is dominant or not; when it is dominant, identify the dominant term \textit{inside} the square root and neglect the non-dominant term.)
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{ 2 x^3 + \sin x^5}{ 3 x^4 + \sin x^6} = \answer{0}. \]
\item Compute the limit
\[ \lim_{x \rightarrow 0} \frac{\sqrt{x + 1}}{\sqrt{x}} = \answer{\infty}. \]
\begin{multipleChoice}
\choice{$x$ dominantes $1$ as $x \rightarrow 0$}
\choice[correct]{$1$ dominates $x$ as $x \rightarrow 0$}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{ (\ln x)^{100}}{x^{0.01}} = \answer{0}. \]
\begin{multipleChoice}
\choice[correct]{$x^{0.01}$ dominates $(\ln x)^{100}$ as $x \rightarrow \infty$}
\choice{$(\ln x)^{100}$ dominates $x^{0.01}$ as $x \rightarrow \infty$}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow 0^+} x^{0.01} (\ln x)^{100} = \answer{\infty} \]
\begin{multipleChoice}
\choice[correct]{$x^{-0.01}$ dominates $(\ln x)^{100}$ as $x \rightarrow 0^+$}
\choice{$(\ln x)^{100}$ dominates $x^{-0.01}$ as $x \rightarrow 0^+$}
\end{multipleChoice}
\item Compute the limit:
\[ \lim_{x \rightarrow \infty} \frac{e^{\ln \ln x}+1}{x^2} = \answer{0} \]
(Use the laws of exponents to write $x^2 = e^{2 \ln x}$.)
\end{itemize}
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[GrowthOrdINF001]

Arrange the functions
\[ x^{x} \qquad \frac{e^{x}}{\ln{x}} \qquad \ln{x} \]
in order from least rate of growth to greatest rate of growth as \(x \rightarrow \infty\). Compare on the basis of magnitude rather than sign, i.e., if a function is negative, take its absolute value first.
\begin{multiplechoice}
\choice{\(\displaystyle x^{x} < \! \! < \frac{e^{x}}{\ln{x}} < \! \! < \ln{x}\)}
\choice{\(\displaystyle \ln{x} < \! \! < x^{x} < \! \! < \frac{e^{x}}{\ln{x}}\)}
\choice{\(\displaystyle \frac{e^{x}}{\ln{x}} < \! \! < \ln{x} < \! \! < x^{x}\)}
\choice{\(\displaystyle x^{x} < \! \! < \ln{x} < \! \! < \frac{e^{x}}{\ln{x}}\)}
\choice{\(\displaystyle \frac{e^{x}}{\ln{x}} < \! \! < x^{x} < \! \! < \ln{x}\)}
\choice[correct]{\(\displaystyle \ln{x} < \! \! < \frac{e^{x}}{\ln{x}} < \! \! < x^{x}\)}
\end{multiplechoice}
\begin{feedback}
General Remarks:
\begin{itemize} \item Higher powers of \(x\) grow faster at infinity than lower powers of \(x\).
\item As \(x \rightarrow \infty\), \(\ln x\) goes to infinity slower than \(x^p\) for any (presumably small) positive constant \(p\).
\item As \(x \rightarrow \infty\), \(e^x\) goes to infinity faster than \(x^n\) for any (presumably large) positive constant \(n\).
\item As \(x \rightarrow \infty\), \(x^x\) goes to infinity faster than any exponential of the form \(e^{cx}\) for any constant \(c\).
\end{itemize}
\end{feedback}

\end{question}

\begin{question}%%%%%[GrowthOrdINF002]

Arrange the functions
\[ \frac{e^{- x}}{\ln{x}} \qquad x^{3} \ln{x} \qquad e^{x} \]
in order from least rate of growth to greatest rate of growth as \(x \rightarrow \infty\). Compare on the basis of magnitude rather than sign, i.e., if a function is negative, take its absolute value first.
\begin{multiplechoice}
\choice[correct]{\(\displaystyle \frac{e^{- x}}{\ln{x}} < \! \! < x^{3} \ln{x} < \! \! < e^{x}\)}
\choice{\(\displaystyle e^{x} < \! \! < \frac{e^{- x}}{\ln{x}} < \! \! < x^{3} \ln{x}\)}
\choice{\(\displaystyle x^{3} \ln{x} < \! \! < e^{x} < \! \! < \frac{e^{- x}}{\ln{x}}\)}
\choice{\(\displaystyle \frac{e^{- x}}{\ln{x}} < \! \! < e^{x} < \! \! < x^{3} \ln{x}\)}
\choice{\(\displaystyle x^{3} \ln{x} < \! \! < \frac{e^{- x}}{\ln{x}} < \! \! < e^{x}\)}
\choice{\(\displaystyle e^{x} < \! \! < x^{3} \ln{x} < \! \! < \frac{e^{- x}}{\ln{x}}\)}
\end{multiplechoice}
\begin{feedback}
General Remarks:
\begin{itemize} \item Higher powers of \(x\) grow faster at infinity than lower powers of \(x\).
\item As \(x \rightarrow \infty\), \(\ln x\) goes to infinity slower than \(x^p\) for any (presumably small) positive constant \(p\).
\item As \(x \rightarrow \infty\), \(e^x\) goes to infinity faster than \(x^n\) for any (presumably large) positive constant \(n\).
\end{itemize}
\end{feedback}

\end{question}

\begin{question}%%%%%[GrowthOrdZERO001]

Arrange the functions
\[ \left( \ln \frac{1}{x} \right)^2 \qquad \frac{e^{- x}}{x^{3}} \ln{x} \qquad x^{3} e^{x} \]
in order from least rate of growth to greatest rate of growth as \(x \rightarrow 0^+\). Compare on the basis of magnitude rather than sign, i.e., if a function is negative, take its absolute value first.
\begin{multiplechoice}
\choice{\(\displaystyle \left( \ln \frac{1}{x} \right)^2 < \! \! < \frac{e^{- x}}{x^{3}} \ln{x} < \! \! < x^{3} e^{x}\)}
\choice[correct]{\(\displaystyle x^{3} e^{x} < \! \! < \left( \ln \frac{1}{x} \right)^2 < \! \! < \frac{e^{- x}}{x^{3}} \ln{x}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3}} \ln{x} < \! \! < x^{3} e^{x} < \! \! < \left( \ln \frac{1}{x} \right)^2\)}
\choice{\(\displaystyle \left( \ln \frac{1}{x} \right)^2 < \! \! < x^{3} e^{x} < \! \! < \frac{e^{- x}}{x^{3}} \ln{x}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3}} \ln{x} < \! \! < \left( \ln \frac{1}{x} \right)^2 < \! \! < x^{3} e^{x}\)}
\choice{\(\displaystyle x^{3} e^{x} < \! \! < \frac{e^{- x}}{x^{3}} \ln{x} < \! \! < \left( \ln \frac{1}{x} \right)^2\)}
\end{multiplechoice}
\begin{feedback}
General Remarks:
\begin{itemize} \item Lower powers of \(x\) grow faster as \(x \rightarrow 0^+\) than higher powers of \(x\).
\item As \(x \rightarrow 0^+\), \(-\ln x = \ln x^{-1}\) goes to \(\infty\) slower than \(x^{-p}\) for any (presumably small) positive \(p\).
\item As \(x \rightarrow 0^+\), \(e^{x} \rightarrow 1\) and so does not influence the growth rate.
\item As \(x \rightarrow 0^+\), \(e^{-x} \rightarrow 1\) and so does not influence the growth rate.
\end{itemize}
\end{feedback}

\end{question}

\begin{question}%%%%%[GrowthOrdZERO005]

Arrange the functions
\[ \frac{x^{3} e^{x}}{\ln{x}} \qquad \frac{e^{- x}}{x^{3}} \qquad \frac{e^{- x}}{x^{3} \ln{x}} \]
in order from least rate of growth to greatest rate of growth as \(x \rightarrow 0^+\). Compare on the basis of magnitude rather than sign, i.e., if a function is negative, take its absolute value first.
\begin{multiplechoice}
\choice{\(\displaystyle \frac{x^{3} e^{x}}{\ln{x}} < \! \! < \frac{e^{- x}}{x^{3}} < \! \! < \frac{e^{- x}}{x^{3} \ln{x}}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3} \ln{x}} < \! \! < \frac{x^{3} e^{x}}{\ln{x}} < \! \! < \frac{e^{- x}}{x^{3}}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3}} < \! \! < \frac{e^{- x}}{x^{3} \ln{x}} < \! \! < \frac{x^{3} e^{x}}{\ln{x}}\)}
\choice[correct]{\(\displaystyle \frac{x^{3} e^{x}}{\ln{x}} < \! \! < \frac{e^{- x}}{x^{3} \ln{x}} < \! \! < \frac{e^{- x}}{x^{3}}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3}} < \! \! < \frac{x^{3} e^{x}}{\ln{x}} < \! \! < \frac{e^{- x}}{x^{3} \ln{x}}\)}
\choice{\(\displaystyle \frac{e^{- x}}{x^{3} \ln{x}} < \! \! < \frac{e^{- x}}{x^{3}} < \! \! < \frac{x^{3} e^{x}}{\ln{x}}\)}
\end{multiplechoice}
\begin{feedback}
General Remarks:
\begin{itemize} \item Lower powers of \(x\) grow faster as \(x \rightarrow 0^+\) than higher powers of \(x\).
\item As \(x \rightarrow 0^+\), \(-\ln x = \ln x^{-1}\) goes to \(\infty\) slower than \(x^{-p}\) for any (presumably small) positive \(p\).
\item As \(x \rightarrow 0^+\), \(e^{x} \rightarrow 1\) and so does not influence the growth rate.
\item As \(x \rightarrow 0^+\), \(e^{-x} \rightarrow 1\) and so does not influence the growth rate.
\end{itemize}
\end{feedback}

\end{question}

\end{document}


\subsection{Kolmogorov space}

In a Kolmogorov (or \(T_0\)) space, for every pair of points there is a neighbourhood containing one but not the other.


\section*{Background}

\subsection*{Neural Radiance Fields}

Neural Radiance Fields (NeRF)~\cite{mildenhall2020nerf} are a recent technique for novel view synthesis. In order to represent a highly-detailed scene, NeRF models a scene as a continuous, volumetric field of varying density, which refracts light at varying degrees. This is based on traditional volumetric rendering techniques, based on the rendering equation:
\[
  I(r) = \int_{t_n}^{t_f} T(t, r) \sigma(r(t)) c(r(t), r_d)dt
\]
Where
\[
    T(t, r) = \exp{-\int_{t_n}^{t} \sigma(r(s))ds}
\].

Where $I(r)$ is the illumination along camera ray $r(t) = r_o + t r_d$. Neural radiance fields are able to accurately reconstruct high-frequency features by recovering $\sigma$, the density at a given point, and $c$, the view-dependent color at a given point by modelling them as MLPs with an additional encoding scheme that allows for differentiation between extremely close points. We evaluate the above equations by performing ray-marching and computing $T(j,r) = \Sigma -\exp\sigma_i c_i$, by partitioning the ray into evenly spaced bins and sampling randomly from within each bin.

There have been a significant number of extensions to NeRF, including optimizations on the encoding for differentiating positions in space~\cite{tancik2020fourfeat}, better sampling approaches~\cite{barron2021mipnerf}, and faster training~\cite{yu2021plenoxels}. This is only a small subset of NeRF variants, and a plethora exist which we do not list.

\subsection*{Dynamic Neural Radiance Fields}

Neural Radiance Fields were designed to only handle static scenes, and thus cannot accurately reconstruct scenes which contain movement, alternative lighting conditions, or other variations.
In order to model dynamic scenes, there have been two diverging approaches.

One subset of approaches directly model the transformation in the time domain, by directly modelling the function $\sigma(x,t)=f(x\in\mathbb{R}^3, t\in[0,1])$, which include works such as HyperNeRF~\cite{park2021hypernerf}, NeRFies~\cite{park2021nerfies}, and Space-Time Invariant Irradiance Fields~\cite{xian2021space}. By directly modelling the variation of the density, these methods are able to reconstruct large deformations in latent spaces and reconstruct a wide variety of transformations from a single radiance field, often allowing for transformations in some latent space that permit for deformation.

The other subset of approaches models dynamics directly as a form of translation. In this case, we are not able to directly move the objects inside the scene since we can only evaluate the NeRF at a given $x$, so instead we invert the translation, instead bending the rays as a function of time, which acts to warp the space being rendered. This allows for effective simulation, and can be post-processed to compute object translation if so desired. The formulation for density is thus better described as $\sigma(x,t)=f(x+\Delta(x,t))$. This formulation enforces a coherent canonical representation, while directly modelling movement, and has been shown to be able to reconstruct both synthetic with D-NeRF~\cite{pumarola2020dnerf} and real scenes in NR-NeRF~\cite{tretschk2021nonrigid}.

The pros of directly including time as a function in the MLP are that we are able to represent a much broader class of functions, in theory every frame may be fully distinct from the previous, but the canonical formulation lends itself to smoothness between frames. Our approach falls into the canonization category, since we are interested in accurately reconstructing smooth movement as opposed to generalizing over many classes of movement.

\subsection*{Bezier Curves}

Bezier curves refer to a class of functions defined as a polynomial parametrized by a set of
control points. They are most commonly used as cubic polynomials: $f(x) = ax^3 + bx^2 + cx + d$,
where x is the variable we are interested in interpolating over. The general formulation for
the Bezier basis functions is defined as $B^n(t) = \Sigma^n_{i=0}
{n \choose i} (1-t)^{n-i} t^i$ where n
is the degree of the Bezier polynomial. In order to give control of the Bezier curve, we
introduce "control points", which weighs different points along the curve differently:
$B^n(t) = \Sigma^n_{i=0} P_i {n \choose i} (1-t)^{n-i} t^i$, where $P_i\in\mathbb{R}^3$ for 3D
movement. For a more comprehensive guide on Bezier splines, we refer the reader to a more
\href{https://pomax.github.io/bezierinfo/index.html}{complete reference}.

\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{bezier_curve.png}
    \caption{
        Bezier Curves are a low dimensional representation of smooth interpolation between a few control points. We leverage them to represent smooth movement and produce a prior over movement. Credit to Wikipedia~\cite{bezier_diagram} for diagram.
    }
    \label{fig:my_label}
\end{figure}