\chapter{Power series and Taylor series}
Polynomials are very well-behaved functions,
and are studied extensively for that reason.
From an analytic perspective, for example, they are smooth,
and their derivatives are easy to compute.

In this chapter we will study \emph{power series},
which are literally ``infinite polynomials'' $\sum_n a_n x^n$.
Armed with our understanding of series and differentiation,
we will see three great things:
\begin{itemize}
	\ii Many of the functions we see in nature
	actually \emph{are} given by power series.
	Among them are $e^x$, $\log x$, $\sin x$.

	\ii Their convergence properties are actually quite well behaved:
	from the string of coefficients,
	we can figure out which $x$ they converge for.

	\ii The derivative of $\sum_n a_n x^n$
	is actually just $\sum_n n a_n x^{n-1}$.
\end{itemize}

\section{Motivation}
To get the ball rolling, let's start with
one infinite polynomial you'll recognize:
for any fixed number $-1 < x < 1$ we have the series convergence
\[ \frac{1}{1-x} = 1 + x + x^2 + \dots \]
by the geometric series formula.

Let's pretend we didn't see this already in
\Cref{prob:geometric}.
So, we instead have a smooth function $f \colon (-1,1) \to \RR$ by
\[ f(x) = \frac{1}{1-x}. \]
Suppose we wanted to pretend that it was equal to
an ``infinite polynomial'' near the origin, that is
\[ (1-x)\inv = a_0 + a_1 x + a_2 x^2 + a_3 x^3 + a_4 x^4 + \dots. \]
How could we find that polynomial,
if we didn't already know?

Well, for starters we can first note that by plugging in $x = 0$
we obviously want $a_0 =1$.

We have derivatives, so actually,
we can then differentiate both sides to obtain that
\[ (1-x)^{-2} = a_1 + 2a_2 x + 3a_3 x^2 + 4a_4 x^3. \]
If we now set $x = 0$, we get $a_1 = 1$.
In fact, let's keep taking derivatives and see what we get.
\begin{alignat*}{8}
	(1-x)^{-1} &{}={}& a_0 &{}+{}& a_1x &{}+{}& a_2x^2 &{}+{}& a_3x^3 &{}+{}& a_4x^4  &{}+{}& a_5x^5 &{}+{}& \dots \\
	(1-x)^{-2} &{}={}& && a_1 &{}+{}& 2a_2 x &{}+{}& 3a_3 x^2 &{}+{}& 4a_4 x^3 &{}+{}& 5a_5 x^4 &{}+{}&\dots \\
	2(1-x)^{-3} &{}={}& && && 2a_2 &{}+{}& 6a_3 x &{}+{}& 12 a_4 x^2 &{}+{}& 20 a_5 x^3 &{}+{}&  \dots \\
	6(1-x)^{-4} &{}={}& &&  &&&& 6a_3 &{}+{}& 24 a_4 x &{}+{}& 60 a_5 x^2 &{}+{}& \dots \\
	24(1-x)^{-5} &{}={}& && && && && 24 a_4 &{}+{}& 120 a_5 x &{}+{}& \dots \\
	&{}\vdotswithin={}.
\end{alignat*}
If we set $x=0$ we find $1 = a_0 = a_1 = a_2 = \dots$
which is what we expect;
the geometric series $\frac{1}{1-x} = 1 + x + x^2 + \dots$.
And so actually taking derivatives was enough to get the right claim!

\section{Power series}
\prototype{$\frac{1}{1-z} = 1 + z + z^2 + \dots$, which converges on $(-1,1)$.}
Of course this is not rigorous,
since we haven't described what the right-hand side is,
much less show that it can be differentiated term by term.
So we define the main character now.

\begin{definition}
	A \vocab{power series} is a sum of the form
	\[ \sum_{n = 0}^\infty a_n z^n
		= a_0 + a_1 z + a_2 z^2 + \dots \]
	where $a_0$, $a_1$, \dots\ are real numbers,
	and $z$ is a variable.
\end{definition}
\begin{abuse}
	[$0^0=1$]
	If you are very careful, you might notice
	that when $z=0$ and $n=0$ we find $0^0$ terms appearing.
	For this chapter the convention is that
	they are all equal to one.
\end{abuse}

Now, if I plug in a \emph{particular} real number $h$,
then I get a series of real numbers $\sum_{n = 0}^{\infty} a_n h^n$.
So I can ask, when does this series converge?
It terms out there is a precise answer for this.

\begin{definition}
	Given a power series $\sum_{n=0}^{\infty} a_n z_n$,
	the \vocab{radius of convergence} $R$ is defined
	by the formula
	\[ \frac 1R = \limsup_{n \to \infty} \left\lvert a_n \right\rvert^{1/n}. \]
	with the convention that $R = 0$ if the right-hand side is $\infty$,
	and $R = \infty$ if the right-hand side is zero.
\end{definition}
\begin{theorem}
	[Cauchy-Hadamard theorem]
	Let $\sum_{n=0}^{\infty} a_n z^n$
	be a power series with radius of convergence $R$.
	Let $h$ be a real number, and consider the infinite series
	\[ \sum_{n=0}^\infty a_n h^n \]
	of real numbers.
	Then:
	\begin{itemize}
		\ii The series converges absolutely if $|h| < R$.
		\ii The series diverges if $|h| > R$.
	\end{itemize}
\end{theorem}
\begin{proof}
	This is not actually hard,
	but it won't be essential, so not included.
\end{proof}
\begin{remark}
	In the case $|h| = R$, it could go either way.
\end{remark}
\begin{example}
	[$\sum z^n$ has radius $1$]
	Consider the geometric series $\sum_{n} z^n = 1 + z + z^2 + \dots$.
	Since $a_n = 1$ for every $n$, we get $R = 1$,
	which is what we expected.
\end{example}

Therefore, if $\sum_n a_n z^n$ is a power
series with a nonzero radius $R > 0$ of convergence,
then it can \emph{also} be thought of as a function
\[ (-R, R) \to \RR
	\quad\text{ by }\quad
	h \mapsto \sum_{n \ge 0} a_n h^n. \]
This is great.
Note also that if $R = \infty$,
this means we get a function $\RR \to \RR$.

\begin{abuse}
	[Power series vs.\ functions]
	There is some subtly going on with ``types'' of objects again.
	Analogies with polynomials can help.

	Consider $P(x) = x^3 + 7x + 9$, a polynomial.
	You \emph{can}, for any real number $h$,
	plug in $P(h)$ to get a real number.
	However, in the polynomial \emph{itself},
	the symbol $x$ is supposed to be a \emph{variable} ---
	which sometimes we will plug in a real number for,
	but that happens only after the polynomial is defined.

	Despite this, ``the polynomial $p(x) = x^3+7x+9$''
	(which can be thought of as the coefficients)
	and ``the real-valued function $x \mapsto x^3+7x+9$''
	are often used interchangeably.
	The same is about to happen with power series:
	while they were initially thought of as a sequence of
	coefficients, the Cauchy-Hadamard theorem
	lets us think of them as functions too,
	and thus we blur the distinction between them.
%	Pedants will sometimes \emph{define} a polynomial
%	to be the sequence of coefficients, say $(9,7,0,1)$,
%	with $x^3+7x+9$ being ``intuition only''.
%	Similarly they would define a power series
%	to be the sequence $(a_0, a_1, \dots)$.
%	We will not go quite this far, but they have a point.
%
%	I will be careful to use ``power series''
%	for the one with a variable,
%	and ``infinite series'' for the sums of real numbers from before.
\end{abuse}

\section{Differentiating them}
\prototype{We saw earlier $1+x+x^2+x^3+\dots$ has derivative $1+2x+3x^2+\dots$.}
As promised, differentiation works exactly as you want.

\begin{theorem}
	[Differentiation works term by term]
	Let $\sum_{n \ge 0} a_n z^n$ be a power series
	with radius of convergence $R > 0$,
	and consider the corresponding function
	\[ f \colon (-R,R) \to \RR \quad\text{by}\quad
		f(x) = \sum_{n \ge 0} a_n x^n. \]
	Then all the derivatives of $f$ exist and
	are given by power series
	\begin{align*}
		f'(x) &= \sum_{n \ge 1} n a_n x^{n-1} \\
		f''(x) &= \sum_{n \ge 2} n(n-1) a_n x^{n-2} \\
		&\vdotswithin=
	\end{align*}
	which also converge for any $x \in (-R, R)$.
	In particular, $f$ is smooth.
\end{theorem}
\begin{proof}
	Also omitted.
	The right way to prove it is to define the notion
	``converges uniformly'',
	and strengthen Cauchy-Hadamard to have
	this is as a conclusion as well.
	However, we won't use this later.
\end{proof}

\begin{corollary}
	[A description of power series coefficients]
	Let $\sum_{n \ge 0} a_n z^n$ be a power series
	with radius of convergence $R > 0$,
	and consider the corresponding function $f(x)$ as above.
	Then
	\[ a_n = \frac{f^{(n)}(x)}{n!}. \]
\end{corollary}
\begin{proof}
	Take the $n$th derivative and plug in $x=0$.
\end{proof}

\section{Analytic functions}
\prototype{The piecewise $e^{-1/x}$ or $0$ function is not analytic,
	but is smooth.}
With all these nice results about power series,
we now have a way to do this process the other way:
suppose that $f \colon U \to \RR$ is a function.
Can we express it as a power series?

Functions for which this \emph{is} true
are called analytic.
\begin{definition}
	A function $f \colon U \to \RR$ is \vocab{analytic} at
	the point $p \in U$
	if there exists an open neighborhood $V$ of $p$ (inside $U$)
	and a power series $\sum_n a_n z^n$ such that
	\[ f(x) = \sum_{n \ge 0} a_n (x-p)^n \]
	for any $x \in V$.
	As usual, the whole function is analytic
	if it is analytic at each point.
\end{definition}
\begin{ques}
	Show that if $f$ is analytic, then it's smooth.
\end{ques}
Moreover, if $f$ is analytic,
then by the corollary above its coefficients are
actually described exactly by
\[ f(x) = \sum_{n \ge 0} \frac{f^{(n)}(p)}{n!} (x-p)^n. \]
Even if $f$ is smooth but not analytic,
we can at least write down the power series;
we give this a name.
\begin{definition}
	For smooth $f$,
	the power series $\sum_{n \ge 0} \frac{f^{(n)}(p)}{n!} z^n$
	is called the \vocab{Taylor series} of $f$ at $p$.
\end{definition}

\begin{example}
	[Examples of analytic functions]
	\listhack
	\label{ex:nonanalytic}
	\begin{enumerate}[(a)]
		\ii Polynomials, $\sin$, $\cos$, $e^x$, $\log$
		all turn out to be analytic.

		\ii The smooth function from before defined by
		\[ f(x) = \begin{cases}
				\exp(-1/x) & x > 0 \\
				0 & x \le 0 \\
			\end{cases}
		\]
		is \emph{not} analytic.
		Indeed, suppose for contradiction it was.
		As all the derivatives are zero,
		its Taylor series would be $0 + 0x + 0x^2 + \dots$.
		This Taylor series does \emph{converge}, but not to the right value ---
		as $f(\eps) > 0$ for any $\eps > 0$, contradiction.
	\end{enumerate}
\end{example}

\begin{theorem}
	[Analytic iff Taylor series has positive radius]
	Let $f \colon U \to \RR$ be a smooth function.
	Then $f$ is analytic if and only if for any point $p \in U$,
	its Taylor series at $p$ has positive radius of convergence.
\end{theorem}

\begin{example}
	It now follows that $f(x) = \sin(x)$ is analytic.
	To see that, we can compute
	\begin{align*}
		f(0) &= \sin 0 = 0 \\
		f'(0) &= \cos 0 = 1 \\
		f''(0) &= -\sin 0 = 0 \\
		f^{(3)}(0) &= -\cos 0 = -1 \\
		f^{(4)}(0) &= \sin 0 = 0 \\
		f^{(5)}(0) &= \cos 0 = 1 \\
		f^{(6)}(0) &= -\sin 0 = 0 \\
		&\vdotswithin=
	\end{align*}
	and so by continuing the pattern
	(which repeats every four) we find the Taylor series is
	\[ z - \frac{z^3}{3!} + \frac{z^5}{5!} - \frac{z^7}{7!} + \dots \]
	which is seen to have radius of convergence $R = \infty$.
\end{example}

Like with differentiable functions:
\begin{proposition}
	[All your usual closure properties for analytic functions]
	The sums, products, compositions, nonzero quotients
	of analytic functions are analytic.
\end{proposition}
The upshot of this is is that most of your usual
functions that occur in nature,
or even artificial ones like $f(x) = e^x + x \sin(x^2)$,
will be analytic, hence describable locally by Taylor series.

\section{A definition of Euler's constant and exponentiation}
We can actually give a definition of $e^x$ using the tools we have now.

\begin{definition}
	We define the map $\exp \colon \RR \to \RR$ by
	using the following power series,
	which has infinite radius of convergence:
	\[ \exp(x) = \sum_{n \ge 0} \frac{x^n}{n!}. \]
	We then define Euler's constant as $e = \exp(1)$.
\end{definition}
\begin{ques}
	Show that under this definition, $\exp' = \exp$.
\end{ques}

We are then settled with:
\begin{proposition}
	[$\exp$ is multiplicative]
	Under this definition,
	\[ \exp(x+y) = \exp(x) \exp(y). \]
\end{proposition}
\begin{proof}
	[Idea of proof.]
	There is some subtlety here with switching
	the order of summation that we won't address.
	Modulo that:
	\begin{align*}
		\exp(x) \exp(y)
		&= \sum_{n \ge 0} \frac{x^n}{n!}
			\sum_{m \ge 0} \frac{y^m}{m!}
		= \sum_{n \ge 0} \sum_{m \ge 0}
			\frac{x^n}{n!} \frac{y^m}{m!} \\
		&= \sum_{k \ge 0} \sum_{\substack{m+n = k \\ m,n \ge 0}}
			\frac{x^n y^m}{n! m!}
		= \sum_{k \ge 0} \sum_{\substack{m+n = k \\ m,n \ge 0}}
			\binom{k}{n} \frac{x^n y^m}{k!} \\
		&= \sum_{k \ge 0} \frac{(x+y)^k}{k!} = \exp(x+y). \qedhere
	\end{align*}
\end{proof}

\begin{corollary}
	[$\exp$ is positive]
	\listhack
	\begin{enumerate}[(a)]
		\ii We have $\exp(x) > 0$ for any real number $x$.
		\ii The function $\exp$ is strictly increasing.
	\end{enumerate}
\end{corollary}
\begin{proof}
	First \[ \exp(x) = \exp(x/2)^2 \ge 0 \]
	which shows $\exp$ is nonnegative.
	Also, $1 = \exp(0) = \exp(x) \exp(-x)$ implies $\exp(x) \ne 0$
	for any $x$, proving (a).

	(b) is just since $\exp'$ is strictly positive (racetrack principle).
\end{proof}

The log function then comes after.
\begin{definition}
	We may define $\log \colon \RR_{>0} \to \RR$
	to be the inverse function of $\exp$.
\end{definition}
Since its derivative is $1/x$ it is smooth;
and then one may compute its coefficients to show it is analytic.

Note that this actually gives us a rigorous way to define
$a^r$ for any $a > 0$ and $r > 0$, namely
\[ a^r \defeq \exp\left( r \log a \right). \]

\section{This all works over complex numbers as well,
except also complex analysis is heaven}
We now mention that every theorem we referred to above
holds equally well if we work over $\CC$,
with essentially no modifications.
\begin{itemize}
	\ii Power series are defined by $\sum_n a_n z^n$ with $a_n \in \CC$,
	rather than $a_n \in \RR$.
	\ii The definition of radius of convergence $R$ is unchanged!
	The series will converge if $|z| < R$.
	\ii Differentiation still works great.
	(The definition of the derivative is unchanged.)
	\ii Analytic still works great for functions
	$f \colon U \to \CC$, with $U \subseteq \CC$ open.
\end{itemize}
In particular, we can now even define complex exponentials,
giving us a function \[ \exp \colon \CC \to \CC \]
since the power series still has $R = \infty$.
More generally if $a > 0$ and $z \in \CC$
we may still define \[ a^z \defeq \exp(z \log a). \]
(We still require the base $a$ to be a positive real
so that $\log a$ is defined, though.
So this $i^i$ issue is still there.)

However, if one tries to study calculus for complex functions
as we did for the real case,
in addition to most results carrying over,
we run into a huge surprise:
\begin{quote}
	\itshape
	If $f \colon \CC \to \CC$ is differentiable,
	it is analytic.
\end{quote}
And this is just the beginning of the nearly unbelievable
results that hold for complex analytic functions.
But this is the part on real analysis,
so you will have to read about this later!

\section{\problemhead}

\begin{problem}
	Find the Taylor series of $\log(1-x)$.
\end{problem}

\begin{dproblem}
	[Euler formula]
	Show that \[ \exp(i \theta) = \cos \theta + i \sin \theta \]
	for any real number $\theta$.
	\begin{hint}
		Because you know all derivatives of $\sin$ and $\cos$,
		you can compute their Taylor series,
		which converge everywhere on $\RR$.
		At the same time, $\exp$ was defined as a Taylor series,
		so you can also compute it.
		Write them all out and compare.
	\end{hint}
\end{dproblem}

\begin{dproblem}
	[Taylor's theorem, Lagrange form]
	Let $f \colon [a,b] \to \RR$ be continuous
	and $n+1$ times differentiable on $(a,b)$.
	Define
	\[ P_n = \sum_{k=0}^n \frac{f^{(k)}(b)}{k!} \cdot (b-a)^k. \]
	Prove that there exists $\xi \in (a,b)$ such that
	\[ f^{(n)}(\xi) = (n+1)! \cdot \frac{f(b) - P_n}{(b-a)^{n+1}}. \]
	This generalizes the mean value theorem
	(which is the special case $n = 0$, where $P_0 = f(a)$).
	\begin{hint}
		Use repeated Rolle's theorem.
		You don't need any of the theory in this chapter to solve this,
		so it could have been stated much earlier;
		but then it would be quite unmotivated.
	\end{hint}
\end{dproblem}

\begin{problem}
	[Putnam 2018 A5]
	\yod
	Let $f \colon \RR \to \RR$ be smooth,
	and assume that $f(0) = 0$, $f(1) = 1$, and $f(x) \ge 0$
	for every real number $x$.
	Prove that $f^{(n)}(x) < 0$ for some positive integer $n$
	and real number $x$.
	\begin{hint}
		Use Taylor's theorem.
	\end{hint}
\end{problem}


\hypertarget{functional-programming-1}{%
\section{Functional Programming 1}\label{functional-programming-1}}

\hypertarget{correctness}{%
\subsection{Correctness}\label{correctness}}

A program should be correct with respect to its specification. e.g.~a
program that computes the sine perfectly well but should compute the
root is clearly not correct.

One can know wether the program is correct

\begin{itemize}
\tightlist
\item
  by testing

  \begin{itemize}
  \tightlist
  \item
    choose particular input•determine correct result for that input
    using test oracle
  \item
    run program under test on the chosen input
  \item
    compare obtained and correct result
  \end{itemize}
\item
  by proving

  \begin{itemize}
  \tightlist
  \item
    no particular input
  \item
    no execution of the program
  \item
    instead apply mathematical rules to program and specification
  \item
    with a finite number of steps prove that something works for a
    infinite number of values
  \end{itemize}
\end{itemize}

\hypertarget{referential-transparency}{%
\subsection{Referential Transparency}\label{referential-transparency}}

\hypertarget{formal-proof}{%
\subsubsection{Formal proof}\label{formal-proof}}

\begin{figure}[H]
\centering
\includegraphics[width=0.25\textwidth]{figures/formalproof1.png}
\caption{Formal proof 1}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.25\textwidth]{figures/formalproof2.png}
\caption{Formal proof 2}
\end{figure}

\hypertarget{equality}{%
\subsubsection{Equality}\label{equality}}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/equality.png}
\caption{Equality}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/leibniz.png}
\caption{Example from Leibniz}
\end{figure}

\hypertarget{functional-program}{%
\subsubsection{Functional Program}\label{functional-program}}

\begin{itemize}
\item
  a functional program consists of

  \begin{enumerate}
  \def\labelenumi{\arabic{enumi}.}
  \tightlist
  \item
    a set of value and function declarations
  \item
    a single expression
  \end{enumerate}
\item
  functional programming is referentially transparent

  \begin{itemize}
  \tightlist
  \item
    values and functions are declared via equality
  \item
    equality then means mathematical equality (if using eager evaluation
    modulo termination)
  \end{itemize}
\item
  referential transparency employed for

  \begin{itemize}
  \tightlist
  \item
    program development, transformation, and proof
  \item
    evaluation
  \end{itemize}
\end{itemize}

\clearpage
\hypertarget{imperative-programming}{%
\subsection{Imperative Programming}\label{imperative-programming}}

\begin{itemize}
\tightlist
\item
  syntax: expressions + commands
\item
  semantics: values + environment + state
\item
  expressions are evaluated in the environment and current state,
  yielding a value
\item
  commands are executed in the environment and current state, yielding a
  new state
\item
  Example:

  \begin{itemize}
  \tightlist
  \item
    assignment command with variable v and expression E -\textgreater{}
    v := E
  \item
    E is evaluated in the environment and current state, yielding value
    t; then t is assigned to the storage cell denoted by v in the
    environment, thus yielding a new state
  \end{itemize}
\item
  proofs of imperative programs are well possible too, but are by far
  more complicated
\item
  possible using HOARE logic
\item
  HOARE triple, with P, Q predicates and C command:

  \begin{itemize}
  \tightlist
  \item
    \{P\} C \{Q\}
  \item
    means: if execution of C starts in a state satisfying P, and
    execution terminates, then the resulting state satisfies Q
  \end{itemize}
\item
  Example:

  \begin{itemize}
  \tightlist
  \item
    proof rule for assignment command v := E
  \item
    \{Q{[}v \textless{}- E{]}\} v := E \{Q\}
  \end{itemize}
\end{itemize}

\hypertarget{conclusion-on-imperative-vs.functional}{%
\subsection{Conclusion on Imperative
vs.~Functional}\label{conclusion-on-imperative-vs.functional}}

\textbf{imperative paradigm}

\begin{itemize}
\tightlist
\item
  syntax: expressions + commands
\item
  semantics: values + environment + state
\item
  expressions are evaluated in the environment and current state,
  yielding a value
\item
  commands are executed in the environment and current state, yielding a
  new state
\end{itemize}

\textbf{functional paradigm}

\begin{itemize}
\tightlist
\item
  syntax: expressions
\item
  semantics: values + environment
\item
  expressions are evaluated in the environment, yielding a value
\end{itemize}

\hypertarget{misuse-of-the-symbol-for-equality}{%
\subsubsection{Misuse of the Symbol for Equality
=}\label{misuse-of-the-symbol-for-equality}}

\begin{itemize}
\tightlist
\item
  assignment like x := x + 1 has not the slightest similarity to
  equality
\item
  it is pronounced ``x becomes (gets, receives) x + 1'' \ldots{}
\item
  \ldots{} but never ever ``x equals (is, is equal to) x + 1''
\item
  a different symbol like := or ← should be used instead
\item
  using the symbol for equality = to denote assignment is a horrendous
  design error of too many programming languages.
\end{itemize}

\hypertarget{evaluation}{%
\subsection{Evaluation}\label{evaluation}}

\begin{itemize}
\tightlist
\item
  strategies

  \begin{itemize}
  \tightlist
  \item
    innermost (call-by-value)
  \item
    outermost (call-by-name)
  \item
    \textbf{lazy (outermost + sharing)}
  \item
    reducible expression, or redex
  \item
    application of a function to its argument expressions
  \end{itemize}
\item
  Example: mult(x,y) = x * y

  \begin{itemize}
  \tightlist
  \item
    mult(1+2, 2+3) has three redexes

    \begin{itemize}
    \tightlist
    \item
      1+2, yielding mult(3,2+3)
    \item
      2+3, yielding mult(1+2,5)
    \item
      mult(1+2,2+3) yielding (1+2)*(2+3)
    \end{itemize}
  \end{itemize}
\end{itemize}

\hypertarget{innermost-evaluation}{%
\subsubsection{Innermost Evaluation}\label{innermost-evaluation}}

innermost redex first; if several, choose leftmost one first

mult (1+2,2+3)\\
= mult (3,2+3)\\
= mult (3,5)\\
= 3*5\\
= 15

\clearpage
\hypertarget{outermost-evaluation}{%
\subsubsection{Outermost Evaluation}\label{outermost-evaluation}}

outermost redex first; if several, choose leftmost one first

mult (1+2,2+3)\\
= (1+2) * (2+3)\\
= 3 * (2+3)\\
= 3 * 5\\
= 15

\hypertarget{lazy-evaluation}{%
\subsubsection{Lazy Evaluation}\label{lazy-evaluation}}

Argument expressions might be evaluated more than once if the
corresponding formal parameters occur several times in the body of the
function. Solution to this problem via sharing:

\begin{itemize}
\tightlist
\item
  keep only a single copy of the argument expression, and maintain a
  pointer to it for each corresponding formal parameter
\item
  evaluate the expression once, and replace it by its value
\item
  access this value through the pointers
\end{itemize}

\hypertarget{summary-of-evaluation}{%
\subsubsection{Summary of evaluation}\label{summary-of-evaluation}}

\begin{itemize}
\tightlist
\item
  an argument is evaluated

  \begin{itemize}
  \tightlist
  \item
    innermost: exactly once
  \item
    outermost: zero or more times
  \item
    lazy: at most once
  \end{itemize}
\item
  whenever there exists an order of evaluation that terminates,
  outermost (and thus lazy) evaluation will find it
\end{itemize}

\begin{tcolorbox}[colback=red!5!white,colframe=red!75!black]
The evaluation strategy in which function arguments are evaluated before the function call is made is known as an "eager" evaluation.

Let us postpone evaluating function arguments until after the function call, and then evaluate the argument only if it actually is needed (e.g. if it appears in an expression in the called function). Such a strategy is "lazy."
\end{tcolorbox}

\clearpage


\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}

\usepackage{parskip}
\usepackage{amsmath}

\title{Backpropagation Mathematics}
\author{Brady Neal}
\date{August 2015}

\begin{document}

\maketitle

\section{Definitions}

If you are ever reading through the math in here and realize you don't know what something such as $w_{ij}^l$ means, just come back to this section and you'll find a concise definition.

Superscripts will always refer to the layer, and subscripts will always refer to the neuron in that layer. All indexing will start from 1, not 0.

$b_i^l$ - bias of the $i$\textsuperscript{th} neuron in the $l$\textsuperscript{th} layer

$w_{ij}^l$ - weight from the $j$\textsuperscript{th} neuron in layer $l - 1$ to the $i$\textsuperscript{th} neuron in layer $l$

$z_i^l$ - weighted input to the activation function of the $i$\textsuperscript{th} neuron in the $l$\textsuperscript{th} layer
\[z_i^l = \sum_j w_{ij}^l a_j^{l - 1} + b_i^l\]
Vectorized version:
\[z^l = w^l a^{l - 1} + b^l\]

The activation function used is the sigmoid function:
\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

$a_i^l$ - activation of the $i$\textsuperscript{th} neuron in the $l$\textsuperscript{th} layer
\[a_i^l = \sigma(z_i^l) = \sigma\left(\sum_j w_{ij}^l a_j^{l - 1} + b_i^l\right)\]
Vectorized version:
\[a^l = \sigma(z^l) = \sigma(w^l a^{l - 1} + b^l)\]

$L$ - number of layers in the neural network

$x^{(i)}$ - feature vector corresponding to the $i$\textsuperscript{th} training example

$y^{(i)}$ - correct output value for the $i$\textsuperscript{th} training example

$a = a^L$ - hypothesis (prediction) of the neural network for a training example (i.e. $a^{(i)}$ is the neural network output that corresponds to $x^{(i)}$)

$\odot$ - the Hadamard product operator (element-wise multiplication between n-dimensional arrays)

The cost function used is the cross entropy error function:
\[C = - \frac{1}{m} \sum_{i = 1}^m \left[y^{(i)} \ln a^{(i)} + (1 - y^{(i)}) \ln \left(1 - a^{(i)}\right)\right]\]

where $m$ is the number of training examples, $a^{(i)}$ is the output layer of the neural network for the $i$\textsuperscript{th} training example, and $y^{(i)}$ is the correct output for the $i$\textsuperscript{th} training example.

The above cost function is over all training examples $x^{(i)}$. Both $y^{(i)}$ and $a^{(i)}$ are commonly vectors, so it is more explicitly written as
\[C = - \frac{1}{m} \sum_{i = 1}^m \sum_j \left[y_j^{(i)} \ln a_j^{(i)} + (1 - y_j^{(i)}) \ln \left(1 - a_j^{(i)}\right)\right]\]
where $y_j^{(i)}$ and $a_j^{(i)}$ are the $j$\textsuperscript{th} elements in the $y^{(i)}$ and $a^{(i)}$ vectors respectively.

However, I will mostly be referring to the cost of a single training example in vectorized form, $C_x$:
\[C_x = y \ln a + (1 - y) \ln (1 - a)\]
This simplifies the notation a great deal as we no longer have to worry about the many indices and summations. Therefore, when I say $C$ below, I'm often referring to $C_x$.

\section{Proofs}

Because we want to find the weights and biases that will minimize the cost function, the ultimate goal of backpropagation is to find the following:
\[\frac{\partial C}{\partial w_{ij}^l} \text{ and } \frac{\partial C}{\partial b_i^l}\]
We need to get four fundamental equations first:
\begin{flalign*}
&\delta_i^L = \frac{\partial C}{\partial a_i^L} \sigma'(z_i^L)
\qquad \qquad \qquad \quad
\delta^L = \nabla_a C \odot \sigma'(z^L) \\
&\delta_j^l = \sum_i w_{ij}^{l + 1} \delta_i^{l + 1} \sigma'(z_j^l)
\qquad \qquad
\delta^l = \left(\left(w^{l + 1}\right)^T \delta^{l + 1}\right) \odot \sigma'(z^l) \\
&\frac{\partial C}{\partial w_{ij}^l} = \delta_i^l a_j^{l - 1}
\qquad \qquad \qquad \qquad
\frac{\partial C}{\partial w^l} = \delta^l \left(a^{l - 1}\right)^T \\
&\frac{\partial C}{\partial b_i^l} = \delta_i^l
\qquad \qquad \qquad \qquad \qquad \enspace
\frac{\partial C}{\partial b^l} = \delta^l
\end{flalign*}

\subsection{An Intermediate Quantity: $\delta_i^l$}
In order to get $\frac{\partial C}{\partial w_{ij}^l}$ and $\frac{\partial C}{\partial b_i^l}$, we must first define an intermediate quantity that we'll call the error:
\[\delta_i^l = \frac{\partial C}{\partial z_i^l}\]
The first step is to get the error in the output layer $L$:
\[\delta_i^L = \frac{\partial C}{\partial z_i^L} = \frac{\partial C}{\partial a_i^L} \frac{d a_i^L}{d z_i^L} \text{ or } \frac{\partial C}{\partial a_i^L} \sigma'(z_i^L)\]
Vectorized:
\[\delta^L = \nabla_a C \odot \sigma'(z^L)\]
Now that we have the error in the output layer, $\delta^L$, we need an equation that relates $\delta^l$ to $\delta^{l + 1}$ in order to inductively solve for $\delta^l$.
\[\delta_j^l = \frac{\partial C}{\partial z_j^l} = \sum_i \frac{\partial C}{\partial z_i^{l + 1}} \frac{\partial z_i^{l + 1}}{\partial z_j^l} = \sum_i \delta_i^{l + 1} \frac{\partial z_i^{l + 1}}{\partial z_j^l}\]
In order to get $\frac{\partial z_i^{l + 1}}{\partial z_j^l}$, recall $z_i^{l + 1}$:
\[z_i^{l + 1} = \sum_j w_{ij}^{l + 1} a_j^l + b_i^{l + 1} = \sum_j w_{ij}^{l + 1} \sigma(z_j^l) + b_i^{l + 1}\]
\[\frac{\partial z_i^{l + 1}}{\partial z_j^l} = w_{ij}^{l + 1} \sigma'(z_j^l)\]
Thus,
\[\delta_j^l = \sum_i \delta_i^{l + 1} \frac{\partial z_i^{l + 1}}{\partial z_j^l} = \sum_i \delta_i^{l + 1} w_{ij}^{l + 1} \sigma'(z_j^l) = \sum_i w_{ij}^{l + 1} \delta_i^{l + 1} \sigma'(z_j^l)\]
Vectorized:
\[\delta^l = \left(\left(w^{l + 1}\right)^T \delta^{l + 1}\right) \odot \sigma'(z^l)\]

\subsection{$\frac{\partial C}{\partial w_{ij}^l}$ and $\frac{\partial C}{\partial b_i^l}$ in terms of $\delta_i^l$}

\[\frac{\partial C}{\partial w_{ij}^l} = \frac{\partial C}{\partial z_i^l} \frac{\partial z_i^l}{\partial w_{ij}^l} = \delta_i^l \frac{\partial z_i^l}{\partial w_{ij}^l}\]
\[\frac{\partial C}{\partial b_i^l} = \frac{\partial C}{\partial z_i^l} \frac{\partial z_i^l}{\partial b_i^l} = \delta_i^l \frac{\partial z_i^l}{\partial b_i^l}\]
In order to get $\frac{\partial z_i^l}{\partial w_{ij}^l}$ and $\frac{\partial z_i^l}{\partial b_i^l}$, recall $z_i^l$:
\[z_i^l = \sum_j w_{ij}^l a_j^{l - 1} + b_i^l\]
\[\frac{\partial z_i^l}{\partial w_{ij}^l} = a_j^{l - 1}
\qquad \qquad
\frac{\partial z_i^l}{\partial b_i^l} = 1\]
Thus,
\[\frac{\partial C}{\partial w_{ij}^l} = \delta_i^l \frac{\partial z_i^l}{\partial w_{ij}^l} = \delta_i^l a_j^{l - 1}\]
\[\frac{\partial C}{\partial b_i^l} = \delta_i^l \frac{\partial z_i^l}{\partial b_i^l} = \delta_i^l\]
Vectorized:
\[\frac{\partial C}{\partial w^l} = \delta^l \left(a^{l - 1}\right)^T\]
\[\frac{\partial C}{\partial b^l} = \delta^l\]

\section{Specific Activation Function and Cost Function}

All of the math in the previous section refers to the activation function as $\sigma$ and the cost function as $C$. Because we defined a specific activation function and cost function at the beginning of the paper, we will now work out the math that will be necessary for an implementation of backpropagation.

In other words, $\sigma'(z_i^L)$ and $\frac{\partial C}{\partial a_i^L}$ are part of the fundamental equations above, so we'll calculate them with respect to our chosen activation function and cost function.

\subsection{Derivative of The Sigmoid Function}
Recall the sigmoid function:
\[\sigma(z) = \frac{1}{1 + e^{-z}} = \left(1 + e^{-z}\right)^{-1}\]
\[\sigma'(z) = -\left(1 + e^{-z}\right)^{-2} \cdot -e^{-z} = e^{-z} \left(1 + e^{-z}\right)^{-2}\]
\[\sigma'(z) = \frac{e^{-z}}{\left(1 + e^{-z}\right)^{2}} = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}}\]
\[\sigma'(z) = \frac{1}{1 + e^{-z}} \left(1 - \frac{1}{1 + e^{-z}}\right)\]
Thus,
\[\sigma'(z) = \sigma(z) \left(1 - \sigma(z)\right)\]

\subsection{Partial Derivative of The Cross Entropy Error Function}
Recall the cross entropy error function:
\[C = - \frac{1}{m} \sum_{i = 1}^m \left[y^{(i)} \ln a^{(i)} + (1 - y^{(i)}) \ln \left(1 - a^{(i)}\right)\right]\]
where $m$ is the number of training examples, $a^{(i)}$ is the output layer of the neural network for the $i$\textsuperscript{th} training example, and $y^{(i)}$ is the correct output for the $i$\textsuperscript{th} training example.
\[\frac{\partial C}{\partial a^{(i)}} = -\left(y^{(i)} \cdot \frac{1}{a^{(i)}} + (1 - y^{(i)}) \cdot \frac{1}{1 - a^{(i)}} \cdot -1\right)\]
\[\frac{\partial C}{\partial a^{(i)}} = \frac{\left(1 - y^{(i)}\right)}{1 - a^{(i)}} - \frac{y^{(i)}}{a^{(i)}} = \frac{a^{(i)}\left(1 - y^{(i)}\right) - y^{(i)}\left(1 - a^{(i)}\right)}{a^{(i)} \left(1 - a^{(i)}\right)}\]
\[\frac{\partial C}{\partial a^{(i)}} = \frac{a^{(i)} - a^{(i)} y^{(i)} - y^{(i)} + a^{(i)} y^{(i)}}{a^{(i)} \left(1 - a^{(i)}\right)}\]
Thus,
\[\frac{\partial C}{\partial a^{(i)}} = \frac{a^{(i)} - y^{(i)}}{a^{(i)} \left(1 - a^{(i)}\right)}\]
Notice that the denominator looks quite familiar. Recall $a^{(i)} = \sigma\left(z^L\right)^{(i)}$, and we can relate the function to $\sigma'$ (let's drop the indices for simplicity):
\[\frac{\partial C}{\partial a} = \frac{a - y}{a (1 - a)} = \frac{a - y}{\sigma(z) \left(1 - \sigma(z)\right)} = \frac{a - y}{\sigma'(z)}\]
So if we ever had to find $\frac{\partial C}{\partial a} \sigma'(z)$, we'd magically get
\[\frac{\partial C}{\partial a} \sigma'(z) = \frac{a - y}{\sigma'(z)} \sigma'(z) = a - y\]

\subsection{Rewriting the Four Fundamental Equations}
Now that we know $\sigma'(z)$ and $\frac{\partial C}{\partial a}$ for our specific activation function and cost function, we can rewrite the backpropogation equations more specifically.

Here are their more generalized versions:
\begin{flalign*}
&\delta_i^L = \frac{\partial C}{\partial a_i^L} \sigma'(z_i^L)
\qquad \qquad \qquad \quad
\delta^L = \nabla_a C \odot \sigma'(z^L) \\
&\delta_j^l = \sum_i w_{ij}^{l + 1} \delta_i^{l + 1} \sigma'(z_j^l)
\qquad \qquad
\delta^l = \left(\left(w^{l + 1}\right)^T \delta^{l + 1}\right) \odot \sigma'(z^l) \\
&\frac{\partial C}{\partial w_{ij}^l} = \delta_i^l a_j^{l - 1}
\qquad \qquad \qquad \qquad
\frac{\partial C}{\partial w^l} = \delta^l \left(a^{l - 1}\right)^T \\
&\frac{\partial C}{\partial b_i^l} = \delta_i^l
\qquad \qquad \qquad \qquad \qquad \enspace
\frac{\partial C}{\partial b^l} = \delta^l
\end{flalign*}

Here are their more specific versions:
\begin{flalign*}
&\delta^L = a^L - y \\
&\delta^l = \left(\left(w^{l + 1}\right)^T \delta^{l + 1}\right) \odot \left(a^l \odot \left(1 - a^l\right)\right) \\
&\frac{\partial C}{\partial w^l} = \delta^l \left(a^{l - 1}\right)^T \\
&\frac{\partial C}{\partial b^l} = \delta^l
\end{flalign*}

\section{Regularization}
In order to reduce overfitting, we're going to add L2 regularization to our cost function.

Recall the cross entropy error function that we used as our cost function. We're now going to call this $C_0$:
\[C_0 = - \frac{1}{m} \sum_{i = 1}^m \left[y^{(i)} \ln a^{(i)} + (1 - y^{(i)}) \ln \left(1 - a^{(i)}\right)\right]\]
Our new cost function with L2 regularization:
\[C = - \frac{1}{m} \sum_{i = 1}^m \left[y^{(i)} \ln a^{(i)} + (1 - y^{(i)}) \ln \left(1 - a^{(i)}\right)\right] + \frac{\lambda}{2m} \sum_{i = 1}^m \sum_j \sum_k {w_{jk}^{(i)}}^2\]
Now that we've changed the cost function, we have to figure out how that changes $\frac{\partial C}{\partial w_{jk}^{(i)}}$ and $\frac{\partial C}{\partial b_j^{(i)}}$ Because we already know $\frac{\partial C_0}{\partial w_{jk}^{(i)}}$ and $\frac{\partial C_0}{\partial b_j^{(i)}}$ let's start by rewriting $C$ in terms of $C_0$:
\[C = C_0 + \frac{\lambda}{2m} \sum_i^m \sum_j \sum_k {w_{jk}^{(i)}}^2\]
Thus,
\[\frac{\partial C}{\partial w_{jk}^{(i)}} = \frac{\partial C_0}{\partial w_{jk}^{(i)}} + \frac{d}{d w_{jk}^{(i)}} \frac{\lambda}{2m} {w_{jk}^{(i)}}^2\]
\[\frac{\partial C}{\partial w_{jk}^{(i)}} = \frac{\partial C_0}{\partial w_{jk}^{(i)}} + \frac{\lambda}{m} w_{jk}^{(i)}\]
Therefore, the new gradient descent update would be:
\[w_{jk}^{(i)} \leftarrow w_{jk}^{(i)} - \alpha \left( \frac{\partial C_0}{\partial w_{jk}^{(i)}} + \frac{\lambda}{m} w_{jk}^{(i)} \right)\]
\[w_{jk}^{(i)} \leftarrow \left( 1 - \frac{\alpha \lambda}{m} \right) w_{jk}^{(i)} - \alpha \frac{\partial C_0}{\partial w_{jk}^{(i)}}\]
where $\alpha$ is the learning rate. \\

Because, $\frac{\lambda}{2m} \sum\limits_{i = 1}^m \sum\limits_j \sum\limits_k {w_{jk}^{(i)}}^2$ has no dependence on $b_j^{(i)}$,
\[\frac{\partial C}{\partial b_j^{(i)}} = \frac{\partial C_0}{\partial b_j^{(i)}}\]

\end{document}


\documentclass{scribe}
\theoremstyle{plain}
\theoremheaderfont{\upshape \bfseries}
\theoremseparator{.}
\theoremsymbol{}
\theorembodyfont{\itshape}
\newtheorem{problem}[theorem]{Problem}
\theoremstyle{empty}
\theoremheaderfont{\upshape \bfseries}
\theorembodyfont{\upshape}
\theoremsymbol{\ensuremath{\square}}
\newtheorem{refproof}{Proof}

\setCourse{Mathematics for Computer Science (30470023)}
\setSemester{Spring 2020}
\setInstructor{Andrew C. Yao}
\setLectureID{3}
\setLectureDate{Mar 2, 2020}
\setScribe{Han Luo}

\begin{document}
\notetitle

\section{Overview}
In this lecture we discribe the solution to the Online Auction Problem. The surprising result is that it is the best to skip first $\frac{1}{\text{e}}$ offers. Then we introduce the concept of conditional probability, and also the distributive principle as well as the chain rule. Finally we introduce the expectation for a random variable.

\section{The Online Auction Problem (Secretary's Problem/Beauty contest)}

In last lecture we introduced the Online Auction Problem:

\begin{problem}
    A music concert ticket is selling. There is a stream of $n$ offers with prices $x_1,x_2,\cdots,x_n$, for what strategy can we get the best price with highest probability?
\end{problem}

\begin{remark}
    The strategy is non-regrettable, means that we cannot go back to the previous offers we rejected.
\end{remark}

\subsection{Formalizing}

We can simply assume that $x_1,x_2,\cdots,x_n$ is a permutation of $1,2,\cdots,n$. The probability space $\mathbb{P}=(\mathcal{U},p)$ is denoted by
\begin{itemize}
    \item $\mathcal{U}$ is the set of all permutations of $\{1,2,\cdots,n\}$;
    \item For all $u\in \mathcal{U}$, $p(u)=\dfrac{1}{n!}$;
    \item $T$ is the event of accepting the best offer (i.e. the price $n$).
\end{itemize}

For integer $k(1\le k<n)$, we denote strategy $k$ as
\begin{itemize}
    \item[1.] Skip the first $k$ offer;
    \item[2.] For $j>k$, take the first $x_j$, such that $x_j>\max\{x_1,x_2,\cdots,x_k\}$.
\end{itemize}
Then the probability of getting the best price $p_{n,k}=\dfrac{|T|}{|\mathcal{U}|}=\dfrac{|T|}{n!}$. To calculate it, we should first obtain the properties of the permutations where we can reach the best price.

A permutation $u=(x_1,x_2,\cdots,x_n)$ where $x_j=n$ is in $T$ iff
\begin{itemize}
    \item[1.] $j>k$;
    \item[2.] $\max\{x_1,x_2,\cdots,x_{j-1}\}=\max\{x_1,x_2,\cdots,x_{k}\}$.
\end{itemize}

To continue the calculation, we first introduce two basic principles.

\subsection{Two basic principles}

\begin{theorem}[Addition principle]
    If $S=S_1\cup S_2\cup\cdots\cup S_m$ is a disjoint union, then
    $$|S|=\sum_{i=1}^m|S_i|.$$
\end{theorem}

\begin{remark}
    As a simple application for addition principle, we have $\text{Pr}(S)\le\sum\limits_{i=1}^m \text{Pr}(S_i)$, the equality holds iff the $S_i$ are disjoint.
\end{remark}

\begin{theorem}[Multiplication principle]
    If $S$ has items characterized by $s\in S, s=(i_1,i_2,\cdots,i_l)$, and $1\le i_j\le c_j$ for all $1\le j\le l$, then
    $$|S|=c_1c_2\cdots c_l.$$
\end{theorem}

\subsection{The solution of the Online Auction Problem}

\begin{lemma} \label{lmm:num-permutation}
    Denote $T_j$ as the set of permutations in $T$ that $x_j=n$, then
    $$|T_j|=n!\cdot\frac{1}{n}\cdot\frac{k}{j-1}.$$
\end{lemma}

We will prove to this lemma later.

Applying the lemma, we get that
$$|T|=\sum_{j>k}|T_j|=n!\cdot\frac{k}{n}\left(\frac{1}{k}+\frac{1}{k+1}+\cdots+\frac{1}{n}\right),$$
so
\begin{align*}
    p_{n,k}&=\frac{k}{n}\left(\frac{1}{k}+\frac{1}{k+1}+\cdots+\frac{1}{n}\right) \\
    & \approx\frac{k}{n}\left(\ln n-\ln k\right) \\
    & =\frac{k}{n}\ln \frac{n}{k}
\end{align*}
by derivation we get that when $k=\dfrac{1}{\text{e}}n$ the probability reaches the maximum $\dfrac{1}{\text{e}}$.

\section{Conditional Probability}

\begin{definition}
    Let $V,W$ be two events such that $\text{Pr}(W)\neq 0$. The conditional probability of $V$ (conditioned on $W$) is defined by
    $$\text{Pr}(V|W)=\frac{\text{Pr}(V\cap W)}{\text{Pr}(W)}.$$
\end{definition}
\begin{remark}
    \begin{itemize}
        \item[1.] $V\cap W$ represents the event that both $V$ and $W$ happen simultaneously;
        \item[2.] The conditional probability can be seen as a restricted probability space $\mathbb{P'}=(W,p')$ with $p'(u)=\dfrac{p(u)}{\text{Pr}(W)}$;
        \item[3.] In particular, define $\text{Pr}(V|W)=0$ if $\text{Pr}(W)=0$.
    \end{itemize}
\end{remark}

\begin{theorem}[Distributive principle]
    Let $W_1, W_2, \cdots, W_m$ be disjoint events, and their union is $W=W_1\cup W_2\cup\cdots\cup W_m$, then for arbitrary event $T$, we have
    $$\Pr(T|W)=\frac{\sum_{i=1}^m\text{Pr}(W_i)\text{Pr}(T|W_i)}{\text{Pr}(W)}.$$
\end{theorem}

\begin{proof}
    Since $W_1,W_2,\cdots,W_m$ are disjoint, we get that $T\cap W_1,T\cap W_2,\cdots,T\cap W_m$ are disjoint and their union is $T\cap W$. So by the addition principle,
\begin{align*}
    \sum_{i=1}^m\text{Pr}(W_i)\text{Pr}(T|W_i) & = \sum_{i=1}^m \text{Pr}(T\cap W_i) \\
    &= \text{Pr}(T\cap W) \\
    &=\text{Pr}(T|W)\text{Pr}(W).
\end{align*}
\end{proof}

This distributive principle can be applied to the Online Auction Problem.

\begin{example}[Proof of Lemma \ref{lmm:num-permutation}]
    In the Online Auction Problem, we denote $W_j(j\ge k+1)$ to be the set of permutations that $x_j=n$, then $\mathcal{U}=W_{k+1}\cup \cdots\cup W_n$, and $\text{Pr}(W_j)=\dfrac{1}{n}$. Since $\text{Pt}(T|W_j)$ is the probability that we can get the best price in the case of $x_j=n$, i.e. the maximum of $x_1,x_2,\cdots,x_{j-1}$ appears in $x_1,x_2,\cdots,x_k$, so $\text{Pt}(T|W_j)=\dfrac{k}{j-1}$. This gives a brief proof to Lemma \ref{lmm:num-permutation}.
\end{example}

\begin{theorem}[Chain Rule]
    Let $V_1, V_2,\cdots, V_k$ be events that not forced to be disjoint, and the event that they happen simutaneously is $T=V_1\cap V_2\cap\cdots\cap V_k$, then
    $$\text{Pr}(T)=\text{Pr}(V_1)\text{Pr}(V_2|V_1)\text{Pr}(V_3|V_1\cap V_2)\cdots\text{Pr}(V_k|V_1\cap V_2\cap \cdots\cap V_{k-1})$$
\end{theorem}

\begin{remark}
    The chain rule is a generalization of the ``multiplication principle''.
\end{remark}

\begin{example}
    In birthday paradox, denote $\bar T$ to be the event that everyone in class has a unique birthday, and $V_j$ is the student \#1, \#2, $\cdots$, \#$j$ have different birthdays, so $T=V_1\cap V_2\cap\cdots\cap V_k$, and
    $$\Pr(V_j|V_1\cap V_2\cap \cdot\cap V_{j-1})=1-\frac{j-1}{n}.$$

    Appling the Chain Rule we get
    $$\text{Pr}(T)=\text{Pr}(V_1)\text{Pr}(V_2|V_1)\cdots\text{Pr}(V_k|V_1\cap V_2\cap \cdots\cap V_{k-1})=\prod_{i=0}^{k-1}\left(1-\frac{i}{n}\right).$$
\end{example}

\section{Expectation}

\begin{definition}[Expectation]
    For a probability space $\mathbb{P}=(\mathcal{U},p)$, a random variable $X$ over $\mathcal{U}$ is a real-valued mapping $X:\mathcal{U}\to (-\infty,+\infty)$.

    The expectation of $X$ is defined by
    $$\mathbb{E}(X)=\sum_{u\in U}X(u)p(u).$$
\end{definition}

\begin{theorem}[linear of expectation] If random variable $X=X_1+X_2$, then
    $$\mathbb{E}(X)=\mathbb{E}(X_1)+\mathbb{E}(X_2).$$
\end{theorem}

\begin{proof} by the definition of expctation,
    $$\mathbb{E}(X)=\sum_{u\in U}p(u)X(u)=\sum_{u\in U}p(u)(X_1(u)+X_2(u))=\mathbb{E}(X_1)+\mathbb{E}(X_2).$$
\end{proof}

\begin{example}
    Throw $n$ coins independently, each coin has bias $b$, i.e.
    $$
    \begin{cases}
    \text{Pr}(X_i=1)=b \\
    \text{Pr}(X_i=0)=1-b
    \end{cases}.
    $$

    Denote $X$ to be the number of $1$s you throw. Then $\mathbb{E}(X)=\mathbb{E}(X_1)+\mathbb{E}(X_2)+\cdots+\mathbb{E}(X_n)=bn$.
\end{example}

\begin{example}
    Take a random permutation $u$ of $\{1,2,\cdots,n\}$, look at its cycle representation, denote by $X(u)$ the number of cycles in its cycle representation.

    To calculate the expectation of $X(u)$, we can write the random variable $X(u)$ to be some easily calculated random variables. Denote $X_i=\dfrac{1}{l}$ be the random variables of $i$, where $l$ is the length of the cycle that $i$ is on. Then $X=\sum\limits_{i=1}^nX_i$.

    So $\mathbb{E}(X)=\sum\limits_{i=1}^n\mathbb{E}(X_i)$. The rest of the problem is solved in homework.
\end{example}

\begin{example}
    We play a game on a rooted tree. In every step we randomly pick a node $v$ and delete $v$ and all its descendents, repeat the step until the tree is empty. Denote by $X$ the number of steps we will make before tree is empty. The result is also shown in homework.
\end{example}

\end{document}


\section{Optimal joint assignment (OJA)}\label{sec:qp}

Since heatmaps generated by the joint predictor are multi-modal, the non-maximum suppression procedure yields multiple possible locations for each joint. The set of joint proposals is represented as $X = \{x_{jp}\}$, where $x_{jp}$ indicates the 2D position of proposal $p \in \{1,...,N_j\}$ associated with joint $j \in J$.
Before applying the optimizer, a subset of proposals $X^* \subseteq X$ should be selected in order to form a complete skeleton, i.e. precisely \emph{one proposal is selected for every joint}. This section will consider how to choose the optimal subset by formulating the problem as an extended optimal assignment problem.

In order to select a complete skeleton proposal from the set of joint proposals $\{x_{jp}\}$, a binary indicator vector $\bvec a_j = \{a_{jp}\} \in \{0, 1\}^{N_j+1}$ is introduced, where $a_{jp} = 1$ indicates that the $p^\text{th}$ proposal for joint $j$ is a correct assignment, and the $p = N_j+1$ position corresponds to a {\em null proposal}, indicating that joint $j$ has no match in this image.
The null proposals are handled as described in each of the energy terms below.
Let $A$ be the jagged array $[\bvec a_j]_{j=1}^J$ containing all assignment variables (for the current frame), and let $X^* = X(A)$ denote the subset of points selected by the binary array $A$.

% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
% \usepackage{multirow}
\begin{table}
    \RawFloats
    \parbox{.48\linewidth}{
        \strut
        \centering
        \begin{tabular}{@{}llll@{}}
        \toprule
        Joint $j$               & Proposal $p$ & $x_{jp}$ & $a_{jp}$ \\
        \midrule
        \multirow{3}{*}{Nose} & 0          & $(0, 2) $   & 1 \\
                            & 1          & $(4, 2) $   & 0 \\
                            & NULL       & ---         & 0 \\
        \midrule
        Upper Leg             & NULL       & ---         & 1 \\
        \midrule
        \multirow{3}{*}{Paw}  & 0          & $(4, 2)$    & 0 \\
                            & 1          & $(8, 10)$   & 1 \\
                            & NULL       & ---         & 0 \\
        \midrule
        \multirow{4}{*}{Tail} & 0          & $(4, 2)$    & 0 \\
                            & 1          & $(8, 10)$   & 0 \\
                            & 2          & $(4, 2)$    & 1 \\
                            & NULL       & ---         & 0 \\
        \bottomrule
        \end{tabular}
        \caption{Example inputs and output to the joint assignment problem. Non maximum suppression applied to predicted heatmap tensors yields a set of proposals $p$ for each each skeleton joint $j$ at 2D location $x_{jp}$. $a_{jp}$ is an illustrative assignment vector which is predicted by the OJA algorithm.}
        \label{tab:oja-example-inputs}
    }
    \hfill
    \parbox{.48\linewidth}{
        \strut
        \centering
        \parbox{\linewidth}{
            \strut
            \centering
            \begin{tabular}{@{}lllll@{}}
            \toprule
            Joint $j$ & \multicolumn{4}{l}{Proposal $p$}                                                 \\
            \midrule
            Nose      & 1 & 0                        & 0                        & \cellcolor[HTML]{808080} \\
            Upper Leg & 1 & \cellcolor[HTML]{808080} & \cellcolor[HTML]{808080} & \cellcolor[HTML]{808080} \\
            Paw       & 0 & 1                        & 0                        & \cellcolor[HTML]{808080} \\
            Tail      & 0 & 0                        & 1                        & 0                        \\
            \bottomrule
            \end{tabular}
            \caption{Assignment variables $A = \bvec a_j = \{a_{jp}\} \in \{0, 1\}^{N_j+1}$ for the current frame stored as a jagged array.}
            \label{tab:oja-proposals}
        }
        \bigskip
        \parbox{\linewidth}{
            \strut
            \centering
            \begin{tabular}{@{}ll@{}}
                \toprule
                Joint $j$ & $X^* = X(A)$ \\
                \midrule
                Nose      & $(0, 2)$     \\
                Upper Leg & ---          \\
                Paw       & $(8, 10)$    \\
                Tail      & $(4, 2)$     \\
                \bottomrule
            \end{tabular}
            \caption{2D joint locations $X^* = X(A)$ selected by the assignment variable $A$}
            \label{tab:oja-selected}
        }
    }
\end{table}

Optimal assignment minimizes the function
\begin{equation}
L(A) = \LL{conf}(A) + \LL{null}(A) + \LL{prior}(A) + \LL{temp}(A) + \LL{cov-sil}(A) + \LL{cov-bone}(A)
\end{equation}
which balances agreement of the joint configuration with the network-supplied {\em confidences}, a learned {\em prior}, {\em temporal} coherence, and {\em coverage} terms which encourage the model to correctly project over the silhouette. Without the coverage terms, this can be optimized as a quadratic program, but better results are obtained by including the coverage terms, and using a genetic algorithm. In addition, the parameters $A$ must satisfy the $J$ constraints $\sum_{p=1}^{N_j+1} a_{jp} = 1$, that exactly one joint proposal (or the null proposal) must be selected for each joint.

\subsection{Basic formulation}

\subsubsection{Network confidences: $\LL{conf}(A)$}

The first energy term $\LL{conf}(A)$ comes from the output of the joint prediction network, which provides a confidence score $y_{jp}$ associated with each joint proposal~$x_{jp}$.  Then $\LL{conf}(A) = \sum_j\sum_p -\lambda_{\text{conf}}\log(y_{jp}) A_{jp}$ is a linear function of $A$,
and $\lambda_{\text{conf}}$ is a tunable parameter to control the relative contribution of the network confidences compared with that of the skeleton prior. Note that if only this term is included, the OJA would simply produce the result of the standard non-maximal suppresion algorithm, selecting the heatmap location with highest network confidence. Note that this function can be rewritten as

\begin{equation}\label{eq:conf-energy}
    \LL{conf}(A) = \lambda_{\text{conf}}\log(y^T) \text{vec}(A)
\end{equation}

\subsubsection{Null proposals: $\LL{null}(A)$}

Under some circumstances, for example when a body part is heavily occluded or ambiguous, \emph{all} available proposals for a given joint may be of poor quality. In such a case, including any of these options in a skeleton configuration may have a detrimental impact on the later model fitting stage. Under these circumstances, it may be preferable to exclude the joint in question from the optimization entirely. Null proposals pay a fixed cost $\lambda_{null}$, effectively acting as a threshold whereby the null proposal will be selected if no other proposal is of sufficient likelihood. Precisely, a jagged array $D$ is defined

\begin{equation}
    D_{jp} = \begin{cases}
        \lambda_{\text{null}} & \text{if } p \text{ is null proposal}\\
        0 & \text{otherwise}
    \end{cases}
\end{equation}

and resulting energy becomes

\begin{equation}\label{eq:null-energy}
    \LL{null}(A) = \text{vec}(D) \text{vec}(A)
\end{equation}

\subsubsection{Skeleton Prior: $\LL{prior}(A)$}

The next energy term is used to discourage anatomically implausible skeletal configurations from being selected. The prior probability of a skeletal assignment $A$ is represented as a multivariate Gaussian distribution over the selected joint positions $X^* = X(A)$

% % Here, x is used as a general 2D coordinate
\begin{equation}
P_{\mathrm{prior}}(A) = \frac{1}{\sqrt[]{(2\pi)^k\left|\Sigma\right|}}\exp\left(-\frac{1}{2}(x^*-\mu)^T\Sigma^{-1}(x^*-\mu)\right)
\end{equation}

The mean $\mu \in \R{2J}$ and covariance $\Sigma \in \R{2J\times 2J}$ terms are obtained from synthetic training examples generated earlier. The prior is shown in \Cref{fig:skeleton-prior}. The objective of the OJA is to find the assignment vector $A$ which maximizes this prior, which is equivalent to minimizing the negative log prior
\begin{equation}
    \LL{prior}(A) = \frac{k}{2}\log(2\pi) + \frac{1}{2}\left|\Sigma\right| + \frac{1}{2}(x^*-\mu)^T\Sigma^{-1}(x^*-\mu)
\end{equation}

This formulation is reducible to a minimization over the Mahalanobis distance, which is given by the summation
\begin{equation}
\LL{prior}(A) = \sum_j^J\sum_p^{N_j}\sum_k^J\sum_q^{N_k}a_{jp}a_{kq}(x_{jp} - \mu_j)\Sigma_{jk}^{-1}(x_{kq}-\mu_k)
\end{equation}

Notice this is a quadratic function of $A$, so $\LL{prior}(A) = \text{vec}(A)^\top Q \text{vec}(A)$ for a fixed matrix $Q$. Precisely, the elements of the matrix $Q$ are precisely the Mahalanobis distance between each individual pair of joint proposals
\begin{equation}
\left[Q\right]_{jp, kq} = (x_{jp} - \mu_j)\Sigma_{jk}^{-1}(x_{kq}-\mu_k)
\end{equation}

Null proposals are simply excluded from the sum, equivalent to marginalizing over their position.

\begin{figure}[t]
\def\bb{\rule{2in}{0pt}\rule{0pt}{1in}}
\begin{tabular}{cccc}
\includegraphics[width=0.25\linewidth]{skeletal_prior_gen/63_rgb.png} & \includegraphics[width=0.25\linewidth]{skeletal_prior_gen/36_rgb.png} & \includegraphics[width=0.25\linewidth]{skeletal_prior/74.png} & \includegraphics[width=0.25\linewidth]{skeletal_prior/108.png} \\
(a) & (b) & (c) & (d) \\
\end{tabular}
\caption{Skeleton Prior: Synthetic quadruped training data examples (rendered with texture to show 3D) generated by sampling pose, shape and position parameters and applying to SMAL model (a), (b). The set of 2D skeletal positions are used to create a vector of means $\mu \in \mathbb{R}^{2J}$ and covariance matrix $\Sigma \in \mathbb{R}^{2J \times 2J}$. The Gaussian distribution constructed can be sampled to create new skeletons, such as those shown in (c), (d).}
\label{fig:skeleton-prior}
\end{figure}

\subsubsection{Temporal Prior: $\LL{temp}(A)$}
% TODO: Consider bringing in qp_defs/IMAGES to help explain and draw matrices Q
A common failure case of the joint prediction network is in situations where a joint position is highly ambiguous, for example between the left and right legs. In such cases, the algorithm will commonly alternate between two equally likely predictions. This leads to large displacements in joint positions between consecutive frames which are difficult for the later model fitting stage to recover from. This can be addressed by introducing a temporal term into the OJA. A prior is imposed on the distance moved by each joint between frame $t_0$ and $t_1$, which is given by a normal distribution with zero mean and variance $\sigma^{2} =e^{\tau|t_1 - t_0 - 1|}$.
The parameter $\tau$ controls the strength of the interaction between distant frames. This results in an additional quadratic term in our objective function, which has the form $L_{temp} = a^\top T^{(t_0, t_1)} a$ for matrix $T^{(t_0, t_1)}$ given by
\begin{equation}
\left[T^{(t_0, t_1)}\right]_{jp, kq} = \begin{cases}
e^{-\alpha|t_1 - t_0 - 1|}||x^{(t_0)}_{jp} - x^{(t_1)}_{kq}||^2 & \text{if } j=k\\
0 & \text{otherwise}
\end{cases}
\end{equation}

\subsection{QP solution.}
A general quadratic program is made up of a quadratic objective function and linear equality and inequality constraints. Thus far, all terms in $L(A)$ are quadratic or linear.

To optimize over a sequence of frames, we construct the block diagonal matrix $\hat{Q}$ whose diagonal elements are the prior matrices $Q^{(t)}$ and off-diagonal elements are the temporal matrices $T^{(t_0, t_1)}$. The confidence term \Cref{eq:conf-energy} and null penalty term \Cref{eq:null-energy} are combined and the vector $\hat{c}$ is obtained by stacking across each frame. The solution vector for the sequence $\hat{a}$ is similarly constructed by stacking the vectorized assignment matrices $A$ across timesteps. The jagged array $B$ is used to formalize the constraint that only one proposal should be selected per joint. Precisely

\begin{equation}
    \left[B\right]_{jp,kq} = \begin{cases}
        1 & \text{if } j=k\\
        0 & \text{otherwise}
    \end{cases}
\end{equation}
and (similarly to $A$) is vectorized and stacked across timesteps to form the constraint $\hat{B}\hat{a} = 1$. Finally, the constraint $\hat{a}(1 - \hat{a})=0$ is applied to ensure binary values of $\hat{a}$. The resulting quadratic program formulation is then given in \Cref{eq:quadprog}.

% \begin{equation}
% \min_{\hat{a}} \quad \hat{a}^T \hat{Q} \hat{a} + \lambda_{temp} \hat{a}^T \hat{T} \hat{a} + \lambda_{conf} \hat{c}_{conf}^T \hat{a} + \lambda_{null} \hat{c}_{null}^T\hat{a}
% \end{equation}

% \[
% \begin{array}{c c} &
%     \begin{array}{c c c} p=1 & p=2 & p=3 \\
%     \end{array}
%     \\
%     \begin{array}{c c c}
%     p=1 \\
%     p=2\\
%     p=3
%     \end{array}
%     &
%     \left[
%     \begin{array}{c c c}
%     0.1 & 0.1 & 0.0 \\
%     0.4 & 1.0 & 0.0 \\
%     0.8 & 0.0 & 0.4
%     \end{array}
%     \right]
% \end{array}
% \]

\begin{mini}
    {\hat{a}}{\hat{a}^T \hat{Q} \hat{a} + \hat{c}^T \hat{a}}{}{}
    \addConstraint{\hat{B}\hat{a} = 1}
    \addConstraint{\hat{a}(1 - \hat{a}) = 0}
    \label{eq:quadprog}
\end{mini}

% \begin{align}
%     \min_{\hat{a}} \quad \hat{a}^T \hat{Q} \hat{a} + \lambda_{temp} \hat{a}^T \hat{T} \hat{a} + \lambda_{conf} \hat{c}_{conf}^T \hat{a} + \lambda_{null} \hat{c}_{null}^T\hat{a} \\
%     \text{subject to } \sum_p^{N_j} \hat{a}_{jp} = 1 \quad \forall j & \quad \text{and } \hat{a}_{jp}(1 - \hat{a}_{jp}) = 0 \quad \forall j,p

% \end{align}

The quadratic program is specified using the open source CVXPY library \cite{diamond2016cvxpy} and solved using the ``\emph{Suggest-and-Improve}'' framework proposed by Park and Boyd \cite{park2017general}. It is initialized by choosing the proposal with the highest confidence for each joint. Appropriate values for the free parameters $\lambda_{\text{conf}, \text{temp}, \text{null}}$ and $\alpha$ were chosen empirically via grid search.

\subsection{Incoporating coverage priors}

The above quadratic formulation is sufficient to correct many errors in the raw output (which we later demonstrate in the experimental section), but suffers from an `overcounting' problem, in which leg joint predictions both cover the same silhouette leg region, leaving another leg empty. We therefore extend the definition of $L(A)$ to include two additional terms.

\def\silhouette{S}

\subsubsection{Silhouette coverage: $\LL{cov-sil}$}

The silhouette coverage term is designed to penalize large silhouette areas with no nearby selected joint. This term requires a precomputed set of silhouette sample points $Z \subseteq \mathbb{R}^2$, which we aim to ``cover'' as best as possible with the set of selected joints. Intuitively, the silhouette is considered well-covered if all sample points are close to \emph{some} selected joint proposal. The set $Z$ is generated from the medial axis transform (MAT)\cite{blum1967transformation} of the silhouette, $Z^{t} = \text{MAT}(\silhouette^{t})$
with a cubed loss strongly penalizing projection outside the silhouette:
\begin{equation}
\LL{cov-sil}(A^{t};X^{t},Z^{t}) = \sum_{i}\min_{j}\|Z_{i}^{t} - \hat{X}_{j}^{t}\|^3
\end{equation}

\begin{figure}[t!]
\begin{floatrow}
\ffigbox{%%%%%%%%%%%%
\def\bb{\rule{2in}{0pt}\rule{0pt}{1in}}
\def\bjb{\rule{0.5in}{0pt}\rule{0pt}{0.25in}}

\begin{center}
\scalebox{-1}[1]{\includegraphics[width=0.49\linewidth]{sil_coverage_new/approx_render_cov_cropped.jpg}}
\scalebox{-1}[1]{\includegraphics[width=0.49\linewidth]{sil_coverage_new/med_axis_overlay_error_cropped.jpg}}
% predictions taken from rs_dog frame 0100
\end{center}
}
{\caption{Silhouette coverage loss. The error (shown in red) is the the distance between the median axis transform (right) and the nearest point on an approximate rendering (left).}
\label{fig:example_errors}}
\ffigbox{
    \raisebox{1 em}{
    \centering
    \includegraphics[trim={0cm 0cm 0cm 0cm}, clip,width=0.45\linewidth]{bone_coverage/skeleton_sil_cropped.jpg}
    \includegraphics[trim={0cm 0cm 0cm 0cm},clip,width=0.45\linewidth]{bone_coverage/bone_error_overlay_cropped.jpg}
    }
}
{\caption{Bone coverage loss. One of the back-right leg joints is incorrectly assigned (left), leading to a large penalty since the lower leg bone crosses outside the dilated silhouette (right).}
\label{fig:cov-bone}
% camel 0020}
}
\end{floatrow}
\end{figure}

\subsubsection{Bone coverage: $\LL{cov-bone}$}

The bone coverage term is used to prevent bones crossing the background. The joint hierarchy is stored in a kinematic tree structure $K = \{\{j,k\} \text{ if joints } j, k \text{ are connected by a bone}\}$.
\begin{equation}
\LL{cov-bone}(A^{t};X^{t},\silhouette^{t},K) = \sum_{\{j,k\} \in K}\biggl(1 - \min_{\lambda \in \big[0:0.1:1\big]}\silhouette^{t}(\hat{X}_{j}^{t} + \lambda(\hat{X}_{j}^{t} - \hat{X}_{k}^{t}))\biggr)
\end{equation}

\subsection{Formulation as a genetic algorithm}
We minimize this more complex objective using a genetic algorithm (GA)\cite{holland1992adaptation}. A genetic algorithm is a method for solving optimization problems using a natural selection process that mimics biological evolution. Unlike the quadratic program described previously, the GA optimization procedure relies on crossover and mutation, rather than by calculating derivatives. In some cases (and as shown in this work), this can lead to much faster convergence properties. The following components are required:

\ss{Genes.} Candidate solutions to the optimization problem are referred to as a set of `genes'. In this case, a gene is the asssignment array $A$ although is represented as a vector of $J$ integers (indicating the proposal ID), rather than one-hot encodings.

\ss{Initial population.} Genes are constructed subject to the constraints given in \Cref{eq:quadprog}. The genetic algorithm is initialized with a population size of 128 genes. Of these, the first 32 are set equal to the max confidence solutions given by the network, which speeds up convergence. The remaining 96 genes are generated by selecting a random proposal for each joint.

\ss{Fitness function.} A fitness function is used to evaluate the quality of a gene. In this setting, the fitness function is precisely the energy $L(A)$ given above which minimizes to yield an optimal skeleton configuration.

\ss{Crossover.} Crossover is a fundemental operation common to evolutionary methods which defines the process of combining the information of two `parent' genes to generate an offspring gene. \emph{Single-point} crossover operates by slicing parent genes each into two parts, and combining first and second parts from different parents to yield the next generation. Following standard practice, the crossover point is randomly selected for each combination.

\ss{Mutation.} Analogous to biological mutation and used to maintain diversity among genes, each gene is assigned some probability of undergoing a {\em mutation}. If a gene
is selected for mutation, between 1 and 4 joints have new proposals randomly assigned.

The genetic algorithm described has weights set empirically and is run for 1000 generations. Examples of errors corrected by the additional coverage energy terms are shown in Fig.~\ref{fig:example_errors} and Fig.~\ref{fig:cov-bone}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\documentclass[12pt,preprint]{aastex}
\documentclass[iop, twocolumn, tighten]{emulateapj}
\usepackage{amsmath}
\usepackage{color}
\definecolor{grey}{rgb}{0.5,0.5,0.5}
\definecolor{red}{rgb}{0.8,0.0,0.0}
\usepackage{listings}

\newcommand{\Vb}{{\boldsymbol V}}
\newcommand{\vb}{{\boldsymbol v}}
%\newcommand{\vtb}{{\boldsymbol \tilde{v}}}
\newcommand{\vtb}{{\tilde{ \boldsymbol  v}}}
\newcommand{\Tf}{{\mathcal{T}}}
\newcommand{\Xt}{{\tilde{X}}}
\newcommand{\Yt}{{\tilde{Y}}}
\newcommand{\Zt}{{\tilde{Z}}}
\newcommand{\Vp}{V_p}
\newcommand{\Vsys}{V_\mathrm{sys}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\slugcomment{}
\shortauthors{}
\shorttitle{}
\bibliographystyle{apj}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{ChopStacks}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\author{Hajime Kawahara\altaffilmark{1,2}}
\email{kawahara@eps.s.u-tokyo.ac.jp}
\altaffiltext{1}{Department of Earth and Planetary Science,
The University of Tokyo, Tokyo 113-0033, Japan}
\altaffiltext{2}{Research Center for the Early Universe,
School of Science, The University of Tokyo, Tokyo 113-0033, Japan}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
ChopStacks is a python code collection for resampling data and stacking them under the flux preservation.
\end{abstract}
\keywords{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Introduction}
We sometimes need to re-sample 1D discrete data (i.e. spectra, light curves, and so on). Of course, the interpolation is one of the major solutions to this problem. However, we sometimes need the flux preservation simultaneously, for instance, when we stack the data.My Python small package, Chopstacks, provides the functions,

\begin{itemize}
\item re-sampling 1D data and preserving flux with an arbitrary array,
\item stacking 1D data with preserving flux,
\item converting the 1D data into the log 1D data with preserving flux.
\end{itemize}

This document describes the background mathematics of Chopstacks. If you want to know how to use, read jupyter notebooks in ipynb directory.

\section{Algorithm}

I consider the situation that the stacked spectrum (we use the term of "spectrum" as an example of 1D data) $\hat{f}(x)$ consists of the sum of $M$-spectra as
\begin{eqnarray}
  \hat{f} (x) = \sum_{m=0}^{M-1} f^{(m)} (x)
\end{eqnarray}
I assume that the stacked spectrum and each spectrum have the unit of [$q/X$], where $q$ is the unit of preserved quantity $Q(x)$ and $X$ is the unit of $x$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{figure}[htb]
%  \includegraphics[width=1.0\linewidth]{FIG1.eps}
%\caption{\label{fig:1}}
%\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The stacked spectrum $\hat{f}$ is discretized to the master bin as
\begin{eqnarray}
  \hat{f}_i &\equiv& \hat{f} (\hat{x}_i)  \\
  \hat{x}_i  &:& i = 0,1..,N-1. \\
  f^{(m)}_j &\equiv& f^{(m)} (x_j)  \\
  x_j  &:& j = 0,1..,N-1.
\end{eqnarray}
The width of each bin $\hat{x}_i$ is defined by $\hat{x}_{i,s}$ and $\hat{x}_{i,e}$ as
\begin{eqnarray}
\Delta \hat{x}_i \equiv \hat{x}_{i,e} - \hat{x}_{i,s}.
\end{eqnarray}
The quantity in the $i$-th bin is written as
\begin{eqnarray}
\label{eq:tot}
\hat{Q}_i = \hat{f}_i \Delta \hat{x}_i
\end{eqnarray}
and
\begin{eqnarray}
Q^{(m)}_j = f^{(m)}_j \Delta x^{(m)}_j.
\end{eqnarray}

The problem is how to distribute the quantity in $\hat{f}_i$ from $f^{(m)}_j$. I denote the overlaped region between $\hat{x}_i$ and $x^{(m)}_j$ by $\Delta^{(m)}_{i,j}$. Then the contribution of the $j$-th bin of the $m$-th spectra to $\hat{Q}_i$  is
\begin{eqnarray}
\Delta Q^{(m)}_{i;j} &=& Q^{(m)}_j \frac{\Delta^{(m)}_{i,j}}{\Delta x^{(m)}_j} \\
                   &=& f^{(m)}_j \Delta^{(m)}_{i,j}
\end{eqnarray}

Hence, the quantity in the i-th stacked bin is
\begin{eqnarray}
\hat{Q}_i &=& \sum_{m=0}^{M-1} \sum_j \Delta Q^{(m)}_{i;j} \\
&=& \sum_{m=0}^{M-1} \sum_j f^{(m)}_j  \Delta^{(m)}_{i,j}.
\end{eqnarray}
Using equation (\ref{eq:tot}), I obtain
\begin{eqnarray}
\hat{f}_i = \sum_{m=0}^{M-1} \sum_{j=0}^{N-1} f^{(m)}_j \frac{\Delta^{(m)}_{i,j}}{\Delta \hat{x}_i}.\nonumber \\
\end{eqnarray}
Though this eqution has $M \times N$ terms, however, most of $\Delta^{(m)}_{i,j}$ is zero. Hence, it is faster if I can find the overlap $(i,j)$ which satisfies $\Delta^{(m)}_{i,j} \ne 0$ for each spectrum.
\begin{eqnarray}
\hat{f}_i = \sum_{m=0}^{M-1} \sum_{j \in \mathrm{overlap}} f^{(m)}_j \frac{\Delta^{(m)}_{i,j}}{\Delta \hat{x}_i}.
\end{eqnarray}

\subsection{implementation}

{\bf chopstacks.py} is the central program of this algorithm. The main subroutine to redistribute the value is {\bf cutput} in {\bf chopstacks.py}. The illustrative explanation of this routine is shown in Figure \ref{fig:ssp}.
The input arrays of {\bf cutput} are
\begin{itemize}
\item xw: walls of the original bins [X], i.e. (N)-array.
\item f: input value [q/X], i.e. (N-1)-array.
\item hxw: walls of the resampled bins [X], i.e. (N)-array.
\item hf: stacking array [q/X], i.e. (N-1)-array,
\end{itemize}
and the output is
\begin{itemize}
\item hf: the input hf + resampled and stacked data.
\end{itemize}
Because 1D data often use the representative values of [X] (usually, the central value of the bin), you need to set $N$-walls between the $(N-1)$-representative values of [X].
If you want to set the walls to be the center of the neighboring representative values, use {\bf buildwall} in chopstacks.py. The edge option specifies the deal of the edge values. If you specify hf=None, then chopstacks.py assumes the initial hf to be a zero array. So, if you just want to divide the 1D data, use hf=None.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[]
  \includegraphics[bb=0 0 615 1030,width=\linewidth]{stackspectrum.png}
\caption{Schematic explanation of {\bf cutput}. The code is shown in appendix. \label{fig:ssp}}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage

\section{Applications}
 Use the spectra with the unit of [$q/X$]. The preservation of $q$ can be checked using {\bf check\_preservation}.

\subsection{Log Bin Convertor}

{\bf analogbin.py} performs the resampling of $f$ [$q/X$] into the logscale of $x$ ($l \equiv \log x$) with a resolution power $R$. The relation between $\delta \log x$ and R is provided by
\begin{eqnarray}
\Delta l \equiv \delta \log x = \frac{\Delta x}{x} = \frac{1}{R}.
\end{eqnarray}
Then the number of the rebinned data is given by
\begin{eqnarray}
\hat{N} = \frac{\log \hat{x}_N  - \log \hat{x}_0}{\Delta l} = R (\log \hat{x}_N  - \log \hat{x}_0).
\end{eqnarray}

The {\bf opt} option specifies the output form. opt=0 will provide
\begin{itemize}
\item $\hat{x}$, walls of $\hat{x}$, $\hat{f}$.
\end{itemize}
opt=1 provides
\begin{itemize}
\item $\log \hat{x}$, walls of $\log \hat{x}$, $\hat{x} \hat{f}$.
\end{itemize}
The latter output is based on the preservation law
\begin{eqnarray}
f d x &=& f \left| \frac{\partial x}{\partial (\log x)}\right| d (\log) x \nonumber \\
&=&   x f d (\log x).
\end{eqnarray}
Subroutine {\bf check\_preservation} can check the degree of preservation of $[q]$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure*}[!htb]
  \includegraphics[bb=0 0 576 432,width=0.49\linewidth]{demo1.png}
  \includegraphics[bb=0 0 576 432,width=0.49\linewidth]{demo2.png}
\caption{Demonstrations of the log binning. The blue line is original data and the red one is the binned data. The right panel shows the oversampling case. \label{fig:2}}
\end{figure*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage
\appendix

\small
\begin{verbatim}
def cutput(xw,f,hxw,hf=None,silent=None):
    #see note 14.8.12 and 14.9.02
    if hf is None:
        if silent is None:
            print "Reset master data (hf) to zero."
        hf=np.zeros(len(hx))

    ind=np.digitize(hxw,xw)
    for i in range(0,len(hxw)-1):

        if ind[i]<len(xw) and ind[i+1]-1 < len(xw)+1 and ind[i]>0:  #cx1,cx2,cx3
            if ind[i]==ind[i+1]:
                hf[i]=hf[i]+f[ind[i]-1] #A0
            else:
                hf[i]=hf[i]+f[ind[i]-1]*(xw[ind[i]]-hxw[i])/(hxw[i+1]-hxw[i]) #B0
                for k in range(ind[i]+1, ind[i+1]):
                    hf[i]=hf[i]+f[k-1]*(xw[k]-xw[k-1])/(hxw[i+1]-hxw[i]) #B1
                hf[i]=hf[i]+f[ind[i+1]-1]*(hxw[i+1]-xw[ind[i+1]-1])/(hxw[i+1]-hxw[i]) #B2

        elif ind[i+1]-1 == len(xw)+1:  #right boundary criterion
            if ind[i]<ind[i+1]:
                hf[i]=hf[i]+f[ind[i]-1]*(xw[ind[i]]-hxw[i])/(hxw[i+1]-hxw[i]) #B0
                for k in range(ind[i]+1, ind[i+1]):
                    hf[i]=hf[i]+f[k-1]*(xw[k]-xw[k-1])/(hxw[i+1]-hxw[i]) #B1

        elif ind[i]==0 and ind[i+1]>0: #left boundary condition
                for k in range(ind[i]+1, ind[i+1]):
                    hf[i]=hf[i]+f[k-1]*(xw[k]-xw[k-1])/(hxw[i+1]-hxw[i]) #B1
                hf[i]=hf[i]+f[ind[i+1]-1]*(hxw[i+1]-xw[ind[i+1]-1])/(hxw[i+1]-hxw[i]) #B2

    return hf
\end{verbatim}

\end{document}


\section{Mathematical Analysis}
\subsection{Derivation of link-level successful transmission probability}
As the first step, we study the successful transmission probability $p_s(r)$ for any given link with distance $r$.
Without loss of generality, the device (i.e., the transmitter) labeled by $x_0$ is assumed to be located at the origin. The other end of this link (i.e., Base Station) with label $y_i$ is with distance $r$ away from the device. By definition, $p_s(r)$ is expressed as follows:
\begin{align}
p_{s} \left( r\right)
& =\mathbb{P}\left\lbrace \frac{H \exp(G) r^{-\gamma}}{\sum_{x_j \in \Phi_{m}} H_{x_j} \exp(G_{x_j}) r_{x_j}^{-\gamma}}  \geq \theta \right\rbrace \nonumber\\
%& =\mathbb{P} \left\lbrace \frac{H_{x_i}  r_{x_i}^{-\gamma}}{\sum_{x_j \in \Phi_{m}} H_{x_j} \exp(G_{x_j}-G_{x_i}) r_{x_j}^{-\gamma}} \geq \theta \right\rbrace \nonumber
\end{align}
Let $I=\sum_{x_j \in \Phi_{m}} H_{x_j} \exp(G_{x_j}) r_{x_j}^{-\gamma}$, which is the cumulative interference suffered for device $x_0$. Thus, we have:
\begin{align}
\label{eq:def_ps}
p_{s}\left( r \right)  &= Pr \left\lbrace H  \geq I \theta\exp(-G)r ^{\gamma}  \right\rbrace  \nonumber\\
% The follwing line can be hidden, not necessary in FINAL_VERSION
&=\mathbb{E}_{G}\left\lbrace  \mathbb{E}_{I} \left[ \exp(-\theta \exp(-g) r^{\gamma}  I ) \vert G = g\right]\right\rbrace   \nonumber\\
&= \mathbb{E}_{G} \left[ \mathcal{L}_{I}\left\lbrace \theta \exp(-G) r^{\gamma}\right\rbrace \right]
\end{align}
It is observed that the $p_s\left( r \right)$ is actually the expectation (with respect to $G$) of a conditional Laplace Transform of cumulative interference $I$ at point $\theta \exp(-G)  r^{\gamma}$.
\begin{align}
\label{eq:interferece-laplace-transform}
&\mathcal{L}_{I}\left( s \right)  = \mathbb{E}\left[ \exp(-sI)\right] \nonumber\\
&= \mathbb{E}\left[ \exp(-s\sum_{x_j \in \Phi_m} H_{x_j} \exp(G_{x_j})  r_{x_j}^{-\gamma})\right] \nonumber \\
&= \mathbb{E}_{\Phi_m} \left\lbrace \prod_{x_j\in \Phi_m} \mathbb{E}\left[\exp(-sH_{x_j} \exp(G_{x_j})  r_{x_j}^{-\gamma})\right] \right\rbrace ,
\end{align}
which is actually the Probability Generating Functional (PGFL) of a Poisson point process.
Applying Campell's theorem,
%\qsong{Here I give the wikipedia link for this theorem, in the final version, I will give a reference of one book: https://en.wikipedia.org/wiki/Campbell's_theorem_(probability)}
\begin{align}
\label{eq:laplace_trans_I}
\mathcal{L}_{I}\left( s \right) &=\exp\left\lbrace -\lambda_m \pi \mathbb{E}\left[ H^{\frac{2}{\gamma}}\exp(\frac{2}{\gamma}G)\right] \Gamma(1-\frac{2}{\gamma})s^{\frac{2}{\gamma}}  \right\rbrace
\end{align}
This result has been derived in~\cite[eq.3.20]{haenggi2009interference}. Since $H$ and $G$ are independent, we have:
% The following two formula can be removed in FINAL_VERSION
%\begin{align}
%\label{eq:lognormal_moment}
%\mathbb{E}\left[ \exp(\frac{2}{\gamma}G)\right] &= \exp \left\lbrace \left( \frac{\sqrt{2}\beta\sigma}{\gamma}\right) ^2\right\rbrace
%\end{align}
%\begin{align}
%\label{eq:expo_moment}
%\mathbb{E}\left[ H^{\frac{2}{\gamma}} \right] &= \Gamma(1+\frac{2}{\gamma})
%\end{align}
%Substituting $(\ref{eq:expo_moment})(\ref{eq:lognormal_moment})$ into $(\ref{eq:laplace_trans_I})$, we have:
\begin{align}
\mathcal{L}_{I}\left( s \right) &= \exp\left\lbrace -p\lambda_m \pi \Gamma(1+\frac{2}{\gamma}) \Gamma(1-\frac{2}{\gamma}) \exp \left( \frac{\sqrt{2}\sigma}{\gamma}\right) ^2 s^{\frac{2}{\gamma}}  \right\rbrace
\end{align}
Let $s=\theta \exp(-G) r^{\gamma}$, thus
\begin{align}
\label{eq:conditional_lp_trans}
\mathcal{L}_{I}\left\lbrace \theta \exp(-G) r^{\gamma}\right\rbrace = \exp\left\lbrace -A r^2\exp(-\frac{2}{\gamma}G)\right\rbrace ,
\end{align}
where $A=p\lambda_m\pi \Gamma(1+\frac{2}{\gamma}) \Gamma(1-\frac{2}{\gamma}) \exp \left( \frac{\sqrt{2}\sigma}{\gamma}\right) ^2 \theta^{\frac{2}{\gamma}}$, $\sigma = 0.1\ln(10)\sigma_{dB}$.

With substitution of $(\ref{eq:conditional_lp_trans})$ into $(\ref{eq:def_ps})$, it remains to calculate the expectation with respect to $G$:
\begin{align}
\label{eq:def_ps_2}
p_{s}(r) &= \mathbb{E}_{G}\left[ \exp(-A r^2 \exp(-\frac{2}{\gamma}G)) \right] \nonumber\\
&=\int_{-\infty}^{+\infty} \exp(-A r^2 e^{x})\frac{1}{\sqrt{2\pi}\frac{2}{\gamma}\sigma} \exp(-\frac{x^2}{2(\frac{2}{\gamma}\sigma)^2}) dx \nonumber\\
&=\mathbb{E}_{X}\left[ \exp(-Ar^2X)\right],
\end{align}
which is the expectation with respect to a new log-normal random variable $X \sim LN(0, \frac{2}{\gamma}\sigma)$.
% The following should be removed in FINAL_VERSION
%\begin{align}
%p_{s}(r) &= \mathbb{E}\left[ \exp(-Ar^2X)\right] \nonumber\\
%&= \mathcal{L}_{X} \left[ Ar^2\right]
%\end{align}
Now we will not consider to get the closed-form expression for $p_{s}(r)$, because in the following the form will simplify the analysis.
%现在我们先不考虑：获取P_s的解析解，因为在下一步计算会更方便
%A closed form expression of the Laplace transform of the lognormal distribution does not
%exist. According to reference~\cite{asmussen2016laplace}, the Laplace transform of a log-normal random variable can be accurately approximated as follows:
%\begin{align}
%\label{eq:laplace-transform-lognormal-form-1}
%\mathcal{L} \left\lbrace X \right\rbrace \left( s \right)
%&= \frac{\exp(-\frac{W(s \sigma_{X}^2 e^{\mu_{X}} )^2 + 2W(s \sigma_{X}^2 e^{\mu_{X}})}{2\sigma_{X}^2})}{\sqrt{1 + W(s \sigma_{X}^2 e^{\mu_{X}})}},
%\end{align}
%where $W\left( \cdot \right)$ is the Lambert W function~\cite{corless1996lambertw}, which is defined as the solution in principal branch of the
%equation $W\left(x\right) e^{W \left( x\right) }= x$.
\subsection{Derivation of outage probability over infinite plane}
\subsubsection{Nearest BS attach method}
As a comparison reference, now we consider probability $P_{f1}$ over an infinite plane using traditional method: the device attach the nearest BS.
The distance to the nearest base station, we denote this distance as $r$. Its PDF, obtained by void probability of PPP, is:
\begin{align}
\label{eq:pdf_nearest_distance}
f\left( r\right)  = 2 \pi \lambda_b  r \exp(-\lambda_b \pi r^2), r \in \left[ 0, +\infty\right]
\end{align}
In this case, it just needs to take the expectation with respect to $r$, to get the outage probability $P_{f1}$.
\begin{align}
P_{f1}&= \mathbb{E}\left[ 1-p_{s}\left(r\right) \right]  \nonumber\\
&= 1-\int_{0}^{+\infty} \mathbb{E}\left[ \exp(-A r^2 X)\right]  2 \pi \lambda_b  r \exp(-\lambda_b \pi r^2) dr \nonumber\\
&= 1 - \mathbb{E}\left[ \frac{1}{\frac{AX}{\pi\lambda_b}+1} \right] \nonumber
\end{align}
For the simplicity  of writing in the following, let $B= \frac{A}{\pi \lambda_b}$. And we now focus on the term $\mathbb{E}\left[ \frac{1}{BX+1} \right]$:
\begin{align}
\label{eq:mean_bx+1_step1}
&\mathbb{E}\left[ \frac{1}{BX+1} \right]  = \int_{-\infty}^{+\infty} \frac{1}{ Be^t+1} \cdot \frac{1}{\sqrt{2\pi} \sigma_X} \exp\left\lbrace -\frac{t^2}{2 \sigma_X^2}\right\rbrace dt \nonumber\\
&= \int_{-\infty}^{+\infty} \frac{1}{ e^t+1} \cdot \frac{1}{\sqrt{2\pi} \sigma_X} \exp\left\lbrace -\frac{(t-\ln(B))^2}{2 \sigma_X^2}\right\rbrace dt
%&= \int_{-\infty}^{+\infty} \frac{1}{1+e^{-(t-\ln(B))}} \cdot \frac{1}{\sqrt{2\pi} \sigma_X} \exp\left\lbrace -\frac{t^2}{2 \sigma_X^2}\right\rbrace dt \nonumber\\
%&= \int_{-\infty}^{+\infty} \frac{1}{1+\exp{-\left( \frac{t-\frac{\ln(B)}{\sigma_X}}{\frac{1}{\sigma_X}}\right) }} \cdot \frac{1}{\sqrt{2\pi}} \exp\left\lbrace -\frac{t^2}{2}\right\rbrace dt \nonumber\\
%& = \int_{-\infty}^{+\infty} \frac{\phi\left( t \right) }{1+\exp{-\left( \frac{t-\frac{\ln(B)}{\sigma_X}}{\frac{1}{\sigma_X}}\right) }} dt,
\end{align}
which is actually a logistic-normal integral. According to~\cite{crooks2009logistic}, this kind of integral can be accurately approximated by another reparameterized logistic function (i.e., sigmoid function). Thus,
\begin{align}
	\label{eq:analytical_result_approach_2}
	P_{f1} &= 1 -\frac{1}{1 + \exp\left\lbrace \left( 1 +\frac{\pi \sigma_X^2}{8} \right)^{-\frac{1}{2}} \ln(B) \right\rbrace} \nonumber\\
	&=1-\frac{1}{1 + B^{\left( 1 +\frac{\pi \sigma^2}{2\gamma^2} \right)^{-\frac{1}{2}}}},
\end{align}
where $B= \frac{p\lambda_m}{\lambda_b}\Gamma(1+\frac{2}{\gamma}) \Gamma(1-\frac{2}{\gamma}) \exp \left( \frac{\sqrt{2}\sigma}{\gamma}\right) ^2 \theta^{\frac{2}{\gamma}}$.
\subsubsection{BS reception diversity method}
We now study the outage probability $P_{f2}$ over an infinite plane when applying BS reception diversity. In this case, one transmission is outage if and only if none of the Base Stations in the infinite plane has received the packet. Thus, $P_{f1}$ is expressed as follows:
\begin{align}
\label{eq:definition_pf1}
P_{f2} &= \mathbb{E}\left[  \prod_{r_i \in \Phi_{b}} (1-p_{s}(r_i)) \right], \text{with } r_i \in \left[ r_{\text{min}}, +\infty\right],
\end{align}
where $r_i$ refers to the distance between the device and BS with label $i$. Note that $r_i$ should be not less than distance to the nearest BS denoted by $r_{\text{min}}$, whose probability density function is given in $(\ref{eq:pdf_nearest_distance})$. Expression $(\ref{eq:definition_pf1})$ is actually the Probability Generating FunctionaL(PGFL) of Point point process $\Phi_{b}$ for Base Station, which states for some function $f(x)$ that $\mathbb{E}\left[ \prod_{x \in \Phi}f(x) \right] = \exp(-\lambda(\int_{\mathbb{R}^2}(1-f(x))dx))$. Thus, $(\ref{eq:definition_pf1})$ can be expressed as follows:
\begin{align}
P_{f2} &= \exp\left\lbrace -\pi \lambda_{b} \mathbb{E}_{r_{\text{min}}} \left[ \int_{r_{\text{min}}}^{+\infty} p_{s}(r)rdr \right] \right\rbrace ,
\end{align}

\begin{align}
P_{f2} &= \exp\left\lbrace -\mathbb{E}_{r_{\text{min}}} \left[ \int_{r_{\text{min}}}^{+\infty} \mathbb{E}_{X}\left[ \exp(-AXr^2)\right] 2\pi\lambda_{b} rdr \right] \right\rbrace \nonumber\\
&= \exp\left\lbrace -\pi \lambda_{b} \mathbb{E}_{r_{\text{min}}} \left[\mathbb{E}_{X}\left[ \int_{r_{\text{min}}}^{+\infty}\exp(-AXr^2)dr\right] \right] \right\rbrace \nonumber\\
&= \exp\left\lbrace -\pi \lambda_{b} \mathbb{E}_{r_{\text{min}}} \left[\mathbb{E}_{X}\left[ \frac{e^{-Ar_{\text{min}}^2X} }{AX} \right] \right] \right\rbrace \nonumber\\
&= \exp\left\lbrace -\pi \lambda_{b} \mathbb{E}_{X} \left[\mathbb{E}_{r_{\text{min}}}\left[ \frac{e^{-Ar_{\text{min}}^2X} }{AX} \right] \right] \right\rbrace \nonumber\\
&= \exp\left\lbrace -\mathbb{E}_{X} \left[ \frac{1}{\left( \frac{AX}{\pi\lambda_b}\right)\left( \frac{AX}{\pi\lambda_b}+1\right) }\right] \right\rbrace \nonumber\\
&= \exp\left\lbrace \mathbb{E}_{X} \left[ \frac{1}{BX + 1}\right] - \mathbb{E}_{X} \left[ \frac{1}{BX}\right]   \right\rbrace
\end{align}

\begin{align}
	\mathbb{E}_{X} \left[ \frac{1}{BX + 1}\right] = \frac{1}{1 + B^{\left( 1 +\frac{\pi \sigma^2}{2\gamma^2} \right)^{-\frac{1}{2}}}}
\end{align}
\begin{align}
	\mathbb{E}_{X} \left[ \frac{1}{BX}\right] =-\frac{1}{\frac{p\lambda_m}{\lambda_b}\Gamma(1+\frac{2}{\gamma}) \Gamma(1-\frac{2}{\gamma})\theta^{\frac{2}{\gamma}}}
\end{align}
Hence, we have:
\begin{align}
	\label{eq:analytical_result_approach_1}
	P_{f2} &= \exp(\frac{1}{1 + B^{\left( 1 +\frac{\pi \sigma^2}{2\gamma^2} \right)^{-\frac{1}{2}}}}-\frac{1}{B\exp\left\lbrace  -\left(\frac{\sqrt{2}\sigma}{\gamma}\right) ^2 \right\rbrace } )，
\end{align}
where $B= \frac{p\lambda_m}{\lambda_b}\Gamma(1+\frac{2}{\gamma}) \Gamma(1-\frac{2}{\gamma}) \exp \left( \frac{\sqrt{2}\sigma}{\gamma}\right) ^2 \theta^{\frac{2}{\gamma}}$.
We surprisingly observe that with BS reception diversity, the limit case, the outage probability has the same formula with the case where just Rayleigh fading is considered. But in practical system, we still observe that shadowing has improved the outage probability, since the wireless network cannot be infinite and the population is always limited.


\section{Equations/Discretizations}
\begin{frame}{Equations \ttnc{BART} solves transport equation}
	\begin{block}{\ttnc{BART} is solving multi-group transport equation with discrete-ordinates in angle}
		\begin{itemize}
			\item Transport equation in operator form with:
			\begin{align}
			&\bfscr{T}_\mm{g,m}\psigm+\bfscr{C}_\mm{g,m}\psigm=\sum\limits_{\substack{1\leq\mm{g'}\leq\mm{G}\\1\leq\mm{m'}\leq\mm{M}}}\left(\bfscr{S}_{\mm{g',m'}\to\mm{g,m}}+\bfscr{F}_{\mm{g',m'}\to\mm{g,m}}\right)\psigmp\\
			&\psigm=\psigm^\mm{inc},\ \vr\in\pd,\ \ndom<0\nno
			\end{align}
%			\begin{center}
				$\bfscr{T}$: Streaming operator\\
				$\bfscr{C}$: Collision operator, $\sigt$\\
				$\bfscr{S}$: Scattering operator\\
				$\bfscr{F}$: Fission operator\\
				$\bfscr{\psi}$: Angular flux\\
				$\mm{g,\ g'}$: Group indicies\\
				$\mm{m,\ m'}$: Angular indices\\
				$\mm{G,\ M}$: Numbers of groups and directions
%			\end{center}
			\item The formulation generalizes to diffusion and nonlinear diffusion
		\end{itemize}
	\end{block}
\end{frame}

\begin{frame}{\ttnc{BART} solves first-order form of transport equations}
	\begin{block}{First-order form and DFEM discretization in space}
		\begin{itemize}
			\item Transport equation has differential order of 1\ in the streaming term, we refer it to as ``first-order" form (FOF)
			\begin{align}
				\ome_\mm{m}\cdot\nabla\psigm+\sigma_{\mm{t,g}}\psigm=\mathcal{Q}_\mm{g,m}\left({\bf \Psi}\right)
			\end{align}
			\begin{align}
			   \mq_\mm{g,m}:=\sum\limits_{\substack{1\leq\mm{g'}\leq\mm{G}\\1\leq\mm{m'}\leq\mm{M}}}\left(\bfscr{S}_{\mm{g',m'}\to\mm{g,m}}+\bfscr{F}_{\mm{g',m'}\to\mm{g,m}}\right)\psigmp
			\end{align}
			\item FOF is discretized using DFEM in \ttt{BART}: given polynomial function space $\mv$,\ $\forall\psigm^*\in\mv$,\ find $\psigm\in\mv$,\ s.t.
			\begin{align}
			\sum\limits_{\mm{e}\in\md}\left[\left(-\omemn\psigms,\psigm\right)+\left(\psigms,\sigtg\psigm\right)\right]_\mm{e}+\sum\limits_{\mm{f}\in\md}\left|\ndom\right|\left<\jjump{\psigms},\tilde{\psi}_\mm{g,m}\right>_\mm{f}\nno\\
			+\sum\limits_{\substack{\mm{f}\in\pd\\\ndom>0}}\ndom\left<\psigms,\psigm\right>_\mm{f}=\sum\limits_{\mm{e}\in\md}\left(\psigms,\mqgm\right)_\mm{e}+\sum\limits_{\substack{\mm{f}\in\pd\\\ndom<0}}\absndom\left<\psigms,\psigm^\mm{inc}\right>_\mm{f}
			\end{align}
		\end{itemize}
	\end{block}
\end{frame}

\begin{frame}{\ttnc{BART} solves second-order forms of transport equations}
	\begin{block}{Second-order forms (SOF) of transport equations}
		\begin{itemize}
			\item FOF can be cast to diffusion-like equations having streaming term with differential order of 2
			\item The casting allows for use of CFEM
			\item BART solves even-parity equation and self-adjoint angular flux equation (SAAF)
		\end{itemize}
	\end{block}
	\begin{block}{SOF example: SAAF}
		\begin{itemize}
			\item SAAF equation
			\begin{align}
				-\omemn\frac{1}{\sigtg}\omemn\psigm+\sigtg\psigm=\mqgm-\frac{1}{\sigtg}\omemn\mqgm
			\end{align}
			\item CFEM discretization
			\begin{align}
				\left(\omemn\psigms,\frac{1}{\sigtg}\omemn\psigm\right)_\md+\left(\psigms,{\sigtg}\psigm\right)_\md+\sum\limits_{\substack{\mm{f}\in\pd\\\ndom>0}}\ndom\left<\psigms,\psigm\right>_\mm{f}\nno\\
				=\left(\psigms+\frac{\omemn\psigms}{\sigtg},\mqgm\right)_\mm{\md}+\sum\limits_{\substack{\mm{f}\in\pd\\\ndom<0}}\absndom\left<\psigms,\psigm^\mm{inc}\right>_\mm{f}
			\end{align}
		\end{itemize}
	\end{block}
\end{frame}

\begin{frame}{\ttnc{BART} solves diffusion and nonlinear diffusion}
	\begin{block}{BART solves diffusion equation}
		\begin{itemize}
			\item Diffusion could be solved within the framework of \ttt{BART}
			\begin{equation}
				-\nabla\frac{1}{3\sigtg}\cdot\nabla\phig+\sigtg\phig=\mq_\mm{g}:=\sum\limits_\mm{g'}\left[\left(\sigsgg+\frac{\chig\nu\sigma_\mm{f,g'}}{k_\mm{eff}}\right)\phigp\right]
			\end{equation}
		\end{itemize}
	\end{block}

	\begin{block}{Nonlinear diffusion for acceleration (NDA)}
		\begin{itemize}
			\item Diffusion can also be written in P$_1$\ form
			\begin{equation}
			\nabla\cdot\vec{J}_\mm{g}+\sigtg\phig=\mq_\mm{g},\quad\vec{J}_\mm{g}=-\frac{1}{3\sigtg}\nabla\phi_\mm{g}\ (\mm{Fick's\ law})
			\end{equation}
			\item NDA is derived from correcting Fick's law using transport corrections
			\item \boxed{Correction} to preserve current is derived from transport high-order solutions (HO)
			\begin{equation}
				\vec{J}_\mm{g}=-\frac{1}{3\sigtg}\nabla\phi_\mm{g}+\boxed{\frac{\sum\limits_{\mm{m}<M}w_\mm{m}\ome_\mm{m}\omemn\psigm^\mm{HO}+\frac{1}{3\sigtg}\nabla\phi_\mm{g}^\mm{HO}}{\phi_\mm{g}^\mm{HO}}\phi}
			\end{equation}
		\end{itemize}
	\end{block}
\end{frame}

\begin{frame}{\ttnc{BART} solves diffusion and nonlinear diffusion (cont'd)}
	\begin{block}{\ttnc{BART} solves in multiple ways}
		\begin{itemize}
			\item CFEM is natural to use to solve diffusion/nonlinear diffusion
			\item DFEM is realizable through penalty method
			\begin{itemize}
				\item Useful when accelerating transport solves with DFEM
			\end{itemize}
			\item A hybrid-FEM is a near-future project for NDA in P$_1$-like form
			\begin{itemize}
				\item Piece-wise constant test function for $\phig$\ while Raviat-Thomas test functions for $\vec{J}_\mm{g}$
			\end{itemize}
		\end{itemize}
	\end{block}
\end{frame}


\section{Semantics}
\label{semantics}

The semantics of LPADs for the case of programs without functions symbols can be given
as follows. An LPAD defines a probability distribution over normal logic programs called
\emph{worlds}. A world is obtained from an LPAD by first grounding it, by
selecting a single head atom for each ground clause and by including in the world
the clause with the selected head atom and the body.
The probability of a world is the product of the probabilities associated to the
heads selected.
The probability of a ground atom (the query) is given by the sum of the probabilities
of the worlds where the query is true.

If the LPAD contains function symbols, the definition is more complex, see
\cite{DBLP:journals/ai/Poole97,DBLP:journals/jair/SatoK01,Rig15-PLP-IW}.

For the semantics of programs with continuous random variables, see \cite{TLP:8688161}
that defines the probability
space for $N$ continuous random variables by considering the Borel $\sigma$-algebra
over $\mathbb{R}^N$
and defines a Lebesgue measure on this set as the probability measure.
The probability space is lifted to cover the entire program using the least model semantics of constraint logic programs. Alternatively,
\cite{Nitti2016} defines the semantics of distributional clauses by resorting to
a stochastic $Tp$ operator.
\verb|cplint| allows more freedom than distributional clauses in the use of continuous random variables
in expressions, for example
\href{http://cplint.eu/e/kalman_filter.pl}{\texttt{kalman\_filter.pl}} would not be allowed by distributional clauses.