\chapter{A refined complexity calculus model: r-Complexity }

\section{Introduction and Motivation}
This chapter will present a new approach in the field of algorithm's computational complexity. Similar to the conventional asymptotic notations proposed in the literature,  \textit{rComplexity} will be expressed as a function $f:\mathbb{N}\longrightarrow\mathbb{R}$, where the function is characterized by the size of the input, while the evaluated value $f(n)$, for a given input size $n$, represents the amount of resources needed in order to compute the result.

This new calculus model aims to produce new asymptotic notations that offer better complexity feedback for similar algorithms, providing subtle insights even for algorithms that are part of the same conventional complexity class $\Theta(g(n))$ denoted by an arbitrary function $g:\mathbb{N}\longrightarrow\mathbb{R}$, in the definition of \textit{Bachmann–Landau} notations. The additional information contained by \textit{rComplexity} classes consists of the fine-granularity obtained by the model based on a refined clustering strategy for functions that used to belong to the same \textit{Bachmann–Landau} group in different complexity classes, established on asymptotic constant analysis.

The classical complexity calculus model is a long-established, verified metric of evaluating an algorithm's performance and it is a valuable measurement in estimating feasibility of computing a considerable algorithm.

However, the model has few shortfalls in making discrepancy between similar algorithms with two similar complexity functions, $v,w:\mathbb{N}\longrightarrow\mathbb{R}$, such that $v(n),w(n) \in \Theta(g(n))$. In order to highlight the lack of distinction, suppose two algorithms $Alg1$ and $Alg2$ that solve exactly the same problem, with the following complexity functions:
$f_{1},f_{2}:\mathbb{N}\longrightarrow\mathbb{R}$ where $f_{1} = x \cdot f_{2}$, with $x \in \mathbb{R}_{+}, \ x > 1$.
If \[ f_{2} \in \Theta(g(n)) \Rightarrow \exists c_{1}, c_{2} \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\ \ c_{1} \cdot g(n) \leq f_{2}(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} \]
Therefore, $f_{1} \in \Theta(g(n))$, as there exists $ c_{1}^{'}, c_{2}^{'} \in \mathbb{R}^{*}_{+}, n_{0}^{'} \in \mathbb{N}^{*}$ such that $ c_{1}^{'} = x \cdot c_{1} $, $ c_{2}^{'} = x \cdot c_{2} $ and $n_{0}^{'} = n_{0}$ and  $\forall n \geq n_{0}^{'} $ :  $c_{1}^{'} \cdot g(n) \leq f_{1}(n) \leq c_{2}^{'} \cdot g(n)$.
\\ \\
Remark that even if $f_{1} > f_{2}$, both complexity functions are part of the same complexity class. This observation implies that two algorithms, whose complexity functions can differ by a constant, even as high as $2020^{2020}$, are part of the same complexity class, even if the actual run-time might differ by over $6676$ orders of magnitude.

\begin{remark}
    The discussion of how big numbers really are is fruitless and worthless. In this discussions the only bound can be expressed by the idiom \textit{sky is the limit}, as numbers such as $2020 \uparrow \uparrow 2020$ defer by colossal orders of magnitude. Knuth's up\-arrow notation is a method of notation for very large integers, introduced by Donald Knuth in 1976. ~\cite{knuth1976mathematics} The idea is based on the fact that multiplication can be viewed as iterated addition and exponentiation as iterated multiplication.
\end{remark}

For comparison, only a $30$ magnitude order between the two complexity functions $f_{1} = 10^{30} \cdot f_{2}$, signify
that if for a given input $n$, if $Alg2$ ends execution in 1 \textit{attosecond} ($10^{-9}$ part of a nanosecond), then $Alg1$ is expected to end execution in about 3 \textit{millenniums}. Despite of the colossal difference in time, classical complexity model is not perceptive between algorithms whose complexity functions differs only through constants. \\ \\
\textbf{\textit{rComplexity}} calculus aims to clarify this issue by taking into deep analysis the preeminent constants that can state major improvements in an algorithm's complexity and can have tremendous effect over total execution time.

\section{Adjusting the Bachmann–Landau notations for rComplexity Calculus}
The following notations and names will be used for describing the asymptotic behavior of a algorithm's complexity characterized by a function, $f:\mathbb{N}\longrightarrow\mathbb{R}$. \\
We define the set of all complexity calculus $\mathcal{F}= \lbrace f:\mathbb{N}\longrightarrow\mathbb{R} \rbrace$
\\Assume that $n, n_{0}\in\mathbb{N}$. Also, we will consider an arbitrary complexity function $g \in \mathcal{F}$.
Acknowledge the following notations $\forall r \neq 0$:

\begin{definition}
    \textbf{Big \textit{r-}Theta}: This set defines the group of mathematical functions similar in magnitude with  $g(n)$ in the study of asymptotic behavior. A set-based description of this group can be expressed as:
    \[\begin{split}
          \Theta_{r}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} \rbrace
    \end{split} \]
\end{definition}

\begin{definition}
    \textbf{Big \textit{r-}O}: This set defines the group of mathematical functions that are known to have a similar or lower
    asymptotic performance in comparison with  $g(n)$. The set of such functions is defined as it follows:
    \[\mathcal{O}_{r}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  r<c, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) \leq c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}

\begin{definition}
    \textbf{Big \textit{r-}Omega}: This set defines the group of mathematical functions that are known to have a similar or higher asymptotic performance in comparison with  $g(n)$. The set of all function is defined as:
    \[\Omega_{r}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+}\ s.t. \ c < r, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) \geq c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}

\begin{definition}
    \textbf{Small \textit{r-}O}:
    This set defines the group of mathematical functions that are known to have a humble
    asymptotic performance in comparison with  $g(n)$. The set of such functions is defined as it follows:
    \[o_{r}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) < c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}
\begin{remark}
    This set is defined for symmetry of the model and it is equal with the set defined by \textbf{Small O} notation in \textit{Bachmann–Landau notations}, as the definition is independent on $r$.
\end{remark}

\begin{definition}
    \textbf{Small \textit{r-}Omega}:
    This set defines the group of mathematical functions that are known to have a commanding asymptotic performance in comparison with  $g(n)$.
    The set of such functions is defined as it follows:
    \[\omega_{r}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) > c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}
\begin{remark}
    This set is defined for symmetry of the model and it is equal with the set defined by \textbf{Small Omega} notation in \textit{Bachmann–Landau notations}, as the definition is independent on $r$.
\end{remark}

\section{Asymptotic Analysis}
Calculus in $rComplexity$ can be performed either using limits of sequences or limits of functions. Consider any two complexity functions $f,g:\mathbb{N}_{+}\longrightarrow\mathbb{R}_{+}$.

\begin{theorem}
    Admittance of a function $f$ in \textbf{Big \textit{r-}Theta} class defined by a function $g$:
    \[ f \in \Theta_{r}(g(n)) \Leftrightarrow \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = r \]
\end{theorem}

\begin{theorem}
    Admittance of a function $f$ in \textbf{Big \textit{r-}O} class defined by a function $g$:
    \[ f \in \mathcal{O}_{r}(g(n)) \Leftrightarrow \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = l,\ l \in \left[ 0, r \right] \]
\end{theorem}

\begin{theorem}
    Admittance of a function $f$ in \textbf{Big \textit{r-}Omega} class defined by a function $g$:
    \[ f \in \Omega_{r}(g(n)) \Leftrightarrow \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = l,\ l \in \left[ r, \infty \right) \]
\end{theorem}

\begin{theorem}
    Admittance of a function $f$ in \textbf{Small \textit{r-}O} class defined by a function $g$:
    \[ f \in o_{r}(g(n)) \Leftrightarrow \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \]
\end{theorem}

\begin{theorem}
    Admittance of a function $f$ in \textbf{Small \textit{r-}Omega} class defined by a function $g$:
    \[ f \in \omega_{r}(g(n)) \Leftrightarrow \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \]
\end{theorem}

\section{Common properties}
This chapter will present new implications in terms of \textit{reflexivity, transitivity, symmetry and projections} compared with conventional \textit{Bachmann–Landau notations}.

\begin{theorem}
    Reflexivity in $rComplexity$: - Big r-Theta notation  \[ f \in \Theta_{r} \left( \frac{1}{r} \cdot f(n) \right)\ \forall r \neq 0 \]
\end{theorem}
\begin{proof}
    Using the definition of $ \Theta_{r}(f(n))$
    \[\begin{split}
          \Theta_{r}(f(n)) = \lbrace f' \in \mathcal{F}\ |\ \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot f(n) \leq f'(n) \leq c_{2} \cdot f(n)\ ,\  \forall n \geq n_{0} \rbrace
    \end{split} \]
    Using substitution $ f(n) \longleftarrow \frac{1}{r} \cdot f(n)$
    \[\begin{split}
          \Theta_{r} \left( \frac{1}{r} \cdot f(n) \right) = \lbrace f' \in \mathcal{F}\ |\ \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \frac{1}{r} \cdot \ c_{1} \cdot f(n) \leq  f'(n) \leq \frac{1}{r} \cdot c_{2} \cdot f(n)\ ,\  \forall n \geq n_{0} \rbrace
    \end{split} \]
    By choosing $c_{1}' = \frac{1}{r} \cdot c_{1}, c_{2}' = \frac{1}{r} \cdot c_{2}$
    \[\begin{split}
          \Theta_{r} \left( \frac{1}{r} \cdot f(n) \right) = \lbrace f' \in \mathcal{F}\ |\ \forall c_{1}', c_{2}' \in \mathbb{R}^{*}_{+} \ s.t. c_{1}'< 1 < c_{2}' , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\  \ c_{1}' \cdot f(n) \leq  f'(n) \leq c_{2}' \cdot f(n)\ ,\  \forall n \geq n_{0} \rbrace
    \end{split} \]
    For any $x \in \mathbb{R}^{*}$,  $\forall c_{1}, c_{2}\ \ s.t. c_{1} \leq 1 \leq c_{2}$, we have $ x \cdot c_{1} \leq x \leq x \cdot c_{2} $. \\
    Thus, if $f:\mathbb{N}\longrightarrow\mathbb{R}$, $\forall c_{1}, c_{2}\ \  s.t. c_{1} \leq 1 \leq c_{2}$, we have $f(n) \cdot c_{1} \leq f(n) \leq f(n) \cdot c_{2}\ \ \forall n \in \mathbb{N}^{*}$ or $f(n) \cdot c_{1} \geq f(n) \geq f(n) \cdot c_{2}\ \ \forall n \in \mathbb{N}^{*}$\\

    Therefore:
    \[\forall c_{1}', c_{2}' \in \mathbb{R}^{*}_{+} \ s.t. c_{1}'< 1 < c_{2}' , \exists n_{0} = 1 s.t.\  \ c_{1}' \cdot f(n) \leq  f(n) \leq c_{2}' \cdot f(n)\ ,\  \forall n \geq n_{0}=1 \Rightarrow \]
    \[ f \in \Theta_{r} \left( \frac{1}{r} \cdot f(n) \right)\ \forall r \neq 0 \]
\end{proof}

\begin{theorem}
    Reflexivity in $rComplexity$: - Big r-O notation
    \[ f \in \mathcal{O}_{r} \left( x \cdot f(n) \right)\ \forall x \geq \dfrac{1}{r} \]
\end{theorem}
\begin{proof}
    :
    Using the definition of $ \mathcal{O}_{r}(f(n))$
    \[\mathcal{O}_{r}(f(n)) = \lbrace f' \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  r<c, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f'(n) \leq c \cdot f(n),\  \forall n \geq n_{0} \rbrace\]
    Using substitution $ f(n) \longleftarrow x \cdot f(n), \forall x \geq \dfrac{1}{r}, \forall r \neq 0$
    \[\mathcal{O}_{r}(x \cdot f(n)) = \lbrace f' \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  r<c, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f'(n) \leq c \cdot x \cdot f(n),\  \forall n \geq n_{0} \rbrace\]
    If $r<c$ and $x \geq \dfrac{1}{r}$, then $c \cdot x \geq 1, \forall r,c,x\ \ r \neq 0$
    Thus, if $f:\mathbb{N}\longrightarrow\mathbb{R}$, we have $ f(n) \leq 1 \cdot f(n) \leq  c \cdot x\cdot f(n) \ \ \forall n \in \mathbb{N}^{*}$.

    Therefore:
    \[\forall x \geq \dfrac{1}{r}, \ \forall c \in \mathbb{R}^{*}_{+} \ s.t.\  r<c , \exists n_{0} = 1 s.t.\  \  f(n) \leq  c \cdot x\cdot f(n) \ \ \forall n \in \mathbb{N}^{*} ,\  \forall n \geq n_{0}=1 \Rightarrow \]
    \[ f \in \mathcal{O}_{r} \left( x \cdot f(n) \right)\ \forall x \geq \dfrac{1}{r} \]
\end{proof}

\begin{theorem}
    Reflexivity in $rComplexity$: - Big r-Omega notation
    \[ f \in \Omega_{r} \left( x \cdot f(n) \right)\ \forall x \leq \dfrac{1}{r} \]
\end{theorem}
\begin{proof}
    Using the definition of $\Omega_{r}(f(n))$
    \[\Omega_{r}(f(n)) = \lbrace f' \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  c<r, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f'(n) \geq c \cdot f(n),\  \forall n \geq n_{0} \rbrace\]
    Using substitution $ f(n) \longleftarrow x \cdot f(n), \forall x \leq \dfrac{1}{r}, \forall r \neq 0$
    \[\Omega_{r}(x \cdot f(n)) = \lbrace f' \in \mathcal{F}\ |\ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  c<r, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f'(n) \geq x \cdot c \cdot f(n),\  \forall n \geq n_{0} \rbrace\]
    If $r>c$ and $x \leq \dfrac{1}{r}$, then $c \cdot x \leq 1, \forall r,c,x\ \ r \neq 0$

    Therefore:
    \[\forall x \leq \dfrac{1}{r}, \ \forall c \in \mathbb{R}^{*}_{+} \ s.t.\  c<r , \exists n_{0} = 1 s.t.\  \  f(n) \geq  c \cdot x\cdot f(n) \ \ \forall n \in \mathbb{N}^{*} ,\  \forall n \geq n_{0}=1 \Rightarrow \]
    \[ f \in \Omega_{r} \left( x \cdot f(n) \right)\ \forall x \leq \dfrac{1}{r} \]
\end{proof}

\begin{remark}
    Reflexivity does not hold in $rComplexity$ for small r-O notation.
    \[ f \notin o_{r}(f(n)) \]

    The reflexivity does not hold for Small \textit{r-}o and Small \textit{r-}Omega, as these two sets are equal with the classical sets defined in \textit{Bachmann–Landau notations}
\end{remark}

\begin{remark}
    Reflexivity does not hold in $rComplexity$ for small r-Omega notation.
    \[ f \notin \omega_{r}(f(n)) \]

    The reflexivity does not hold for Small \textit{r-}o and Small \textit{r-}Omega, as these two sets are equal with the classical sets defined in \textit{Bachmann–Landau notations}
\end{remark}

\begin{theorem}
    Transitivity in $rComplexity$ - Big r-Theta notation:  \\  $ f \in \Theta_{r}(g(n)), g \in \Theta_{r'}(h(n)) \Rightarrow  f \in \Theta_{r \cdot r'}(h(n))$
\end{theorem}
\begin{proof}
    If $ f \in \Theta_{r}(g(n)) \Rightarrow \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} $ \\
    If $ g \in \Theta_{r'}(h(n)) \Rightarrow \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r' < c_{2} , \exists n_{0}' \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot h(n) \leq g(n) \leq c_{2} \cdot h(n)\ ,\  \forall n \geq n_{0}' $ \\
    Thus, if $ f \in \Theta_{r}(g(n))$ and $ g \in \Theta_{r'}(h(n))$ \\ $\Rightarrow \forall c_{1}, c_{2}, c_{1}', c_{2}' \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r < c_{2}, c_{1}' < r' < c_{2}' , \exists n''_{0}=max(n_{0}, n'_{0}) \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1}' \cdot c_{1} \cdot h(n) \leq c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n) \leq c_{2}' \cdot c_{2} \cdot h(n)\ ,\  \forall n \geq n''_{0} $ \\
    Let $c_{1}'' = c_{1} \cdot c_{1}' , c_{2}'' = c_{2} \cdot c_{2}'$. \\
    Then, $c_{1}'' < r \cdot r' < c_{2}''$ \\
    Therefore, $\forall c_{1}'', c_{2}'' \in \mathbb{R}^{*}_{+} \ s.t. c_{1}'' < r \cdot r' < c_{2}'' \ \ \exists n''_{0}=max(n_{0}, n'_{0}) \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1}'' \cdot h(n) \leq f(n) \leq c_{2}'' \cdot h(n)\ ,\  \forall n \geq n''_{0} \Rightarrow f \in \Theta_{r \cdot r'}(h(n))$ \\

\end{proof}

All other Transitivity properties hold as well in $rComplexity$ Calculus. The proof is similar with the above.
\begin{theorem}
    Transitivity in $rComplexity$ - Big r-O notation

    $ f \in \mathcal{O}_{r}(g(n)), g \in \mathcal{O}_{r}(h(n)) \Rightarrow  f \in \mathcal{O}_{r \cdot r'}(h(n))$
\end{theorem}
\begin{theorem}
    Transitivity in $rComplexity$ - Big r-Omega notation

    $ f \in \Omega_{r}(g(n)), g \in \Omega_{r}(h(n)) \Rightarrow  f \in \Omega_{r \cdot r'}(h(n))$
\end{theorem}
\begin{theorem}
    Transitivity in $rComplexity$ - Small r-O notation

    $ f \in o_{r}(g(n)), g \in o_{r'}(h(n)) \Rightarrow  f \in o_{r \cdot r'}(h(n))$
\end{theorem}
\begin{theorem}
    Transitivity in $rComplexity$ - Small r-Omega notation

    $ f \in \omega_{r}(g(n)), g \in \omega_{r'}(h(n)) \Rightarrow  f \in \omega_{r \cdot r'}(h(n))$
\end{theorem}

\begin{theorem}
    \textbf{Symmetry in $rComplexity$:}  \\  $ f \in \Theta_{r}(g(n)) \Rightarrow g \in \Theta_{\frac{1}{r}}(f(n)) $
\end{theorem}

\begin{proof}
    $ f \in \Theta_{r}(g(n)) \Rightarrow \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} $
    \\ Using the substitution $c_{1}' = \dfrac{c_{1}}{r}, c_{2}' = \dfrac{c_{2}}{r} \Rightarrow \forall c_{1}', c_{2}' \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < 1 < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1}' \cdot g(n) \leq \dfrac{1}{r} \cdot f(n) \leq c_{2}' \cdot g(n)\ ,\  \forall n \geq n_{0} $
    \\ The previous inequality can be re-written as:
    \[\begin{cases}
          g(n) \leq \dfrac{1}{c_{1}'} \cdot \dfrac{1}{r} \cdot f(n) \\ g(n) \geq \dfrac{1}{c_{2}'} \cdot \dfrac{1}{r} \cdot f(n)
    \end{cases}\]
    \\ Using notation $c_{1}'' = \dfrac{1}{c_{2}'}, c_{2}'' = \dfrac{1}{c_{1}'}$ the inequality becomes:  \\
    $\forall c_{1},'' c_{2}'' \in \mathbb{R}^{*}_{+} \ s.t. c_{1}'' < r < c_{2}'' , \exists n_{0} \in \mathbb{N}^{*}\ $
    \[ {c_{1}''} \cdot \dfrac{1}{r} \cdot f(n) \leq g(n) \leq {c_{2}'} \cdot \dfrac{1}{r} \cdot f(n)\ \ \forall n \geq n_{0} \]
    Thus, using the definition of $ \Theta_{r}(f(n))$, $ f \in \Theta_{r}(g(n)) \Rightarrow g \in \Theta_{\frac{1}{r}}(f(n)) $.
\end{proof}

\begin{theorem}
    \textbf{Transpose symmetry in $rComplexity$:}  \\  $ f \in \mathcal{O}_{r}(g(n)) \Leftrightarrow g \in \Omega_{\frac{1}{r}}(f(n)) $
\end{theorem}

\begin{proof}
    Using the property of $ \mathcal{O}_{r}(f(n))$ and $f \in \mathcal{O}_{r}(g(n))$:
    \begin{enumerate}[label=(\roman*)]
        \item
        \[\mathcal{O}_{r}(g(n)) \supseteq \lbrace f \rbrace \]
        \item
        \[ \ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  r<c, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  g(n) \leq c \cdot f(n),\  \forall n \geq n_{0} \]
    \end{enumerate}
    We can rewrite the previous relation as following:
    \[ \ \forall c  \in \mathbb{R}^{*}_{+} \ s.t.\  r<c, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  \dfrac{1}{c} \cdot g(n) \leq f(n),\  \forall n \geq n_{0} \]
    Substituting the constant $ c' = \dfrac{1}{c}$:
    \[ \ \forall c'  \in \mathbb{R}^{*}_{+} \ s.t.\  c' < \dfrac{1}{r}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) \geq c' \cdot g(n) ,\  \forall n \geq n_{0} \]

    Therefore:
    \[\Omega_{\frac{1}{r}}(f(n)) \supseteq \lbrace g \rbrace \]
\end{proof}
\begin{theorem}
    \textbf{Transpose symmetry in $rComplexity$:}
    $ f \in o_{r}(g(n)) \Leftrightarrow g \in \omega_{\frac{1}{r}}(f(n)) $
\end{theorem}

\begin{proof}
    The Transpose symmetry hold for Small \textit{r-}o and Small \textit{r-}Omega, as these two sets are equal with the classical sets defined in \textit{Bachmann–Landau notations}
\end{proof}

\begin{theorem}
    \textbf{Projection in $rComplexity$:}  \\  $ f \in \Theta_{r}(g(n)) \Leftrightarrow f \in \mathcal{O}_{r}(g(n)), f \in \Omega_{r}(g(n)) $
\end{theorem}
\begin{proof}
    The proof is straightforward, usind the definitions of \textbf{Big \textit{r-}Theta}, \textbf{Big \textit{r-}O} and \textbf{Big \textit{r-}Omega}. \\
    " $\Rightarrow$ " $ f \in \Theta_{r}(g(n)) \Rightarrow f \in \mathcal{O}_{r}(g(n)), f \in \Omega_{r}(g(n)) $
    Using the definition of $\Theta_{r}(g(n))$, $ f \in \Theta_{r}(g(n)) \Rightarrow$
    \[\forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} \]
    By splitting the inequality:
    \[\begin{cases}
          \forall c_{1} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r , \exists n_{0} \in \mathbb{N}^{*}\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) ,\  \forall n \geq n_{0} \\ \forall  c_{2} \in \mathbb{R}^{*}_{+} \ s.t. r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ s.t.\ f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0}
    \end{cases}\]
    Therefore:
    \[\begin{cases}
          f \in \mathcal{O}_{r}(g(n)) \\ f \in \Omega_{r}(g(n))
    \end{cases}\]

    " $\Leftarrow$ " $ f \in \mathcal{O}_{r}(g(n)), f \in \Omega_{r}(g(n)) \Rightarrow f \in \Theta_{r}(g(n)) $
    Using the definitions of \textbf{Big \textit{r-}O} and \textbf{Big \textit{r-}Omega}:
    \[\begin{cases}
          \forall c_{1} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r , \exists n_{0} \in \mathbb{N}^{*}\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) ,\  \forall n \geq n_{0} \\ \forall  c_{2} \in \mathbb{R}^{*}_{+} \ s.t. r < c_{2} , \exists n_{0}' \in \mathbb{N}^{*}\ s.t.\ f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0}'
    \end{cases}\]
    By choosing $n_{0}'' = max(n_{0}, n_{0}')$:
    \[\begin{cases}
          \forall c_{1} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r ,\Rightarrow\ \ c_{1} \cdot g(n) \leq f(n) ,\  \forall n \geq n_{0}'' \\ \forall  c_{2} \in \mathbb{R}^{*}_{+} \ s.t. r < c_{2} , \Rightarrow \ f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0}''
    \end{cases}\]
    By merging the inequalities, we have:
    \[\forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < r < c_{2} , \exists n_{0}'' = max(n_{0}, n_{0}' \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0}'' \]
    Therefore $f \in \Theta_{r}(g(n))$
\end{proof}

\section{Interdependence properties}
An interesting property of the \textbf{Big \textit{r-}Theta},  \textbf{Big \textit{r-}O} and \textbf{Big \textit{r-}Omega} classes is the simple technique of conversion between various values for the $r$s parameters. The following results arise:

\begin{theorem}
    \textbf{Big r-Theta conversion}:
    \[  f \in \Theta_{r}(g) \Rightarrow f \in \Theta_{q} \left( \frac{q}{r} \cdot g \right) \ \forall r,q \in \mathbb{R}_{+}\]
\end{theorem}

\begin{proof}
    $ f \in \Theta_{r}(g(n)) \Rightarrow \forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1}< r < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} $ \\
    Therefore, $\forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < q < c_{2} , \exists n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ \frac{r}{q} \cdot c_{1} \cdot g(n) \leq f(n) \leq \frac{r}{q} \cdot c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0}$. \\
    Thus, $f \in \Theta_{q} \left( \frac{q}{r} \cdot g \right)$.
\end{proof}

Another interesting result is obtained by multiplying the last equation by $\frac{q}{r}$: \\
$\forall c_{1}, c_{2} \in \mathbb{R}^{*}_{+} \ s.t. c_{1} < q < c_{2} , \exists n_{0}' = n_{0} \in \mathbb{N}^{*}\ \\ s.t.\ \ c_{1} \cdot g(n) \leq \frac{q}{r} \cdot f(n) \leq \frac{r}{q} \cdot g(n)\ ,\  \forall n \geq n_{0}$
\begin{corollary}
    \[ \frac{q}{r} \cdot f \in \Theta_{q} \left( g \right)\].
\end{corollary}

\begin{theorem}
    \textbf{Big r-O Conversion}:
    \[  f \in \mathcal{O}_{r}(g) \Rightarrow f \in \mathcal{O}_{q} \left( \frac{q}{r} \cdot g \right) \ \forall r,q \in \mathbb{R}_{+}\]
\end{theorem}
\begin{proof}
    Is similar with the proof for Big r-Theta, with the inequalities becoming $f(n) \leq c \cdot g(n)$.
\end{proof}
\begin{corollary}
    The following conversion relationship arises:
    \[  f \in \mathcal{O}_{r}(g) \Rightarrow \frac{q}{r} \cdot f \in \mathcal{O}_{q} \left(  g \right) \ \forall r,q \in \mathbb{R}_{+}\]
\end{corollary}
\begin{theorem}
    \textbf{Big r-Omega Conversion}:
    \[  f \in \Omega_{r}(g) \Rightarrow f \in \Omega_{q} \left( \frac{q}{r} \cdot g \right) \ \forall r,q \in \mathbb{R}_{+}\]
\end{theorem}
\begin{proof}
    Is similar with the proof for Big r-Theta, with the inequalities becoming $f(n) \geq c \cdot g(n)$.
\end{proof}
\begin{corollary}
    The following conversion relationship arises:
    \[  f \in \Omega_{r}(g) \Rightarrow \frac{q}{r} \cdot f \in \Omega_{q} \left( g \right) \ \forall r,q \in \mathbb{R}_{+}\]
\end{corollary}

Furthermore, we present some interesting results regarding the connection between $rComplexity$ functions and the correspondent class in the $Bachmann-Landau$ notations. For Small notations, we already presented the connection in definition section.

For Big notations $(\Theta, \mathcal{O}, \Omega)$, we present further some results:

\begin{theorem}
    Relationship between Big r-Theta and Big Theta:
    \[  f \in \Theta_{r}(g) \Rightarrow f \in \Theta (g) \]
    \[  f \in \Theta(g) \Rightarrow \exists r \in \mathbb{R}_{+}\ f \in \Theta_{r}(g) \]
\end{theorem}

\begin{theorem}
    Relationship between Big r-O and Big O:
    \[  f \in \mathcal{O}_{r}(g) \Rightarrow f \in \mathcal{O} (g) \]
    \[  f \in \mathcal{O}(g) \Rightarrow \exists r \in \mathbb{R}_{+}\ f \in \mathcal{O}{r}(g) \]
\end{theorem}

\begin{theorem}
    Relationship between Big r-Omega and Big Omega:
    \[  f \in \Omega_{r}(g) \Rightarrow f \in \Omega (g) \]
    \[  f \in \Omega(g) \Rightarrow \exists r \in \mathbb{R}_{+}\ f \in \Omega_{r}(g) \]
\end{theorem}

\section{Addition properties}
Similar to the calculus in $Bachmann-Landau$ notations (including Big-O arithmetic), a useful technique can be obtained by analyzing the behavior of $r-$Complexity classes in regards to addition exercises.

\begin{itemize}
    \item Addition in \textbf{Big r-Theta}:  \\
    The following relations hold for any correctly defined functions $f, g, f', g', h:\mathbb{N}\longrightarrow\mathbb{R}$, where $ h(n) = f'(n) + g'(n)\  \forall n \in  \mathbb{N} $, where $f',g'$ are two arbitrary functions such that $ f' \in \Theta_{r}(f), g' \in \Theta_{q}(g) $ and $r,q \in \mathbb{R}_{+}$:
    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow  h \in \Theta_{q}(g) $. \\
    \end{theorem}
    \begin{proof}
        $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow \lim_{n\to\infty} \dfrac{f'(n)}{g'(n)} = 0 \Rightarrow \lim_{n\to\infty} \dfrac{g'(n) + f'(n)}{g'(n)} = 1 $ and using the result from asymptotic analysis section, we have $ g'(n) + f'(n) = h(n) \in \Theta_{1}(g')$. \\ Using reflexivity property, $ h(n) \in \Theta_{q}(g)$.
    \end{proof}

    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow  h \in \Theta_{r}(f) $. \\
    \end{theorem}
    \begin{proof}
        Using the last result, by swapping $f \leftarrow g, g \leftarrow f$ and considering the commutativity of addition, we obtain the proof for this statement.
    \end{proof}

    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow  h \in \Theta_{r} \left( f + \dfrac{r}{q} \cdot g \right) $. \\
    \end{theorem}
    \begin{proof}

        $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t \Rightarrow \lim_{n\to\infty} \dfrac{f'(n)}{g'(n)} = t \cdot \dfrac{r}{q} = t' \Rightarrow \lim_{n\to\infty} \dfrac{g'(n) + f'(n)}{g'(n)} = t' + 1 $ and using the result from asymptotic analysis section, we have $ g'(n) + f'(n) = h(n) \in \Theta_{t' + 1}(g')$. \\
        Using reflexivity property, $ h(n) \in \Theta_{t' + 1} (q \cdot g)$. \\
        Using the conversion technique, we have $ h(n) \in \Theta_{r}( \dfrac{1}{r} \cdot t \cdot \dfrac{r}{q} + 1) \cdot q \cdot g)$. \\
        By swapping back $t$, asymptotically, we can establish: $ h(n) \in \Theta_{r} \left( \dfrac{1}{r} \cdot ( \dfrac{f(n)}{g(n)} \cdot \dfrac{r}{q} + 1) \cdot q \cdot g \right) $ \\
        Therefore $ h \in \Theta_{r} \left( f + \dfrac{r}{q} \cdot g \right) $.
    \end{proof}

    \item Addition in \textbf{Big r-O}: \\
    The following relations hold for any correctly defined functions $f, g, f', g', h:\mathbb{N}\longrightarrow\mathbb{R}$, where $ h(n) = f'(n) + g'(n)\  \forall n \in \mathbb{N} $, where $f',g'$ are two arbitrary functions such that $ f' \in \mathcal{O}_{r}(f), g' \in \mathcal{O}_{q}(g) $ and $r,q \in \mathbb{R}_{+}$:
    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow  h \in \mathcal{O}_{q}(g) $. \\
    \end{theorem}

    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-O class.
    \hfill\break

    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow  h \in \mathcal{O}_{r}(f) $. \\
    \end{theorem}

    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-O class.
    \hfill\break
    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow  h \in \mathcal{O}_{r} \left( f + \dfrac{r}{q} \cdot g \right) $. \\
    \end{theorem}

    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-O class.
    \hfill\break

    \item Addition in \textbf{Big r-Omega}:
    The following relations hold for any correctly defined functions $f, g, f', g', h:\mathbb{N}\longrightarrow\mathbb{R}$, where $ h(n) = f'(n) + g'(n)\  \forall n \in \mathbb{N} $, where $f',g'$ are two arbitrary functions such that $ f' \in \Omega_{r}(f), g' \in \Omega_{q}(g) $ and $r,q \in \mathbb{R}_{+}$:
    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow  h \in \Omega_{q}(g) $. \\
    \end{theorem}
    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-O class.
    \hfill\break

    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow  h \in \Omega_{r}(f) $. \\
    \end{theorem}
    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-Omega class.
    \hfill\break
    \begin{theorem}
        \textbf{If} $ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow  h \in \Omega_{r} \left( f + \dfrac{r}{q} \cdot g \right) $. \\
    \end{theorem}
    \textit{Proof}:
    Is similar with the proof for addition in Big Theta, using conversion presented for Big r-Omega class.
    \hfill\break

    \item Addition in \textbf{Small r-O}:
    Same property as described in $Bachmann-Landau$ notations.
    \item Addition in \textbf{Small r-Omega}:
    Same property as described in $Bachmann-Landau$ notations.
\end{itemize}

In a relax notation (consider that by any  $r-$Complexity class notation, we denote an arbitrary function part of the class), the following relations can be settled:

If $\lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0$:
\begin{itemize}
    \item \textbf{Big r-Theta}:
    \begin{lemma}
        \[  \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow \Theta_{r}(f) + \Theta_{q}(g) = \Theta_{q}(g)\]
    \end{lemma}
    \item \textbf{Big r-O}:
    \begin{lemma}
        \[  \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow \mathcal{O}_{r}(f) + \mathcal{O}_{q}(g) = \mathcal{O}_{q}(g)\]
    \end{lemma}
    \item \textbf{Big r-Omega}:
    \begin{lemma}
        \[  \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = 0 \Rightarrow \Omega_{r}(f) + \Omega_{q}(g) = \Omega_{q}(g)\]
    \end{lemma}
\end{itemize}

If $\lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty$:
\begin{itemize}
    \item \textbf{Big r-Theta}:
    \begin{lemma}
        \[  \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow \Theta_{r}(f) + \Theta_{q}(g) = \Theta_{r}(f)\]
    \end{lemma}
    \item \textbf{Big r-O}:
    \begin{lemma}
        \[ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow \mathcal{O}_{r}(f) + \mathcal{O}_{q}(g) = \mathcal{O}_{r}(f)\]
    \end{lemma}
    \item \textbf{Big r-Omega}:
    \begin{lemma}
        \[ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = \infty \Rightarrow \Omega_{r}(f) + \Omega_{q}(g) = \Omega_{r}(f)\]
    \end{lemma}
\end{itemize}

If $\lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+}$:
\begin{itemize}
    \item \textbf{Big r-Theta}:
    \begin{lemma}
        \[ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow \Theta_{r}(f) + \Theta_{q}(g) = \Theta_{r}(f + \dfrac{r}{q} \cdot g)\]
    \end{lemma}
    \item \textbf{Big r-O}:
    \begin{lemma}
        \[ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow \mathcal{O}_{r}(f) + \mathcal{O}_{q}(g) = \mathcal{O}_{r}(f + \dfrac{r}{q} \cdot g)\]
    \end{lemma}
    \item \textbf{Big r-Omega}:
    \begin{lemma}
        \[ \lim_{n\to\infty} \dfrac{f(n)}{g(n)} = t, \ t \in \mathbb{R}_{+} \Rightarrow \Omega_{r}(f) + \Omega_{q}(g) = \Omega_{r}(f + \dfrac{r}{q} \cdot g)\]
    \end{lemma}
\end{itemize}

The proofs of these Lemmas are imminent from the corresponding theorems.


\section{A few examples, double complexes, Dress sseq}\label{dress-sseq}
Way back in 905 I remember computing the cohomology ring of $\CP^n$ using Poincar\'e duality.
Let's do it fresh using the fiber sequence
$$
S^1\to S^{2n+1}\to \CP^n
$$
where $S^1$ acts on $S^{2n+1}$.
Here we know the cohomology of the fiber and the total space, but not the cohomology of the base.
Let's look at the cohomology sseq for this.
Then
$$
E_2^{s,t} = H^s(\CP^n;H^t(S^1)) \simeq H^s(\CP^n)\otimes H^t(S^1) Rightarrow H^{s+t}(S^{2n+1})
$$
The isomorphism $H^s(\CP^n;H^t(S^1)) \simeq H^s(\CP^n)\otimes H^t(S^1)$ follows from the UCT.

We know at least that $\CP^n$ is simply connected by the lexseq of homotopy groups.
I don't have to worry about local coefficients.
Let's work with the case $S^5$.
We know that $\CP^n$ is simply connected, so the one-dimensional cohomology is $0$.
The only way to kill $E_2^{0,1}$ is by sending it via $d_2$ to $E_2^{2,0}$.
Is this map surjective?
Yes, it's an isomorphism.

Now I'm going to give names to the generators of these things; see the below diagram.
$E_2^{2,1}$ is in total degree $3$ and so we have to get rid of it.
I will compute $d_2$ on this via Leibniz:
$$
d_2(xy) = (d_2 x)y - x d_2y = (d_2 x)y = y^2
$$
which gives (iterating the same computation):
\begin{sseqdata}[name=example1,classes={draw = none},degree={#1}{-#1+1},classes={inner sep=1ex}]
    \class["\Z x"](0,1)
    \class["\Z"](0,0)
    \class["0"](1,0)
    \class["0"](1,1)
    \class["\Z y"](2,0)
    \class["\Z xy"](2,1)
    \class["0"](3,0)
    \class["0"](3,1)
    \class["\Z y^2"](4,0)
    \class["\Z xy^2"](4,1)
    \d["d_2"]2(0,1)
    \d["d_2"]2(2,1)
\end{sseqdata}
\begin{equation*}
        \printpage[name=example1,page=2]
\end{equation*}
This continues until the end where you reach $\Z xy^{??}$ which is a permanent cycle since it lasts until the $E_\infty$-page.

Another example:
let $C_m$ be the cyclic group of order $m$ sitting inside $S^1$.
How can we analyse $S^{2n+1}/C_m=:L$?
This is the lens space.
We have a map $S^{2n+1}/C_m\to S^{2n+1}/S^1 = \CP^n$.
This is a fiber bundle whose fiber is $S^1/C_m$.
The spectral sequence now runs:
$$
E^2_{s,t} = H_s(\CP^n)\otimes H_t(S^1/C_m) \Rightarrow H_{s+t}(L)
$$
We know the whole $E^2$ term now:
\begin{sseqdata}[name=example2,classes={draw = none},homological Serre grading,classes={inner sep=1ex}]
    \class["\Z"](0,1)
    \class["\Z"](0,0)
    \class["0"](1,0)
    \class["0"](1,1)
    \class["\Z"](2,0)
    \class["\Z"](2,1)
    \class["0"](3,0)
    \class["0"](3,1)
    \class["\Z"](4,0)
    \class["\Z"](4,1)
    \d["m"]2(2,0)
\end{sseqdata}
\begin{equation*}
        \printpage[name=example2,page=2]
\end{equation*}
In cohomology, we have something dual:
\begin{sseqdata}[name=example3,classes={draw = none},degree={#1}{-#1+1},classes={inner sep=1ex}]
    \class["u"](0,1)
    \class["1"](0,0)
    \class["0"](1,0)
    \class["0"](1,1)
    \class["y"](2,0)
    \class["uy"](2,1)
    \class["0"](3,0)
    \class["0"](3,1)
    \class["y^2"](4,0)
    \class["uy^2"](4,1)
    \d["m"]2(0,1)
    \d["m"]2(2,1)
\end{sseqdata}
\begin{equation*}
        \printpage[name=example3,page=2]
\end{equation*}
What's the ring structure?
We get that $H^\ast(L) = \Z[y,v]/(my,y^{n+1},yv,v^2)$ where $|v| = 2n+1$ and $|y| = 2$.
By the way, when $m=1$, this is $\RP^{2n+1}$.
This is a computation of the cohomology of odd real projective spaces.
Remember that odd projective spaces are orientable and you're seeing that here because you're picking up a free abelian group in the top dimension.
\subsection{Double complexes}
$A_{s,t}$ is a bigraded abelian group with $d_h:A_{s,t}\to A_{s-1,t}$ and $d_v:A_{s,t}\to A_{s,t-1}$ such that $d_vd_h = d_hd_v$.
Assume that $\{(smt):s+t=n,A_{s,t}\neq 0\}$ is finite for any $n$.
Then
$$
(tA)_n = \bigoplus_{s+t=n}A_{s,t}
$$
Under this assumption, there's only finitely many nonzero terms.
I like this personally because otherwise I'd have to decide between the direct sum and the direct product, so we're avoiding that here.
It's supposed to be a chain complex.
Here's the differential:
$$
d(a_{s,t}) = d_ha_{s,t} + (-1)^s d_v a_{s,t}
$$
Then $d^2 = 0$, as you can check.
\begin{question}
    What is $H_\ast(tA_\ast)$?
\end{question}
Define a filtration as follows:
$$
F_p(tA)_n = \bigoplus_{s+t=n,s\leq p}A_{s,t}\subseteq (tA)_n
$$
This kinda obviously gives a filtered complex.
Let's compute the low pages of the sseq.
What is $\gr_s(tA)$?
Well
$$
\gr_s(tA)_{s+t} = (F_s/F_{s-1})_{s+t} = A_{s,t}
$$
This associated graded object has its own differential $\gr_s(tA)_{s+t}=A_{s,t}\xar{d_v} A_{s,t-1} = \gr_s(tA)_{s+t-1}$.
Let $E^0_{s,t} = \gr_s(tA)_{s+t} = A_{s+t}$, so that $d^0 = d_v$.
Then $E^1 = H(E^0_{s,t},d^0) = H(A_{s,t};d_v) =: H^v_{s,t}(A)$.
So computing $E^1$ is ez.
Well, what's $d^1$ then?

To compute $d^1$ I take a vertical cycle that and the differential decreases the ... by $1$, so that $d^1$ is induced by $d_h$.
This means that I can write $E^2_{s,t} = H^h_{s,t}(H^v(A))$.
\begin{question}
    You can also do $^\prime E^2_{s,t} = H^v_{s,t}(H^h(A))$, right?
\end{question}
Rather than do that, you can define the transposed double complex $A^\mathsf{T}_{t,s} = A_{s,t}$, and $d^\mathsf{T}_h(a_{s,t}) = (-1)^s d_v(a_{s,t})$ and $d^\mathsf{T}_v(a_{s,t}) = (-1)^t d_h a_{s,t}$.
When I set the signs up like that, then
$$
tA^\mathsf{T} \simeq tA
$$
\emph{as complexes} and not just as groups (because of those signs).
Thus, you get a spectral sequence
$$
^\mathsf{T}E^2_{s,t} = H^v_{s,t}(H^h(A))
$$
converging to the same thing.
I'll reserve telling you about Dress' construction until Monday because I want to give a double complex example.
It's not ... it's just a very clear piece of homological algebra.
\begin{example}[UCT]
    For this, suppose I have a (not necessarily commutative) ring $R$.
    Let $C_\ast$ be a chain complex, bounded below of right $R$-modules, and let $M$ be a left $R$-module.
    Then I get a new chain complex of abelian groups via $C_\ast\otimes_R M$.
    What is $H(C_\ast\otimes_R M)$?
    I'm thinking of $M$ as some kind of coefficient.
    Let's assume that each $C_n$ is projective, or at least flat, for all $n$.

    Shall we do this?

    Let $M\leftarrow P_0 \leftarrow P_1\leftarrow \cdots$ be a projective resolution of $M$ as a left $R$-module.
    Then $H_\ast(P_\ast)\xar{\simeq} M$.
    Form $C_\ast\otimes_R P_\ast$: you know how to do this!
    I'll define $A_{s,t}$ to be $C_s\otimes_R P_t$.
    It's got two differentials, and it's a double complex.
    Let's work out the two sseqs.

    Firstly, let's take it like it stands and take homology wrt $P$ first.
    I'm organizing it so that $C$ is along the base and $P$ is along the fiber.
    What is the vertical homology $H^v(A_{\ast,\ast})$?
    If the $C$ are projective then tensoring with them is exact, so that $H^v(A_{s,\ast}) = C_s\otimes_R H_\ast(P_\ast)$, so that
    $E^1_{s,t} = H^v_{s,t}(A_{\ast,\ast}) = C_s\otimes M$ if $t=0$ and $0$ otherwise.
    The spectral sequence is concentrated in one row.
    Thus,
    $$E^2_{s,t} = \begin{cases}
	H_s(C_\ast\otimes_R M) & \text{ if }t = 0\\
	0 & \text{else}
    \end{cases}$$
    This is canonically the same thing as $E^\infty_{s,0}\simeq H_s(tA)$.

    Let me go just one step further here.
    The game is to look at the \emph{other} spectral sequence, where I do horizontal homology first.
    Then $H^h(A_{\ast,\ast}) = H_t(C_\ast)\otimes P_s$ again because the $P_\ast$ are projective.
    Thus,
    $$
    E^2_{s,t} = H^v(H^h(A_{\ast,\ast})) = \Tor^R_s(H_t(C),M)\Rightarrow H_{s+t}(C_\ast\otimes_R M)
    $$
    That's the \emph{universal coefficients spectral sequence}.

    What happens if $R$ is a PID?
    Only two columns are nonzero, and $E^2_{0,n} = H_n(C)\otimes_R M$ and $E^2_{1,n-1} = \Tor_1(H_{n-1}(C),M)$.
    This exactly gives the universal coefficient exact sequence.
\end{example}
Later we'll use this stuff to talk about cohomology of classifying spaces and Grassmannians and Thom isomorphisms and so on.


\section{The Valency Theorem; Euler Characteristic; The Riemann-Hurwitz Theorem}
\subsection{The Valency Theorem}
We want to relate the branching data of an analytic map and the topology of a compact Riemann surface.
\begin{theorem}[Valency Theorem]
    Suppose $f:R\to S$ is a non-constant analytic map between compact Riemann surfaces $R,S$, then the function $n:S\to\mathbb N$ defined by
    $$n(q)=\sum_{p\in f^{-1}(\{q\})}m_f(p)$$
    is constant on $S$.
\end{theorem}
\begin{proof}
    By the identity principle, $f^{-1}(\{q\})$ is finite, therefore $n$ is well-defined.
    As $S$ is connected, it suffices to show that $n$ is locally constant.
    Let $q_0\in S$ and $f^{-1}(q_0)=\{p_1,\ldots,p_k\}$.
    Our goal is to find local coordinates about the $p_i$ and $q_0$ such that $f$ is a power map.
    Let $(\psi,V)$ be a chart about $q_0$ such that $\psi(q_0)=0$, then there exists disjoint charts $(\phi_1,U_1),\ldots,(\phi_k,U_k)$ such that $p_i\in U_i$ and $\psi\circ f\circ\phi_i^{-1}(z)=z^{m_f(p_i)}$.\\
    Let $U=\bigcup_iU_i$, then $U$ is open, so $R\setminus U$ is closed subset of $R$, hence is compact.
    So $K=f(R\setminus U)$ is compact and hence closed as $S$ is Hausdorff.
    Set $V'=V\setminus K$.
    Now quite obviously $f^{-1}(V')\subset U$.
    Set $U_i'=U_1\cap f^{-1}(V')$, then $f^{-1}(V')=\bigcup_iU_i'$, then in the charts $(\phi_i,U_i')$ and $(\psi,V')$, $f$ takes the form of power maps, therefore $n(q)=n(q_0)$ for any $q\in V'$.
\end{proof}
\begin{definition}
    This constant $n$ is called the valency or degree $\deg f$ of $f$.
\end{definition}
\begin{example}
    If $f$ is a polynomial, then the degree of $f$ equals the polynomial degree of $f$.
\end{example}
\begin{corollary}[Fundamental Theorem of Algebra]
    Any nonconstant polynomial of degree $d$ has exactly $d$ zeros in $\mathbb C$.
\end{corollary}
\begin{proof}
    Extend it analytically to $\mathbb C_\infty\to\mathbb C_\infty$.
\end{proof}


\section{Cubic spline interpolation}
	\begin{frame}{Cubic spline interpolation}
		\begin{block}{}
			\begin{itemize}
				\item $s_{j}$ on $[x_{j},x_{j+1}] \rightarrow$ cubic polynomial:
                $\newline \ \ \ s_{j}(x)=a_{j}+b_{j}(x-x_{j})+c_{j}(x-x_{j})^{2}+
                d_{j}(x-x_{j})^{3}$
                \item $s_{j}(x_{j+1})=f(x_{j+1})$
                \item $s_{j}(x_{j+1})=s_{j+1}(x_{j+1})$
                \item $s^{'}_{j}(x_{j+1})=s^{'}_{j+1}(x_{j+1})$
                \item $s^{''}_{j}(x_{j+1})=s^{''}_{j+1}(x_{j+1})$
			\end{itemize}
		\end{block}
        \begin{figure}[h]
			\includegraphics[width=.55\linewidth]{img/4/spline_img_4}
		\end{figure}
	\end{frame}


\chapter{The pull-through relation}\label{app:pull-through-relation}

\begin{prop}\label{prop:pull-through-relation}
\thmtitle{Pull-through relation}
\thmstatement{
  For any non-commuting $x_1,\ld,x_n$, and $y$ for which addition, subtraction and multiplication are defined, $x_1\cd x_ny=\pr{\mp}^nyx_1\cd x_n+\sum_{k=1}^n\pr{\mp}^{n-k}x_1\cd[x_k,y]_{\pm}\cd x_n$, where $[x,y]_{\pm}\equiv xy\pm yx$.
}
\thmproof{
  The $n=1$ case follows from the definition of the commutator brackets: $xy=\mp yx+[x,y]_{\pm}$.
  Now, assume it holds for $n$ and consider the $n+1$ case.
  Since $x_1\cd x_{n+1}y=x_1\cd x_n(\mp yx_{n+1}+[x_{n+1},y]_{\pm})$, we find
  \begin{align*}
    x_1\cd x_{n+1}y
  =&\
  \mp
    \pr{
      \pr{\mp}^nyx_1\cd x_n
    +
      \sum_{k=1}^n
      \pr{\mp}^{n-k}
      x_1\cd [x_k,y]_{\pm}\cd x_n
    }
    x_{n+1}
  +
    x_1\cd x_n[x_{n+1},y]_{\pm}
  \\=&\
    \pr{\mp}^{n+1}
    yx_1\cd x_{n+1}
  +
    \sum_{k=1}^{n+1}
    \pr{\mp}^{n+1-k}
    x_1\cd[x_k,y]_{\pm}\cd x_{n+1}
  \end{align*}
  and, by induction, the result holds for all $n$.
}
\end{prop}


\chapter{Functions and Their Graphs}

You can think of a function as a machine: you put something into the
machine, it processes it, and out comes something else. Just as we
often use the variable $x$ to stand in for a number, we often use the
variable $f$ to stand in for a function.

For example, we might ask, ``Let the function $f$ be defined like this:

\begin{equation*}
f(x) = -5x^2 + 12x + 2
\end{equation*}

What is the value of $f(3)$?''

You would run the number 3 through ``the machine'': $-5(3^2) + 12(3) + 2 = -7$. The answer would be ``$f(3)$ is $7$''.

Some functions are not defined for every possible input. For example:

\begin{equation*}
  f(x) = \frac{1}{x}
\end{equation*}

  This is defined for any $x$ except 0. The set of values that a function can process is called its \textit{domain}.

\begin{Exercise}[title={Domain of a function}, label=function_domain]

  Let the function $f$ be given by $f(x) = \sqrt{x - 3}$.  What is its domain?

\end{Exercise}
\begin{Answer}[ref=function_domain]
  You can only take the square root of nonnegative numbers, so the
  function is only defined when $x - 3 \geq 0$.  Thus the domain is
  all real numbers greater than or equal to 3.
\end{Answer}

\section{Graphs of Functions}

If you have a function $f$, its graph is the set of pairs $(x, y)$
such that $y = f(x)$.  We usually draw a picture of this set, so we
will use the term \textit{graph} to mean the set and also the picture.

Here is the graph of the function $f(x) = -5x^2 + 12x + 2$:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-1,xmax=3.5,
        ymin=-10,ymax=11,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue,<->] expression[domain=-0.7:3.05,samples=100]{(-5)*(x^2) + (12 * x) + 2};
    \end{axis}
\end{tikzpicture}

(Note that is just part of the graph: it goes infinitely in both
directions, so I could not fit the whole thing on the page.)

Here is the graph of the function $f(x) = \frac{1}{x}$:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-7,xmax=7,
        ymin=-7,ymax=7,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue,<->] expression[domain=-6.5:-0.15,samples=100]{1/x};
        \addplot[no marks,sdkblue,<->] expression[domain=0.15:6.5,samples=100]{1/x};
    \end{axis}
\end{tikzpicture}

\begin{Exercise}[title={Draw a graph}, label=draw_graph]

  Let the function $f$ be given by $f(x) = -3x + 3$. Sketch its graph.

\end{Exercise}
\begin{Answer}[ref=draw_graph]

  The graph of this function is line. Its slope is -3.  It intersects the y axis at $(0, 3)$

\begin{tikzpicture}
    \begin{axis}[
        xmin=-1,xmax=3,
        ymin=-7,ymax=7,
        xtick={1},
        ytick={3},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue,<->] expression[domain=-0.75:2.74,samples=100]{-3 * x + 3};
    \end{axis}
\end{tikzpicture}

\end{Answer}

\section{Can this be expressed as a function?}

Note that not all sets can be expressed as graphs of functions.  For
example, here is the set of points $(x,y)$ such that $x^2 + y^2 = 9$:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-3.5,xmax=3.5,
        ymin=-3.5,ymax=3.5,
        ytick={-3,-2,-1,0,1,2,3},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{sqrt(9 - x^2)};
        \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{-1 * sqrt(9 - x^2)};
    \end{axis}
\end{tikzpicture}

This cannot be the graph of a function because what would $f(0)$ be? 3
or -3?  This set fails what we call ``the vertical line test'': If any
vertical line contains more than one point from the set, it isn't the graph
of a function.  For example, the vertical line $x = 2$ would cross
the graph twice:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-3.5,xmax=3.5,
        ymin=-3.5,ymax=3.5,
        ytick={-3,-2,-1,0,1,2,3},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{sqrt(9 - x^2)};
        \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{-1 * sqrt(9 - x^2)};
        \addplot [thick, dashed] coordinates {(2,-2.5)(2,2.5)};

    \end{axis}

\end{tikzpicture}

\section{Inverses}

Some functions have inverse functions. If a function $f$is a machine that turns
number $x$ into $y$, the inverse (usually denoted $f^{-1}$) is the machine that turns $y$ back
into $x$.

For example, let $f(x) = 5x + 1$. Its inverse is
$f^{-1}(x) = (x - 1)/5$. (Spot check it: $f(3) = 16$ and $f^{-1}(16) = 3$)

Does the function $f(x) = x^3$ have an inverse? Yes, $f^{-1}(x) =
\sqrt[3]{x}$. Let's plot the function (solid line) and its inverse (dashed):

\begin{tikzpicture}
    \begin{axis}[
        xmin=-3.5,xmax=3.5,
        ymin=-3.5,ymax=3.5,
        ytick={-3,-2,-1,0,1,2,3},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
        \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{x^3};
        \addplot[no marks,sdkblue,dashed] expression[domain=0:3,samples=100]{x^(1/3)};
        \addplot[no marks,sdkblue,dashed] expression[domain=-3:0,samples=100]{-1 * (-1 * x)^(1/3)};
    \end{axis}
\end{tikzpicture}

The inverse is the same as the function, just with its axes swapped.
This tells us how to solve for an inverse: We swap $x$ and $y$ and
solve for $y$.

For example, if you are given the function $f(x) = 5x + 1$, its graph
is all $(x,y)$ such that $y = 5x + 1$.  The graph of its inverse is
all $(x, y)$ such that $x = 5y + 1$. So you solve for $y$: $y = (x -
1)/5$.

Not every function has an inverse.  For example, $f(x) = x^2$.  Note
that $f(2) = f(-2) = 4$.  What would $f^{-1}(4)$ be? 2 or -2?  This
implies the ``horizontal line test'': If any horizontal line contains
more than one point of a function's graph, that function has no
inverse.

\begin{tikzpicture}
    \begin{axis}[
        xmin=-3.5,xmax=3.5,
        ymin=-1, ymax=6,
        ytick={-1,0,1,2,3,4,5,6},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
      \addplot[no marks,sdkblue] expression[domain=-3:3,samples=100]{x^2};
      \addplot [thick, dashed] coordinates {(-3,4)(3,4)};
    \end{axis}
\end{tikzpicture}

In some problems, you need an inverse and you don't really need the
whole domain, so you trim the domain to a set you can define an
inverse on. This allow you to say things like ``If we restrict the domain to
the nonnegative numbers, the function $f(x) = x^2 - 5$ has an inverse:
$f^{-1}(x) =\sqrt{x + 5}$.

This begs the question: What is the domain of the inverse function $f^{-1}$?

If we let $X$ be the domain of $f$, we can run every member of $X$
through ``the machine'' $f$ and gather them in a set on the other
side. This set would be the \textit{image} of $f$. (Some people call
this the \textit{range} of $f$.)

What is the image of $f(x) = x^2 - 5$? It is the set of all real
numbers greater than or equal to -5. We write this

\begin{equation*}
  \{ x \in {\rm I\!R} | x \geq -5 \}
  \end{equation*}

Now we have the words we need: \textbf{The image of the function is the domain
  of the inverse function.}

In our example, we can use any number greater
than or equal to -5 as input into the inverse function.

\begin{tikzpicture}
    \begin{axis}[
        xmin=-5.5,xmax=7.5,
        ymin=-6, ymax=5,
        xtick={-3, 2},
        ytick={-5, 1},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        ]
      \addplot[no marks,sdkblue, ->] expression[domain=0:3,samples=100]{x^2 - 5} node[right] {$y = x^2 - 5$};
      \addplot [thick, dashed, red, ->] coordinates {(-0.05,-5)(-0.05,4.5)}
      node [draw, red, left, align=left, yshift=-0.6cm, xshift=-0.1cm] {image of $f$ \textit{or}\\ domain of $f^{-1}$};
      \addplot [thick, dashed, blue, ->] coordinates {(-0,-5.1)(6.75,-5.1)}
      node [draw, align=left, above, blue, yshift=0.1cm, xshift=-1.3cm] {domain of $f$ \textit{or}\\image of $f^{-1}$};
    \end{axis}
\end{tikzpicture}

\begin{Exercise}[title={Find the inverse}, label=simple_inverse]

  Let $f(x) = (x - 3)^2 + 2$.  Sketch the graph.

  Using all the real numbers as a domain, does this function have an inverse?

  How would you restrict the domain to make the function invertible?

  What is the inverse of that restricted function?

  What is the domain of the inverse?

\end{Exercise}
\begin{Answer}[ref=simple_inverse]

  This graph is the graph of $y = x^2$ that has been moved to the right by three units and up two units:

\begin{tikzpicture}
    \begin{axis}[
        xmin=-1,xmax=7,
        ymin=-1,ymax=7,
        xtick={3, 6},
        ytick={2, 4},
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={$x$},
        ylabel={$y$},
        ]
      \addplot[no marks,sdkblue,<->] expression[domain=-2.5:6.5,samples=100]{(x - 3)^2 + 2};
      \addplot[dashed] coordinates {(-1, 2)(4,2)};
      \addplot[dashed] coordinates {(3, -1)(3, 3)};
    \end{axis}
\end{tikzpicture}

To prevent any horizontal line from containing more than one point of
the graph, you would need to use the left or the right side: Either
$\{x \in {\rm I\!R}  | x \leq 3\}$ or $\{x {\rm I\!R}| x \geq 3\}$. Most people will choose the
right side; the rest of the solution will assume that you did too.

To find the inverse we swap $x$ and $y$: $x = (y -3)^2 + 2$

The we solve for $y$ to get the inverse: $y = \sqrt{x - 2} + 3$

You can take the square root of nonnegative numbers. So the function
$f^{-1}(x) = \sqrt{x - 2} + 3$ is defined whenever $x$ is greater than
or equal to 2.

\end{Answer}

\section{Graphing Calculators}

One really easy way to understand your function better is to use a graphing
calculator.  Desmos is a great, free online graphing calculator.

In a web browser, go to Desmos: \url{https://www.desmos.com/calculator}

In the field on the left, enter the function $y = x^2 - x - 6$. (For
the exponent, just prefix it with a caret symbol: ``x\^2''.)

\includegraphics[width=0.85\textwidth]{Desmos.png}


\chapter{Neural Networks}
\label{chapter:neural networks}

Applicability of models that comprised linear combinations of \textbf{fixed basis functions} is limited by the curse of dimensionality.In order the apply such models to large-scale problems,it is necessary to adapt the basis functions to the data.

Support vector machines and relevance vector machine \textbf{choose a subset from a fixed set of basis functions} and results in much sparser models.

An alternative approach is to fix the number of basis functions in advance but allow them to be \textbf{adaptive},in other words to use parametric forms for the basis function
\section{Feed-forward Network Functions}
The linear models are based on linear combinations of fixed nonlinear basis functions $\phi_j(\vec{x})$ and take the form
\begin{align}
    y(\vec{x},\vec{w}) = f(\sum_{j=1}^{M}w_j\phi_j(\vec{x})) = f(\vec{w}^T\vec{\phi}(\vec{x}))
\end{align}
where $f(\cdot)$ is a nonlinear activation function in the case of classification and is the identity in the case of regression.

Neural network model can be described a series of functional transformations.First construct $M$ linear combinations of the input variables $x_1,\ldots,x_D$ in the form
\begin{align}
    a_j^{(l+1)} &= \sum_{i=1}^{D}w_{ji}^{(l)}x_i+w_{j0}^{(l)} \\
	    &= \vec{w}^{(l)}\vec{x}^{(l)}
\end{align}
where $j=1,\ldots,M$,and the superscript $(l)$ indicates the $l-the$ layer of the network.We refer the parameters $w_{ji}^{(l)}$ as \textbf{weights} and $w_{j0}^{(l)}$ as \textbf{biases}.The quantity $a_j$ are known as \textbf{activations}.Each of them is then transformed using \textbf{activation function} $h(\cdot)$ to give
\begin{align}
    z_j=h(a_j)
\end{align}
These quantities correspond to the outputs of the basis functions in linear model that,in the context of neural networks,are called \textbf{hidden units}.The nonlinear functions $h(\cdot)$ are generally chosen to be sigmoidal or the '$tanh$' function.

The transformation of the following layer of the network,combine these values to give \textbf{output unit activations}
\begin{align}
    a_k=\sum_{j=1}^{M}w_{kj}^{(2)}z_j+w_{k0}^{(2)}
\end{align}
where $k=1,\ldots,K$,and $K$ is the total number of outputs.

\begin{figure}
    \includegraphics[ ]{figures/prml/Figure5.1.jpg}
    \caption{Network diagram for two-layer neural network}
\end{figure}

For regression,$y_k=a_k$ and binary classification,we can use $y_k=\sigma(a_k)=\dfrac{1}{1+\exp(-a)}$.Finally,for multiclass problems,a \textbf{softmax} activation function is used.\textbf{Unit activation function} is discussed later.

The process of evaluating the output of the network diagram is interpreted as a \textbf{forward propagation} of information through the network.Deterministic rather than stochastic variables of internal nodes do not represent probabilistic graphical models.Each (hidden or output) unit in such network computes a function given by
\begin{align}
    z_k=h(\sum_j w_{kj}z_j)
\end{align}
where the sum runs over all units that send connections to unit $k$ (including the bias parameter).

\subsection{Weight-space symmetries}
Consider a two-layer network as shown before.For $M$ hidden units,there will be $M$ 'sign-flip' symmetries,i.e.,$2^M$ equivalent weight vectors.Similarly,interchanging the values of all the weights(and the bias) can give rise to the same mapping function from inputs to outputs represented by the network.There will be $M!$ equivalent weight vectors.The network have an overall weight-space symmetry factor of $M!2^M$.

\section{Network Training}
Given a training set comprising a set of input vectors ${\vec{x}_n}$,where $n=1,\ldots,N$,together with a corresponding set of target vectors ${\vec{t}_n}$,we minimize the error function.We can provide a probabilistic interpretation to the network outputs.

For regression,and for the moment we consider a single target variable $t$ that can take any real value.We assume that $t$ has a Gaussian distribution with an $\vec{x}$-dependent mean,which is given by the output of the neural network,so that
\begin{align}
    p(t|\vec{w},\vec{w})=\mathcal{N}(t|y(\vec{x},\vec{w}),\beta^{-1})
\end{align}
where $\beta$ is the precision (inverse variance) of the Gaussian noise.Maximize the likelihood function as in linear models.View the network as having an output activation function that id the identity,so that $y_k=a_k$.The corresponding sum-of-squares error function has the property
\begin{align}\label{eqn:output activation function derivative}
    \dfrac{\partial E}{\partial a_k}=\dfrac{\partial E}{\partial y_k}=y_k - t_k
\end{align}

For binary classification in which $t\in{0,1}$,consider a network having a single output whose activation function is a logistic sigmoid
\begin{align}
    y=\sigma(a)=\dfrac{1}{1+\exp(-a)}
\end{align}
We can interpret $y(\vec{x},\vec{w})$ as the conditional probability $p(\mathcal{C}_1|\vec{x})$.the conditional distribution of targets given inputs is then a Bernoulli distribution of the form
\begin{align}
    p(t|\vec{x},\vec{w})=y(\vec{x},\vec{w})^t\{{1-y(\vec{x},\vec{w})}^{1-t}\}
\end{align}

For $K$ separate binary classifications,the conditional distribution of the targets is
\begin{align}
    p(\vec{t}|\vec{x},\vec{w})=\prod_{k=1}^{K}y_k(\vec{x},\vec{w})^t_k[1-y(\vec{x},\vec{w})]^{1-t_k}
\end{align}
Taking the negative logarithm of the corresponding likelihood function,we get \textbf{cross-entropy} error function
\begin{align}
    E(\vec{w})=-\sum_{n=1}^{N}\sum_{k=1}^{K}{t_{nk}\ln y_{nk}+(1-t_{nk})\ln(1-y_{nk})}
\end{align}
The derivative of the error function with respect to the \textbf{activation} for a particular output unit is
\begin{align}
    \dfrac{\partial E_n}{\partial a_k}
    &=\dfrac{\partial {-\sum_{k=1}^{K}{t_{nk}\ln y_{nk}+(1-t_{nk})\ln(1-y_{nk})} }}{\partial a_k} \\
    &=-{t_{k} \dfrac{y_{k}(1-y_{k})}{y_{k}} +(1-t_{k}\dfrac{-y_{k}(1-y_{k})}{1-y_{k}}) } \\
    &={y_{k}-t_{k}}
\end{align}

Following the logistic regression,the \textbf{output unit activation} is given by the \textbf{softmax} function
\begin{align}
    y_k(\vec{x},\vec{w})=\dfrac{\exp(a_k(\vec{x},\vec{w}))}{\sum_j\exp(a_j(\vec{x},\vec{w})) }
\end{align}

\subsection{Parameter optimization}
Turn next to the task of finding a weight vector $\vec{w}$ which minimizes the chosen function $E(\vec{w})$.
\begin{figure}
    \includegraphics{figures/prml/Figure5.5.jpg}
    \caption{geometrical view of the error function $E(\vec{w})$ as a surface sitting over weight space.A is a local minimum and B is the global minimum.The local gradient of the error surface is given by the vector $\nabla E$.}
\end{figure}
Resort to \textbf{iterative} numerical procedures.Most techniques involve choosing some initial value $\vec{w}^{(0)}$ for weight vector and then moving through weight space in a succession of steps of the form
\begin{align}
    \vec{w}^{(\eta+1)}=\vec{w}^{(\eta)}+\Delta \vec{w}^{(\eta)}
\end{align}

Many algorithms make use of gradient information,evaluating $\nabla E(\vec{w})$  at the new weight vector $\vec{w}^{(\eta+1)}$.

\subsection{Local quadratic approximation}
Consider the \textbf{Taylor expansion} of $E(\vec{w})$ around some point in weight space.

\subsection{Use of gradient information}
It is possible to evaluate the gradient of an error function efficiently by means of the backpropagation procedure.Because each evaluation of $\nabla E$ brings $W$ items of information,we might hope to find the minimum of the function in $O(W)$ gradient evaluations.Each such evaluation takes only $O(W)$ steps and so the minimum can be now found in $O(W^2)$ steps.
\subsection{Gradient descent optimization}
\textbf{Batch gradient descent}
\begin{align}
    \vec{w}^{(\tau+1)}=\vec{w}^{(\tau)}-\eta\nabla E(\vec{w})^{(\tau)}
\end{align}
where the parameter $\eta>0$ is known as the \textbf{learning rate}.

More efficient methods than this poor algorithm are \textbf{conjugate gradients} and \textbf{quasi-Newton} methods,which are more robust and much faster.\textbf{On-line} version of gradient has also proved useful in practice.On-line gradient descent,known as \textbf{sequential gradient descent } or \textbf{stochastic gradient descent},makes an update to the weight vector based on one data point at a time,so that
\begin{align}
    \vec{w}^{(\tau+1)}=\vec{w}^{(\tau)}-\eta \nabla E(\vec{w}^{(\tau)})
\end{align}

\section{Error Backpropagation}
The goal in this section is to find an efficient technique for evaluating the gradient of an error function $E(\vec{w})$ for a feed-forward neural network.This can be achieved using a local message passing scheme in which information is sent alternately forwards and backwards through the network,known as \textbf{error backpropagation} or \textbf{backprop}.

\subsection{Evaluation of error-function derivatives}
Many error functions of practical interest,for instance those defined by maximum likelihood for a set of i.i.d data,comprise a sum of terms,on for each data point,so that
\begin{align}
    E(\vec{w})=\sum_{n=1}^{N}E_n(\vec{w})
\end{align}

In a general feed-forward network,each unit computes a weighted sum of its inputs of the form
\begin{align}
    a_j=\sum_i w_{ji} z_i
\end{align}
where $z_i$ is the activation of a unit that sends a connection to unit $j$ and $w_{ji}$ is the weight associated with that connection.The sum is transformed by an activation function $h(\cdot)$ to give the activation $z_j$ of unit $j$ in the form
\begin{align}
    z_j=h(a_j)
\end{align}

Evaluate the derivate of $E_n$ with respect to a weight $w_{ji}$,applying the \textbf{chain rule} for partial derivatives
\begin{align}
    \dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial \vec{a}_j}\dfrac{a_j}{\partial w_{ji}}
\end{align}
Now introduce a notation
\begin{align}
    \delta_j \equiv \dfrac{\partial E_n}{\partial a_j}
\end{align}
referred as \textbf{errors}.
Using the sum form
\begin{align}
    \dfrac{\partial a_j}{\partial w_{ji}} = z_i
\end{align}
Substituting them back
\begin{align}
    \dfrac{\partial E_n}{\partial w_{ji}}=\delta_j z_i.
\end{align}
For the output units,we have
\begin{align}
    \delta_k=y_k-t_k
\end{align}

For hidden units,make use of the chain rule for partial derivatives
\begin{align}
    \delta_j\equiv \dfrac{\partial E_n}{\partial a_j}=
    \sum_k \dfrac{\partial E_n}{\partial a_k}\dfrac{\partial a_k}{\partial a_j} \\
\end{align}
where the sum runs over all units $k$ to which unit $j$ sends connections.
\begin{align}
    \because \delta_k &= \dfrac{\partial E_n}{\partial a_k}\\
    \therefore
    \delta_j\equiv \dfrac{\partial E_n}{\partial a_j} &=
    \sum_k \dfrac{\partial E_n}{\partial a_k}\dfrac{\partial a_k}{\partial a_j} \\
    &=\sum_k \delta_k\dfrac{\partial a_k}{\partial a_j} \\
    &=\sum_k \delta_k h'(a_j)w_{kj} \\
    &= h'(a_j)\sum_k w_{kj}\delta_k
\end{align}

\subsubsection{back-propagation algorithm wrap-up}

We'll use $\vec{w}_{ij}^{l}$ to denote the \textbf{weight} for the connection from the $k$th neuron in the $(l-1)$th layer to the $j$th neuron in the $(l)$th layer.And $b_j^l$  for the bias of the $j$th neuron in the $l$th layer, $a_j^l$ for the activation(weighted input) of the $j$th neuron in the $l$th layer,$\sigma$ denotes the element-wise activation function,$z_j^l$ denotes the output of $j$th neuron in the $l$th layer.
Then we have
\begin{align}
	\vec{a}^l &\equiv \vec{W}^l \vec{z}^{l-1}+\vec{b}^l \\
	\vec{z}^{l} &\equiv \sigma(\vec{a}^{l})
\end{align}

$s\odot t$ denotes \textbf{element-wise product} of two matrices (vectors):
\begin{align}
(s\odot t)_j = s_jt_j
\end{align}

Define the error $\sigma_j^l$ of neuron $J$ in the layer $l$ by
\begin{align}
\vec{\delta}^l_j \equiv \frac{\partial C}{\partial \vec{a}^l_j}
\end{align}

Using \textbf{chain rule of derivatives},the error in the output layer is:
\begin{eqnarray}
\delta^L_j &=& \frac{\partial C}{\partial a^L_j} \\
&=& \sum_k \frac{\partial C}{\partial z^L_k} \frac{\partial z^L_k}{\partial a^L_j} \\
&=& \frac{\partial C}{\partial z^L_j} \frac{\partial z^L_j}{\partial a^L_j},\text{if output only depends on same neuron's input } \\
&=& \frac{\partial C}{\partial \vec{z}^L_j} \sigma'(\vec{a}^L_j).
\end{eqnarray}
And the matrix-based form is:
\begin{eqnarray}
\delta^L & = & \nabla_z C \odot \sigma'(a^L). \\
\delta^L & = &\Sigma'(a^L) \nabla_z C,
\end{eqnarray}
where $\Sigma'(z^L)$ is a square matrix whose diagonal entries are the values $\sigma'(z^L)$, and whose off-diagonal entries are zero.

Propagation (recursive relationship) in hidden layer:
\begin{eqnarray}
\delta_j^l &=& \frac{\partial C}{\partial a^l_j}\\
&=& \sum_k \frac{\partial C}{\partial a^{l+1}_k} \frac{\partial a^{l+1}_k}{\partial a^l_j}\\
&=& \sum_k \frac{\partial a^{l+1}_k}{\partial a^l_j} \delta^{l+1}_k, \\
\because a^{l+1}_k &=& \sum_j w^{l+1}_{kj} z^l_j +b^{l+1}_k = \sum_j w^{l+1}_{kj} \sigma(a^l_j) +b^{l+1}_k. \\
\therefore \frac{\partial a^{l+1}_k}{\partial a^l_j} &=& w^{l+1}_{kj} \sigma'(a^l_j) \\
\therefore
\delta^l_j &=& \sum_k w^{l+1}_{kj}  \delta^{l+1}_k \sigma'(a^l_j) \\
\delta^l &=& ((\vec{W}^{l+1})^T \delta^{l+1}) \odot \sigma'(\vec{a}^l), \\
\delta^l &=& \Sigma'(a^l) (w^{l+1})^T \delta^{l+1}.\\
\end{eqnarray}

Rate of change of cost with respect to bias in the network:
\begin{eqnarray}
\frac{\partial C}{\partial b^l_j}
&=\frac{\partial C}{\partial \vec{a}^l_j}\frac{\partial \vec{a}^l_j}{\partial b^l_j} \\
&= \frac{\partial C}{\partial \vec{a}^l_j} \cdot 1
&=\vec{\delta}^l_j
\end{eqnarray}

Rate of change of the cost with respect to any weight matrix in the network:
\begin{align}
    \frac{\partial C}{\partial w^l_{jk}}
&= \frac{\partial C}{\partial \vec{a}^l_j}\frac{\partial \vec{a}^l_j}{\partial w^l_{jk}}\\
&= z^{l-1}_k \delta^l_j \\
\frac{\partial C}{\partial w} &= \vec{\delta_{\rm out}} \vec{z_{\rm in}}^T,
\end{align}

When dealing with \textbf{batch} training instances, we sum derivatives with respect to parameters over these instances.

\subsection{Efficiency of back propagation}

\subsection{The Jacobian matrix}
Back-propagation can also be applied to the calculation of other derivatives.Consider the evaluation of the \textbf{Jacobian matrix},whose elements are given by the derivatives of the network outputs with respects to the inputs
\begin{align}
    J_{ki}=\dfrac{\partial y_k}{\partial x_i}
\end{align}

\section{The Hessian Matrix}

\section{Regularization in Neural Networks}

\section{Mixture Density Networks}

\section{Bayesian Neural Networks}


\section{Legendre Polynomials}

The routines in this section compute Legendre polynomials,  normalized associated Legendre polynomials and their derivatives in formats useful to scattering. Associated Legendre polynomials are computed at both positive and negative $m$ and automatically include the Condon-Shortley phase.

\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Legendre Polynomials, $P_l(x)$}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The Legendre polynomials satisfy the equation
\begin{equation}
\dfrac{d}{dx} \left[ (1-x^2)\dfrac{d}{dx}P_l(x)\right] + l(l+1)P_l(x) = 0
\end{equation}

for $-1 \le x \le 1$, with the following recurrence relation
\begin{equation}
(l+1)P_{l+1}(x) = (2l+1)xP_l(x) - lP_{l-1}(x) \label{plrec}
\end{equation}

The routine \texttt{legendrePl} returns $P_l(x)$ on a 2D array with degrees $0$ to $L$ along rows and evaluation points along columns.

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/legendrePl.m}
}

\clearpage
\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Legendre Polynomial Derivative, $d/dx P_l(x)$}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The derivative of the Legendre polynomials satisfies either of the following recurrence relations
\begin{equation}
\dfrac{x^2-1}{l}\dfrac{d}{dx}P_l(x) = xP_l(x) - P_{l-1}(x)
\end{equation}

\begin{equation}
(2l+1)P_l(x) = \dfrac{d}{dx}(P_{l+1}(x) - P_{l-1}(x))
\end{equation}

The first has a divide by zero when $x$ is 1 or -1.  Rearranging the second equation we get the following recurrence
\begin{equation}
\dfrac{d}{dx}P_{l+1}(x) = (2l+1)P_l(x) + \dfrac{d}{dx} P_{l-1}(x)
\end{equation}

\noindent which is safe on $x=[-1, 1]$.  Initial conditions are the special cases $P'_0(x) = 0$ and $P'_1(x) = 1$, and the derivative has the feature that $P'_l(1) = l(l+1)/2$.  The routine \texttt{legendrePlp} is like \texttt{legendrePl}: it returns $d/dx P_l(x)$ on a 2D array with degrees $0$ to $L$ along rows and evaluation points along columns.

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/legendrePlp.m}
}

\clearpage
\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Normalized Associated Legendre Polynomials, $\widetilde{P}_l^m(x)$}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The associated Legendre polynomials, $P_l^m(x)$, are given by
\begin{equation}
P_l^m(x) = \dfrac{(-1)^m}{2^l l!} (1-x^2)^{m/2} \dfrac{d^{l+m}}{dx^{l+m}} (x^2 - 1)^l \label{plm}
\end{equation}

\noindent for integers $l$ and $m$ such that $l \ge 0$ and $-l \le m \le l$.  The Condon-Shortley phase is included in the definition of the associated Legendre polynomials. The following identity relates positive and negative $m$
\begin{equation}
P_l^{-m}(x) = (-1)^m \dfrac{(l-m)!}{(l+m)!} P_l^m(x)\
\label{eq2}
\end{equation}

The normalized associated Legendre polynomials $\widetilde P_l^m(x)$ are recommended for scattering computations because they avoid direct computation of factorials. These are given by
\begin{equation}
\widetilde P_l^m(x) = \sqrt{\dfrac{(l + \frac{1}{2})(l-m)!}{(l+m)!}} P_l^m(x) \label{plmnorm}
\end{equation}

\noindent where
\begin{equation}
\widetilde P_l^{-m}(x) = (-1)^m \widetilde{P}_l^m(x)
\end{equation}

The routine \texttt{Plm} returns the normalized associated Legendre polynomials for $l=0,...,L$, all $\pm m$, linearly indexed (see Chapter \ref{chap:wavefunctions} on linear indexing). The result is returned with harmonics indexed along rows, and evaluation points along columns. It calls Matlab's \texttt{legendre} with the \texttt{'norm'} option, which returns $m = 0,...,l$ and an extra factor of $(-1)^m$. \texttt{Plm} takes out the extra $(-1)^m$ from Matlab's normalization so that the one factor of $(-1)^m$ in \eqref{plm} remains. %Use the switch \texttt{'scalar'} to include the $(l,m) = (0,0)$ term. The total number of harmonics is $N = L^2 + 2L$ or $N = L^2 + 2L + 1$ for scalar.

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/Plm.m}
}

%
%\subsection{Normalized Associated Legendre Polynomials Derivative, $d/dx\widetilde{P}_l^m(x)$}
%
%The derivative of the associated Legendre polynomials has several recurrence relations that are given in terms of the non-differentiated functions, but they all have the problem of divide by zero at $\pm1$. A mixed recurrence on $l$ can be derived starting with the unnormalized recurrence
%\eq{(l-m+1)P_{l+1}^m(x) = (2l+1)xP_l^m(x) - (l+m)P_{l-1}^m(x)}
%
%and differentiating, which gives
%\eq{(l-m+1)\dfrac{d}{dx}P_{l+1}^m(x) =  (2l+1)\left( P_l^m(x) +  x\dfrac{d}{dx}P_l^m(x)\right)- (l+m)\dfrac{d}{dx}P_{l-1}^m(x) \label{dpplus}}
%
%Substituting the expression for the normalized Legendre polynomials into \eqref{dpplus} and simplifying
%%
%%\eq{\sqrt{\dfrac{(l+m)!}{(l + \frac{1}{2})(l-m)!}} \widetilde P_l^m(x) =  P_l^m(x)}
%%
%%\ea{(l-m+1)\sqrt{\dfrac{(l+1+m)!}{(l +1+ \frac{1}{2})(l+1-m)!}} \dfrac{d}{dx}\widetilde P_{l+1}^m(x) &=&  (2l+1)\sqrt{\dfrac{(l+m)!}{(l + \frac{1}{2})(l-m)!}} \left( \widetilde P_l^m(x)   +  x\dfrac{d}{dx}\widetilde P_l^m(x)\right)\\
%%\ &\ & - (l+m)\sqrt{\dfrac{(l-1+m)!}{(l -1 + \frac{1}{2})(l-1-m)!}} \dfrac{d}{dx}\widetilde P_{l-1}^m(x)  \label{dpplus} }
%%
%%\ea{(l-m+1)\sqrt{\dfrac{(l+1+m)(l+m)}{(l +1+ \frac{1}{2})(l+1-m)(l-m)}} \dfrac{d}{dx}\widetilde P_{l+1}^m(x) &=&  (2l+1)\sqrt{\dfrac{(l+m)}{(l + \frac{1}{2})(l-m)}} \left( \widetilde P_l^m(x)   +  x\dfrac{d}{dx}\widetilde P_l^m(x)\right)\\
%%\ &\ & - (l+m)\sqrt{\dfrac{1}{(l -1 + \frac{1}{2})}} \dfrac{d}{dx}\widetilde P_{l-1}^m(x)  \label{dpplus} }
%
%\ea{  \dfrac{d}{dx}\widetilde P_{l+1}^m(x) &=&   a_{l,m} \left( \widetilde P_l^m(x)   +  x\dfrac{d}{dx}\widetilde P_l^m(x)\right) -  b_{l,m} \dfrac{d}{dx}\widetilde P_{l-1}^m(x)  }
%%
%%\ea{a_{l,m} &=& \dfrac{1}{(l-m+1)}\sqrt{\dfrac{(l +1+ \frac{1}{2})(l+1-m)(l-m)}{(l+1+m)(l+m)}}  (2l+1)\sqrt{\dfrac{(l+m)}{(l + \frac{1}{2})(l-m)}} \\
%%b_{l,m} &=&  \dfrac{1}{(l-m+1)}\sqrt{\dfrac{(l +1+ \frac{1}{2})(l+1-m)(l-m)}{(l+1+m)(l+m)}}(l+m)\sqrt{\dfrac{1}{(l -1 + \frac{1}{2})}}}
%
%\ea{a_{l,m} &=&  2 \sqrt{\dfrac{(l + \frac{3}{2})(l + \frac{1}{2}) }{(l-m+1)(l+m+1) }}   \\
%b_{l,m} &=&   \sqrt{\dfrac{(l +\frac{3}{2}) (l-m)(l+m)}{(l - \frac{1}{2})(l-m+1)(l+m+1) }} }
%
%This is initialized along the $m = l$ diagonal with
%\eq{ \dfrac{d}{dx} \widetilde P_{0}^0(x) = 0}
%\eq{  \dfrac{d}{dx} \widetilde P_{l+1}^l(x) = \sqrt{\dfrac{(2l+1)(l + \frac{3}{2})}{(l + \frac{1}{2})}} \left(\widetilde P_l^l(x) + x \dfrac{d}{dx} \widetilde P_{l}^l(x) \right)}
%
%which comes from differentiating $P_{l+1}^l (x) = x(2l+1)P_l^l(x)$ and normalizating.
%
%%\eq{\sqrt{\dfrac{(l+m)!}{(l + \frac{1}{2})(l-m)!}} \widetilde P_l^m(x) =  P_l^m(x)}
%
%
%%\eq{ \sqrt{\dfrac{(l+1+m)!}{(l + 1 + \frac{1}{2})(l+1-m)!}} \widetilde P_{l+1}^m(x) = x(2l+1) \sqrt{\dfrac{(l+m)!}{(l + \frac{1}{2})(l-m)!}} \widetilde P_l^m(x) }
%%
%%\eq{ \sqrt{\dfrac{(l+1+l)!}{(l + 1 + \frac{1}{2})}} \widetilde P_{l+1}^l(x) = x(2l+1) \sqrt{\dfrac{(l+l)!}{(l + \frac{1}{2})}} \widetilde P_l^l(x) }
%%
%%\eq{ \sqrt{\dfrac{(2l+1)}{(l + 1 + \frac{1}{2})}} \widetilde P_{l+1}^l(x) = x(2l+1) \sqrt{\dfrac{1}{(l + \frac{1}{2})}} \widetilde P_l^l(x) }
%
%%\eq{  \widetilde P_{l+1}^l(x) = c_{l,m} x  \widetilde P_l^l(x) }
%
%
%This recurrence requires both degrees less than or equal to $l$.  This is given by the routine \texttt{Plmp}, and has the same structure as \texttt{Plm}.
%
%{\footnotesize
%\VerbatimInput{\code/LegendrePolynomials/Plmp.m}
%}
%
%

\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Normalized Associated Legendre Polynomials Derivative, $d/dx\widetilde{P}_l^m(x)$}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The derivative of the normalized associated Legendre polynomials is found by substituting \eqref{plmnorm} into the following non-normalized recurrence relation for the associated Legendre polynomial derivative:
\eq{(x^2-1) \dfrac{d}{dx} P_l^m(x) = lx P_l^m(x) - (l+m)P_{l-1}^m(x)}

which gives
\begin{equation}
\dfrac{d}{dx}\widetilde P_l^m(x) = \dfrac{1}{x^2-1}\left( lx \widetilde P_l^m(x) - \sqrt{\dfrac{(l+1/2)}{(l-1/2)}}\sqrt{(l+m)(l-m)} \widetilde P_{l-1}^m(x)\right)
\end{equation}

This recurrence only requires $\widetilde P_l^m(x)$ to be computed at degrees less than or equal to $l$.  This is computed by the routine \texttt{Plmp}, and has the same structure as \texttt{Plm}, which returns $l=0,...,L$, all $\pm m$, linearly indexed (see Chapter \ref{chap:wavefunctions}). This version is not suitable at the end points $x = [-1, 1]$. In fact, the difficultly of computing the derivative of the associated Legendre polynomials at the end-points is well-known, and other routines can be found elsewhere. However, we use this routine in scattering problems when the poles do not need to be sampled, for example, when using Gauss-Legendre quadrature as the basis for spherical harmonic transforms (see Chapter \ref{chap:fmm}).

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/Plmp.m}
}

\clearpage
\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Normalized Associated Legendre Polynomials, $m\widetilde{P}_l^m(\cos\theta)/\sin\theta $}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The variant $m\widetilde{P}_l^m(\cos\theta)/\sin\theta $ is needed for vector wave functions. Numerically, direct division by $\sin\theta$ is a problem at the poles, however, analytically, dividing by $\sin\theta$ is harmless because $P_l^m(\cos\theta) \sim \sin^m\theta$.  With the right recurrence relation, it can be computed directly.  Start with the following unnormalized recurrence relation
\begin{equation}
\dfrac{m}{\sin\theta}P_l^m(\cos\theta) = -\dfrac{1}{2}\left(P_{l-1}^{m+1}(\cos\theta) + (l+m-1)(l+m)P_{l-1}^{m-1}(\cos\theta) \right)
\end{equation}

Then substituting \eqref{plmnorm} and canceling like factorials
\begin{eqnarray}
\dfrac{m}{\sin\theta}\widetilde P_l^m(\cos\theta)& =& \dfrac{1}{2}\sqrt{\dfrac{l-\frac{1}{2}}{l+\frac{1}{2}}}\left(\sqrt{(l-m)(l-m-1)}\widetilde P_{l-1}^{m+1}(\cos\theta)\right. \nonumber \\
\ & \ & +\left. \sqrt{(l+m)(l+m-1)}\widetilde P_{l-1}^{m-1}(\cos\theta) \right)
\end{eqnarray}

This is computed by the routine \texttt{mPlmsin}, and has similar structure to \texttt{Plm}.  It returns degrees $l = 1,..,L$, all $m$, linearly indexed along rows with evaluation points along columns.  We exclude the monopole because this variant is only used for vector wave functions. It uses the initial condition
\begin{equation}
\dfrac{(1)}{\sin\theta}\widetilde P_1^1(\cos\theta) = \sqrt{\dfrac{3}{4}}
\end{equation}

Technically this initial condition should have a negative sign. However, we iterate on the output of the Matlab's \texttt{legendre}, which has an extra negative sign. Therefore, we adjust the negative signs at the end of the routine so that the outputs will match a direct computation of the quantity $m\widetilde{P}_l^m(\cos\theta)/\sin\theta $ when using \texttt{Plm}.

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/mPlmsin.m}
}

\addtocontents{toc}{\protect\setcounter{tocdepth}{1}}
\subsection{Normalized Associated Legendre Polynomials Derivative, $d/d\theta\widetilde{P}_l^m(\cos\theta)$}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

The variant $d/d\theta\widetilde{P}_l^m(\cos\theta)$ is the derivative of the normalized associated Legendre polynomials with respect to $\theta$ when the argument is $\cos\theta$.  Start with the recurrence relation for unnormalized Legendre polynomials
\begin{eqnarray}
\dfrac{d}{d\theta}P_l^m(\cos\theta) &= &-\sin\theta \dfrac{d}{d\cos\theta}P_l^m(\cos\theta)  \\
\ &=& -\dfrac{1}{2}\left((l+m)(l-m+1)P_l^{m-1}(\cos\theta) -P_l^{m+1}(\cos\theta)\right)
\end{eqnarray}

Substituting the expression for the normalized Legendre polynomials and canceling like factorials
\eq{
\dfrac{d}{d\theta}\widetilde P_l^m(\cos\theta) = \dfrac{1}{2}\left(\sqrt{(l+m)(l-m+1)}\widetilde P_l^{m-1}(\cos\theta) - \sqrt{(l-m)(l+m+1)}\widetilde P_l^{m+1}(\cos\theta)\right)
}
%
%\begin{eqnarray}
%\dfrac{d}{d\theta}\widetilde P_l^m(\cos\theta) &=& \dfrac{1}{2}\left(\sqrt{(l+m)(l-m+1)}\widetilde P_l^{m-1}(\cos\theta)\right. \nonumber \\
%\ & \ & -\left. \sqrt{(l-m)(l+m+1)}\widetilde P_l^{m+1}(\cos\theta)\right)
%\end{eqnarray}

This recurrence only requires degrees equal to $l$.  This is given by the routine \texttt{Plmp2}, and has similar structure to \texttt{Plm}.  It returns degrees $l = 1,..,L$, all $m$, linearly indexed in rows and evaluation points along columns.  We exclude the monopole because this variant is used for vector wave functions, and because the derivative for $(l,m) = (0,0)$ is zero.  It uses the initial condition
\begin{equation}
\dfrac{d}{d\theta}\widetilde{P}_l^0(\cos\theta) = -\sqrt{l(l+1)}\widetilde{P}_l^1(\cos\theta)
\end{equation}

{\footnotesize
\VerbatimInput{\code/LegendrePolynomials/Plmp2.m}
}