% Copyright ©2015 The gonum Authors. All rights reserved.
% Use of this source code is governed by a BSD-style
% license that can be found in the LICENSE file.

\documentclass{article}

\usepackage{amsmath,amsfonts}
\usepackage[margin=4cm]{geometry}

\title{Louvain algorithm for undirected and directed graphs}
\author{The {\tt gonum} Authors}

\begin{document}

\maketitle

The algorithm attempts to find communities (highly connected sub-graphs),
and it does this by minimising the modularity function
\begin{equation}
	Q(c) = \frac{1}{2m}\sum_i\sum_j\left[ A_{ij} - \gamma \frac{k_ik_j}{2m} \right] \delta_{ij}(c),
\end{equation}
where $c$ is a partition of nodes into subsets or communities,
$A_{ij}$ is the edge weight between nodes $i$ and $j$,
$\gamma$ is a tuning parameter,
\begin{equation}
m = \frac{1}{2}\sum_i\sum_jA_{ij},
\end{equation}
\begin{equation}
k_i = \sum_j{A_{ij}},
\end{equation}
and
\begin{equation}
	\delta_{ij}(c) = \left \{ \begin{array}{ll}
	1 & \text{if} \quad  c(i) = c(j) \\
	0 & \text{otherwise} \end{array} \right ..
\end{equation}
Here $c(i)$ denotes the community to which node $i$ belongs
in the partitioning $c$.

The algorithm finds a hierarchical community structure by iterating
between two phases:
\begin{enumerate}
	\item Find a set of communities that minimise $Q$.
	\item Construct a new graph, whose nodes are the communities
	found in the preceding phase one step.
\end{enumerate}
Each iteration of these two phases is called a `pass'.
In this way, the algorithm obtains a nested community structure,
where at each level $Q$ is minimised for the relevant graph.
We consider this process in more detail, in particular looking
at phase one first in the first pass, when each node is a single
node, and then how this generalises to later passes when each node
is a community.

\section{Undirected Graphs}

\subsection{Initial Pass}
\label{sec:initialPass}

The initial pass is simple as the initial pass uses the original graph,
and in all following passes graphs constructed in the previous pass's
phase two are used.
Here we will consider this initial simple formulation for phase one, and
in Section~\ref{sec:laterPasses} we consider how this generalises for
passes two and onwards.
Phase one works by initially allocating each node to a separate community,
and then iterating through each node $a$ and checking if moving it into
a different community $\beta$ will reduce $Q$.
If there are possible moves that will reduce $Q$, $a$ is moved into the
the community which will generate the largest reduction in $Q$.
This process is continued until there are no moves left to reduce $Q$
further, meaning a local minimum for $Q$ has been achieved.
Then the algorithm moves to phase two (constructing a new graph where
each node in the new graph is a community in the old graph).

Note that we assume the original graph to be simple and undirected.
First, we introduce some notation that will be useful:
Let $c(i)$ denote the community to which node $i$ belongs,
and let $\alpha$ be the community that the node $a$ mentioned above
belongs to, i.e., $\alpha = c_a$.
Then we define
\newcommand{\Stot}[1]{\Sigma_{\text{tot}}^{#1}}
\begin{equation}
	\Stot{\alpha} = \sum_{i \in \alpha}\sum_{j}A_{ij} = \sum_{i \in \alpha}k_i,
\end{equation}
\newcommand{\kin}[2]{k_{#1}^{#2}}
\begin{equation}
	\kin{i}{\alpha} = \sum_{j \in \alpha}A_{ij},
\end{equation}
and
\newcommand{\Sin}[1]{\Sigma_{\text{in}}^{#1}}
\begin{equation}
	\Sin{\alpha} = \sum_{i \in \alpha}\sum_{j \in \alpha}A_{ij} = \sum_{i \in \alpha}\kin{i}{\alpha}.
\end{equation}

We are interested in how $Q$ will change if we move a node $a$ from its
current community $\alpha$, to a new community $\beta$.
This will have two effects, it will remove the terms from $Q$
related to $a$ in $\alpha$, which we will call $Q^-$ and it will add terms
related to $a$ in $\beta$, which we will call $Q^+$.
The total change in $Q$ caused by the movement of $a$ from $\alpha$ to $\beta$ is
\begin{equation}
	\Delta Q = Q^{+} - Q^{-},
\end{equation}
where
\begin{align*}
Q^- &= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2\sum_{i \in \alpha, \, i \neq a} \left( A_{ia} - \gamma \frac{k_ik_a}{2m} \right) \right] \\
	&= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2 \left( \kin{a}{\alpha} -A_{aa}\right) - \gamma \frac{2k_a}{2m}\sum_{i \in \alpha, \, i \neq a} k_i \right] \\
	&= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2 \left( \kin{a}{\alpha} -A_{aa}\right) - \gamma \frac{2k_a}{2m}\left( \Stot{\alpha} - k_a \right)  \right], \\
\end{align*}
and
\begin{align*}
Q^+ &= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2\sum_{i \in \beta} \left( A_{ia} - \gamma \frac{k_ik_a}{2m} \right) \right] \\
		&= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2\kin{a}{\beta} - \gamma \frac{2k_a}{2m}\sum_{i \in \beta} k_i \right] \\
		&= \frac{1}{2m}\left[ \left( A_{aa} - \gamma \frac{k_a^2}{2m} \right)
+ 2\kin{a}{\beta} - \gamma \frac{2k_a\Stot{\beta}}{2m} \right]. \\
\end{align*}
The first term in both these expressions ($Q^-$ and $Q^+$) is the same, and so cancels:
\begin{equation}
\Delta Q = \frac{1}{2m}\left[ \left( 2\kin{a}{\beta} - \gamma \frac{2k_a\Stot{\beta}}{2m} \right)
		- \left( 2 \left( \kin{a}{\alpha} -A_{aa}\right) - \gamma \frac{2k_a}{2m}\left( \Stot{\alpha} - k_a \right) \right) \right].
\end{equation}

\subsection{Later Passes}
\label{sec:laterPasses}

In phase two a `meta-graph' is constructed where nodes correspond to
the communities found in the preceding phase one step, and edge weight
between two such communities (nodes, in the meta-graph)
$\alpha$ and $\beta$ are defined to be
\begin{equation}
	A_{\alpha \beta}^* = \sum_{i \in \alpha}\sum_{j \in \beta}A_{ij}.
	\label{eqn:Aij*}
\end{equation}
Note that $i$ and $j$ refer to nodes in the original graph, not nodes
in the previous graph, and so holds any meta-graph, not just the first.
Also note that this definition of $A^*_{\alpha \beta}$ allows for
$A^*_{\alpha \alpha}$ to be non-zero as
\begin{equation}
A_{\alpha \alpha}^* = \sum_{i \in \alpha}\sum_{j \in \alpha}A_{ij} = \Sin{\alpha}.
\end{equation}

In this newly constructed graph, $\alpha$ and $\beta$ are nodes, but
also refer to communities (sets of nodes) in the original graph, and I
use these two interpretations interchangeably.
This should be the only ambiguous bit of notation in this document, I hope.

The results of Section~\ref{sec:initialPass} generalise to these meta-graphs,
and the generalised results mirror those of Section~\ref{sec:initialPass} closely
-- I distinguish the new results from those of Section~\ref{sec:initialPass} by a
superscript $*$.
I use $i$ and $j$ to denote nodes of the original graph as in Section~\ref{sec:initialPass},
and use $z$ and $w$ to denote nodes of the meta-graph (communities of the original).
I use analogous notation to Section~\ref{sec:initialPass}, $c^*(z)$, to
denote the community to which node $z$ of the meta-graph belongs,
and let $\mathfrak{a}$ be the community that the node $\alpha$ belongs to
($c^*(\alpha) = \mathfrak{a}$), i.e.
\begin{equation}
	\mathfrak{a} = \{z | c^*(z) = c^*(\alpha) \}.
\end{equation}

Given this notation, we can observe that
\begin{equation}
m^* = \frac{1}{2}\sum_{z}\sum_{w}{A_{zw}^*} = \frac{1}{2}\sum_{z}\sum_{w}{\sum_{i \in z}\sum_{j \in w}A_{ij}} = \frac{1}{2}\sum_i\sum_jA_{ij} = m,
\end{equation}
\begin{equation}
k_{z}^* = \sum_{w}{A_{zw}^*} =  \sum_{w}{\sum_{i \in z}\sum_{j \in w}A_{ij}} = \sum_{i \in z}\sum_{j}A_{ij} = \Stot{z},
\end{equation}
\begin{equation}
	\Stot{\mathfrak{a} *} = \sum_{z \in \mathfrak{a}}\sum_{w}A_{zw}^* = \sum_{z \in \mathfrak{a}}k_z^* = \sum_{z \in \mathfrak{a}}\Stot{z},
\end{equation}
\begin{equation}
	\kin{z}{\mathfrak{a} *} = \sum_{w \in \mathfrak{a}}{A_{zw}^*} = \sum_{w \in \mathfrak{a}}{\sum_{i \in z}\sum_{j \in w}A_{ij}},
\end{equation}
and
\begin{equation}
\Sin{\mathfrak{a} *} = \sum_{z \in \mathfrak{a}}\sum_{w \in \mathfrak{a}}A_{zw}^* = \sum_{z \in \mathfrak{a}}\kin{z}{\mathfrak{a} *} = \sum_{z \in \mathfrak{a}}\sum_{w \in \mathfrak{a}}{\sum_{i \in z}\sum_{j \in w}A_{ij}}.
	%\label{eqn:Sin}
\end{equation}

If we let $\mathfrak{b}$ denote the community to which we are considering moving $\alpha$,
then the expression for $\Delta Q$ from Section~\ref{sec:initialPass} trivially generalises to
\begin{equation}
\Delta Q = \frac{1}{2m}\left[ \left( 2 \kin{\alpha}{\mathfrak{b} *} - \gamma \frac{2k_{\alpha}^*\Stot{\mathfrak{b} *}}{2m} \right)
		- \left( 2\left( \kin{\alpha}{\mathfrak{a} *} - A_{\alpha \alpha}^* \right) - \gamma \frac{2k_{\alpha}^*}{2m} \left( \Stot{\mathfrak{a} *} - k_{\alpha}^* \right ) \right) \right] \\
\end{equation}

\section{Directed Graphs}
\label{sec:directedGraphs}

It is of interest to consider how this generalises to directed graphs.
If we are to treat incoming and outgoing nodes equally, there are several
thoughts on how to extend the algorithm to directed graphs, of which we
will explore three:
\begin{itemize}
	\item Construct an undirected graph first, and then use the undirected case.
	\item Generalise the expressions from the undirected case to the directed case,
	we will consider two different suggestions for such generalisations.
\end{itemize}
We will show that one of the two `generalisation of expressions' approaches is
equivalent to constructing an undirected graph, and the other is not.

\subsection{Construction of an undirected graph}
A simple approach to generalising to directed graphs is to construct
an undirected graph with edge weights
\begin{equation}
A_{ij} = B_{ij} + B_{ji},
\label{eqn:undirectedAB}
\end{equation}
and simply use the undirected algorithm.
Another suggestion is to average the directed edges to make
an undirected graph, i.e. to use a directed graph with edge weights
\begin{equation}
A_{ij} = \frac{B_{ij} + B_{ji}}{2}.
\end{equation}
This raises an important question: does scaling all edge weights across
the entire graph by a constant affect the results of the algorithm?
Hopefully not, but worth checking.
We can follow this through the results for the undirected graph by
substituting $A_{ij}^{(1)} = pA_{ij}$, $p \in \mathbb{R}$, and
distinguishing the new expressions by a superscript ${(1)}$. These
new expressions are:
\begin{equation}
m^{(1)} = \frac{1}{2}\sum_i\sum_jpA_{ij}  = p\frac{1}{2}\sum_i\sum_j A_{ij} = pm ,
\end{equation}
\begin{equation}
k_i^{(1)} = \sum_j{pA_{ij}} = p\sum_j{A_{ij}} = pk_i,
\end{equation}
and so
\begin{align*}
	Q^{(1)}(c) &= \frac{1}{2pm}\sum_i\sum_j\left[ pA_{ij} - \gamma \frac{pk_ipk_j}{2pm} \right] \delta_{ij}(c) \\
	&= \frac{1}{2m}\sum_i\sum_j\left[ A_{ij} - \gamma \frac{k_ik_j}{2m} \right] \delta_{ij}(c) \\
	&= Q(c)
\end{align*}
Note that as we have shown $Q^{(1)} = Q$ there is no need to go into the remainder of the terms
involved in the algorithm, as they all derive from $Q$.

\subsection{First generalisation of expressions approach}

One suggested extension to directed graphs is to modify the expressions
involved by adding the `from' case and the `to' case for each term.
If we let $B_{ij}$ be the edge weight between nodes $i$ and $j$ in
the directed graph, and distinguishing these extended expressions by
a superscript $(2)$, the extended expressions become:
\begin{equation}
m^{(2)} = \frac{1}{2}\left ( \sum_i\sum_jB_{ij} + \sum_i\sum_jB_{ji}\right)  = \frac{1}{2}\sum_i\sum_j \left( B_{ij} + B_{ji} \right) ,
\end{equation}
\begin{equation}
k_i^{(2)} = \sum_jB_{ij} + \sum_jB_{ji} = \sum_j{\left( B_{ij} + B_{ji} \right)},
\end{equation}
and similarly
\begin{equation}
	Q^{(2)}(c) = \frac{1}{2m}\sum_i\sum_j\left[ \left( B_{ij} + B_{ji} \right) - \gamma \frac{k_i^{(2)}k_j^{(2)}}{2m} \right] \delta_{ij}(c).
\end{equation}

Note how this is equivalent to the construction of an undirected graph as
per Equation~(\ref{eqn:undirectedAB}). Similarly to above,
there is no need to go into the remainder of the terms
involved in the algorithm, as they all derive from $Q$.

\subsection{Second generalisation of expressions approach}

Another approach to generalising the expressions to the
directed case, that still treats incoming and outgoing edges
as equally important, is to propose an alternative modularity
expression:
\newcommand{\dkin}[1]{k_{#1}^{\text{in}}}
\newcommand{\dkout}[1]{k_{#1}^{\text{out}}}
\begin{equation}
Q^{(3)}(c) = \frac{1}{2m}\sum_i\sum_j\left[ 2B_{ij} - 2\gamma \frac{\dkin{i}\dkout{j}}{2m} \right] \delta_{ij}(c), \\
\end{equation}
where
\begin{equation}
\dkout{i} = \sum_j{B_{ij}}
\quad \quad \text{and} \quad \quad
\dkin{i} = \sum_j{B_{ji}},
\end{equation}
so $k_i^{(2)} = \dkin{i} + \dkout{i}$.
Note I leave the factor of two in the expression for $Q^{(3)}$ so that it
remains as comparable to that for $Q^{(2)}$ as possible.
There is no need for alternative $m$, as it will still be the same as above.
$Q^{(3)}$ will differ from $Q^{(2)}$ in two ways.
Firstly, as $k_i^{(2)} = \dkin{i} + \dkout{i}$,
\begin{align*}
\sum_i\sum_j k_i^{(2)} k_j^{(2)} \delta_{ij}(c)  &= \sum_i\sum_j  (\dkin{i} + \dkout{i}) (\dkin{j} + \dkout{j}) \delta_{ij}(c) \\
 &= \sum_i\sum_j \left[ (\dkin{i}\dkin{j} + \dkout{i}\dkout{j}) + (\dkin{i}\dkout{j} + \dkin{j}\dkout{i}) \right] \delta_{ij}(c). \\
 &= \sum_i\sum_j \left[ (\dkin{i}\dkin{j} + \dkout{i}\dkout{j}) + 2\dkin{i}\dkout{j} \right] \delta_{ij}(c), \\
\end{align*}
and similarly,
\begin{equation}
\sum_i\sum_j  \left( B_{ij} + B_{ji} \right) \delta_{ij}(c) = 2\sum_i\sum_j B_{ij} \delta_{ij}(c).
\end{equation}
From these two expressions, we can see that
\begin{equation}
Q^{(3)} - Q^{(2)} = \frac{1}{2m}\sum_i\sum_j \gamma \frac{\dkin{i}\dkin{j} + \dkout{i}\dkout{j}}{2m} \delta_{ij}(c).
\end{equation}

\section{Directed Graphs in more detail}
\label{sec:directedGraphsDetail}

In Section \ref{sec:directedGraphs} we essentially showed three
things:
\begin{itemize}
	\item How an undirected graph could be constructed from a directed
	graph, thereby allowing the undirected algorithm to be used for
	directed graphs.
	\item How scaling all edge weights by a non-zero constant would not
	affect the modularity function.
	\item An alternative approach to extending the algorithm to
	directed graphs that is not equivalent to first reducing it
	to an undirected graph.
\end{itemize}
It is this third point that we will explore here.
Analogously to Sections \ref{sec:initialPass} and \ref{sec:laterPasses} we will
break this up into the initial pass and the later passes.

\subsection{Initial pass}
\label{sec:initialPassDirected}

Continuing with the notation of Section \ref{sec:initialPass}, in which
$c(i)$ denotes the community to which node $i$ belongs,
and $\alpha = c(a)$, we define
\newcommand{\dinStot}[1]{\Sigma_{\text{tot}}^{\text{in}(#1)}}
\newcommand{\doutStot}[1]{\Sigma_{\text{tot}}^{\text{out}(#1)}}
\begin{equation}
	\doutStot{\alpha} = \sum_{i \in \alpha}\sum_{j}B_{ij} = \sum_{i \in \alpha}\dkout{i}
	\quad \quad \text{and} \quad \quad
	\dinStot{\alpha} = \sum_{i \in \alpha}\sum_{j}B_{ji} = \sum_{i \in \alpha}\dkin{i},
\end{equation}
\newcommand{\dinkin}[2]{k_{#1}^{\text{in}(#2)}}
\newcommand{\doutkin}[2]{k_{#1}^{\text{out}(#2)}}
\begin{equation}
	\doutkin{i}{\alpha} = \sum_{j \in \alpha}B_{ij}
	\quad \quad \text{and} \quad \quad
	\dinkin{i}{\alpha} = \sum_{j \in \alpha}B_{ji},
\end{equation}
and we will entertain one more ambiguous notation choice:
%\newcommand{\Sin}[1]{\Sigma_{\text{in}}^{#1}}
\begin{equation}
	\Sin{\alpha} = \sum_{i \in \alpha}\sum_{j \in \alpha}B_{ij} = \sum_{i \in \alpha}\doutkin{i}{\alpha} = \sum_{i \in \alpha}\dinkin{i}{\alpha}.
\end{equation}

Analogously to Section \ref{sec:initialPass}, we are interested in how
$Q^{(3)}$ will change if we move a node $a$ from its
current community $\alpha$, to a new community $\beta$,
and analogously this will have two effects -- it will remove the terms
from $Q^{(3)}$ related to $a$ in $\alpha$, which we will call $Q^{-(3)}$
and it will add terms related to $a$ in $\beta$, which we will call $Q^{+(3)}$.
The total change in $Q^{(3)}$ caused by the movement of $a$ from $\alpha$ to $\beta$ is
\begin{equation}
	\Delta Q^{(3)} = Q^{+(3)} - Q^{-(3)},
\end{equation}
where
\begin{align*}
Q^{-(3)} &= \frac{1}{2m}\left[ \left( 2B_{aa} - 2\gamma \frac{\dkin{a}\dkout{a}}{2m} \right)
+ \sum_{i \in \alpha, \, i \neq a} \left( 2B_{ia} + 2B_{ai} - 2\gamma \frac{\dkin{i}\dkout{a}}{2m} - 2\gamma \frac{\dkin{a}\dkout{i}}{2m} \right) \right] \\
				 &= \frac{1}{2m}\left[ \left( 2B_{aa} - 2\gamma \frac{\dkin{a}\dkout{a}}{2m} \right)
+ 2(\dinkin{a}{\alpha} - B_{aa}) + 2(\doutkin{a}{\alpha} - B_{aa}) \hdots \right . \\
 & \quad \quad \quad \quad \quad \quad \left .
- \frac{2\gamma\dkout{a}}{2m} (\dinStot{\alpha} - \dkin{a}) - \frac{2\gamma\dkin{a}}{2m} (\doutStot{\alpha} - \dkout{a}) \right] \\
\end{align*}
and
\begin{align*}
Q^{+(3)} &= \frac{1}{2m}\left[ \left( 2B_{aa} - 2\gamma \frac{\dkin{a}\dkout{a}}{2m} \right)
+ \sum_{i \in \beta} \left( 2B_{ia} + 2B_{ai} - 2\gamma \frac{\dkin{i}\dkout{a}}{2m} - 2\gamma \frac{\dkin{a}\dkout{i}}{2m} \right) \right] \\
				 &= \frac{1}{2m}\left[ \left( 2B_{aa} - 2\gamma \frac{\dkin{a}\dkout{a}}{2m} \right)
+ 2\dinkin{a}{\beta} + 2\doutkin{a}{\beta} - \frac{2\gamma\dkout{a}}{2m} \dinStot{\beta}  - \frac{2\gamma\dkin{a}}{2m} \doutStot{\beta} \right] \\
\end{align*}
Similarly to Section \ref{sec:initialPass}, the first term in both these expressions is the same, and so cancels, leaving:
\begin{align*}
\Delta Q^{(3)} &= \frac{2}{2m}\left[
\left( \dinkin{a}{\beta} + \doutkin{a}{\beta} - \frac{\gamma\dkout{a}}{2m} \dinStot{\beta}  - \frac{\gamma\dkin{a}}{2m} \doutStot{\beta} \right) \right. \\
& \hspace{-1cm}
- \left. \left( (\dinkin{a}{\alpha} - B_{aa}) + (\doutkin{a}{\alpha} - B_{aa}) - \frac{\gamma\dkout{a}}{2m} (\dinStot{\alpha} - \dkin{a}) - \frac{\gamma\dkin{a}}{2m} (\doutStot{\alpha} - \dkout{a})  \right) \right] \\
 &= \frac{2}{2m}\left[ (\dinkin{a}{\beta}-\dinkin{a}{\alpha}) + (\doutkin{a}{\beta}-\doutkin{a}{\alpha}) + 2B_{aa} \right. \\
& \hspace{-1cm} \left.
- \frac{\gamma\dkout{a}}{2m} (\dinStot{\beta}-\dinStot{\alpha}) - \frac{\gamma\dkin{a}}{2m} (\doutStot{\beta} - \doutStot{\alpha}) - \frac{2\gamma\dkin{a}\dkout{a}}{2m} \right]
\end{align*}

\subsection{Later passes}
\label{sec:laterPassesDirected}

In phase two a `meta-graph' is constructed where nodes correspond to
the communities found in the preceding phase one step, and edge weight
between two such communities (nodes, in the meta-graph)
$\alpha$ and $\beta$ are defined to be
\begin{equation}
	B_{\alpha \beta}^* = \sum_{i \in \alpha}\sum_{j \in \beta}B_{ij}.
	\label{eqn:Bij*}
\end{equation}
Note that $i$ and $j$ refer to nodes in the original graph, not nodes
in the previous graph, and so holds any meta-graph, not just the first.
Also note that this definition of $B^*_{\alpha \beta}$ allows for
$B^*_{\alpha \alpha}$ to be non-zero, in fact
\begin{equation}
B_{\alpha \alpha}^* = \sum_{i \in \alpha}\sum_{j \in \alpha}B_{ij} = \Sin{\alpha}.
\end{equation}

In this newly constructed graph, $\alpha$ and $\beta$ are nodes, but
also refer to communities (sets of nodes) in the original graph, and I
use these two interpretations interchangeably, completely analogously to
Section \ref{sec:laterPasses}.

The results of Section~\ref{sec:initialPassDirected} generalise to these meta-graphs,
and the generalised results mirror those of Section~\ref{sec:initialPassDirected} closely
-- I distinguish the new results from those of Section~\ref{sec:initialPassDirected} by a
superscript $*$.
I use $i$ and $j$ to denote nodes of the original graph as in Sections~\ref{sec:initialPass}
and \ref{sec:initialPassDirected},
and use $z$ and $w$ to denote nodes of the meta-graph (communities of the original).
I use analogous notation to Section~\ref{sec:initialPass}, $c^*(z)$, to
denote the community to which node $z$ of the meta-graph belongs,
and let $\mathfrak{a}$ be the community that the node $\alpha$ belongs to,
i.e., $\mathfrak{a} = c^*(\alpha) $.

Given this notation, we get all the same results as in \ref{sec:laterPasses}, but
each split into two cases `out' and `in', separating by direction, essentially, so
\newcommand{\dkinStar}[1]{k_{#1}^{\text{in} *}}
\newcommand{\dkoutStar}[1]{k_{#1}^{\text{out} *}}
\begin{equation}
\dkoutStar{z} = \sum_w{B_{zw}^*} = \sum_w\sum_{i \in z}\sum_{j \in w}B_{ij} = \sum_{i \in z}\sum_jB_{ij} = \doutStot{z},
\end{equation}
\begin{equation}
\dkinStar{z}  = \sum_w{B_{wz}^*} = \sum_w\sum_{i \in z}\sum_{j \in w}B_{ji} = \sum_{i \in z}\sum_jB_{ji} = \dinStot{z},
\end{equation}
\newcommand{\dinStotStar}[1]{\Sigma_{\text{tot}}^{\text{in}(#1) *}}
\newcommand{\doutStotStar}[1]{\Sigma_{\text{tot}}^{\text{out}(#1) *}}
\begin{equation}
	\doutStotStar{\mathfrak{a}} = \sum_{z \in \mathfrak{a}}\sum_{w}B_{zw}^* = \sum_{z \in \mathfrak{a}}\dkoutStar{z} = \sum_{z \in \mathfrak{a}}\doutStot{z},
\end{equation}
\begin{equation}
	\dinStotStar{\mathfrak{a}}  = \sum_{z \in \mathfrak{a}}\sum_{w}B_{wz}^* = \sum_{z \in \mathfrak{a}}\dkinStar{z}  = \sum_{z \in \mathfrak{a}}\dinStot{z},
\end{equation}
\newcommand{\dinkinStar}[2]{k_{#1}^{\text{in}(#2) *}}
\newcommand{\doutkinStar}[2]{k_{#1}^{\text{out}(#2) *}}
\begin{equation}
	\doutkinStar{z}{\mathfrak{a}} = \sum_{w \in \mathfrak{a}}{B_{zw}^*} = \sum_{w \in \mathfrak{a}}{\sum_{i \in z}\sum_{j \in w}B_{ij}},
\end{equation}
\begin{equation}
	\dinkinStar{z}{\mathfrak{a}}  = \sum_{w \in \mathfrak{a}}{B_{wz}^*} = \sum_{w \in \mathfrak{a}}{\sum_{i \in z}\sum_{j \in w}B_{ji}},
\end{equation}
and
\begin{equation}
\Sin{\mathfrak{a} *} = \sum_{z \in \mathfrak{a}}\sum_{w \in \mathfrak{a}}A_{zw}^* = \sum_{z \in \mathfrak{a}}\kin{z}{\mathfrak{a} *} = \sum_{z \in \mathfrak{a}}\sum_{w \in \mathfrak{a}}{\sum_{i \in z}\sum_{j \in w}A_{ij}}.
	%\label{eqn:Sin}
\end{equation}

If we let $\mathfrak{b}$ denote the community to which we are considering moving $\alpha$,
then the expression for $\Delta Q$ from Section~\ref{sec:initialPassDirected} simply generalises as
\begin{align*}
\Delta Q^{(3)} &= \frac{2}{2m}\left[ (\dinkinStar{\alpha}{\mathfrak{b}}-\dinkinStar{\alpha}{\mathfrak{a}}) + (\doutkinStar{\alpha}{\mathfrak{b}}-\doutkinStar{\alpha}{\mathfrak{a}}) + 2B_{\alpha\alpha}^* \right. \\
& \hspace{-1cm} \left.
- \frac{\gamma\dkoutStar{\alpha}}{2m} (\dinStotStar{\mathfrak{b}}-\dinStotStar{\mathfrak{a}}) - \frac{\gamma\dkinStar{\alpha}}{2m} (\doutStotStar{\mathfrak{b}} - \doutStotStar{\mathfrak{a}}) - \frac{2\gamma\dkinStar{\alpha}\dkoutStar{\alpha}}{2m} \right]
\end{align*}

\end{document}


\section{Introduction}

Consider the pure bending of an isotropic and homogeneous thin clamped
plate.
Let $\Omega$ denote the midline of the plate, $\Gamma$ denote the
boundary of $\Omega$ and $w$ denote the vertical displacement.
The buckling eigenvalue problem consists of finding
values $k^{2}$ such that the boundary value problem

\begin{align}
\Delta (\Delta + k^2) w &= 0 \quad \textrm{in} \quad \Omega \label{eq:buck1} \; , \\
w &= 0 \quad \textrm{on} \quad \Gamma \label{eq:buck2} \; ,\\
\dwdn &=0 \quad \textrm{on} \quad \Gamma \label{eq:buck3}
\end{align}
has a non-trivial solution $w$.
It is well known that the values $k^2$ are necessarily real
and positive and there is a countable collection
of such values
$0 < k_{1}^{2} \leq k_{2}^2 \leq \ldots \uparrow \infty$,
counting multiplicites.

\begin{remark}
  When $k = i\alpha$, the differential equation
  \cref{eq:buck1} is known as the modified biharmonic
  equation. As there appears to be no preferred
  name for the equation with real-valued $k$,
  we will refer to \cref{eq:buck1} as the
  oscillatory biharmonic equation.
\end{remark}

The eigenvalues (and eigenfunctions)
of the boundary value problem \cref{eq:buck1,eq:buck2,eq:buck3}
describe the critical buckling load (and deflection)
of a thin plate under
a compressive force applied along the edge. While the
isotropy and homogeneity assumptions are restrictive,
the equations are still of interest in materials design
(CITATIONS). The analytical properties of the eigenvalues and
eigenfunctions are also of interest from a mathematical
perspective, often as they contrast with the eigenvalues
and eigenmodes of an idealized drum (CITATIONS). For instance,
it is well-known that the first eigenfunction of the
drum problem (i.e. the eigenvalue problem of the Laplace
equation with homogeneous Dirichlet boundary conditions)
is either nonpositive or nonnegative on any domain.
In contrast, it is possible for the first buckling
eigenfunction of a convex domain with analytic boundary
to take both positive and negative values \cite{antunes2011buckling}.

A related problem is that of the Stokes eigenvalue problem
which is to find values $k^2$ such that
\begin{align}
\Delta (\Delta + k^2) w &= 0 \quad \textrm{in} \quad \Omega \label{eq:stokes1} \; , \\
\nabla w &= 0 \quad \textrm{on} \quad \Gamma \label{eq:stokes2} \; ,\\
\end{align}
has a non-trivial solution $w$.
While the buckling eigenvalues are identical to the Stokes
eigenvalues on simply connected domains, on multiply
connected domains it turns out that the buckling eigenvalues
are a subset of the Stokes eigenvalues.

In this note, we derive integral conditions on the stream function
associated with the Stokes eigenfunction
to determine whether the Stokes eigenvalue is also a
buckling eigenvalue on multiply connected domains.
Thus, given the Stokes eigenvalues
of the domain and the stream functions associated with the
corresponding eigenfunctions,
the buckling eigenvalues can be determined using a simple
post-processing step.
Given $k^2$ to be a Stokes eigenvalue, we also derive
a well-conditioned second-kind integral equation for computing
the stream function associated with the
Stokes eigenfunction.

Numerical simulations have long played an important
role in the analyses described above and there are a number
of available methods (note that some of the references
included here are written in terms of the related
Stokes eigenvalue problem, which has equivalent
eigenvalues on simply connected domains). We do not
seek to review the literature here but will provide
some context for the present work. Finite element methods
are ubiquitous in mechanical engineering applications
and are flexible with respect to both the governing
equations and the shape of the domain. Such a method
would first obtain a ``weak'' form of the equations
\cref{eq:buck1,eq:buck2,eq:buck3} and then
approximate the solution space in some
finite dimensional basis, resulting in a discrete model
for the buckling problem.
The eigenvalues $k^2$ are then
approximated by the eigenvalues of the discrete
model \cite{johnson1974beam,rannacher1979nonconforming,
  jia2009approximation,carstensen2014guaranteed}.

\note{needs work}

The rest of this paper is organized as follows ...


\section{Linear Transformation and Matrix}

\subsection{Linear Transformation}

\begin{definition}
	A \cindex{linear transformation}  from $V$ to $W$ is a function $T: V \rightarrow W$ that:
	\begin{enumerate}
		\item $T(x+y) = T(x) + T(y)$
		\item $T(c x) = c T(x)$
	\end{enumerate}
\end{definition}

The two linear transformation verification criteria could be combined into one: prove that
\begin{equation}
    T(cx + y) = cTx+Ty
\end{equation}

The \cindex{identity transformation}  $\mathrm{I}_v : V \rightarrow V$ is defined as $\mathrm{I}_v(x) = x$.

The \cindex{zero transformation}  $T_0: V \rightarrow W$ is defined as $T_0 = 0$.

\begin{definition}
	Let $T:V \rightarrow W$ be linear. the \cindex{null space}  $\nullspace{T}$ of $T$ is the set $\set{x \in V : T(x) = 0}$. It is also called the \cindex{kernel} of $T$. It measures how much  information is lost by the transformation $T$.
\end{definition}

\begin{definition}
	The \cindex{range}  of $T$ is defined as $\rangespace{T} = \set{ {T(x):x \in V}}$. It measures how much information is retained by the transformation $T$.
\end{definition}

\begin{theorem}
	Let $T: V \rightarrow W$ be linear. If $\beta=\set{v_i}$ is a basis for $V$, then
	\begin{equation}
		\rangespace{T} =  \vectorspan{T(\beta)} = \vectorspan{\set{T(v_i)}}
	\end{equation}
\end{theorem}

\begin{definition}
	Let $T: V \rightarrow W$ be linear. the \cindex{nullity}  of $T$ is the dimension of $\nullspace{T}$. The \cindex{rank} \label{rankdefinition} of $T$ is the dimension of $\rangespace{T}$.
\end{definition}

\begin{theorem}[\cindex{Dimension Theorem}]
	If $V$ is finite dimensional, $T:V\rightarrow W$ is linear, then
	\begin{equation}
		\dimension{\nullspace{T}} + \dimension{\rangespace{T}} = \dimension{T}
	\end{equation}
\end{theorem}

\begin{proof}
	expand nullity set to a basis and prove the image of extra parameters are independent.
\end{proof}

\begin{theorem}\label{uniquelineartransformation}
	Let $V:\set{v_i}$ and $W:\set{w_i}$ be vector space over $F$, and their dimensions are the same. Then there exists a unique linear transformation $T:V \rightarrow W$ such that $T(v_i) = w_i$.
\end{theorem}

\begin{proof}
    For $\displaystyle x = \sum_{i=1}^{n} a_i v_i$, define $T:V \rightarrow W$ that $\displaystyle T(x) = \sum_{i=1}^n a_i w_i$.
\end{proof}

\thmref{uniquelineartransformation} is useful when proving two functions are the same.

\begin{theorem}
    Let $T: V \rightarrow W$ be a linear transformation. $T$ is one-to-one if and only if $\nullspace{T} = \{ 0 \}$.
\end{theorem}

% matrix representation
\subsection{Matrix Representation}

\begin{definition}
	A \cindex{ordered basis}  for $V$ is a basis for $V$ with a specific order.
\end{definition}

\begin{definition}
	$\set{e_1, e_2, \dots, e_n}$ is the \cindex{standard ordered basis}  for $F^n$. $\set{1, x, \dots, x^n}$ is the \cindex{standard ordered basis}  for $P_n (F)$.
\end{definition}

\begin{definition}
	Let $\beta = \set{u_1, u_2, \dots, u_n}$ be an ordered basis for $V$. $\forall x \in V$, let $\set{a_1, a_2, \dots, a_n}$ be the unique scalar such that
	\begin{equation*}
		x = \sum_{i=1}^n a_i u_i
	\end{equation*}

	the \cindex{coordinate vector}  of $x$ relative to $\beta$, is defined as
	\begin{equation}
		\coordinate{x}_\beta = \begin{bmatrix}
		a_1 \\
		a_2 \\
		\vdots \\
		a_n
		\end{bmatrix}
	\end{equation}

	Note that $\coordinate{u_i}_\beta = e_i$.
\end{definition}

\begin{definition}
	Let $V$ with ordered basis $\beta=\set{v_i}$, $W$ with ordered basis $\gamma:\set{w_i}$, $T:V \rightarrow W$ be linear. There exists unique scala $a_{ij} \in F$ such that
	\begin{equation}
		T(v_j) = \sum_{i=1}^m a_{ij} w_j
	\end{equation}

	The $m \times n$ \cindex{matrix}\footnote{The word matrix is Latin for womb which is the same root as matrimony. The idea is that a matrix is a receptacle for holding numbers. } $A$ defined by $A_{ij}=a_{ij}$ is the \cindex{matrix representation}  of $T$ in the ordered basis $\beta$ and $\gamma$ and write $A=\coordinate{T}_\beta^\gamma$. If $V = W$ and $\beta = \gamma$, we write $A=\coordinate{T}_\beta$.
	\qed
\end{definition}

	Note that the $j$-th column of $A$ is $\coordinate{T(v_j)}_\gamma$: $\coordinate{T}_\beta^\gamma = \coordinate{\dots, \coordinate{T(v_j)}_\gamma, \dots}$.

	Note that $T$ is the relationship between two basis. The value of $T$ might be the same as basis, for example when they are operators on $F^n$, but $T$ and basis are different objects. It is easy to confuse them, especially on $F^n$.

\begin{theorem}
	If $\mathrm{U},T:V \rightarrow W$ are linear transformation that $\coordinate{\mathrm{U}}_\beta^\gamma = \coordinate{T}_\beta^\gamma$, then $\mathrm{U} = T$.
\end{theorem}

\begin{definition}
	\cindex{$\mathcal{L}(V,W)$} contains all linear transformation from $V$ to $W$.
\end{definition}

\begin{theorem}
	Let $T$,$\mathrm{U}$ be linear transformation over $V$ and $W$,
	\begin{enumerate}
		\item $\coordinate{T + \mathrm{U}}_\beta^\gamma = \coordinate{T}_\beta^\gamma  + \coordinate{\mathrm{U}}_\beta^\gamma $
		\item $\coordinate{a T }_\beta^\gamma = a \coordinate{T}_\beta^\gamma $ for all scalar $a$
	\end{enumerate}
\end{theorem}

\begin{theorem}
	let $T:V\rightarrow W$ and $\mathrm{U}:W\rightarrow Z$. Then $\mathrm{UT}: V \rightarrow Z$ is linear.
\end{theorem}

\begin{definition}
	Let $T:V\rightarrow W$ and $\mathrm{U}:W\rightarrow Z$ be linear transformation. $A_{m \times n}=\coordinate{\mathrm{U}}_\alpha^\beta$ and $B_{n \times p}=\coordinate{T}_\beta^\gamma$ where $\alpha=\set{v_i}$, $\beta=\set{w_i}$, $\gamma=\set{z_i}$. Define the \cindex{product} of matrix $AB$ as:
	\begin{equation}
		(AB)_{ij} = \sum_{k=1}^n A_{ik} B_{kj}
	\end{equation}

	then
	\begin{equation}
	    \coordinate{UT}_\alpha^\gamma = \coordinate{U}_\beta^\gamma \coordinate{T}_\alpha^\beta
	\end{equation}
\end{definition}

\begin{proof}
	For product $AB=\coordinate{UT}_\alpha^\gamma$, we have
	\begin{equation}
		\begin{aligned}
			(UT)(v_j) &= U(T(v_j)) = U \left( \sum_{k=1}^m B_{kj} w_k \right) = \sum_{k=1}^m B_{kj} U(w_k) \\
			&= \sum_{k=1}^m B_{kj} \left( \sum_{i=1}^p A_{ik} z_i \right) = \sum_{k=1}^m  \left( \sum_{i=1}^p A_{ik} B_{kj} \right)  z_i \\
			&= \sum_{i=1}^p C_{ij} z_i
		\end{aligned}
	\end{equation}
\end{proof}

\begin{definition}
	the \cindex{Kronecker delta}  $\delta_{ij}$ is defined as
	\begin{equation}
		\delta_{ij} = \begin{cases}
			1 & \text{, if } i = j \\
			0 & \text{, if } i \neq j
 		\end{cases}
	\end{equation}
\end{definition}

\begin{definition}
	The $n\times n$ \cindex{identity matrix} \cindex{$I_n$} is defined as $\left(I_n \right)_{ij} = \delta_{ij}$.
\end{definition}

\begin{theorem}
	Let $u_j$ and $v_j$ be the $j$th column of $AB$ and $B$, then
	\begin{enumerate}
		\item $u_j = A v_j$ : $AB = \coordinate{A v_1, A v_2, \dots, A v_j, \dots, A v_p}$
		\item $v_j = B e_j$ : $B = B \times I_n$
	\end{enumerate}
\end{theorem}

\begin{theorem} Let $T:V \rightarrow W$ be linear, we have
	\begin{equation}
		\coordinate{T(u)}_\gamma = \coordinate{T}_\beta^\gamma \coordinate{u}_\beta
	\end{equation}
\end{theorem}

\begin{proof}
	Fix $u \in V$, and define linear transformation $f: F \rightarrow V$ by $f(a) = a u$ and $g: F \rightarrow W$ by $g(a) = a T(u)$. Let $a=\{1\}$ be the standard basis of $F$. Notice that $g=Tf$. we have:
	\begin{equation}
		[T(u)]_\gamma = [g(1)]_\gamma = [g]_\alpha^\gamma = [Tf]_\alpha^\gamma = \coordinate{T}_\beta^\gamma [f]_\alpha^\beta = \coordinate{T}_\beta^\gamma [f(1)]_\beta = \coordinate{T}_\beta^\gamma [u]_\beta
	\end{equation}
\end{proof}

Note: in the above proof, a vector could be treated as a linear transformation from a field to vector space.

\begin{definition}
	Let $A$ be an $m \times n$ matrix. The mapping \cindex{$L_A$} that $L_A: F^n \rightarrow F^m$ defined by $L_A (x) = A x$ is called \cindex{left-multiplication transformation} . \qed
\end{definition}

A linear transformation is different from matrix:
\begin{enumerate}
    \item Matrix is finite dimensional, so it defines relation only in finite dimension space. A linear transformation could be of any dimension.
    \item For a transformation, its matrix representation depends on the chosen basis.
\end{enumerate}

\begin{theorem}
    \begin{equation}
        \begin{aligned}
            \begin{cases}
            [L_A]_\alpha^\beta &= A \\
            L_{\coordinate{T}_\alpha^\beta} &= T
        \end{cases}
        \end{aligned}
    \end{equation}
\end{theorem}

\subsection{Inverse}

\begin{definition}
	Let $T: V\rightarrow W$ and $\mathrm{U}:W \rightarrow V$ be linear. $\mathrm{U}$ is an \cindex{inverse}   of $T$ if $\mathrm{TU} = I_W$ and $\mathrm{UT} = I_V$. If $T$ has an inverse, $T$ is \cindex{invertable}  , which is denoted as $T^{-1}$.
\end{definition}

\begin{theorem}
    $(\mathrm{UT})^{-1} = T^{-1} \mathrm{U}^{-1}$.
\end{theorem}

\begin{definition}
	Let $A$ be $n \times n$ matrix. $A$ is invertable if there is an $n \times n$ matrix $B$ that $AB=BA=I$.
\end{definition}

\begin{theorem}
    if $T$ is invertible,
	\begin{equation*}
		\coordinate{T^{-1}}_\gamma^\beta = \left(\coordinate{T}_\beta^\gamma \right)^{-1}
	\end{equation*}
\end{theorem}
\begin{proof}
	\begin{equation*}
		I_n = \coordinate{I_V}_\beta = \coordinate{T^{-1} T}_\beta = \coordinate{T^{-1}}_\gamma^\beta \coordinate{T}_\beta^\gamma
	\end{equation*}
\end{proof}

\begin{definition}
	$V$ is \cindex{isomorphic} to $W$ if there exists a linear transformation $T:V\rightarrow W$ that is invertible. $T$ is called an \cindex{isomorphism}  from $V$ to $W$.
\end{definition}

\begin{theorem}
	$V$ is isomorphic to $W$ if $\dimension{V} = \dimension{W}$.
\end{theorem}

\begin{proof}
	If the dimensions are the same, choose basis $\beta$ of $V$ and $\gamma$ of $W$ and create a linear mapping $T:\beta \rightarrow \gamma$ by \thmref{uniquelineartransformation}.
\end{proof}

\begin{theorem}
	Let $V$ be a vector space over $F$. Then $V$ is isomorphic to $F^n$ $\Leftrightarrow$ $\dimension{V} = n$.
\end{theorem}

\begin{theorem}
	The function $\Phi: \mathcal{L}(V,M) \rightarrow M_{m \times n}(F)$ defined by $\Phi (T) = \coordinate{T}_\beta^\gamma$, is an isomorphism. The dimension has relation that
	\begin{equation}
		\dimension{\mathcal{L}(V,M)} = \dimension{V} \times \dimension{W}
	\end{equation}
\end{theorem}

\subsection{Change of Coordinate Matrix}

\begin{theorem}
	Let $\beta$ and $\beta^\prime$ be two ordered basis of $V$. Let $Q = \coordinate{I_V}_{\beta^\prime}^\beta$, then
	\begin{enumerate}
		\item $Q$ is invertible.
		\item $\forall \alpha \in V$, $\coordinate{\alpha}_\beta = Q \coordinate{\alpha}_{\beta^\prime} = \coordinate{I_V}_{\beta^\prime}^\beta \coordinate{\alpha}_{\beta^\prime}$.
	\end{enumerate}

	$Q= \coordinate{I_V}_{\beta^\prime}^\beta$ is called \cindex{change of coordinate matrix} that changes from $\beta^\prime$-coordinates to $\beta$-coordinates.
\end{theorem}

\begin{proof}
    $\forall \alpha \in V $,  $\coordinate{\alpha}_\beta = \coordinate{I_V (\alpha)}_\beta =  \coordinate{I_V}_{\beta^\prime}^\beta \coordinate{\alpha}_{\beta^\prime} = Q \coordinate{\alpha}_{\beta^\prime}$.
\end{proof}

If $Q$ changes $\beta^\prime$-coordinate into $\beta$-coordinate, $Q^{-1}$ changes $\beta$-coordinate into $\beta^\prime$-coordinate.

\begin{definition}
	A \cindex{linear operator} is a linear transformation that map from $V$ to itself.
\end{definition}

\begin{theorem}\label{twoindextransform}
	If $T$ is a linear operator on $V$, then
	\begin{equation}
		\coordinate{T}_{\beta^\prime} = \coordinate{I_V}_{\beta}^{\beta^\prime} \coordinate{T}_\beta \coordinate{I_V}_{\beta^\prime}^\beta= Q^{-1} \coordinate{T}_\beta Q
	\end{equation}
\end{theorem}

\begin{proof}
    $Q \coordinate{T}_{\beta^\prime} = [I]_{\beta^\prime}^\beta \coordinate{T}_{\beta^\prime}^{\beta^\prime} = [I T]_{\beta^\prime}^\beta = [T I]_{\beta^\prime}^\beta = \coordinate{T}_\beta^\beta [I]_{\beta^\prime}^\beta = \coordinate{T}_\beta Q$.
\end{proof}

\begin{theorem}
    Let $A \in M_{n \times n} (F)$, and $\gamma:\set{a_i}$ is an ordered basis for $F^n$. Then $\coordinate{L_A}_\gamma = Q^{-1} A Q$, where $Q = \coordinate{a_1, a_2, \dots, a_n}$.
\end{theorem}

\begin{proof}
    $\coordinate{L_A}_I = A$, so
    \begin{equation*}
        [L_A]_\gamma = \coordinate{I_V}_I^\gamma \times \coordinate{L_A}_I \times \coordinate{I_V}_\gamma^I = \coordinate{I_V}_I^\gamma \times A \times \coordinate{I_V}_\gamma^I
    \end{equation*}

    A take aways is that $Q$ is the change of coordinate matrix from $\gamma$ to $I$.
\end{proof}

\begin{theorem} \label{specialchangeofcoordinates}
	Let $T:V\rightarrow W$, $\beta$ and $\beta^\prime$ are ordered basis of $V$, $\gamma$ and $\gamma^\prime$ are ordered basis of $W$. Then
	\begin{equation}
		\coordinate{T}_{\beta^\prime}^{\gamma^\prime} = \coordinate{I_W}_\gamma^{\gamma^\prime} \coordinate{T}_\beta^\gamma \coordinate{I_V}_{\beta^\prime}^\beta
	\end{equation}
\end{theorem}

\begin{example}
There is an example of the usage of change of coordinate matrix: do reflection operation $T$ against a line $y = a x$. Let $\beta$ be the standard basis of $R^2$ and $\beta^\prime$ be the standard basis of $R^2$ after the rotation of $y = a x$. The operation $T$ has a matrix representation in $\beta^\prime$
	\begin{equation*}
		\coordinate{T}_{\beta^\prime} = \begin{bmatrix}
			1 & 0 \\
			0 & -1
		\end{bmatrix}
	\end{equation*}
	Then calculate $\coordinate{T}_\beta$ based on $\coordinate{T}_{\beta^\prime}$.
\end{example}

\begin{definition}
	$B$ is \cindex{similar} to $A$ if there is an invertible matrix $Q$ that $B = Q^{-1} A Q$.
\end{definition}

\begin{theorem}
If $T$ is a linear operator on finite dimension vector space $V$, and if $\beta$ and $\beta^\prime$ are any ordered basis of $V$, then $\coordinate{T}_{\beta^\prime}$ is similar to $\coordinate{T}_\beta$.
\end{theorem}

\subsection{Quotient Space}

\begin{definition}
    Let subspace $U \subset V$, The \cindex{affine subset}  $v + U$ of $V$ is defined as:
    \begin{equation}
        v + U = \{ v + u: u \in U\}
    \end{equation}
\end{definition}

\begin{definition}
    Let subspace $U \subset V$. Then the \cindex{quotient space} $V/U$ is defined as:
    \begin{equation}
        V/U = \{ v + U: v \in V \}
    \end{equation}
\end{definition}

\begin{definition}
    Let subspace $U \subset V$. The \cindex{quotient map} $\pi: V \rightarrow V/U$ is defined as:
    \begin{equation}
        \pi(v) = v + U
    \end{equation}
\end{definition}

\begin{theorem}
    \begin{equation}
        \dimension{V/U} = \dimension{V} - \dimension{U}
    \end{equation}
\end{theorem}

\begin{proof}
    Define $\pi : V \rightarrow V/U$. The null space is $U$.
\end{proof}

\begin{theorem}
    Define $\tilde{T}: V/\nullspace{T} \rightarrow W$ by:
    \begin{equation*}
        \tilde{T}\left(v + \nullspace{T} \right) = Tv
    \end{equation*}
    Then $\tilde{T}$ is an isomorphism between $V/\nullspace{T}$ and $T$.
\end{theorem}
\begin{proof}
    If $u + \nullspace{T} = v + \nullspace{T}$, then $u - v \in \nullspace{T}$. So $T(u - v) = T(u) - T(v) = 0$ and $T(u) = T(v)$.
\end{proof}

% dual space section
\subsection{Dual Space}

\begin{definition}
	A \cindex{linear functional} is a linear transformation that map from $V$ into $F$.
\end{definition}

\begin{definition}
	An $i$-th coordinate function $f_i$ with respect to basis $\beta$ is defined as $f_i(x) = a_i$ where
	\begin{equation*}
		\coordinate{x}_\beta = \begin{bmatrix}
			a_1 \\
			a_2 \\
			\vdots \\
			a_n
		\end{bmatrix} = \begin{bmatrix}
			f_1(a) \\
			f_2(a) \\
			\vdots \\
			f_n(a)
		\end{bmatrix}
	\end{equation*}
\end{definition}

\begin{definition}
	The \cindex{dual space}\label{dualspacedefinition} of $V$ is the vector space $V^* = \mathcal{L}(V,F)$. The \cindex{double dual space} $V^{**}$ is the dual space of $V^*$.
\end{definition}

The dimension of dual space is $\dimension{V^*} = \dimension{\mathcal{L}(V,F)} = \dimension{V} \times \dimension{F} = \dimension{V}$.

\begin{definition}
	Let $\beta = \set{x_i}$ be an ordered basis for finite dimensional vector space $V$. Define $f_i (x) = a_i$ where
	\begin{equation*}
	    \coordinate{x}_\beta = \begin{bmatrix}
	        a_1 \\
	        a_2 \\
	        \vdots \\
	        a_n
	    \end{bmatrix}
	\end{equation*}

	$f_i$ is the $i$-th coordinate function with respect to basis $\beta$. let $\beta^*=\set{f_i}$. Then $\beta^*$ is an ordered basis for $V^*$, and $\forall f \in V^*$, we have
	\begin{equation}
		f = \sum_{i=1}^n f(x_i) f_i
	\end{equation}
	$\beta^*$ is called the \cindex{dual basis} of $\beta$.
\end{definition}
\begin{proof}
	Let $g =\displaystyle \sum_{i=1}^n f(x_i) f_i$, we have
	\begin{equation*}
	g(x_j) = \left( \sum_{i=1}^n f(x_i) f_i \right) (x_j) = \sum_{i=1}^n f(x_i) f_i (x_j) = \sum_{i=1}^n f(x_i) \delta_{ij} =f(x_j)
	\end{equation*}
\end{proof}

\begin{theorem}
	Let $V$ and $W$ be vector space over $F$ with ordered basis $\beta$ and $\gamma$. For any linear transformation $T:V \rightarrow W$, the mapping $T^t: W^* \rightarrow V^*$ defined as $T^\top (g) = gT, \forall g \in W^*$ is a linear transformation with property that $\left[T^\top \right]_{\gamma^*}^{\beta^*} = \left(\left[T \right]_\beta^\gamma \right)^\top$.
\end{theorem}
\begin{proof}
	Let $\beta = \{x_i\}$ and $\gamma=\{y_i\}$ with dual basis $\beta^*=\{f_i\}$ and $\gamma^*=\{g_i\}$, $A=\coordinate{T}_\beta^\gamma$. we have
	\begin{equation*}
		T^\top (g_j) = g_j T = \sum_{s=1}^n (g_j T) (x_s) f_s
	\end{equation*}

	So the row $i$, column $j$ entry of $[T^\top]_{\gamma^*}^{\beta^*}$ is
	\begin{equation*}
	(g_j T)(x_i) = g_j (T(x_i))= g_j \left( \sum_{k=1}^m A_{kj} y_k \right) = \sum_{k=1}^m A_{kj} g_j(y_k)= \sum_{k=1}^m A_{kj} \delta_{kj} = A_{ji}
	\end{equation*}

	Hence $\left[T^\top \right]_{\gamma^*}^{\beta^*} = A^\top $.
\end{proof}

\begin{definition}
    For $U \subset V$, the \cindex{annihilator} of $U$, denoted as $U^0_V$, is defined as
    \begin{equation*}
        U^0_V = \{ \phi \in V^*: \phi(u) = 0, \forall u \in U \}
    \end{equation*}
    So the annihilator map $U$ to $0$. For vectors in $V - U$, the mapping could be any result. The annihilator is a subspace.
\end{definition}

\begin{theorem}
    \begin{equation}
        \dimension{U} + \dimension{U_V^0} = \dimension{V}
    \end{equation}
\end{theorem}

\begin{proof}
    Define $i \in \mathcal{L}(U,V)$ that $i(u) = u, \forall u \in U$. $i^* \in \mathcal{L}(V^*,U^*)$. So
    \begin{equation*}
        \dimension{\rangespace{i^*} } +\dimension{\nullspace{i^*}} = \dimension{V^*}
    \end{equation*}
    By definition, $\nullspace{i^*} = U^0_V$. Also $\rangespace{i^*} = U^*$.
\end{proof}

\begin{theorem}
    Let $V$ and $W$ be two finite-dimentional vector space, and $T \in \mathcal{L}(V,W)$. Then:
    \begin{enumerate}
        \item $\nullspace{T^*}  = (\rangespace{T} )^0$
        \item $\rangespace{T^*} = (\nullspace{T} )^0$
        \item $\dimension{\rangespace{T^*}} = \dimension{\text{range } T}$
        \item $\dimension{\nullspace{T^*}} = \dimension{\nullspace{T}} + \dimension{W} - \dimension{V}$
    \end{enumerate}
\end{theorem}

\begin{proof}
    Suppose $\varphi \in \text{null } T^*$. Then $ 0 = T^*(\varphi) = \varphi T$. Then
    \begin{equation*}
        0 = (\varphi T)(v) = \varphi (Tv)
    \end{equation*}
    So $\varphi \in (\text{range } T)^0_W$.

    \begin{equation*}
        \begin{aligned}
            \dimension{\rangespace{T^*}} &= \dimension{W^*} - \dimension{\nullspace{T^*}} \\
            &= \dimension{W} - \dimension{\rangespace{T}^0} \\
            &= \dimension{\rangespace{T}}
        \end{aligned}
    \end{equation*}

    \begin{equation*}
        \begin{aligned}
            \dimension{\nullspace{T^*}} &= \dimension{\rangespace{T}^0}\\
            &= \dimension{W} - \dimension{\rangespace{T}} \\
            &= \dimension{W} - (\dimension{V} - \dimension{\nullspace{T}} ) \\
            &= \dimension{W} + \dimension{\nullspace{T}} - \dimension{V}
        \end{aligned}
    \end{equation*}
\end{proof}

\begin{definition}
    For vector $x \in V$, define $\hat{x}: V^* \rightarrow F $ by $\hat{x}(f) = f(x)$. $\hat{x}$ is a linear functional on $V^*$, so $\hat{x} \in V^{**}$.
\end{definition}

\begin{theorem}
    Define $\psi : V \rightarrow V^{**}$ by $\psi (x) = \hat{X}$.  Then $\psi$ is an isomorphism.
\end{theorem}

\begin{theorem}
    Let $V$ be a finite dimension vector space with dual space $V^*$. Every ordered basis for $V^*$ is the dual basis for some basis for $V$.
\end{theorem}

\begin{center}
    \begin{tikzcd}
V \arrow[rrddd, "V^*"{name=U}] \arrow[rrrr, "T"] && & & W \arrow[llddd, "W^*"'{name=W}] \\
\\
\\
&& F
\arrow[rightarrow, from=W, to=U, "T^\top"]
\end{tikzcd}
\end{center}


\section{Application: Least squares approximations and curve fitting}
\label{sec:least-squares}

\begin{outcome}
  \begin{enumerate}
  \item Find least squares approximations for a system of equations.
  \item Find best fit lines and parabolas for a set of data points.
  \end{enumerate}
\end{outcome}

In this section, we will consider the problem of finding approximate
solutions to a system of linear equations $A\vect{v}=\vect{b}$. This
can be useful when the system is inconsistent, but we would still like
to find a ``best'' answer. For example, consider the following system
of equations.
\begin{equation*}
  \begin{array}{rcrcl}
    0.1 x &+& 0.2 y &=& 0.3, \\
    0.2 x &+& 0.5 y &=& 0.7, \\
    0.7 x &+& 0.2 y &=& 0.9. \\
  \end{array}
\end{equation*}
This system has the solution $(x,y)=(1,1)$, so it is clearly
consistent. Now imagine that we introduce some small inaccuracies into
the equations. The inaccuracies might perhaps be due to round-off
errors, or due to measurement errors if the coefficients are obtained
from experimental data. We might end up with the following system of
equations:
\begin{equation*}
  \begin{array}{rcrcl}
    0.1000001 x &+& 0.2 y &=& 0.3, \\
    0.2 x &+& 0.5 y &=& 0.7, \\
    0.7 x &+& 0.2 y &=& 0.9. \\
  \end{array}
\end{equation*}
Except for a tiny error in one of the coefficients, this is the same
system of equations as before. We would expect that such a small error
does not affect the result much. However, this last system of
equations is inconsistent; it has no solutions at all. You can see
this by observing that $(x,y)=(1,1)$ is still the unique solution to
the last two equations; substituting this into the first equation, we
get $0.3000001 = 0.3$, which almost, but not exactly, true. What we
would like to find in a situation like this is an ``approximate
solution'', i.e., numbers $x$ and $y$ such that each of the three
equations ``almost'' holds. We can formulate the problem more
precisely as follows:

\begin{problem}{Least squares approximation problem}{least squares-approximation}
  Given a (possibly inconsistent) system of equations
  $A\vect{v}=\vect{b}$, find $\vect{v}$ such that
  \begin{equation*}
    \norm{A\vect{v} - \vect{b}}
  \end{equation*}
  is as small as possible. We call such a vector $\vect{v}$ a
  \textbf{least squares approximation}%
  \index{least squares approximation}%
  \index{approximation!least squares} for the system of equations.
\end{problem}

To see why this is called a ``least squares'' approximation, consider
a system of equations
\begin{equation*}
  \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
    a_{11}x_1 &+& \cdots &+& a_{1n}x_n &=& b_1, \\
    a_{21}x_1 &+& \cdots &+& a_{2n}x_n &=& b_2, \\
    \multicolumn{7}{c}{\cdots} \\
    a_{m1}x_1 &+& \cdots &+& a_{mn}x_n &=& b_m. \\
  \end{array}
\end{equation*}
We can write this in matrix form $A\vect{v}=\vect{b}$, where
\begin{equation*}
  A = \begin{mymatrix}{ccc}
    a_{11} & \cdots & a_{1n} \\
    \vdots & \ddots & \vdots \\
    a_{m1} & \cdots & a_{mn} \\
  \end{mymatrix},
  \quad
  \vect{b} = \begin{mymatrix}{c} b_1 \\ \vdots \\ b_m \end{mymatrix},
  \quad\mbox{and}\quad
  \vect{v} = \begin{mymatrix}{c} x_1 \\ \vdots \\ x_n \end{mymatrix}.
\end{equation*}
Then
\begin{equation*}
  A\vect{v} - \vect{b} ~=~
  \begin{mymatrix}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
    a_{11}x_1 &+& \cdots &+& a_{1n}x_n &-& b_1 \\
    \multicolumn{7}{c}{\cdots} \\
    a_{m1}x_1 &+& \cdots &+& a_{mn}x_n &-& b_m \\
  \end{mymatrix},
\end{equation*}
and therefore
\begin{equation*}
  \norm{A\vect{v} - \vect{b}}^2 ~=~
  (a_{11}x_1 + \cdots + a_{1n}x_n - b_1)^2 + \ldots
  + (a_{m1}x_1 + \cdots + a_{mn}x_n - b_m)^2.
\end{equation*}
Therefore, minimizing $\norm{A\vect{v}-\vect{b}}$ is the same as
minimizing the sum of the squares of the errors of all the equations,
where the error of each equation is defined to be the difference
between its left-hand side and right-hand side.

We note that $\norm{A\vect{v}-\vect{b}}=0$ if and only if
$A\vect{v}=\vect{b}$. Therefore, if the system of equations
$A\vect{v}=\vect{b}$ is consistent, then its least squares
approximations are exactly the solutions of the system of equations in
the usual sense.

The least squares approximation problem has a very elegant solution,
provided by the following proposition.

\begin{proposition}{Solution of the least squares approximation problem}{least-squares-approximation}
  A vector $\vect{v}$ is a least squares approximation of the system
  of equations $A\vect{v}=\vect{b}$ if and only if
  \begin{equation*}
    A^TA\vect{v} = A^T\vect{b}.
  \end{equation*}
\end{proposition}

\begin{proof}
  Let $\vect{a}_1,\ldots\vect{a}_n$ be the columns of the matrix
  $A$. Recall that $\sspan\set{\vect{a}_1,\ldots\vect{a}_n}$ is called
  the \textbf{column space}%
  \index{column space}%
  \index{matrix!column space} of $A$, which we write as $\col(A)$.  If
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{c} x_1 \\ \vdots \\ x_n \end{mymatrix},
  \end{equation*}
  is any vector, then by the definition of matrix multiplication, we
  have
  \begin{equation*}
    A\vect{v} = x_1\vect{a}_1 + \ldots + x_n\vect{a}_n.
  \end{equation*}
  Therefore, a vector is of the form $A\vect{v}$ if and only if it is
  an element of $\col(A)$. In particular, the equation
  $A\vect{v}=\vect{b}$ has a solution if and only if $\vect{b}$ is an
  element of the column space of $A$. For $\vect{v}$ to be a least
  squares approximation, we want $\norm{A\vect{v}-\vect{b}}$ to be as
  small as possible. This means that we are looking for the element of
  $\col(A)$ that is closest to $\vect{b}$.
  \begin{center}
    \begin{tikzpicture}[x={(1cm,-0.1cm)},y={(0.4cm,0.5cm)},z={(0cm,1cm)},scale=0.9]
      \filldraw[draw=red!80,fill=red!10](-4,-3,0) -- (4,-3,0) -- (4,3,0) -- (-4,3,0) -- cycle;
      \path[red] (2,0,0) node[right] {$\col(A)$};
      \draw[->,thick,blue!80!black](0,0,0) -- node[left, pos=0.6] {$\vect{b}$} (2,1.5,2.5);
      \draw[->,thick,blue!80!black](0,0,0) -- node[above, pos=0.6] {$A\vect{v}$} (2,1.5,0);
      \draw[->,thick,blue!80!black](2,1.5,2.5) -- node[right] {$A\vect{v}-\vect{b}$} (2,1.5,0);
      \fill (0,0,0) circle [radius=2.2pt] node [left=3pt] {$\vect{0}$};
      \fill (2,1.5,2.5) circle [radius=2.2pt];
      \fill (2,1.5,0) circle [radius=2.2pt];
    \end{tikzpicture}
  \end{center}
  From Proposition~\ref{prop:projection-subspace}, we know that this
  happens when $A\vect{v}-\vect{b}$ is orthogonal to $\col(A)$. Since
  $\col(A)=\sspan\set{\vect{a}_1,\ldots\vect{a}_n}$, this is
  equivalent to saying that $A\vect{v}-\vect{b}$ is orthogonal to each of
  the vectors $\vect{a}_1,\ldots\vect{a}_n$. Therefore,
  \begin{equation*}
    \vect{a}_i^T(A\vect{v}-\vect{b})=0
  \end{equation*}
  for $i=1,\ldots,n$. Since $\vect{a}_1^T,\ldots\vect{a}_n^T$ are the
  rows of the matrix $A^T$, this system of $n$ equations is equivalent
  to the single equation
  \begin{equation*}
    A^T(A\vect{v}-\vect{b})=\vect{0},
  \end{equation*}
  or equivalently,
  \begin{equation*}
    A^TA\vect{v} = A^T\vect{b}.
  \end{equation*}
\end{proof}

\begin{example}{Least squares approximation}{least-squares-approximation}
  Find the least squares approximation for the system of equations
  \begin{equation*}
    \begin{array}{r@{~}c@{~}r@{~}c@{~}r@{~}c@{~}r}
      2x &+& 2y &+& 2z &=&  1, \\
      x  &-&  y &-&  z &=& -2, \\
      -x &-&  y &+& 2z &=&  4, \\
      2x &+& 2y &-&  z &=& -8. \\
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  We first write the system in matrix form $A\vect{v}=\vect{b}$, where
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      2 & 2 & 2 \\
      1 & -1 & -1 \\
      -1 & -1 & 2 \\
      2 & 2 & -1 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{b} = \begin{mymatrix}{r} 1 \\ -2 \\ 4 \\ -8 \end{mymatrix}.
  \end{equation*}
  By Proposition~\ref{prop:least-squares-approximation}, the least
  squares approximation is given by the solution of the system of
  equations $A^TA\vect{v} = A^T\vect{b}$. We calculate:
  \begin{equation*}
    A^TA =
    \begin{mymatrix}{rrrr}
      2 & 1 & -1 & 2 \\
      2 & -1 & -1 & 2 \\
      2 & -1 & 2 & -1 \\
    \end{mymatrix}
    \begin{mymatrix}{rrr}
      2 & 2 & 2 \\
      1 & -1 & -1 \\
      -1 & -1 & 2 \\
      2 & 2 & -1 \\
    \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      10 & 8 & -1 \\
      8  & 10 & 1 \\
      -1 & 1 & 10 \\
    \end{mymatrix},
  \end{equation*}
  \begin{equation*}
    A^T\vect{b} =
    \begin{mymatrix}{rrrr}
      2 & 1 & -1 & 2 \\
      2 & -1 & -1 & 2 \\
      2 & -1 & 2 & -1 \\
    \end{mymatrix}
    \begin{mymatrix}{r} 1 \\ -2 \\ 4 \\ -8 \end{mymatrix}
    =
    \begin{mymatrix}{r} -20 \\ -16 \\ 20  \end{mymatrix}.
  \end{equation*}
  Therefore, we must solve the system of equations
  \begin{equation*}
    \begin{mymatrix}{rrr}
      10 & 8 & -1 \\
      8  & 10 & 1 \\
      -1 & 1 & 10 \\
    \end{mymatrix}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    =
    \begin{mymatrix}{r} -20 \\ -16 \\ 20  \end{mymatrix}.
  \end{equation*}
  After some row operations, we find that the unique solution is
  $(x,y,z) = (-1, -1, 2)$.  We can double-check this answer as
  follows. We calculate
  \begin{equation*}
    A\vect{v} - \vect{b}
    ~=~
    \begin{mymatrix}{rrr}
      2 & 2 & 2 \\
      1 & -1 & -1 \\
      -1 & -1 & 2 \\
      2 & 2 & -1 \\
    \end{mymatrix}
    \begin{mymatrix}{r} -1 \\ -1 \\ 2 \end{mymatrix}
    - \begin{mymatrix}{r} 1 \\ -2 \\ 4 \\ -8 \end{mymatrix}
    ~=~
    \begin{mymatrix}{r} 0 \\ -2 \\ 6 \\ -6 \end{mymatrix}
    - \begin{mymatrix}{r} 1 \\ -2 \\ 4 \\ -8 \end{mymatrix}
    ~=~ \begin{mymatrix}{r} -1 \\ 0 \\ 2 \\ 2 \end{mymatrix}
  \end{equation*}
  and check that this vector is orthogonal to every column of
  $A$. Since this is the case, our answer is correct.
\end{solution}

An important application of least squares approximations is
\textbf{curve fitting}%
\index{curve fitting}%
\index{fitting!curve fitting}: finding the ``best'' function of a given type
(for example, linear, quadratic) to fit a given series of data points.
The next two examples show how to use least square approximations to
solve curve fitting problems.

\begin{example}{Least squares line}{least-squares-line}
  A company has collected daily data on temperature and ice cream
  sales over a period of one week. The data is as follows:
  \begin{equation*}
    \begin{array}{c@{~~~}|c|@{~~~}c}
      \mbox{Date} & \mbox{Peak temperature} & \mbox{Sales} \\\hline
      \mbox{July 1}  & 17\degC & \$320  \\   % 300 1 -5   1
      \mbox{July 2}  & 22\degC & \$570  \\   % 600 1  0  -3 -1  1  1
      \mbox{July 3}  & 26\degC & \$850  \\   % 840 1  4   1 -1 -1
      \mbox{July 4}  & 20\degC & \$470  \\   % 480 1 -2      0 -1
      \mbox{July 5}  & 24\degC & \$750  \\   % 720 1  2      2  1  1
      \mbox{July 6}  & 23\degC & \$620  \\   % 660 1  1   1       -2  1
      \mbox{July 7}  & 23\degC & \$680  \\   % 660 1  1              -1
    \end{array}
  \end{equation*}
  The company is interested in predicting how temperature will affect
  future sales. Find a function of the form $y=a+bx$ that best fits
  the data, where $x$ is temperature in degrees Celsius and $y$ is
  sales in dollars. By ``best fit'', we mean that the sum of the
  square of the errors should be as small as possible (where each
  error is the difference between the dollar amount predicted by the
  formula $y=a+bx$ and the actual dollar amount).  Such a function is
  called a \textbf{least squares line}%
  \index{least squares line}%
  \index{curve fitting!least squares line}%
  \index{line fitting}%
  \index{fitting!line fitting} or a \textbf{linear regression}%
  \index{linear regression}%
  \index{curve fitting!linear regression} for the data.
\end{example}

\begin{solution}
  We first write down a system of equations that expresses the
  relationship $y=a+bx$ for each of the seven data points $(x,y)$:
  \begin{equation*}
    \begin{array}{r@{~}c@{~}r@{~}c@{~}r}
      a &+& 17b &=& 320, \\
      a &+& 22b &=& 570, \\
      a &+& 26b &=& 850, \\
      a &+& 20b &=& 470, \\
      a &+& 24b &=& 750, \\
      a &+& 23b &=& 620, \\
      a &+& 23b &=& 680. \\
    \end{array}
  \end{equation*}
  This is a system of seven equations in two variables (the variables
  are $a$ and $b$). This system is likely inconsistent, because there
  are more equations than variables, and also because it is unlikely
  that the relationship between temperature and sales is exactly (as
  opposed to approximately) linear. Instead, we find the least squares
  approximation for the system of equations. We let
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      1 & 17 \\
      1 & 22 \\
      1 & 26 \\
      1 & 20 \\
      1 & 24 \\
      1 & 23 \\
      1 & 23 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{b} =
    \begin{mymatrix}{r}
      320 \\
      570 \\
      850 \\
      470 \\
      750 \\
      620 \\
      680 \\
    \end{mymatrix}.
  \end{equation*}
  By Proposition~\ref{prop:least-squares-approximation}, we must solve
  the equation $A^TA\vect{v} = A^T\vect{b}$. We calculate
  \begin{equation*}
    A^TA = \begin{mymatrix}{cc}
      7 & 155 \\
      155 & 3483
    \end{mymatrix}
    \quad\mbox{and}\quad
    A^T\vect{b} =
    \begin{mymatrix}{c}
      4260 \\
      97380
    \end{mymatrix}.
  \end{equation*}
  So the system of equations we must solve is
  \begin{equation*}
    \begin{mymatrix}{cc}
      7 & 155 \\
      155 & 3483
    \end{mymatrix}
    \begin{mymatrix}{c} a \\ b \end{mymatrix}
    =
    \begin{mymatrix}{c}
      4260 \\
      97380
    \end{mymatrix}.
  \end{equation*}
  After some row operations, we find the unique solutions
  $(a,b)=(-720,60)$. This means that the desired linear approximation
  is $y=-720 + 60x$. The following plot shows this function along with
  the original data points.
  \begin{center}
    \begin{tikzpicture}[xscale=0.7,yscale=0.7]
      \draw[->] (15,2) -- (28,2) node[right] {$x$ (temperature)};
      \draw[->] (15,2) -- (15,10) node[above] {$y$\makebox[0in][l]{ (sales)}};
      \draw (15,2) -- +(0,-0.5) node[below] {15};
      \draw (20,2) -- +(0,-0.5) node[below] {20};
      \draw (25,2) -- +(0,-0.5) node[below] {25};
      \draw (15,2)  -- +(-0.5,0) node[left] {200};
      \draw (15,4)  -- +(-0.5,0) node[left] {400};
      \draw (15,6)  -- +(-0.5,0) node[left] {600};
      \draw (15,8)  -- +(-0.5,0) node[left] {800};
      \draw (15,10) -- +(-0.5,0) node[left] {1000};
      \fill (17,3.2) circle [radius=3.1pt];
      \fill (22,5.7) circle [radius=3.1pt];
      \fill (26,8.5) circle [radius=3.1pt];
      \fill (20,4.7) circle [radius=3.1pt];
      \fill (24,7.5) circle [radius=3.1pt];
      \fill (23,6.2) circle [radius=3.1pt];
      \fill (23,6.8) circle [radius=3.1pt];
      \draw[red,thick] (15.5,2.1) -- (27.5,9.3);
    \end{tikzpicture}
  \end{center}
  The following table compares the observed data to the computed
  linear regression (sorted by increasing temperature). It also shows
  the error for each data point.
  \begin{equation*}
    \begin{array}{c@{~~~}|c|@{~~~}c|@{~~~}c}
      \mbox{Temperature} & \mbox{Actual sales} & \mbox{Best linear fit} & \mbox{Error}\\\hline
      17 & 320 & 300 & -20 \\
      20 & 470 & 480 & +10 \\
      22 & 570 & 600 & +30 \\
      23 & 620 & 660 & +40 \\
      23 & 680 & 660 & -20 \\
      24 & 750 & 720 & -30 \\
      26 & 850 & 840 & -10 \\
    \end{array}
  \end{equation*}
  The sum of the squares of the errors is $20^2 + 10^2 + 30^2 + 40^2 + 20^2 + 30^2 + 10^2 = 4400$.
\end{solution}

\begin{example}{Least squares parabola}{least-squares-parabola}
  \index{least squares parabola}%
  \index{curve fitting!least squares parabola}
  Find a quadratic polynomial that is the best fit for the following
  data points:
  \begin{equation*}
    \begin{array}{rcr@{~}r}
      (x_1,y_1) &=& ( 3,& 23.5), \\
      (x_2,y_2) &=& ( 4,& 13.5), \\
      (x_3,y_3) &=& ( 5,& 12.5), \\
      (x_4,y_4) &=& ( 6,&  5.5), \\
      (x_5,y_5) &=& ( 7,&  9.0), \\
      (x_6,y_6) &=& ( 8,&  8.0), \\
      (x_7,y_7) &=& ( 9,& 19.0). \\
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  We are looking for an equation of the form
  $y=a+bx+cx^2$. Substituting each of the seven data points into the
  equation, we obtain 7 equations in the unknowns $a$, $b$, and $c$:
  \begin{equation*}
    \begin{array}{r@{~}c@{~}r@{~}c@{~}rcr}
      a &+& 3b &+&  9c &=& 23.5, \\
      a &+& 4b &+& 16c &=& 13.5, \\
      a &+& 5b &+& 25c &=& 12.5, \\
      a &+& 6b &+& 36c &=&  5.5, \\
      a &+& 7b &+& 49c &=&  9.0, \\
      a &+& 8b &+& 64c &=&  8.0, \\
      a &+& 9b &+& 81c &=& 19.0. \\
    \end{array}
  \end{equation*}
  We write this in matrix form at $A\vect{v} = \vect{b}$, where
  \begin{equation*}
    A =
    \begin{mymatrix}{rrr}
      1 & 3 &  9 \\
      1 & 4 & 16 \\
      1 & 5 & 25 \\
      1 & 6 & 36 \\
      1 & 7 & 49 \\
      1 & 8 & 64 \\
      1 & 9 & 81 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{b} =
    \begin{mymatrix}{r}
      23.5 \\
      13.5 \\
      12.5 \\
      5.5 \\
      9.0 \\
      8.0 \\
      19.0 \\
    \end{mymatrix}.
  \end{equation*}
  To find the least squares approximation, we calculate
  \begin{equation*}
    A^TA = \begin{mymatrix}{ccc}
      7 & 42 & 280 \\
      42 & 280 & 2016 \\
      280 & 2016 & 15316 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    A^T\vect{b} =
    \begin{mymatrix}{c}
      91 \\
      518 \\
      3430 \\
    \end{mymatrix}
  \end{equation*}
  and solve the system of equations $A^TA\vect{v} = A^T\vect{b}$, i.e.,
  \begin{equation*}
    \begin{mymatrix}{ccc}
      7 & 42 & 280 \\
      42 & 280 & 2016 \\
      280 & 2016 & 15316 \\
    \end{mymatrix}
    \begin{mymatrix}{c} a \\ b \\ c \end{mymatrix}
    ~=~
    \begin{mymatrix}{c}
      91 \\
      518 \\
      3430 \\
    \end{mymatrix}.
  \end{equation*}
  After doing some row operations, we find that the unique solution is
  $(a,b,c) = (67,-19,1.5)$. Therefore, the desired quadratic
  approximation is $y = 67 - 19x + 1.5x^2$. The following plot shows
  this function along with the original data points.
  \begin{center}
    \def\X{*2.0}
    \def\Y{*0.4}
    \begin{tikzpicture}[scale=0.5, domain=2.67:10, samples=50]
      \draw[->] (0\X,0\Y) -- (12\X,0\Y) node[right] {$x$};
      \draw[->] (0\X,0\Y) -- (0\X,27\Y) node[above] {$y$};
      \draw (0\X,0\Y) -- +(0,-0.5) node[below] {0};
      \draw (5\X,0\Y) -- +(0,-0.5) node[below] {5};
      \draw (10\X,0\Y) -- +(0,-0.5) node[below] {10};
      \draw (0\X,0\Y)  -- +(-0.5,0) node[left] {0};
      \draw (0\X,5\Y)  -- +(-0.5,0) node[left] {5};
      \draw (0\X,10\Y)  -- +(-0.5,0) node[left] {10};
      \draw (0\X,15\Y)  -- +(-0.5,0) node[left] {15};
      \draw (0\X,20\Y) -- +(-0.5,0) node[left] {20};
      \draw (0\X,25\Y) -- +(-0.5,0) node[left] {25};
      \fill ( 3\X, 23.5\Y) circle [radius=4.4pt];
      \fill ( 4\X, 13.5\Y) circle [radius=4.4pt];
      \fill ( 5\X, 12.5\Y) circle [radius=4.4pt];
      \fill ( 6\X,  5.5\Y) circle [radius=4.4pt];
      \fill ( 7\X,  9.0\Y) circle [radius=4.4pt];
      \fill ( 8\X,  8.0\Y) circle [radius=4.4pt];
      \fill ( 9\X, 19.0\Y) circle [radius=4.4pt];
      \draw[red,thick] plot (\x\X,{(67-19*\x+1.5*\x*\x)\Y});
    \end{tikzpicture}
  \end{center}
  The following table shows the original data, the values of the
  best fit parabola, and the error for each data point.
  \begin{equation*}
    \begin{array}{c@{~~~}|c|@{~~~}c|@{~~~}c}
      x & \mbox{Actual $y$} & \mbox{Best fit parabola} & \mbox{Error} \\\hline
      3 &  23.5 &  23.5 & ~~~0.0 \\
      4 &  13.5 &  15.0 &   +1.5 \\
      5 &  12.5 & ~~9.5 &   -3.0 \\
      6 & ~~5.5 & ~~7.0 &   +1.5 \\
      7 & ~~9.0 & ~~7.5 &   -1.5 \\
      8 & ~~8.0 &  11.0 &   +3.0 \\
      9 &  19.0 &  17.5 &   -1.5 \\
    \end{array}
  \end{equation*}
  The sum of the squares of the errors is $0^2 + 1.5^2 + 3^2 + 1.5^2 + 1.5^2 + 3^2 + 1.5^2 = 27$.
\end{solution}


\section{Gradient derivatives}
\label{sec:appendix:gradient-derivatives}

\subsection{Weight matrix construction}
\label{sec:appendix:gradient-derivatives:weight-matrix-construction}
For clarity the weight matrix construction is defined using scalar notation
\begin{equation}
W_{h_\ell, h_{\ell-1}} = \tanh(\hat{W}_{h_\ell, h_{\ell-1}}) \sigma(\hat{M}_{h_\ell, h_{\ell-1}})
\end{equation}

The of the loss with respect to $\hat{W}_{h_\ell, h_{\ell-1}}$ and $\hat{M}_{h_\ell, h_{\ell-1}}$ is then derived using backpropagation.
\begin{equation}
\begin{aligned}
\frac{\partial\mathcal{L}}{\partial \hat{W}_{h_\ell, h_{\ell-1}}} &= \frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}} \frac{\partial W_{h_\ell, h_{\ell-1}}}{\partial \hat{W}_{h_\ell, h_{\ell-1}}} \\
&= \frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}} (1 - \tanh^2(\hat{W}_{h_\ell, h_{\ell-1}})) \sigma(\hat{M}_{h_\ell, h_{\ell-1}}) \\
\frac{\partial\mathcal{L}}{\partial \hat{M}_{h_\ell, h_{\ell-1}}} &= \frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}} \frac{\partial W_{h_\ell, h_{\ell-1}}}{\partial \hat{M}_{h_\ell, h_{\ell-1}}} \\
&= \frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}} \tanh(\hat{W}_{h_\ell, h_{\ell-1}}) \sigma(\hat{M}_{h_\ell, h_{\ell-1}}) (1 - \sigma(\hat{M}_{h_\ell, h_{\ell-1}}))
\end{aligned}
\end{equation}

As seen from this result, one only needs to consider $\frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}}$ for $\mathrm{NAC}_{+}$ and $\mathrm{NAC}_{\bullet}$, as the gradient with respect to $\hat{W}_{h_\ell, h_{\ell-1}}$ and $\hat{M}_{h_\ell, h_{\ell-1}}$ is a multiplication on $\frac{\partial\mathcal{L}}{\partial W_{h_\ell, h_{\ell-1}}}$.

\subsection{Gradient of \texorpdfstring{$\mathrm{NAC}_{\bullet}$}{NAC-mul}}
\label{sec:appendix:gradient-derivatives:gradient-nac-mul}

The $\textrm{NAC}_\bullet$ is defined using scalar notation.
\begin{equation}
z_{h_\ell} = \exp\left(\sum_{h_{\ell-1}=1}^{H_{\ell-1}} W_{h_{\ell}, h_{\ell-1}} \log(|z_{h_{\ell-1}}| + \epsilon) \right)
\end{equation}

The gradient of the loss with respect to $W_{h_\ell, h_{\ell-1}}$ can the be derived using backpropagation.
\begin{equation}
\begin{aligned}
\frac{\partial z_{h_\ell}}{\partial W_{h_\ell, h_{\ell-1}}} &= \exp\left(\sum_{h'_{\ell-1}=1}^{H_{\ell-1}} W_{h_{\ell}, h'_{\ell-1}} \log(|z_{h'_{\ell-1}}| + \epsilon) \right) \log(|z_{h_{\ell-1}}| + \epsilon) \\
&= z_{h_\ell} \log(|z_{h_{\ell-1}}| + \epsilon)
\end{aligned}
\end{equation}

We now wish to derive the backpropagation term $\delta_{h_\ell} = \frac{\partial \mathcal{L}}{\partial z_{h_\ell}}$, because $z_{h_\ell}$ affects $\{z_{h_{\ell+1}}\}_{h_{\ell+1}=1}^{H_{\ell+1}}$ this becomes:
\begin{equation}
\delta_{h_\ell} = \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} = \sum_{h_{\ell+1}=1}^{H_{\ell+1}} \frac{\partial \mathcal{L}}{\partial z_{h_{\ell+1}}} \frac{\partial z_{h_{\ell+1}}}{\partial z_{h_\ell}} = \sum_{h_{\ell+1}=1}^{H_{\ell+1}} \delta_{h_{\ell+1}} \frac{\partial z_{h_{\ell+1}}}{\partial z_{h_\ell}}
\end{equation}

To make it easier to derive $\frac{\partial z_{h_{\ell+1}}}{\partial z_{h_\ell}}$ we re-express the $z_{h_\ell}$ as $z_{h_{\ell+1}}$.
\begin{equation}
z_{h_{\ell+1}} = \exp\left(\sum_{h_{\ell}=1}^{H_{\ell}} W_{h_{\ell+1}, h_{\ell}} \log(|z_{h_{\ell}}| + \epsilon) \right)
\end{equation}

The gradient of $\frac{\partial z_{h_{\ell+1}}}{\partial z_{h_\ell}}$ is then:
\begin{equation}
\begin{aligned}
\frac{\partial z_{h_{\ell+1}}}{\partial z_{h_\ell}} &= \exp\left(\sum_{h_{\ell}=1}^{H_{\ell}} W_{h_{\ell+1}, h_{\ell}} \log(|z_{h_{\ell}}| + \epsilon) \right) W_{h_{\ell+1}, h_{\ell}} \frac{\partial \log(|z_{h_{\ell}}| + \epsilon)}{\partial z_{h_\ell}} \\
&= \exp\left(\sum_{h_{\ell}=1}^{H_{\ell}} W_{h_{\ell+1}, h_{\ell}} \log(|z_{h_{\ell}}| + \epsilon) \right) W_{h_{\ell+1}, h_{\ell}} \frac{\mathrm{abs}'(z_{h_{\ell}})}{|z_{h_{\ell}}| + \epsilon} \\
&= z_{h_{\ell+1}} W_{h_{\ell+1}, h_{\ell}} \frac{\mathrm{abs}'(z_{h_{\ell}})}{|z_{h_{\ell}}| + \epsilon}
\end{aligned}
\end{equation}

$\mathrm{abs}'(z_{h_{\ell}})$ is the gradient of the absolute function. In the paper we denote this as $\mathrm{sign}(z_{h_{\ell}})$ for brevity. However, depending on the exact definition used there may be a difference for $z_{h_{\ell}} = 0$, as $\mathrm{abs}'(0)$ is undefined. In practicality this doesn't matter much though, although theoretically it does mean that the expectation of this is theoretically undefined when $E[z_{h_{\ell}}] = 0$.

\subsection{Gradient of NMU}
\label{sec:appendix:gradient-derivatives:gradient-nmu}

In scalar notation the NMU is defined as:
\begin{equation}
z_{h_\ell} = \prod_{h_{\ell-1}=1}^{H_{\ell-1}} \left(W_{h_{\ell-1},h_\ell} z_{h_{\ell-1}} + 1 - W_{h_{\ell-1},h_\ell} \right)
\end{equation}

The gradient of the loss with respect to $W_{h_{\ell-1},h_\ell}$ is fairly trivial. Note that every term but the one for $h_{\ell-1}$, is just a constant with respect to $W_{h_{\ell-1},h_\ell}$. The product, except the term for $h_{\ell-1}$ can be expressed as $\frac{z_{h_\ell}}{W_{h_{\ell-1},h_\ell} z_{h_{\ell-1}} + 1 - W_{h_{\ell-1},h_\ell}}$. Using this fact, the gradient can be expressed as:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial w_{h_{\ell}, h_{\ell - 1}}} = \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} \frac{\partial z_{h_\ell}}{\partial w_{h_{\ell}, h_{\ell - 1}}} = \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} \frac{z_{h_\ell}}{W_{h_{\ell-1},h_\ell} z_{h_{\ell-1}} + 1 - W_{h_{\ell-1},h_\ell}} \left(z_{h_{\ell-1}} - 1\right)
\end{equation}

Similarly, the gradient $\frac{\partial \mathcal{L}}{\partial z_{h_\ell}}$ which is essential in backpropagation can equally easily be derived as:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial z_{h_{\ell-1}}} = \sum_{h_\ell = 1}^{H_\ell} \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} \frac{\partial z_{h_\ell}}{\partial z_{h_{\ell-1}}} = \sum_{h_\ell = 1}^{H_\ell} \frac{z_{h_\ell}}{W_{h_{\ell-1},h_\ell} z_{h_{\ell-1}} + 1 - W_{h_{\ell-1},h_\ell}} W_{h_{\ell-1},h_\ell}
\end{equation}


\section{Non Relativistic Theory}

    Using the non relativistic Schr\"odinger equation, the ground state
    wave-function for a hydrogenic atom is given by
    equation~\ref{eq:schrodinger-ground}.
    The corresponding electron density for the ground state is given by
    \(
        \rho(\mb{r}) = \psi_{1s}^{*} \psi_{1s}
                     = \frac{1}{\pi a_{0}^{3}} e^{-Zr/a_0}
    \).
    We know that for a hydrogenic atom we can assume spherical symmetry,
    and therefore by substituting the charge density into
    equation~\ref{eq:nff-spherical} we get the non relativistic normal
    form factor for hydrogenic atoms.
    \begin{equation*}
        f_0(q) = \frac{4}{q} \left( \frac{Z}{a_0} \right)^3
                 \int_0^\infty e^{-2Zr/a_0} \sin(qr) r \; dr
    \end{equation*}
    \begin{equation} \label{eq:nff-nonrelativistic}
        \boxed{
            f_0(q) = \left( \frac{2Z}{a_0} \right)^4
                     \left[ \left( \frac{2Z}{a_0} \right)^2 + q^2 \right]^{-2}
        }
    \end{equation}
    Here we have $q = 4\pi \sin(\theta/2)/\lambda$ $\InvAngstrom$ as stated earlier. This
    result was has also been obtained by Hubbell~\cite{Hubbell-1975} and
    Pirenne~\cite{Pirenne}, however in a slightly modified form due to a
    different definition of $q$ which does not include the factor of $4\pi$.


\chapter{Power law phenomenon}
\section{The power law}
Initially the simplest distribution assumed for the degree distribution on a social network is a \textit{Normal Gaussian}. But it is verified that the distribution of the degree is proportional to $ f(k) = \frac{1}{k^\beta} $ with $ \beta  $ close to 2. A function $ f(k) $ that decrease as increase $ k $ is called \textit{power law}
\subsection{Difference between a normal distribution and a power law}
The normal distribution is a representation of a balanced phenomenon, where in the mean there is the maximum frequency and in the slopes decrease the frequency.\\
In the power law instead is a representation of a imbalanced phenomenon where, with small values of x we have high frequency and with high values x we have a small frequency. Some example of power law phenomenon are the richness, number of relations in a social network.
\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{img/power_law_example}
	\caption{Degree distribution on a Facebook social network}
	\label{fig:powerlawexample}
\end{figure}
\subsection{Check if a degree distribution is a power law}
In order to check if a degree distribution is a power law it must be in the form $ f(k) = \alpha \cdot \dfrac{1}{k^{\beta}} $.
We can do a linear regression in log-log scale of $ f(k) $ that is
\begin{align*}
\log(f(k)) = \log\left(\dfrac{\alpha}{k^{\beta}}\right) = \log(\alpha) - \beta \cdot \log(k)
\end{align*}
and should be a straight line to represent a power law

\section{Generative model of social network based on degree distributions}
In this section we study some generative models based on degree distribution of real graph. The generative models aim to estimate the $ p(x) $, the distribution of data that in this case is a social network
\subsection{Erd\"{o}s-Gilbert-R\'{e}nyi model}
This model is very simple, in fact it consider the probability \textit{p} that exists an edge between two nodes is independent to any other. \\ In statistical terms we can consider the distribution edge as a $ Bernoulli(k;p) $.\\ In this case we define the graph as $ G(n,p) $ where n is the number of nodes and p the probability that exists an edge.\\
The expected number of edges is:
%In prof handouts the formula is wrong, this is taken from https://economics.mit.edu/files/4621
\begin{align*}
\E \left[ Edges \right]= \sum_{1\leq x \leq y \leq n} (1\cdot p + 0 \cdot (1-p)) = \sum_{1\leq x \leq y \leq n} p = \binom{n}{2} p = \dfrac{n(n-1)}{2} p
\end{align*}
Since in a indirect graph an edge is present in the two extremes of the edge the expected degree of a node is $ 2 \cdot   \frac{n(n-1)}{2} p $.\\
The probability that a node have \textit{k} neighbours is a binomial distribution  so we can express as $ Bin(k; n, p) = \binom{n}{k} \cdot p^k \cdot (1-p) $. %also here an error on prof handouts
\\
This model for \textit{n} and \textit{k} sufficiently big approximate to a normal distribution
\subsection{Chung-lu model}
In this model the probability that exists an edge between two nodes is proportional to the expected degree of the pair of nodes.
\begin{itemize}
	\item Initially we set $ \textbf{d} = (d_1 \ldots d_n) $, the expected degree of each node.
	\item The probability that exists an edge between the node \textit{i} and \textit{j} is \begin{align*}  p(E_{ij}) = \dfrac{d_i d_j}{\sum_k d_k} 	\end{align*}
\end{itemize}
If the degree distribution \textbf{d} is a power law, then the graph generated is a power law
\subsection{Barabasi-Albert model}
The previous models are static, in fact they have a fixed number of nodes. \\This model is dynamic so it mean that doesn't require the number of nodes a priori but they can be added always.\\
With this dynamism try to give an explain to creation of a social network at difference of the previous models.\\
Every time a node is inserted an edge is created with the \textit{preferential attachment} system.
\subsubsection{Preferential attachment system}
Every time a node \textit{j} is added, a direct edge is created with the following two possibilities:
	\begin{itemize}
		\item with probability \textit{p} choose a random node $ i < j $
		\item with probability $ (1-p) $ choose a node with probability proportional to the in-degree of the node
	\end{itemize}
The latter option is called \textit{rich-get-richer rule}. In fact if you have a  high number of relation you are more likely to make more.
\subsubsection{Degree distribution}
When a node \textit{j} is created, the probability that exists the edge $ (j, i)) $ with $ j > i $ is:
\begin{align*}
P(E_{ji}) &= \dfrac{1}{j-1} \cdot p + \dfrac{G_i(j-1)}{\sum_{h \leq (j-1)} G_h(j-1)} \cdot (1-p) \\
&=  \dfrac{1}{j-1} \cdot p + \dfrac{G_i(j-1)}{j-1} \cdot (1-p)
\end{align*}
where $ G_i(j-1) $ is the in-degree of the node \textit{i} at the time $ j-1 $
\subsubsection{Deterministic $ G_j(t) $}
We want to find an approximation of $ G_j(t) $ that give deterministically the in-degree of a node $ l $ at time $ t $. Let us denote as $ g_l(t) $ the approximation of $ G_l(t) $. The increase of in-degree can be expressed from a differential equation:
\begin{align*}
	& \dfrac{d(g_l(t))}{dt} = \dfrac{p}{t} + \dfrac{ g_l(t) \cdot (1-p)}{t} = \dfrac{p + g_l(t) (1-p)}{t} \\
	& \dfrac{d(g_l(t))}{dt} \cdot \dfrac{1}{p + g_l(t) (1-p)} =  \dfrac{1}{t} \\
	& \int \dfrac{d(g_l(t))}{dt} \cdot \dfrac{1}{p + g_l(t) (1-p)} dt = \int   \dfrac{1}{t}  dt \\
	& g_l(t) \cdot \dfrac{\ln(p + g_l(t) (1-p))}{1-p} + c' = \ln(t) + c'' \\
	& g_l(t) \cdot  \ln(p + g_l(t) (1-p)) =  \ln(t) \cdot (1-p) + c \\
	& e^{\ln(p + g_l(t) (1-p))} = e^{\ln(t) \cdot (1-p) + c} \\
	&e^{\ln(p + g_l(t) (1-p))} = e^{\ln(t) \cdot (1-p) } \cdot e^c \\
	&(p + g_l(t) (1-p)) = t^{(1-p) } \cdot e^c \\
	& g_l(t) = \dfrac{t^{1-p} \cdot e^c - p}{1-p}
\end{align*}
When we insert the node $ l $ at time $ l $ we have that $ g_l(l) = 0  $ so we have
\begin{align*}
 & 0= \dfrac{l^{1-p} \cdot e^c - p}{1-p}	 \\
 & \dfrac{p}{1-p} =\dfrac{l^{1-p} \cdot e^c}{1-p} \\
 &	e^c = \dfrac{p}{l^{1-p}}
\end{align*}
if we substitute the latter expression in $ g_l(t) $ we have
\begin{align*}
	 &g_l(t) = \dfrac{t^{1-p} \cdot \dfrac{p}{l^{1-p}} - p}{1-p}  \\
	 &g_l(t) = p \cdot \left( \left(\frac{t}{l}\right)^{(1-p)}  -1 \right) \cdot \dfrac{1}{1-p}
\end{align*}
\noindent
We can use the latter expression to estimate the number of nodes that at time \textit{t} have degree at least \textit{k}. So we write
\begin{align*}
	& \left( \left(\frac{t}{l}\right)^{(1-p)}  -1 \right) \cdot \dfrac{p}{1-p} \geq k \\\\
	&\text{Then we pone respect to l: } \\
	& \left(\frac{t}{l}\right)^{(1-p)} \geq k \cdot \dfrac{1-p}{p} + 1 \\
	& \frac{t}{l} \geq \left(k \cdot \dfrac{1-p}{p} + 1\right)^{\frac{1}{1-p}}
	\end{align*}
Note that the next expression is the fraction of nodes at time \textit{t} that have at least in-degree \textit{k}:
\begin{align*}
	 & \frac{l}{t}\leq \left(k \cdot \dfrac{1-p}{p} + 1\right)^{-\frac{1}{1-p}} \\
	  & l \leq t \cdot \left(k\cdot \dfrac{1-p}{p} + 1\right)^{-\frac{1}{1-p}}
\end{align*}
To have the nodes that have \textit{exactly k in-degree} at \textit{time t} we must take the opposite of derivative (why?) of $  \left(k \cdot \dfrac{1-p}{p} + 1\right)^{-\frac{1}{1-p}} $ respect to k.\\ The result is
\begin{center}
	$ \dfrac{1}{1-p} \left( k \dfrac{1-p}{p} + 1 \right)^{-\left(1 + \frac{1}{1-p}\right)} $
\end{center}
We can infer from the latter expression that the distribution of in-degree nodes is a power law with exponent $ \beta =  1 + \frac{1}{1-p}$.\\
From
\begin{itemize}
	\item  when $ p \simeq 1 $ we exponent disappear then we tend to Erd\"{o}s-Gilbert-R\'{e}nyi model.
	\item when $ p \simeq 0 $ we apply almost ever the Preferential Attachment  System,  so the new nodes will more probably link to others with high in-degree, that is very similar to what happen with Chung-lu model.
	\item The distribution disappear when $ p \propto k^{-2} $ %Not understand how
%	in fact we have\\ $ \left( k \dfrac{1-k^{-2}}{k^{-2}} + 1 \right)^{-\dfrac{1-k^{-2}+1}{1-k^{-2}}}  = \left( k \dfrac{1-k^{-2} + k^{-2}}{k^{-2}} \right) ^ {\dfrac{k^{-2}-2}{1-k^{-2}}}= k^{3 \cdot \dfrac{k^{-2}-2}{1-k^{-2}}}$
\end{itemize}


%---------------------------Scaled Jacobian-----------------------------
\section{Scaled Jacobian}

The scaled Jacobian is the minimum of
the Jacobian at each corner divided by the lengths of the 2 edge vectors
(which is the minimum sine of the included angles):
\[
q =
  \min \left\{ \frac {\alpha_0} {\normvec{L_0} \normvec{L_3}},
               \frac {\alpha_1} {\normvec{L_1} \normvec{L_0}},
               \frac {\alpha_2} {\normvec{L_2} \normvec{L_1}},
               \frac {\alpha_3} {\normvec{L_3} \normvec{L_2}}
  \right\}.
\]
Note that if any edge has $L< DBL\_MIN$, we take $q = 0$.

\quadmetrictable{scaled Jacobian}%
{$1$}%                                      Dimension
{$[0.3,1]$}%                                Acceptable range
{$[-1,1]$}%                                 Normal range
{$[-1,1]$}%                                 Full range
{$1$}%                                      Unit square
{\cite{knu:00}}%                            Citation
{v\_quad\_scaled\_jacobian}%                Verdict function name


\title{Bayesian Neural Network}

\subsection{Bayesian Neural Network}

A Bayesian neural network is a neural network with a prior
distribution on its weights \citep{neal2012bayesian}.

Consider a data set $\{(\mathbf{x}_n, y_n)\}$, where each data point
comprises of features $\mathbf{x}_n\in\mathbb{R}^D$ and output
$y_n\in\mathbb{R}$. Define the likelihood for each data point as
\begin{align*}
  p(y_n \mid \mathbf{w}, \mathbf{x}_n, \sigma^2)
  &=
  \text{Normal}(y_n \mid \mathrm{NN}(\mathbf{x}_n\;;\;\mathbf{w}), \sigma^2),
\end{align*}
where $\mathrm{NN}$ is a neural network whose weights and biases form
the latent variables $\mathbf{w}$. Assume $\sigma^2$ is a
known variance.

Define the prior on the weights and biases $\mathbf{w}$ to be the standard normal
\begin{align*}
  p(\mathbf{w})
  &=
  \text{Normal}(\mathbf{w} \mid \mathbf{0}, \mathbf{I}).
\end{align*}

Let's build the model in Edward. We define a 3-layer Bayesian neural
network with $\tanh$ nonlinearities.
\begin{lstlisting}[language=Python]
from edward.models import Normal

def neural_network(x):
    h = tf.tanh(tf.matmul(x, W_0) + b_0)
    h = tf.tanh(tf.matmul(h, W_1) + b_1)
    h = tf.matmul(h, W_2) + b_2
    return tf.reshape(h, [-1])

N = 40  # number of data points
D = 1   # number of features

W_0 = Normal(loc=tf.zeros([D, 10]), scale=tf.ones([D, 10]))
W_1 = Normal(loc=tf.zeros([10, 10]), scale=tf.ones([10, 10]))
W_2 = Normal(loc=tf.zeros([10, 1]), scale=tf.ones([10, 1]))
b_0 = Normal(loc=tf.zeros(10), scale=tf.ones(10))
b_1 = Normal(loc=tf.zeros(10), scale=tf.ones(10))
b_2 = Normal(loc=tf.zeros(1), scale=tf.ones(1))

x = tf.cast(x_train, dtype=tf.float32)
y = Normal(loc=neural_network(x), scale=0.1 * tf.ones(N))
\end{lstlisting}
This program builds the model assuming the features \texttt{x\_train}
already exists in the Python environment. Alternatively, one can also
define a TensorFlow placeholder,
\begin{lstlisting}
x = tf.placeholder(tf.float32, [N, D])
\end{lstlisting}
The placeholder must be fed with data later during inference.

A toy demonstration is available in the \href{/getting-started}{Getting Started} section.
Source code is available at
\href{https://github.com/blei-lab/edward/blob/master/examples/bayesian_nn.py}
{\texttt{examples/bayesian\_nn.py}} in the Github repository.

\subsubsection{References}\label{references}


\section{Small Tips and Tricks}\label{subsec:tipsAndTricks}

We end this tutorial by giving some helpful tips and tricks, and giving pointers
where to find more information about HOL4.

\subsection{Extending the Simplifier}
While proving the gaussian sum in \autoref{sec:hol_ex1} we had to explicitly
tell the simplifier that it should also use the definition of function
\lstinline{sum} while proving properties about the function.
In a pen-and-paper proof, one would never explicitly write this down and as such
it is desirable to have the same convenience in HOL4.
We can do so by slightly changing the definition of \lstinline{sum}:
%
\begin{lstlisting}
Definition sum_def[simp]:
  sum 0 = 0 /\
  sum n = n + sum (n-1)
End
\end{lstlisting}

By appending \lstinline{[simp]} to the name of the function, HOL4 automatically
adds \lstinline{sum_def} to the list of theorems used by the simplifier.
A similar mechanism exists for adding theorems to the simplifier.
However, this mechanic has to be used with caution as it is very easy to make
the simplifier diverge.

As an example, suppose we used the old definition of \lstinline{sum} which
defines the function as a recursive function not in equational style:

\begin{lstlisting}
Definition sum_def[simp]:
  sum n = if (n = 0) then 0 else n + sum (n-1)
End
\end{lstlisting}

If we restart the proof for the gaussian sum now, and run through the first two
tactics only (\lstinline{nltac `Induction on 'n'. simplify.`}) HOL4 will just
keep running.
As a rule of thumb, it is recommended to be conservative and rather mention a
definition than adding it to the simplifier.
The machinery can be useful for (non-recursive) abbreviations.
For theorems, one should refrain from adding commutativity or associativity
theorems, but adding theorems of the form $\forall x. P x \rightarrow Q x$, where
$Q$ does not depend on $P$ should be fine.

\subsection{Making Proof Scripts More Robust}
The most cumbersome work once a proof has been developed is making sure that it
remains correct even when versions of HOL4 change.
We give some simple recommendations that have proven quite useful over time.

First, we recommend commenting larger case splits and induction proofs.
While it may seem obvious now which case is being worked on by the proofscript
this might not be the case in a month, or a year of time after writing the
initial version.

Second, we recommend using tactics like
\lstinline{first_x_assum, last_assum, qpat_x_assum}.
These tactics are independent of the specific order of assumptions and thus make
the proof more robust to additional assumptions, or their removal.

\subsection{Getting More Help}
This small tutorial has only covered the basics.
More reference material can be found on \url{https://hol-theorem-prover.org/#doc}.
We especially recommend looking at the documentation of the emacs mode
(\url{https://hol-theorem-prover.org/hol-mode.html}), and the description manual.

The help index located at \lstinline{<HOLDIR>/help/HOLindex.html} provides
documentation for a lot of tactics and contains signature files for all of the
HOL4 distributions libraries and theories.

Finally, the HOL-info mailing list
(\url{https://sourceforge.net/projects/hol/lists/hol-info}) is a good place to
ask further questions, as well as the \texttt{\#hol} channel of the Slack of the
CakeML project (\url{https://join.slack.com/t/cakeml/shared_invite/MjM1NjEyODgxODkzLTE1MDQzNjgwMTUtYjI4YTdlM2VmMQ}).


LazySets provides ways to convert one set representation to another.
If a conversion is not possible due to restrictions in the represented class of sets, LazySets provides ways to obtain approximations.
Turning to an approximate but simpler set representation is also interesting for answering questions efficiently that would otherwise be computationally expensive.

\subsection{Conversion}

LazySets extends Julia's \code{convert} function for converting between set representations. The first argument is the target type and the second argument is the source set.
%
Below are three mathematically equivalent representations of the interval $\X = [0, 1] \subseteq \R$:

\begin{minipage}{\linewidth}
	\vspace{-\abovedisplayskip}
	\begin{lstlisting}
julia> X = Interval(0, 1)
Interval{Float64,
  IntervalArithmetic.Interval{Float64}}([0, 1])

julia> convert(Hyperrectangle, X)
Hyperrectangle{Float64, Vector{Float64},
  Vector{Float64}}([0.5], [0.5])

julia> convert(Zonotope, X)
Zonotope{Float64, Vector{Float64},
  Matrix{Float64}}([0.5], [0.5])
	\end{lstlisting}
\end{minipage}
There are even more possibilities, such as representing $\X$ as an intersection of half-spaces (try \code{convert(HPolytope, X)}).

\smallskip

With multiple dispatch it is easy to define less obvious conversions, e.g., to convert the Cartesian product of an interval and a two-dimensional hyperrectangle to a three-dimensional zonotope:

\begin{minipage}{\linewidth}
	\vspace{-\abovedisplayskip}
	\begin{lstlisting}
julia> X = rand(Interval)

julia> Y = rand(Hyperrectangle, dim=2)

julia> Z = convert(Zonotope, X × Y)
Zonotope{Float64, ...}

julia> dim(Z)
3
\end{lstlisting}
\end{minipage}

\subsection{Approximation}\label{sec:approximation}

\begin{figure}
	\hfill
	\includegraphics[height=30mm]{img/overapproximate}
	\hfill
	\includegraphics[height=30mm]{img/polardirs}
	\hfill\
	\vspace*{1mm}
	\caption{Left picture: Overapproximation of the polytope from Fig.~\ref{fig:supfunc} (orange) with a hyperrectangle (blue) and two zonotopes. The zonotope generators were synthesized from three (red) resp.\ five (green) polar directions (right). Observe that the approximations are pairwise incomparable.}
	\label{fig:overapproximate}
\end{figure}

In many applications we do not require exact results but are content with an approximation. To still give mathematical guarantees, one usually aims for either over- or underapproximations.

We can use the support function to get an overapproximation: For every nonempty compact convex set $X \subseteq \R^n$ and $D \subseteq \R^n$ we have
\begin{equation*}
	X \subseteq \bigcap_{d \in D} \{d^T x \leq \rho(d, X)\}
\end{equation*}
and equality holds for $D = \R^n$.

\smallskip

LazySets has predefined common template directions such as \code{OctDirections(2)} for directions normal to a regular octagon in two dimensions. Fig.~\ref{fig:supfunc} (right) illustrates the evaluation of overapproximating the set $X$ with octagonal directions, resulting in a polygon with eight constraints. Apart from common fixed template directions there are also options for parametric uniform directions in two (\code{PolarDirections}) or three (\code{SphericalDirections}) dimensions or for a custom set of directions (\code{CustomDirections}).

\begin{minipage}{\linewidth}
\vspace{-\abovedisplayskip}
\begin{lstlisting}
julia> Xoct = overapproximate(X, OctDirections(2))

julia> length(constraints_list(Xoct))
8
\end{lstlisting}
\end{minipage}

In two dimensions, LazySets can compute $\varepsilon$-close overapproximations using a method by Kamenev~\cite{kamenev1996algorithm} later refined in~\cite{lotov2008modified}.
%
It is used via \code{overapproximate(X, $\varepsilon$)}, where $\varepsilon$ is the specified tolerance.
%
The higher-dimensional extension is not implemented yet.
%
On the other hand, a higher-dimensional set can be lazily projected using the support function to a lower-dimensional subspace, where the available method applies.

\smallskip

In some applications, we may want to ensure that the result has a specific set type.
The smallest bounding box is available by specifying the second argument type. It yields a \code{Hyperrectangle}, which is more efficient to work with.

\begin{minipage}{\linewidth}
	\vspace{-\abovedisplayskip}
	\begin{lstlisting}
julia> overapproximate(X, Hyperrectangle)

julia> box_approximation(X) # alias
\end{lstlisting}
\end{minipage}

We can use \code{overapproximate(P, Zonotope, D)}, where \code{P} is a polytope and \code{D} is a vector of directions used as candidates for the generators, to obtain a zonotope (in any dimension). We show some example overapproximations in Fig.~\ref{fig:overapproximate}.
Underapproximations can be obtained using the function \code{underapproximate}.