\chapter{Network Analysis}

\newthought{Network analysis can tell us a lot about the most imporant elements of the graph and how well-connected each element is.} A common way is to count the number of connections, shortest paths to neighbors, and shapes that the network takes (i.e. number of triangles).

Some common measures include:
\begin{itemize}
    \item \emph{average degree}, which corresponds to the average number of edges per node. For lastfm, this would mean the average number of artists with whom the genre tag is shared. The higher the number, the more densely connected the network is. A similar measure is graph density, which corresponds to the ratio between number of edges and all possible number of edges.
    \begin{marginfigure}
        \centering
        \includegraphics[scale=0.3]{graph.png}
        \caption{The average degree of the above graph is 2.5 (The number of edges for each node divided by the number of nodes.). The degree of the blue node is 5, while the degree for the red node is 3. The degree centrality for the blue node is 0.5 (half of all edges are connected to it!), while for the red node it is 0.3.}
    \end{marginfigure}
    \item \emph{degree}, which is the number of edges per node. The higher the number, the better connected the node is. For lastfm, this would mean the artist with the highest degree would be the one that shares the most genre tags with other artists.
    \item \emph{degree centrality}, which is the ratio of nodes that connect to the given node. A high ratio would mean a node that can reach the most other nodes and that the node is very important in graph structure. In the case of lastfm, this would be the artist that represents the most widely used genre (perhaps pop, rock, jazz).
\end{itemize}

But enough talk, let us observe this in practice! A widget that computes these statistics is called \widget{Network Analysis}.

\begin{figure*}[h!]
    \centering
    \newcommand{\hierclust}{\includegraphics[scale=0.45]{net-analysis-graph.png}}
    \newcommand{\imageview}{\includegraphics[scale=0.45]{net-analysis-node.png}}
    \infinitewidthbox{
    \stackinset{r}{-0.45\linewidth}{t}{0.0\linewidth}{\imageview}{\hierclust}\hspace{6cm}
    }
\end{figure*}

\newpage

\begin{wrapfigure}{o}{0.8\textwidth}
    \includegraphics[scale=0.6]{workflow.png}
    \label{fig:embedding}
\end{wrapfigure}

Some statistics relate to graph properties, while others relate to node properties. Select the above three measures. Node statistics will be added to the data table, which means we can observe it in \widget{Network Explorer}.

Sending additional node data to Network Explorer enables us to use the computed statistics in a visualization. We can color the nodes and set their size to degree centrality. This will expose the most central nodes - the ones that share the most tags with their neighbors. Unsurprisingly, we find The Rolling Stones and Britney Spears in the bunch.

\vspace{-0.2cm}
\begin{figure*}[h]
  \centering
  \includegraphics[width=\linewidth]{network-explorer.png}%
  \caption{$\;$}
\end{figure*}
\vspace{-0.3cm}


\chapter{Vector-valued Functions}

In the last chapter, you calculated the flight of the shell.  For any
time $t$, you could find a vector $[distance, height]$. This can be
thought of as a function $f$ that takes a number and returns a
2-dimensional vector.  We call this a \newterm{vector-valued} function
from $\mathbb{R} \rightarrow \mathbb{R}^2$.

We often make a vector-valued function by defining several real-valued
functions.  For example, if you threw a hammer with an initial upward
speed of 12 m/2 and a horizontal speed of 4 m/s along the $x$ axis from
the point $(1, 6, 2)$, its position at time $t$ (during its flight) would be given by:

$$f(t) = [4t + 1, 6, -4.8t^2 + 12t + 2]$$

That is, $x$ is increasing with $t$, $y$ is constant, and $z$ is a parabola.

\tdplotsetmaincoords{80}{20}
\begin{tikzpicture} [scale=0.5, tdplot_main_coords, axis/.style={->,sdkblue},
vector/.style={-stealth,black,very thick},
vector guide/.style={dashed,sdkblue}]

%standard tikz coordinate definition using x, y, z coords
\coordinate (O) at (0,0,0);

%draw axes
\draw[axis] (0,0,0) -- (12,0,0) node[anchor=north east]{$x$};
\draw[axis] (0,0,0) -- (0,7,0) node[above]{$y$};
\draw[axis] (0,0,0) -- (0,0,10) node[anchor=south]{$z$};

\draw[thick,draw=black,
      domain=0:2.60563,samples=300,variable=\t]
      plot ({4*\t + 1},6, {-4.9*\t^2 + 12*\t + 2});
\draw[dashed,draw=sdkblue] (1, 6, 0) -- (11.42252, 6, 0);
\draw[dashed,draw=sdkblue] (11.42252, 0, 0) -- (11.42252, 6, 0);
\draw[dashed,draw=sdkblue] (1, 6, 0) -- (1, 6, 2);
\draw[dashed,draw=sdkblue] (1, 6, 0) -- (1, 6, 2);
\draw[dashed,draw=sdkblue] (1, 6, 0) -- (1, 0, 0);

\filldraw[black] (1,6,2) circle(4pt) node [left]{$f(0) = [1,6,2]$};
\filldraw[black] (5,6,9.1) circle(4pt) node [above]{$f(1) = [5, 6, 9.1]$};
\draw[dashed,draw=sdkblue] (5, 6, 0) -- (5, 6, 9.1);
\filldraw[black] (9,6,6.4) circle(4pt) node [right]{$f(2) = [9, 6, 6.4]$};
\draw[dashed,draw=sdkblue] (9, 6, 0) -- (9, 6, 6.4);
\filldraw[black] (11.42252,6,0) circle(4pt) node [right] {$f(2.6) = [11.4, 6, 0]$};
\end{tikzpicture}

\section{Finding the velocity vector}

Now that we have its position vector, we can differentiate each
component separately to get its velocity as a vector-valued function:

$$f'(t) = [4, 0, -9.8t + 12]$$

That is, the velocity is constant along the $x$-axis, zero along the
$y$-axis, and decreasing with time along the $z$ axis.

\tdplotsetmaincoords{80}{20}
\begin{tikzpicture} [scale=0.5, tdplot_main_coords, axis/.style={->,sdkblue},
vector/.style={-stealth,black,very thick},
vector guide/.style={dashed,sdkblue}]

%standard tikz coordinate definition using x, y, z coords
\coordinate (O) at (0,0,0);

%draw axes
\draw[axis] (0,0,0) -- (14,0,0) node[anchor=north east]{$x$};
\draw[axis] (0,0,0) -- (0,7,0) node[above]{$y$};
\draw[axis] (0,0,0) -- (0,0,10) node[anchor=south]{$z$};

\draw[thick,dashed,draw=black,
      domain=0:2.60563,samples=300,variable=\t]
      plot ({4*\t + 1},6, {-4.9*\t^2 + 12*\t + 2});

\filldraw[black] (1,6,2) circle(4pt);
\draw[->, thick, draw=black] (1,6,2) -- (5, 6, 14) node [right] {$f'(0) = [4,0,12]$};
\draw[dashed, draw=sdkblue] (1,6,2) -- (5,6,2) -- (5,6,14);
\filldraw[black] (5,6,9.1) circle(4pt);
\draw[->, thick, draw=black] (5,6,9.1) -- (9, 6, 11.3) node [right] {$f'(1) = [4,0,2.2]$};
\draw[dashed, draw=sdkblue] (6,6,9.1) -- (9,6,9.1) -- (9,6,11.3);
\filldraw[black] (9,6,6.4) circle(4pt);
\draw[->, thick, draw=black] (9,6,6.4) -- (13, 6, -1.2) node [right] {$f'(2) = [4,0,-7.6]$};
\filldraw[black] (11.42252,6,0) circle(4pt);
\draw[dashed, draw=sdkblue] (9,6,6.4) -- (13,6,6.4) -- (13,6,-1.2);
\end{tikzpicture}

\section{Finding the acceleration vector}

Now that we have its velocity, we can get its acceleration as a vector-valued function:

$$f''(t) = [0, 0, -9.8]$$

There is no acceleration along the $x$ or $y$ axes. It is accerating
down at a constant $9.8 m/s^2$.

\tdplotsetmaincoords{80}{20}
\begin{tikzpicture} [scale=0.5, tdplot_main_coords, axis/.style={->,sdkblue},
vector/.style={-stealth,black,very thick},
vector guide/.style={dashed,sdkblue}]

%standard tikz coordinate definition using x, y, z coords
\coordinate (O) at (0,0,0);

%draw axes
\draw[axis] (0,0,0) -- (14,0,0) node[anchor=north east]{$x$};
\draw[axis] (0,0,0) -- (0,7,0) node[above]{$y$};
\draw[axis] (0,0,0) -- (0,0,10) node[anchor=south]{$z$};

\draw[thick,dashed,draw=black,
      domain=0:2.60563,samples=300,variable=\t]
      plot ({4*\t + 1},6, {-4.9*\t^2 + 12*\t + 2});

\filldraw[black] (1,6,2) circle(4pt);
\draw[->, thick, draw=black] (1,6,2) -- (1, 6, -7.8) node [right] {$f''(0) = [0,0,-9.8]$};
\filldraw[black] (5,6,9.1) circle(4pt);
\draw[->, thick, draw=black] (5,6,9.1) -- (5, 6, -0.7) node [below] {$f''(1) = [0,0,-9.8]$};
\filldraw[black] (9,6,6.4) circle(4pt);
\draw[->, thick, draw=black] (9,6,6.4) -- (9, 6, -3.4) node [right] {$f''(2) = [0,0,-9.8]$};
\filldraw[black] (11.42252,6,0) circle(4pt);
\end{tikzpicture}


%================================
\section{Metrics versus Topologies}
%================================

%--------------------------------
\begin{definition}
	Let $X$ be a set, and let $\rho$ and $\rho'$ be metrics on $X$. We say that $\rho$ and $\rho'$ are \textit{topologically equivalent} if they induce the same topology on $X$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	$\rho $ and $\rho'$ are \textit{Lipschitz equivalent} iff there exist $c, C \in \mathbb R_{>0}$ such that for all $x,y \in X$,
	$$
	c \rho(x,y) \le \rho'(x,y) \le C\rho(x,y).
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{lemma}
	Lipschitz equivalence implies topological equivalence.
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof}
	As $\rho$ and $\rho'$ are Lipschitz equivalent, by definition, there exist $c \in \mathbb R_{>0}$ such that for all $x,y \in X$,
	$$
	c \rho(x, y) \le \rho'(x, y).
	$$

	Given $r > 0$ and $x \in X$,
	$$
	B_{c\rho}(x, r) = \left\{ y \in X : c\rho(x, r) < r \right\}
	$$
	and
	$$
	B_{\rho'}(x,r) = \{ y \in X : \rho'(x,r) < r \}.
	$$

	As $r$ is non-underestimated compared to $\rho'$, then
	$$
	B_{\rho'}(x,r) \supseteq B_{c\rho}(x, r) = B_{\rho}\left(x, \frac{1}{c}r \right)
	$$
	is an open neighbourhood of $x$ in $(X, \rho')$ and is a subset

	Let $U \in \mathcal T_{\rho'}$, then for some $\varepsilon > 0$,
	$$
	U \supseteq B_{\rho'} (x, \varepsilon) \supseteq B_{\rho}\left( x, \frac{1}{c}r \right).
	$$
	Thus $U$ is open with respect to $\rho$, i.e., $U \in \mathcal T_{\rho}$.

	It is not necessary to prove converse for there always exists $C \in \mathbb R_{>0}$ such that $c = \frac{1}{C}$.
\end{proof}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		%--------------------------------
		\item For all $p \ge 0$, $\rho_p: \mathbb R^n \times \mathbb R^n \to \mathbb R_{\ge 0}$ are topologically equivalent.
		%--------------------------------

		%--------------------------------
		\item On $C[a,b]$, $\rho_1$ and $\rho_\infty$ induce different topologies, hence they are not topologically equivalent, and in particular, they are not Lipschitz equivalent. As Lipschitz equivalence implies topological equivalence, but not vice versa. So Lipschitz in-equivalence do nothing to the proof the topological in-equivalence between $\rho_1$ and $\rho_\infty$.
		%--------------------------------

		%--------------------------------
		\item $\rho_p$ and $\rho_\text{disc}$ on $\mathbb Z$ are topologically equivalent. Firstly, topology $\mathcal T_{\rho_p} = \mathcal P(\mathbb Z)$, because for all $B_{\rho_p} (x, \varepsilon)$ for all $x \in \mathbb Z$ and $\varepsilon \in \mathbb R_{(0,1)}$,  $B_{\rho_p}(x, \varepsilon) = \{x\}$. Thus, for all, $U \in \mathcal P(\mathbb Z)$,
			$$
			U = \bigcup_{x \in U} B_{\rho_p}(x, \varepsilon) = \bigcup_{x \in U} \{x\} \in \mathcal T_{\rho_p}.
			$$
			Thus $\mathcal P(\mathbb Z) \subseteq \mathcal T_{\rho_p}$, but $\mathcal T_{\rho_p} \subseteq \mathcal P(\mathbb Z)$, so $\mathcal P(\mathbb Z) = \mathcal T_{\rho_p}$. Thus $\mathcal T_{\rho_p} = \mathcal T_\text{disc}$.
		%--------------------------------
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{definition}
	A topological space $(X, \mathcal T)$ is \textit{metrizable} iff $\mathcal T$ is induced by some metric on $X$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item Let $(\mathbb Z, \mathcal T)$ with
		$$
		\mathcal T = \{ U \in \mathcal P(\mathbb Z) : |U| \le 1 \},
		$$
		Then $\mathcal T$ is not induced by any metric. Suppose it were, then all open set $U \in \mathcal T$ should be monotone, and for all $\varepsilon > 0$, and for all $x \in \mathbb Z$, $B(x, \varepsilon)$ should be monotone. But if $\mathcal T$ is induced by some metric, then for all $I \in \mathcal P(X)$ with $|I| > 1$, a set
		$$
		W = \bigcup_{x \in I} B(x, \varepsilon) \in \mathcal T,
		$$
		then $|W| > 1$, which is contradicted to the conditions.
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{definition} \
	\begin{enumerate}[(i)]
		\item A topological space $(X, \mathcal T)$ is said to be $T_1$ iff every monotone in $\mathcal P(X)$ is closed.
		\item A topological space $(X, \mathcal T)$ is said to be $T_2$ or \textit{Hausdorff} iff
			$$
			\forall x, y \in X \ (x \ne y), \ \exists U, W \in \mathcal T \ (U \cap W = \emptyset), \quad x \in U \land y \in W.
			$$
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item $(X, \mathcal T_{\rho_\text{disc}})$ is $T_1$, for as any set $U \subseteq X$ is closed for $X \setminus U \in \mathcal T_{\rho_\text{disc}}$ as well. It is also Hausdorff, because for all $x,y \in X$, $\{x\}, \{y\} \in \mathcal T_{\rho_\text{disc}}$ and $\{x\} \cap \{y\} = \emptyset$ if $x \ne y$.
		\item On the other hand, $(X, \{\emptyset, X\})$ is $T_1$ iff $|X| = 1$. And $(X, \{\emptyset, X\})$ is not Hausdorff, because there exist $x,y \in X$ with $x \ne y$, the only open set contains $x$ is $X$, and the only open set contains $y$ is $X$. Clearly, $X \cap X$
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{lemma} \
	\begin{enumerate}[(i)]
		\item Every metrizable space is Hausdorff.
		\item Every Hausdorff topological space is $T_1$.
	\end{enumerate}
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof} \
	\begin{enumerate} [(i)]
		\item Let $(X, \rho)$ be metric space, then for all $x, y \in X$, let $r = \frac{\rho(x,y)}{2}$. Suppose $(X, \rho)$ is not Hausdorff, i.e., there is $z \in B(x, r) \cap B(y, r)$. By metric axioms, we have
			$$
			\rho(x, z) + \rho(y, z) \ge \rho (x,y) = 2r.
			$$
			But $z \in B(x, r)$ implies that $\rho(x,z) < r$, and $z \in B(y, r)$ implies that $\rho(y,z) < r$, then we have
			$$
			\rho(x, z) + \rho(y, z) < \rho(x,y),
			$$
			which is contradicted to the metric axioms.

		\item (Just an outline...) Let $(X, \mathcal T)$ be Hausdorff. Suppose $X$ is not $T_1$, then there is $\{x\} \subseteq X$ which is not closed. Then there must be a smallest $V \supsetneq \{x\}$ which is closed (Why?). Then there must be a smallest $U \in \mathcal T$ with $U \supseteq V$ (Why?). Then for all $x, y \in U$, there is no disjoint $U_x, U_y$ such that $U_x \ni x$ and $U_y \ni y$.
	\end{enumerate}
\end{proof}
%--------------------------------

%--------------------------------
\begin{definition}
	Let $(X, \mathcal T)$ be a topological space, let $\{x_n\}_{n = 1}^\infty$ be a sequence in $X$, and let $x \in X$. Then $\{x_n\}$ \textit{converges} in $X$ iff there is an $x \in X$, for all $U \in \mathcal T$ with $x \in U$, $U$ contains all but finite elements in $\{x_n\}$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item If $(X, \mathcal T)$ is metrizable, i.e., there is a metric $\rho$ can induce $\mathcal T$. If $\{x_n\} \subseteq X$ converges in $X$, then there exists $x \in X$, for all $\varepsilon > 0$, $B(x, \varepsilon)$ contains all but finite elements in $\{x_n\}$.
		\item If $\mathcal T$ is a discrete topology, a sequence $\{x_n\}$ converges in $(X, \mathcal T)$ iff there is an $N$ such that for all $n \ge N$, $x_n = x_{n + 1}$.
		\item If $\mathcal T$ is an indiscrete topology, then any $\{x_n\} \subseteq X$ converges to any point in $X$, for there is only one non-empty open set which is $X$ itself.
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{lemma}
	In Hausdorff topological space, any convergent sequence converges to at most one point.
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof}
	Let $(X, \mathcal T)$ be a Hausdorff topological space. Suppose there is a sequence $\{x_n\}$ converges to $x,y \in X$ with $x \ne y$. By the definition of topological convergence, there are $U_x, U_y \in \mathcal T$ both contains all but finite elements in $\{x_n\}$. $U_x \cap U_y$ must be non-empty (Explain!). $x,y \in U_x \cap U_y$, for if they were not, by Hausdorff property, there must be open $V_x \subseteq U_x$ and $V_y \subseteq U_x$ with $V_x \ni x$ and $V_y \ni y$, and they both contains all but finite elements in $\{x_n\}$, which is not possible. Thus, there is no such open sets $V_x \ni x$ and $V_y \ni y$ with $V_x \cap V_y = \emptyset$, which implies $(X, \mathcal T)$ is not Hausdorff. This is a contradition.
\end{proof}
%--------------------------------

%--------------------------------
\begin{definition} \
	\begin{enumerate}[(i)]
		\item A topological space $(X, \mathcal T)$ is \textit{regular} iff for all closed sets $V \subseteq X$ and $x \in X$ with $x \notin V$, there exist disjoint open sets $U,W \subseteq X$ such that $V \subseteq U$ and $x \in W$.
		\item $(X, \mathcal T)$ is \textit{normal} iff for all disjoint closed sets $V, Z \subseteq X$, there exist disjoint open sets $U, W \subseteq X$ such that $V \subseteq U$ and $Z \subseteq W$.
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{note}
	[To do] \
	\begin{enumerate}
		\item Can I find a regular space which is not normal?
		\item Can I find a normal space which is not regular?
		\item Does regular implies normal or normal implies regular?
	\end{enumerate}
\end{note}
%--------------------------------

%


\chapter{Pythagorean Theorem}

Watch's Khan Academy's Intro to the Pythagorean Theorem video at \url{https://youtu.be/AA6RfgP-AHU}.

If you have a right triangle, the edges that touch the right angle are
called \emph{the legs}.  The third edge, which is always the longest,
is known as \emph{the hypotenuse}. The Pythagorean Theorem gives us
the relationship between the length of the legs and the length of the
hypotenuse.

\begin{tikzpicture}[scale=1.2]
  \coordinate [circle, fill, inner sep=1pt] (a) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b) at (0,4) ;
  \coordinate [circle, fill, inner sep=1pt] (c) at (3,0) ;
  \draw (a)--node [outer sep=3pt, left]{Length $a$}(b);
  \draw (b)--node[outer sep=3pt, right]{Length $c$}(c) ;
  \draw (c)--node[outer sep=3pt, below]{Length $b$}(a) ;
  \pic [draw, angle eccentricity=1.5] {right angle = c--a--b};
\end{tikzpicture}

The Pythagorean Theorem tells us that $a^2 + b^2 = c^2$.\index{Pythagorean theorem}

For example, if one leg has length 3 and the other has length 4, then
$a^2 + b^2 = 3^2 + 4^2 = 25$. Thus $c^2$ must equal 25. So you know
the hypotenuse must be of length 5.

(In reality, it almost never works out to be such a tidy number. For
example, what is the length of the hypotenuse if the two legs are 3
and 6? $a^2 + b^2 = 3^2 + 6^2 = 45$.  The length of the hypotenuse is
the square root of that: $\sqrt{45} = \sqrt{9 \times 5} = 3 \sqrt{5}$,
which is approximately 6.708203932499369.)

\begin{Exercise}[title={Find the Missing Length}, label=missingsides]
  What is the missing measure?
  \begin{multicols}{2}
Leg 1 = 6, Leg 2 = 8, Hypotenuse = ? \\(It should be a whole number.)

Leg 1 = 5, Leg 2 = ?, Hypotenuse = 13 \\(It should be a whole number.)

Leg 1 = ?, Leg 2 = 15, Hypotenuse = 17 \\(It should be a whole number.)

Leg 1 = 3, Leg 2 = 3, Hypotenuse = ? \\(It is an irrational number. Give the exact answer and then use a calculator to get an approximation.)
\end{multicols}
\end{Exercise}
\begin{Answer}[ref=missingsides]
  10 because $6^2 + 8^2 = 10^2$

  12 because $5^2 + 12^2 = 13^2$

  8 because $8^2 + 15^2 = 17^2$

  $3\sqrt{2} \approx 4.24$ because $3^2 + 3^2 = \left(3 \sqrt{2}\right)^2$
\end{Answer}

\section{Distance between Points}

What is the distance between these two points?\index{distance using Pythagorean theorme}

\begin{tikzpicture}
  % axis
  \draw[thick, <->] (0, -4.1) -- (0, 4.1);
  \draw[thick, <->] (-4.1, 0) -- (4.1, 0);
  \coordinate [circle, fill, inner sep=2pt] (a) at (-1,-2) ;
  \coordinate [circle, fill, inner sep=2pt] (b) at (1,3) ;
  \node [left] at (a) {(-1,-2)};
  \node [right] at (b) {(1,3)};
  % grid
  \draw[help lines, step = 1cm] (-4, -4) grid (4, 4);

\end{tikzpicture}

We can draw a right triangle and use the Pythagorean Theorem:

\begin{tikzpicture}
  % axis
  \draw[thick, <->] (0, -4.1) -- (0, 4.1);
  \draw[thick, <->] (-4.1, 0) -- (4.1, 0);
  \coordinate [circle, fill, inner sep=2pt] (a) at (-1,-2) ;
  \coordinate [circle, fill, inner sep=2pt] (b) at (1,3) ;
    \node [left] at (a) {(-1,-2)};
  \node [right] at (b) {(1,3)};
  \coordinate (c) at (1, -2);

  \draw [thick] (a) -- (b);
  \draw [thick] (a) -- node[outer sep = 3pt, below]{Length 2}(c);
  \draw [thick] (c) -- node[rotate=90, outer sep = 3pt, below]{Length 5}(b);
  \draw[help lines, step = 1cm] (-4, -4) grid (4, 4);

\end{tikzpicture}

The distance between the two points is $\sqrt{2^2 + 5^2} = \sqrt{29}
\approx 5.385165$. That is, you square the change in $x$ and add it to
the square of the change in $y$. The distance is the square root of
that sum.

\section{Distance in 3 Dimensions}

What if the point is in three dimensional space?  That is, you move 2
meters East, 8 meters North, and 4 meters up in the air. How far are
you from where you started?  You just square each, sum them and take the square root:
$\sqrt{2^2 + 8^2 + 4^2} = \sqrt{84} = 2\sqrt{21} \approx 9.165$ meters.\index{distance!in 3 dimensions}

\begin{tikzpicture}
  \draw [thick, ->] (0,0,0) -- (9,0,0) node[outer sep = 1pt, right]{North} ;
  \draw [thick, ->]  (0,0,0) -- (0,3,0) node[outer sep = 1pt, above]{Up} ;
  \draw [thick, ->] (0,0,0) -- (0,0,4) node[outer sep = 1pt, below]{East} ;

    \draw [dashed]  (8,0,0) -- node[outer sep = 1pt, right]{2}  (8,0,2);

  \draw [dashed]  (0,0,2) -- node[outer sep = 1pt, below]{8} (8,0,2);
  \draw [dashed]  (8,0,2) -- node[outer sep = 1pt, right]{4} (8,4,2);
  \draw [thick]  (0,0,0) --  (8,4,2) node[circle, fill, inner sep=2pt]{};
  \node [left] at (5, 2.7, 1){$\sqrt{2^2 + 8^2 + 4^2} \approx 9.165$};

\end{tikzpicture}


%!TEX root = TDT4265-Summary.tex
\section{Intensity transformations}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Basic functions}
Image intensity levels are defined in the range $[0, L-1]$. $0$ is black, $L$ is white.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=.8\linewidth]{images/basic-transformations}
    \caption{Some basic grey-level transformations}
    \label{fig:basic-transformations}
\end{figure}

\subsubsection{Negative}
Inverts the intensity levels of the image (white becomes black and so on). Enhances details in dark regions.
\begin{equation}
    s = L - 1 - r
\end{equation}

\subsubsection{Log transformation}
Used to expand dark areas and compress bright areas, i.e. give more shadow detail. Good for displaying images with very large dynamic range.
\begin{equation}
    s = c \log(1 + r), \quad r \geq 0
\end{equation}

\subsubsection{Gamma transformation}
Can compress blacks or whites depending on the gamma value. Similar to the log transformation, but more versatile. Many devices have an inherent gamma transform response that can be compensated for by applying the opposite gamma transform. Can be used to improve detail by darkening a washed-out image or brightening a too dark image.
\begin{equation}
    s = c r^\gamma
\end{equation}

\subsubsection{Piecewise linear transformation}
Use a piecewise linear function as the transformation (as opposed to the smooth transformations in Figure \ref{fig:basic-transformations}). Many uses, depending on the function:
\begin{itemize}
    \item Contrast stretching: Increase the dynamic range of the grey levels, similar to a a sigmoid function. (Can also be done with a smooth function.)
    \item Grey-level slicing: Highlight a range of grey-levels. Either leave other values as they are or darken all other values.
    \item Bit-plane slicing: Consider e.g. an 8-bit image to be made of 8 bit `planes'. Extracting one or some of these is called bit-plane slicing. The most significant plane can be extracted to form a binary black or white image, for instance.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Histogram processing}

\subsubsection{Histogram equalization}
The cumulative probability density (CPD) of an image is
\begin{equation}
    F(r) = \int_0^r p_r (w) \dif w ,
\end{equation}
which leads to the transformation
\begin{equation}
    s = T(r) = (L-1) F(r).
\end{equation}
$p_r(w)$ is the probability for a pixel to be of intensity $w$. This transformation makes the histogram flat/uniform, which improves contrast by utilizing darks and lights equally much, rather than compressing the image into midtones.

\subsubsection{Histogram matching}
Sometimes you don't want a flat histogram, but a specific, nonflat distribution. Histogram matching is the method of transforming an image so that its histogram matches any given distribution.

\subsubsection{Local histogram processing}
The previous two methods are global, but sometimes a local method is better. This is done by iterating through pixels, looking at and changing the histogram of its neighborhood, and updating the pixel value according to the altered local histogram. An example is histogram equalization for a small moving window.


%================================================
\section{Breast Cancer Dataset}
%================================================
	%----------------------------------------------
	\subsection{Problem and Approach}
	%----------------------------------------------
		\begin{frame}
		\frametitle{Breast Cancer Data Classification}

		\begin{itemize}
			\item Small dataset (569 points with 30 features, 2 labels, nearly balanced).
			\item 30 features: numeric such as radius, texture, concavity, etc.
			\item 2 classes: benign and malignant
			\item Very simple and computationally easy, but with far more dynamic feature set.
			\item Analsis Tools Considered:
				\begin{itemize}
					\item PCA Clustering (qualitative)
					\item K Nearest Neighbor Classifier Accuracy (quantitative)
					\item K Nearest Neighbor Confusion Matrix (quantitative)
					\item Persistence Diagrams (qualitative)
					\item Bottleneck Distance Matrix (quantitative)
				\end{itemize}
		\end{itemize}

		\end{frame}

		%----------------------------------------------
	\subsection{Results}
	%----------------------------------------------

		\begin{frame}
		\frametitle{PCA Clustering}

		\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/cancer_PCA.png}
				\caption{PCA Clustering of the Breast Cancer Dataset. Visually, we can see that there is some separation, but the data lies close even after PCA.}
		\end{figure}

		\end{frame}

		\begin{frame}
			\frametitle{K Nearest Neighbors Classifier}

			\begin{itemize}
				\item Pre-processing:
				\begin{itemize}
					\item We split the data into a 30\% train and 70\% test sets and keep the data balanced.
					\item We also shuffle the data.
					\item We scale the data such that it is within $(0,1)$ using a min-max scaler.
				\end{itemize}
				\item The function class we consider is trivial since it is dependent only on the training data given.
				\item We can view this as a weighting function that weights the $k$-nearest points to the input with $\frac{1}{k}$ and all other points $0$.
				\item Classification Accuracy: 91 percent
				\item Training Accuracy: 94 percent
			\end{itemize}
		\end{frame}

		\begin{frame}
			\frametitle{Confusion Matrix for Cancer Data}

			\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/confusion_matrix_cancer.png}
				\caption{Confusion Matrix of Cancer Data. The data is pretty nicely classified in this binary case.}
		\end{figure}
		\end{frame}

		\begin{frame}
		\frametitle{Persistence Diagrams of Digits: Benign}

		\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/PD_H0_benign.png}
				\caption{Persistence Diagram for Benign.}
		\end{figure}
		\end{frame}

		\begin{frame}
		\frametitle{Persistence Diagrams of Digits: Malignant}

		\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/PD_H0_malignant.png}
				\caption{Persistence Diagram for Malignant.}
		\end{figure}
		\end{frame}

		\begin{frame}
		\frametitle{Persistence Images of Digits: Benign}

		\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/PI_H0_benign.png}
				\caption{Persistence Image for Benign.}
		\end{figure}
		\end{frame}

		\begin{frame}
		\frametitle{Persistence Images of Digits: Malignant}

		\begin{figure}
				\centering
				\includegraphics[scale=0.5]{images/PI_H0_malignant.png}
				\caption{Persistence Image for Malignant.}
		\end{figure}
		\end{frame}

		\begin{frame}
		\frametitle{Pairwise Bottleneck Distances}

		\begin{itemize}
			\item The bottleneck distances:
				\begin{itemize}
					\item With themselves: 0 (control)
					\item Benign, Malignant: 25.95355225
				\end{itemize}
		\end{itemize}

		A downside to the Bottleneck Distance as a measure is that quantitatively it can be hard to distinguish for simple data what a good distance is. In the binary case, unless you are comparing for hyperparameter search, this metric is close to useless.
		\end{frame}


\documentclass{article}
\usepackage[minionint,mathlf,textlf]{MinionPro} % To gussy up a bit
\usepackage[margin=1in]{geometry}
\usepackage{graphicx} % For .eps inclusion
%\usepackage{indentfirst} % Controls indentation
\usepackage[compact]{titlesec} % For regulating spacing before section titles
\usepackage{adjustbox} % For vertically-aligned side-by-side minipages
\usepackage{array, mathrsfs, mhchem, amsmath} % For centering of tabulars with text-wrapping columns
\usepackage{hyper ref}
\usepackage[autolinebreaks,framed,numbered]{mcode}
\pagenumbering{gobble}
\setlength\parindent{0 cm}
\begin{document}
\large

\section*{The probability distribution of a diffusing particle}
Let  $G(x,t)$ be the probability distribution of a certain particle, created at the origin at time zero, which diffuses in $n$ dimensions. According to Fick's second law of diffusion\footnote{Notice that Fick's second law is intended to describe the change in concentration profile for an ensemble of molecules, but we have applied it to the probability distribution of a single particle.}:
\[ \frac{\partial G(\mathbf{x},t)}{\partial t} = D \Delta G(\mathbf{x},t) + \delta(x,t) \]
Rearrange and perform a Fourier transform ($\mathbf{x} \mapsto \mathbf{k}$,t$\mapsto \omega$):
\begin{eqnarray}
\mathcal{L}G(\mathbf{x},t) = \left[ \frac{\partial}{\partial t} - D \Delta \right]G(\mathbf{x},t) & = & \delta(\mathbf{x},t) \nonumber \\
\left(i\omega + D|\mathbf{k}|^2 \right) \tilde{g}(\mathbf{k},\omega) & = & 1 \nonumber \\
\tilde{g}(\mathbf{k},\omega) & = & \frac{1}{i\omega + D|\mathbf{k}|^2} \nonumber
\end{eqnarray}
Now perform an inverse Fourier transform ($\omega \mapsto$t) using a contour integral on the upper half plane to enclose the singularity at $\omega=iD|\mathbf{k}|^2$. We apply the residue theorem to evaluate this integral:
\begin{eqnarray}
\tilde{g}(\mathbf{k},t) & = & \frac{1}{2\pi} \int_{-\infty}^{\infty} \frac{e^{it\omega}}{i\omega + D|\mathbf{k}|^2}\,d\omega \nonumber \\
& = & \frac{1}{2\pi} \oint_{UHP} \frac{e^{it\omega}}{i\omega + D|\mathbf{k}|^2}\,d\omega \nonumber \\
& = & \frac{2\pi i}{2\pi} \textrm{Res} \left( \frac{\omega - iD|\mathbf{k}|^2}{i\omega + D|\mathbf{k}|^2}e^{it\omega}, iD|\mathbf{k}|^2 \right) \nonumber \\
& = & \lim_{\omega \to iD|\mathbf{k}|^2} e^{it\omega} \nonumber \\
& = & e^{-Dt|\mathbf{k}|^2} \nonumber
\end{eqnarray}
Now we can perform another set of inverse Fourier transforms, completing the square to make a Gaussian function and shifting horizontally to simplify:

\begin{eqnarray}
G(\mathbf{x},t) & = & \frac{1}{(2\pi)^n} \int^{\infty}_{-\infty} \cdots \int^{\infty}_{-\infty} e^{i\mathbf{k} \cdot \mathbf{x}}e^{-Dt|\mathbf{k}|^2}\,dk_1 \cdots \,dk_n \nonumber \\
& = & \frac{e^{-|\mathbf{x}|^2/4Dt}}{(2\pi)^n} \int^{\infty}_{-\infty} \cdots \int^{\infty}_{-\infty} \exp\left(\sum_{j=1}^n -Dtk_j^2 + ik_jx_j +  \frac{x_j^2}{4Dt} \right) \,dk_1 \cdots \,dk_n \nonumber \\
& = & \frac{e^{\frac{-|\mathbf{x}|^2}{4Dt}}}{(2\pi)^n} \int^{\infty}_{-\infty} \cdots \int^{\infty}_{-\infty} \prod_{j=1}^n \exp \left(-Dtk_j^2 + ik_jx_j + \frac{x_j^2}{4Dt} \right) \,dk_1 \cdots \,dk_n \nonumber \\
& = & \frac{e^{\frac{-|\mathbf{x}|^2}{4Dt}}}{(2\pi)^n} \prod_{j=1}^n \int^{\infty}_{-\infty} \exp \left(-Dtk_j^2 + ik_jx_j + \frac{x_j^2}{4Dt} \right) \,dk_j \nonumber \\
& = & \frac{e^{\frac{-|\mathbf{x}|^2}{4Dt}}}{(2\pi)^n} \prod_{j=1}^n \int^{\infty}_{-\infty} \exp \left( \left(ik_j\sqrt{Dt} + \frac{x_j}{2\sqrt{Dt}}\right)^2 \right) \,dk_j \nonumber \\
& = & \frac{e^{\frac{-|\mathbf{x}|^2}{4Dt}}}{(2\pi)^n} \prod_{j=1}^n \int^{\infty}_{-\infty} e^{-Dtk_j^2} \,dk_j = \frac{e^{\frac{-|\mathbf{x}|^2}{4Dt}}}{(2\pi)^n} \prod_{j=1}^n \sqrt{\frac{\pi}{Dt}} = \left( \frac{\pi}{Dt} \right)^{n/2}{(2\pi)^n} e^{\frac{-|\mathbf{x}|^2}{4Dt}} \nonumber \\
& = & \frac{1}{\left(4\pi Dt\right)^{n/2}} \exp\left(-|\mathbf{x}|^2/4Dt\right) \nonumber
\end{eqnarray}
$G(\mathbf{x},t)$, the Green's function for a diffusing particle whose location at time $t=0$ is known with absolute accuracy, is a Gaussian function with mean zero. Its mean square displacement -- which in this case is also equal to the variance in position -- is:
\begin{eqnarray*}
\sigma^2 = \left< |\mathbf{x}|^2 \right> & = & \frac{1}{\left(4\pi Dt\right)^{n/2}} \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} |\mathbf{x}|^2 \exp\left(-|\mathbf{x}|^2/4Dt\right) \,dx_1 \cdots \,dx_n \\
& = & \frac{1}{\left(4\pi Dt\right)^{n/2}} \int_{-\infty}^{\infty} \cdots  \int_{-\infty}^{\infty} \exp\left(- \sum_{i=2}^{n} x_i^2/4Dt\right) \left[ \int_{-\infty}^{\infty} |\mathbf{x}|^2 e^{\frac{-x_1^2}{4Dt}} \,dx_1 \right]\, dx_2 \cdots \,dx_n
\end{eqnarray*}
Focus on how to evaluate the innermost interval, using Gaussian integral identities:
\begin{eqnarray*}
\int_{-\infty}^{\infty} |\mathbf{x}|^2 \exp \left( \frac{-x_1^2}{4Dt} \right) \,dx_1 & = & \int_{-\infty}^{\infty} x_1^2 \exp \left( \frac{-x_1^2}{4Dt} \right) \,dx_1 + \int_{-\infty}^{\infty} \left(\sum_{i=2}^{n} x_i^2\right) \exp \left( \frac{-x_1^2}{4Dt} \right) \,dx_1 \\
& = & 2Dt\sqrt{4\pi Dt} + \left(\sum_{i=2}^{n} x_i^2\right) \sqrt{4\pi Dt} \\
& = & \sqrt{4\pi Dt} \left[ 2Dt + \sum_{i=2}^{n} x_i^2 \right]
\end{eqnarray*}
Plugging this result into the calculation for mean square distance, we get:
\begin{eqnarray*}
\left< |\mathbf{x}|^2 \right>  & = & \frac{1}{\left(4\pi Dt\right)^{(n-1)/2}} \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} \exp\left(- \sum_{i=2}^{n} x_i^2/4Dt\right) \left[ 2Dt + \sum_{i=2}^{n} x_i^2 \right] \,dx_2 \cdots \,dx_n \\
& = & \frac{1}{\left(4\pi Dt\right)^{(n-2)/2}} \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} \exp\left(- \sum_{i=3}^{n} x_i^2/4Dt\right) \left[ 4Dt + \sum_{i=3}^{n} x_i^2 \right] \,dx_3 \cdots \,dx_n \\
& = & 2nDt
\end{eqnarray*}
\end{document}

lectures/Lecture 23 - Diffusion and Stochastic Differential Equations/greens function diffusion.tex


\chapter{Mathematical notation}

The table below gives a description of some of the commonly used symbols in this manual.
Note that some symbols are used to denote multiple concepts, and therefore there are some repeated
entries.

\begin{center}
\begin{longtable}{cl}
\hline\hline
Symbol & Definition\\
\hline\hline
\endfirsthead
%
\multicolumn{2}{c}{{\tablename} -- Continued} \\[0.5ex]
\hline\hline
Symbol & Definition\\
\hline\hline
\endhead
%This is the footer for all pages except the last page of the table...
  \\[0.5ex]
  \multicolumn{2}{l}{{Continued on Next Page\ldots}} \\
\endfoot
%This is the footer for the last page of the table...
  \hline
\endlastfoot
%
%
\multicolumn{2}{l}{{\bf Mathematical symbols}} \\ \hline
%
$\nabla$     & gradient operator \\
$\partial$   & domain boundary \\
\hline
%
\multicolumn{2}{l}{{\bf Greek}} \\ \hline
%
$\alpha$     & thermal expansion coefficient\\
$\beta$      & saline contraction coefficient\\
$\gamma$     & isothermal compressibility coefficient\\
$\eta$       & free surface height\\
$\eta_{\textrm{eq}}$ & equilibrium tide\\
$\kaptens$   & diffusivity tensor\\
$\lambda$    & longitude\\
$\mu$        & dynamic or molecular viscosity\\
$\nu$        & kinematic viscosity\\
$\phi$       & latitude\\
$\phi$       & test function\\
$\psi$       & trial function\\
$\rho$       & density\\
$\rho_0$     & constant background density\\
$\rho'$      & perturbation density\\
$\sigtens$   & stress tensor\\
$\bmtau$     & unit tangent vector\\
$\bmOmega$   & rotation vector\\
\hline
%
\multicolumn{2}{l}{{\bf Latin}} \\ \hline
%
$\vec{b}$      & buoyancy vector\\
$c_p$       & specific heat constant\\
$e$         & specific internal energy\\
$E$         & total specific energy\\
$f$         & Coriolis parameter\\
$\bmg$      & gravity vector\\
$\ktens$    & thermal conductivity tensor\\
$\mat{M}$   & mass matrix\\
$\metric$   & metric tensor\\
$\bmn$      & unit normal vector\\
$p$         & pressure\\
$p_a$       & air pressure\\
$p_h$       & hydrostatic pressure\\
$p'$        & perturbation pressure\\
$\bmq$      & thermal conduction\\
$R_E$       & radius of the Earth\\
$s$         & entropy\\
$S$         & salinity\\
$T$         & temperature\\
$\bmu$      & velocity vector\\
$w$         & enthalpy\\

\end{longtable}
\end{center}


\lab{Linear Systems}{Linear Systems}
\objective{The fundamental problem of linear algebra is solving the linear system $A\x = \b$, given that a solution exists.
There are many approaches to solving this problem, each with different pros and cons.
In this lab we implement the LU decomposition and use it to solve square linear systems.
We also introduce SciPy, together with its libraries for linear algebra and working with sparse matrices.
}

\section*{Gaussian Elimination} % =============================================

The standard approach for solving the linear system $A\x = \b$ on paper is reducing the augmented matrix $\left[A\mid\b\right]$ to row-echelon form (REF) via \emph{Gaussian elimination}, then using back substitution.
The matrix is in REF when the leading non-zero term in each row is the diagonal term, so the matrix is upper triangular.

At each step of Gaussian elimination, there are three possible operations: swapping two rows, multiplying one row by a scalar value, or adding a scalar multiple of one row to another.
Many systems, like the one displayed below, can be reduced to REF using only the third type of operation.
First, use multiples of the first row to get zeros below the diagonal in the first column, then use a multiple of the second row to get zeros below the diagonal in the second column.
%
\begin{align*}
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
1 & 4 & 2 & 3 \\
4 & 7 & 8 & 9 \\
\end{array}\right]
\longrightarrow
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
\textcolor{red}0 & 3 & 1 & 2 \\
4 & 7 & 8 & 9 \\
\end{array}\right]
\longrightarrow
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
0 & 3 & 1 & 2 \\
\textcolor{red}0 & 3 & 4 & 5 \\
\end{array}\right]
\longrightarrow
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
0 & 3 & 1 & 2 \\
0 & \textcolor{red}0 & 3 & 3
\end{array}\right]
\end{align*}

Each of these operations is equivalent to left-multiplying by a \emph{type III elementary matrix}, the identity with a single non-zero non-diagonal term.
If row operation $k$ corresponds to matrix $E_k$, the following equation is $E_3E_2E_1A = U$.
%
\begin{align*}
\left[\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & -1 & 1 \\
\end{array}\right]
\left[\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
-4 & 0 & 1 \\
\end{array}\right]
\left[\begin{array}{ccc}
1 & 0 & 0 \\
-1 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}\right]
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
1 & 4 & 2 & 3 \\
4 & 7 & 8 & 9 \\
\end{array}\right]
=
\left[\begin{array}{ccc|c}
1 & 1 & 1 & 1 \\
0 & 3 & 1 & 2 \\
0 & 0 & 3 & 3
\end{array}\right]
\end{align*}

However, matrix multiplication is an inefficient way to implement row reduction.
Instead, modify the matrix in place (without making a copy), changing only those entries that are affected by each row operation.

\begin{lstlisting}
>>> import numpy as np

>>> A = np.array([[1, 1, 1, 1],
...               [1, 4, 2, 3],
...               [4, 7, 8, 9]], dtype=np.<<float>>)

# Reduce the 0th column to zeros below the diagonal.
>>> A[1,0:] -= (A[1,0] / A[0,0]) * A[0]
>>> A[2,0:] -= (A[2,0] / A[0,0]) * A[0]

# Reduce the 1st column to zeros below the diagonal.
>>> A[2,1:] -= (A[2,1] / A[1,1]) * A[1,1:]
>>> print(A)
[[ 1.  1.  1.  1.]
 [ 0.  3.  1.  2.]
 [ 0.  0.  3.  3.]]
\end{lstlisting}

Note that the final row operation modifies only part of the third row to avoid spending the computation time of adding $0$ to $0$.

If a $0$ appears on the main diagonal during any part of row reduction, the approach given above tries to divide by $0$.
Swapping the current row with one below it that does not have a $0$ in the same column solves this problem.
This is equivalent to left-multiplying by a type II elementary matrix, also called a \emph{permutation matrix}.

\begin{warn} % Gaussian Elimination is numerically unstable!
Gaussian elimination is not always numerically stable.
In other words, it is susceptible to rounding error that may result in an incorrect final matrix.
Suppose that, due to roundoff error, the matrix $A$ has a very small entry on the diagonal.
\begin{align*}
A = \left[\begin{array}{cc}
10^{-15} & 1 \\
-1 & 0 \\
\end{array}\right]
\end{align*}
Though $10^{-15}$ is essentially zero, instead of swapping the first and second rows to put $A$ in REF, a computer might multiply the first row by $10^{15}$ and add it to the second row to eliminate the $-1$.
The resulting matrix is far from what it would be if the $10^{-15}$ were actually $0$.
\begin{align*}
\left[\begin{array}{cc}
10^{-15} & 1 \\
-1 & 0 \\
\end{array}\right]
\longrightarrow
\left[\begin{array}{cc}
10^{-15} & 1 \\
0 & 10^{15} \\
\end{array}\right]
\end{align*}

Round-off error can propagate through many steps in a calculation. %, resulting in garbage output.
The NumPy routines that employ row reduction use several tricks to minimize the impact of round-off error, but these tricks cannot fix every matrix.
\end{warn}

\begin{problem} % Program simple row reduction to REF.
Write a function that reduces an arbitrary square matrix $A$ to REF.
You may assume that $A$ is invertible and that a $0$ will never appear on the main diagonal (so only use type III row reductions, not type II).
Avoid operating on entries that you know will be $0$ before and after a row operation.
Use at most two nested loops.

Test your function with small test cases that you can check by hand.
Consider using \li{np.random.randint()} to generate a few manageable tests cases.
\label{prob:ref-row-reduction}
\end{problem}

\subsection*{The LU Decomposition} % ------------------------------------------

The \emph{LU decomposition} of a square matrix $A$ is a factorization $A=LU$ where $U$ is the \textbf{upper} triangular REF of $A$ and $L$ is the \textbf{lower} triangular product of the type III elementary matrices whose inverses reduce $A$ to $U$.
% Thus, the LU decomposition is an efficient way of storing the REF and how we got there.
The LU decomposition of $A$ exists when $A$ can be reduced to REF using only type III elementary matrices (without any row swaps).
However, the rows of $A$ can always be permuted in a way such that the decomposition exists.
If $P$ is a permutation matrix encoding the appropriate row swaps, then the decomposition $PA = LU$ always exists.

Suppose $A$ has an LU decomposition (not requiring row swaps).
Then $A$ can be reduced to REF with $k$ row operations, corresponding to left-multiplying the type III elementary matrices $E_1, \ldots, E_k$.
% Then $U = E_k \ldots E_2E_1A,$ where $U$ is the REF of $A$.
Because there were no row swaps, each $E_i$ is lower triangular, so each inverse $E_i^{-1}$ is also lower triangular.
Furthermore, since the product of lower triangular matrices is lower triangular, $L$ is lower triangular:
\begin{align*}
E_k\ldots E_2E_1 A = U\quad \longrightarrow\quad A &= (E_k \ldots E_2E_1)^{-1} U \\
&= E_1^{-1}E_2^{-1}\ldots E_k^{-1}U \\
&= LU.
\end{align*}

Thus, $L$ can be computed by right-multiplying the identity by the matrices used to reduce $U$.
However, in this special situation, each right-multiplication only changes one entry of $L$, matrix multiplication can be avoided altogether.
The entire process, only slightly different than row reduction, is summarized below. % in Algorithm \ref{alg:LU-Decomposition}.
%
\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{LU Decomposition}{$A$}
    \State $m, n \gets \shape{A}$
        \Comment{Store the dimensions of $A$.}
    \State $U \gets \makecopy{A}$
        \Comment{Make a copy of $A$ with \li{np.copy()}.}
    \State $L \gets I_m$
        \Comment{The $m\times m$ identity matrix.}
    \For{$j=0 \ldots n-1$}
        \For{$i=j+1 \ldots m-1$}
            \State $L_{i,j} \gets U_{i, j}/U_{j, j}$
            \State $U_{i,j:} \gets U_{i,j:} - L_{i,j}U_{j,j:}$
        \EndFor
    \EndFor
    \State \pseudoli{return} $L, U$
\EndProcedure
\end{algorithmic}
\caption{}
\label{alg:LU-Decomposition}
\end{algorithm}

\begin{problem}
Write a function that finds the LU decomposition of a square matrix.
You may assume that the decomposition exists and requires no row swaps.
\label{prob:LU-Decomposition}
\end{problem}

\subsection*{Forward and Backward Substitution} % -----------------------------

If $PA = LU$ and $A\x = \b$, then $LU\x = PA\x = P\b$.
This system can be solved by first solving $L\y = P\b$, then $U\x = \y$.
Since $L$ and $U$ are both triangular, these systems can be solved with backward and forward substitution.
We can thus compute the $LU$ factorization of $A$ once, then use substitution to efficiently solve $A\x=\b$ for various values of $\b$.

Since the diagonal entries of $L$ are all $1$, the triangular system $L\y = \b$ has the form
\begin{align*}
\left[\begin{array}{ccccc}
1      & 0      & 0      & \cdots & 0 \\
l_{21} & 1      & 0      & \cdots & 0 \\
l_{31} & l_{32} & 1      & \cdots & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
l_{n1} & l_{n2} & l_{n3} & \cdots & 1
\end{array}\right]
\left[\begin{array}{c}
y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_n \\
\end{array}\right]
=
\left[\begin{array}{c}
b_1 \\ b_2 \\ b_3 \\ \vdots \\ b_n \\
\end{array}\right].
\end{align*}
Matrix multiplication yields the equations
%
\begin{align}
\nonumber y_1 &= b_1, & y_1 &= b_1, \\
\nonumber l_{21}y_1 + y_2 &= b_2, & y_2 &= b_2 - l_{21}y_1, \\
% \nonumber l_{31}y_1 + l_{21}y_2 + y_3 &= b_3 & y_3 &= b_3 - l_{31}y_1 - l_{32}y_2 \\
\nonumber & \vdots & \vdots & \\
\sum_{j=1}^{k-1}l_{kj}y_j + y_k &= b_k, & y_k &= b_k - \sum_{j=1}^{k-1}l_{kj}y_j.
\label{eq:forward-substitution}
\end{align}
The triangular system $U\x = \y$ yields similar equations, but in reverse order:
\begin{align*}
\left[\begin{array}{ccccc}
u_{11} & u_{12} & u_{13} & \cdots & u_{1n} \\
0      & u_{22} & u_{23} & \cdots & u_{2n} \\
0      & 0      & u_{33} & \cdots & u_{3n} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0      & 0      & 0      & \cdots & u_{nn}
\end{array}\right]
\left[\begin{array}{c}
x_1 \\ x_2 \\ x_3 \\ \vdots \\ x_n \\
\end{array}\right]
=
\left[\begin{array}{c}
y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_n \\
\end{array}\right],
\end{align*}
\begin{align}
\nonumber u_{nn}x_n &= y_n, & x_n &= \frac{1}{u_{nn}}y_n, \\
\nonumber u_{n-1,n-1}x_{n-1} + u_{n-1,n}x_{n} &= y_{n-1}, & x_{n-1} &= \frac{1}{u_{n-1,n-1}}\left(y_{n-1} - u_{n-1,n}x_{n}\right),\\
\nonumber & \vdots & \vdots & \\
\sum_{j=k}^{n}u_{kj}x_j &= y_k, & x_k &= \frac{1}{u_{kk}}\left(y_k - \sum_{j=k+1}^{n}u_{kj}x_j\right).
\label{eq:backward-substitution}
\end{align}

\begin{problem} % Program back and forward substitution.
Write a function that, given $A$ and $\b$, solves the square linear system $A\x = \b$.
Use the function from Problem \ref{prob:LU-Decomposition} to compute $L$ and $U$, then use (\ref{eq:forward-substitution}) and (\ref{eq:backward-substitution}) to solve for $\y$, then $\x$.
You may again assume that no row swaps are required ($P = I$ in this case).
\label{prob:substitute-solve}
\end{problem}

\section*{SciPy} % ============================================================

SciPy \cite{scipy} is a powerful scientific computing library built upon NumPy.
It includes high-level tools for linear algebra, statistics, signal processing, integration, optimization, machine learning, and more.

SciPy is typically imported with the convention \li{import scipy as sp}.
However, SciPy is set up in a way that requires its submodules to be imported individually.%
\footnote{SciPy modules like \lif{linalg} are really \emph{packages}, which are not initialized when SciPy is imported alone.}

\begin{lstlisting}
>>> import scipy as sp
>>> hasattr(sp, "stats")            # The stats module isn't loaded yet.
<<False>>

>>> from scipy import stats         # Import stats explicitly. Access it
>>> hasattr(sp, "stats")            # with 'stats' or 'sp.stats'.
<<True>>
\end{lstlisting}

\subsection*{Linear Algebra} % ------------------------------------------------

NumPy and SciPy both have a linear algebra module, each called \li{linalg}, but SciPy's module is the larger of the two.
Some of SciPy's common \li{linalg} functions are listed below.
% See \url{http://docs.scipy.org/doc/scipy/reference/linalg.html} for more documentation.
%
\begin{table}[H]
\centering
\begin{tabular}{r|l}
    Function & Returns \\ \hline
    \li{det()} & The determinant of a square matrix. \\
    \li{eig()} & The eigenvalues and eigenvectors of a square matrix. \\
    \li{inv()} & The inverse of an invertible matrix. \\
    \li{norm()} & The norm of a vector or matrix norm of a matrix. \\
    \li{solve()} & The solution to $A\x = \b$ (the system need not be square).
\end{tabular}
\end{table}

This library also includes routines for computing matrix decompositions.

\begin{lstlisting}
>>> from scipy import linalg as la

# Make a random matrix and a random vector.
>>> A = np.random.random((1000,1000))
>>> b = np.random.random(1000)

# Compute the LU decomposition of A, including pivots.
>>> L, P = la.lu_factor(A)

# Use the LU decomposition to solve Ax = b.
>>> x = la.lu_solve((L,P), b)

# Check that the solution is legitimate.
>>> np.allclose(A @ x, b)
<<True>>
\end{lstlisting}

As with NumPy, SciPy's routines are all highly optimized.
However, some algorithms are, by nature, faster than others.

\begin{problem} % Time ways to solve Ax = b with scipy.linalg.
Write a function that times different \li{scipy.linalg} functions for solving square linear systems.

For various values of $n$, generate a random $n \times n$ matrix $A$ and a random $n$-vector $\b$ using \li{np.random.random()}.
Time how long it takes to solve the system $A\x = \b$ with each of the following approaches:
%
\begin{enumerate}
\item Invert $A$ with \li{la.inv()} and left-multiply the inverse to $\b$.
\item Use \li{la.solve()}.
\item Use \li{la.lu_factor()} and \li{la.lu_solve()} to solve the system with the LU decomposition.
\item Use \li{la.lu_factor()} and \li{la.lu_solve()}, but only time \li{la.lu_solve()} (not the time it takes to do the factorization with \li{la.lu_factor()}).
\end{enumerate}
%
Plot the system size $n$ versus the execution times.
Use log scales if needed.
\label{prob:linsystems-timing1}
\end{problem}

\begin{warn}
Problem \ref{prob:linsystems-timing1} demonstrates that computing a matrix inverse is computationally expensive.
In fact, numerically inverting matrices is so costly that there is hardly ever a good reason to do it.
Use a specific solver like \li{la.lu_solve()} whenever possible instead of using \li{la.inv()}.
\end{warn}

\subsection*{Sparse Matrices} % -----------------------------------------------

Large linear systems can have tens of thousands of entries.
Storing the corresponding matrices in memory can be difficult: a $10^{5} \times 10^{5}$ system requires around 40 GB to store in a NumPy array (4 bytes per entry $\times\ 10^{10}$ entries).
This is well beyond the amount of RAM in a normal laptop.

In applications where systems of this size arise, it is often the case that the system is \emph{sparse}, meaning that most of the entries of the matrix are $0$.
% Taking advantage of the sparse structure of these matrices uses less memory and decreases computation time.
SciPy's \li{sparse} module provides tools for efficiently constructing and manipulating 1- and 2-D sparse matrices.
A \li{sparse} matrix only stores the nonzero values and the positions of these values.
For sufficiently sparse matrices, storing the matrix as a \li{sparse} matrix may only take megabytes, rather than gigabytes.

For example, diagonal matrices are sparse.
Storing an $n \times n$ diagonal matrix in the na\"{i}ve way means storing $n^2$ values in memory.
It is more efficient to instead store the diagonal entries in a 1-D array of $n$ values.
In addition to using less storage space, this allows for much faster matrix operations: the standard algorithm to multiply a matrix by a diagonal matrix involves $n^3$ steps, but most of these are multiplying by or adding $0$.
A smarter algorithm can accomplish the same task much faster.

SciPy has seven sparse matrix types.
Each type is optimized either for storing sparse matrices whose nonzero entries follow certain patterns, or for performing certain computations.

\begin{table}[H]
\centering
\begin{tabular}{c|c|l}
    Name & Description & \multicolumn{1}{c}{Advantages}
    \\ \hline
    \li{bsr_matrix} & Block Sparse Row & Specialized structure. \\
    \li{coo_matrix} & Coordinate Format & Conversion among sparse formats. \\
    \li{csc_matrix} & Compressed Sparse Column & Column-based operations and slicing.\\
    \li{csr_matrix} & Compressed Sparse Row & Row-based operations and slicing. \\
    \li{dia_matrix} & Diagonal Storage & Specialized structure. \\
    \li{dok_matrix} & Dictionary of Keys & Element access, incremental construction. \\
    \li{lil_matrix} & Row-based Linked List & Incremental construction.
\end{tabular}
\end{table}

\subsubsection*{Creating Sparse Matrices} % -----------------------------------

A regular, non-sparse matrix is called \emph{full} or \emph{dense}.
Full matrices can be converted to each of the sparse matrix formats listed above.
However, it is more memory efficient to never create the full matrix in the first place.
There are three main approaches for creating sparse matrices from scratch.

\begin{itemize} % Sparse Construction Formats

\item \textbf{Coordinate Format}:
When all of the nonzero values and their positions are known, create the entire sparse matrix at once as a \li{coo_matrix}.
All nonzero values are stored as a coordinate and a value.
This format also converts quickly to other sparse matrix types.

\begin{lstlisting}
>>> from scipy import sparse

# Define the rows, columns, and values separately.
>>> rows = np.array([0, 1, 0])
>>> cols = np.array([0, 1, 1])
>>> vals = np.array([3, 5, 2])
>>> A = sparse.coo_matrix((vals, (rows,cols)), shape=(3,3))
>>> print(A)
  (0, 0)    3
  (1, 1)    5
  (0, 1)    2

# The toarray() method casts the sparse matrix as a NumPy array.
>>> print(A.toarray())              # Note that this method forfeits
[[3 2 0]                            # all sparsity-related optimizations.
 [0 5 0]
 [0 0 0]]
\end{lstlisting}

\item \textbf{DOK and LIL Formats}:
If the matrix values and their locations are not known beforehand, construct the matrix incrementally with a \li{dok_matrix} or a \li{lil_matrix}.
Indicate the size of the matrix, then change individual values with regular slicing syntax.

\begin{lstlisting}
>>> B = sparse.lil_matrix((2,6))
>>> B[0,2] = 4
>>> B[1,3:] = 9

>>> print(B.toarray())
[[ 0.  0.  4.  0.  0.  0.]
 [ 0.  0.  0.  9.  9.  9.]]
\end{lstlisting}

\item \textbf{DIA Format}: Use a \li{dia_matrix} to store matrices that have nonzero entries on only certain diagonals.
The function \li{sparse.diags()} is one convenient way to create a \li{dia_matrix} from scratch.
Additionally, every sparse matrix has a \li{setdiags()} method for modifying specified diagonals.
\begin{lstlisting}
# Use sparse.diags() to create a matrix with diagonal entries.
>>> diagonals = [[1,2],[3,4,5],[6]]     # List the diagonal entries.
>>> offsets = [-1,0,3]                  # Specify the diagonal they go on.
>>> print(sparse.diags(diagonals, offsets, shape=(3,4)).toarray())
[[ 3.  0.  0.  6.]
 [ 1.  4.  0.  0.]
 [ 0.  2.  5.  0.]]

# If all of the diagonals have the same entry, specify the entry alone.
>>> A = sparse.diags([1,3,6], offsets, shape=(3,4))
>>> print(A.toarray())
[[ 3.  0.  0.  6.]
 [ 1.  3.  0.  0.]
 [ 0.  1.  3.  0.]]

# Modify a diagonal with the setdiag() method.
>>> A.setdiag([4,4,4], 0)
>>> print(A.toarray())
[[ 4.  0.  0.  6.]
 [ 1.  4.  0.  0.]
 [ 0.  1.  4.  0.]]
\end{lstlisting}

\item \textbf{BSR Format}: Many sparse matrices can be formulated as block matrices, and a block matrix can be stored efficiently as a \li{bsr_matrix}.
Use \li{sparse.bmat()} or \li{sparse.block_diag()} to create a block matrix quickly.
\begin{lstlisting}
# Use sparse.bmat() to create a block matrix. Use 'None' for zero blocks.
>>> A = sparse.coo_matrix(np.ones((2,2)))
>>> B = sparse.coo_matrix(np.full((2,2), 2.))
>>> print(sparse.bmat([[  A , None,  A  ],
                       [None,  B  , None]], <<format>>='bsr').toarray())
[[ 1.  1.  0.  0.  1.  1.]
 [ 1.  1.  0.  0.  1.  1.]
 [ 0.  0.  2.  2.  0.  0.]
 [ 0.  0.  2.  2.  0.  0.]]

 # Use sparse.block_diag() to construct a block diagonal matrix.
 >>> print(sparse.block_diag((A,B)).toarray())
[[ 1.  1.  0.  0.]
 [ 1.  1.  0.  0.]
 [ 0.  0.  2.  2.]
 [ 0.  0.  2.  2.]]
 \end{lstlisting}
\end{itemize}

\begin{info} % plt.spy()
% One way to view a sparse matrix is to convert it to a NumPy array with the \li{toarray()} method.
If a sparse matrix is too large to fit in memory as an array, it can still be visualized with Matplotlib's \li{plt.spy()}, which colors in the locations of the non-zero entries of the matrix.
\begin{lstlisting}
>>> from matplotlib import pyplot as plt

# Construct and show a matrix with 50 2x3 diagonal blocks.
>>> B = sparse.coo_matrix([[1,3,5],[7,9,11]])
>>> A = sparse.block_diag([B]*50)
>>> plt.spy(A, markersize=1)
>>> plt.show()
\end{lstlisting}
%
\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/spydemo.pdf}
\end{figure}
\end{info}

% This problem will be used later in the Iterative Solvers lab.
\begin{problem} % Construct a large sparse matrix.
% Write a function that accepts an integer $n$.
Let $I$ be the $n\times n$ identity matrix, and define
\[
A =\left[\begin{array}{ccccc}
B & I &      &        & \\
I & B &  I   &        & \\
  & I & \ddots & \ddots & \\
  &   & \ddots & \ddots & I \\
  &   &        &    I   & B
\end{array}\right],
\qquad
B = \left[\begin{array}{rrrrr}
-4 &  1 &      &        & \\
 1 & -4 &  1   &        & \\
   &  1 & \ddots & \ddots & \\
   &    & \ddots & \ddots & 1 \\
   &    &        &    1   & -4
\end{array}\right],
\]
where $A$ is $n^2\times n^2$ and each block $B$ is $n\times n$.
The large matrix $A$ is used in finite difference methods for solving Laplace's equation in two dimensions, $\frac{\partial^2u}{\partial x^2} + \frac{\partial^2u}{\partial y^2} = 0$.

Write a function that accepts an integer $n$ and constructs and returns $A$ as a sparse matrix.
Use \li{plt.spy()} to check that your matrix has nonzero values in the correct places.
% \\ (Hint: Consider using \li{setdiag()} method of sparse matrices to fill in some of the diagonals.)
\label{prob:sparse-construction}
\end{problem}

\begin{comment}
\begin{info} % Note about banded matrices.
A \emph{banded} matrix is a square matrix whose only non-zero entries are on the main diagonal and on some diagonals on either side.
If the nonzero entries are confined to the three central diagonals, the matrix is also called \emph{tri-diagonal}.
Banded matrices arise naturally in many applications, including numerical methods for solving certain kinds of differential equations.
\end{info}
\end{comment}

\subsubsection*{Sparse Matrix Operations} % -----------------------------------

Once a sparse matrix has been constructed, it should be converted to a \li{csr_matrix} or a \li{csc_matrix} with the matrix's \li{tocsr()} or \li{tocsc()} method.
The CSR and CSC formats are optimized for row or column operations, respectively.
To choose the correct format to use, determine what direction the matrix will be traversed.% (row-wise or column-wise).

For example, in the matrix-matrix multiplication $AB$, $A$ is traversed row-wise, but $B$ is traversed column-wise.
Thus $A$ should be converted to a \li{csr_matrix} and $B$ should be converted to a \li{csc_matrix}.

\begin{lstlisting}
# Initialize a sparse matrix incrementally as a lil_matrix.
>>> A = sparse.lil_matrix((10000,10000))
>>> for k in range(10000):
...     A[np.random.randint(0,9999), np.random.randint(0,9999)] = k
...
>>> A
<<<10000x10000 sparse matrix of type '<type 'numpy.float64'>'
    with 9999 stored elements in LInked List format>>>

# Convert A to CSR and CSC formats to compute the matrix product AA.
>>> Acsr = A.tocsr()
>>> Acsc = A.tocsc()
>>> Acsr.dot(Acsc)
<<<10000x10000 sparse matrix of type '<type 'numpy.float64'>'
    with 10142 stored elements in Compressed Sparse Row format>>>
\end{lstlisting}

Beware that row-based operations on a \li{csc_matrix} are very slow, and similarly, column-based operations on a \li{csr_matrix} are very slow.

% SciPy's \li{sparse} methods work on NumPy arrays and dense matrices, but they take longer than using the usual methods for handling full matrices.

\begin{warn} % scipy.sparse matrices act differently than NumPy arrays.
Many familiar NumPy operations have analogous routines in the \li{sparse} module.
These methods take advantage of the sparse structure of the matrices and are, therefore, usually significantly faster.
However, SciPy's \li{sparse} matrices behave a little differently than NumPy arrays.

\begin{table}[H]
\centering
\begin{tabular}{r|c|c}
    Operation & \li{numpy} & \li{scipy.sparse}
    \\ \hline
    Component-wise Addition & \li{A + B} & \li{A + B} \\
    Scalar Multiplication & \li{2 * A} & \li{2 * A} \\
    Component-wise Multiplication  & \li{A * B} & \li{A.multiply(B)} \\
    Matrix Multiplication & \li{A.dot(B)}, \li{A @ B} & \li{A * B},\ \li{A.dot(B)}, \li{A @ B} \\
\end{tabular}
\end{table}

Note in particular the difference between \li{A * B} for NumPy arrays and SciPy sparse matrices.
Do \textbf{not} use \li{np.dot()} to try to multiply sparse matrices, as it may treat the inputs incorrectly.
The syntax \li{A.dot(B)} is safest in most cases.
\end{warn}

SciPy's sparse module has its own linear algebra library, \li{scipy.sparse.linalg}, designed for operating on sparse matrices.
Like other SciPy modules, it must be imported explicitly.

\begin{lstlisting}
>>> from scipy.sparse import linalg as spla
\end{lstlisting}

\begin{problem} % Time scipy.sparse.linalg.spsolve() against sp.linalg.solve().
Write a function that times regular and sparse linear system solvers.

For various values of $n$, generate the $n^{2} \times n^{2}$ matrix $A$ described in Problem \ref{prob:sparse-construction} and a random vector $\b$ with $n^2$ entries.
Time how long it takes to solve the system $A\x = \b$ with each of the following approaches:
%
\begin{enumerate}
\item Convert $A$ to CSR format and use \li{scipy.sparse.linalg.spsolve()} (\li{spla.spsolve()}).
% Do not include the time it takes to convert $A$ to CSR format.
\item Convert $A$ to a NumPy array and use \li{scipy.linalg.solve()} (\li{la.solve()}).
% Do not include the time it takes to convert $A$ to a NumPy array.
\end{enumerate}
In each experiment, only time how long it takes to solve the system (not how long it takes to convert $A$ to the appropriate format).

Plot the system size $n^{2}$ versus the execution times.
As always, use log scales where appropriate and use a legend to label each line.
\end{problem}

\begin{warn} % Don't try to invert sparse matrices.
Even though there are fast algorithms for solving certain sparse linear system, it is still very computationally difficult to invert sparse matrices.
In fact, the inverse of a sparse matrix is usually not sparse.
There is rarely a good reason to invert a matrix, sparse or dense.
\end{warn}

See \url{http://docs.scipy.org/doc/scipy/reference/sparse.html} for additional details on SciPy's \li{sparse} module.

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Improvements on the LU Decomposition} % --------------------------

\subsubsection*{Vectorization} % - - - - - - - - - - - - - - - - - - - - - - -

Algorithm \ref{alg:LU-Decomposition} uses two loops to compute the LU decomposition.
With a little vectorization, the process can be reduced to a single loop.

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{Fast LU Decomposition}{$A$}
    \State $m, n \gets \shape{A}$
    \State $U \gets \makecopy{A}$
    \State $L \gets I_m$
    \For{$k=0 \ldots n-1$}
        \State $L_{k+1:,k} \gets U_{k+1:,k}/U_{k,k}$
        \State $U_{k+1:,k:} \gets U_{k+1:,k:} - L_{k+1:,k}U_{k,k:}\trp$
        \label{state:outer-product}
    \EndFor
    \State \pseudoli{return} $L, U$
\EndProcedure
\end{algorithmic}
\caption{}
\end{algorithm}

Note that step \ref{state:outer-product} is an \emph{outer product}, not the regular dot product ($\x\y\trp$ instead of the usual $\x\trp\y$).
Use \li{np.outer()} instead of \li{np.dot()} or \li{@} to get the desired result.

\subsubsection*{Pivoting} % - - - - - - - - - - - - - - - - - - - - - - - - - -

Gaussian elimination iterates through the rows of a matrix, using the diagonal entry $x_{k,k}$ of the matrix at the $k$th iteration to zero out all of the entries in the column below $x_{k,k}$ ($x_{i,k}$ for $i\ge k$).
This diagonal entry is called the \emph{pivot}.
Unfortunately, Gaussian elimination, and hence the LU decomposition, can be very numerically unstable if at any step the pivot is a very small number.
Most professional row reduction algorithms avoid this problem via \emph{partial pivoting}.

The idea is to choose the largest number (in magnitude) possible to be the pivot by swapping the pivot row\footnote{\emph{Complete pivoting} involves row and column swaps, but doing both operations is usually considered overkill.}
 with another row before operating on the matrix.
For example, the second and fourth rows of the following matrix are exchanged so that the pivot is $-6$ instead of $2$.
%
\begin{align*}
\left[\begin{array}{cccc}
\times & \times & \times & \times\\
0 & 2 & \times & \times\\
0 & 4 & \times & \times\\
0 & \textcolor{blue}{-6} & \textcolor{blue}\times & \textcolor{blue}\times\\
\end{array}\right]
\longrightarrow
\left[\begin{array}{cccc}
\times & \times & \times & \times\\
0 & \textcolor{blue}{-6} & \textcolor{blue}\times & \textcolor{blue}\times\\
0 & 4 & \times & \times\\
0 & 2 & \times & \times\\
\end{array}\right]
\longrightarrow
\left[\begin{array}{cccc}
\times & \times & \times & \times\\
0 & \textcolor{blue}{-6} & \textcolor{blue}\times & \textcolor{blue}\times\\
0 & \textcolor{red}0 & \textcolor{red}\times & \textcolor{red}\times\\
0 & \textcolor{red}0 & \textcolor{red}\times & \textcolor{red}\times\\
\end{array}\right]
\end{align*}

A row swap is equivalent to left-multiplying by a type II elementary matrix, also called a \emph{permutation matrix}.
%
\begin{align*}
\left[\begin{array}{cccc}
1 & 0 & 0 & 0 \\
0 & 0 & 0 & 1 \\
0 & 0 & 1 & 0 \\
0 & 1 & 0 & 0 \\
\end{array}\right]
\left[\begin{array}{cccc}
\times & \times & \times & \times\\
0 & 2 & \times & \times\\
0 & 4 & \times & \times\\
0 & \textcolor{blue}{-6} & \textcolor{blue}\times & \textcolor{blue}\times\\
\end{array}\right]
=
\left[\begin{array}{cccc}
\times & \times & \times & \times\\
0 & \textcolor{blue}{-6} & \textcolor{blue}\times & \textcolor{blue}\times\\
0 & 4 & \times & \times\\
0 & 2 & \times & \times\\
\end{array}\right]
\end{align*}

For the LU decomposition, if the permutation matrix at step $k$ is $P_k$, then $P = P_k\ldots P_2P_1$ yields $PA = LU$.
The complete algorithm is given below.

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{LU Decomposition with Partial Pivoting}{$A$}
    \State $m, n \gets \shape{A}$
    \State $U \gets \makecopy{A}$
    \State $L \gets I_m$
    \State $P \gets [0,\ 1,\ \ldots,\ n-1]$
        \Comment{See tip \ref{tip:lu-decomposition-with-pivoting-P} below.}
    \For{$k=0 \ldots n-1$}
        \State Select $i \ge k$ that maximizes $|U_{i,k}|$
        \State $U_{k,k:} \leftrightarrow U_{i,k:}$
            \Comment{Swap the two rows.}
        \State $L_{k,:k} \leftrightarrow L_{i,:k}$
            \Comment{Swap the two rows.}
        \State $P_{k} \leftrightarrow P_{i}$
            \Comment{Swap the two entries.}
        \State $L_{k+1:,k} \gets U_{k+1:,k}/U_{k,k}$
        \State $U_{k+1:,k:} \gets U_{k+1:,k:} - L_{k+1:,k}U_{k,k:}\trp$
    \EndFor
    \State \pseudoli{return} $L, U, P$
\EndProcedure
\end{algorithmic}
\caption{}
\end{algorithm}

The following tips may be helpful for implementing this algorithm:
\begin{enumerate}
    \item Since NumPy arrays are mutable, use \li{np.copy()} to reassign the rows of an array simultaneously.
    \item Instead of storing $P$ as an $n \times n$ array, fancy indexing allows us to encode row swaps in a 1-D array of length $n$.
    Initialize $P$ as the array $[0, 1, \ldots, n]$.
    After performing a row swap on $A$, perform the same operations on $P$.
    Then the matrix product $PA$ will be the same as \li{A[P]}.
    \label{tip:lu-decomposition-with-pivoting-P}
\end{enumerate}

\begin{lstlisting}
>>> A = np.zeros(3) + np.vstack(np.arange(3))
>>> P = np.arange(3)
>>> print(A)
[[ 0.  0.  0.]
 [ 1.  1.  1.]
 [ 2.  2.  2.]]

# Swap rows 1 and 2.
>>> A[1], A[2] = np.copy(A[2]), np.copy(A[1])
>>> P[1], P[2] = P[2], P[1]
>>> print(A)                        # A with the new row arrangement.
[[ 0.  0.  0.]
 [ 2.  2.  2.]
 [ 1.  1.  1.]]
>>> print(P)                        # The permutation of the rows.
[0 2 1]
>>> print(A[P])                     # A with the original row arrangement.
[[ 0.  0.  0.]
 [ 1.  1.  1.]
 [ 2.  2.  2.]]
\end{lstlisting}

There are potential cases where even partial pivoting does not eliminate catastrophic numerical errors in Gaussian elimination, but the odds of having such an amazingly poor matrix are essentially zero.
The numerical analyst J.H. Wilkinson captured the likelihood of encountering such a matrix in a natural application when he said, ``Anyone that unlucky has already been run over by a bus!''

\subsubsection*{In Place} % - - - - - - - - - - - - - - - - - - - - - - - - - -

The LU decomposition can be performed in place (overwriting the original matrix $A$) by storing $U$ on and above the main diagonal of the array and storing $L$ below it.
The main diagonal of $L$ does not need to be stored since all of its entries are $1$.
This format saves an entire array of memory, and is how \li{scipy.linalg.lu_factor()} returns the factorization.

\subsection*{More Applications of the LU Decomposition} % ---------------------

The LU decomposition can also be used to compute inverses and determinants with relative efficiency.

\begin{itemize}
\item \textbf{Inverse}:
$(PA)^{-1} = (LU)^{-1} \ \Longrightarrow\ A^{-1}P^{-1} = U^{-1}L^{-1} \ \Longrightarrow\ LUA^{-1} = P$.
Solve $LU\a_i = \p_i$ with forward and backward substitution (as in Problem \ref{prob:substitute-solve}) for every column $\p_i$ of $P$.
Then
\begin{align*}
A^{-1} =
\left[\begin{array}{c|c|c|c}
&&&\\
\a_1&\a_2&\cdots&\a_n
\\&&&
\end{array}\right],
\end{align*}
the matrix where $\a_k$ is the $k$th column.
% Even this ``efficient'' method of inverting a matrix is computationally expensive.

\item \textbf{Determinant}: % \det(P^{-1})\det(L)\det(U) =
$\det(A) = \det(P^{-1}LU) = \frac{\det(L)\det(U)}{\det(P)}$.
The determinant of a triangular matrix is the product of its diagonal entries. Since every diagonal entry of $L$ is 1, $\det(L) = 1$.
Also, $P$ is just a row permutation of the identity matrix (which has determinant $1$), and a single row swap negates the determinant.
Then if $S$ is the number of row swaps, the determinant is
\[\det(A) = (-1)^S\prod_{i=1}^nu_{ii}.\]
\end{itemize}

\subsection*{The Cholesky Decomposition} % ------------------------------------

A square matrix $A$ is called \emph{positive definite} if $\z\trp A\z > 0$ for all nonzero vectors $\z$.
In addition, $A$ is called \emph{Hermitian} if $A = A\hrm = \overline{A\trp}$.
If $A$ is Hermitian positive definite, it has a \emph{Cholesky Decomposition} $A = U\hrm U$ where $U$ is upper triangular with real, positive entries on the diagonal.
This is the matrix equivalent to taking the square root of a positive real number.

The Cholesky decomposition takes advantage of the conjugate symmetry of $A$ to simultaneously reduce the columns \emph{and} rows of $A$ to zeros (except for the diagonal).
It thus requires only half of the calculations and memory of the LU decomposition.
Furthermore, the algorithm is \emph{numerically stable}, which means, roughly speaking, that round-off errors do not propagate throughout the computation.
% This decomposition is used when possible to solve least squares, optimization, and state estimation problems.

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{Cholesky Decomposition}{$A$}
\State $n,n \gets \shape{A}$
\State $U \gets$ \li{np.triu(}$A$\li{)}
    \Comment{Get the upper-triangular part of $A$.}
\For{$i=0 \ldots n-1$}
    \For{$j=i+1 \ldots n-1$}
        \State $U_{j,j:} \gets U_{j,j:} - U_{i,j:}\overline{U_{ij}}/U_{ii}$
    \EndFor
    \State $U_{i,i:} \gets U_{i,i:}/\sqrt{U_{ii}}$
\EndFor
\State \pseudoli{return} $U$
\EndProcedure
\end{algorithmic}
\caption{}
\end{algorithm}
%
As with the LU decomposition, SciPy's \li{linalg} module has optimized routines,\\\li{la.cho_factor()} and \li{la.cho_solve()}, for using the Cholesky decomposition.

\begin{comment} % This uses the decomposition A = LL^H.
The entries of $L$ are calculated as follows.
%
\begin{align*}
&L_{i,j} = \frac{1}{L_{j,j}}\left(A_{i,j} -\sum_{k=1}^{j-1}{L_{i,k}L_{j,k}}\right) \mbox{ for $i>j$} \\ \\
&L_{i,i} = \sqrt{A_{i,i} - \sum_{k=1}^{i-1}{L_{i,k}L_{i,k}}}.
\end{align*}

Notice that the entries of $L$ are defined recursively, with dependencies as diagrammed in Figure \ref{fig:cholesky-decomposition-order}.
Thus, an implementation of the Cholesky decomposition must compute the entries of $L$ in the correct order.

\begin{figure}[H]
\begin{tikzpicture}[red dot/.style={draw, circle, fill=red, red},
    norm/.style={draw=none}, xscale=1.5, yscale=1.5]

\begin{scope}[shift={(4,0)}]
\draw [-,ultra thick](-.2,0)--(-.2,2.5);
\draw [-,ultra thick](2.7,0)--(2.7,2.5);
\draw [-,ultra thick](-.2,0)--(0,0);
\draw [-,ultra thick](-.2,2.5)--(0,2.5);
\draw [-,ultra thick](2.7,2.5)--(2.5,2.5);
\draw [-,ultra thick](2.7,0)--(2.5,0);

\node[norm,black](bk1)at(.25,2.25){\LARGE \textbullet};
\node[norm,black](bk2)at(.75,1.75){\LARGE \textbullet};
\node[norm,black!25!](b1)at(1.25,1.25){\LARGE \textbullet};
\node[norm,black](bk3)at(1.75,.75){\LARGE \textbullet};
\node[norm, black](bk4)at(2.25,.25){\LARGE \textbullet};
\node[norm, black!25!](b2)at(.25,1.25){\LARGE \textbullet};
\node[norm,black!25!](b3)at(.75,1.25){\LARGE \textbullet};
\node[norm,black!25!](b4)at(.25,.25){\LARGE \textbullet};
\node[norm,black!25!](b3)at(.75,.25){\LARGE \textbullet};
\node[norm, shadecolor](r1)at(1.25,.25){\Huge \textbullet};
\end{scope}

\begin{scope}
\draw [-,ultra thick](-.2,0)--(-.2,2.5);
\draw [-,ultra thick](2.7,0)--(2.7,2.5);
\draw [-,ultra thick](-.2,0)--(0,0);
\draw [-,ultra thick](-.2,2.5)--(0,2.5);
\draw [-,ultra thick](2.7,2.5)--(2.5,2.5);
\draw [-,ultra thick](2.7,0)--(2.5,0);

\node[norm, shadecolor](r1)at(1.75,.75){\Huge \textbullet};
\node[norm,black](bk1)at(.25,2.25){\LARGE \textbullet};
\node[norm,black](bk2)at(1.25,1.25){\LARGE \textbullet};
\node[norm,black](bk3)at(.75,1.75){\LARGE \textbullet};
\node[norm,black](bk4)at(2.25,.25){\LARGE \textbullet};
\node[norm,black!25!](bk1)at(.25,.75){\LARGE \textbullet};
\node[norm,black!25!](bk1)at(.75,.75){\LARGE \textbullet};
\node[norm, black!25!](bk1)at(1.25,.75){\LARGE \textbullet};
\end{scope}
\end{tikzpicture}
\caption{The entries of $L$ in the Cholesky decomposition are defined recursively.
To calculate the green entry, you need to know each of the light gray entries.}
\label{fig:cholesky-decomposition-order}
\end{figure}
\end{comment}

% TODO: Thomas algorithm for solving tridiagonal systems?


\subsection{Randomness Analysis}
\label{sec:evaluation-randomness-analysis}
The randomness degree of the adopted pseudo-random number generator has been evaluated with the following standard tests:

\begin{itemize}
	\item \textbf{Spectral Test:} this test is considered one of the most intuitive and widely adopted tests to assess the quality of linear congruential generators \cite{knuth1981art}.
	It relies on the fact that the output of such generators form lines or hyperplanes when plotted on 2 or more dimensions. The less the distance between these lines or planes, the better is the generator.
	In fact, a smaller distance between lines or planes highlights a better uniform distribution.

	In Figure~\ref{fig:evaluation-randomness-spectral-16807},\ref{fig:evaluation-randomness-spectral-48271} and \ref{fig:evaluation-randomness-spectral-50812} we show the test results for generators $(16807,2^{31}-1)$, $(48271,2^{31}-1)$ and $(50812,2^{31}-1)$, respectively.

	The results show that the adopted generator $(50812,2^{31}-1)$ is much better than $(16807, 2^{31}-1)$, which has been widely adopted in the past, and it is really similar to $(48271,2^{31}-1)$, which is the current 32-bit de-facto standard, according to \cite{leemis2006discrete}.

	\item \textbf{Test of Extremes:} this test relies on the fact that if $U=U_{0},...,U_{d-1}$ is an independent identically distributed sequence of $Uniform(0,1)$ random variables, then $\max(U)^{d}$ is also a $Uniform(0,1)$. The test leverages this property to measures, for every stream, how much the generated random values differ from the theoretical uniform distribution.

	Given a number of streams $s$ and a level of confidence $c=1-\alpha$, the more the total number of fails is close to the expected value $s \cdot \alpha$, the better is the generator.

	In Figure~\ref{fig:evaluation-randomness-extremes-50812} we show the results for the adopted generator $(508012,2^{31}-1, 256)$ with sample size $n=10000$, $k=1000$ bins, sequence size $d=5$ and $95\%$ level of confidence.

	The adopted generator shows critical values $v_{min}=913$ and $v_{max}=1088$ and $14$ total fails ($7$ lower and $7$ upper fails), that is not far from the theoretical accepted number of fails, i.e. $256*0.05=13$.
	The proposed generator successfully passed the test with a $94.531\%$ level of confidence.

	\item \textbf{Kolmogorov-Smirnov Test:} the test measures, at a given level of confidence, the biggest vertical distance between the theoretical cumulative distribution function and the empirical cumulative distribution function.
	The more the recorded distance $d$ is less than the critical value $d^*$ for the considered level of confidence, the better the generator is.
	As the Kolmogorov-Smirnov analysis relies on pre-calculated randomness statistics, we have chosen to take into account the statistics obtained by the previous Test of Extremes.

	In Figure~\ref{fig:evaluation-randomness-kolmogorov-smirnov-50812} we show the test results for the proposed generator $(50812,2^{31}-1, 256)$ with a $95\%$ level of confidence.
	%
	The proposed generator successfully passed the test, as $d=0.041<0.081=d^*$.

\end{itemize}

We can conclude that the adopted generator $(50812,2^{31}-1, 256)$ provides a satisfying randomness degree.

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/evaluation-randomness-spectral-16807}
	\caption{The Spectral Test for the generator $(16807,2^{31}-1, 1)$ in the interval $(0, 10^{-3})$.}
	\label{fig:evaluation-randomness-spectral-16807}
\end{figure}

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/evaluation-randomness-spectral-48271}
	\caption{The Spectral Test for the generator $(48271,2^{31}-1, 1)$ in the interval $(0, 10^{-3})$.}
	\label{fig:evaluation-randomness-spectral-48271}
\end{figure}

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/evaluation-randomness-spectral-50812}
	\caption{The Spectral Test for the generator $(50812,2^{31}-1, 1)$ in the interval $(0, 10^{-3})$.}
	\label{fig:evaluation-randomness-spectral-50812}
\end{figure}

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/evaluation-randomness-extremes-50812}
	\caption{The Test of Extremes with $d=5$ to evaluate the randomness of the random number generator $(50812,2^{31}-1, 256)$.}
	\label{fig:evaluation-randomness-extremes-50812}
\end{figure}

\begin{figure}
	\includegraphics[width=\columnwidth]{fig/evaluation-randomness-kolmogorov-smirnov-50812}
	\caption{The Kolmogorov-Smirnov Analysis (leveraging the Test of Extremes with $d=5$) to evaluate the randomness of the random number generator $(50812,2^{31}-1, 256)$ with $95\%$ confidence.}
	\label{fig:evaluation-randomness-kolmogorov-smirnov-50812}
\end{figure}


\documentclass[../main.tex]{subfiles}

\begin{document}
        \section{Velocity, Acceleration, \& Frenet Frames}
        In previous lectures, we have introduced vectors \(\vec{T}\), \(\vec{N}\), and \(\vec{B}\) which are related to parameterized curves. We have also considered \(\vec{v}\) and \(\vec{a}\), which describe the velocity and acceleration of parameterized curves. We wish to find formulas relating these vectors. We also wish to find formulas for \(\vec{N}\) and \(\vec{B}\) which do not depend on arc length parameterization.

        We begin be considering the components of the acceleration vector \(\vec{a}(t)\). In particular, we wish to consider the components of the acceleration vector \emph{in the Frenet frame}, whose basis is \(\{\vec{T},\vec{N},\vec{B}\}\). In order to express \(\vec{a}\) in the Frenet frame, it is helpful to write
        \[
        \vec{v}=\vec{r}'(t)=|\vec{r}'|\vec{T}=v\vec{T}
        \]
        and
        \[
        \vec{a}=\vec{v}'(t)=\frac{d}{dt}v\vec{T}=\frac{dv}{dt}\vec{T}+v\frac{d\vec{T}}{dt}
        \]
        but we may use the definition of \(\vec{N}\) to express this as
        \[
        \vec{a}=\frac{dv}{dt}\vec{T}+v|\vec{T}'(t)|\vec{N}
        \]
        which may yet again be simplified using the definition of \(\kappa\) to yield
        \[
        \vec{a}=\frac{dv}{dt}\vec{T}+v^2\kappa\vec{N}
        \]
        which describes \(\vec{a}\) in the Frenet frame. Note that it has two main ``parts": there is the tangential component \(v'\vec{T}\), which describes the change in velocity parallel to the curve, and the normal component \(v^2\kappa\vec{N}\), which describes the change in direction of the curve, and is always directed towards the centre of curvature. Note that there is no \(\vec{B}\) component, so the acceleration vector is always parallel to the osculating plane.

        We also wish to consider another method of computing \(\vec{B}\) and \(\vec{N}\) which does not depend on finding the arc length parameterization. We already have
        \[
        \vec{a}=v'\vec{T}+v^2\kappa\vec{N}
        \]
        so we consider \(\vec{v}\times\vec{a}\). Since \(\vec{v}=v\vec{T}\), we have
        \[
        \vec{v}\times\vec{a}=v\vec{T}\times(v'\vec{T}+v^2\kappa\vec{N})=v^3\kappa\vec{B}
        \]
        which recovers a previously obtained formula for curvature, since if we take the absolute value of \(\vec{v}\times\vec{a}\), we obtain
        \[
        |\vec{v}\times\vec{a}|=|v^3\kappa\vec{B}|=v^3\kappa
        \]
        which yields
        \[
        \kappa=\frac{|\vec{v}\times\vec{a}|}{v^3}
        \]
        The same expression may also be used to obtain an expression for \(\vec{B}\), namely
        \[
        \vec{B}=\frac{\vec{v}\times\vec{a}}{v^3\kappa}=\frac{\vec{v}\times\vec{a}}{|\vec{v}\times\vec{a}|}
        \]
        which yields an expression for \(\vec{B}\) which is independent of parameterization. Since we already have an expression for \(\vec{T}\) which is independent of parameterization, we may obtain one for \(\vec{N}\) using the cross product.

        There is also another method for computing the components of the acceleration vector. Since
        \[
        \vec{a}=a_T\vec{T}+a_N\vec{N}
        \]
        we may find the components of \(\vec{a}\) by considering the projections of \(\vec{v}\) on \(\vec{T}\) and \(\vec{N}\), or formally
        \[
        a_T=\vec{a}\cdot\vec{T}=\frac{\vec{a}\cdot\vec{v}}{|\vec{v}|}
        \]
        and
        \[
        a_N=\vec{a}\cdot\vec{N}=\sqrt{a^2-a_T^2}
        \]

        \begin{example}{}{}
                Find \(\vec{T}\), \(\vec{N}\), \(\vec{B}\), and \(\kappa\) for the curve parameterized by \(\vec{r}=2t\i+t^2\j+\frac{t^3}{3}\k\).
                \tcblower
                We have
                \[
                \vec{v}=2\i+2t\j+t^2\k
                \]
                and as a result
                \[
                v=t^2+2
                \]

                We also have
                \[
                \vec{a}=2\j+2t\k
                \]
                using which we may compute
                \[
                \vec{v}\times\vec{a}=2t^2\i-4t\j+4\k
                \]
                and
                \[
                |\vec{v}\times\vec{a}|=2(t^2+2)
                \]

                Using these, we may compute the desired quantities. First,
                \[
                \kappa=\frac{|\vec{v}\times\vec{a}|}{v^3}=\frac{2(t^2+2)}{(t^2+2)^3}=\frac{2}{(t^2+2)^2}
                \]
                then
                \[
                \vec{T}=\frac{1}{t^2+2}(2\i+2t\j+t^2\k)
                \]
                as well as
                \[
                \vec{B}=\frac{\vec{v}\times\vec{a}}{|\vec{v}\times\vec{a}|}=\frac{1}{t^2+2}(t^2\i-2t\j+2\k)
                \]
                and finally
                \[
                \vec{N}=\vec{B}\times\vec{T}
                \]
                which is algebraically messy, and therefore the closed form is elided.
        \end{example}
        \begin{example}{}{}
                Consider the curve parameterized by \(\vec{r}=\cos{t}\i+t\j+t^2\k\). Find \(\kappa\), \(\vec{T}\), \(\vec{N}\), and \(\vec{B}\) at \(t=0\).
                \tcblower
                We have
                \[
                \vec{v}(0)=-\sin{0}\i+\j+2(0)\k=\j
                \]
                and as a result
                \[
                v=1
                \]

                We also have
                \[
                \vec{a}(0)=-\cos{t}\i+2\k
                \]
                using which we may compute
                \[
                \vec{v}\times\vec{a}=2\i+\k
                \]
                and
                \[
                |\vec{v}\times\vec{a}|=\sqrt{5}
                \]

                Using these, we may compute the desired quantities. First,
                \[
                \kappa=\frac{|\vec{v}\times\vec{a}|}{v^3}=\sqrt{5}
                \]
                then
                \[
                \vec{T}=\frac{\vec{v}}{v}=\j
                \]
                as well as
                \[
                \vec{B}=\frac{\vec{v}\times\vec{a}}{|\vec{v}\times\vec{a}|}=\frac{1}{\sqrt{5}}(2\i+\k)
                \]
                and finally
                \[
                \vec{N}=\vec{B}\times\vec{T}=\frac{1}{\sqrt{5}}(-\i+2\k)
                \]
        \end{example}
\end{document}


\subsection{Least squares approximation}

It should not be surprising to hear that many problems do not have a
perfect solution, and in these cases the objective is always to try to
do the best possible. For example what does one do if there are no
solutions to a system of linear equations $A\vect{x}=\vect{b}$? It
turns out that what we do is find $\vect{x}$ such that $A\vect{x}$ is
as close to $\vect{b}$ as possible. A very important technique that
follows from orthogonal projections is that of the least square
approximation\index{least square approximation}, and allows us to do
exactly that.

We begin with a lemma.

Recall that we can form the image of an $m \times n$-matrix $A$ by
$\func{im}(A) = \set{A\vect{x} \mid \vect{x} \in
\R^n }$. Rephrasing Theorem~\ref{thm:orthogonal-projection} using
the subspace $W=\func{im}(A)$  gives the
equivalence of an orthogonality condition with a minimization
condition\index{orthogonality and minimization}. The following picture
illustrates this orthogonality condition and geometric meaning of this
theorem.

\begin{center}
\begin{tikzpicture}
\draw[fill=lightgray](-3,0,2)--(-2,0.75,0)--(2,0.75,0)--(1,0,2)--(-3,0,2);
\draw(-2,0,0) rectangle (-1.8, 0.2,0);
\draw[thick,->](0,0,0)--(-2,0,0);
\draw[thick,->](0,0,0)--(-2,2,0);
\draw[red,thick,->](-2,2,0)--(-2,0,0);
\draw[thick,->](0,0,0)--(-0.5,0,1);
\draw[blue, thick, ->](-2,2,0)--(-0.5,0,1);
\node[right] at (0,0,0){$0$};
\node[above] at (-1,0,0){$\vect{z}=A\vect{x}$};
\node[above right] at (-1,1,0){$\vect{y}$};
\node[left] at (-2,1,0){$\vect{y}-\vect{z}$};
\node[right] at (-0.25,0,1){$\vect{u}$};
\node at (1,0.5,0){$A(\R^n)$};
\end{tikzpicture}
\end{center}

\begin{theorem}{Existence of minimizers}{existence-minimizer-hs}\label{existence-minimizer-hs}
Let $\vect{y}\in \R^{m}$ and let $A$ be an $m\times n$-matrix.

Choose $\vect{z}\in W= \func{im}(A)$ given by $\vect{z} =
\proj_{W}(\vect{y})$, and let $\vect{x} \in \R^{n}$ such that $\vect{z}=A\vect{x}$.

Then
\begin{enumerate}
\item $\vect{y} - A\vect{x} \in W^{\perp}$
\item $\norm{\vect{y} - A\vect{x}} < \norm{\vect{y} - \vect{u}}$ for all $\vect{u} \neq \vect{z} \in W$
\end{enumerate}
\end{theorem}

We note a simple but useful observation.

\begin{lemma}{Transpose and dot product}{transpose-and-dot-prod}
Let $A$ be an $m\times n$-matrix. Then
\begin{equation*}
A\vect{x} \dotprod \vect{y} = \vect{x}\dotprod A^T\vect{y}
\end{equation*}
\end{lemma}

\begin{proof}
This follows from the definitions:
\[ A\vect{x} \dotprod \vect{y}=\sum_{i,j}a_{ij}x_{j} y_{i}
=\sum_{i,j}x_{j} a_{ji} y_{i}= \vect{x} \dotprod A^T\vect{y}
\] \end{proof}

The next corollary gives the technique of least squares.

\begin{corollary}{Least squares and normal equation}{normal-equation}
A specific value of $\vect{x}$ which solves the problem of Theorem~\ref{thm:existenceminimizerhs} is obtained by solving the equation
\begin{equation*}
A^TA\vect{x}=A^T\vect{y}
\end{equation*}
Furthermore, there always exists a solution to this system of equations.
\end{corollary}

\begin{proof}
For $\vect{x}$ the minimizer of Theorem~\ref{thm:existenceminimizerhs}, $(
\vect{y}-A\vect{x}) \dotprod A \vect{u} =0$ for all $\vect{u} \in \R^{n}$ and from
Lemma~\ref{lem:transpose-and-dot-prod}, this is the same as saying
\begin{equation*}
A^T(\vect{y}-A\vect{x}) \dotprod \vect{u}=0
\end{equation*}
for all $\vect{u} \in \R^{n}$. This implies
\begin{equation*}
A^T\vect{y}-A^TA\vect{x}=\vect{0}.
\end{equation*}
Therefore, there is a solution to the equation of this corollary, and it
solves the minimization problem of Theorem~\ref{thm:existenceminimizerhs}.
\end{proof}

Note that $\vect{x}$ might not be unique but $A\vect{x}$, the closest
point of $A(\R^{n})$ to $\vect{y}$ is unique as was shown in the
above argument.

Consider the following example.

\begin{example}{Least squares solution to a system}{}
Find a least squares solution to the system
\begin{equation*}
\begin{mymatrix}{rr}
2 & 1 \\
-1 & 3 \\
4 & 5
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{c}
2 \\
1 \\
1
\end{mymatrix}
\end{equation*}
\end{example}

\begin{solution}
First, consider whether there exists a real solution. To do so, set up the augmented matrix given by
\begin{equation*}
\begin{mymatrix}{rr|r}
2 & 1 & 2 \\
-1 & 3 & 1 \\
4 & 5 & 1
\end{mymatrix}
\end{equation*}
The {\rref} of this augmented matrix is
\begin{equation*}
\begin{mymatrix}{rr|r}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix}
\end{equation*}

It follows that there is no real solution to this system. Therefore we wish to find the least squares solution. The normal equations are
\begin{eqnarray*}
A^T A \vect{x} &=& A^T \vect{y} \\
\begin{mymatrix}{rrr}
2 & -1 & 4 \\
1 & 3 & 5
\end{mymatrix} \begin{mymatrix}{rr}
2 & 1 \\
-1 & 3 \\
4 & 5
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} &=&\begin{mymatrix}{rrr}
2 & -1 & 4 \\
1 & 3 & 5
\end{mymatrix} \begin{mymatrix}{c}
2 \\
1 \\
1
\end{mymatrix}
\end{eqnarray*}
and so we need to solve the system
\begin{equation*}
\begin{mymatrix}{rr}
21 & 19 \\
19 & 35
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{r}
7 \\
10
\end{mymatrix}
\end{equation*}
This is a familiar exercise and the solution is
\begin{equation*}
\begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{c}
\vspace{0.05in}\frac{5}{34} \\
\vspace{0.05in}\frac{7}{34}
\end{mymatrix}
\end{equation*}
\end{solution}

Consider another example.

\begin{example}{Least squares solution to a system}{}
Find a least squares solution to the system
\begin{equation*}
\begin{mymatrix}{rr}
2 & 1 \\
-1 & 3 \\
4 & 5
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{c}
3 \\
2 \\
9
\end{mymatrix}
\end{equation*}
\end{example}

\begin{solution}
First, consider whether there exists a real solution. To do so, set up the augmented matrix given by
\begin{equation*}
\begin{mymatrix}{rr|r}
2 & 1 & 3 \\
-1 & 3 & 2 \\
4 & 5 & 9
\end{mymatrix}
\end{equation*}
The {\rref} of this augmented matrix is
\begin{equation*}
\begin{mymatrix}{rr|r}
1 & 0 & 1 \\
0 & 1 & 1 \\
0 & 0 & 0
\end{mymatrix}
\end{equation*}

It follows that the system has a solution given by $x=y=1$. However we can also use the normal equations and find
the least squares solution.
\begin{equation*}
\begin{mymatrix}{rrr}
2 & -1 & 4 \\
1 & 3 & 5
\end{mymatrix} \begin{mymatrix}{rr}
2 & 1 \\
-1 & 3 \\
4 & 5
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{rrr}
2 & -1 & 4 \\
1 & 3 & 5
\end{mymatrix} \begin{mymatrix}{r}
3 \\
2 \\
9
\end{mymatrix}
\end{equation*}
Then
\begin{equation*}
\begin{mymatrix}{rr}
21 & 19 \\
19 & 35
\end{mymatrix} \begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{c}
40 \\
54
\end{mymatrix}
\end{equation*}

The least squares solution is
\begin{equation*}
\begin{mymatrix}{c}
x \\
y
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
1
\end{mymatrix}
\end{equation*}
which is the same as the solution found above.
\end{solution}

An important application of Corollary~\ref{cor:normal-equation} is the
problem of finding the least squares regression
line\index{regression line} in statistics.  Suppose you are given points in the $xy$ plane
\begin{equation*}
\set{(x_{1},y_{1}),  (x_{2},y_{2}),\ldots,  (x_{n},y_{n})   }
\end{equation*}
and you would like to find constants $m$ and $b$ such that the line $\vect{y}=m\vect{x}+b$
goes through all these points. Of course this will be impossible in general.
Therefore, we try to find $m,b$ such that the line will be as close as possible. The desired system
is
\begin{equation*}
\begin{mymatrix}{c}
y_{1} \\
\vdots \\
y_{n}
\end{mymatrix} =\begin{mymatrix}{cc}
x_{1} & 1 \\
\vdots & \vdots \\
x_{n} & 1
\end{mymatrix} \begin{mymatrix}{c}
m \\
b
\end{mymatrix}
\end{equation*}
which is of the form $\vect{y}=A\vect{x}$. It is desired to choose $m$
and $b$ to make
\begin{equation*}
\norm{A\begin{mymatrix}{c}
m \\
b
\end{mymatrix} -\begin{mymatrix}{c}
y_{1} \\
\vdots \\
y_{n}
\end{mymatrix}} ^{2}
\end{equation*}
as small as possible. According to Theorem~\ref{thm:existenceminimizerhs} and
Corollary~\ref{cor:normal-equation}, the best values for $m$ and $b$ occur as the
solution to
\begin{equation*}
A^{T}A\begin{mymatrix}{c}
m \\
b
\end{mymatrix} =A^{T}\begin{mymatrix}{c}
y_{1} \\
\vdots \\
y_{n}
\end{mymatrix} ,\ \;\mbox{where}\; A=\begin{mymatrix}{cc}
x_{1} & 1 \\
\vdots & \vdots \\
x_{n} & 1
\end{mymatrix}
\end{equation*}
Thus, computing $A^{T}A$,
\begin{equation*}
\begin{mymatrix}{cc}
\sum_{i=1}^{n}x_{i}^{2} & \sum_{i=1}^{n}x_{i} \\
\sum_{i=1}^{n}x_{i} & n
\end{mymatrix} \begin{mymatrix}{c}
m \\
b
\end{mymatrix} =\begin{mymatrix}{c}
\sum_{i=1}^{n}x_{i}y_{i} \\
\sum_{i=1}^{n}y_{i}
\end{mymatrix}
\end{equation*}
Solving this system of equations for $m$ and $b$ (using Cramer's rule for example) yields:
\begin{equation*}
m=
\frac{-\paren{\sum_{i=1}^{n}x_{i}} \paren{\sum_{i=1}^{n}y_{i}}
+\paren{\sum_{i=1}^{n}x_{i}y_{i}} n}{\paren{
\sum_{i=1}^{n}x_{i}^{2}} n-\paren{\sum_{i=1}^{n}x_{i}} ^{2}}
\end{equation*}
and
\begin{equation*}
b=\frac{-\paren{\sum_{i=1}^{n}x_{i}} \sum_{i=1}^{n}x_{i}y_{i}+\paren{
\sum_{i=1}^{n}y_{i}} \sum_{i=1}^{n}x_{i}^{2}}{\paren{
\sum_{i=1}^{n}x_{i}^{2}} n-\paren{\sum_{i=1}^{n}x_{i}} ^{2}}.
\end{equation*}

Consider the following example.

\begin{example}{Least squares regression line}{least-squares-line}
Find the least squares regression line $\vect{y}=m\vect{x}+b$ for the following set of data points:
\[ \set{(0,1), (1,2), (2,2), (3,4), (4,5) } \]
\end{example}

\begin{solution}
In this case we have $n=5$ data points and we obtain:
\begin{equation*}
\begin{array}{ll}
\sum_{i=1}^{5}x_{i} = 10 & \sum_{i=1}^{5}y_{i} =  14 \\
\\
\sum_{i=1}^{5}x_{i}y_{i}  =  38 & \sum_{i=1}^{5}x_{i}^{2}  =   30\\
\end{array}
\end{equation*}
and hence
\begin{eqnarray*}
m &=& \frac{- 10 * 14 + 5*38}{5*30-10^2} = 1.00 \\
\\
b &=& \frac{- 10 * 38 + 14*30}{5*30-10^2} = 0.80
\end{eqnarray*}

The  least squares regression line for the set of data points is:
\[ \vect{y} = \vect{x}+0.8 \]

One could use this line to approximate other values for the data. For
example for $x=6$ one could use $y(6)=6+0.8=6.8$ as an approximate
value for the data.

The following diagram shows the data points and the corresponding regression line.

\begin{center}
\begin{tikzpicture}
\begin{axis}[
	xlabel=x,
	ylabel=y,
	xmin = -1,
	xmax = 5,
	ymin = -1,
	ymax = 6,
	axis lines = center,
	axis on top = true,
	domain = 0:5,
	legend style = {at={(1.03,-0.03)}},
	%legend style = {legend pos=south east}
	]

	\addplot [mark =none,color=blue, ultra thick] {x + 0.8};
	\addlegendentry{Regression Line}
	\addplot coordinates{
	(0,1)
	(1,2)
	(2,2)
	(3,4)
	(4,5)
	};
	\addlegendentry{Data Points}
\end{axis}
\end{tikzpicture}
\end{center}
\end{solution}

One could clearly do a least squares fit for curves of the form $
y=ax^{2}+bx+c$ in the same way. In this case you want to solve as well as
possible for $a,b$, and $c$ the system
\begin{equation*}
\begin{mymatrix}{ccc}
x_{1}^{2} & x_{1} & 1 \\
\vdots & \vdots & \vdots \\
x_{n}^{2} & x_{n} & 1
\end{mymatrix} \begin{mymatrix}{c}
a \\
b \\
c
\end{mymatrix} =\begin{mymatrix}{c}
y_{1} \\
\vdots \\
y_{n}
\end{mymatrix}
\end{equation*}
and one would use the same technique as above. Many other similar problems
are important, including many in higher dimensions and they are all solved
the same way.