\chapter{Vector spaces}
This is a pretty light chapter.
The point of it is to define what a vector space and a basis are.
These are intuitive concepts that you may already know.

\section{The definitions of a ring and field}
\prototype{$\ZZ$, $\RR$, and $\CC$ are rings; the latter two are fields.}

I'll very informally define a ring/field here,
in case you skipped the earlier chapter.
\begin{itemize}
	\ii A \textbf{ring} is a structure with a \emph{commutative}
	addition and multiplication, as well as subtraction, like $\ZZ$.
	It also has an additive identity $0$ and multiplicative identity $1$.

	\ii If the multiplication is invertible like in $\RR$ or $\CC$,
	(meaning $\frac 1x$ makes sense for any $x \neq 0$),
	then the ring is called a \textbf{field}.
\end{itemize}
In fact, if you replace ``field'' by ``$\RR$'' everywhere in what follows,
you probably won't lose much.
It's customary to use the letter $R$ for rings, and $k$ or $K$ for fields.

Finally, in case you skipped the chapter on groups, I should also mention:
\begin{itemize}
	\ii An \textbf{additive abelian group} is a structure
	with a commutative addition, as well as subtraction,
	plus an additive identity $0$.
	It doesn't have to have multiplication.
	A good example is $\RR^3$ (with addition componentwise).
\end{itemize}

\section{Modules and vector spaces}
\prototype{Polynomials of degree at most $n$.}
You intuitively know already that $\RR^n$ is a ``vector space'':
its elements can be added together,
and there's some scaling by real numbers.
Let's develop this more generally.

Fix a commutative ring $R$.
Then informally,
\begin{moral}
	An $R$-module is any structure where you can add two elements
	and scale by elements of $R$.
\end{moral}
% You can think of the $R$-module as consisting of soldiers
% being commanded by the ring $R$.
Moreover, a \vocab{vector space} is just a module whose commanding ring
is actually a field.
I'll give you the full definition in a moment,
but first, examples\dots

\begin{example}
	[Quadratic polynomials, aka my favorite example]
	My favorite example of an $\RR$-vector space is the
	set of polynomials of degree at most two, namely
	\[ \left\{ ax^2+bx+c \mid a,b,c \in \RR \right\}. \]
	Indeed, you can add any two quadratics, and multiply by constants.
	You can't multiply two quadratics to get a quadratic,
	but that's irrelevant -- in a vector space there need not
	be a notion of multiplying two vectors together.

	In a sense we'll define later, this vector space
	has dimension $3$ (as expected!).
	% But I hope you can see why this is kind of true!
	\label{example:quadratic_vector_space}
\end{example}
\begin{example}[All polynomials]
	The set of \emph{all} polynomials with real coefficients is an
	$\RR$-vector space, because you can \emph{add any two polynomials}
	and \emph{scale by constants}.
\end{example}

\begin{example}
	[Euclidean space]
	\listhack
	\begin{enumerate}[(a)]
		\ii The complex numbers
		\[ \left\{ a+bi \mid a,b \in \RR \right\} \]
		form a real vector space. As we'll see later,
		it has ``dimension $2$''.
		\ii The real numbers $\RR$ form a real vector space of dimension $1$.
		\ii The set of 3D vectors
		\[ \left\{ (x,y,z) \mid x,y,z \in \RR \right\} \]
		forms a real vector space, because you can add any two triples
		component-wise. Again, we'll later explain
		why it has ``dimension $3$''.
	\end{enumerate}
\end{example}

\begin{example}
	[More examples of vector spaces]
	\listhack
	\begin{enumerate}[(a)]
		\ii The set \[ \QQ[\sqrt 2] = \left\{ a + b \sqrt 2 \mid a, b \in \QQ \right\} \]
		has a structure of a $\QQ$-vector space in the obvious fashion:
		one can add any two elements, and scale by rational numbers.
		(It is not a real vector space -- why?)
		\ii The set \[ \left\{ (x,y,z) \mid x+y+z = 0 \text{ and } x,y,z \in \RR \right\} \]
		is a $2$-dimensional real vector space.
		\ii The set of all functions $f : \RR \to \RR$ is also a real vector space
		(since the notions $f+g$ and $c \cdot f$ both make sense for $c \in \RR$).
	\end{enumerate}
\end{example}

Now let me write the actual rules for how this multiplication behaves.
\begin{definition}
	Let $R$ be a commutative ring.
	An $R$-\vocab{module} starts with an additive abelian group $M = (M,+)$
	whose identity is denoted $0 = 0_M$.
	We additionally specify a left multiplication by elements of $R$.
	This multiplication must satisfy the following properties
	for $r, r_1, r_2 \in R$ and $m, m_1, m_2 \in M$:
	\begin{enumerate}[(i)]
		\ii $r_1 \cdot (r_2 \cdot m) = (r_1r_2) \cdot m$.
		\ii Multiplication is distributive, meaning
		\[ (r_1+r_2) \cdot m = r_1 \cdot m + r_2 \cdot m
			\text{ and }
			r \cdot (m_1 + m_2) = r \cdot m_1 + r \cdot m_2. \]
		\ii $1_R \cdot m = m$.
		\ii $0_R \cdot m = 0_M$.
		(This is actually extraneous;
		one can deduce it from the first three.)
	\end{enumerate}
	If $R$ is a field we say $M$ is an $R$-\vocab{vector space};
	its elements are called \vocab{vectors}
	and the members of $R$ are called \vocab{scalars}.
\end{definition}

\begin{abuse}
	In the above, we're using the same symbol $+$ for the addition of $M$
	and the addition of $R$.
	Sorry about that, but it's kind of hard to avoid, and the point
	of the axioms is that these additions should be related.
	I'll try to remember to put $r \cdot m$ for the multiplication of the module
	and $r_1r_2$ for the multiplication of $R$.
\end{abuse}

\begin{ques}
	In \Cref{example:quadratic_vector_space},
	I was careful to say ``degree at most $2$'' instead of ``degree $2$''.
	What's the reason for this?
	In other words, why is
	\[ \left\{ ax^2 + bx + c \mid a,b,c \in \RR, a \neq 0  \right\} \]
	not an $\RR$-vector space?
\end{ques}

A couple less intuitive but somewhat important examples\dots
\begin{example}[Abelian groups are $\ZZ$-modules]
	(Skip this example if you're not comfortable with groups.)
	\begin{enumerate}[(a)]
		\ii The example of real polynomials
		\[ \left\{ ax^2+bx+c \mid a,b,c \in \RR \right\} \]
		is also a $\ZZ$-module!
		Indeed, we can add any two such polynomials,
		and we can scale them by integers.
		\ii The set of integers modulo $100$, say $\ZZ/100\ZZ$,
		is a $\ZZ$-module as well. Can you see how?
		\ii In fact, \emph{any} abelian group $G = (G,+)$ is a $\ZZ$-module.
		The multiplication can be defined by
		\[ n \cdot g = \underbrace{g+\dots+g}_{\text{$n$ times}}
		\qquad (-n) \cdot g = n \cdot (-g)\]
		for $n \ge 0$. (Here $-g$ is the additive inverse of $g$.)
	\end{enumerate}
\end{example}
\begin{example}
	[Every ring is its own module]
	\listhack
	\begin{enumerate}[(a)]
	\ii $\RR$ can be thought of as an $\RR$-vector space over itself.
	Can you see why?

	\ii By the same reasoning,
	we see that \emph{any} commutative ring $R$ can be thought of
	as an $R$-module over itself.
	\end{enumerate}
\end{example}

\section{Direct sums}
\prototype{$\{ax^2+bx+c\} = \RR \oplus x\RR \oplus x^2\RR$, and
$\RR^3$ is the sum of its axes.}
Let's return to \Cref{example:quadratic_vector_space}, and consider
\[ V = \left\{ ax^2+bx+c \mid a,b,c \in \RR \right\}.  \]
Even though I haven't told you what a dimension is,
you can probably see that this vector space ``should have'' dimension $3$.
We'll get to that in a moment.

The other thing you may have noticed is that somehow
the $x^2$, $x$ and $1$ terms don't ``talk to each other''.
They're totally unrelated.
In other words, we can consider the three sets
\begin{align*}
	x^2\RR &\defeq \left\{ ax^2 \mid a \in \RR \right\} \\
	x\RR &\defeq \left\{ bx \mid b \in \RR \right\} \\
	\RR &\defeq \left\{ c \mid c \in \RR \right\}.
\end{align*}
In an obvious way, each of these can be thought of as a ``copy'' of $\RR$.

Then $V$ quite literally consists of the ``sums of these sets''.
Specifically, every element of $V$ can be written \emph{uniquely}
as the sum of one element from each of these sets.
This motivates us to write
\[ V = x^2\RR \oplus x\RR \oplus \RR. \]
The notion which captures this formally is the \vocab{direct sum}.

\begin{definition}
	Let $M$ be an $R$-module.
	Let $M_1$ and $M_2$ be subsets of $M$ which are themselves $R$-modules.
	Then we write $M = M_1 \oplus M_2$ and say $M$ is a \vocab{direct sum}
	of $M_1$ and $M_2$
	if every element from $M$ can be written uniquely as the sum
	of an element from $M_1$ and $M_2$.
\end{definition}
\begin{example}[Euclidean plane]
	Take the vector space $\RR^2 = \left\{ (x,y) \mid x \in \RR, y \in \RR \right\}$.
	We can consider it as a direct sum of its $x$-axis and $y$-axis:
	\[ X = \left\{ (x,0) \mid x \in \RR  \right\}
		\text{ and }
		Y = \left\{ (0,y) \mid y \in \RR \right\}. \]
	Then $\RR^2 = X \oplus Y$.
\end{example}

This gives us a ``top-down'' way to break down modules
into some disconnected components.

By applying this idea in reverse, we can also construct
new vector spaces as follows.
In a very unfortunate accident, the two names and notations for technically
distinct things are exactly the same.
\begin{definition}
	Let $M$ and $N$ be $R$-modules.
	We define the \vocab{direct sum} $M \oplus N$
	to be the $R$-module whose elements are pairs $(m,n) \in M \times N$.
	The operations are given by
	\[ (m_1, n_1) + (m_2, n_2) = (m_1+m_2, n_1+n_2). \]
	and
	\[ r \cdot (m, n) = (r \cdot m, r \cdot n). \]
\end{definition}

For example, while we technically wrote $\RR^2 = X \oplus Y$,
since each of $X$ and $Y$ is a copy of $\RR$,
we might as well have written $\RR^2 \cong \RR \oplus \RR$.

\begin{abuse}
	The above illustrates an abuse of notation in the way we write a direct sum. The symbol $\oplus$ has two meanings.
	\begin{itemize}
		\ii If $V$ is a \emph{given} space and $W_1$ and $W_2$ are subspaces, then $V = W_1 \oplus W_2$ means that ``$V$ \emph{splits} as a direct sum $W_1 \oplus W_2$'' in the way we defined above.
		\ii If $W_1$ and $W_2$ are two \emph{unrelated} spaces, then $W_1 \oplus W_2$ is \emph{defined} as the vector space whose \emph{elements} are pairs $(w_1, w_2) \in W_1 \times W_2$.
	\end{itemize}
	You can see that these definitions ``kind of'' coincide.
\end{abuse}

In this way, you can see that $V$ should be isomorphic
to $\RR \oplus \RR \oplus \RR$;
we had $V = x^2\RR \oplus x\RR \oplus \RR$,
but the $1$, $x$, $x^2$ don't really talk to each other
and each of the summands is really just a copy of $\RR$ at heart.

\begin{definition}
	We can also define, for every positive integer $n$, the module
	\[ M^{\oplus n}
		\defeq \underbrace{M \oplus M \oplus \dots \oplus M}_{\text{$n$ times}}. \]
\end{definition}

\section{Linear independence, spans, and basis}
\prototype{%
	$\left\{ 1,x,x^2 \right\}$ is a basis of
	$\left\{ ax^2 + bx + c \mid a,b,c \in \RR \right\}$.}

The idea of a basis, the topic of this section,
gives us another way to capture the notion that
\[ V = \left\{ ax^2+bx+c \mid a,b,c \in \RR \right\} \]
is sums of copies of $\{1,x,x^2\}$.
This section should be very intuitive, if technical.
If you can't see why the theorems here ``should'' be true,
you're doing it wrong.

Let $M$ be an $R$-module now.
We define three very classical notions that you likely are already familiar with.
If not, fall upon your notion of Euclidean space or $V$ above.
\begin{definition}
	A \vocab{linear combination} of some vectors $v_1, \dots, v_n$
	is a sum of the form $r_1 v_1 + \dots + r_n v_n$,
	where $r_1, \dots, r_n \in R$.
	The linear combination is called \vocab{trivial}
	if $r_1 = r_2 = \dots = r_n = 0_R$, and \vocab{nontrivial} otherwise.
\end{definition}
\begin{definition}
	Consider a finite set of vectors $v_1, \dots, v_n$ in a module $M$.
	\begin{itemize}
		\ii It is called \vocab{linearly independent} if there
		is no nontrivial linear combination with value $0_M$.
		(Observe that $0_M = 0 \cdot v_1 + 0 \cdot v_2 + \dots + 0 \cdot v_n$
		is always true -- the assertion is that there is no other
		way to express $0_M$ in this form.)
		\ii It is called a \vocab{generating set} if every $v \in M$ can be written as
		a linear combination of the $\{v_i\}$.
		If $M$ is a vector space we say it is \vocab{spanning} instead.
		\ii It is called a \vocab{basis} (plural \vocab{bases})
		if every $v \in M$ can be written
		\emph{uniquely} as a linear combination of the $\{v_i\}$.
	\end{itemize}
	The same definitions apply for an infinite set, with the proviso
	that all sums must be finite.

\end{definition}
So by definition, $\left\{ 1,x,x^2 \right\}$ is a basis for $V$.
It's not the only one: $\{2, x, x^2\}$ and $\{x+4, x-2, x^2+x\}$
are other examples of bases, though not as natural.
However, the set $S = \{3+x^2, x+1, 5+2x+x^2\}$ is not a basis;
it fails for two reasons:
\begin{itemize}
	\ii Note that
	$0 = (3+x^2) + 2(x+1) - (5+2x+x^2)$.
	So the set $S$ is not linearly independent.
	\ii It's not possible to write $x^2$ as a sum of elements of $S$.
	So $S$ fails to be spanning.
\end{itemize}
With these new terms, we can say a basis is a linearly independent and spanning set.

\begin{example}[More example of bases]
	\listhack
	\begin{enumerate}[(a)]
		\ii Regard $\QQ[\sqrt2]
		= \left\{ a + b \sqrt 2 \mid a, b \in \QQ \right\}$
		as a $\QQ$-vector space.
		Then $\{1, \sqrt 2\}$ is a basis.
		\ii If $V$ is the set of all real polynomials,
		there is an infinite basis $\{1, x, x^2, \dots\}$.
		The condition that we only use finitely many terms just says
		that the polynomials must have finite degree (which is good).
		\ii Let $V = \{ (x,y,z) \mid x+y+z=0 \text{ and } x,y,z \in \RR\}$.
		Then we expect there to be a basis of size $2$, but unlike previous examples
		there is no immediately ``obvious'' choice.
		Some working examples include:
		\begin{itemize}
			\ii $(1,-1,0)$ and $(1,0,-1)$,
			\ii $(0,1,-1)$ and $(1,0,-1)$,
			\ii $(5,3,-8)$ and $(2,-1,-1)$.
		\end{itemize}
	\end{enumerate}
\end{example}

\begin{exercise}
	Show that a set of vectors is a basis if and only if
	it is linearly independent and spanning.
	(Think about the polynomial example if you get stuck.)
\end{exercise}

Now we state a few results which assert
that bases in vector spaces behave as nicely as possible.
\begin{theorem}[Maximality and minimality of bases]
	\label{thm:vector_best}
	Let $V$ be a vector space over some field $k$
	and take $e_1, \dots, e_n \in V$. The following are equivalent:
	\begin{enumerate}[(a)]
		\ii The $e_i$ form a basis.
		\ii The $e_i$ are spanning, but no proper subset is spanning.
		\ii The $e_i$ are linearly independent, but adding any other
		element of $V$ makes them not linearly independent.
	\end{enumerate}
\end{theorem}
\begin{remark}
	If we replace $V$ by a general module $M$ over a commutative ring $R$,
	then (a) $\implies$ (b) and (a) $\implies$ (c) but not conversely.
\end{remark}
\begin{proof}
	Straightforward, do it yourself if you like.
	The key point to notice is that you need to divide by scalars for the converse direction,
	hence $V$ is required to be a vector space instead of just a module
	for the implications (b) $\implies$ (a) and (c) $\implies$ (a).
\end{proof}

\begin{theorem}
	[Dimension theorem for vector spaces]
	If a vector space $V$ has a finite basis,
	then every other basis has the same number of elements.
\end{theorem}
\begin{proof}
	We prove something stronger:
	Assume $v_1, \dots, v_n$ is a spanning set
	while $w_1, \dots, w_m$ is linearly independent. We claim that $n \ge m$.
	\begin{ques}
		Show that this claim is enough to imply the theorem.
	\end{ques}

	Let $A_0 = \{v_1, \dots, v_n\}$ be the spanning set.
	Throw in $w_1$: by the spanning condition,
	$w_1 = c_1 v_1 + \dots + c_n v_n$.
	There's some nonzero coefficient, say $c_n$.
	Thus \[ v_n = \frac{1}{c_n} w_1 - \frac{c_1}{c_n}v_1 - \frac{c_2}{c_n}v_2 - \dots. \]
	Thus $A_1 = \{v_1, \dots, v_{n-1}, w_1\}$ is spanning.
	Now do the same thing, throwing in $w_2$,
	and deleting some element of the $v_i$ as before to get $A_2$;
	the condition that the $w_i$ are linearly independent
	ensures that some $v_i$ coefficient
	must always not be zero.
	Since we can eventually get to $A_m$, we have $n \ge m$.
\end{proof}
\begin{remark}
	[Generalizations]
	\begin{itemize}
		\ii The theorem is true for an infinite basis as well
		if we interpret ``the number of elements'' as ``cardinality''.
		This is confusing on a first read through, so we won't elaborate.
		\ii In fact, this is true for modules over any commutative ring.
		Interestingly, the proof for the general case proceeds by reducing
		to the case of a vector space.
	\end{itemize}
\end{remark}

The dimension theorem, true to its name,
lets us define the \vocab{dimension} of
a vector space as the size of any finite basis, if one exists.
When it does exist we say $V$ is \vocab{finite-dimensional}.
So for example,
\[ V = \left\{ ax^2 + bx + c \mid a,b,c \in \RR \right\} \]
has dimension three, because $\left\{ 1,x,x^2 \right\}$ is a basis.
That's not the only basis: we could as well have written
\[ \left\{ a(x^2-4x) + b(x+2) + c \mid a,b,c \in \RR \right\} \]
and gotten the exact same vector space.
But the beauty of the theorem is that no matter how we try
to contrive the generating set, we always will get exactly three elements.
That's why it makes sense to say $V$ has dimension three.

On the other hand, the set of all polynomials
$\RR[x]$ is \emph{infinite-dimensional}
(which should be intuitively clear).

A basis $e_1, \dots, e_n$ of $V$ is really cool
because it means that to specify $v \in V$,
I only have to specify $a_1, \dots, a_n \in k$,
and then let $v = a_1 e_1 + \dots + a_n e_n$.
You can even think of $v$ as $\left( a_1, \dots, a_n \right)$.
% In a way I'll make precise in a moment, $V$ is actually isomorphic to just $k^{\oplus n}$.
To put it another way, if $V$ is a $k$-vector space we always have
\[ V = e_1 k \oplus e_2 k \oplus \dots \oplus e_n k. \]

\section{Linear maps}
\prototype{Evaluation of $\{ax^2+bx+c\}$ at $x=3$.}
We've seen homomorphisms and continuous maps.
Now we're about to see linear maps, the structure preserving maps
between vector spaces. Can you guess the definition?

\begin{definition}
	Let $V$ and $W$ be vector spaces over the same field $k$.
	A \vocab{linear map} is a map $T \colon V \to W$ such that:
	\begin{enumerate}[(i)]
		\ii We have $T(v_1 + v_2) = T(v_1) + T(v_2)$
		for any $v_1, v_2 \in V$.\footnote{In group language,
			$T$ is a homomorphism $(V,+) \to (W,+)$.}
		\ii For any $a \in k$ and $v \in V$, $T(a \cdot v) = a \cdot T(v)$.
	\end{enumerate}
	If this map is a bijection (equivalently, if it has an inverse),
	it is an \vocab{isomorphism}.
	We then say $V$ and $W$ are \vocab{isomorphic}
	vector spaces and write $V \cong W$.
\end{definition}

\begin{example}[Examples of linear maps]
	\listhack
	\begin{enumerate}[(a)]
		\ii For any vector spaces $V$ and $W$ there is a trivial linear map sending everything to $0_W \in W$.
		\ii For any vector space $V$, there is the identity isomorphism $\id : V \to V$.
		\ii The map $\RR^3 \to \RR$ by $(a,b,c) \mapsto 4a+2b+c$ is a linear map.
		\ii Let $V$ be the set of real polynomials of degree at most $2$.
		The map $\RR^3 \to V$ by $(a,b,c) \mapsto ax^2+bx+c$ is an \emph{isomorphism}.
		\ii Let $V$ be the set of real polynomials of degree at most $2$.
		The map $V \to \RR$ by $ax^2+bx+c \mapsto 9a + 3b + c$
		is a linear map, which can be described as ``evaluation at $3$''.
		\ii Let $W$ be the set of functions $\RR \to \RR$.
		The evaluation map $W \to \RR$ by $f \mapsto f(0)$ is a linear map.
		\ii There is a map of $\QQ$-vector spaces $\QQ[\sqrt2] \to \QQ[\sqrt2]$
		called ``multiply by $\sqrt2$''; this map sends $a+b\sqrt2 \mapsto 2b + a\sqrt2$.
		This map is an isomorphism, because it has an inverse ``multiply by $1/\sqrt2$''.
	\end{enumerate}
\end{example}

In the expression $T(a \cdot v) = a \cdot T(v)$, note that the first $\cdot$ is the multiplication of $V$ and the second $\cdot$ is the multiplication of $W$.
Note that this notion of isomorphism really only cares about the size of the basis:
\begin{proposition}[$n$-dimensional vector spaces are isomorphic]
	If $V$ is an $n$-dimensional vector space, then
	$V \cong k^{\oplus n}$.
\end{proposition}
\begin{ques}
	Let $e_1$, \dots, $e_n$ be a basis for $V$.
	What is the isomorphism?
	(Your first guess is probably right.)
\end{ques}
\begin{remark}
	You could technically say that all finite-dimensional vector
	spaces are just $k^{\oplus n}$ and that no other space is worth
	caring about.
	But this seems kind of rude.
	Spaces often are more than just triples: $ax^2+bx+c$ is a polynomial,
	and so it has some ``essence'' to it that you'd lose if you
	compressed it into $(a,b,c)$.

	Moreover, a lot of spaces, like the set of vectors $(x,y,z)$ with $x+y+z=0$,
	do not have an obvious choice of basis.
	Thus to cast such a space into $k^{\oplus n}$
	would require you to make arbitrary decisions.
\label{rem:vector_spaces_have_essence}
\end{remark}

\section{What is a matrix?}
Now I get to tell you what a matrix is:
it's a way of writing a linear map in terms of bases.

Suppose we have a finite-dimensional
vector space $V$ with basis $e_1, \dots, e_m$
and a vector space $W$ with basis $w_1, \dots, w_n$.
I also have a map $T \colon V \to W$ and I want to tell you what $T$ is.
It would be awfully inconsiderate of me to try and tell you what $T(v)$
is at every point $v$.
In fact, I only have to tell you what $T(e_1)$, \dots, $T(e_m)$ are,
because from there you can work out
$T(a_1 e_1 + \dots + a_m e_m)$ for yourself:
\[ T(a_1 e_1 + \dots + a_m e_m) = a_1 T(e_1) + \dots + a_m T(e_m). \]
Since the $e_i$ are a basis, that tells you all you need to know about $T$.
\begin{example}
	[Extending linear maps]
	Let $V = \left\{ ax^2+bx+c \mid a,b,c \in \RR \right\}$.
	Then $T(ax^2+bx+c) = aT(x^2) + bT(x) + cT(1)$.
\end{example}

Now I can even be more concrete.
I could tell you what $T(e_1)$ is, but seeing as I have a basis of $W$,
I can actually just tell you what $T(e_1)$ is in terms of this basis.
Specifically, there are unique $a_{11}, a_{21}, \dots, a_{n1} \in k$ such that
\[ T(e_1) = a_{11} w_1 + a_{21} w_2 + \dots + a_{n1} w_n. \]
So rather than telling you the value of $T(e_1)$ in some abstract space $W$,
I could just tell you what $a_{11}, a_{21}, \dots, a_{n1}$ were.
Then I'd repeat this for $T(e_2)$, $T(e_3)$, all the way up to $T(e_m)$,
and that would tell you everything you need to know about $T$.

That's where the matrix $T$ comes from!
It's a concise way of writing down all $mn$ numbers
I need to tell you.
To be explicit, the matrix for $T$ is defined as the array
\begin{align*}
	T &= \underbrace{%
	\begin{bmatrix}
		\mid & \mid & & \mid \\
		T(e_1) & T(e_2) & \dots & T(e_{m}) \\
		\mid & \mid & & \mid \\
	\end{bmatrix}
	}_{\text{$m$ columns}} \Bigg\} \text{$n$ rows} \\
	&=
	\begin{bmatrix}
		a_{11} & a_{12} & \dots & a_{1m} \\
		a_{21} & a_{22} & \dots & a_{2m} \\
		\vdots & \vdots & \ddots & \vdots \\
		a_{n1} & a_{n2} & \dots & a_{nm}
	\end{bmatrix}.
\end{align*}

\begin{example}
	[An example of a matrix]
	Here is a concrete example in terms of a basis.
	Let $V = \RR^3$ with basis $e_1$, $e_2$, $e_3$
	and let $W = \RR^2$ with basis $w_1$, $w_2$.
	If I have $T \colon V \to W$
	then uniquely determined by three values, for example:
	\begin{align*}
		T(e_1) &= 4w_1 + 7w_2 \\
		T(e_2) &= 2w_1 + 3w_2 \\
		T(e_3) &= w_1
	\end{align*}
	The columns then correspond to $T(e_1)$, $T(e_2)$, $T(e_3)$:
	\[
		T =
		\begin{bmatrix}
			4 & 2 & 1 \\
			7 & 3 & 0
		\end{bmatrix}
	\]
\end{example}

\begin{example}
	[An example of a matrix after choosing a basis]
	We again let $V = \left\{ ax^2 + bx + c \right\}$
	be the vector space of polynomials of degree at most $2$.
	We fix the basis $1$, $x$, $x^2$ for it.

	Consider the ``evaluation at $3$'' map,
	a map $V \to \RR$.
	We pick $1$ as the basis element of the RHS;
	then we can write it as a $1 \times 3$ matrix
	\[ \begin{bmatrix}
			1 & 3 & 9
		\end{bmatrix} \]
	with the columns corresponding to $T(1)$, $T(x)$, $T(x^2)$.
\end{example}

From here you can actually work out for yourself
what it means to multiply two matrices.
Suppose we have picked a basis for three spaces $U$, $V$, $W$.
Given maps $T \colon U \to V$ and $S \colon V \to W$,
we can consider their composition $S \circ T$, i.e.
\[ U \taking{T} V \taking{S} W. \]
Matrix multiplication is defined exactly so that the matrix $ST$
is the same thing we get from interpreting the composed function $S \circ T$ as a matrix.
\begin{exercise}
	Check this for yourself!
	For a concrete example let $\RR^2 \taking{T} \RR^2 \taking{S} \RR^2$
	by $T(e_1) = 2e_1+3e_2$ and $T(e_2) = 4e_1+5e_2$,
	$S(e_1) = 6e_1+7e_2$ and $S(e_2) = 8e_1+9e_2$.
	Compute $S(T(e_1))$ and $S(T(e_2))$ and see how it compares
	to multiplying the matrices associated to $S$ and $T$.
\end{exercise}
In particular, since function composition is associative,
it follows that matrix multiplication is as well.
To drive this point home,
\begin{moral}
	A matrix is the laziest possible way to specify
	a linear map from $V$ to $W$.
\end{moral}

This means you can define concepts like the determinant or the trace of a matrix
both in terms of an ``intrinsic'' map $T \colon V \to W$
and in terms of the entries of the matrix.
Since the map $T$ itself doesn't refer to any basis,
the abstract definition will imply that the numerical definition
doesn't depend on the choice of a basis.

\section{Subspaces and picking convenient bases}
\prototype{Any two linearly independent vectors in $\RR^3$.}
% A submodule is exactly what you think it is.
\begin{definition}
	Let $M$ be a left $R$-module.
	A \vocab{submodule} $N$ of $M$ is a module $N$
	such that every element of $N$ is also an element of $M$.
	If $M$ is a vector space then $N$ is called a \vocab{subspace}.
\end{definition}

\begin{example}[Kernels]
	The \vocab{kernel} of a map $T \colon V \to W$ (written $\ker T$) is
	the set of $v \in V$ such that $T(v) = 0_W$.
	It is a subspace of $V$, since it's closed under addition and scaling (why?).
\end{example}
\begin{example}[Spans]
	Let $V$ be a vector space and $v_1, \dots, v_m$ be any vectors of $V$.
	The \vocab{span} of these vectors is defined as the set
	\[ \left\{ a_1 v_1 + \dots + a_m v_m \mid a_1, \dots, a_m \in k \right\}. \]
	Note that it is a subspace of $V$ as well!
\end{example}
\begin{ques}
	Why is $0_V$ an element of each of the above examples?
	In general, why must any subspace contain $0_V$?
\end{ques}

Subspaces behave nicely with respect to bases.
\begin{theorem}[Basis completion]
	\label{thm:basis_completion}
	Let $V$ be an $n$-dimensional space, and $V'$ a subspace of $V$.
	Then
	\begin{enumerate}[(a)]
		\ii $V'$ is also finite-dimensional.
		\ii If $e_1, \dots, e_m$ is a basis of $V'$, then there exist
		$e_{m+1}, \dots, e_n$ in $V$ such that
		$e_1, \dots, e_n$ is a basis of $V$.
	\end{enumerate}
\end{theorem}
\begin{proof}
	Omitted, since it is intuitive and the proof is not that enlightening.
	(However, we will use this result repeatedly later on,
	so do take the time to internalize it now.)
\end{proof}

A very common use case is picking a convenient basis for a map $T$.
\begin{theorem}[Picking a basis for linear maps]
	\label{thm:linear_map_basis}
	Let $T \colon V \to W$ be a map of finite-dimensional vector spaces,
	with $n = \dim V$, $m = \dim W$.
	Then there exists a basis $v_1, \dots, v_n$ of $V$
	and a basis $w_1, \dots, w_m$ of $W$,
	as well as a nonnegative integer $k$, such that
	\[
		T(v_i) =
		\begin{cases}
			w_i & \text{if $i \le k$} \\
			0_W & \text{if $i > k$}.
		\end{cases}
	\]
	Moreover $\dim \ker T = n-k$ and $\dim T\im(V) = k$.
\end{theorem}
\begin{proof}[Sketch of Proof]
	You might like to try this one yourself before reading on:
	it's a repeated application of \Cref{thm:basis_completion}.

	Let $\ker T$ have dimension $n-k$.
	We can pick $v_{k+1}, \dots, v_{n}$ a basis of $\ker T$.
	Then extend it to a basis $v_1, \dots, v_n$ of $V$.
	The map $T$ is injective over the span of $v_1, \dots, v_k$
	(since only $0_V$ is in the kernel) so its images in $W$ are linearly independent.
	Setting $w_i = T(v_i)$ for each $i$,
	we get some linearly independent set in $W$.
	Then extend it again to a basis of $W$.
\end{proof}

This theorem is super important,
not only because of applications but also
because it will give you the right picture in your head
of how a linear map is supposed to look.
I'll even draw a cartoon of it to make sure you remember:

\begin{center}
\begin{asy}
	unitsize(0.7cm);
	real d = 3;

	filldraw(box( (-3*d/2,3.5), (-d/2,-4.5) ), opacity(0.1)+lightcyan, blue);
	filldraw(box( (d/2,3.5), (3*d/2,-5.5) ), opacity(0.1)+lightred, red);

	label(scale(1.5)*"$V$", (-d,4), blue);
	dot("$e_1$", (-d,3), dir(180), blue);
	dot("$e_2$", (-d,2), dir(180), blue);
	label("$\vdots$", (-d,1), blue);
	dot("$e_k$", (-d,0), dir(180), blue);
	dot("$e_{k+1}$", (-d,-1), dir(180), blue);
	dot("$e_{k+2}$", (-d,-2), dir(180), blue);
	label("$\vdots$", (-d,-3), dir(180), blue);
	dot("$e_n$", (-d,-4), dir(180), blue);

	label(scale(1.5)*"$W$", (d,4), red);
	dot("$f_1$", (d,3), dir(0), red);
	dot("$f_2$", (d,2), dir(0), red);
	label("$\vdots$", (d,1), red);
	dot("$f_k$", (d,0), dir(0), red);
	dot("$f_{k+1}$", (d,-1), dir(0), red);
	dot("$f_{k+2}$", (d,-2), dir(0), red);
	dot("$f_{k+3}$", (d,-3), dir(0), red);
	label("$\vdots$", (d,-4), dir(0), red);
	dot("$f_m$", (d,-5), dir(0), red);

	label("$T$", (0,3), dir(90));
	draw( (-d,3)--(d,3), EndArrow, Margin(3,3) );
	draw( (-d,2)--(d,2), EndArrow, Margin(3,3) );
	draw( (-d,0)--(d,0), EndArrow, Margin(3,3) );
	draw( (-d,-1)--(0,-1), EndArrow, Margin(3,3) );
	draw( (-d,-2)--(0,-2), EndArrow, Margin(3,3) );
	draw( (-d,-4)--(0,-4), EndArrow, Margin(3,3) );
	label("$0$", (0,-1));
	label("$0$", (0,-2));
	label("$0$", (0,-4));

	draw( (5.5,3)--(6,3)--(6,0)--(5.5,0));
	label("$\operatorname{im} T$", (6, 1.5), dir(0));
	draw( (-5.5,-1)--(-6,-1)--(-6,-4)--(-5.5,-4) );
	label("$\ker T$", (-6, -2.5), dir(180));
\end{asy}
\end{center}

In particular, for $T \colon V \to W$,
one can write $V = \ker T \oplus V'$,
so that $T$ annihilates its kernel while sending $V'$
to an isomorphic copy in $W$.

A corollary of this (which you should have expected anyways)
is the so called rank-nullity theorem,
which is the analog of the first isomorphism theorem.
\begin{theorem}
	[Rank-nullity theorem]
	\label{thm:rank_nullity}
	Let $V$ and $W$ be finite-dimensional vector spaces.
	If $T \colon V \to W$, then
	\[ \dim V = \dim \ker T + \dim \img T. \]
\end{theorem}
\begin{ques}
	Conclude the rank-nullity theorem from \Cref{thm:linear_map_basis}.
\end{ques}

\section{A cute application: Lagrange interpolation}
Here's a cute application\footnote{Source: Communicated to me
by Joe Harris at the first Harvard-MIT Undergraduate Math Symposium.}
of linear algebra to a theorem from high school.
\begin{theorem}
	[Lagrange interpolation]
	Let $x_1, \dots, x_{n+1}$ be distinct real numbers
	and $y_1, \dots, y_{n+1}$ any real numbers.
	Then there exists a \emph{unique}
	polynomial $P$ of degree at most $n$
	such that \[ P(x_i) = y_i \] for every $i$.
\end{theorem}
When $n = 1$ for example, this loosely
says there is a unique line joining two points.
\begin{proof}
	The idea is to consider the vector space $V$
	of polynomials with degree at most $n$,
	as well as the vector space $W = \RR^{n+1}$.
	\begin{ques}
		Check that $\dim V = n + 1 = \dim W$.
		This is easiest to do if you pick a basis for $V$,
		but you can then immediately forget about the basis
		once you finish this exercise.
	\end{ques}
	Then consider the linear map $T : V \to W$ given by
	\[ P \mapsto \left( P(x_1), \dots, P(x_{n+1}) \right). \]
	This is indeed a linear map because,
	well, $T(P+Q) = T(P)+T(Q)$ and $T(cP) = cT(P)$.
	It also happens to be injective: if $P \in \ker T$,
	then $P(x_1) = \dots = P(x_{n+1}) = 0$,
	but $\deg P \le n$ and so $P$ can only be the zero polynomial.

	So $T$ is an injective map between vector spaces of the same dimension.
	Thus it is actually a bijection, which is exactly what we wanted.
\end{proof}

\section{(Digression) Arrays of numbers are evil}
\label{sec:basis_evil}
As I'll stress repeatedly, a matrix represents a
\emph{linear map between two vector spaces}.
Writing it in the form of an $m \times n$ matrix
is merely a very convenient way to see the map concretely.
But it obfuscates the fact that this map is,
well, a map, not an array of numbers.

If you took high school precalculus, you'll see everything done in terms of matrices.
To any typical high school student, a matrix is an array of numbers.
No one is sure what exactly these numbers represent,
but they're told how to magically multiply these arrays to get more arrays.
They're told that the matrix
\[ \begin{bmatrix}
		1 & 0 & \dots & 0 \\
		0 & 1 & \dots & 0 \\
		\vdots & \vdots & \ddots & \vdots \\
		0 & 0 & \dots & 1 \\
	\end{bmatrix} \]
is an ``identity matrix'', because when you multiply
by another matrix it doesn't change.
Then they're told that the determinant is some magical combination of these
numbers formed by this weird multiplication rule.
No one knows what this determinant does,
other than the fact that $\det(AB) = \det A \det B$,
and something about areas and row operations and Cramer's rule.

Then you go into linear algebra in college, and you do more magic
with these arrays of numbers.
You're told that two matrices $T_1$ and $T_2$ are similar if
\[ T_2 = ST_1S\inv \] for some invertible matrix $S$.
You're told that the trace of a matrix $\Tr T$ is the sum of the diagonal entries.
Somehow this doesn't change if you look at a similar matrix,
but you're not sure why.
Then you define the characteristic polynomial as
\[ p_T(X) = \det (XI - T). \]
Somehow this also doesn't change if you take a similar matrix,
but now you really don't know why.
And then you have the Cayley-Hamilton theorem in all its black magic:
$p_T(T)$ is the zero map.  Out of curiosity you Google the proof,
and you find some ad-hoc procedure which still leaves you
with no idea why it's true.

This is terrible. What's so special about $T_2 = ST_1S\inv$?
Only if you know that the matrices are linear maps does this make sense:
$T_2$ is just $T_1$ rewritten with a different choice of basis.

I really want to push the opposite view.
Linear algebra is the study of \emph{linear maps},
but it is taught as the study of \emph{arrays of numbers},
and no one knows what these numbers mean.
And for a good reason: the numbers are meaningless.
They are a highly convenient way of encoding the matrix,
but they are not the main objects of study,
any more than the dates of events are the main objects of study in history.

The other huge downside is that people get the impression
that the only (real) vector space in existence is $\RR^{\oplus n}$.
As explained in \Cref{rem:vector_spaces_have_essence},
while you \emph{can} work this way if you're a soulless robot,
it's very unnatural for humans to do so.

When I took Math 55a as a freshman at Harvard,
I got the exact opposite treatment:
we did all of linear algebra without writing down a single matrix.
During all this time I was quite confused.
What's wrong with a basis?
I didn't appreciate until later that this approach was the
morally correct way to treat the subject: it made it clear what was happening.

Throughout the Napkin, I've tried to strike a balance between these
two approaches, using matrices when appropriate to illustrate
the maps and to simplify proofs, but ultimately writing
theorems and definitions in their \emph{morally correct} form.
I hope that this has both the advantage of giving the ``right'' definitions
while being concrete enough to be digested.
But I would like to say for the record that,
if I had to pick between the high school approach and the 55a approach,
I would pick 55a in a heartbeat.

\section{A word on general modules}
\prototype{$\ZZ[\sqrt2]$ is a $\ZZ$-module of rank two.}
I focused mostly on vector fields (aka modules over a field) in this chapter
for simplicity, so I want to make a few remarks about
modules over a general commutative ring $R$ before concluding.

Firstly, recall that for general modules,
we say ``generating set'' instead of ``spanning set''.
Shrug.

The main issue with rings is that our key theorem \Cref{thm:vector_best}
fails in spectacular ways.
For example, consider $\ZZ$ as a $\ZZ$-module over itself.
Then $\{2\}$ is linearly independent, but it cannot be extended to a basis.
Similarly, $\{2,3\}$ is spanning, but one cannot cut it down to a basis.
You can see why defining dimension is going to be difficult.

Nonetheless, there are still analogs of some of the definitions above.
\begin{definition}
	An $R$-module $M$ is called \vocab{finitely generated} if it has a finite generating set.
\end{definition}
\begin{definition}
	An $R$-module $M$ is called \vocab{free} if it has a basis.
	As said before, the analogue of the dimension theorem holds,
	and we use the word \vocab{rank} to denote the size of the basis.
	As before, there's an isomorphism $M \cong R^{\oplus n}$ where $n$ is the rank.
\end{definition}
\begin{example}[An example of a $\ZZ$-module]
	The $\ZZ$-module
	\[ \ZZ[\sqrt2] = \left\{ a + b\sqrt 2 \mid a,b \in \ZZ \right\} \]
	has a basis $\{1, \sqrt 2\}$, so we say it is
	a free $\ZZ$-module of rank $2$.
\end{example}

\begin{abuse}
	[Notation for groups]
	Recall that an abelian group
	can be viewed a $\ZZ$-module (and in fact vice-versa!),
	so we can (and will) apply these words to abelian groups.
	We'll use the notation $G \oplus H$ for two abelian groups $G$ and $H$
	for their Cartesian product, emphasizing the fact that $G$ and $H$ are abelian.
	This will happen when we study algebraic number theory and homology groups.
\end{abuse}

\section{\problemhead}
General hint:
\Cref{thm:linear_map_basis} will be your best friend
for many of these problems.

\begin{dproblem}
	Let $V$ and $W$ be finite-dimensional vector spaces
	with nonzero dimension, and consider linear maps $T \colon V \to W$.
	Complete the following table by writing
	``sometimes'', ``always'', or ``never'' for each entry.
	\begin{center}
	\begin{tabular}[h]{c|ccc}
		& $T$ injective & $T$ surjective & $T$ isomorphism \\ \hline
		If $\dim V > \dim W$\dots \\
		If $\dim V = \dim W$\dots \\
		If $\dim V < \dim W$\dots \\
	\end{tabular}
	\end{center}
	\begin{hint}
		Use the rank-nullity theorem.
		Also consider the zero map.
	\end{hint}
	\begin{sol}
		\begin{center}
		\begin{tabular}[h]{c|ccc}
		& $T$ injective & $T$ surjective & $T$ isomorphism \\ \hline
		If $\dim V > \dim W$\dots & never & sometimes & never\\
		If $\dim V = \dim W$\dots & sometimes & sometimes & sometimes \\
		If $\dim V < \dim W$\dots & sometimes & never & never \\
		\end{tabular}
		\end{center}
		Each ``never'' is by the rank-nullity theorem.
		Each counterexample is obtained by the zero map
		sending every element of $V$ to zero;
		this map is certainly neither injective or surjective.
	\end{sol}
\end{dproblem}

\begin{dproblem}
	[Equal dimension vector spaces are usually isomorphisms]
	\label{prob:equal_dimension}
	Let $V$ and $W$ be finite-dimensional vector
	spaces with $\dim V = \dim W$.
	Prove that for a map $T \colon V \to W$,
	the following are equivalent:
	\begin{itemize}
		\ii $T$ is injective,
		\ii $T$ is surjective,
		\ii $T$ is bijective.
	\end{itemize}
	\begin{sol}
		It essentially follows by \Cref{thm:linear_map_basis}.
	\end{sol}
\end{dproblem}

\begin{problem}
	[Multiplication by $\sqrt5$]
	Let $V = \QQ[\sqrt5] = \left\{ a + b \sqrt 5 \right\}$
	be a two-dimensional $\QQ$-vector space,
	and fix the basis $\{1, \sqrt 5\}$ for it.
	Write down the $2 \times 2$ matrix with rational coefficients
	that corresponds to multiplication by $\sqrt 5$.
	\begin{hint}
		$a + b \sqrt 5 \mapsto \sqrt 5 a + 5b$.
	\end{hint}
	\begin{sol}
		Since $1 \mapsto \sqrt5$ and $\sqrt5 \mapsto 5$,
		the matrix is
		$\begin{bmatrix}
			0 & 5 \\
			1 & 0
		\end{bmatrix}$.
	\end{sol}
\end{problem}

\begin{problem}
	[Multivariable Lagrange interpolation]
	Let $S \subset {\mathbb Z}^2$ be a set of $n$ lattice points.
	Prove that there exists a nonzero two-variable polynomial $p$
	with real coefficients, of degree at most $\sqrt{2n}$,
	such that $p(x,y) = 0$ for every $(x,y) \in S$.
\end{problem}

\begin{problem}
	[Putnam 2003]
	Do there exist polynomials $a(x)$, $b(x)$, $c(y)$, $d(y)$ such that
	\[ 1 + xy + (xy)^2 = a(x)c(y) + b(x)d(y) \]
	holds identically?
	\begin{hint}
		Plug in $y = -1, 0, 1$. Use dimensions of $\RR[x]$.
	\end{hint}
\end{problem}

\begin{problem}[TSTST 2014]
	\gim
	Let $P(x)$ and $Q(x)$ be arbitrary polynomials with real coefficients,
	and let $d$ be the degree of $P(x)$.
	Assume that $P(x)$ is not the zero polynomial.
	Prove that there exist polynomials $A(x)$ and $B(x)$ such that
	\begin{enumerate}[(i)]
		\ii Both $A$ and $B$ have degree at most $d/2$,
		\ii At most one of $A$ and $B$ is the zero polynomial,
		\ii $P$ divides $A+Q \cdot B$.
	\end{enumerate}
	\begin{hint}
		Interpret as $V \oplus V \to W$ for suitable $V$, $W$.
	\end{hint}
	\begin{sol}
		Let $V$ be the space of real polynomials with degree at most $d/2$
		(which has dimension $1 + \left\lfloor d/2 \right\rfloor$),
		and $W$ be the space of real polynomials modulo $P$ (which has dimension $d$).
		Then $\dim (V \oplus V) > \dim W$.
		So the linear map $V \oplus V \to W$ by $(A,B) \mapsto A + Q \cdot B$
		has a kernel of positive dimension
		(by rank-nullity, for example).
	\end{sol}
\end{problem}

\begin{sproblem}
	[Idempotents are projection maps]
	\label{prob:idempotent}
	Let $P \colon V \to V$ be a linear map,
	where $V$ is a vector space
	(not necessarily finite-dimensional).
	Suppose $P$ is \vocab{idempotent},
	meaning $P(P(v)) = P(v)$ for each $v \in V$,
	or equivalently $P$ is the identity on its image.
	Prove that \[ V = \ker P \oplus \img P. \]
	Thus we can think of $P$ as \emph{projection}
	onto the subspace $\img P$.
\end{sproblem}

\begin{sproblem}
	\label{prob:endomorphism_eventual_lemma}
	\gim
	Let $V$ be a finite dimensional vector space.
	Let $T \colon V \to V$ be a linear map,
	and let $T^n \colon V \to V$ denote $T$ applied $n$ times.
	Prove that there exists an integer $N$ such that
	\[ V = \ker T^N \oplus \img T^N. \]
	\begin{hint}
		Use the fact that the infinite chain of subspaces
		\[ \ker T \subseteq \ker T^2 \subseteq \ker T^3 \subseteq \dots \]
		and the similar chain for $\img T$ must eventually stabilize
		(for dimension reasons).
	\end{hint}
	\begin{sol}
		Consider
		\[
			\{0\} \subsetneq \ker S \subseteq \ker S^2 \subseteq \ker S^3 \subseteq \dots
			\text{  and  }
			V \supsetneq \img S \supseteq \img S^2 \supseteq \img S^3 \supseteq \dots.
		\]
		For dimension reasons, these subspaces must eventually stabilize:
		for some large integer $N$,
		$\ker T^N = \ker T^{N+1} = \dots$
		and $\img T^N = \img T^{N+1} = \img T^{N+2} = \dots$.
		When this happens, $\ker T^N \bigcap \img T^N = \{0\}$,
		since $T^N$ is an automorphism of $\img T^N$.
		On the other hand, by Rank-Nullity we also have
		$\dim \ker T^N + \dim \img T^n = \dim V$.
		Thus for dimension reasons, $V = \ker T^N \oplus \img T^N$.
	\end{sol}
\end{sproblem}

%Now consider the sequences


\documentclass{report}

\usepackage[latin1]{inputenc}
\usepackage[danish]{babel}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,bm,mathtools}
\usepackage{enumitem}
\usepackage{forest}
\usepackage[margin=1in]{geometry}

\begin{document}
\section*{AD - assignment 1}
Determine the asymptotic behavior of the following three CPSBC pricing scheme using the master theorem.% and assuming $p(1)=1$, and $p(2)=2$.
\subsection*{Task 1}
\begin{enumerate}
	\item $p(n) = 8 p(n/2) +n^2$
\end{enumerate}
For this recurrence, we have $a = 8, b = 2, f(n) = n^2$, and thus we have that $n^{\log_2(8)} = n^3 = \Theta(n^3)$. Since $f(n) = O(n^{3-\varepsilon})$, where $\varepsilon = 1$, we can apply case 1 of the master method. Hence
$$
p(n) = \Theta(n^3)
$$
%
\begin{enumerate}[resume]
	\item $p(n) = 8 p(n/4) +n^3$
\end{enumerate}
For this recurrence, we have $a = 8, b = 4, f(n) = n^3$, and thus we have that $n^{\log_4(8)} = n^{3/2} = \Theta(n^{3/2})$. Since $f(n) = \Omega(n^{3/2 + \varepsilon})$, where $\varepsilon = 3/2$ case 3 applies if we furthermore can show that $a f(n/b) \leq c f(n), \forall n>=n_0, c<1$. So
\begin{align*}
af(n/b) &= 8f(n/4) = 8\left(\frac{n}{4}\right)^3 = \frac{8}{4^3}n^3\\
				&\leq cn^3 = cf(n)
\end{align*}
where the inequality hold for all $n>1$ and for $c > 8/4^3$. Hence we can easily pick a $c<1$ and the regularity condition holds so
$$
p(n) = \Theta(n^3)
$$
\begin{enumerate}[resume]
	\item $p(n) = 10 p(n/9) +n\log_2n$
\end{enumerate}
For this recurrence, we have $a = 10, b = 9, f(n) = n\log_2n$. Since $\log(n) = o(n^k)$ for $k>0$ and $\log_9(10)>1$ it follows that $f(n) = O(t)$ we are in case 1 of the master theorem, hence
$$
p(n) = \Theta(n^{\log_9(10)})
$$
\subsection*{Task 2}
For the following price schemes use the substitution method.
\begin{enumerate}
	\item $p(n) = p(n/2) + p(n/3) + n$
\end{enumerate}
The recursion tree can be seen in Figure~\ref{fig: task2} for the first few levels. The longest path from the root to a leaf is to follow the left branch. $n \rightarrow n/2 \rightarrow n/2^2 \rightarrow \cdots \rightarrow 1$. Hence the height of the tree is a most $\lg(n)$. The work done at level $i$ is $(1/2 + 1/3)^i n = (5/6)^i n$ which is decreasing. If the tree was completely binary there would be at most $2^{\log_2(n)} = n$ leaves each with a cost of 1, hence all the leaves would at most be done in $O(n)$ time. We expect the solution to the recurrence to be at most the height of the tree times the work at the worst level in the tree. Hence we guess that $p(n) = O(n\log_2(n))$. To show it we use the substitution method (without showing the initial step). So assume $p(n) \leq c n \log_2(n) - n$
\begin{align*}
p(n) &= p(n/2) + p(n/3) +n \\
		 &=\leq c\frac{n}{2}\log_2(n/2) - \frac{n}{2} + c\frac{n}{3}\log_2(n/3) -\frac{n}{3} +n\\
		 &\leq c\frac{n}{2}\log_2(n/2) - n + c\frac{n}{3}\log_2(n/3) \\
		 &= (5/6)cn\log_2(n) - n - c\frac{n}{2}\log_2(2) - c\frac{n}{3}\log_2(3)\\
		 &\leq cn\log_2(n) - n
\end{align*}
where the inequalities hold for $c>0$. Hence
$$
p(n) = O(n\log_2(n))
$$

\begin{figure}[b]
\centering
\begin{forest}
[$n$
	[$\frac{n}{2}$
		[$\frac{n}{2^2}$
			[$\frac{n}{2^3}$
				[$\frac{n}{2^4}$]
				[$\frac{n}{2^3 3}$]
			]
			[$\frac{n}{2^2 3}$
				[$\frac{n}{2^3 3}$]
				[$\frac{n}{2^2 3^2}$]
			]
		]
		[$\frac{n}{2 \cdot 3}$
			[$\frac{n}{2^2 3}$
				[$\frac{n}{2^3 3}$]
				[$\frac{n}{2^2 3^2}$]
			]
			[$\frac{n}{2 \cdot 3^2}$
				[$\frac{n}{2^2 3^2}$]
				[$\frac{n}{2 \cdot 3^3}$]
			]
		]
	]
	[$\frac{n}{3}$
		[$\frac{n}{2 \cdot 3}$
			[$\frac{n}{2^2 3}$
				[$\frac{n}{2^3 3}$]
				[$\frac{n}{2^2 3^2}$]
			]
			[$\frac{n}{2 \cdot 3^2}$
				[$\frac{n}{2^2 3^2}$]
				[$\frac{n}{2 \cdot 3^3}$]
			]
		]
		[$\frac{n}{3^2}$
			[$\frac{n}{2 \cdot 3^2}$
				[$\frac{n}{2^2 3^2}$]
				[$\frac{n}{2 \cdot 3^3}$]
			]
			[$\frac{n}{3^3}$
				[$\frac{n}{2 \cdot 3^3}$]
				[$\frac{n}{3^4}$]
			]
		]
	]
]
\end{forest}
\caption{Three structure for the recurrence $p(n) = p(n/2) + p(n/3) + n$. The node indicates how much work need to be done on that node.}
\label{fig: task2}
\end{figure}

\begin{enumerate}[resume]
	\item $p(n) = \sqrt{n}p(\sqrt{n})+\sqrt{n}$
\end{enumerate}
When describing the recurrence tree that we use to create a guess for the asymptotic cost, we will neglect cases where $n$ is not a power of two. For level $i$ of the tree, the number of nodes will be the number of nodes in the previous levels time the square root of the current level. Hence the number of nodes on level $i$ is $\prod_{j=0}^(i-1) n^{1/2^(j+1)}$, the cost of each node in level $i$ of the tree is $n^{1/2^(i+1)}$. So the the total cost of each level $i$ in the tree is
\begin{align*}
\prod_{j=0}^{i-1} n^{1/2^{j+1}} n^{1/2^{i+1}} = \prod_{j=0}^{i} n^{1/2^{j+1}}
																							= \prod_{j=1}^{i+1} n^{(1/2)^j}
																							= n^{\sum_{j=1}^{i+1} (1/2)^j} \leq n
\end{align*}
So the cost of all nodes in a level is a most $n$. Our guess are again that the asymptotic behavior is a most the height of the tree times the cost of the worst level in the tree. So lets try to determine the height of the tree. Note that after each recursive call the square root is taken, hence after $k$ calls the size is $n^{1/2^k} = 2^{\log_2(n)/2^k}$. Solving this for a size 2, yields
\begin{align*}
2^{\log_2(n)/2^k} = 2
\shortintertext{taking $\log_2$}
\frac{\log_2(n)}{2^k} = 1
\shortintertext{multiply by $2^k$}
\log_2(n) = 2^k
\shortintertext{taking $\log_2$}
k = \log_2(\log_2(n))
\end{align*}
So our guess to use in the substitution method is that $p(n) \leq n\log_2\log_2(n) -d$. So
\begin{align*}
p(n) &= \sqrt{n}p(\sqrt{n})+\sqrt{n} \\
		 &\leq \sqrt{n}\left(\sqrt{n}\log_2\log_2(\sqrt{n}) - d\right)+\sqrt{n}\\
		 &\leq n\log_2\log_2(n) - d\sqrt{n} + \sqrt{n}
\end{align*}
The last line in the equation above, hold if there exist a lower order term $d>0$ where $d\sqrt{n} + \sqrt{n} = d$. Solving for $d$ yields $d = \frac{\sqrt{n}}{\sqrt{n} - 1}$ which is indeed a lower order term, larger than 0 from $n>2$. Hence
$$
p(n) = O(n\log_2\log_2(n))
$$
\end{document}


\documentclass[epsfig,10pt,fullpage]{article}

\newcommand{\LabNum}{4}
\newcommand{\CommonDocsPath}{../../common/docs}
\input{\CommonDocsPath/preamble.tex}

\begin{document}

\centerline{\huge OpenCL}
~\\
\centerline{\huge Laboratory Exercise \LabNum}
~\\
\centerline{\large Classification of Handwritten Digits using Machine Learning: Linear Classifier}
~\\

In this exercise you will learn about a machine learning approach to classifying handwritten digits.
You will create software and hardware implementations of a linear classifier that accepts images of handwritten digits ranging 0 to 9 and determines which digit is present in each image.

\section*{The MNIST Handwritten Digits Database}

In this exercise you will use the MNIST handwritten digits database,
which is widely used by scientists around the world to train and test handwritten digit recognition machines.
The database provides 28x28-pixel 8-bit grayscale images, each containing a single handwritten digit ranging 0 to 9. An example of
an image from the MNIST database is shown in Figure~\ref{fig:mnist_image}.

\begin{figure}[H]
   \begin{center}
       \hspace*{1.5cm}\includegraphics[scale=0.7]{figures/fig_mnist_image}
   \end{center}
   \caption{An example handwritten digit image from the MNIST database}
	\label{fig:mnist_image}
\end{figure}

The database contains a training set of 60000 images which are used to train the classifier, and a test set of 10000 images which are used to test the classifier's accuracy.
The 60000 training images have already been used to train the weights for the linear classifier.
Your task is to use the provided weights to implement the linear classifier and test its classification accuracy on the 10000 test images.

\section*{The Linear Classifier}

\begin{figure}[H]
   \begin{center}
       \includegraphics[scale=1]{figures/fig_linear_classifier_equation}
   \end{center}
   \caption{The equation for calculating the score of a class in the linear classifier}
	\label{fig:linear_classifier_equation}
\end{figure}

The linear classifier evaluates the likelihood (or score) of the input image being each of the possible digits.
The score is calculated using the equation shown in Figure~\ref{fig:linear_classifier_equation}, where j ranges 0 to 783 to span the pixels and their corresponding weights.
Each digit has its own set of weights leading to a total of 10*784 = 7840 weights for the classifier.
After calculating the scores of all the digits, the digit whose score is highest is chosen as the classification result.
An example of the calculations is shown in Figure~\ref{fig:linear_classifier}, where the linear classifier receives an input image containing the digit 3 and
outputs the correct classification result.

\begin{figure}[H]
   \begin{center}
       \includegraphics[scale=0.58]{figures/fig_linear_classifier}
   \end{center}
   \caption{The linear classifier operating on an input image containing the digit ``3'' and yielding the correct result}
	\label{fig:linear_classifier}
\end{figure}

\section*{Part I}
\noindent
Implement a program in the C++ language that performs linear classification of MNIST handwritten digits.
Your program must use the trained weights stored in files \textit{weights\_0}, \textit{weights\_1}, ... , \textit{weights\_9}, in the \textit{/design\_files/weights\_fp/} directory.
Test your program by classifying the 10000 MNIST test images stored in the file \textit{/design\_files/t10k-images-idx3-ubyte}.
Compare the results of your classifier to the correct labels stored in the file \textit{/design\_files/t10k-labels-idx1-ubyte} to calculate
your classifier's accuracy as: \[Accuracy = (\#\ of\ correct\ results )/ 10000 * 100\% \]
Compile and execute your program on a DE-Series Linux platform.
Upon completion, your program should output the classification accuracy and the runtime in milliseconds.

\subsubsection*{Parsing the MNIST Database Files}
The MNIST database is provided in .idx files. The file formats of the MNIST images and labels files are shown in Figure~\ref{fig:mnist_file_format}.
Your program must open and parse these files to extract the image pixels and labels. Each pixel is 8 bits and represents a grayscale value.
Each label is also 8 bits, and its value can range from 0 to 9 to indicate the correct classification of the corresponding image.
The 32-bit integers stored in these files are stored in most-significant byte (MSB)-first format, which
must first be converted to least significant byte (LSB)-first format before being used on the ARM processors of the DE-series boards.

\lstset{}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{12.7 cm}
\begin{lstlisting}
TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  10000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label
The labels values are 0 to 9.
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-.23in}\caption{File formats of the MNIST files. Source: http://yann.lecun.com/exdb/mnist/}
\label{fig:mnist_file_format}
\end{figure}

\subsubsection*{Parsing the Weights Files}

Trained weights for use in this part of the exercise are provided in the directory \textit{/design\_files/weights\_fp/}. There are 10 weights files named
\textit{weights\_0}, \textit{weights\_1}, ... , \textit{weights\_9} that correspond to their respective digits.
Each file contains 784 32-bit floating point weights that can be parsed using the code shown in
Figure~\ref{fig:parse_weights_code}.

\lstset{language=C++}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{16 cm}
\begin{lstlisting}
#define FEATURE_COUNT 784

// weights must be an array of sufficient size (>= 784)
bool read_weights_file(char *filename, float *weights) {
	FILE *f = fopen(filename, "rb");
	if (f == NULL){
		printf("ERROR: could not open %s\n",filename);
		return false;
	}
	int read_elements = fread(weights, sizeof(float), FEATURE_COUNT, f);
	fclose(f);

	if (read_elements != FEATURE_COUNT){
		printf("ERROR: read incorrect number of weights from %s\n", filename);
		return false;
	}
	return true;
}
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-0.33in}\caption{Function that parses a weights file}
\label{fig:parse_weights_code}
\end{figure}

\section*{Part II}

Implement an OpenCL kernel that performs linear classification of handwritten digits. Use the function prototype shown in Figure~\ref{fig:kernel_v1}.
The kernel reads the input array \texttt{images} which contains 10000 images. Each image in the array is 784 chars long, for a total of 10000 * 784 = 7840000 chars in the array.
The kernel reads the input array \texttt{weights} which contains 10 sets of 784 weights, for a total of 7840 floats in the array.
The kernel writes the classification results to the output array \texttt{results} which will contain 10000 chars when the kernel finishes.
Each char in the \texttt{results} array will contain a value from 0 to 9 indicating the classification result.

\lstset{}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{16 cm}
\vspace{-0.13in}\begin{lstlisting}
__attribute__((reqd_work_group_size(10000,1,1)))
__kernel void linear_classifier(global const unsigned char * restrict images,
								constant float * restrict weights,
								global unsigned char * restrict results)
{
	... Your Code ...
}
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-0.33in}\caption{Function prototype for the linear classifier kernel}
\label{fig:kernel_v1}
\end{figure}

In previous exercises you manually created local memories to cache frequently used values and reduce memory access overhead.
In this exercise, you will use Intel FPGA SDK for OpenCL's automatic cache generation feature to cache the linear classifier's weights.
This is done by assigning the \texttt{constant}
keyword to the weights array which tells the compiler that the values in the array are constant and should be cached to improve performance.
By default the Intel FPGA SDK for OpenCL creates a cache that is 16384 bytes large to store the values.
The cache size can be changed when compiling your kernel
using the argument \texttt{-const-cache-bytes=<N>}, where <N> is the number of bytes desired.
Select a cache size that is sufficiently large to minimize cache misses.

~\\
Maximize the performance of your design by unrolling your loop(s) until you run out of FPGA resources.
Modify the SW implementation from Part I to invoke your OpenCL kernel.
Ensure that your OpenCL kernel produces the same classification accuracy as your software implementation in Part I.
Determine the runtime of your OpenCL kernel. How does its performance compare to that of the software implementation?

\section*{Part III}

Open the ``Area Analysis of System'' page in the compilation report of your design in Part II. By examining the detailed breakdowns,
you will see that the majority of resource utilization comes from implementing floating-point arithmetic operations.
How many ALUTs, FFs, and DSPs does a floating-point add require? How many ALUTs, FFs, and DSPs does a floating-point multiply require?
How many floating-point add operations could you fit into your FPGA, assuming you are implementing nothing else?
How many floating-point multiply operations could you fit into your FPGA, assuming you are implementing nothing else?
Note that you can get the total number of available ALUTs, FFs, and DSPs on the ``Summary'' page of the report.

\section*{Part IV}

In this part of the exercise, you will modify your implementation from Part II to use fixed-point arithmetic instead of
floating-point arithmetic.

\subsubsection*{Fixed-Point Arithmetic}

The fixed-point datatype stores real numbers using a fixed number of digits after the decimal point.
Fixed-point data consists of an arbitrary number of bits which are divided into two sets.
The set of most significant bits represents the whole number to the left of the decimal point,
while the set of least significant bits represents the fractional portion to the right of the decimal point.
As an example, a 16-bit fixed-point number can be divided into a 12-bit set and an 4-bit set, where the most
significant 12 bits represent the whole number, and the least significant 4 bits represent the fractional number.
Such a fixed-point representation is called ``12.4 fixed-point format''.
Each bit contributes 2\textsuperscript{x} to the real number value, where x is the bit's position offset from the decimal point.
Figure~\ref{fig:fixed_point} shows an example of a 16-bit value which is interpreted as a 12.4 fixed-point value and as a 8.8 fixed-point value.

\begin{figure}[H]
   \begin{center}
       \includegraphics[scale=0.70]{figures/fig_fixed_point}
   \end{center}
   \caption{Interpreting a 16-bit value as 12.4 and 8.8 fixed-point values}
	\label{fig:fixed_point}
\end{figure}

An advantage of using fixed-point arithmetic is that the operations require fewer hardware resources to implement
compared to floating-point arithmetic.

~\\
This means that a fixed-point circuit can be smaller and more power-efficient than its floating-point equivalent.
Alternatively, you could achieve higher performance by implementing more fixed-point units with the same hardware resources.

~\\
Fixed-point multiplication can be computed using an integer multiplier circuit, with the caveat that the circuit
does not track the decimal point locations of the multiplicands and the product.
It is therefore the designer's responsibility to track the decimal point locations.
The product's decimal point location (offset from the LSB) is determined by adding the offsets in the multiplicands.
For example, if you multiply a 12.4 fixed-point number by a 8.8 fixed-point number, the decimal point in the product
will be offset from the LSB by 4 + 8 = 12 bits. In other words, the product would have a 12-bit fraction.

~\\
Fixed-point addition can be computed using an integer adder circuit, provided that two summands have the same decimal point location.
When adding two fixed-point values whose decimal point locations differ, one of the summands must be shifted to match the other prior to addition.
The resulting sum will have the same decimal point location as its summands.

\subsubsection*{Modifying Your Kernel to Use Fixed-Point Arithmetic}

Modify your host program to parse the fixed-point weights in the directory \textit{/design\_files/weights\_fxp/}.
These fixed-point weights are stored in 16.16 fixed-point format.
Modify your kernel to accept the fixed-point weights as an \texttt{int} array, as shown in Figure~\ref{fig:kernel_v2}.
When multiplying the 16.16 fixed-point weights by the 8-bit pixels, the pixels can be considered to be
8.0 fixed-point numbers.

\lstset{language=C}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{16 cm}
\begin{lstlisting}
__attribute__((reqd_work_group_size(10000,1,1)))
__kernel void linear_classifier(global const unsigned char * restrict images,
								constant int * restrict weights,
								global unsigned char * restrict results)
{
	... Your Code ...
}
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-0.33in}\caption{Function prototype for the linear classifier kernel that uses fixed point weights}
\label{fig:kernel_v2}
\end{figure}

Use the \texttt{aoc} flag \texttt{-c} to obtain a compilation report for your modified kernel and examine the resource utilization of
implementing fixed-point addition and multiplication. Determine the per-adder and per-multiplier resource cost and compare them
to the numbers you determined in Part III.
Unroll your loops to fully utilize the resources of the FPGA and maximize the kernel's performance.
How many times can you unroll your loop before running out of resources, and how does that compare to your design from Part II?
Compile your design and use it to classify the MNIST test images.
What is the runtime of your new kernel and how does it compare to that of your kernel from Part II?
Is your classification accuracy affected? Why or why not? In what situations would changing from
floating-point arithmetic to fixed-point arithmetic affect (and not affect) classification accuracy?

\input{\CommonDocsPath/copyright.tex}
\end{document}


*  Yuyan: Deep Neural network is usually under-determined.  If there are a set of parameters will give accurate prediction under various perturbations or disturbances or distractions,  then these would be a good set of parameters.

Why a good set of parameters should be robust with respects to drop-out?

*  Xiaodong:

Dropping out will reduce the number of parameters, which will decrease the generalization ability (泛化能力）, namely decrease the inaccuracy in the test data.   This has nothing to do what activations are used.

Theoretical starting point:
\begin{quote}
$\#\Theta$ is sufficiently large that the underlying DNN model is accurate.
\end{quote}

The starting point, or the ideal case,  is that $\# \Theta \approx \# X$.

Dropout is only efficient if $\# \Theta \gg \# X$.

* Hongxuan:

This gives a random perturbation in the gradient method.  This will help to jump out local minimizers ...

* Juncai:  These parameters should have special structures, namely these matrices, multiplied by a random cut-matrices (consisting of $0$ and $1$) on their right lead to the final results that have some uniform properties.

$$
f^{j+1}=\theta^{j+1}\circ g \theta^{j}\circ g \circ f^{j-1}
$$
Drop-out:  Let $D_j$ be a diagonal matrix with $0$ or $1$ as diagonal, say $25\%$ $0$.
$$
f^{j+1}=\theta^{j+1}\circ g \circ \theta^{j}\circ {\color{red}D_j}\circ g \circ f^{j-1}
$$
There are two conceptional understandings, one is
$$
f^{j+1}=\theta^{j+1}\circ g \circ \theta^{j}\circ[ {\color{red}D_j}\circ g \circ f^{j-1}]
$$
This means that we drop out $25\%$ the output values
$$
f^{j+1}=\theta^{j+1}\circ g \circ [\theta^{j}\circ {\color{red}D_j}]\circ g \circ f^{j-1}
$$
This means that we drop out $25\%$ neurons.

This may be related to the following fact:

\begin{quote}\it
Given $Q\in \mathbb R^{n\times m}$, let $D={\rm diag}(d_i)$ with $d_i\in {\rm rand}(0,1)$.  then, for large singular values
$$
\sigma_i(QD-Q)\le \epsilon \sigma_i(Q)
$$
\end{quote}

* Deep Learning Book:

It is important to understand that a large portion of the power of dropoutarises from the fact that the masking noise is applied to the hidden units. This can be seen as a form of highly intelligent, adaptive destruction of the information content of the input rather than destruction of the raw values of the input. For example, if the model learns a hidden unithithat detects a face by finding thenose, then dropping $h_i$ corresponds to erasing the information that there is a nose in the image. The model must learn another $h_i$, that either redundantly encodes the presence of a nose or detects the face by another feature, such as the mouth. Traditional noise injection techniques that add unstructured noise at the input are not able to randomly erase the information about a nose from an image of a face unless the magnitude of the noise is so great that nearly all the information inthe image is removed. Destroying extracted features rather than original values allows the destruction process to make use of all the knowledge about the input distribution that the model has acquired so far.

Another important aspect of dropout is that the noise is multiplicative. If the noise were additive with fixed scale, then a rectified linear hidden unit $h_i$ with added noise $\epsilon$ could simply learn to have $h_i$ become very large in order to make the added noise $\epsilon$ insignifiant by comparison. Multiplicative noise does not allow such a pathological solution to the noise robustness problem.

\newpage
\section{Mathematical versus practical}

\subsection{Mathematical}
Mathematically speaking, for a given application, we have the following conclusion
\begin{enumerate}
\item There exist a number of smoother disjoint manifold that  contain all the possible data.
\item If the $N=\#\Theta$ is sufficiently large, then there exists a DDN with these paramaters that can ``accurately'' classify the underlying manifolds
\item If the $m=\# X$ is sufficiently large and ``well-distributed'' for training, then the aforementioned DDN with at least one set of parameters $\Theta^{\rm opt}$ and, as a result, there is no ``over-fitting''!
\end{enumerate}

\subsection{Practical}
We will use training data to get an approximation of $\Theta^{\rm opt}$.  More precisely
$$
\Theta^*=\arg\min {1\over m}\sum_{i=1}^m|f(x_i,\Theta)-y_i|^2.
$$
Question:
$$
\Theta^*\approx \Theta^{\rm opt}?
$$
Note
\begin{quote}
  $\Theta^*$ is determined by the training data!
\end{quote}

Xu:

\begin{quote}
If  $m\gg 1$ and ``good'', then $\Theta^*\approx \Theta^{\rm opt}?$
\end{quote}
Practically speaking,
\begin{enumerate}
\item $N$ is unknown.
\item $N\gg 1$, but $m\ll N$
\end{enumerate}

\subsubsection{Case 1:  The DNN model is good enough}

\subsubsection{Case 2:  The DNN model is  NOT good enough}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\section{Lecture 28}
\subsection{Lecture Notes - Differential Cross Section}
\subsubsection{Review}
Cross section = area of ring of radius $b$ and width $db$. Particles hitting the ring between $b$ and $b + db$ are scattered by an angle between $\theta$ and $\theta + d\theta$. Scattered onto a larger ring on a sphere with scallering nucleus in center. The solid angle of the entire ring is:
\[d\Omega = \frac{2\pi R\sin\theta Rd\theta}{R^2} = 2\pi\sin\theta d\theta\]
And the solid angle of a small area is given by:
\[d\Omega = \frac{d\phi R\sin\theta Rd\theta}{R^2} = \sin\theta d\phi d\theta\]
This notion is useful as often our particle detectors cover a certain fraction of the area.
\begin{center}
    \includegraphics[scale=0.5]{Lecture-28/l28-img1.png}
\end{center}
The ratio between the $d\sigma$ of the first ring and the $d\Omega$ of the second ring is the the differential cross section, which is:
\[\dod{\sigma}{\Omega} = \frac{b}{\sin\theta}\abs{\dod{b}{\theta}}\]
Where $d\sigma = 2\pi b db$ and $d\Omega = 2\pi \sin\theta d\theta$. The task today is then to obtain $b$ in terms of $\theta$ ($b(\theta)$). Today, we will consider the case where our target is fixed (i.e. a heavy target) and try to describe the trajectory of this scenario. This is just the two body central force problem (as covered in PHYS 216), with the only difference that the orbits are open and not closed.

\subsubsection{Calculation of the differential cross-section - The Kepler Approach}
We first remark that this chapter is interesting as not only is it highly relevant to research fields (e.g. particle physics) but also puts something that we know (the two-body problem) into a new context. We now move onto solving this two-body central force problem. The total energy of the system in polar coordinates is given by:
\[E = \frac{m}{2}\left(\dot{r}^2 + r^2\dot{\phi}^2\right) + U(r) = \text{Const}\]
Where $U(r)$ is the scattering potential. Here the energy is conserved. Another quantity that is conserved is the angular momentum:
\[L = mr^2\dot{\phi} = \text{Const}\]
From this, we can immediately write:
\[r^2\dot{\phi}^2 = \frac{L^2}{m^2r^2}\]
Hence sustituting this into the energy expression to obtain everything in terms of $r$, we have:
\[E = \frac{m}{2}\left(\dot{r}^2 + \frac{L^2}{m^2r^2}\right) + U(r)\]
We can therefore solve for $\abs{\dot{r}}$:
\[\abs{\dot{r}} = \sqrt{\frac{2E}{m} - \frac{2U(r)}{m} - \frac{L^2}{m^2r^2}}\]
We have a two body problem, but it is effectively a one body problem. We may introduce an "effective potential":
\[U_{eff} = U(r) + \frac{L^2}{2mr^2}\]
Since we know that $\phi(t)$ is monotonic, we may write:
\[\abs{\dot{\phi}} = \frac{\abs{L}}{mr^2}\]
Dividing the $\dot{\phi}$ equation by the $\dot{r}$ equation, we get:
\[\frac{\abs{\dot{\phi}}}{\abs{\dot{r}}} = \abs{\dod{\phi}{r}} = \frac{\frac{\abs{L}}{mr^2}}{\sqrt{\frac{2E}{m} - \frac{2U(r)}{m} - \frac{L^2}{m^2r^2}}}\]
We can from this expression obtain the trajectory in the following way. The particle is incoming with impact parameter $b$, and deflects off by angle $\theta$. There is some closest point of approach $r_{min}$ that divides the trajectory into two pieces, that is, the trajectory is symmetric about $r_{min}$. Call the angle at this point $\Delta \phi/2$. This can be visualized as follows:
\begin{center}
    \includegraphics[scale=0.7]{Lecture-28/l28-img2.png}
\end{center}
We therefore have that:
\[\Delta \phi(r) = 2\int_{r_{min}}^\infty \frac{\frac{L}{r^2}}{\sqrt{2m(E-U) - \frac{L^2}{r^2}}}\]
Note that we obtain $r_{min}$ by solving for the turnaround point, i.e the energy equals the effective potential (the kinetic energy is zero) so:
\[2m(E - U) = \frac{L^2}{r_{min}^2}\]
However, all of our equations are in terms of $L$ and $E$;  should recast these in terms of physical parameters. Set:
\[E = T_{\infty} = \frac{m}{2}v_{\infty}^2\]
and
\[L = \abs{\v{r} \times \v{p}_{\infty}} = mbv_{\infty}\]
Hence we may write:
\[\Delta \phi = 2b\int_{r_{min}}^\infty \frac{dr}{r^2\sqrt{1 - \frac{2U(r)}{mv_{\infty}^2} - \frac{b^2}{r^2}}}\]
So we have expressed the integral purely in terms of the mass, velocity, and impact parameter, which are all measureable variables. Note that this solution isn't really scattering; this is a very general solution to the two-body problem.
We can now apply it to some relevant scattering scenarios to obtain the differential cross section.

\subsubsection{Example - Hard Sphere Differential Cross-Section}
In this case, $r_{min} = R$ (closest point of approach is just the surface of the sphere), and $V(r) = 0$ for $r > R$ (the particles don't see each other). For this case, we have:
\[\Delta \phi = 2b\int_R^\infty\frac{dr}{r^2\sqrt{1 - \frac{b^2}{r^2}}}\]
The middle term is zero as the potential is zero in the region of interest. This integral is solvable analytically, but we may very well just look this up instead of undergoing a tedious calculation:
\[\Delta \phi = 2\arcsin(\frac{b}{R})\]
Rearranging, we get:
\[b = R\sin(\frac{\Delta \phi}{2})\]
According to the picture we have drawn, $\Delta \phi + \theta = \pi$, so we can convert this to be in terms of $\theta$, which works out to be:
\[b = R\cos(\frac{\theta}{2}) = b(\theta)\]
Now, all is left to do is to calculate the differential cross section:
\[\dod{\sigma}{\Omega} = \frac{b}{\sin\theta}\abs{\dod{b}{\theta}} = \frac{R\cos(\frac{\theta}{2})}{\sin\theta}\abs{-\frac{R}{2}\sin(\frac{\theta}{2})} = \frac{R^2}{2\sin\theta}\cos(\frac{\theta}{2})\sin(\frac{\theta}{2}) = \frac{R^2}{2\sin\theta}\frac{1}{2}\sin(\theta) = \frac{R^2}{4}\]
Hence calculating $\sigma_{tot}$ (total cross section) we have:
\[\sigma_{tot} = \int \dod{\sigma}{\Omega}d\Omega = \int \frac{R^2}{4}d\Omega = \pi R^2\]
Which is a good sanity check, as it agrees with what we found on Monday!

\subsubsection{Example - Coloumb Potential Differential Cross-Section}
In this case, we have that:
\[U = -\frac{\beta}{r}\]
Where $\beta = -kqQ < 0$ (repulsive). The procedure is exactly the same, but the integral is just harder.
\[\Delta \phi = 2b\int_{r_{min}}^\infty \frac{dr}{r\sqrt{r^2 + \frac{2\beta r}{mv_{\infty}} - b^2}}\]
\[\Delta \phi = \left.2\arccos(\frac{1 - \frac{mv_\infty^2 b^2}{\beta}}{\sqrt{1 + \left(\frac{mv_{\infty}^2b}{\beta}\right)^2}})\right|_{r_{min}}^{\infty}\]
Evaluating at the bounds, we get:
\[\Delta \phi = 2\arccos(\frac{1}{\sqrt{1 + \left(\frac{mv_\infty b}{\beta}\right)^2}})\]
(this is the term at infinity, the term at $r_{min}$ vanishes, as the value is $2\arccos(-1) = 2\pi$ and $\phi = \phi + 2\pi$). Hence solving for $b(\theta)$ we have:
\[b(\theta) = \frac{1}{mv_{\infty}^2}\sqrt{\frac{1}{\sin^2(\frac{\theta}{2})} - 1}\]
Using this, we find the differential cross section:
\[\frac{d\sigma}{d\Omega} = \frac{k^2q_1^2q_2^2}{16E^2}\frac{1}{\sin^4(\frac{\theta}{2})}\]
Plotting the Rutherford cross section, we find:

Where we have a characteristic divergence at the origin and a rapid decrease, with a value of $1$ at $\pi$.
\begin{center}
    \includegraphics[scale=0.5]{Lecture-28/l28-img3.png}
\end{center}
Question: What is the total cross section?
\begin{s}
$\sigma = \int\dod{\sigma}{\Omega}d\Omega = \infty$ due to the divergence at the origin. This can be traced back to the $\frac{1}{r}$ type behavior of the Columb potential.
\end{s}
A remark: One could state that this (classical treatment) is nonsense and we require a quantum treatment. Luckily, the classical treatment actually does work out in this case (due to the Coulomb potential in particular) but for a full description we require a quantum formulation, which is something we can expect to see in Graduate school QM. Next day we will look at the center of mass lab frame, and how we treat scattering off of nuclei that are not infinitely heavy.
\end{document}


\chapter{Multidimensional scaling}
\label{mds}

%\section{Principal co-ordinates analysis - multidimensional scaling}

We have looked at clustering techniques which partition an $n \times n$ matrix of dissimilarities into groups of like individuals.   Whilst it is possible to visualise differences between individuals in terms of dendrograms, it might be more useful to have a direct representation of the relationship between individuals.   We are now going to consider techniques which let us visualise the relative distances bewteen individuals in a low dimensional structure.  Some early ideas were set out by \cite{Richardson:1938}, and the algebraic techniques required to reconstruct a configuration from distances were established by \cite{Young+Householder:1938}.   \cite{Torgerson:1952} set out the foundations for this work, but developments in the technique associated with the name \emph{principal co-ordinates analysis} were given by \cite{Gower:1966}.   Arguably, the tecnique is most commonly referred to as \emph{scaling}, often as classical scaling.

\section{Metric Scaling}

Consider (any) distance matrix $\boldsymbol{\Delta}$.   It is metric if elements of $\boldsymbol{\Delta}$ satisfy the metric (triangle) inequality $\delta_{ij} \leq \delta_{ik} + \delta_{kj}$ for all $i$, $j$, $k$.

Classical metric multidimensional scaling is based on the $n \times n$ distance or dissimilarity matrix, we will note later some connections with principal components.   Here, we will first consider a matrix $\boldsymbol{\Delta}$ of \emph{Euclidean} distances between objects, such that $\delta_{ij}$ is the distance between points $i$ and $j$.   These $n$ points can be represented in $n-1$ dimensional space.   We are looking for a configuration of $n$ points such that the distance $d_{ij}$ between points $i$ and $j$ equals the dissimilarity $\delta_{ij}$ for all $i$, $j$.   The dimensionality of this configuration is $q$ such that we are seeking to reconstruct an $n \times q$ matrix $\boldsymbol{X}$.

We can very easily get from an $n \times p$ matrix $\boldsymbol{X}$ to a Euclidean distance matrix.   If we first form the $n \times n$ matrix $\boldsymbol{Q}$ then $\boldsymbol{Q} = \boldsymbol{X}\boldsymbol{X}^{T}$.   Considering this one element at a time:

\begin{equation}
q_{rs} = \sum_{j = 1}^{p} x_{rj}x_{sj}
\end{equation}

We know that the Euclidean distance is given by:

\begin{eqnarray*}
\delta_{rs}^{2} &=& \sum_{j=1}^{p} (x_{rj} - x_{sj})^{2} \\
 &=& \sum_{j=1}^{p} (x_{rj}^{2} + \sum_{j=1}^{p} (x_{sj}^{2} - 2 \sum_{j=1}^{p} x_{rj}x_{sj} \\
&=&  q_{rr} + q_{ss} - 2q_{rs}
\end{eqnarray*}

So given $\boldsymbol{X}$ we can find $\boldsymbol{Q} = \boldsymbol{X}\boldsymbol{X}^{T}$ and hence find the Euclidean distance.
%proof pg 106

What we want to do now is reverse this process.    We should note that our recovered $n \times q$ matrix is not uniquely defined - it is only defined up to translation, rotation and reflection.   To fix this, we will usually assume $\boldsymbol{X}$ has been centred (i.e. make column means zero, such that $\sum_{i=1}^{n} y_{ij} = 0)$.    It may be noted here that we don't necessarily want to recover an $n \times p$ matrix, we can reconstruct a matrix with up to $n-1$ columns, but as with other dimension reduction techniques we hope to find an $n \times q$ matrix with $q < p$; clearly if $q = 2$ it will be easy to visualise the results.   So, to recover $\boldsymbol{Q}$ from $\boldsymbol{D}$:

\begin{eqnarray}
\sum_{r=1}^{n} d_{rs}^{2} = trace(\boldsymbol{Q}) + n q_{ss}\\
\sum_{s=1}^{n} d_{rs}^{2} = n q_{rr} + trace(\boldsymbol{Q}) \\
\sum_{r=1}^{n} d_{rs}^{2} \sum_{s=1}^{n} d_{rs}^{2} = 2 n trace(\boldsymbol{Q})
\end{eqnarray}

By rearranging this and manipulating the equations above which lead us to our distance matrix, we can recover elements of $\boldsymbol{Q}$ from $\boldsymbol{D}$ using a double centering procedure:

\begin{displaymath}
q_{ij} = -\frac{1}{2} ( d_{ij}^{2} - d_{i \cdot}^{2} -  d_{\cdot j}^{2} +  d_{\cdot \cdot}^{2})
\end{displaymath}

The dots denote means taken over the relevant indices.

% mention double centering here
In summary, to find $\boldsymbol{Q}$ given $\boldsymbol{D}$:

\begin{itemize}
\item Square it element by element
\item Double centre it
  \begin{itemize}
  \item  subtract column means
  \item subtract row means
  \item add overall mean
  \end{itemize}
\item Multiply by $-\frac{1}{2}$.
\end{itemize}

%some people approximate Q with a similarity matrix

Having found $\boldsymbol{Q}$ ($\boldsymbol{Q} = \boldsymbol{XX}^{T}$), all we need is to find a suitable $\boldsymbol{X}$, which sounds like some kind of matrix square root problem.   Given Euclidean distances, $\boldsymbol{Q}$ is symmetric and we can do this using the spectral decomposition $\boldsymbol{Q} = \boldsymbol{E \Lambda E}^{T}$ where $\boldsymbol{\Lambda} = \lambda_{1}, \lambda_{2}, \ldots, \lambda_{n}$, a diagnonal matrix of ordered eigenvalues and $\boldsymbol{E}$ is the matrix whose columns are the corresponding (normalised) eigenvectors.   If  $\boldsymbol{Q} = \boldsymbol{E \Lambda}^{\frac{1}{2}} \boldsymbol{\Lambda^{\frac{1}{2}} E}^{T}= \boldsymbol{X} \boldsymbol{X}^{T}$ then $\boldsymbol{X} =  \boldsymbol{E \Lambda}^{\frac{1}{2}}$ and we have recovered the co-ordinates from the inter-point distances.

\begin{displaymath}
\boldsymbol{X} = \left( \sqrt{\lambda_{1}} \left( \begin{array}{r} e_{11} \\ e_{12} \\ \vdots \\ e_{1n} \end{array} \right), \ldots,   \sqrt{\lambda_{n}} \left( \begin{array}{r} e_{n1} \\ e_{n2} \\ \vdots \\ e_{nn} \end{array} \right) \right)
\end{displaymath}

So if we want a one dimensional representation, we just use  $\left( \sqrt{\lambda_{1}} \boldsymbol{e}_{1} \right) $, for a two dimensional representation we would use  $\left( \sqrt{\lambda_{1}} \boldsymbol{e}_{1},   \sqrt{\lambda_{2}} \boldsymbol{e}_{2} \right)$

\subsection{Similarities with principal components analysis}

A short diversion noting a few similarities with principal component analysis may be in order.   Consider the centred data matrix $\boldsymbol{X}$:

\begin{displaymath}
C = \frac{1}{n-1} \boldsymbol{X}^{T} \boldsymbol{X}
\end{displaymath}

Principal components come from an eigenanalysis of $\boldsymbol{C}$, here we denote the eigenvalues of $\boldsymbol{C}$ by $\mu_{i}$ and associated eigenvectors by $\boldsymbol{a}_{i}$:

\begin{eqnarray*}
\boldsymbol{C} \boldsymbol{a}_{i} &=& \mu_{i} \boldsymbol{a}_{i}\\
\frac{1}{n-1}\boldsymbol{X}^{T} \boldsymbol{X} \boldsymbol{a}_{i} &=& \mu_{i} \boldsymbol{a}_{i}\\
\boldsymbol{X}^{T} \boldsymbol{X} \boldsymbol{a}_{i} &=& (n-1)\mu_{i} \boldsymbol{a}_{i}\\
\boldsymbol{X}\boldsymbol{X}^{T} \boldsymbol{X} \boldsymbol{a}_{i} &=& (n-1)\mu_{i} \boldsymbol{X} \boldsymbol{a}_{i}\\
\boldsymbol{Q} \underbrace{\boldsymbol{X} \boldsymbol{a}_{i}}_{\boldsymbol{z}_{i}} &=& (n-1)\mu_{i} \underbrace{\boldsymbol{X} \boldsymbol{a}_{i}}_{\boldsymbol{z}_{i}}
\end{eqnarray*}

So $\boldsymbol{X} \boldsymbol{a}_{i} = \boldsymbol{z}_{i}$ is an eigenvector of $\boldsymbol{Q}$ with corresponding eigenvalue $(n-1) \mu_{i}$.   If we want a normalised eigenvector it may be worth noting that the length can be found as follows:

\begin{eqnarray*}
||\boldsymbol{z}_{i}||^{2} = \boldsymbol{z}_{i}^{T}\boldsymbol{z}_{i} = \boldsymbol{a}_{i} \boldsymbol{X}^{T} \boldsymbol{X} \boldsymbol{a}_{i} &=& (n-1) \boldsymbol{a}_{i}^{T} \boldsymbol{C} \boldsymbol{a}_{i}\\
 &=& (n-1) \boldsymbol{a}_{i}^{T} \mu_{i} \boldsymbol{a}_{i}\\
 &=& (n-1) \mu_{i} \frac{\boldsymbol{a}_{i}^{T} \boldsymbol{a}_{i}}{||\boldsymbol{a}_{i}||^{2}}\\
 &=& (n-1) \mu_{i}
\end{eqnarray*}

So, $||\boldsymbol{z}_{i}|| = \sqrt{(n-1) \mu_{i}}$.   Hence a normalised eigenvector for $\boldsymbol{Q}$ takes the form $\frac{1}{(n-1)\mu_{i}} \boldsymbol{Xa}_{i}$ with eigenvalue $(n-1) \mu_{i}$

Therefore, our eigenvalues and eigenvectors found from multidimensional scaling / principal co-ordinates analysis are related to those found from decomposition of the covariance of the scaled data matrix:

\begin{eqnarray*}
\lambda_{i} = (n-1) \mu_{i}\\
e_{i} = \frac{1}{\sqrt{\lambda_{i}}} \boldsymbol{Xa}_{i}
\end{eqnarray*}

Remember that $\boldsymbol{Z} = \boldsymbol{XA}^{T}$, where:

\begin{displaymath}
\boldsymbol{A} = \left( \begin{array}{c} \boldsymbol{a}_{1}^{T} \\  \boldsymbol{a}_{2}^{T} \\ \vdots \\ \boldsymbol{a}_{n}^{T}   \end{array} \right)
\end{displaymath}

So $\boldsymbol{Xa}_{i}$:

\begin{displaymath}
 \left( \boldsymbol{Xa}_{1} \boldsymbol{Xa}_{2}, \ldots, \boldsymbol{Xa}_{n} \right)
\end{displaymath}
in other words, this is our matrix of principal component scores.

%This technique uses eigen decompositions (which we can do in our sleep by now) and a similarity matrix.   The thing to emphasise is that we are decomposing an $n \times n$ matrix of similarities between points, rather than in principal components where we had a $p \times p$ matrix of correlations / variance-covariances.

\section{Visualising multivariate distance}
\label{visdist}

Consider the US Arrests data (we have looked at this a few times).   If you still have the distance matrix \texttt{spot} created earlier, you can run principal co-ordinates analysis quite easily:

\singlespacing
\begin{verbatim}
> spot <- dist(USArrests, method = "euclidean")
> what <- cmdscale(spot)
> plot(what[,1], what[,2],
   xlab = "Axis 1", ylab = "Axis 2",
   main = "US Arrests")
> identify(what[,1], what[,2], row.names(USArrests))
\end{verbatim}
\onehalfspacing

By default you extract two variables, and you don't get the eigen values.   You can alter the function call if you want to change that (see \texttt{?cmdscale})

You might like to use \texttt{identify()} to check that very odd points according to principal co-ordinates are also very odd according to your cluster analysis.

\section{Assessing the quality of fit}

One measure of discrepancy is given by:

\begin{displaymath}
\varphi = \sum_{i=1}^{n} \sum_{j=1}^{n} (\delta_{ij}^{2} - d_{ij}^{2})
\end{displaymath}

and it can be shown \citep{Mardia+etal:1979}%, page 406}
 that:

\begin{displaymath}
\varphi = 2n(\lambda_{q+1} + \ldots + \lambda_{n})
\end{displaymath}

So, if we fit a model with $q = n-1 = 49$ (in order to measure the size of the discarded eigenvalues - most of these eigenvalues are zero hence warnings about eigenvalues below zero):

\singlespacing
\begin{verbatim}
> what <- cmdscale(spot, eig = TRUE, k = 49)
> 2 * dim(USArrests)[1] * sum(what$eig[3:49])
\end{verbatim}
\onehalfspacing

We should find that this give the same value as a direct comparison of the distance matrices formed from the data and the $q=2$ dimensional representation.  (Note that we convert the distance matrices into ordinary matrices, and then we carry out vectorised operations to take squares, subtract elementwise and sum)

\singlespacing
\begin{verbatim}
what <- cmdscale(spot, eig = TRUE, k = 2)
delta <- as.matrix(dist(USArrests, upper = TRUE))
d <- as.matrix(dist(what$points, upper = TRUE))
sum(as.vector(delta)^2 - as.vector(d)^2)
\end{verbatim}
\onehalfspacing

This has clear analogies with the percent trace measure used in principal components analysis:

\begin{equation}
\frac{\sum_{i=1}^{q} \lambda_{i}}{\sum_{i=1}^{p} \lambda_{i}}
\end{equation}

\singlespacing
\begin{verbatim}
> what <- cmdscale(spot, eig = TRUE, k = 4)
> what$eig / sum(what$eig)
[1] 0.9655342206 0.0278173366 0.0057995349 0.0008489079
\end{verbatim}
\onehalfspacing

Considerable work has been carried out on Goodness of fit measures for scaling problems.   If $\boldsymbol{\Delta}$ is based on a measure other than the Euclidean then our reconstructed $\boldsymbol{Q}$ may not be positive semi-definite (in other words we find some negative eigenvalues and imaginary co-ordinates).   If $\boldsymbol{Q} = \sum_{i=1}^{q} \lambda_{i} \boldsymbol{e}_{i} \boldsymbol{e}_{i}^{T}$ then \cite{Mardia:1978} suggests the discrepancy measure:

\begin{equation}
\varphi \prime = trace(\boldsymbol{Q} - \hat{\boldsymbol{Q}}^{2})
\end{equation}.   Following \cite{Eckart+Young:1936} we would use:

\begin{equation}
\frac{\sum_{i=1}^{q} \lambda_{i}}{\sum_{i=1}^{p} |\lambda_{i}|}
\end{equation}

or following \cite{Mardia:1978} we would use:

\begin{equation}
\frac{\sum_{i=1}^{q} \lambda_{i}^{2}}{\sum_{i=1}^{p} \lambda_{i}^{2}}
\end{equation}

\subsection{Sammon Mapping}

Classical metrical scaling works on orthogonal projections, and has the attractive property of an exact analytical solution.   This is quite restrictive, \cite{Sammon:1969} suggested minimising the discrepancy measure:

\begin{displaymath}
\varphi \prime \prime =  \sum_{i=1}^{n} \sum_{j=1}^{n} (\delta_{ij} - d_{ij})^{2}
\end{displaymath}

This has no analytical solution, and numerical methods must be used.   But it is worth noting that a set of disparities are generated:

\begin{equation}
\label{rsssammon}
\hat{d}_{ij} = a + b \delta_{ij}
\end{equation}

This should look like a reasonably familiar formula, and you should have guessed that residual sums of squares from \ref{rsssammon} yields another discrepancy measure.   This measure can (should / must) be normalised with reference to its size $\sum_{i=1}^{n} \sum_{j=1}^{n} d_{ij}^{2}$, giving what is called the \textbf{st}andardised \textbf{re}sidual \textbf{s}um of \textbf{s}quares:

\begin{equation}
STRESS = \left( \frac{\sum_{i=1}^{n} \sum_{j=1}^{n} (d_{ij} - \hat{d}_{ij})^{2}}{\sum_{i=1}^{n} \sum_{j=1}^{n} d_{ij}^{2}} \right) ^{\frac{1}{2}}
\end{equation}

\begin{equation}
SSTRESS = \left( \frac{\sum_{i=1}^{n} \sum_{j=1}^{n} (d_{ij}^{2} - \hat{d}_{ij}^{2})^{2}}{\sum_{i=1}^{n} \sum_{j=1}^{n} d_{ij}^{4}} \right) ^{\frac{1}{2}}
\end{equation}

Normalisation means that both these measures take on values between 0 and 1, lower values indicate better fit.   Values below 0.1 are usually considered adequate, but it may be worth noting that \cite{Kruskal:1964a} suggests values below 0.2 give a poor fit, values below 0.1 a fair fit, values below 0.05 a good fit, values below 0.025 an excellent fit.

%\sum_{i=1}^{q} \lambda_{i}
%\sum_{i=1}^{n} \lambda_{i}
%abs(v) max(v,0)

%F: a numeric vector of length 2, equal to say (g.1,g.2), where
%          g.i = (sum{j=1..k} lambda[j]) / (sum{j=1..n} T.i(lambda[j])),
%          where lambda[j] are the eigenvalues (sorted decreasingly),
%          T.1(v) = abs(v), and T.2(v) = max(v, 0).

%Iff $d_{ij} = d_{ik} + d_{kj}$ then principal co-ordinates also acts as a method of metric scaling.   You will be given further directed reading on this topic in the relevant lecture.

%principle co-ordinates analysis wk 104-109

%\section{Modern approaches}
%\label{modernmds}

%%% Local Variables: ***
%%% mode:latex ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


\documentclass{article}

\usepackage{listings}
\lstset{
  basicstyle=\ttfamily,
  keywordstyle=\ttfamily,
  escapeinside={(*@}{@*)},
}

\usepackage{xcolor}
\newcommand\hlBlame[1]{\colorbox{red!25}{#1}}

\begin{document}

\section{Debugging and Functional Programming}

You will be shown OCaml programs that \emph{do not type-check}, the code
implicated by the type checker will be \hlBlame{highlighted}. You will
also be given English comments explaining what the program \emph{should}
do, as well as assertions that \emph{should} type check \emph{and} succeed.

\subsection{Concatenating Strings}

\subsubsection{Control}
\begin{lstlisting}
  (* "sepConcat sep [s1;s2;s3]" should insert "sep"
     between "s1", "s2", and "s3", and concatentate
     the result. *)

  let rec sepConcat sep sl =
    match sl with
    | [] -> ""
    | h::t ->
        let f a x = a ^ (sep ^ x) in
        let base = [] in
        (*@\hlBlame{List.fold\_left f base sl}@*)

  assert( sepConcat "," ["foo"; "bar"; "baz"] = "foo,bar,baz" )
\end{lstlisting}

\begin{enumerate}
\item Why is \verb!sepConcat! not well-typed?
\item Fix the \verb!sepConcat! program.
\end{enumerate}

\subsubsection{Treatment}
\begin{lstlisting}
  (* "sepConcat sep [s1;s2;s3]" should insert "sep"
     between "s1", "s2", and "s3", and concatentate
     the result. *)

  let rec sepConcat sep sl =
    match sl with
    | [] -> ""
    | h::t ->
        let f a x = a ^ (sep ^ x) in
        let base = (*@\hlBlame{[]}@*) in
        List.fold_left f base sl

  assert( sepConcat "," ["foo"; "bar"; "baz"] = "foo,bar,baz" )
\end{lstlisting}

\begin{enumerate}
\item Why is \verb!sepConcat! not well-typed?
\item Fix the \verb!sepConcat! program.
\end{enumerate}

\subsection{Padding Lists}

\subsubsection{Control}
\begin{lstlisting}
  (* "padZero xs ys" returns a pair "(xs', ys')"
     where the shorter of "xs" and "ys" has been
     left-padded by zeros until both lists have
     equal length. *)

  let rec clone x n =
    if n <= 0 then
      []
    else
      x :: clone x (n - 1)

  let padZero l1 l2 =
    let n = List.length l1 - List.length l2 in
    if n < 0 then
      (clone 0 ((-1) * n) @ l2, l2)
    else
      (l1, (*@\hlBlame{clone 0 n}@*) :: l2)

  assert( padZero [1;2] [1] = ([1;2], [0;1]) )
\end{lstlisting}

\subsubsection{Treatment}
\begin{lstlisting}
  (* "padZero xs ys" returns a pair "(xs', ys')"
     where the shorter of "xs" and "ys" has been
     left-padded by zeros until both lists have
     equal length. *)

  let rec clone x n =
    if n <= 0 then
      []
    else
      x :: clone x (n - 1)

  let padZero l1 l2 =
    let n = List.length l1 - List.length l2 in
    if n < 0 then
      (clone 0 ((-1) * n) @ l2, l2)
    else
      (l1, clone 0 n (*@\hlBlame{::}@*) l2)

  assert( padZero [1;2] [1] = ([1;2], [0;1]) )
\end{lstlisting}

\begin{enumerate}
\item Why is \verb!padZero! not well-typed?
\item Fix the \verb!padZero! program.
\end{enumerate}

\subsection{Big-Integer Arithmetic}

\subsubsection{Control}
\begin{lstlisting}
  (* "mulByDigit d [n1;n2;n3]" should multiply the
     "big integer" "[n1;n2;n3]"
     by the single digit "d". *)

  let rec mulByDigit d n =
    match List.rev n with
    | []   -> []
    | h::t -> [(*@\hlBlame{mulByDigit d t}@*); (h * d) mod 10]

  assert( mulByDigit 4 [2;5] = [1;0;0] )
\end{lstlisting}

\begin{enumerate}
\item Why is \verb!mulByDigit! not well-typed?
\item Fix the \verb!mulByDigit! program.
\end{enumerate}

\subsubsection{Treatment}
\begin{lstlisting}
  (* "mulByDigit d [n1;n2;n3]" should multiply the
     "big integer" "[n1;n2;n3]"
     by the single digit "d". *)

  let rec mulByDigit d n =
    match List.rev n with
    | []   -> []
    | h::t -> (*@\hlBlame{[mulByDigit d t; (h * d) mod 10]}@*)

  assert( mulByDigit 4 [2;5] = [1;0;0] )
\end{lstlisting}

\begin{enumerate}
\item Why is \verb!mulByDigit! not well-typed?
\item Fix the \verb!mulByDigit! program.
\end{enumerate}

\end{document}