\chapter{Compactness}
One of the most important notions of topological spaces is that of \emph{compactness}.
It generalizes the notion of ``closed and bounded'' in Euclidean space
to any topological space
(e.g.\ see \Cref{thm:bzw}).

For metric spaces, there are two equivalent ways of formulating compactness:
\begin{itemize}
	\ii A ``natural'' definition using \emph{sequences}, called sequential compactness.
	\ii A less natural definition using open covers.
\end{itemize}
As I alluded to earlier, sequences in metric spaces are super nice,
but sequences in general topological spaces \emph{suck} (to the point where
I didn't bother to define convergence of general sequences).
So it's the second definition that will be used for general spaces.

\section{Definition of sequential compactness}
\prototype{$[0,1]$ is compact, but $(0,1)$ is not.}
To emphasize, compactness is one of the
\emph{best} possible properties that a metric space can have.
\begin{definition}
	A \vocab{subsequence} of an infinite sequence
	$x_1, x_2, \dots$ is exactly what it sounds like:
	a sequence $x_{i_1}, x_{i_2}, \dots$
	where $i_1 < i_2 < \cdots$ are positive integers.
	Note that the sequence is required to be infinite.
\end{definition}
Another way to think about this is ``selecting infinitely many terms''
or ``deleting some terms'' of the sequence, depending on whether
your glass is half empty or half full.

\begin{definition}
	A metric space $M$ is \vocab{sequentially compact} if
	every sequence has a subsequence which converges.
\end{definition}
This time, let me give some non-examples before the examples.
\begin{example}
	[Non-examples of compact metric spaces]
	\listhack
	\begin{enumerate}[(a)]
		\ii The space $\RR$ is not compact: consider the sequence $1,2,3,4,\dots$.
		Any subsequence explodes, hence $\RR$ cannot possibly be compact.
		\ii More generally, if a space is
		not bounded it cannot be compact.
		(You can prove this if you want.)
		\ii The open interval $(0,1)$ is bounded but not compact:
		consider the sequence $\frac12, \frac13, \frac14, \dots$.
		No subsequence can converge to a point in $(0,1)$ because the sequence ``converges to $0$''.
		\ii More generally, any space which is not complete cannot be compact.
	\end{enumerate}
\end{example}

Now for the examples!
\begin{ques}
	Show that a finite set is compact.
	(Pigeonhole Principle.)
\end{ques}
\begin{example}[Examples of compact spaces]
	Here are some more examples of compact spaces.
	I'll prove they're compact in just a moment;
	for now just convince yourself they are.
	\begin{enumerate}[(a)]
		\ii $[0,1]$ is compact. Convince yourself of this!
		Imagine having a large number of dots in the unit interval\dots
		\ii The surface of a sphere, $S^2 = \left\{ (x,y,z) \mid x^2+y^2+z^2=1 \right\}$ is compact.
		\ii The unit ball $B^2 = \left\{ (x,y) \mid x^2+y^2 \le 1 \right\}$ is compact.
		\ii The \vocab{Hawaiian earring} living in $\RR^2$ is compact:
		it consists of mutually tangent circles of radius $\frac 1n$ for each $n$,
		as in \Cref{fig:hawaiian}.
	\end{enumerate}
\end{example}
\begin{figure}[ht]
	\centering
	\begin{asy}
		size(4cm);
		for (int n=1; n<=15; ++n) draw(CP(dir(0)/n, origin));
		fill(CP(dir(0)/15, origin), black);
	\end{asy}
	\caption{Hawaiian Earring.}
	\label{fig:hawaiian}
\end{figure}

To aid in generating more examples, we remark:
\begin{proposition}[Closed subsets of compacts]
	Closed subsets of sequentially compact sets are compact.
\end{proposition}
\begin{ques}
	Prove this. (It should follow easily from definitions.)
\end{ques}

We need to do a bit more work for these examples, which we do in the next section.

\section{Criteria for compactness}
%Quick note: right now I've only defined compactness for metric spaces.
%In the next section I'll define compactness for general spaces, but
%all the results in this section will still remain true.
%However the proofs become much harder (in particular, \Cref{thm:tychonoff}
%becomes notoriously difficult).
%So you should assume all spaces in this section are metric spaces.

\begin{theorem}
	[Tychonoff's theorem]
	\label{thm:tychonoff}
	If $X$ and $Y$ are compact spaces, then so is $X \times Y$.
\end{theorem}
\begin{proof}
	\Cref{prob:tychonoff}.
\end{proof}

We also have:
\begin{theorem}[The interval is compact]
	\label{thm:interval_compact}
	$[0,1]$ is compact.
\end{theorem}
\begin{proof}
	Killed by \Cref{thm:bzw};
	however, here is a sketch of a direct proof.
	Split $[0,1]$ into $[0,\half] \cup [\half,1]$.
	By Pigeonhole, infinitely many terms of the sequence lie in the left half (say);
	let $x_1$ be the first one and then keep only the terms in the left half after $x_1$.
	Now split $[0, \half]$ into $[0,\frac14] \cup [\frac14,\half]$.
	Again, by Pigeonhole, infinitely many terms fall in some half; pick one of them, call it $x_2$.
	Rinse and repeat.
	In this way we generate a sequence $x_1$, $x_2$, \dots which is Cauchy,
	implying that it converges since $[0,1]$ is complete.
\end{proof}

Now we can prove the main theorem about Euclidean space:
in $\RR^n$, compactness is equivalent to being ``closed and bounded''.
\begin{theorem}[Bolzano-Weierstra\ss]
	A subset of $\RR^n$ is compact if and only if it is closed and bounded.
	\label{thm:fakeBW}
\end{theorem}
\begin{ques}
	Why does this imply the spaces in our examples are compact?
\end{ques}
\begin{proof}
	Well, look at a closed and bounded $S \subseteq \RR^n$.
	Since it's bounded, it lives inside some box $[a_1, b_1] \times [a_2, b_2] \times \dots \times [a_n, b_n]$.
	By Tychonoff's theorem, since each $[a_i, b_i]$ is compact the entire box is.
	Since $S$ is a closed subset of this compact box, we're done.
\end{proof}

One really has to work in $\RR^n$ for this to be true!
In other spaces, this criterion can easily fail.
\begin{example}[Closed and bounded but not compact]
	Let $S = \{s_1, s_2, \dots\}$ be any infinite set equipped with the discrete metric.
	Then $S$ is closed (since all convergent sequences are constant sequences)
	and $S$ is bounded (all points are a distance $1$ from each other)
	but it's certainly not compact since the sequence $s_1, s_2, \dots$ doesn't converge.
\end{example}

The Bolzano-Weierstrass theorem, which is \Cref{thm:bzw}, tells you exactly
which sets are compact in metric spaces in a geometric way.

\section{Compactness using open covers}
\prototype{$[0,1]$ is compact.}
There's a second related notion of compactness which I'll now define.
The following definitions might appear very unmotivated, but bear with me.
\begin{definition}
	An open cover of a topological space $X$
	is a collection of open sets $\{U_\alpha\}$
	(possibly infinite or uncountable) which \emph{cover} it:
	every point in $X$ lies in at least one of the $U_\alpha$,
	so that \[ X = \bigcup U_\alpha. \]
	Such a cover is called an \vocab{open cover}.

	A \vocab{subcover} is exactly what it sounds like:
	it takes only some of the $U_\alpha$,
	while ensuring that $X$ remains covered.
\end{definition}

Some art:
\begin{center}
\begin{asy}
size(12cm);
path blob = (-8,3)..(-10,1)..(-9.4,0)..(-8.2,-3)..(-8,-4)
	..(-2,-4.3)..(2,-4.2)..(8,-4)
	..(8.6,-2)..(8.2,0.5)..(8,3)
	..(4,3.3)..(0,3.1)..(-6,2.9)..cycle;
void open_ball(pair O, real r, pen p) {
	dot(O, p);
	filldraw(CR(O, r), p+opacity(0.1), p+dashed);
}

filldraw(blob, cyan+opacity(0.2), blue+0.7);
open_ball((-7,0.8), 3.5, red);
open_ball((-4.7,-1.2), 4.6, orange);
open_ball((-1.7,-0.3), 3.9, red);
open_ball((1.4,0.9), 2.9, yellow);
open_ball((2.5,-0.9), 3.7, orange);
open_ball((0.4,-3.7), 1.3, yellow);
open_ball((4.3,0.8), 2.7, red);
open_ball((6.3,1.8), 2.4, orange);
open_ball((6.0,-2.4), 4.0, yellow);

label("$X$", (8,3), dir(5), blue);

label(scale(2)*"$X = \bigcup_\alpha U_\alpha$", (0,4), blue);
\end{asy}
\end{center}

\begin{definition}
	A topological space $X$ is \vocab{quasicompact}
	if \emph{every} open cover has a finite subcover.
	It is \vocab{compact} if it is also Hausdorff.
\end{definition}
\begin{remark}
	The ``Hausdorff'' hypothesis that I snuck in
	is a sanity condition which is not worth worrying about unless you're
	working on the algebraic geometry chapters,
	since all the spaces you will deal with are Hausdorff.
	(In fact, some authors don't even bother to include it.)
	For example all metric spaces are Hausdorff
	and thus this condition can be safely ignored
	if you are working with metric spaces.
\end{remark}
What does this mean? Here's an example:
\begin{example}[Example of a finite subcover]
	Suppose we cover the unit square $M = [0,1]^2$ by
	putting an open disk of diameter $1$ centered at every point
	(trimming any overflow).
	This is clearly an open cover because,
	well, every point lies in \emph{many} of the open sets,
	and in particular is the center of one.

	But this is way overkill -- we only need about four
	of these circles to cover the whole square.
	That's what is meant by a ``finite subcover''.
	\begin{center}
		\begin{asy}
			size(4cm);
			draw(shift( (-0.5,-0.5) )*unitsquare, black+1);
			real d = 0.4;
			real r = 0.5;
			draw(CR(dir( 45)*d, r), dotted);
			draw(CR(dir(135)*d, r), dotted);
			draw(CR(dir(225)*d, r), dotted);
			draw(CR(dir(315)*d, r), dotted);
		\end{asy}
	\end{center}
\end{example}

Why do we care?
Because of this:
\begin{theorem}[Sequentially compact $\iff$ compact]
	A metric space $M$ is sequentially compact if and only if it is compact.
	\label{thm:compactness_metric}
\end{theorem}
We defer the proof to the last section.

This gives us the motivation we wanted for our definition.
Sequential compactness was a condition that made sense.
The open-cover definition looked strange,
but it turned out to be equivalent.
But we now prefer it, because we have seen that
whenever possible we want to resort to open-set-only based definitions:
so that e.g.\ they are preserved under homeomorphism.

\begin{example}[An example of non-compactness]
	The space $X = [0,1)$ is not compact in either sense. % chktex 9
	We can already see it is not sequentially compact, because it is not even complete (look at $x_n = 1 - \frac 1n$).
	To see it is not compact under the covering definition, consider the sets
	\[ U_m = \left[0, 1 - \frac{1}{m+1} \right) \] % chktex 9
	for $m = 1, 2, \dots$. Then $X = \bigcup U_i$; hence the $U_i$ are indeed a cover.
	But no finite collection of the $U_i$'s will cover $X$.
\end{example}
\begin{ques}
	Convince yourself that $[0,1]$ \emph{is} compact;
	this is a little less intuitive than it being sequentially compact.
\end{ques}
\begin{abuse}
	Thus, we'll never call a metric space ``sequentially compact'' again
	--- we'll just say ``compact''.
	(Indeed, I kind of already did this in the previous few sections.)
\end{abuse}

\section{Applications of compactness}
Compactness lets us reduce \emph{infinite} open covers to finite ones.
Actually, it lets us do this even if the open covers are \emph{blithely stupid}.
Very often one takes an open cover consisting
of an open neighborhood of $x \in X$ for every single point $x$ in the space;
this is a huge number of open sets,
and yet compactness lets us reduce to a finite set.

To give an example of a typical usage:
\begin{proposition}[Compact $\implies$ totally bounded]
	Let $M$ be compact. Then $M$ is totally bounded.
\end{proposition}
\begin{proof}[Proof using covers]
	For every point $p \in M$, take an $\eps$-neighborhood of $p$, say $U_p$.
	These cover $M$ for the horrendously stupid reason that each point $p$ is
	at the very least covered by its open neighborhood $U_p$.
	Compactness then lets us take a finite subcover.
\end{proof}

Next, an important result about maps between compact spaces.
\begin{theorem}[Images of compacts are compact]
	Let $f \colon X \to Y$ be a continuous function, where $X$ is compact.
	Then the image \[ f\im(X) \subseteq Y \] is compact.
\end{theorem}
\begin{proof}[Proof using covers]
	Take any open cover $\{V_\alpha\}$ in $Y$ of $f\im(X)$.
	By continuity of $f$,
	it pulls back to an open cover $\{U_\alpha\}$ of $X$.
	Thus some finite subcover of this covers $X$.
	The corresponding $V$'s cover $f\im(X)$.
\end{proof}
\begin{ques}
	Give another proof using the sequential definitions
	of continuity and compactness.
	(This is even easier.)
\end{ques}

Some nice corollaries of this:
\begin{corollary}
	[Extreme value theorem]
	Let $X$ be compact and consider a continuous function $f : X \to \RR$.
	Then $f$ achieves a \emph{maximum value} at some point,
	i.e.\ there is a point $p \in X$ such that $f(p) \ge f(q)$ for any
	other $q \in X$.
\end{corollary}
\begin{corollary}
	[Intermediate value theorem]
	Consider a continuous function $f: [0,1] \to \RR$.
	Then the image of $f$ is of the form $[a,b]$ for some real numbers $a \le b$.
\end{corollary}

\begin{proof}[Sketch of Proof]
	The point is that the image of $f$ is compact in $\RR$,
	and hence closed and bounded.
	You can convince yourself that the closed sets are just unions of closed intervals.
	That implies the extreme value theorem.

	When $X=[0,1]$, the image is also connected,
	so there should only be one closed interval in $f\im([0,1])$.
	Since the image is bounded, we then know it's of the form $[a,b]$.
	(To give a full proof, you would use the so-called \emph{least upper bound}
	property, but that's a little involved for a bedtime story;
	also, I think $\RR$ is boring.)
\end{proof}

\begin{example}
	[$1/x$]
	The compactness hypothesis is really important here.
	Otherwise, consider the function
	\[ (0,1) \to \RR \quad \text{ by } \quad
		x \mapsto \frac 1x. \]
	This function (which you plot as a hyperbola) is not bounded;
	essentially, you can see graphically that the issue
	is we can't extend it to a function on $[0,1]$ because it explodes near $x=0$.
\end{example}

One last application: if $M$ is a compact metric space,
then continuous functions $f : M \to N$
are continuous in an especially ``nice'' way:
\begin{definition}
	A function $f : M \to N$ of metric spaces
	is called \vocab{uniformly continuous}
	if for any $\eps > 0$, there exists a $\delta > 0$
	(depending only on $\eps$) such that
	whenever $d_M(x,y) < \delta$ we also have $d_N(fx, fy) < \eps$.
\end{definition}
The name means that for $\eps > 0$,
we need a $\delta$ that works for \emph{every point} of $M$.
\begin{example}[Uniform continuity]
	\listhack
	\begin{enumerate}[(a)]
		\ii The functions $\RR$ to $\RR$ of the form
		$x \mapsto ax+b$ are all uniformly continuous,
		since one can always take $\delta = \eps/|a|$ (or $\delta=1$ if $a=0$).
		\ii Actually, it is true that a differentiable function $\RR \to \RR$
		with a bounded derivative is uniformly continuous.
		(The converse is false for the reason that uniformly continuous
		doesn't imply differentiable at all.)
		\ii The function $f : \RR \to \RR$ by $x \mapsto x^2$
		is \emph{not} uniformly continuous, since for large $x$,
		tiny $\delta$ changes to $x$ lead to fairly large changes in $x^2$.
		(If you like, you can try to prove this formally now.)

		Think $f(2017.01) - f(2017) > 40$;
		even when $\delta = 0.01$, one can still cause large changes in $f$.

		\ii However, when restricted to $(0,1)$ or $[0,1]$
		the function $x \mapsto x^2$ becomes uniformly continuous.
		(For $\eps > 0$ one can now pick for example $\delta = \min\{1,\eps\}/3$.)

		\ii The function $(0,1) \to \RR$ by $x \mapsto 1/x$ is \emph{not}
		uniformly continuous (same reason as before).
	\end{enumerate}
\end{example}

Now, as promised:
\begin{proposition}[Continuous on compact $\implies$ uniformly continuous]
	If $M$ is compact and $f \colon M \to N$ is continuous,
	then $f$ is uniformly continuous.
\end{proposition}
\begin{proof}[Proof using sequences]
	Fix $\eps > 0$, and assume for contradiction that for every $\delta = 1/k$
	there exists points $x_k$ and $y_k$ within $\delta$ of each
	other but with images $\eps > 0$ apart.
	By compactness, take a convergent subsequence $x_{i_k} \to p$.
	Then $y_{i_k} \to p$ as well, since the $x_k$'s and $y_k$'s are close to each other.
	So both sequences $f(x_{i_k})$ and $f(y_{i_k})$ should converge to $f(p)$ by sequential continuity,
	but this can't be true since the two sequences are always $\eps$ apart.
\end{proof}

\section{(Optional) Equivalence of formulations of compactness}
We will prove that:
\begin{theorem}
	[Heine-Borel for general metric spaces]
	For a metric space $M$, the following are equivalent:
	\begin{enumerate}[(i)]
		\ii Every sequence has a convergent subsequence,
		\ii The space $M$ is complete and totally bounded, and
		\ii Every open cover has a finite subcover.
	\end{enumerate}
\end{theorem}
We leave the proof that (i) $\iff$ (ii) as \Cref{thm:bzw};
the idea of the proof is much in the spirit of \Cref{thm:interval_compact}.
\begin{proof}
	[Proof that (i) and (ii) $\implies$ (iii)]
	We prove the following lemma, which is interesting in its own right.
	\begin{lemma}
		[Lebesgue number lemma]
		Let $M$ be a compact metric space and $\{U_\alpha\}$ an open cover.
		Then there exists a real number $\delta > 0$,
		called a \vocab{Lebesgue number} for that covering,
		such that the $\delta$-neighborhood of any point
		$p$ lies entirely in some $U_\alpha$.
	\end{lemma}
	\begin{subproof}[Proof of lemma]
		Assume for contradiction that for every $\delta = 1/k$
		there is a point $x_k \in M$
		such that its $1/k$-neighborhood isn't contained in any $U_\alpha$.
		In this way we construct a sequence $x_1$, $x_2$, \dots;
		thus we're allowed to take a subsequence which converges to some $x$.
		Then for every $\eps > 0$ we can find an integer $n$ such that $d(x_n, x) + 1/n < \eps$;
		thus the $\eps$-neighborhood at $x$ isn't contained in any $U_\alpha$ for every $\eps > 0$.
		This is impossible, because we assumed $x$ was covered by some open set.
	\end{subproof}
	Now, take a Lebesgue number $\delta$ for the covering.
	Since $M$ is totally bounded, finitely many $\delta$-neighborhoods cover the space,
	so finitely many $U_\alpha$ do as well.
\end{proof}

\begin{proof}
	[Proof that (iii) $\implies$ (ii)]
	One step is immediate:
	\begin{ques}
		Show that the covering condition $\implies$ totally bounded.
	\end{ques}
	The tricky part is showing $M$ is complete.
	Assume for contradiction it isn't and thus that the sequence $(x_k)$ is Cauchy,
	but it doesn't converge to any particular point.
	\begin{ques}
		Show that this implies for each $p \in M$, there is an $\eps_p$-neighborhood $U_p$
		which contains at most finitely many of the points of the sequence $(x_k)$.
		(You will have to use the fact that $x_k \not\to p$ and $(x_k)$ is Cauchy.)
	\end{ques}
	Now if we consider $M = \bigcup_p U_p$ we get a
	finite subcover of these open neighborhoods;
	but this finite subcover can only cover finitely
	many points of the sequence, by contradiction.
\end{proof}

\section{\problemhead}
The later problems are pretty hard;
some have the flavor of IMO 3/6-style constructions.
It's important to draw lots of pictures so one can tell what's happening.
% I'd be happy to learn of any easier ones.
Of these \Cref{thm:bzw} is definitely my favorite.

\begin{problem}
	Show that the closed interval $[0,1]$ and
	open interval $(0,1)$ are not homeomorphic.
	\begin{hint}
		$[0,1]$ is compact.
	\end{hint}
	\begin{sol}
		Compactness is preserved under homeomorphism,
		but $[0,1]$ is compact while $(0,1)$ is not.
	\end{sol}
\end{problem}

\begin{problem}
	Let $X$ be a topological space with the discrete topology.
	Under what conditions is $X$ compact?
	\begin{hint}
		If and only if it is finite.
	\end{hint}
\end{problem}

\begin{problem}
	[The cofinite topology is quasicompact only]
	We let $X$ be an infinite set and equip it with the
	\vocab{cofinite topology}:
	the open sets are the empty set and complements of finite sets.
	This makes $X$ into a topological space.
	Show that $X$ is quasicompact but not Hausdorff.
\end{problem}

\begin{problem}
	[Cantor's intersection theorem]
	\label{prob:cantor_intersect}
	Let $X$ be a compact topological space, and suppose
	\[ X = K_0 \supseteq K_1 \supseteq K_2 \supseteq \dots \]
	is an infinite sequence of nested nonempty closed subsets.
	Show that $\bigcap_{n \ge 0} K_n \neq \varnothing$.
\end{problem}

%\begin{sproblem}[Compact Implies Bounded]
%	Let $f : X \to \RR$ be a continuous function,
%	where $X$ is a compact topological space.
%	Show that $f$ is bounded.
%	\begin{hint}
%		Immediate by the fact that the image of $f$ is compact,
%		and hence bounded.
%		Remember this!
%	\end{hint}
%\end{sproblem}

\begin{problem}
	[Tychonoff's theorem]
	Let $X$ and $Y$ be compact metric spaces. Show that $X \times Y$ is compact.
	(This is also true for general topological spaces,
	but the proof is surprisingly hard,
	and we haven't even defined $X \times Y$ in general yet.)
	\label{prob:tychonoff}
	\begin{hint}
	Suppose $p_i = (x_i, y_i)$ is a sequence in $X \times Y$ ($i=1,2,\dots$).
	Take a sub-sequence such that the $x$-coordinate converges
	(throwing out some terms).
	Then take a sub-sequence of \emph{that} sub-sequence
	such that $y$-coordinate converges (throwing out more terms).
	\end{hint}
	\begin{sol}
	Suppose $p_i = (x_i, y_i)$ is a sequence in $X \times Y$ ($i=1,2,\dots$).
	Looking on the $X$ side, some subsequence converges:
	for the sake of illustration say it's $x_1, x_4, x_9, x_{16}, \dots \to x$.
	Then look at the corresponding sequence $y_1, y_4, y_9, y_{16}, \dots$.
	Using compactness of $Y$, it has a convergent subsequence, say
	$y_1, y_{16}, y_{81}, y_{256}, \dots \to y$.
	Then $p_1, p_{16}, p_{81}, \dots$ will converge to $(x,y)$.

	One common mistake is to just conclude
	that $(x_n)$ has a convergent subsequence
	and that $(y_n)$ does too.
	But these sequences could be totally unrelated.
	For this proof to work,
	you do need to apply compactness of $X$ first,
	and then compactness of $Y$ on the resulting \emph{filtered}
	sequence like we did here.
	\end{sol}
\end{problem}

\begin{dproblem}
	[Bolzano-Weierstra\ss\ theorem for general metric spaces]
	\gim
	Prove that a metric space $M$ is sequentially compact
	if and only if it is complete and totally bounded.
	\label{thm:bzw}
	\begin{hint}
		Mimic the proof of \Cref{thm:interval_compact}.
		The totally bounded condition lets you do Pigeonhole.
	\end{hint}
\end{dproblem}

\begin{problem}
	[Almost Arzel\`a-Ascoli theorem]
	\gim
	Let
	$f_1, f_2, \ldots \colon [0,1] \to [-100,100]$
	be an \vocab{equicontinuous} sequence of functions, meaning
	\[
		\forall \eps > 0 \quad
		\exists \delta > 0 \quad
		\forall n \;
		\forall x,y \quad
		\left( \left\lvert x-y \right\rvert <\delta
		\implies \left\lvert f_n(x) - f_n(y) \right\rvert < \eps \right)
	\]
	Show that we can extract a subsequence $f_{i_1}, f_{i_2}, \dots$
	of these functions such that for every $x \in [0,1]$,
	the sequence $f_{i_1}(x)$, $f_{i_2}(x)$, \dots\ converges.
\end{problem}

\begin{problem}
	\gim
	Let $M = (M,d)$ be a bounded metric space.
	Suppose that whenever $d'$ is another metric on $M$
	for which $(M,d)$ and $(M,d')$ are homeomorphic
	(i.e.\ have the same open sets), then $d'$ is also bounded.
	Prove that $M$ is compact.
\end{problem}

\begin{problem}
	\yod
	In this problem a ``circle''
	refers to the boundary of a disk with \emph{nonzero} radius.
	\begin{enumerate}[(a)]
		\ii Is it possible to partition the plane $\RR^2$
		into disjoint circles?
		\ii From the plane $\RR^2$ we delete two distinct points $p$ and $q$.
		Is it possible to partition the remaining points into disjoint circles?
	\end{enumerate}
	\begin{hint}
		The answer to both parts is no.

		For (a) use \Cref{prob:cantor_intersect}.

		For (b), color each circle in the partition
		based on whether it contains $p$ but not $q$,
		$q$ but not $p$, or both.
	\end{hint}
	\begin{sol}
		Part (a) follows by the Cantor intersection theorem
		(\Cref{prob:cantor_intersect}).
		Assume for contradiction such a partition existed.
		Take any of the circles $C_0$, and let $K_0$ denote the closed disk
		with boundary $C_0$.
		Now take the circle $C_1$ passing through the center of $C_0$,
		and let $K_1$ denote the closed disk with boundary $C_1$.
		If we repeat in this way,
		we get a nested sequence $K_0 \supseteq K_1 \supseteq \dots$
		and the radii of $C_i$ approach zero
		(since each is at most half the previous once).
		Thus some point $p$ lies in $\bigcap_n K_n$ which is impossible.

		Now for part (b),
		again assume for contradiction a partition into circles exists.
		Color a circle magenta if it contains $p$ but not $q$
		and color a circle cyan if it contains $q$ but not $p$.
		Color $p$ itself magenta and $q$ itself cyan as well.
		Finally, color a circle neon yellow if it contains both $p$ and $q$.
		(When we refer to coloring a circle,
		we mean to color all the points on it.)

		By repeating the argument in (a) there are no circles
		enclosing neither $p$ nor $q$.
		Hence every point is either magenta, cyan, or neon yellow.
		Now note that given any magenta circle,
		its interior is completely magenta.
		Actually, the magenta circles can be totally ordered
		by inclusion (since they can't intersect).
		So we consider two cases:
		\begin{itemize}
		 \ii If there is a magenta circle which is maximal by inclusion
		 (i.e.\ a magenta circle not contained in any other magenta circle)
		 then the set of all magenta points is just a closed disk.
		 \ii If there is no such magenta circle,
		 then the set of magenta points can also be expressed
		 as the union over all magenta circles of their interiors.
		 This is a union of open sets, so it is itself open.
		 \end{itemize}

		We conclude the set of magenta points is
		either a closed disk or an open set.
		Similarly for the set of cyan points.
		Moreover, the set of such points is convex.

		To finish the problem:
		\begin{itemize}
		\ii Suppose there are no neon yellow points.
		If the magenta points form a closed disk,
		then the cyan points are $\mathbb R^2$ minus a disk which is not convex.
		Contradiction. So the magenta points must be open.
		Similarly the cyan points must be open.
		But $\mathbb R^2$ is connected,
		so it can't be written as the union of two open sets.
		\ii Now suppose there are neon yellow points.
		We claim there is a neon yellow circle minimal by inclusion.
		If not, then repeat the argument of (a) to get a contradiction,
		since any neon yellow circle must have diameter the distance from $p$ to $q$.
		So we can find a neon yellow circle $\mathscr C$ whose
		interior is all magenta and cyan.
		Now repeat the argument of the previous part,
		replacing $\mathbb R^2$ by the interior of $\mathscr C$.
		 \end{itemize}
	\end{sol}
\end{problem}


\chapter{More properties of the discriminant}
I'll remind you that the discriminant of a number field $K$ is given by
\[
\Delta_K \defeq \det
\begin{bmatrix}
	\sigma_1(\alpha_1) & \dots & \sigma_n(\alpha_1) \\
	\vdots & \ddots & \vdots \\
	\sigma_1(\alpha_n) & \dots & \sigma_n(\alpha_n) \\
\end{bmatrix}^2
\]
where $\alpha_1$, \dots, $\alpha_n$ is a $\ZZ$-basis for $K$,
and the $\sigma_i$ are the $n$ embeddings of $K$ into $\CC$.

Several examples, properties, and equivalent definitions follow.

\section\problemhead
\begin{sproblem}[Discriminant of cyclotomic field]
	\label{prob:discrim_cyclotomic_field}
	Let $p$ be an odd rational prime and $\zeta_p$ a primitive $p$th root of unity.
	Let $K = \QQ(\zeta_p)$.
	Show that \[ \Delta_K = (-1)^{\frac{p-1}{2}} p^{p-2}. \]
	\begin{hint}
		Direct linear algebra computation.
	\end{hint}
\end{sproblem}

\begin{sproblem}[Trace representation of $\Delta_K$]
	\gim
	Let $\alpha_1$, \dots, $\alpha_n$ be a basis for $\OO_K$.
	Prove that
	\[ \Delta_K
		=
		\det
		\begin{bmatrix}
			\TrK(\alpha_1^2) & \TrK(\alpha_1\alpha_2) & \dots & \TrK(\alpha_1\alpha_n) \\
			\TrK(\alpha_2\alpha_1) & \TrK(\alpha_2^2) & \dots & \TrK(\alpha_2\alpha_n) \\
			\qquad\vdots & \qquad\vdots & \ddots & \qquad\vdots \\
			\TrK(\alpha_n\alpha_1) & \TrK(\alpha_n\alpha_2) & \dots & \TrK(\alpha_n\alpha_n) \\
		\end{bmatrix}.
	\]
	In particular, $\Delta_K$ is an integer.
	\label{prob:trace_discriminant}
	\begin{hint}
		Let $M$ be the ``embedding'' matrix.
		Look at $M^\top M$, where $M^\top$ is the transpose matrix.
	\end{hint}
\end{sproblem}

\begin{sproblem}[Root representation of $\Delta_K$]
	The \vocab{discriminant} of a quadratic polynomial $Ax^2+Bx+C$ is defined as $B^2-4AC$.
	More generally, the polynomial discriminant of a polynomial $f \in \ZZ[x]$ of degree $n$ is
	\[ \Delta(f) \defeq c^{2n-2} \prod_{1 \le i < j \le n} \left( z_i - z_j \right)^2 \]
	where $z_1, \dots, z_n$ are the roots of $f$, and $c$ is the leading coefficient of $f$.

	Suppose $K$ is monogenic with $\OO_K = \ZZ[\theta]$.
	Let $f$ denote the minimal polynomial of $\theta$ (hence monic).
	Show that \[ \Delta_K = \Delta(f). \]
	\label{prob:root_discriminant}
	\begin{hint}
		Vandermonde matrices.
	\end{hint}
\end{sproblem}

\begin{problem}
	Show that if $K \neq \QQ$ is a number field then $\left\lvert \Delta_K \right\rvert > 1$.
	\begin{hint}
		$M_K \ge 1$ must hold. Bash.
	\end{hint}
\end{problem}

\begin{problem}
	[Brill's theorem]
	For a number field $K$ with signature $(r_1, r_2)$, show that
	$\Delta_K > 0$ if and only if $r_2$ is even.
\end{problem}

\begin{problem}
	[Stickelberger theorem]
	\kurumi
	Let $K$ be a number field. Prove that \[ \Delta_K \equiv 0 \text{ or } 1 \pmod 4. \]
	% P N
\end{problem}


\documentclass{scrartcl}

\usepackage{riley}
\usepackage[tt={monowidth=true}]{cfr-lm}
%\usepackage{riley-libertine}
\usepackage{graphicx}

\newcommand{\measpace}{\Omega}
\newcommand{\system}{\mathcal D}
\newcommand{\propersetminus}{\begin{smallmatrix} \setminus\\\supseteq \end{smallmatrix}}%{\mathop{\setminus}\limits_{\supseteq}}
\DeclareMathOperator*{\bigdisjunion}{\coprod}
\newcommand{\disjunion}{\mathbin{\amalg}}
\newcommand{\ordinals}{\texttt{Ord}}
\newcommand{\inclass}{\in}
\newcommand{\notinclass}{\notin}

\DeclareMathOperator{\successor}{succ}

\usepackage{import}
\newcommand{\figinput}[1]{\import{figs/}{#1}}

\begin{document}

\section{Ordinals and transfinite induction}
\newcommand{\setofords}{\mathcal X}
\begin{defn}[The ordinals, \(\ordinals\)]
  These are defined inductively:
  \begin{itemize}
  \item \(0\define\emptyset \inclass \ordinals\)
  \item If \(n\inclass\ordinals\) then
    \[
      \successor(n)\define n \cup \{n\} \inclass \ordinals
    \]
  \item If \(\setofords \subseteq \ordinals\) is a set, then
    \[
      \sup \setofords \define \bigcup\setofords \inclass \ordinals
    \]
  \end{itemize}
  In other words, an ordinal is the set of all smaller ordinals.
In particular, the last definition precludes the ordinals from being a set.
\(\ordinals\) is bigger than any set. They form a class.

If \(\alpha = \successor(n)\) then \(\alpha\) is a \emph{successor ordinal}.
Otherwise, \(\alpha\) is a \emph{limit ordinal}.
\end{defn}
\begin{theorem}[\(\ordinals\) is well-ordered]\label{ordinals are well ordered}
  Every nonempty class of ordinals contains a least element.
\end{theorem}
\begin{proof}
  Let \(\setofords\) be a nonempty class of ordinals. Then it suffices to show
  \[
   I\define\inf\setofords = \bigcap \setofords \in\setofords
  \]
  Suppose \(I\notin\setofords\). Then for any \(x\in\setofords\), \(I<x\).
  Consequently, for any \(x\in\setofords\), \(\successor(I)\leq x\). But then
  \(\successor(I)=I\), a contradiction.
\end{proof}

\begin{theorem}[Transfinite induction]
  For a logical statement \(\phi\), if
  \begin{enumerate}
  \item \(\phi(0)\) holds,
  \item \(\phi(n)\implies\phi(\successor(n))\) \footnote{strictly speaking, this follows
    from (1) and  (3) but is usually easier to prove first}
  \item \label{strong induction}If, for all \(n<\alpha\), \(\phi(n)\) is true then \(\phi(\alpha)\) is true:
    \[
      (\forall n<\alpha) \phi(n)\implies \phi(\alpha)
    \]
  \end{enumerate}
  Then \(\phi\) holds for all ordinals.
\end{theorem}
\begin{proof}
  \newcommand{\badset}{\setofords}
  Let \(\badset\) be the class where \(\phi\) does not hold. Suppose
  \(\badset\neq\emptyset\). Then let for all ordinals \(n< \inf\badset\),
  \(\phi(n)\) by hypothesis. Hence, by (3), \(\phi(\inf\badset)\) holds. But then
  \(\inf\badset\notin\badset\), contradicting \namecref{ordinals are well ordered}.
  Therefore, \(\badset\) is empty.
\end{proof}

\begin{theorem}[\(\ordinals\) fixed point theorem] \label{ordinal fixed point theorem}
  Let \((X,\leq)\) be a complete ordered space.
  Suppose \(F:X\to X\) is nondecreasing.
  Define
  \begin{align*}
    F^0(x)&\define x \\
    F^{\successor(n)}(x)&\define F(F^n(x))
    \shortintertext{If \(\alpha\) is a limit ordinal,}
    F^{\alpha}(x) &\define \sup_{\beta < \alpha} F^\beta(x)
  \end{align*}
  Then for any \(x\in X\), \(F^\alpha(x)\)  is eventually the smallest fixed
  point of \(F\) greater than or equal to \(x\).
\end{theorem}
\begin{proof}
  Fix \(x\). Then \(F^\blank(x):\ordinals\to X\) is nondecreasing by
  construction. Because \(\ordinals\) is bigger than any set, \(F^\blank(x)\)
  cannot be injective. Hence there is some smallest \(\alpha\inclass\ordinals\)
  such that \(F^\alpha(x)=F^{\gamma}(x)\) with \(\gamma > \alpha\). But
  \(\alpha<\successor(\alpha)\leq\beta\), so by monotonicity,
  \[
    F^{\alpha}(x)\leq F^{\successor(\alpha)}(x) \leq F^{\beta}(x)=F^{\alpha}(x)
  \]
  hence
  \[
    F(F^\alpha(x)) = F^{\successor(\alpha)}(x)=F^{\alpha}(x)
  \]
  so \(F\) fixes \(F^{\alpha}(x)\).

  Suppose \(y\) is a fixed point and \(x\leq y\leq F^\alpha(x)\). Then there is
  some smallest \(\beta\) such that \(y\leq F^\beta(x)\):
  \begin{itemize}
  \item If \(\beta=\successor(\gamma)\), then
    \[
      F^\gamma(x) \leq y \leq f(F^\gamma(x)) = F^\beta(x)
    \]
    But by monotonicity, \(F(F^\gamma(x))\leq f(y) =y\) by assumption \(y\) is fixed.
    Thus \(y=F^\gamma(x)\) for some \(\gamma\leq\alpha\).
  \item If \(\beta\) is a limit ordinal, then  for all \(\gamma <\beta\),
    \[
      F^\gamma(x) \leq y \leq F^\beta(x)=\sup_{\hat\gamma < \beta} F^{\hat\gamma}(x) = y
    \]
    so \(F^{\beta}(x)=y\).
  \end{itemize}
\end{proof}
By characterizing \(\sigma\)-algebras and \(\lambda\)-systems as fixed points of
set functions, this result becomes a powerful tool for proving results about
generated \(\sigma\)-algebras and \(\lambda\)-systems.
\section{\(\pi\)--\(\lambda\) systems and theorem}
Special notation:
\begin{itemize}
\item I'll use \(\disjunion\) to represent \emph{disjoint} unions,
\item  \(\upunion\) to
  represent \emph{increasing} unions,
\item and \(\propersetminus\) to represent
  \emph{proper} relative complements: \(A\setminus B\) when \(B\subseteq A\).
\end{itemize}
\begin{defn}[\(\lambda\)-system aka Dynkin system] \label{original dynkin system def}
  \(\system\) is a Dynkin system on \(\measpace\) iff
  \begin{itemize}
  \item \(\measpace\in\system\)
  \item Closure under \emph{proper} relative complements:
    \[
      A\subseteq B \implies B\propersetminus A \in \system
    \]
  \item  Closure under \emph{increasing} \(\sigma\)-unions:
    \[
      A_1\subseteq A_2 \subseteq \dots \in \system \implies \upunion A_n \in \system
    \]
  \end{itemize}
\end{defn}
Allowing arbitrary complements would give a \(\sigma\)-algebra as
\[
  (\measpace\setminus A)\setminus B = (A\cup B)^C = A^C \cap B^C
\]
allowing arbitrary \(\sigma\)--unions and intersections. \\
\figinput{relcomp.pdf_tex}

\begin{theorem}[Alternative definition of \(\lambda\)-system] \label{equivalent dynkin system def}
  Equivalently,
  \begin{itemize}
  \item \(\measpace\in\system\)
  \item Closure under \emph{global} complements: \[A\in \system \implies A^C\in\system\]
  \item Closure under \emph{disjoint} \(\sigma\)-unions:
    \[
      \{A_i\in \system\}, A_i\cap A_j=\emptyset \implies \bigdisjunion A_i \in \system
    \]
  \end{itemize}
\end{theorem}
\begin{proof}
  Suppose \(\system\) meets the criteria of \namecref{equivalent dynkin system def}.
 Then \(\system\) contains proper relative complements as \(A\propersetminus B =
 (A^C\cup B)^C\): \\
 \figinput{proper-comp-from-global-comp.pdf_tex}\\
 But by taking proper relative complements, we can convert an ascending
 \(\sigma\)-union into a disjoint \(\sigma\)-union: if \(A_n\uparrow A\), let
 \begin{align*}
   B_1 &= A_1 \in\system\\
   B_n &= A_n\propersetminus \bigcup_{i<n}A_{i} \in\system\\
         \shortintertext{Then \(\{B_n\}\) are pairwise-disjoint and}
    A &= \upunion A_n =  \bigdisjunion_n B_n \in \system
 \end{align*}
 proving sufficiency.

 For necessity, suppose \(\system\) satisfies the criteria of
 \namecref{original dynkin system def}. As \(A^C =\measpace\setminus A\), a proper
 complement, it suffices to show closure under disjoint unions. First, note that
 it is closed under finite disjoint unions:
 \[
   A \disjunion B = ((\measpace\propersetminus A)\propersetminus B)^C \in \system
 \]\\
 \figinput{disjointbinunion-from-relcomp.pdf_tex}\\
 because the complements are proper by disjointness.
 Suppose \(B_i\)
 are pairwise disjoint. Then \(B=\bigdisjunion B_i\in\system\) by hypothesis.
 \begin{align*}
   A_1 &= B_1 \in\system \\
   A_n &= B_n\disjunion A_{n-1} \in \system
         \shortintertext{therefore}
   B&=\bigdisjunion B_n=\upunion A_n \in\system
 \end{align*}
\end{proof}
\begin{defn}[\(\pi\)-system]
  A \(\pi\)-system is closed under finite intersections.
\end{defn}
\begin{theorem}
  If \(\system\) is a \(\pi\)-system and \(\lambda\)-system then it is a \(\sigma\)-algebra.
\end{theorem}
\begin{proof}
  First, prove \(\system\) is an algebra.
  Note that \(\system\) is closed under binary union as \(A\cup B = (A^C\cap
  B^C)^C \) and \(A^C,B^C\in\system\) because \(\system\) is a \(\lambda\)-system, so
  \(A^C\cap B^C\in\system\) because it is a \(\pi\)-system, and its complement is in
  \(\system\) by closure under complements. In particular, \(\system\) must be
  closed under \emph{arbitrary} relative complements as
  \[
    A\setminus B = A\propersetminus ({A\cap B})
  \]

  Suppose \(A_i\) is a countable sequence of sets in \(\system\). Then let
  \begin{align*}
    B_1 &= A_1 \\
    B_n &= A_n \setminus \bigdisjunion_{i<n}B_{i} \\
    \shortintertext{so \(B_i\) are pairwise-disjoint and}
    \bigcup A_n &= \bigdisjunion B_n \in \system
  \end{align*}
  proving closure under \(\sigma\)-unions.
\end{proof}

\newcommand{\lambdify}{\Lambda}
\newcommand{\lambdagen}{\lambda}
\newcommand{\pisystem}{P}
\newcommand{\dsystem}{\system}
\begin{theorem}[Dynkin's \(\pi\)--\(\lambda\) theorem]
  If \(\pisystem\subseteq \dsystem\) where \(\pi\) is a \(\pi\)-system and \(\dsystem\) is a
  \(\lambda\)-system, then
  \[
    \sigma(\pisystem) \subseteq \dsystem
  \]
\end{theorem}
\begin{proof}
  As \(\lambdagen(\pisystem)\) is the minimal \(\lambda\)-system containing
  \(\pisystem\),
  \[
    \lambdagen(\pisystem)\subseteq\dsystem
  \]
  It suffices to show \(\lambdagen(\pisystem)\) is a
  \(\pi\)-system because then it must be a \(\sigma\)-algebra.

  Let \(U(X)\) indicate the set of disjoint \(\sigma\)-unions in \(X\):
  \[
    U(X)\define\left\{\bigdisjunion_{i\in \N} A_i: A_i\in X, A_i\cap A_j=\emptyset\right\}
  \]
  Let \(C(X)\) indicate the set of complements in \(X\):
  \[
    C(X)\define\left\{ A^C:A\in X \right\}
  \]
  And define
  \[
    \lambdify(X)\define X\cup U(X) \cup C(X) = U(X)\cup C(X) :
    (2^{2^\measpace},\subseteq ) \to (2^{2^\measpace},\subseteq)
  \]
  By construction, \(\lambdify(X)=X\) iff \(X\) is a \(\lambda\)-system. By monotonicity
  (\(X\subseteq\lambdify(X)\)), the minimum fixed point of \(\lambdify\) containing \(X\)
  is the \(\lambda\)-system generated by \(X\). By \namecref{ordinal fixed point
    theorem},
  \newcommand{\stopord}{\omega}
  \[
    \lambdify^\stopord(\pisystem)=\lambda(\pisystem)
  \]
  for some \(\stopord\inclass\ordinals\).

  \newcommand{\nextord}{5}
  We show for any \(\alpha\in\ordinals\), the intersection of
  any two elements in \(\lambdify^\alpha(\pisystem)\) is in
  \(\lambdify^{\nextord(\alpha)}(\pisystem)\).

  \begin{enumerate}
  \item As $\nextord(0)\geq 0$, this is true for \(\alpha=0\) by the hypothesis that \(\pisystem\) is a \(\pi\)-system.
  \item Any element in \(\lambdify^{\successor(\alpha)}(\pisystem)\) can be written as
    \(A\), \(A^C\), or \(\bigdisjunion A_i\) with \(A,A_i\in\lambdify^{\alpha}\).
    Writing an intersection of elements in
    \(\lambdify^{\successor(\alpha)}\) in terms of proper complements and disjoint
    unions of intersections of elements in \(\Lambda^\alpha(\pisystem)\):
      \begin{align*}
        &A\cap B^C &&= A\propersetminus (B\cap A) = (A^{C} \disjunion \grayoverbrace{(B \cap A)}{\in\lambdify^{\nextord(\alpha)}(\pisystem)})^{C}
        &&\in \lambdify^{\nextord(\alpha)+3}(\pisystem) \\
        %
        &A^C\cap B^C &&= (A\cup B)^C = \left((A\cap B^C)\disjunion B  \right)^C
        &&\in \lambdify^{\nextord(\alpha)+5}(\pisystem) \\
        %
        & \left(\bigdisjunion_{i} A_i\right) \cap
        \left(\bigdisjunion_j B_j\right) &&= \bigdisjunion_{i,j}
                                            \grayoverbrace{A_i\cap B_j}{\in\lambdify^{\nextord(\alpha)}(\pisystem)}
        &&\in
        \lambdify^{\nextord(\alpha)+1}(\pisystem) \\
        %
       & A^C \cap \bigdisjunion B_j &&= \bigdisjunion \grayunderbrace{A^C\cap
           B_j}{\in\lambdify^{\nextord(\alpha)+3}(\pisystem)} &&\in\lambdify^{\nextord(\alpha)+4}(\pisystem)
      \end{align*}
          which are all in \(\lambdify^{\nextord(\alpha)+5}(\pisystem)= \lambdify^{\nextord(\alpha+1)}\).
  \item
    If \(\alpha\) is a limit ordinal, then set
    \[
      \nextord\alpha \geq \nextord\sup_{\beta <\alpha}\beta = \sup_{\beta <
        \alpha}\nextord\beta
    \]
    For any two \(A,B\in\lambdify^\alpha\) there must be some \(n_A,n_B<\alpha\)
    such that \(A\in \lambdify^{n_A}(\pisystem)\) and
    \(B\in\lambdify^{n_B}(\pisystem)\). Hence, if \(n=\max(n_A,n_B)\), then
    \(A\cap B\in \lambdify^{\nextord n}(\pisystem)\subseteq
    \lambdify^{\nextord\alpha}(\pisystem)\).
  \end{enumerate}
  But this process terminates at \(\omega\), hence
  \(\lambdify^{3\omega}(\pisystem)=\lambdify^\omega(\pisystem)=\lambdagen(\pisystem)\);
  \(\lambdagen(\pisystem)\) is closed under intersections.
\end{proof}
\end{document}


\documentclass[11pt]{lab-exercise}

\begin{document} %****************************************************
\maketitle
\pagestyle{exercise-style}
\thispagestyle{empty}
%=============================================
Download the file \verb|lab_14_files.zip|, un-zip it.
%---------------------------------------------
\section{Direction Fields and Solution Curves}
%---------------------------------------------
Given the following ODE and the initial condition,
$$
\frac{dy}{dt} = -y(t) - 5 e^{-t} \sin(5t), \quad y(0) = -2, \quad t \in [0, 3].
$$
\begin{enumerate}[a.]
    \item In \verb|lab_14_script.m|, define anoymous function \verb`dydt` to be the right-hand side of the ODE.
    \item Define \verb`a`, \verb`b` to be the left and right endpoint of the interval $[0, 3]$, respectively.
    \item Define \verb`tStep` to be the step size $\Delta t = 0.01$.
    \item Define \verb`tSpan` to be a vector starting from \verb`a` to \verb`b` with step size \verb`tStep` using colon notation.
    \item At line 20, use \verb`ode23` to solve the ODE.
    \item At line 40, plot \verb`tSol(:, i)` against \verb`ySol(:, i)` with line style specified in the cell array \verb`LineStyle`.
    \item Run the script and see whether it works. If it does work, add more initial conditions at line 17: $y(0) = 0$, $y(0) = 2$, $y(0) = 4$.
\end{enumerate}
%---------------------------------------------
\section{System of ODEs}
%---------------------------------------------
Next, use the built-in ODE solver \verb`ode45` to solve the system of ODEs for Problem 4 of Homework 7. The system of ODEs is as follows:
$$
\begin{cases}
    y'_1(t) = y_3, \\
    y'_2(t) = y_4, \\
    y'_3(t) = -2 y_1 + (3/2) y_2, \\
    y'_4(t) = (4/3) y_1 - 3 y_2, \\
\end{cases}
\quad
\mathbf{y}(0) =
\begin{bmatrix}
    -1 \\ 4 \\ 1 \\ 1
\end{bmatrix},
\quad
t \in [0, 15].
$$
\begin{enumerate}[a.]
    \item Uncomment line 49 through 88, define an anoymous function (you can refer to the example in reference page for \verb`ode45`. To open the reference page, type \verb`doc ode45` in the Command Window).
    \item Repeat the steps in Part I to define \verb`a`, \verb`b`, \verb`tStep`, \verb`tSpan`, and \verb`y0`.
    \item Use \verb`ode45` to solve the system of ODEs at line 57.
    \item At line 63, plot \verb`t` against \verb`y(:, i)` with line style specified in the cell array \verb`LineStyle`.
    \item At line 73, plot \verb`y(:, 1)` against \verb`y(:, 3)`.
\end{enumerate}
%---------------------------------------------
At last, run the script \verb|lab_14_script.m|. Upload the script file \verb|lab_14_script.m|, and figure files \verb|lab_14_figure_1.pdf|, \verb|lab_14_figure_2.pdf|, \verb|lab_14_figure_3.pdf| to Overleaf. Recompile, and submit the generated \verb|.pdf| file on WyoCourses.

\section{Lipsum}
\lipsum.
\end{document}


\documentclass[12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{xcolor}
\usepackage[T1]{fontenc}
\usepackage{pagecolor}
\usepackage{amssymb}
\usepackage{lmodern}
\usepackage{mathtools, nccmath}
\usepackage{courier}
\usepackage[overload]{empheq}
\usepackage[inline, shortlabels]{enumitem}
\usepackage{amsmath}
\usepackage{mathtools}
\definecolor{myyellow}{RGB}{225,225,100}
\definecolor{myred}{RGB}{220,100,100}
\definecolor{mygreen}{RGB}{120,225,120}
\definecolor{myblue}{RGB}{100,200,255}
\definecolor{mypurple}{RGB}{200,140,255}
\definecolor{myorange}{RGB}{255,150,50}
\color{white}
\pagecolor{black}
\title{Introduction to Linear algebra \#1}
\author{@all.about.mathematics}

\begin{document}
\maketitle
\large
\section{Introduction}
This new series of posts is designed to introduce concepts in linear algebra with some problems and solutions!
\newline In this post, we review the very basics of linear algebra, including some \textcolor{myyellow}{properties of matrices, elementary row operations and elementary matrices}

\newpage
\section{Properties of matrices}
\textcolor{myyellow}{Theorem 1: If $A$ is invertible, then so is $adj \:A$ }
\medskip

\noindent{Proof: We start with the formula }
\begin{equation}
A \:(adj\:A)= (\det A)\: I
\end{equation}
Since $\exists A^{-1}\iff \det A \neq 0$, we divide both sides of (1) by $\det A$
$$\left(\frac{1}{\det A}A\right)(adj \: A)=I$$
Therefore, $\left(\frac{1}{\det A}A\right)$ is the inverse of $adj \: A$.
\medskip

\noindent \textcolor{myyellow}{Theorem 2: If $\exists A^{-1}, ({A^T})^{-1}=({A^{-1}})^T$}
\medskip

\noindent{Proof: $$A^{-1} A=I\implies (A^{-1} A)^T=I$$
Since $(BC)^T=C^TB^T$,
$$\implies A^T ({A^{-1}})^T=I\implies ({A^{-1}})^T=({A^T})^{-1}$$}

\newpage
\section{Elementary row operations}
There are 3 types of elementary row operations (EROs):
\medskip

\noindent{\textcolor{myred}{Type I: Swapping row $i$ and row $j$}}

\noindent{This is expressed as \textcolor{myred}{$\mathbf{r}_i\longleftrightarrow\mathbf{r}_j\:,\:i\neq j$}}

\noindent{The inverse of this operation is \textcolor{myred}{itself}, and applying this operation on a matrix multiplies it's determinant by \textcolor{myred}{$-1$}}
\medskip

\noindent{\textcolor{mygreen}{Type II: Multiplying row $i$ by a nonzero constant $k$}}

\noindent{This is expressed as \textcolor{mygreen}{$k\mathbf{r}_i\longrightarrow\mathbf{r}_i\:,\:k\neq 0$}}

\noindent{The inverse of this operation is \textcolor{mygreen}{$\frac{1}{k}\mathbf{r}_i\longrightarrow\mathbf{r}_i$}, and applying this operation on a matrix multiplies it's determinant by \textcolor{mygreen}{$k$}}
\medskip

\noindent{\textcolor{myblue}{Type III: Add row $i$ multiplied by $k$ to row $j$}}

\noindent{This is expressed as \textcolor{myblue}{$k\mathbf{r}_i+r_j\longrightarrow\mathbf{r}_j\:,\:k\neq 0\:,\: i\neq j$}}

\noindent{The inverse of this operation is \textcolor{myblue}{$-k\mathbf{r}_i+r_j\longrightarrow\mathbf{r}_j$}, and applying this operation on a matrix \textcolor{myblue}{does not change its determinant.}}
\medskip

\noindent{Another way to represent EROs are by \textcolor{mypurple}{elementary matrices}. Let $B$ be a matrix obtained from a square matrix $A$ by applying an ERO. Let $E$ be obtained from $I$ by \textcolor{mypurple}{applying the same ERO}. Then \textcolor{mypurple}{$B=EA$}, and $E$ is the elementary matrix representing such an ERO. }

\medskip

\noindent{\textcolor{myorange}{Important theorem:}

\noindent{We can turn square matrix $A$ into $I$ with EROs $\implies \exists A^{-1}$} }

\medskip

\noindent{Proof: Suppose we apply the EROs $e_1, e_2, \cdots, e_n$ to turn $A$ into $I$. Let $E_1, E_2, \cdots, E_n$ be the elementary matrices that represent the EROs respectively. Then we have}
$$E_n\cdots E_2 \:E_1\: A=I$$
$$\implies A^{-1}= E_n\cdots E_2 \:E_1\: $$
\end{document}

Algebra/all.about.mathematics' questions/Linear algebra 1.tex


\paragraph{Divide \& conquer optimization}
	For recurrence
	$$f(i) = \min_{k<i}\{b(k)+c[k][i]\}$$
	$k(i) \leq k(i+1)$ holds true if $c[a][c]+c[b][d]<c[a][d]+c[b][c]$.

	\subparagraph{LBN technique}
	Implement the optimization in $O(n\log^2n)$.
	\begin{algorithmic}
		\Function{update}{$l,r,a,b$}
			\Comment{update $f[l\cdots r]$ with $f[a\cdots b]$}
			\If {$l\le r$}
				\State $m\gets (l+r)/2$
				\State $newk\gets\arg_i\min_{a\le i \le b}f[i]+cost(i,m)$
				\State $newf\gets f[u]+cost(u,m)$
				\If{$newf < f[m]$}
					\State $f[m]\gets newf$
					\State $k[m]\gets newk$
					\State \Call{update}{$l,m-1,a,newk$}
				\EndIf
				\State \Call{update}{$m+1,r,newk,b$}
			\EndIf
		\EndFunction
		\Function{solve}{$l,r$}
			\Comment{compute $f[l\cdots r]$}
			\If {$l<r$}
				\State $m\gets (l+r)/2$
				\State \Call{solve}{$l,m$}
				\State \Call{update}{$m+1,r,\max(l,k[m]),m$}
				\State \Call{solve}{$m+1,r$}
			\EndIf
		\EndFunction
	\end{algorithmic}

\paragraph{Knuth optimization}
	For recurrence
	$$f(i,j) = \min_{i<k<j}\{f(i,k)+f(k,j)\}+c[i][j]$$
	$k(i,j-1) \leq k(i,j) \leq k(i+1,j)$ holds true if $c[a][c]+c[b][d]<c[a][d]+c[b][c]$.


\subsection{Constant Elasticity of Substitution (CES) production function}

For some constant \(r\).

\(Q=A[\alpha_i X_i^r]^{\dfrac{1}{r}}\)


%!TEX root = ../Thesis.tex
\chapter{Pendulum - Hamilton's Equations and Numerical Analysis}

\section{Equations of Motion}
\begin{figure}[ht!]
\centering
\includegraphics[scale=1.2]{fig/pendulum.pdf}
\caption{The pendulum}
\label{fig:pe}
\end{figure}
\begin{description}
\item[Step 0 \quad Lagrangian $L$] \ \\[0.5cm]
Generalized coordinate: $\theta$. \\[0.2cm]
$T = \frac{1}{2} m \dot{x}^2 = \frac{1}{2} m l^2 \theta^2.$ \\[0.2cm]
$V = -m g l \cos{\theta}$ \\[0.2cm]
$L(\theta, \dot{\theta}) = T - V = \frac{1}{2} m l^2 \theta^2 + m g l \cos{\theta}$

\item[Step 1 \quad Generalized momentum $p_\theta$] \ \\[0.2cm]
$p_\theta(\theta,\dot{\theta}) = \dfrac{\partial L}{\partial \dot{\theta}} = m l^2 \dot{\theta}$

\item[Step 2 \quad Transform $\dot{\theta}$] \ \\[0.5cm]
$\dot{\theta} = \dot{\theta}(\theta, p_\theta) = \dfrac{p_\theta}{m l^2}$

\item[Step 3 \quad The Hamiltonian $H$] \ \\[0.5cm]
$H(\vec{q}, \vec{p}, t) = \sum\limits_{i=1}^n p_i \dot{q_i} - L$ \\[0.5cm]
\begin{align}
\notag H &= p_\theta \dot{\theta} - \left(\dfrac{1}{2} m l^2 \dot{\theta}^2 + m g l \cos{\theta}\right) \\
\notag &= p_\theta \left(\dfrac{p_\theta}{m l^2} \right) - \left(\dfrac{1}{2} m l^2 \left(\dfrac{p_\theta}{m l^2}\right)^2 + m g l \cos{\theta}\right) \\
&= \dfrac{p_\theta^2}{2 m l^2} - m g l \cos{\theta}
\end{align}
, which we again recognize simply as the total energy. The first term is the kinetic energy in terms of the momentum $p_\theta$ and the second term is the potential energy $V$.

\item[Step 4 \quad Hamilton's Equations of Motion]
\begin{align}
\begin{split}
\label{eq:pe-eom}
\dot{\theta} = +\dfrac{\partial H}{\partial p_\theta} &= \dfrac{p_\theta}{m l^2} ,
\\[0.2cm]
\dot{p_\theta} = -\dfrac{\partial H}{\partial \theta} &= - m g l \sin{\theta}
\end{split}
\end{align}
\end{description}

\section{Numerical Analysis}
We will now solve the pendulum's equations of motion, \eqref{eq:ho-eom}. For simplicity we'll set $k = m = 1$. Note that by choice of $m=l=1$, the momentum $p_\theta$ is actually equal to the angular velocity $\omega$. The first step is to discretize the equations
\begin{alignat}{2}
&\dod{\theta}{t} = p  & \qquad \implies \qquad &\Delta \theta = p \Delta t \\[0.5cm]
&\dod{p}{t} = -\sin{\theta} & \qquad \implies \qquad &\Delta p = - \sin{\theta} \Delta t
\end{alignat}
This is a non-linear system of PDEs with no analytical solution. The small angle approximation $\sin{\theta} = \theta$ makes this system identical to the harmonic oscillator. However we're interested in solving the exact system where large oscillations are permitted. We will not be able to compare our numerical solutions to an analytical one (since none exists), but we can gain valuable insights nonetheless.

Once again, in the explicit Euler use old time step values $i$, implicit Euler use new time step values $i+1$ and the symplectic Euler use mixed time step values. As we will see, now the implicit Euler necessitates finding roots numerically.
\subsection{Explicit Euler algorithm}
\begin{align}
\begin{split}
\label{al:pe-euler_e}
\theta_{i+1} &= p_i \Delta t + \theta_i \\
p_{i+1} &= - \sin{\theta_i} \Delta t + p_i
\end{split}
\end{align}

\subsection{Implicit Euler algorithm}
\begin{align}
\begin{split}
\label{al:pe-euler_i}
\theta_{i+1} &= p_{i+1} \Delta t + \theta_i \\
p_{i+1} &= - \sin{\theta_{i+1}} \Delta t + p_i
\end{split}
\end{align}
The new time step $i+1$ values are then found by substituting on the the equations into the other, then solving numerically with a root finder algorithm. For example we can substitute the expression for $p_{i+1}$ into $\theta_{i+1}$ and solve the resulting equation
\begin{align}
0 = \theta_i - \theta_{i+1} + (p_i - sin{\theta_{i+1}})\Delta t
\end{align}
Now run root finding for $\theta_{i+1}$, guessing $\theta_i$ as initial guess.

\subsection{Symplectic Euler algorithm}
\begin{align}
\begin{split}
\label{al:pe-euler_s1}
\theta_{i+1} &= p_{i+1} \Delta t + \theta_i \\
p_{i+1} &= - \sin{\theta_i} \Delta t + p_i
\end{split}
\end{align}
or
\begin{align}
\begin{split}
\label{al:pe-euler_s2}
\theta_{i+1} &= p_{i} \Delta t + \theta_i \\
p_{i+1} &= - \sin{\theta_{i+1}} \Delta t + p_i
\end{split}
\end{align}
Algorithms \crefrange{al:pe-euler_e}{al:pe-euler_s1} was implemented in Python (Appendix \ref{app:pe}).

\section{Plots}
\begin{figure}
    \centering
        \subbottom[Explicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_y(x)_euler_explicit.pdf}
            \label{fig:pe_y(x)_euler_explicit}
        }
        \subbottom[Implicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_y(x)_euler_implicit.pdf}
            \label{fig:pe_y(x)_euler_implicit}
        }
        \subbottom[Symplectic Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_y(x)_euler_symplectic.pdf}
            \label{fig:pe_y(x)_euler_symplectic}
        }
        \caption{Position y(x)}
    \label{fig:pe_y(x)_euler}
\end{figure}

\begin{figure}
    \centering
        \subbottom[Explicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_omega(t)_euler_explicit.pdf}
            \label{fig:pe_omega(t)_euler_explicit}
        }
        \subbottom[Implicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_omega(t)_euler_implicit.pdf}
            \label{fig:pe_omega(t)_euler_implicit}
        }
        \subbottom[Symplectic Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_omega(t)_euler_symplectic.pdf}
            \label{fig:pe_omega(t)_euler_symplectic}
        }
        \caption{Momentum p(t), equivalent to angular velocity $\omega(t)$ by choice of $m=l=1$}
    \label{fig:pe_omega(t)_euler}
\end{figure}

\begin{figure}
    \centering
        \subbottom[Explicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_E(t)_euler_explicit.pdf}
            \label{fig:pe_E(t)_euler_explicit}
        }
        \subbottom[Implicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_E(t)_euler_implicit.pdf}
            \label{fig:pe_E(t)_euler_implicit}
        }
        \subbottom[Symplectic Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_E(t)_euler_symplectic.pdf}
            \label{fig:pe_E(t)_euler_symplectic}
        }
        \caption{Energy E(t), note y-scale on \ref{fig:pe_E(t)_euler_symplectic}}
    \label{fig:pe_E(t)_euler}
\end{figure}

\begin{figure}
    \centering
        \subbottom[Explicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_phase-space_euler_explicit.pdf}
            \label{fig:pe_phase-space_euler_explicit}
        }
        \subbottom[Implicit Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_phase-space_euler_implicit.pdf}
            \label{fig:pe_phase-space_euler_implicit}
        }
        \subbottom[Symplectic Euler]{
            \includegraphics[scale=0.24]{fig/pe/pe_phase-space_euler_symplectic.pdf}
            \label{fig:pe_phase-space_euler_symplectic}
        }
        \caption{Phase-space $p(\theta) = \omega(\theta)$}
    \label{fig:pe_phase-space_euler}
\end{figure}
Things to note this time from the figures \crefrange{fig:pe_y(x)_euler}{fig:pe_phase-space_euler}:
\begin{itemize}
    \item The initial conditions are set such that the pendulum barely has enough energy to go all the way around. Hence the implicit Euler method, known for loosing energy, does not make it all the way around, see fig \ref{fig:pe_y(x)_euler_implicit} as opposed to the explicit and symplectic, fig \ref{fig:pe_y(x)_euler_explicit} and \ref{fig:pe_y(x)_euler_symplectic}.
    \item Symplectic once again keeps the energy $E(t)$ almost constant, and conserves it seemingly perfectly on average over a cycle.
    \item The implicit Euler is in a closed orbit in the phase-space diagram as expected from the position plot \ref{fig:pe_y(x)_euler_implicit}, but spirals inward as expected from the energy plot \ref{fig:pe_E(t)_euler_implicit}, whereas explicit and symplectic steadily increases their angles from the initial angle.
\end{itemize}
To summarize, we have seen that the symplectic version of the Euler method for numerical integration is superior because it conserves the quantity that we most care about the reduced 3-body system: energy. In more general terms symplectic integrators conserves the Hamiltonian $H$, which in our cases is the same is the total energy $E$ as discussed in the chapter on using Hamiltonian mechanics. Using symplectic Euler going forward, we are now ready to solve the restricted 3-body problem.


\subsection{Tensor inverses}

For second order tensors we have:

\begin{itemize}
\item \(T_j^i\)
\item \(T_{ij}\)
\item \(T^{ij}\)
\end{itemize}

For each of these we can define an inverse:

\begin{itemize}
\item \(T_i^jU_j^k=\delta_i^k\)
\item \(T_{ij}U^{jk}=\delta_i^k\)
\item \(T^{ij}U_{jk}=\delta_i^k\)
\end{itemize}

\subsubsection{Notation for inverses}

If we have \(T_{ij}U^{jk}=\delta_i^k\), we can instead write:

\(T_{ij}T^{jk}=\delta_i^k\)


\section{Kernels}
\smallskip \hrule height 2pt \smallskip
If the data is not linearly separable and/or you want a wiggly boundary, use kernels.

Can use for things you can't handle with slack variables.  % wk 8 audio

Start w/ original feature space, use polynomial of degree d, and get a higher dimensional space
    then do a linear classifier in this space.   % wk 8 audio
\hfill \\  \hfill \\

Can give you nonlinear boundaries (good) but the feature space can get really large really quickly.

Example mapping of data that is not linearly separable to a separable higher dimension space: \hfill \\
\includegraphics[width=2.5in]{figures/example_kernel_separation.pdf}  \hfill \\

The decision boundary would be a circle.  % wk 8 audio
\hfill \\

If you can't figure out what features you should use, this is a good approach.  % wk 8 audio.
You don't need the features themselves (??).   \hfill \\
?? You just need the kernel/similarity? matrix.??   \hfill \\
? gram matrix?   \hfill \\

General idea: \hfill \\
If $\bm{x}$ is in $R^n$, then $\phi(\bm{x})$ is in $R^m$ for $m>n$. \hfill \\
We can now learn feature weights $\bm{w}$ in $R^m$ and predict using $y = sign(\bm{w} \cdot \phi(\bm{x}))$. \hfill \\
\hfill \\

\textbf{A linear function in the higher dimensional space will be non-linear in the original space.}  \hfill \\
\hfill \\

% wk 8 audio.
Say you had a 100,000 points, each of which have a100-dimensional (binary) feature vector.
The chances of being able to find a hyperplane that divides random assignments of binary values is very good.
Almost all of those vertices are empty.
So your chance of finding a desirable hyperplane increases dramatically.
??? Did I get this right?  Seems like we need to be comparing a feature vector to a transformed feature vector.
???
\hfill \\  \hfill \\

\subsubsection{Danger of Mapping to a Higher Dimensional Space:}
The number of terms in a polynomial of degree $d$ for $m$ input features is
$\displaystyle {d + m - 1}\choose{d} $ $ \displaystyle = \frac{d + m - 1}{d!(m-1)!}$. \hfill \\
This grows fast!  For $d=6$, $m=100$ you get about 1.6 billion terms.  \hfill \\

We are taking a dot product of $m$ rows for the feature space times $d$ columns of polynomial.
But you also have terms for combinations of features.  E.g.
\includegraphics[width=0.8in]{figures/example_kernel.pdf}

\subsubsection{Efficient dot-product of polynomials}
For polynomials of degree exactly $d$, the dot product in higher dimensional space can be written as a dot product in lower dimensional space.  \hfill \\
For $m=2$ (2 features): \hfill \\
\includegraphics[width=2.5in]{figures/kernel_dot_polynomials.pdf}  \hfill \\
note: here the $.$ is a dot product, not horizontal space fill like a previous lecture. \hfill \\
$u_1$ is feature 1's value, and $u_2$ is feature 2's value. \hfill \\
Polynomial of exactly degree $d$ means we \underline{don't} have a constant bias term like in homework 3 with $[1, \sqrt{2}u, u^2]$  \hfill \\
This is just factoring.  \hfill \\
\hfill \\

Proof not shown, but for any $d$: \hfill \\
$K(u, v) = \bm{\phi}(u).\bm{\phi}(v) = (u.v)^d$

\subsection{The "Kernel Trick"}

We like the idea of going to higher dim, but we don't like the idea of operating in it.  Expensive. \hfill \\
\hfill \\

With this trick, we won't ever compute a dot product in the expensive space.   % wk 8 audio
Not all $\phi$ satisfy, but there are big categories that do it.  % wk 8 audio
Need to be able to represent a higher dim space as one in a lower dim space.  % wk 8 audio
Usually exponentiation is used.  %wk 8 audio.
%But it doesn't work with all projections, .  Just special ones.
\hfill \\ \hfill \\

A \textbf{kernel function} defines a dot product in some feature space:
\begin{align*}
	K(\bm{u}, \bm{v}) =\bm{\phi}(\bm{u}) \cdot \bm{\phi}(\bm{v})
\end{align*}
Where $\bm{u}$, $\bm{v}$ are points in your training or test set, each with their own features. \hfill \\
Or $\bm{u}$ could be a data point and $\bm{v}$ could be the $\bm{w}$ of the hyperplane that you need to dot against to classify new points.
 \hfill \\

 If $\bm{u}$, $\bm{v}$ each have two dimensions (2 features): \hfill \\

\includegraphics[width=3.1in]{figures/example_kernel_math.pdf} \hfill \\
Thus, a kernel function \textit{implicitly} maps data to a high-dimensional space without the need to compute each $\bm{\phi}(\bm{x})$ explicitly.   \hfill \\
If we had $d = 100$ instead, our savings would be huge.
 \hfill \\

Translation for this particular kernel: \hfill \\
We want to get the perks of the high dimensional space given by $\bm{\phi}(\bm{x}) = [1, x_1^2, \sqrt{2}x_1 x_2, x_2^2, \sqrt{2} x_1, \sqrt{2} x_2]$ where $\bm{x}$ is either $\bm{u}$ or $\bm{v}$.  \hfill \\
But this is computationally expensive because there are many terms in that thing.  \hfill \\
If we take the dot product of two vectors $\bm{\phi}(\bm{u})$, $\bm{\phi}(\bm{v})$ we get some magic: \hfill \\
\begin{align*}
	\bm{\phi}(\bm{u}) \cdot \bm{\phi}(\bm{v}) &= [1, u_1^2, \sqrt{2}u_1 u_2, u_2^2, \sqrt{2} u_1, \sqrt{2} u_2]  \cdot \quad \mbox{(line wrapped)} \\
			& \quad \quad [1, v_1^2, \sqrt{2}v_1 v_2, v_2^2, \sqrt{2} v_1, \sqrt{2} v_2]   \\
		&=  (1 + \bm{u} \cdot \bm{v})^2
\end{align*}
Can do $(1 + \bm{u} \cdot \bm{v})^2$ with a dot product that only includes multiplying $[u_1, u_2]^T[v_1, v_2]$.     \hfill \\
That's a lot less computation/memory expensive than the full dot product above.  \hfill \\
If that leads to good separation, you are a happy machine learner!   \hfill \\
\textbf{This is true for other kernels in general.}   \hfill \\
($(1 + \bm{u} \cdot \bm{v})^2$ would take other forms for other cases).   \hfill \\
\hfill \\

It is essential that everything you want to do with your transformed vectors is representable by simple dot products.
If you can't simplify it down to dot products of the original vector then there's no point.

\subsection{"Kernel Trick" for the Perceptron}

\subsubsection{Intro to $ \displaystyle w = \sum_k a^k \phi(x^k)$}
If we wanted to apply a transformation of the data $\phi(x^i)$ to each of our $x^i$ training points, we would do: \hfill \\
For $t = 1 \dots T$, $i = 1, \dots n$:
\begin{itemize}
	\item $ y = sign(w \cdot \phi(x^i))$
	\item if $y \neq y^i$:  (if class wasn't predicted correctly)
	\begin{itemize}
		\item $ \quad w = w + y^i \phi(x^i)$  % update $w$:
		%\item $a^i += y^i$
	\end{itemize}
\end{itemize}

We can re-write $w$ as $ \displaystyle \sum_j y^j x^j$ for one loop over the data, if $j$ is the index of the points that were wrongly classified and thus contributed to the weights $w$.
but we have multiple epochs: $t = 1 \dots T$.
The point might be classified wrong the first time but right the 2nd and 3rd times.  \hfill \\
So we can use different notation to keep track of how each point adds to the weights
	depending on whether it was predicted correctly and whether it was a $y=1$ or $y = -1$.  \hfill \\

This looks like $ \displaystyle w = \sum_k a^k \phi(x^k)$   \hfill \\
We get $a^k$ from the modified loop over the data points shown below. \hfill \\
Note: we switch from index $i$ over the data points to $k$ because we might see data points that are identical in the training set.
We don't need to keep track of those $\phi(x^k)$ points separately with separate $k$ indices and hence $a^k$ values if they are the same.
Just keep summing up their signs as needed to modify $w$.

New protocol: \hfill \\
For $t = 1 \dots T$, $i = 1, \dots n$:
\begin{itemize}
	\item $ y = sign(w \cdot \phi(x^i)$
	\item if $y \neq y^i$:  (if class wasn't predicted correctly)
	\begin{itemize}
		%\item $ \quad w = w + y^i \phi(x^i)$  % update $w$:
		\item $ \quad a^i += y^i$
	\end{itemize}
\end{itemize}

For example, say you loop over data point $k$ with $\phi(x^k) = [1, 2, 3]$ and label $y^k = -1$ three times.
If you got it wrong the first two times and right the third time you would have this point contributing
$w = -[1, 2, 3] -[1, 2, 3] + 0[1, 2, 3]$. \hfill \\
In the loop we would have had $a^k = -1 -1$ for the two passes through that we got it wrong.  \hfill \\

If we kept track of those $a$ values for all $i$ or $k$ data points, then we get that thing above:
$ \displaystyle w = \sum_k a^k \phi(x^k)$

\subsubsection{The Kernelized Perceptron} % $w = \sum_k a^k \phi(x^k)$ in

Protocol: \hfill \\
Set $a^i = 0$ for each example $i$ \hfill \\
For $t = 1 \dots T$, $i = 1, \dots n$:  \hfill \\
\begin{itemize}
	\item $ \displaystyle y = w \cdot \phi(x^i) $ \hfill \\
			 $ \displaystyle \quad \mbox{use } w = \sum_k a^k \phi(x^k) $  \hfill \\
			$ \displaystyle \quad = sign( ( \sum_k a^k \phi(x^k))  \cdot \phi(x^i))$  \hfill \\
			$ \displaystyle  \quad = sign( \sum_k a^k K(x^k, x^i) )$
	\item if $y \neq y^i$:  (if class wasn't predicted correctly)
	\begin{itemize}
		%\item $ \quad w = w + y^i \phi(x^i)$  % update $w$:
		\item $ \quad a^i += y^i$
	\end{itemize}
\end{itemize}

The points:
\begin{itemize}
	\item We never compute the features explicitly. \hfill \\
		If $x^i$ has 3 features, $\phi(x^i)$ might have 9 items such as polynomials.  \hfill \\
		But we don't have to calculate those 9 features if they correspond to a Kernel. \hfill \\
		We might be able to use something like the simplification to $(1 + \bm{u} \cdot \bm{v})^2$
			simplification just before this.
	\item We compute the dot products in "closed form": $K(u, v) = \phi(u) \cdot \phi(v)$ \hfill \\
		A "closed form" expression is a mathematical expression that can
			be evaluated in a finite number of operations.
		(You can calculate it efficiently; it is possible you would have an infinite number of features.)  % Erick 2/24

\hfill \\

\end{itemize}

\subsubsection{Kernelized Perceptron Example}

\includegraphics[width=3.4in]{figures/kernelized_perceptron_example.pdf}

\subsubsection{Common Kernels}

\includegraphics[width=2.0in]{figures/common_kernels.pdf} \hfill \\
How do we pick the kernel?
You will get a poor result if you pick the wrong Kernel.
There is no theory that tells you which kernel to use, so try a few and see which works bets.  \hfill \\

For our class:
\begin{itemize}
	\item try the polynomial of up to degree d.
    	\item Gaussian is probably the most common.
    	\item Sigmoid pushes one side down and the other side up.
\end{itemize}

\subsubsection{Kernels: Overfitting}

With Kernels we have a huge feature space, so over-fitting can happen. \hfill \\
It turns out, however, that it can be robust to overfitting. \hfill \\
People have spent a lot of time figuring out regularization for different Kernel methods. \hfill \\
Can do things like limiting the number of updates the Perceptron does. \hfill \\
SVMs have have a clearer story for avoiding overfitting.  \hfill \\

Do keep in mind that \textbf{everything overfits sometimes}!

You can also control by:
\begin{itemize}
	\item Choosing a better Kernel
	\item Varying parameters of the Kernel, such as the width of Gaussian.
	\item  limit dimensionality, add regularization terms (e.g. L1, L2)
\end{itemize}

\subsubsection{Kernels in Logistic Regression}
Had: \hfill \\
$P(Y=0 | \bm{X}= \bm{x}, \bm{w}, w_0 )= \frac{1}{1 + \exp(w_0 + \bm{w}\cdot \bm{x})}$ \hfill \\
 \hfill \\

We can define weights in terms of data points: \hfill \\
$\displaystyle  \bm{w} = \sum_j \alpha^j \phi(\bm{x}^j)$  \hfill \\
That leads to
\begin{align*}
	P(Y=0 | \bm{X}= \bm{x}, \bm{w}, w_0 ) &= \frac{1}{1 + \exp(w_0 +  \sum_j  \alpha^j \phi(\bm{x}^j) \cdot \phi(\bm{x}))} \\
		& = \frac{1}{1 + \exp(w_0 +  \sum_j  \alpha^j K(\bm{x}^j , \bm{x})} \\
\end{align*}

We can derive the gradient descent rule on $\alpha^j$, $w_0$.  ???  \hfill \\

Similar tricks for all linear models: SVMs, etc.

\subsubsection{Kernels: overfitting \& number of parameters}
Most of the time Kernels are better behaved for over-fitting (? than \_\_\_).  But they are more expensive.
Why?
For the perceptron, we optimize $\bm{w}$.  Each time, we update $\bm{w}$.
        The dimensionality of $\bm{w}$ is $d$ or $d+1$, depending on how you consider it.
    With kernel we are optimizing $\bm{a}$.
        Dimensionality of $\bm{a}$ is way bigger than $d$, the dimensionality of the data.
    We are optimizing for way more dimensions.  \hfill \\

We are explaining points based on other points in the data set
How similar is a new point to x1, x2, xn.
That's why we are worried about how many summations/iterations you will do.

\subsubsection{Kernels for SVM = not easy}.
For SVM using the kernel isn't as easy to apply.  In Logistic regression it is easy to optimize.
    SVM requires you to write stuff down on paper.

\subsection{Kernel Summary}
Summary:
\begin{itemize}
    \item Expand the data to a higher dimension
    \item might be expensive
    \item arrive at kernel trick: approximates dot prod on higher dim space by forming dot prod in original space
    \item we derived polynomial kernels.
\end{itemize}

Question: When I see a new instance in Kernelized perceptron, do I need to add it to my matrix?   \hfill \\
Anser: no.  If you do the matrix, you compute it all in the beginning.   \hfill \\
Note: for homework we did it on-line.  \hfill \\

Method:
\begin{itemize}
        \item Separate training and test data.
        \item During training, give n data points all beforehand.  Allows you to form matrix.
        \item Train to get a values and kernel matrix.
        \item Then done training.
        \item Go to testing phase.  Give new instance and ask to classify.
        \item Don't need to add new instance to kernel matrix.
        \item Compute similarity to all of the points you have seen.
\end{itemize}

\textbf{You do need to store the points you got right.}
Good to stick to sparse \_\_\_ for a .


\subsection{Classification of Elliptic Functions}
We already classified the meromorphic functions on $\mathbb C_\infty$.
Remarkably, we can do something similar for the torus $\mathbb C/\Lambda$.
\begin{theorem}
    Let $f$ be an elliptic functions with periods $\Lambda$.
    There exists rational functions $Q_1,Q_2$ such that
    $$f(z)=Q_1(\wp(z))+Q_2(\wp(z))\wp^\prime(z)$$
    where $\wp=\wp_\Lambda$.
    Furthermore, $f$ is even, then $Q_2=0$.
\end{theorem}
\begin{proof}
    First assume $f$ is even.
    Now $f,\wp$ both have finitely many branch points by compactness.
    So we may choose distinct $c,d\in\mathbb C$ which are not branch points of $f,\wp$.
    Now consider the function $z\mapsto (f(z)-c)/(f(z)-d)$ which is analytic (as it is the composition of $f$ with a M\"obius transformation), even, and has simple zeros and poles not at ramification points of $\wp$.
    Note that we can invert M\"obius transformations nicely, therefore we can WLOG assume that $f$ has these properties.
    Since $f$ is even, the zeros of $f$ can be written as $\{\pm a_1,\ldots,\pm a_m\}$ where $a_i\neq\pm a_j$ for $i\neq j$.
    Likewise we can write the poles of $f$ as $\{\pm b_1,\ldots,\pm b_n\}$.
    We can write down an elliptic function with the same zeros and poles, namely
    $$g(z)=\frac{(\wp(z)-\wp(a_1))\cdots (\wp(z)-\wp(a_m))}{(\wp(z)-\wp(b_1))\cdots (\wp(z)-\wp(b_n))}$$
    So $f/g$ is elliptic but does not have any zeros or poles, therefore is constant.
    The claim follows.\\
    If $f$ is odd, then $f(z)/\wp^\prime(z)$ is even hence $f$ can be written as the desired form as well by the previous part.
    For general $f$, simply write it as the sum of odd and even parts
    $$f(z)=\frac{f(z)+f(-z)}{2}+\frac{f(z)-f(-z)}{2}$$
    which completes the proof.
\end{proof}


\section{Overview of Supervised Learning}

\subsection{Exercise 2.1}
This question is poorly worded. We assume that the question means there exists some model that, given $x$,
predicts a vector $\hat{y}$ such that $\hat{y}_k$ is the probability that $x$ belongs to class $k$. We then
have that
\begin{align*}
        \text{arg}\min_k \norm{t_k - \hat{y}} &= \text{arg}\min_k \sum_{i = 1}^K \big((t_k)_i - \hat{y}_i\big)^2 \\
                                    &= \text{arg}\min_k (1 - \hat{y}_k)^2 + \sum_{i = 1, i \neq k}^K \hat{y}_i^2 \\
                                    &= \text{arg}\min_k (1 - \hat{y}_k)^2 + \sum_{i = 1, i \neq k}^K \hat{y}_i^2 - \sum_{i = 1}^K \hat{y}_i^2\\
                                    &= \text{arg}\min_k (1 - \hat{y}_k)^2 - \hat{y}_k^2 \\
                                    &= \text{arg}\min_k 1 - 2\hat{y}_k \\
                                    &= \text{arg}\max_k \hat{y}_k
\end{align*}


\section{Method}

In this section we cover the precise details of how to construct the objective and constraint equations,
given that we represent $\bm{x}(t)$ as a cubic spline.

\subsection{Assumptions}

To make this problem tractable, we will make two simplifying assumptions:
\begin{itemize}
  \setlength\itemsep{0em}
  \item the trajectory will be a cubic spline
  \item the knot points of the spline will be given
\end{itemize}

\subsection{Outline}

The problem outlined in Section \ref{sec:ProblemStatement} can be posed as a quadratic program,
where $\mathcal{H}$ and $\bf{f}$ are the quadratic and linear cost matrices,
$\mathcal{A}$ and $\bf{b}$ are the equality constraints, and
$\bm{z}$ is the vector of decision variables: the coefficients of the cubic spline.

\begin{align}
  & \quad  & \quad & \quad &
    \text{minimize:}  & \quad  & \tfrac{1}{2} \bm{z}^T \mathcal{H} \bm{z} + \bm{f}^T \bm{z}
  & \quad  & \quad &  \quad &\\
  & \quad  & \quad & \quad &
    \text{subject to:}  & \quad & \mathcal{A} \bm{z} = \bm{b}
  & \quad  & \quad & \quad &
\end{align}

This is a special quadratic program: it does not have inequality constraints.
As a result, this quadratic program can be solved as the linear system shown below,
where $\bm{w}$ is a vector of Lagrange multipliers for the constraint equations.

\begin{equation}

\begin{bmatrix}
\mathcal{H} & \mathcal{A}^T \\
\mathcal{A} & \bm{0}
\end{bmatrix}
\cdot
\begin{bmatrix}
\bm{z} \\
\bm{w}
\end{bmatrix}
=
\begin{bmatrix}
-\bm{f}  \\
\bm{b}
\end{bmatrix}

\label{eqn:linearSplineSystem}
\end{equation}

Once we have constructed the matrices
$\mathcal{H}$,  $\bf{f}$,  $\mathcal{A}$, and $\bf{b}$,
we can then solve the linear system (\ref{eqn:linearSplineSystem})
for the decision variables to $\bm{z}$: the spline coefficients.

\subsection{Block Matrices}

Note that the matrices $\mathcal{H}$ and $\mathcal{A}$ are large and sparse,
as shown in Figure \ref{fig:SparsityPattern}.
These matrices should be implemented as sparse matrices, and the linear system solved using a sparse solver.

\begin{figure}[ht]
	\centering
  \includegraphics[width=\textwidth]{fig/SparsityPattern.pdf}
  \caption{The sparsity pattern for the spline shown in Figure \ref{fig:DataFittingExampleFigure}.
           It has 20 cubic segments and thus 4*20 = 80 decision variables.
           Notice that the quadratic cost matrix is perfectly block diagonal.
           The first 3*(20-1) rows of the constraint matrix are perfectly block diagonal,
             representing the continuity constraints.
           The final six rows are different, representing the boundary constraints.}
  \label{fig:SparsityPattern}
\end{figure}

The matrices $\mathcal{H}$ and $\mathcal{A}$ are constructed from smaller blocks,
where each block corresponds to the coefficients of a single segment. For example,
$\mathcal{H}_j$ is the $4 \times 4$ block in the $\mathcal{H}$ matrix that corresponds to the
spline segment $j$.
Similarly, $\mathcal{A}_{k,j}$ is the $3 \times 4$ block of coefficients
for the $k^\text{th}$ set of constraints on the coefficients of spline segment $j$.

\subsection{Spline Definition}

We will represent the trajectory as a cubic spline, with knot points $T_j$ where $j \in \{ 0 \dots M\}$.
A single segment of the spline is given below,
where $j$ is the segment index, and
$\tau = t - T_j$ is the time since the start of the segment.
The function $\bm{x}(t)$ is constructed by chaining together each of the segments $\bm{x}_j(t)$ in order.

\begin{equation}
  \bm{x}_j(\tau) = \bm{A}_j \tau^3 + \bm{B}_j \tau^2 + \bm{C}_j \tau + \bm{D}_j
\end{equation}

The derivatives of this spline are easily computed, as shown below:
\begin{equation}
  \dot{\bm{x}}_j(\tau)
    = \frac{d}{dt} \bm{x}_j(\tau)
    = 3 \bm{A}_j \tau^2 + 2 \bm{B}_j \tau + \bm{C}_j
\end{equation}
\begin{equation}
  \ddot{\bm{x}}_j(\tau)
    = \frac{d}{dt} \dot{\bm{x}}_j(\tau)
    = 6 \bm{A}_j \tau + 2 \bm{B}_j
\end{equation}
\begin{equation}
  \dddot{\bm{x}}_j(\tau)
    = \frac{d}{dt} \ddot{\bm{x}}_j(\tau)
    = 6 \bm{A}_j
\end{equation}

We will represent the coefficients of a single segment of the spline as
$\bm{z}_j = [\bm{D}_j, \bm{C}_j, \bm{B}_j, \bm{A}_j]^T$,
and the coefficients of the entire spline as
$\bm{z} = [\bm{z}_0, \bm{z}_1, \dots, \bm{z}_{M-1}]^T$.

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\subsection{Spline Continuity}

The spline coefficients are constructed such that value, slope, and curvature are continuous.
This can be written as a set of three equations at each knot point, where
$h_j = T_{j+1} - T_j$ is the duration of segment $j$.
\begin{equation}
  \bm{x}_j(h_j) - \bm{x}_{j+1}(0) = \bm{0}
\end{equation}
\begin{equation}
  \dot{\bm{x}}_j(h_j) - \dot{\bm{x}}_{j+1}(0) = \bm{0}
\end{equation}
\begin{equation}
  \ddot{\bm{x}}_j(h_j) - \ddot{\bm{x}}_{j+1}(0) = \bm{0}
\end{equation}

These equations form the blocks on the diagonals of the $\mathcal{A}$ matrix.
Each set of three constraints at a knot point populates two blocks of the constraint matrix,
one for the lower segment and one for the upper segment.

\begin{equation}
\begin{bmatrix}
\mathcal{A}_{k, j} & \mathcal{A}_{k, j+1}
\end{bmatrix}
\cdot
\begin{bmatrix}
\bm{z_j} \\
\bm{z_{j+1}}
\end{bmatrix}
=
\bm{b}_k
\label{eqn:continuityEquationsSymbols}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\begin{equation}

\mathcal{A}_{k, j}
=
\begin{bmatrix}
1 & h_j & h_j^2  & h_j^3 \\
0 & 1   & 2 h_j  & 3 h_j^2 \\
0 & 0   & 2      & 6 h_j
\end{bmatrix}

\quad \quad \quad

\mathcal{A}_{k, j+1}
=
\begin{bmatrix}
-1 & 0 & 0  & 0 \\
0 & -1   & 0  & 0 \\
0 & 0   & -2      & 0
\end{bmatrix}

\label{eqn:continuityEquationExplicitPartOne}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\begin{equation}

\bm{z}_j
=
\begin{bmatrix}
\bm{D}_j \\
\bm{C}_j \\
\bm{B}_j \\
\bm{A}_j
\end{bmatrix}

\quad \quad \quad

\bm{z}_{j+1}
=
\begin{bmatrix}
\bm{D}_{j+1} \\
\bm{C}_{j+1} \\
\bm{B}_{j+1} \\
\bm{A}_{j+1}
\end{bmatrix}

\quad \quad \quad

\bm{b}_k
=
\begin{bmatrix}
\bm{0} \\
\bm{0} \\
\bm{0}
\end{bmatrix}

\label{eqn:continuityEquationExplicitPartTwo}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\subsection{Boundary Constraints}

The boundary constraints are similar to the continuity equations,
but they use only a single block in the $\mathcal{A}$ matrix and a non-zero block in the $\bm{b}$ matrix.
To keep notation simple through this section,
we will define $L = M-1$ to be the index of the final segment of the spline.

\begin{align}
  \bm{x}_0(0) = \bm{x}_0  & \quad &  \bm{x}_L(h_L) = \bm{x}_T \\
  \dot{\bm{x}}_0(0) = \dot{\bm{x}}_0  & \quad & \dot{\bm{x}}_L(h_L) = \dot{\bm{x}}_T \\
  \ddot{\bm{x}}_0(0) = \ddot{\bm{x}}_0  & \quad & \ddot{\bm{x}}_L(h_L) = \ddot{\bm{x}}_T
\end{align}

The lower boundary equation can be written:

\begin{equation}
\mathcal{A}_{\ell, 0}
\cdot
\bm{z_0}
=
\bm{b}_\ell
\label{eqn:lowerBoundaryConstraintSymbols}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\begin{equation}

\mathcal{A}_{\ell, 0}
=
\begin{bmatrix}
  1 & 0 & 0  & 0 \\
  0 & 1   & 0  & 0 \\
  0 & 0   & 2      & 0
\end{bmatrix}

\quad \quad \quad

\bm{z}_0
=
\begin{bmatrix}
  \bm{D}_0 \\
  \bm{C}_0 \\
  \bm{B}_0 \\
  \bm{A}_0
\end{bmatrix}

\quad \quad \quad

\bm{b}_\ell
=
\begin{bmatrix}
  \bm{x}_0 \\
  \dot{\bm{x}}_0 \\
  \ddot{\bm{x}}_0
\end{bmatrix}

\label{eqn:lowerBoundaryConstraint}
\end{equation}

The upper boundary equation can be written:

\begin{equation}
\mathcal{A}_{\ell+1, L}
\cdot
\bm{z_L}
=
\bm{b}_{\ell+1}
\label{eqn:upperBoundaryConstraintSymbols}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\begin{equation}

\mathcal{A}_{\ell+1, L}
=
\begin{bmatrix}
1 & h_L & h_L^2  & h_L^3 \\
0 & 1   & 2 h_L  & 3 h_L^2 \\
0 & 0   & 2      & 6 h_L
\end{bmatrix}

\quad \quad \quad

\bm{z}_L
=
\begin{bmatrix}
  \bm{D}_L \\
  \bm{C}_L \\
  \bm{B}_L \\
  \bm{A}_L
\end{bmatrix}

\quad \quad \quad

\bm{b}_{\ell+1}
=
\begin{bmatrix}
  \bm{x}_T \\
  \dot{\bm{x}}_T \\
  \ddot{\bm{x}}_T
\end{bmatrix}

\label{eqn:upperBoundaryConstraint}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\subsection{Objective Function}

Unlike the constraint matrices, the objective function matrices are made of up a sum
of many components. A single segment will have a term from the minimum-jerk component,
as well as a term for each point in the data set that is on the time-domain of that segment.
Thus, we can rewrite the objective function (\ref{eqn:continuousObjectiveFunction}) as
sum over segments:

\begin{equation}
  \bm{J} =  \sum_{j=0}^{M-1} \bm{J}_j
\end{equation}

Similarly, the cost of a single segment can be constructed as follows,
where $\mathcal{I}$ is the set of points such that $t_i \in [T_j, T_{j+1})$.

\begin{equation}
  \bm{J}_j =  \alpha J_j^\text{ smooth}
  + \frac{T}{N} \sum_{i \in I_j}  J_{j}^\text{i}
\end{equation}

In practice, the segment cost is implemented by
summing blocks of the $\mathcal{H}$ and $\bm{f}$ matrices.
These blocks are thus computed:

\begin{equation}
  \bm{H}_j =  \alpha \bm{H}_j^\text{ smooth}
  + \frac{T}{N} \sum_{i \in I_j}  \bm{H}_{j}^\text{i}
\end{equation}

\begin{equation}
  \bm{f}_j =  \alpha \bm{f}_j^\text{ smooth}
  + \frac{T}{N} \sum_{i \in I_j}  \bm{f}_{j}^\text{i}
\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\subsection{Data-Fitting}
The data-fitting term in the objective function is given by:
\begin{equation}
  \frac{T}{N}  \sum_{i=0}^N \big( \bm{x}(t_i) - \bar{\bm{x}}_i \big)^2
\end{equation}

We will construct the equations for a single  point $\{t_i, \bar{\bm{x}}_i\}$ from the data set,
where $t_i \in [T_j, T_{j+1})$.
In other words, the point is on segment $j$.
The squared-error between the point and the segment is thus given by:

\begin{equation}
  J_j^i = \bigg( \bm{x}_j(t_i-T_j) - \bar{\bm{x}}_i\bigg) ^ 2
\end{equation}

After doing a bit of algebra, the block matrices for this equation are given below,
where $\tau_{ij} = t_i-T_j$ is the time between the data point $i$ and segment $j$.
Note that $\mathcal{H}_j^i$ is symmetric.
These matrices can them be computed for each point in the data set.

\begin{equation}
  \mathcal{H}_j^i
  =
  \begin{bmatrix}
    1 & \tau_{ij} & \tau_{ij}^2 & \tau_{ij}^3 \\
    \tau_{ij} & \tau_{ij}^2 & \tau_{ij}^3  & \tau_{ij}^4\\
    \tau_{ij}^2 & \tau_{ij}^3  & \tau_{ij}^4 & \tau_{ij}^5\\
    \tau_{ij}^3  & \tau_{ij}^4 & \tau_{ij}^5& \tau_{ij}^6\\
  \end{bmatrix}

  \quad \quad \quad

  \bm{f}_j^i
  \begin{bmatrix}
    -\bar{\bm{x}}_i \\
    -\bar{\bm{x}}_i \, \tau_{ij}\\
    -\bar{\bm{x}}_i \, \tau_{ij}^2\\
    -\bar{\bm{x}}_i \, \tau_{ij}^3\\
  \end{bmatrix}^T

\end{equation}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%

\subsection{Smoothing Objective}

We enforce a smooth spline by adding a term to minimize the integral of the curvature-rate-squared.
This is simple to compute for a cubic spline:

\begin{equation}
  \dddot{\bm{x}}_j^2(t) = 36\bm{A}_j^2
\end{equation}

The corresponding blocks in the $\mathcal{H}$ and $\bf{f}$ matrices are:

\begin{equation}
  \mathcal{H}_j^\text{ smooth}
  =
  \begin{bmatrix}
    0 & 0 & 0 & 0 \\
    0 & 0 & 0 & 0 \\
    0 & 0 & 0 & 0 \\
    0 & 0 & 0 & 36
  \end{bmatrix}

  \quad \quad \quad

  \bm{f}_j^\text{ smooth}
  \begin{bmatrix}
    \bm{0} \\
    \bm{0} \\
    \bm{0} \\
    \bm{0}
  \end{bmatrix}^T

\end{equation}


\documentclass[12pt,titlepage]{article}

\usepackage{amsmath,amssymb}
\usepackage{hyperref}
\usepackage{pgfplots}
\usepackage{siunitx}
\usepackage{placeins}
\usepackage{url}

\title{The Stroop Effect \\ \vspace{10pt} \large{\it Udacity Deep Learning Nanodegree Statistics Project}}
\author{George Yu}
\date{\today}

\begin{document}

\maketitle

\section{Basic Information}
  \begin{itemize}
    \item {\bf Independent Variable}: the congruency of the words and the ink colors
    \item {\bf Dependent Variable}: the time for the participant to name the ink colors in a list of words
  \end{itemize}

  According to \cite{wiki:stroop-effect}, the participants will take longer to name the ink colors for the incongruent words, so the we should perform a one-tailed test on a sample. The null hypothesis is that for the population, there is no difference in both groups of words, and the alternative hypothesis is that the time taken for the incongruent group will be longer than the congruent group in the population, or,
  \begin{align*}
    &\mathbf{H_0}\hspace{-2pt}:\ \mu_\mathrm{i} - \mu_\mathrm{c} \leqslant 0 \\
    &\mathbf{H_a}\hspace{-2pt}:\ \mu_\mathrm{i} - \mu_\mathrm{c} > 0,
  \end{align*}
  where $\mu_\mathrm{i}$ is the population mean of the time for the incongruent words, and $\mu_\mathrm{c}$ is the population mean of the time for the congruent words.

  Since each participant will go through and record from each of the two conditions, so this experiment is a within-subject design, namely, it tests a sample of $n=24$ with two conditions. Therefore, a dependent samples $t$-test should be used. The reason for using a $t$-test rather than a $z$-test is that the we do not know the population parameters $\mu$ and $\sigma$.

  To conduct the $t$-test, we make the following assumptions:
  \begin{itemize}
    \item The population is approximately normally distributed
    \item The sample drawn from the population should be random
    \item The sample data can estimate population parameters
    \item The dependent variable (time to read the words) is continuous
    \item The pair-differences $\overline{x}_\mathrm{d}$ are independent of each other
  \end{itemize}

\section{Dataset}
  \autoref{fig:histogram} shows a visualization of the dataset. From this graph, we can see that the incongruent group are mainly distributed to the right of the congruent group. The congruent group is mainly distributed between 8.63 and 19.28, while the incongruent group is mainly distributed between 13.96 and 24.61.

  For the congruent group, the mean is $\overline{x}_\mathrm{c} \approx 14.05$ with a sample standard deviation of $s_\mathrm{c} \approx 3.56$, and for the incongruent group, we have $\overline{x}_\mathrm{i} \approx 22.02$ and $s_\mathrm{i} \approx 4.80$.

  \begin{figure}[h]
  \centering
    \begin{tikzpicture}
    \begin{axis}[
      width=0.8\textwidth,
      height=0.6\textwidth,
      x tick label style={xshift=-0.65cm},
      xlabel=Time (\si{s}),
      x label style={at={(axis description cs:0.5,-0.03)},anchor=north},
      enlarge y limits=upper,
      minor y tick num = 1,
      ybar interval=0.8,
      area style,
    ]
    \addplot+[ybar interval,mark=no] plot coordinates { (8.63, 11) (13.96, 11) (19.28, 2) (24.61, 0) (29.93, 0) (35.26, 0) (40.59, 0) };
    \addplot+[ybar interval,mark=no] plot coordinates { (8.63, 0) (13.96, 8) (19.28, 12) (24.61, 2) (29.93, 1) (35.26, 1) (40.59, 0) };
    \legend{Congruent,Incongruent}
    \end{axis}
    \end{tikzpicture}
    \caption{Data of }
    \label{fig:histogram}
  \end{figure}

\FloatBarrier
\section{Statistical Test}
  First subtract the incongruent group with the congruent group, we get a new distribution of differences with $\overline{x}_\mathrm{d} \approx 7.96$ and $s_\mathrm{d} \approx 4.86$.

  Using $\alpha = 0.05$, we obtain that our one-tailed $t$-critical value with $n=24$ (degrees of freedom $\nu = 23$), is $t^*=1.711$. Now, we can obtain the $t$-statistic,
  $$ t = \frac{\overline{x}_\mathrm{d} - \mu_0}{s_\mathrm{d}/\sqrt{n}}, $$
  where $\mu_0$ is the expected mean difference, which is $\mu_\mathrm{i} - \mu_\mathrm{c} = 0$, thus,
  $$ t = \frac{7.96}{4.86 / \sqrt{24}} \approx 8.02. $$

  We can see that $t$ is statistically significant, since $p < .0001$, in conclusion,
  $$ t(23) = 8.02,\ p < .0001,\ \textrm{one-tailed}. $$
  So, $\overline{x}_\mathrm{d}$ is significant at $p<0.05$, in other words, the difference in the two test groups are very unlikely due to chance, and since we are estimating the population parameter, we have $\mu_\mathrm{i} - \mu_\mathrm{c} > 0$, therefore we reject the null hypothesis. The result is what I expected.
  The $95\%$ confidence interval $CI$ is,
  $$ 95\%\ CI = (5.91, 10.01). $$

\section{Evaluation}
  Cohen's d is about 1.64, and the $r^2$ value is approximately $0.74$, which means that about 74\% of the variation in the time taken to name the ink colors can be explained by the congruency of the words and the ink colors. Other factors that might have influenced the time could be the carry-over effect, the order of the tests that the participants take will perhaps influence their results.

  We can conduct a similar experiment but using a between-subject design, where two independent samples are selected randomly, and one group will be reading the congruent words where the other group will be reading the incongruent words. This experiment can then eliminate the possible inaccuracies caused by the carry-over effect.

\medskip

\bibliographystyle{unsrt}
\bibliography{stroop}

\end{document}