\chapter{Sets}
\label{chapter:sets}
\marginurl{%
  Sets:\\\noindent
  Introduction to Mathematical Reasoning \#6
}{youtu.be/bshBV2H4Sqo}
\section{The Intuitive Definition of a Set}
A set is one of the two most important concepts in mathematics. Many mathematical
statements involve ``an integer $n$'' or ``a real number $a$''. Set theory
notation provides a simple way to express that $a$ is a real number or that $a$
is a natural number. However, this language is much more expressible and it is
impossible to imagine modern mathematics without this notation.

As in the previous chapter it is difficult to define a set formally so we give
a less formal definition which should be enough to use the notation.
A \emph{set} is a well-defined collection of objects. Important examples of
sets are:
\begin{enumerate}
  \item $\R$ a set of reals,
    \nomenclature[S]{$\R$}{denotes the set of all real numbers}
  \item $\Z$ the set of integers\footnote{``Z'' stands for the German word
    Zahlen (``numbers'').},
    \nomenclature[S]{$\Z$}{denotes the set of all integers}
  \item $\N$ the set of natural numbers\footnote{%
      Note that in the literature
      there are two different traditions: in one $0$ is a natural number, in
      another it is not; in this book we are going to assume that $0$ is not a
      natural number.
    },
    \nomenclature[S]{$\N$}{denotes the set of all integers greater than $0$}
  \item $\Q$ a set of rational numbers,
    \nomenclature[S]{$\Q$}{denotes the set of all rational numbers}
  \item $\mathbb{C}$ a set of complex numbers.
    \nomenclature[S]{$\C$}{denotes the set of all complex numbers}
\end{enumerate}
Usually, sets are denoted by a single letter.

Objects in a set are called \emph{elements} of the set and we denote the
statement ``x is in the set $E$''\footnote{%
  The symbol $\in$ was first used by Giuseppe Peano 1889 in his work
  ``Arithmetices principia, nova methodo exposita''. Here he wrote on page X:
  ``The symbol $\in$ means is. So $a \in b$ is read as a \emph{is} a b;
  \dots''
  The symbol itself is a stylized lowercase Greek letter epsilon
  (``$\epsilon$''), the first letter of the word  \textgreek{esti}, which means
  ``is''.
}
by the formula $x \in E$ and the negation of this statement by $x \not\in E$.
For example, we proved that $\sqrt{2} \not\in \Q$.
\nomenclature[S]{$x \in X$}{says that $x$ is an element of $X$}
\nomenclature[S]{$x \not\in Y$}{says that $x$ is not an element of $Y$}

\section{Basic Relations Between Sets}
Many problems in mathematics are problems of determining whether two descriptions
of sets are describing the same set or not. For example, when we learn how to
solve quadratic equations of the form $ax^2 + bx + c = 0$ ($a, b, c \in \R$) we
learn how to list the elements of the set $\set[ax^2 + bx + c = 0]{x \in \R}$.

We say that two sets $A$ and $B$ are equal if they contain the
same elements (we denote it by $A = B$). If all the elements of $A$ belong to
$B$ we say that $A$ is a subset of $B$ and denote it by
$A \subseteq B$\footnote{%
  In the literature there are three symbols for ``subset'': $\subseteq$,
  $\subsetneq$, and $\subset$. $A \subseteq B$ means that $A$ is a subset of
  $B$ and we allow $A = B$ and $A \subsetneq B$ means that $A$ is a subset of
  $B$ and we forbid $A = B$. However, there is a problem with the third symbol,
  some people use it as a synonym of $\subseteq$ and some use it as a synonym of
  $\subsetneq$. Due to this ambiguity we are going to avoid using it in this
  book.
}.

For example, $\Q \subseteq \R$ since any rational number is
also a real number. A special set is an empty set i.e. the set that does not
have elements, we denote it $\emptyset$.

\nomenclature[S]{$A \subseteq B$}{says that $A$ is a subset of $B$}

\nomenclature[S]{$\emptyset$}{denotes the set that does not have elements}

\begin{exercise}
  \label{exercise:inclusion}

  Which of the following sets are included in which? Recall that a number is
  prime iff it is an integer greater than $1$ and divisible only by $1$ and
  itself.
  \begin{enumerate}
    \item The set of all positive integers less than $10$.
    \item The set of all prime numbers less than $11$.
    \item The set of all odd numbers greater than $1$ and less than $6$.
    \item The set of all positive integers less than $10$.
    \item The set whose only elements are $1$ and $2$.
    \item The set whose only element is $1$.
    \item The set of all prime numbers less than $11$.
  \end{enumerate}
\end{exercise}

\subsection{Diagrams}
If we think of a set $A$ as represented by all the points within a circle or
any other closed figure, then it is easy to represent the notion of $A$ being a
subset of another set $B$ also represented by all the points within a circle.
We just put a circle labeled by $A$ inside of the circle labeled by $B$. We can
also diagram an equality by drawing a circle labeled by both $A$ and $B$. (see
fig.~\ref{figure:euler-diagram}). Such diagrams are called Euler diagrams and
it is clear that one may draw Euler diagrams for more than two sets.

\begin{figure}
    \centering
    \subfloat[$B \subseteq A$]{
      \begin{tikzpicture}[scale=0.6]
        \draw (20:2cm) circle (3cm) node [yshift=8ex] {$A$};
        \draw (0:2cm) circle (1.5cm) node {$B$};
      \end{tikzpicture}
    }
    \qquad\qquad
    \subfloat[$A = B$] {
      \begin{tikzpicture}[scale=0.6]
        \draw (20:2cm) circle (3cm) node [yshift=8ex] {$A, B$};
      \end{tikzpicture}
    }
    \caption{Euler diagrams for subset and equality relations}
    \label{figure:euler-diagram}
\end{figure}

\subsection{Descriptions of Sets}
In this section we describe how to define new sets, this notation is also
known as \emph{set-builder notation}.

\paragraph{Listing elements.} The simplest way to define a set is just to list
the elements. For example
\begin{enumerate}
  \item $\set{1, 2, \pi}$ is the set consisting of three elements 1, 2, and
    $\pi$, and
  \item $\set{1, 2, 3, \dots}$ is the set of all positive integers i.e. it is
    the set $\N$.
\end{enumerate}

\paragraph{Conditional definitions.}
We may also describe a set using some constraint e.g we may list all the even
numbers using the following formula $\set[n \text{ is even}]{n \in \Z}$
(we read it as ``the set of all integers $n$ such that $n$ is even'').

Using this we may also define the set of all integers from $1$ to $m$, we
denote it $[m]$; i.e. $[m] = \set[0 < n \le m]{n \in \N}$.
\nomenclature[S]{$[n]$}{denotes the set of all the integers from $1$ to n}

\paragraph{Constructive definitions.} Another way to construct a set of all
even numbers is to use the constructive definition of a set:
$\set[k \in \Z]{2k}$.

We may also describe a set of rational numbers using this description:
$\Q = \set[a \in \Z, b \in \N]{a / b}$ (note that we may also use a mix of
a conditional and constructive definitions,
$\Q = \set[a, b \in \Z, b \neq 0]{a / b}$).

\begin{exercise}
  Describe a set of perfect squares using constructive type of definition.
\end{exercise}

\subsection{Disjoint Sets}
Two sets are \emph{disjoint} iff they do not have common elements. We also
say that two sets are \emph{overlapping} iff they are not disjoint i.e. they
share at least one element.

More generally, $A_1$, \dots, $A_\ell$ are pairwise disjoint iff $A_i$ is
disjoint with $A_j$ for all $i \neq j \in [\ell]$

\begin{exercise}
  Of the sets in Exercise~\ref{exercise:inclusion}, which are disjoint from
  which?
\end{exercise}

\section{Operations over Sets.}
Another way to describe a set is to apply operation to other sets. Let $A$ and
$B$ be sets.
\begin{figure}
  \centering
  \subfloat[$A \cup B$]{
    \begin{tikzpicture}[scale=0.8]
        \fill[yellow] (45:2cm) circle (1.5cm);
        \fill[yellow] (0:2cm) circle (1.5cm);

        \draw (45:2cm) circle (1.5cm) node [above] {$A$};
        \draw (0:2cm) circle (1.5cm) node [below] {$B$};
    \end{tikzpicture}
  }
  \qquad\qquad
  \subfloat[$A \cap B$]{
    \begin{tikzpicture}[scale=0.8]
      \begin{scope}
          \clip (45:2cm) circle (1.5cm);
          \fill[yellow] (0:2cm) circle (1.5cm);
      \end{scope}
      \draw (45:2cm) circle (1.5cm) node [above] {$A$};
      \draw (0:2cm) circle (1.5cm) node [below] {$B$};
    \end{tikzpicture}
  }
  \vskip 0.25cm
  \subfloat[$A \setminus B$]{
    \begin{tikzpicture}[scale=0.8]
      \begin{scope}[even odd rule]
        \clip (0:2cm) circle (1.5cm) (-0.25,-0.25) rectangle (3,3);
        \fill[yellow] (45:2cm) circle (1.5cm);
      \end{scope}
      \draw (45:2cm) circle (1.5cm) node [above] {$A$};
      \draw (0:2cm) circle (1.5cm) node [below] {$B$};
    \end{tikzpicture}
  }
  \qquad\qquad
  \subfloat[$A \Delta B$]{
    \begin{tikzpicture}[scale=0.8]
      \begin{scope}[even odd rule]
        \clip (0:2cm) circle (1.5cm) (45:2cm) circle (1.5cm);
        \fill[yellow] (45:2cm) circle (1.5cm);
        \fill[yellow] (0:2cm) circle (1.5cm);
      \end{scope}

      \draw (45:2cm) circle (1.5cm) node [above] {$A$};
      \draw (0:2cm) circle (1.5cm) node [below] {$B$};
    \end{tikzpicture}
  }
  \caption{Euler diagrams for set operations}
\end{figure}

The first example of the operations on sets is the \emph{union} operation.
The union of $A$ and $B$ is the set containing all the elements of $A$ and all
the elements of $B$ i.e.
$A \cup B = \set[x \in A \text{ or } x \in B]{x}$\footnote{%
  Note that this definition is not correct since in the conditional definitions
  we have to specify the set $x$ belongs to and we cannot do this here.
}.
\nomenclature[S]{$A \cup B$}{denotes the union of two sets $A$ and $B$}

Another example of such an operation is \emph{intersection}. The
intersection of $A$ and $B$ is the set of all the elements belonging to both
$A$ and $B$ i.e $A \cap B = \set[x \in A \text{ and } x \in B]{x}$\footnote{%
  You may notice that in the definition of the union we use disjunction and
  in the definition of intersection we use conjunction. Actually this is the
  reason the symbol of the conjunction is similar to the symbol of intersection
  and the symbol of the disjunction is similar to the symbol of union.
}.
\nomenclature[S]{$A \cap B$}{denotes the intersection of two sets $A$ and $B$}

The third operation we are going to discuss this lecture is
\emph{set difference}. If $A$ and $B$ are some sets, then
$A \setminus B = \set[x \in A \text{ and } x \not\in B]{x}$.
\nomenclature[S]{$A \setminus B$}{denotes the difference of two sets $A$ and $B$}

The last operation is \emph{symmetric difference}. If $A$ and $B$ are some
sets, then $A \Delta B = (A \setminus B) \cup (B \setminus A)$. Note that
alternatively $A \Delta B = (A \cup B) \setminus (A \cap B)$

\begin{exercise}
  Describe the set
  $\set[n \text{ is even}]{n \in \N} \cap \set[n \in \N]{3n}$.
\end{exercise}

\begin{theorem}
\label{theorem:set-equalities}
  Let $A$, $B$, and $C$ be some sets. Then we have the following identities.
  \begin{description}
    \item[(associativity)] $A \cup (B \cup C) = (A \cup B) \cup C$ and
      $A \cap (B \cap C) = (A \cap B) \cap C$.
    \item[(commutativity)] $A \cup B = B \cup A$ and $A \cap B = B \cap A$.
    \item[(distributivity)] $A \cup (B \cap C) = (A \cup B) \cap (A \cup C)$
      and $A \cap (B \cup C) = (A \cap B) \cup (A \cap C)$.
  \end{description}
\end{theorem}
\begin{proof}
  One may prove these properties using the Euler diagrams. Alternatively they
  can be proven by definitions. Let us prove only the first part of the
  distributivity, the rest is Exercise~\ref{exercise:set-equalities}.

  Our proof consists of two parts in the first part we prove that
  $A \cup (B \cap C) \subseteq (A \cup B) \cap (A \cup C)$.
  Suppose that $x \in A \cup (B \cap C)$. Then $x \in A$ or $x \in (B \cap C)$.
  \begin{itemize}
    \item If $x \in A$, then $x \in (A \cup B)$ and $x \in (A \cup C)$ i.e.
      $x \in ((A \cup B) \cap (A \cup C))$.
    \item If $x \in (B \cap C)$, then $x \in B$ and $x \in C$. Which implies
      that $x \in (A \cup B)$ and $x \in (A \cup C)$. As a result,
      $x \in ((A \cup B) \cap (A \cup C))$.
  \end{itemize}
\end{proof}

\begin{exercise}
\label{exercise:set-equalities}
  Prove the rest of the equalities in Theorem~\ref{theorem:set-equalities}.
\end{exercise}

Probably the most difficult concept connected to sets is
the concept of a power set. Let $A$ be some set, then the set of
all possible subsets of $A$ is denoted by $2^A$ (sometimes this set is denoted
by $\mathcal{P}(A)$) and called the power set of $A$. In other words $2^A =
\set[B \subseteq A]{B}$.

\nomenclature[S]{$2^A$}{denotes the set of all the subsets of the set $A$}

\begin{warning}
  Please do not forget about two extremal elements of the power set $2^A$: the
  empty set and $A$ itself.
\end{warning}

\noindent For example if $A = \set{1, 2, 3}$, then
\[
  2^A = \set{\emptyset, \set{1}, \set{2}, \set{3}, \set{1, 2},
  \set{1, 3}, \set{2, 3}, \set{1, 2, 3}}.
\]

\section{The Well-ordering Principle}
Using the set notation we may finally justify the proof of the statement
that $2^n > n$ for all positive integers $n$ from the video about mathematical
induction. In order to do this let us first formulate the following theorem.
\begin{theorem}
\label{theorem:well-ordering}
  Let $A \subseteq \Z$ be a non-empty set. We say that $b \in \Z$ is a lower
  bound for the set $A$ iff $b \le a$ for all $a \in A$. Additionally, we say
  that the set $A$ is bounded if there is a lower bound for $A$.

  Given this, if $A$ is bounded, then there is a lower bound $a \in A$ for the set $A$
  (we say that $a$ is the minimum of the set $A$).
\end{theorem}
Note that this theorem also states that any subset of natural numbers have a
minimum.

Recall that we wish to prove that $2^n > n$ for all positive $n$.
Assume that it is not true, in this case the set
$A = \set[2^n < n]{n \in \N}$ is non-empty. Denote by $n_0$ the minimum of the
set $A$, $n_0$ exists by Theorem~\ref{theorem:well-ordering}. We may consider
the following two cases.
\begin{itemize}
  \item If $n_0 = 1$, then it leads to a contradiction since $2 = 2^1 > 1$.
  \item Otherwise, note that $1 \le n_0 - 1 < n_0$, hence,
    $2^{n_0 - 1} > n_0 - 1$. So $2^{n_0} > 2n_0 - 2 \ge n_0$. Which is a
    contradiction with the definition of $n_0$.
\end{itemize}

Finally, we prove Theorem~\ref{theorem:well-ordering}.
\begin{proof}[Proof of Theorem~\ref{theorem:well-ordering}]
  Let $b$ be a lower bound for the set $A$. Assume that there is no minimum of
  the set $A$. Let $P(n)$ be the statement that
  $n \notin A$.

  First, we are going to prove that $P(n)$ is true for all
  $n \ge b$. The base case is true since if $b \in A$, then $b$ is the minimum
  of $A$ which contradicts to the assumption that there is no minimum of $A$.
  The induction step is also clear, by the induction hypothesis we know that
  $P(b)$, \dots, $P(k)$ are true, hence, $(k + 1) \in A$ implies that $k + 1$
  is the minimum of $A$.

  Now we prove that $A$ is empty. Assume the opposite i.e. assume that there is
  $x \in A$. Note that $x \ge b$ since $b$ is a lower bound of $A$. However,
  $P(x)$ is true which implies that $x \notin A$. Therefore the assumption was
  false and $A$ is empty, but this contradicts to the fact that $A$ is non-
  empty.
\end{proof}

\begin{chapterendexercises}
  \exercise Find the power sets of $\emptyset$, $\set{1}$, $\set{1, 2}$,
    $\set{1, 2, 3, 4}$. How many elements in each of this sets?
  \exercise[recommended] Prove that
    \begin{itemize}
      \item $A \subseteq B \iff A \cup B = B$,
      \item $A \subseteq B \iff A \cap B = A$.
    \end{itemize}
  \exercise Let $A$ be a subset of a set $U$ we call this set a universe.
    We say that the set $\overline{A} = U \setminus A$ is a complement of $A$
    in $U$. Show the following equalities
    \begin{itemize}
      \item $\overline{\overline{A}} = A$.
      \item $\overline{A \cup B} = \overline{A} \cap \overline{B}$.
      \item $\overline{A \cap B} = \overline{A} \cup \overline{B}$.
    \end{itemize}
  \exercise[recommended] Let us define an intersection of more than two sets as follows.
    Let $A_1$, \dots, $A_n$ be some sets. Then
    \begin{itemize}
      \item $\bigcap_{i = 1}^1 A_i = A_1$ and
      \item $\bigcap_{i = 1}^{k + 1} A_i = \left( \bigcap_{i = 1}^k A_i \right)
        \cap A_{k + 1}$.
    \end{itemize}
    \nomenclature[S]{$\bigcap_{i = 1}^k A_i$}{denotes $A_1 \cap \dots \cap
    A_k$}

    Show that $\bigcap_{i = 1}^n \set[i \le x \le n]{x \in \N} =
    \set{n}$ for all integers $n > 0$.
    \begin{solution}
      There are two solutions for this problem which provide two important
      points of view.

      \begin{enumerate}
        \item We may prove the statement using induction by $k$. Note that the
          statement does not involve $k$ so we need to prove a bit different
          statement, we prove
          $\bigcap_{i = 1}^k \set[i \le x \le n]{x \in \N} =
            \set{k, k + 1, \dots, n}$.
          The base case for $k = 1$ is clear since
          $\bigcap_{i = 1}^1 \set[i \le x \le n]{x \in \N} =
            \set[1 \le x \le n]{x \in \N} = \set{1, 2, \dots, n}$.

          The induction step is also easy to see. By the induction hypothesis
          $\bigcap_{i = 1}^k \set[i \le x \le n]{x \in \N} =
            \set{k, k + 1, \dots, n}$.
          Note that
          \begin{multline*}
            \bigcap_{i = 1}^{k + 1} \set[i \le x \le n]{x \in \N} = \\
            \left(
              \bigcap_{i = 1}^k \set[i \le x \le n]{x \in \N}
            \right) \cap
            \set[k + 1 \le x \le n]{x \in \N} =  \\
            \set{k, k + 1, \dots, n} \cap \set{k + 1, \dots, n} =
            \set{k + 1, \dots, n}.
          \end{multline*}
          Thus for $k = n$,
          $\bigcap_{i = 1}^n \set[i \le x \le n]{x \in \N} = \set{n}$.
        \item Another approach is to prove the statement using induction by $n$.
          The base case is also clear,
          $\bigcap_{i = 1}^1 \set[i \le x \le n]{x \in \N} = \set{1}$.

          Let us now prove the induction step. By the induction hypothesis
          $\bigcap_{i = 1}^k \set[i \le x \le k]{x \in \N} = \set{k}$.
          Hence,
          \begin{multline*}
            \bigcap_{i = 1}^k \set[i \le x \le k + 1]{x \in \N} = \\
            \bigcap_{i = 1}^k
              \left(
                \set[i \le x \le k]{x \in \N} \cup \set{k + 1}
              \right)
            \stackrel{*}{=} \\
            \left(\bigcap_{i = 1}^k \set[i \le x \le k]{x \in \N} \right) \cup
            \set{k + 1} =
            \set{k, k + 1};
          \end{multline*}
          note that all these equalities are clear except the equiality marked
          by $*$, we prove it later.

          Thus by the definition of the union
          \begin{multline*}
            \bigcap_{i = 1}^{k + 1} \set[i \le x \le k + 1]{x \in \N} = \\
            \left(
              \bigcap_{i = 1}^k \set[i \le x \le k + 1]{x \in \N}
            \right) \cap \\
            \set[k + 1 \le x \le k + 1]{x \in \N}= \\
            \left(
              \bigcap_{i = 1}^k \set[i \le x \le k + 1]{x \in \N}
            \right) \cap
            \set{k + 1} = \\
            \set{k, k + 1} \cap \set{k + 1} =
            \set{k + 1}.
          \end{multline*}

          Now we need to prove that if $A_1, \dots, A_\ell$, and $B$ are some sets,
          then $\bigcap_{i = 1}^\ell (A_i \cup B) =
          \left(\bigcap_{i = 1}^\ell A_i\right) \cup B$.
          We prove it also using induction by $\ell$. For $\ell = 1$, the
          statement is clear since $\bigcap_{i = 1}^1 (A_i \cup B) = A_1 \cup B
          = \left(\bigcap_{i = 1}^\ell A_i\right) \cup B$.
          Now we need to check the induction step. By the induction hypothesis
          $\bigcap_{i = 1}^k (A_i \cup B) =
            \left(\bigcap_{i = 1}^k A_i\right) \cup B$. Note that
          \begin{multline*}
            \bigcap_{i = 1}^{k + 1} (A_i \cup B) =
            \bigcap_{i = 1}^k (A_i \cup B) \cap (A_{k + 1} \cup B) = \\
            \left(\left(\bigcap_{i = 1}^k A_i\right) \cup B\right) \cap
              (A_{k + 1} \cup B) =  \\
            \left(
              \left(
                \bigcap_{i = 1}^k A_i
              \right) \cap A_{k + 1}
            \right) \cup B =
            \left(\bigcap_{i = 1}^{k + 1} A_i\right) \cup B.
          \end{multline*}
      \end{enumerate}
    \end{solution}
  \exercise Let us define a union of more than two sets as follows.
      Let $A_1$, \dots, $A_n$ be some sets. Then
      \begin{itemize}
        \item $\bigcup_{i = 1}^1 A_i = A_1$ and
        \item $\bigcup_{i = 1}^{k + 1} A_i =
          \left( \bigcup_{i = 1}^k A_i \right) \cup A_{k + 1}$.
      \end{itemize}
      \nomenclature[S]{$\bigcup_{i = 1}^k A_i$}{denotes $A_1 \cup \dots \cup
      A_k$}

      Show that $\bigcup_{i = 1}^n \range{i} = \range{n}$ for all integers
      $n > 0$.
      \begin{solution}
        First, note that $\range{k - 1} \cup \range{k} = \range{k}$ for any
        $k > 1$.

        Now we are ready to prove the statement, we prove it using induction by
        $n$. The base case is true since $\bigcup_{i = 1}^1 \range{i} =
        \range{1}$ by the definition. The induction step is also true since by
        the induction hypothesis $\bigcup_{i = 1}^{k - 1} \range{i} =
        \range{k - 1}$ and by the definition of the union
        $\bigcup_{i = 1}^k \range{i} =
        \left(\bigcup_{i = 1}^{k - 1} \range{i}\right) \cup \range{k}$.
        Hence, $\bigcup_{i = 1}^k \range{i} =
          \left( \bigcup_{i = 1}^{k - 1} \range{i} \right) \cup \range{k} =
          \range{k - 1} \cup \range{k} = \range{k}$.
      \end{solution}
  \exercise[recommended] Let $\Omega$ be some set and $A_1, \dots, A_n \subseteq \Omega$.
    Show that $\bigcup_{i = 1}^n A_i =
      \set[{\exists i \in \range{n}\ x \in A_i}]{x \in \Omega}$.
  \exercise Let $A_1$, \dots, $A_n$ be some sets.
    Show that $\bigcup_{i = 1}^n (A_i \cap B) =
    \left(\bigcup_{i = 1}^n A_i\right) \cap B$.
  \exercise Show that $A \Delta (B \Delta C) = (A \Delta B) \Delta C$.
  \exercise[recommended]
    Let $\mathbb{R}^{m \times n}$ be the set of all matrices $m \times n$ and
    $\mathbb{R}^n$ be the set of $n$ dimensional vectors.
    Show that for any matrix $A \in \mathbb{R}^{m \times n}$
    ($n > m$) there is a nonzero vector $x \in \mathbb{R}^n$ such that $Ax = 0$.
\end{chapterendexercises}


\documentclass{standalone}
\begin{document}
\chapter{Applications of Calculus}
\section{Integrals}
\subsection{Volume of Revolution}

If part	of a curve and the area underneath is rotated about a straight line, the solid formed is called a solid of revolution.\\

Consider the graph $y=f(x)$ and suppose we  rotate the part of the curve from $x=a$ to $x=b$ about the x-axis. If the shaded area of $f(x)$ is notated about the $x$-axis the following shape is formed:

\begin{multicols}{2}
	\begin{center}
		\includegraphics[scale=0.5]{app_of_calc_integ_1}
	\end{center}
	\begin{center}
		~\\
		\includegraphics[scale=0.5]{app_of_calc_integ_2}
	\end{center}
\end{multicols}

Suppose that the solid formed is cut into sections as shown. Let $PQRS$ be a typical section. If the cuts are reasonably close to each other, $PQRS$ approximates a cylinder with height $\delta x$ and radius $y$ as shown below:
\begin{center}
	\qquad \qquad \includegraphics[scale=1]{app_of_calc_integ_3}
\end{center}
The volume, $\delta v$, of $PQRS$ is given by:
$$\delta V \simeq \pi y^2 \delta x$$
Thus, the volume $V$ of solid $PQRS$ is given by:
$$V\simeq \sum_{x=a}^{b} \pi y^2 \delta x$$
This summation approaches $V$ as $\delta x \to 0$
$$V = \lim\limits_{\delta x \to 0}   \sum_{x=a}^{b} \pi y^2 \delta x$$
$$\quad\boxed{V = \int_{a}^{b} \pi y^2 \, dx}$$
\hrulefill
\begin{example}
	Find the volume generated when the area  between $y=e^x$, the x-axis
\end{example}

\begin{alignat*}{2}
	&   & V & =\int_{0}^{1} \pi y \, dx \\
\end{alignat*}

\begin{example}
	Find the volume generated when the area defined by the inequalities $y\leq x^2$ and $y\geq x$ is rotated about the $x$-axis.
\end{example}
\begin{multicols}{2}
	\begin{tikzpicture}
		\begin{axis}[
			width=8cm,
			height=6cm,
			axis line style={-},
			xmin=0,
			xmax=2.5,
			ymin=0,
			ymax=2.5,
			xtick={}, % remove all ticks from x-axis
			ytick={}, % ditto for y-axis
			xlabel=$x$,
			ylabel=$y$,
			axis lines=center, % default is to make a box around the axis
			samples=100]
			\addplot [name path=A,samples=501, domain=0:2, black] {sqrt(x)};
			\addplot [name path=B,samples=501, domain=0:2, green]  {x};
			\draw[pattern=north east lines,
			intersection segments={
				of=A and B,
				sequence={L2--R2[reverse]}
			}];
		\end{axis}
	\end{tikzpicture}
	\begin{center}

	\end{center}

\end{multicols}
\begin{alignat*}{2}
	&   & V_{e-c} & = V_e - V_c                                                      \\
	&   &         & = \pi \int_{0}^{1} \sqrt{x}^2 \, dx - \pi \int_{0}^{1} x^2 \, dx \\
\end{alignat*}
\begin{example}
	Find the volume generated when the area in the first quadrant bounded by the circle $x=4\cos\theta, y=4\sin\theta$ rotates completely about the $x$-axis.
\end{example}
\begin{example}

	Find the volume when the region defined by  $y\geq x^2+1,\quad x\geq0$ and $y\leq2$ is rotated about the $y$-axis.
\end{example}
\begin{tikzpicture}
	\begin{axis}[
		width=8cm,
		height=6cm,
		axis line style={-},
		xmin=0,
		xmax=2.5,
		ymin=0,
		ymax=2.5,
		xtick={}, % remove all ticks from x-axis
		ytick={}, % ditto for y-axis
		xlabel=$x$,
		ylabel=$y$,
		axis lines=center, % default is to make a box around the axis
		samples=100]
		\path[name path=axis] (axis cs:0,0) -- (axis cs:0,2);
		\addplot [name path=A, domain=0:2, black] {2};
		\addplot [name path=B, domain=0:2, black]  {x^2 + 1};
		\draw[pattern=north east lines,
		%TODO fix this
		intersection segments={
			of=B and A,
			of=axis and B,
			of=A and axis,
			sequence={L2--R	2[reverse]}
		}];
		%	\draw[pattern=north east lines,
		%	intersection segments={
		%		of=axis and B,
		%	}];
	\end{axis}
\end{tikzpicture}
\subsection{Length of an arc of a curve}
To find the length of an arc of a curve we use the method of summing small elements of one length.\\
Suppose that arc $PQ$, of length $\delta s$, is such an element. Then, the length $S$ of the curve $AB$ is given by:
\begin{multicols}{2}
	\begin{center}
		$\sum_{x=x_1}^{x_2} \delta S$
	\end{center}
	\begin{center}
		\includegraphics[width=0.7\linewidth]{app_of_integ_4}
	\end{center}
\end{multicols}

As $\delta s$ is very small, it can be approximated to the hypothenuse of the $\triangle PQR$. Thus:
\begin{alignat*}{2}
	&            & (\delta x)^2 + (\delta y)^2                    & \simeq (\delta s)^2                                                                 \\
	& \implies   & 1 + \left( \frac{\delta y}{\delta x}\right) ^2 & \simeq \left( \frac{\delta S}{\delta x}\right) ^2                                   \\
	& \implies   & \delta S                                       & \simeq \sqrt{1+\left( \frac{\delta y}{\delta x}\right)^2 } \delta x                 \\
	& \therefore & S                                              & \simeq \sum_{x=x_1}^{x_2} \sqrt{1 + \left(\frac{\delta y}{\delta x}\right)}\delta x
\end{alignat*}
~\\
~\\
As $\delta x \leftarrow 0, (\frac{\delta y}{\delta x}) \leftarrow \frac{dy}{dx} \quad $and$ \quad s = \lim_{\delta x \to 0} \sum_{x=x_1}^{x_2} \sqrt{1+\left( \frac{\delta y}{\delta x}\right)^2}  \delta x$
~\\
~\\
~\\

\begin{center}
	\begin{tcolorbox}[center title,hbox,
		lifted shadow={1mm}{-2mm}{3mm}{0.1mm}%
		{black!50!white}]
		\begin{varwidth}{\textwidth}
			\begin{center}
				$\therefore \quad S = \int_{x_1}^{x_2} \sqrt{1 + \left(\frac{dy}{dx}\right)^2}\, dx$
			\end{center}
		\end{varwidth}
	\end{tcolorbox}
\end{center}

Knowing the Cartesian equation of the curve, the necessary integration can be carried out.\\

Let us now consider a curve $S$ parametrically in terms of $t$.

We can use again :
\begin{alignat*}{2}
	&          & (\delta s )^2                            & \simeq (\delta x)^2 + (\delta y)^2                                                                         \\
	& \implies & \left(\frac{\delta s}{\delta t}\right)^2 & \simeq \left(\frac{\delta x}{\delta t}\right)^2 + \left(\frac{\delta y}{\delta t}\right)^2                 \\
	& \implies & \delta s                                 & \simeq \sqrt{\left(\frac{\delta x}{\delta t}\right)^2 + \left(\frac{\delta y}{\delta t}\right)^2} \delta t
\end{alignat*}

~\\
as $\delta t \to 0\quad ; \quad \frac{\delta x}{\delta t} \to \frac{dx}{dt} \quad \text{and} \quad \left(\frac{\delta y}{\delta t}\right) \to \frac{dy}{dt}$\\
~\\

$$\therefore S = \lim_{\delta t \to 0} \sum_{t=t_1}^{t_2} \sqrt{\left(\frac{\delta x}{\delta t}\right)^2 + \left(\frac{\delta y}{\delta t}\right)^2} \delta t$$

\begin{center}
	\begin{tcolorbox}[center title,hbox,    %%<<---- here
		lifted shadow={1mm}{-2mm}{3mm}{0.1mm}%
		{black!50!white}]
		\begin{varwidth}{\textwidth}
			\begin{center}
				$S = \int_{t_1}^{t_2} \sqrt{\left(\frac{dx}{dt}\right)^2 + \left(\frac{dy}{dt}\right)^2}\, dt$
			\end{center}
		\end{varwidth}
	\end{tcolorbox}
\end{center}
\section{Rates of change}
The notation $\dfrac{dy}{dx}$ denotes the rate of change of $y$ w.r.t.x. Suppose that $x$ and $y$ are quantities a such as length and volume respectively. Then $\dfrac{dy}{dx}$ denotes the rate of change of volume w.r.t some length. In this section we use differentiation to deal with practical problems involving rates of change.

\begin{example}
	A spherical balloon is blown up so that its volume increases at a constant rate of $2\text{cm}^3/s$. Find the rate of increase of the radius when the volume of the balloon is $50\text{cm}^3$
\end{example}

\begin{multicols}{2}
	$\od{V}{t} = 2$

\end{multicols}
\begin{example}
	A container with water is in the form of an inverted hollow cone with a iaerrentical angle of $30\degree$. Water drips out from the vertex at the rate $3\text{cm}^2/s$. Find the rate at which the surface area in contact with the water is changing when there are $8\pi\text{cm}^3$ of water remaining in the cone.

	\begin{figure}
		\includegraphics[scale=0.5]{app_of_calc_der_1}
	\end{figure}
	\begin{multicols}{2}
		\begin{align*}
			\od{V}{t} & = -3                            \\
			V         & = \frac{1}{3} \pi r^2 h         \\
			V         & = \frac{1}{3} \pi r^2 \sqrt{3}r \\
			V         & = \frac{\sqrt{3}}{3}\pi r^3     \\
			\od{V}{t} & = \sqrt{3}\pi  r^2
		\end{align*}
		\begin{align*}
			\text{Required derivative: } \od{A}{t} &          \\
			A                                      & = \pi rl
		\end{align*}

	\end{multicols}
\end{example}
\end{document}


\lab{Correlation and Covariance}{Correlation and Covariance}
\label{lab:Stats1}

\objective{Explore applications of inner product spaces to topics in statistics.}

% Source of the datasets: http://people.sc.fsu.edu/~jburkardt/datasets/regression/regression.html
% under the GNU LGPL liscence

\section*{Shifting Data by the Mean}
When analyzing numerical data, it is often useful to transform the data to have an average value of 0.
This process, which we call shifting by the mean,
is easily accomplished by simply subtracting the mean of the data from each value.
The resulting shifted data now shows how each data point deviates from the mean.
This form makes it easier to spot outliers, judge the spread of the data, and compare with other data sets.

Consider Table \ref{tab:data} representing students scores in a class.\\

\begin{figure}
\begin{center}
\begin{tabular}{|c|r|r|r|r|}
	\hline
Student & Homework & Exam 1  & Exam 2 & Final \\
\hline
S1  & 89 & 91 & 77 & 75 \\
S2  & 67 & 72 & 76 & 66 \\
S3  & 72 & 77 & 69 & 70 \\
S4  & 56 & 60 & 55 & 61 \\
S5  & 92 & 98 & 89 & 86 \\
S6  & 83 & 88 & 90 & 84 \\
S7  & 45 & 60 & 55 & 48 \\
\hline
Average  & 72 & 78 & 73 & 70\\
\hline
\end{tabular}\\
\end{center}
\caption{Homework and exam scores for 7 students in a class.}
\label{tab:data}
\end{figure}

We can shift our data set by subtracting each column by its average value.
This makes it so that the average of each column is zero.
If $W$ represents the matrix of scores, the following Python command will shift each column by the mean.
\begin{lstlisting}
>>> W = np.array([[89, 91, 77, 75],
                  [67, 72, 76, 66],
                  [72, 77, 69, 70],
                  [56, 60, 55, 61],
                  [92, 98, 89, 86],
                  [83, 88, 90, 84],
                  [45, 60, 55, 48]])
>>> column_means = W.mean(axis=0)
>>> X = W - column_means
>>> X
array([[ 17.,  13.,   4.,   5.],
       [ -5.,  -6.,   3.,  -4.],
       [  0.,  -1.,  -4.,   0.],
       [-16., -18., -18.,  -9.],
       [ 20.,  20.,  16.,  16.],
       [ 11.,  10.,  17.,  14.],
       [-27., -18., -18., -22.]])
\end{lstlisting}

Once we have subtracted out the mean from a vector, it is a simple task to compute its variance and standard deviation.
The \emph{variance} of an $n$-dimensional vector $v$, often denoted $\sigma^2$, is defined by the formula
\[
\sigma^2 = \frac{1}{n}\displaystyle\sum_{i=1}^n (v_i - \mu)^2,
\]
where $\mu$ is the mean of $v$. The \emph{standard deviation} of $v$ is simply $\sigma$, i.e. the square
root of the variance. These quantities measure the spread of the entries of the vector. When all of the entries are clustered
closely around the mean, the variance is small. See Figure \ref{fig:variance} for an illustration of variance and mean-shifted data.

(Note: \emph{unbiased variance} is a quantity closely related to variance, but defined slightly differently. It comes into play
in particular for small datasets or vectors. Further discussion is beyond the scope of this lab, but be aware that there is
more than one type of variance.)

\begin{figure}
\includegraphics[width=\textwidth]{variance}
\caption{On the left, the red values come from a vector with relatively high variance,
         and the blue values come from a vector with relatively low variance.
         On the right, the green values have been shifted by the mean from the blue values.}
\label{fig:variance}
\end{figure}

Calculating the variance and the standard deviation in Python is straight forward.
Given the shifted data $X$, we simply square each entry, sum along the columns, and divide by the number of rows
to obtain the variance. Take the square root of the result to get the standard deviation.
\begin{lstlisting}
>>> var = (X**2).sum(axis=0)/X.shape[0]
>>> var
array([ 260.     ,  193.42857,  176.28571,  151.14285])
>>> std = np.sqrt(var)
>>> std
array([ 16.124515,  13.907860,  13.277263,  12.294017])
\end{lstlisting}
Observe that the scores for the homework exhibited the largest amount of spread.

As with many things in Python, there is an even shorter, more convenient way to calculate these quantities.
We may calculate the variance and standard deviation of the columns directly from $W$, without having to shift
by the mean.
\begin{lstlisting}
>>> W.var(axis = 0)
array([ 260.     ,  193.42857,  176.28571,  151.14285])
>>> W.std(axis = 0)
array([ 16.124515,  13.907860,  13.277263,  12.294017])
\end{lstlisting}

\begin{problem}
\label{prob:shiftdata}
Import the dataset contained in the \li{weight_age_fat.txt} file. The first row is a header, and contains the names
of each column, but no actual data. There are five columns, but the first two do not contain any data of interest.
The last three columns contain data on the weight (in kilograms), age (in years), and blood fat content of 25
individuals. Distinct data entries are separated by whitespace.

Write a function \li{shiftByMean} that shifts the columns of an input array by their respective means, and returns the result.
Next, write a function \li{computeVariance} that calculates and returns the variance of each column of an input array.
Finally, write a function \li{reportStDev} that accepts no parameters and returns nothing, and simply contains a print
statement that prints out the name of the column with the smallest standard deviation as well as the numerical value of
its standard deviation.

\end{problem}

\section*{The Inner Product and Angles Formula}

Inner products give information about lengths of vectors and angles between vectors.
Recall that the standard inner product on $\mathbb{R}^n$ between vectors $v$ and $u$ is given by
\[
\ipt{v}{u} = v^T u = \displaystyle\sum_{i=1}^n v_iu_i.
\]
We can take advantage of convenient syntax in NumPy to quickly calculate the inner product between two vectors:
\begin{lstlisting}
>>> v = np.array([1., -2., 4.])
>>> u = np.array([2., 3., -1.])
>>> v.dot(u)
-8.0
\end{lstlisting}
The 2-norm of a vector can be easily recovered from the inner product:
\begin{lstlisting}
>>> v_norm = np.sqrt(v.dot(v))
>>> v_norm
4.5825756949558398
\end{lstlisting}
Another option to calculate the norm of an array is to use the \li{numpy.linalg.norm} function, which has the
capability of computing a variety of different types of norms.
It is also convenient for computing norms of columns or rows of matrices, as follows:
\begin{lstlisting}
>>> from numpy.linalg import norm
>>> # calculate norms of the columns of X
>>> norm(X, axis=0)
array([42.661, 36.797, 35.128, 32.527])
\end{lstlisting}

Recall that the angle $\theta$ between two nonzero
vectors $v$ and $u$ is given by
\[
\cos{\theta} = \frac{\ipt{v}{u}}{\norm{v}\norm{u}}
\]
where $\norm{v} = \sqrt{\ipt{v}{v}}$ denotes the 2-norm of $x$.
By bringing the constants into the inner product, we see that the angle satisfies
\[
\cos{\theta} = \left\langle\frac{v}{\norm{v}},\frac{u}{\norm{u}}\right\rangle.
\]
Hence, if we view the columns of $X$ as vectors in $\mathbb{R}^7$, we can find the cosine of the angles between these columns by dividing each
by its length, and then computing pairwise inner products.
To divide the columns by their respective lengths, we may execute the following code.
\begin{lstlisting}
>>> Y = X/norm(X, axis=0)
>>> Y
array([[ 0.39848615,  0.35329218,  0.11386819,  0.15371887],
       [-0.11720181, -0.16305793,  0.08540114, -0.12297509],
       [ 0.        , -0.02717632, -0.11386819,  0.        ],
       [-0.37504578, -0.48917378, -0.51240685, -0.27669396],
       [ 0.46880723,  0.54352643,  0.45547275,  0.49190037],
       [ 0.25784398,  0.27176321,  0.4839398 ,  0.43041282],
       [-0.63288976, -0.48917378, -0.51240685, -0.67636301]])
\end{lstlisting}

Finally, we get the cosines of the angles between columns by computing $Y^T Y$. To justify this calculation, consider
the $(i,j)^{th}$ entry of $Y^T Y$:
\begin{align*}
(Y^T Y)_{i,j} &= \sum_{k=1}^n (Y_{i,k})(Y^T)_{k,j} \\
&= \sum_{k=1}^n Y_{i,k}Y_{j,k} \\
&= \left\langle Y_{:,i}, Y_{:,j} \right\rangle,
\end{align*}
where $Y_{:,i}$ denotes the $i^{th}$ column of $Y$, and $Y_{:,j}$ the $j^{th}$ column.
Thus, we may obtain the cosine of the angle between each pair of columns of $X$ by computing $Y^T Y$,
which can be easily done in Python as follows:
\begin{lstlisting}
>>> np.dot(Y.T,Y)
array([[ 1.        ,  0.97782999,  0.89014869,  0.94908967],
       [ 0.97782999,  1.        ,  0.90978843,  0.92490144],
       [ 0.89014869,  0.90978843,  1.        ,  0.9276955 ],
       [ 0.94908967,  0.92490144,  0.9276955 ,  1.        ]])
\end{lstlisting}

We remark that the diagonals are always equal to one because the angle between a vector and itself is zero, and the cosine of zero is one.

%It is also worth noting that the matrix resulting from the operation $A \cdot A^T$ is symmetric and positive definite, so it compliant
%with the Cholesky decomposition that was discussed at the end of Lab \ref{lab:LUdecomp}.
% ^^ I'm not totally sure what is going on with this last statement. What exactly is A \cdot A^T? I'm assuming we mean A^T A, in which case
% the matrix is in fact only guaranteed to be positive semi-definite. It still has a Cholesky decomposition in theory, although I think it can only be
% computed for positive definite matrices. Anyway, I don't think the comment is relevant, so I have it removed for now.

\section*{Correlation}
In statistics, \emph{correlation} is a broad term that refers to various types of statistical relationships and dependence between
variables of interest.
In this setting, the \emph{Pearson correlation coefficient} of two vectors $u$ and $v$ is defined to be the cosine of the angle between the
vectors $\overline{u}$ and $\overline{v}$, where $\overline{u}$ is the vector obtained by subtracting the mean of $u$ from each of its entries,
and similarly for $\overline{v}$.
Two vectors are said to be
\begin{itemize}
\item Perfectly correlated if their correlation coefficient is equal to 1.
\item Positively correlated if their correlation coefficient is between 0 and 1.
\item Uncorrelated if their correlation coefficient is equal to 0.
\item Negatively correlated if their correlation coefficient is between -1 and 0.
\item Perfectly anticorrelated if their correlation coefficient is equal to -1.
\end{itemize}

If we have an array whose columns are the vectors of interest, we may calculate a correlation matrix
by first shifting the columns by their mean, and then computing the matrix of angles as described in the
previous section. The $(i,j)$ entry of the resulting correlation matrix gives the correlation coefficient
of columns $i$ and $j$.

\begin{problem}
Write a function \li{corrMatrix} that calculates and returns the correlation matrix of the columns of the input array.
\end{problem}
Again, NumPy provides a method for computing the correlation matrix of the columns or rows of an array via the function \li{np.corrcoef}.
Be aware of the keyword argument \li{rowvar}.
Nonzero integer values for \li{rowvar} will yield the correlation matrix of the rows, whereas a value of 0 for \li{rowvar} will produce the correlation matrix of the columns.
\begin{lstlisting}
>>> np.corrcoef(W, rowvar=0)
array([[ 1.        ,  0.97782999,  0.89014869,  0.94908967],
       [ 0.97782999,  1.        ,  0.90978843,  0.92490144],
       [ 0.89014869,  0.90978843,  1.        ,  0.9276955 ],
       [ 0.94908967,  0.92490144,  0.9276955 ,  1.        ]])
\end{lstlisting}
Time both your implementation as well as that of NumPy. Which is faster?

The notion of correlation is important in establishing linear relationships between measurements.
Parallel vectors pointing in the same direction are perfectly correlated, since they lie in the same line.
Orthogonal vectors, on the other hand, are uncorrelated.
Given two vectors of the same length, one can visually check for correlation by viewing a scatter plot.
This can be done in Python as follows:
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> x = np.random.rand(100)
>>> y = np.random.rand(100)
>>> plt.scatter(x,y)
>>> plt.show()
>>> plt.clf()
\end{lstlisting}
You will observe that the scatter plot does not indicate any obvious linear relationship between the two
vectors. You can calculate the correlation coefficient and confirm that it is close to zero.
See Figure \ref{fig:correlation} for examples of correlated and uncorrelated data.

\begin{figure}[h]
\includegraphics[width=\textwidth]{correlation}
\caption{Left: scatter plot of two slightly positively correlated vectors.
         Middle: scatter plot of two uncorrelated vectors.
         Right: scatter plot of two highly negatively correlated vectors.}
\label{fig:correlation}
\end{figure}

\begin{problem}
Import the data contained in the \li{mortality.txt} file. The first 17 rows are headers, providing the names of the
17 columns, and do not contain data. The first column is simply an index column, and may also be omitted.
The following 16 columns provide various demographic and environmental data for 60 countries, with the
final column giving the mortality rate for that country.
Distinct data entries are separated by whitespace.

Write a function that prints the answers to the following three questions, and generates plots as
described below.
Between which pair of distinct columns is the highest correlation?
Which column (apart from the last column) has the most negative correlation coefficient with mortality rate?
Which column is most nearly uncorrelated with mortality rate (i.e. the correlation coefficient is closest to 0)?
Make scatter plots of all three of these pairs of columns, and include them on the same subplot panel.
In all relevant cases, the mortality rate column should be the second argument passed to \li{plt.scatter}.
\end{problem}

It's important to understand that the high correlation between quantities does not necessarily imply
a causal relationship. For example, high correlation between violent crime rates and ice cream sales has
been observed. This does not mean that ice cream causes crime or that increases in crime makes people want
to eat more ice cream. Rather, both rates happen to increase during the summer and decrease during the winter.

Another import consideration is that the correlation coefficient does not provide information about
non-linear relationships and dependencies between data (see Figure \ref{fig:non_linear}). Any thorough analysis will go well beyond a simple
calculation of the correlation matrix.

\begin{figure}
\includegraphics[width=\textwidth]{nonlinear_dependence}
\caption{There is a clear nonlinear relationship present in this data, but the Pearson correlation
         coefficient, which has a value of -0.00056, fails to capture it.}
\label{fig:non_linear}
\end{figure}

\section*{Covariance}
Related to the notion of correlation is that of covariance.
The covariance of two vectors measures how they jointly vary together.
In mathematical terms, given two vectors $u$ and $v$ in $\mathbb{R}^n$, with $\bar{u}$ and $\bar{v}$ denoting the mean-shifted
versions of these vectors (as before), we define the \emph{covariance} of $u$ and $v$ to be
\[
\text{cov}(u,v) = \frac{\ipt{\bar{u}}{\bar{v}}}{n-1}.
\]
If $u$ and $v$ are close to parallel and both have high variance, then their covariance will be large in magnitude.
On the other hand, if $u$ and $v$ are nearly orthogonal (that is, nearly uncorrelated), or one has very small variance,
then their covariance will be small in magnitude.

Just as with correlation, we can generate a covariance matrix for a collection of vectors that contains the covariance for each set of vectors.
Specifically, given a collection of vectors $x_1,\ldots, x_m$, we define the covariance matrix $C$ for this collection to be the $m \times m$ matrix satisfying
\[
C_{i,j} = \text{cov}(u,v).
\]
If we let $X$ be the matrix whose columns are the mean-shifted vectors $\bar{x_1}, \ldots, \bar{x_m}$, note that
\begin{equation}
C = \frac{1}{n-1}X^TX.
\label{eq:cov}
\end{equation}
This gives us a very straight-forward way to compute the covariance matrix in Python:
\begin{lstlisting}
>>> # we will calculate the covariance matrix of W
>>> # subtract out the mean from the columns
>>> X = W-W.mean(axis=0)
>>> C = X.T.dot(X)/(X.shape[0]-1)
array([[ 303.33333333,  255.83333333,  222.33333333,  219.5       ],
       [ 255.83333333,  225.66666667,  196.        ,  184.5       ],
       [ 222.33333333,  196.        ,  205.66666667,  176.66666667],
       [ 219.5       ,  184.5       ,  176.66666667,  176.33333333]])
\end{lstlisting}
As usual, there is a nice way to compute the covariance matrix in NumPy, using the function \li{np.cov}. This function
also has the keyword argument \li{rowvar} which determines whether the covariance of the rows or of the columns is computed.
\begin{lstlisting}
>>> np.cov(W, rowvar=0)
array([[ 303.33333333,  255.83333333,  222.33333333,  219.5       ],
       [ 255.83333333,  225.66666667,  196.        ,  184.5       ],
       [ 222.33333333,  196.        ,  205.66666667,  176.66666667],
       [ 219.5       ,  184.5       ,  176.66666667,  176.33333333]])
\end{lstlisting}

\begin{problem}
Compute and report the covariance matrix of the \li{mortality.txt} data set.
\end{problem}

Note that the covariance matrix is symmetric.
Indeed, from equation \ref{eq:cov}, we can see that it is positive semi-definite and has nice spectral properties.
Observe from that same equation that the spectral decomposition of the covariance matrix is closely related to the singular value decomposition of $X$.
It turns out that the covariance matrix and its spectral properties contain much information about relationships, patterns, and structure in the data.
Several statistical techniques in data compression, dimensionality reduction, and information retrieval rely on discovering and exploiting this structure,
and hence the covariance matrix and the singular value decomposition play an important role in data analysis.

\section*{Latent Semantic Indexing}
We now have the tools to build a simple, yet effective, system for indexing and retrieving documents according to their semantic content.
The method that we will implement is called \emph{Latent Semantic Indexing}, or LSI.

Suppose we have a collection of documents, which we call a \emph{corpus}, and we wish to find the particular documents
that are most closely related to a given sequence of search terms.
This problem arises when searching a database of newspaper articles, an email inbox, a scientific journal, and in many more settings.
When our corpus consists of many documents, it becomes infeasible to read, or even skim, each document on our own.
We need to utilize our mathematical and computational skills to facilitate the process.
To begin, we create a numerical representation of our corpus.
A common approach, known as the \emph{vector space model}, is as follows.
First, specify a list $v$ of words of interest (this could just be all the unique words found in the corpus).
Let $m$ be the length of the list $v$. Then we can represent a given document by a $m$-dimensional vector
\[
d = \begin{bmatrix}
d_1 & d_2 & \cdots & d_m
\end{bmatrix}^T,
\]
where $d_i \in \mathbb{R}$ gives the contribution of the $i$-th term in the list $v$ to the document.
For our purposes in this lab, we define $d_i$ to be the number of times term $i$ occurs in the document.
The following code snippet shows how to compute the vector representation of a document given a list of words.
We use a particular data structure \li{Counter}, which counts the number of occurrences of each unique element in a list.
\begin{lstlisting}
>>> from collections import Counter
>>> # initialize the word list and the document
>>> v = ["the", "great", "live", "math", "things", "do", "me", "you"]
>>> doc = "If you cannot do great things do small things in a great way"
>>> # split the document into a list of words, make counter
>>> count = Counter(doc.split())
>>> # create the vector representation
>>> d = np.array([count[w] for w in v])
>>> print d
[0 2 0 0 2 2 0 1]
\end{lstlisting}
If our corpus consists of $n$ documents, then we can represent the entire corpus by a $m \times n$ matrix
$D$ whose columns are just the vector representations of the documents.
In most real-world applications, $m$ and $n$ are very large (in the several thousands, or more).
Thankfully, the matrix $D$ will often be very sparse, since each particular document contains only a small subset
of the entire list of words.
Thus, we can use the sparse libraries in SciPy to efficiently store and compute with this matrix.

We now use the singular value decomposition to reduce the dimensionality of the matrix representation of our
corpus in a way that optimally decouples the semantic content of the documents.
More precisely, we specify the number $t$ of greatest singular values we wish to retain, and calculate the truncated SVD of the matrix $D$,
yielding a rank-$t$ approximation
\[
D_t = U_t\Sigma_tV_t^T,
\]
where $U_t$ and $V_t$ are orthogonal matrices of size $m \times t$ and $n \times t$, respectively, and $\Sigma_t$
is a $t \times t$ diagonal matrix containing the $t$ largest singular values of $D$.
A typical value for $t$ is in the low hundreds.
We can compute this in Python as follows:
\begin{lstlisting}
>>> from scipy.sparse import linalg as la
>>> # assume D is a sparse matrix
>>> t = 100
>>> U, Sig, VT = la.svds(D, k=t)
\end{lstlisting}
Although $U_t$ and $V_t$ are not sparse, they are much more feasible to compute with given that $t \ll m$.
Observe that the $i$-th column of $V_t^T$ is $\Sigma_t^{-1}U_t^Td_i$, a transformed version of the $i$-th document vector.
It turns out that the matrix $\Sigma_t^{-1}U_t^T$ defines a projection from $\mathbb{R}^m$ onto a $t$-dimensional subspace
in such a way that the projections of semantically similar documents have high correlation in the subspace, and
semantically differing documents will have low correlation.
Thus, if $q$ is the vector representation of a query, we can find the document in the corpus with the closest semantical
content to $q$ by projecting $q$ down to the $t$-dimensional subspace, calculating its correlation with each of the
projected document vectors in the corpus, and then selecting the document with the highest such correlation.
\begin{comment}
The discussion above is still lacking, and doesn't quite hit the nail on the head.
But I don't want it to become to wordy and bloated, so I'll leave it as is for now.
\end{comment}
\begin{lstlisting}
>>> # assume that docs is a list of the original documents
>>> # assume that q is the vector representation of a query
>>> # first, project q to the subspace
>>> q = (1/S)*(U.T.dot(q))
>>> # now calculate correlation of q with each column of VT
>>> q -= q.mean()           # shift by mean
>>> VT -= VT.mean(axis=0)
>>> q /= norm(q)            # normalize
>>> VT /= norm(VT, axis=0)
>>> coefs = VT.T.dot(q)
>>> # select the index of the highest correlation
>>> ind = np.argmax(coefs)
>>> print docs[ind]
\end{lstlisting}
\begin{problem}
Do LSI and stuff.
\end{problem}

As a final note, the concepts of variance, standard deviation, correlation, and covariance apply to a much broader class of objects
known as \emph{random variables}, which are central to statistics and probability theory. These topics will be
covered in detail later on.


\subsection{System}
$$
\ddot{x}(t) = -x(t) - 0.1\dot{x}(t) + u
$$
$$
\begin{bmatrix}
	\dot{x}_1\\
	\dot{x}_2
\end{bmatrix} = \begin{bmatrix}
	-1 & 0\\
	0 & -0.1
\end{bmatrix}\begin{bmatrix}
	{x}_1\\
	{x}_2
\end{bmatrix} +\begin{bmatrix}
	0\\1
\end{bmatrix}u
$$
$$
A = \begin{bmatrix}
	-1 & 0\\
	0 & -0.1
\end{bmatrix}, \quad B = \begin{bmatrix}
	0\\1
\end{bmatrix}, \qquad Q = \begin{bmatrix}
	1 & 0\\
	0 & 1
\end{bmatrix}, \qquad R = 1, \qquad H = \begin{bmatrix}
	1 & 0\\
	0 & 1
\end{bmatrix},
$$
\subsection{part a}
Gradient tolerance is: $10^{-4}$

\subsubsection{figures}
\newpage
\begin{itemize}
	\item Steepest Descent
	\begin{itemize}
		\item Quadratic Interpolation
		\begin{figure}[H]
			\caption{Steepest Descent and Quadratic Interpolation}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q2/part a Steepest Descent + Quadratic Interpolation.png}
		\end{figure}
		\item Golden Section
		\begin{figure}[H]
			\caption{Steepest Descent and Golden Section}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q2/part a Steepest Descent + Golden Section.png}
		\end{figure}
	\end{itemize}
	\item BFGS
	\begin{itemize}
		\item Quadratic Interpolation
		\begin{figure}[H]
			\caption{BFGS and Quadratic Interpolation}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q2/part a BFGS + Quadratic Interpolation.png}
		\end{figure}
		\item Golden Section
		\begin{figure}[H]
			\caption{BFGS and Golden Section}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q2/part a BFGS + Golden Section.png}
		\end{figure}
	\end{itemize}
\end{itemize}
\subsubsection{result}
\begin{itemize}
	\item Time
	\begin{table}[h]
		\caption {Time compare between four methods}
		\begin{center}
			\begin{tabular}{ |c|c|c|c| }
				\hline
				\multicolumn{2}{|c|}{Steepest Descent} &
				\multicolumn{2}{|c|}{BFGS} \Tstrut\\
				\hline
				Quadratic Interpolation & Golden Section & Quadratic Interpolation &
				Golden Section \Tstrut\\
				\hline
				$17.000\sec$ & $24.353\sec$ & $3.905\sec$ & $4.985\sec$\Tstrut\\
				\hline
			\end{tabular}
		\end{center}
	\end{table}
	\item Number of Cost calculation
	\begin{table}[h]
		\caption {Number of Cost calculation compare between four methods}
		\begin{center}
			\begin{tabular}{ |c|c|c|c| }
				\hline
				\multicolumn{2}{|c|}{Steepest Descent} &
				\multicolumn{2}{|c|}{BFGS} \Tstrut\\
				\hline
				Quadratic Interpolation & Golden Section & Quadratic Interpolation &
				Golden Section \Tstrut\\
				\hline
				$1285$ & $1922$ & $273$ & $373$\Tstrut\\
				\hline
			\end{tabular}
		\end{center}
	\end{table}
	\item Number of Gradient calculation
	\begin{table}[h]
		\caption {Number of Gradient calculation compare between four methods}
		\begin{center}
			\begin{tabular}{ |c|c|c|c| }
				\hline
				\multicolumn{2}{|c|}{Steepest Descent} &
				\multicolumn{2}{|c|}{BFGS} \Tstrut\\
				\hline
				Quadratic Interpolation & Golden Section & Quadratic Interpolation &
				Golden Section \Tstrut\\
				\hline
				$51$ & $51$ & $11$ & $11$\Tstrut\\
				\hline
			\end{tabular}
		\end{center}
	\end{table}
\end{itemize}
\subsubsection{Four iteration for BFGS and Quadratic interpolation}
		\begin{figure}[H]
	\caption{BFGS and Quadratic Interpolation with four iteration}
	\centering
	\includegraphics[width=11.5cm]{../Figure/Q2/part a_2 BFGS + Quadratic Interpolation.png}
\end{figure}
	\begin{table}[H]
	\caption {four iteration and gradient tolerance compare}
	\begin{center}
		\begin{tabular}{ c|c|c|c| }
			\cline{2 - 4}
			 &Time & Number of Cost calculation & Number of Gradient calculation\Tstrut\\
			 \hline
			 \multicolumn{1}{ |c|  } {Four iteration} & $3.905_{\sec}$ & $273$ & $11$ \Tstrut\\
			\hline
			\multicolumn{1}{ |c|  }{Gradient tolerance} & $1.586_{\sec}$ & $100$ & $4$ \Tstrut\\
			\hline
		\end{tabular}
	\end{center}
\end{table}
Between BFGS and Steepest Descent, BFGS if more faster but Steepest Descent is more easy to use. Quadratic interpolation can be faster when cost function use so much process but when we are so close to answer the function doesn't work well so we must increase gradient tolerance.


\subsection{Image and Kernel}

If $T:\;\R^m\rightarrow \R^n$ then $\mathrm{Im}(T)\subset \R^n$ and
$\mathrm{ker}(T)\subset \R^m$ because the associated matrix $A$ is $n\times m$ in dimension.\newline

\noindent
Both are closed under linear combinations:
\begin{itemize}
    \item If $\tb y_1 ,\tb y_2 \in \mathrm{Im}(T)$ then $a \tb{y}_{1}+b \tb{y}_{2} \in \mathrm{Im}(T)$ as well
    \item If $\tb x_1 ,\tb x_2 \in \mathrm{Ker}(T)$ then $a\tb x_1 +b\tb x_2 \in \mathrm{Ker}(T)$ as well
\end{itemize}

\subsection{Subspaces}

Collection of vectors in $\R^n$ is called a subspace in $\R^n$ if collection is nonempty and closed under linear combinations.
Examples (and counterexamples):

\begin{itemize}
    \item $W=\left\{\left(\begin{array}{c}
        3 s \\
        2+5 s
        \end{array}\right) \mid s \in \mathbb{R}\right\} \subset \mathbb{R}^{2}$
        is not a subspace because $\tb{0}$ is not contained within the set, so not closed under linear combinations
    \item $W=\left\{\left(\begin{array}{l}
        x_{1} \\
        x_{2} \\
        x_{2}
        \end{array}\right) \mid 2 x_{1}+x_{2}-x_{3}=0\right\}$ is a subspace due to matrix representation and
        the image of this matrix containing $\tb{0}$ due to $T(\tb{0}=\tb{0})$
\end{itemize}

\noindent
$\R^2$ is not a subspace of $\R^3$ because though a plane can be drawn in $\R^3$, its components will be of the form
$\begin{bmatrix}x\\ y\\k\end{bmatrix}$, where $k$ is fixed. Since $\R^3$ vectors always have 3 coordinates, they can't represent $\R^2$.
$\R^2$ can only be represented by $\R^2$ vectors. Thus, $R^n$ is not a subspace of $\R^{n+1}$.\\

\noindent
\textbf{\textit{Claim:}} Span of a set of vectors in $\R^n$ is a subspace of $\R^n$.\\
\textbf{\textit{Proof:}}\\
Let $S=\{\tb v_1 ,\tb v_2 ,\cdots\tb v_m \}$. Let $\tb w ,\tb{y}\in \mathrm{span}(S)$.
Thus, $\tb w =\sum c_i\tb v_i$ and $\tb{y}=\sum d_i\tb v_i $ where $d_i,c_i\in \R$.

\begin{align*}
    a\tb w +b\tb{y}&=a\sum c_i\tb v_i+b\sum d_i\tb v_i\\
    &=\sum a c_{i} \tb v_{i}+\sum b d_{i} \tb v_i\\
    &=\sum (a c_i+b d_i) \tb v_i \in \operatorname{span}(S)
\end{align*}

\noindent
List of subspaces in $\R^2$ would be $\R^2$, $\{t\tb{v}\mid t\in\R\}$, $\{\tb{0}\}$.

\subsection{Intersection and Union}

If $V$ and $W$ are collections of vectors in $\R^n$:
\begin{itemize}
    \item $V \cap W=\{\tb{x} \mid \tb{x} \in V \text { and } \tb{x} \in W\}$ is the intersection
    \item $V \cup W=\{\tb{x} \mid \tb{x} \in V \text { or } \tb{x} \in W\}$ is the union
\end{itemize}

\subsection{Redundant Vectors}

If for some transformation $T$ there exists the following:

\[\operatorname{im}(T)=\operatorname{span}\left\{\left(\begin{array}{l}
    1 \\
    1 \\
    1
    \end{array}\right),\left(\begin{array}{l}
    2 \\
    2 \\
    2
    \end{array}\right),\left(\begin{array}{l}
    1 \\
    2 \\
    3
    \end{array}\right),\left(\begin{array}{l}
    2 \\
    3 \\
    4
    \end{array}\right)\right\}\]

There are redundant vectors in this case. The minimum number of vectors in the span is 2,
for $\tb{0}$ cannot be produced then. With 3 vectors in $\R^3$, any one can be the result of linear combinations of the other 2.
So, it would be appropriate to say that:

\[\operatorname{im}(T)=\left\{\left(\begin{array}{l}
    1 \\
    1 \\
    1
    \end{array}\right),\left(\begin{array}{l}
    1 \\
    2 \\
    3
    \end{array}\right)\right\}\]

These are then \textbf{linearly independent}. This set forms a \textbf{basis} for that set of vectors.
Thus, the basis can be found for any matrix. The basis of $I_n$ is then $\{\tb{e}_1,\tb{e}_2\cdots \tb{e}_n\}$.

\subsection{Intersection and Union}

\begin{framed}
    \noindent If $V$ and $W$ are subspaces of $\R^n$, then $V\cap W$
    is a subspace of $\R^n$ and $V\cup W$ is \textbf{not} a subspace of $\R^n$.
\end{framed}

\noindent
An intersection is the items contained in both sets, so $\tb{0}\in V\cap W$.
If $\tb{v},\tb{w}\in V\cap W$, then $\tb{v},\tb{w}\in V$ and $\tb{v},\tb{w}\in W$. This means that
$\tb{v}+\tb{w}\in V$ and $\tb{v}+\tb{w}\in W$ so $\tb{v}+\tb{w}\in V\cap W$.
Similarly, if some $k\tb{v}\in V\cap W$ where $k\in \R$ then $k\tb{v}\in V$ and $k\tb{v}\in V$.
Thus, $V\cap W$ is a subspace of $\R^n$.\\

\noindent
The union is the items contained in either set.
If $V=\mathrm{span}(\tb{e}_2)$ and $W=\mathrm{span}(\tb{e}_1)$, then let $\tb{y}=\begin{bmatrix}0\\ y\end{bmatrix}\in V$
and $\tb{x}=\begin{bmatrix}x\\ 0\end{bmatrix}\in W$. Thus, $\tb{y},\tb{x}\in V\cup W$. However,
$a\tb{x}+b\tb{y}\notin V\cup W$ where $a,b\in \R$.


%%%%%%%%%%%%%%%%%%%%%%%%%
% Expectation Integrals %
%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Expectation Integrals}

\begin{proposition}{(Unknown)}{}

    Let $A,B \subseteq \Omega$. Then the following equalities hold:

        \begin{itemize}
            \setlength{\parskip}{0em}
            \item $\Indicator{A^C} = 1 - \Indicator{A}$,
            \item $\Indicator{A \cap B} = \Indicator{A}\Indicator{B}$.
            \item $\Indicator{A \cup B} = \Indicator{A} + \Indicator{B} - \Indicator{A \cap B}$.
        \end{itemize}

\end{proposition}

\begin{lemma}{3.3}{}

    Let $X$ be a \emph{non-negative} random variable. Then there exists a sequence of \emph{non-negative, simple} random variables $X_n$ converging to $X$ for every $\omega \in \Omega$.

    \Hint $h_n(x)=\min\{\floor{2^nx}/2^n, n\}$ is non-negative, simple and increasing, approaching $x$. Consider $X_n \coloneqq h(X) \to X$.

\end{lemma}

\begin{lemma}{}{Simple Function Integral Properties}

    Let $f,g: \Omega \to \overline{\mathbb{R}}$ be a \emph{non-negative}, simple functions and $a,b \geq 0$. Then the following holds:

    \begin{itemize}
        \setlength{\parskip}{0em}
        \item $\int_{\Omega} f \, d\mu \geq 0$,
        \item $\int_{\Omega} (af + bg) \, d\mu = a\int_{\Omega} f + b\int_{\Omega}g \, d\mu$.
    \end{itemize}

\end{lemma}

\begin{corollary}{}{Positive Integral over Set}

    Let $A \subseteq \Omega$ and $f: \Omega \to \overline{\mathbb{R}}$ a \emph{non-negative} measurable function. Then $\int_A f \, d\mu \geq 0$.

\end{corollary}

\begin{lemma}{3.3}{General}

    Let $f: \Omega \to \overline{\mathbb{R}}$ be a \emph{non-negative}, measurable function. The there exists a sequence $f_n$ of \emph{non-negative}, simple functions such that:

    \begin{align*}
        \lim_{n \to \infty} f_n = f
    \end{align*}

    \Hint Use $h_n$ from Lemma 3.3's hint.

\end{lemma}

\begin{exercise}{3.5}{}

    Let $A \in \CalF$ s.t. $\mu(A) = 0$. Then for \emph{any} measurable function $f: \Omega \to \overline{\mathbb{R}}$:

        \begin{align*}
            \int_A f \,d\mu = 0.
        \end{align*}

\end{exercise}

\begin{exercise}{3.6}{}

    Let $f: \Omega \to \mathbb{R}$ be a measurable function, then:

        \begin{enumerate}[(i)]
            \setlength{\parskip}{0em}
            \item For any $c \in \mathbb{R}$ and $A \in \CalF$:

                \begin{align*}
                    \int_A cf \, d\mu = c \int_A f \, d\mu,
                \end{align*}

            provided the integral exists.
            \item For any $A, B \in \CalF$, such that $A \cap B = \emptyset$:

                \begin{align*}
                    \int_{A \cup B} f \,d\mu = \int_{A} f \,d\mu + \int_{B} f \,d\mu,
                \end{align*}

            provided the left-hand or right-hand side is well-defined.
        \end{enumerate}

\end{exercise}

\begin{theorem}{3.8}{Monotone Convergence}

    Let $(f_n)_{n=1}^{\infty}$ be increasing sequence of non-negative, measurable functions $f_n: \Omega \to \overline{\mathbb{R}}$, converging to some $f$. Then:

        \begin{align*}
            \int_{\Omega} \lim_{n \to \infty} f_n \,d\mu = \lim_{n \to \infty} \int_{\Omega} f_n \,d\mu
        \end{align*}

\end{theorem}

\begin{theorem}{3.14}{Lebesgue Integral as Riemann Integral}

    Let $f: \mathbb{R} \to \mathbb{R}$ be a Borel-function such that:

        \begin{enumerate}[(i)]
            \setlength{\parskip}{0em}
            \item the Riemann integral $\int_{-\infty}^{\infty} f(x) \, dx$ exists and
            \item the Riemann integral $\int_{-\infty}^{\infty} |f(x)| \, dx < \infty$, i.e. is finite,
        \end{enumerate}

    then the Lebesgue integral $\int_{\mathbb{R}} f(x) \lambda(dx)$ \emph{exists} and

        \begin{align*}
            \int_{\mathbb{R}} f(x) \lambda(dx) = \int_{-\infty}^{\infty} f(x) \, dx,
        \end{align*}

    i.e. the Lebesgue integral is equal to the Riemann integral.

\end{theorem}

\begin{exercise}{3.15}{}

    Let $\nu$ be a measure that is absolutely continuous with respect to measure $\mu$ and density $g$, then $\mu(g < 0) = 0$. Moreover, $\nu$ is a probability measure $\Leftrightarrow$ $g \geq 0$ $\mu$-a.e. and $\int_{\Omega} g \, d\mu = 1$.

\end{exercise}

\begin{proposition}{3.16}{}

    Let $\nu$ and $\mu$ be measures on \SigmaAlgebra\ $\CalF$ such that $\nu$ is absolutely continuous with respect to $\mu$ and density $g$. Then for every $\CalF$-measurable function $f$ the following holds:

        \begin{align*}
            \int_{\Omega} f \, d\nu = \int_{\Omega} fg \, d\mu,
        \end{align*}

    whenever one of the integrals exists.

\end{proposition}

\begin{remark}{3.3}{}

    Let $(\Omega, \mathcal{F}, \mu)$ be measure space, $f: \Omega \to \overline{\mathbb{R}}$ \emph{non-negative} $\mathcal{F}$-measurable, then

        \begin{align*}
            \mu(f \geq \lambda) \leq \lambda^{-\alpha} \int_{\Omega} f^{\alpha} \,d\mu \quad \forall \lambda > 0, \alpha > 0.
        \end{align*}

\end{remark}

\begin{lemma}{3.10}{Fatou's Lemma}

    Let $(f_n)_{n=1}^{\infty}$ be a sequence of \emph{non-negative}, measurable functions $f: \Omega \to \overline{\mathbb{R}}$, then

        \begin{align*}
            \int_{\Omega} \liminf_{n \to \infty} f_n \, d\mu \leq \liminf_{n \to \infty} \int_{\Omega} f_n \, d\mu.
        \end{align*}

\end{lemma}

\begin{corollary}{3.11}{Fatou's Lemma Extension}

    Let $(f_n)_{n=1}^{\infty}$ be a sequence of measurable functions $f: \Omega \to \overline{\mathbb{R}}$. Then

        \begin{enumerate}[(i)]
            \setlength{\parskip}{0em}
            \item if there exists a $g \in L_1(\Omega, \CalF, \mu)$, i.e. $\int_{\Omega} |g| \, d\mu < \infty$ such that $g \leq f_n$ for all $n$, then:

            \begin{align*}
                \int_{\Omega} \liminf_{n \to \infty} f_n \, d\mu \leq \liminf_{n \to \infty} \int_{\Omega} f_n \, d\mu.
            \end{align*}
            \item if there exists a $g \in L_1(\Omega, \CalF, \mu)$, i.e. $\int_{\Omega} |g| \, d\mu < \infty$ such that $g \geq f_n$, then:

            \begin{align*}
                \int_{\Omega} \limsup_{n \to \infty} f_n \, d\mu \geq \limsup_{n \to \infty} \int_{\Omega} f_n \, d\mu.
            \end{align*}
        \end{enumerate}

\end{corollary}

\begin{theorem}{3.12}{Lebegue's Theorem on Dominated Convergence}

    Let $(f_n)_{n=1}^{\infty}$ be a sequence of Borel functions $f_n: \Omega \to \overline{\mathbb{R}}$ converging to some $f: \Omega \to \overline{\mathbb{R}}$. Assume there exists a (non-negative) Borel functions $g$ such that $|f_n| \leq g$ for any $n \geq 1$ and $\int_{\Omega} g \, d\mu < \infty$. Then the following two statements hold:

        \begin{enumerate}[(i)]
            \setlength{\parskip}{0em}
            \item
                \begin{align*}
                    \int_{\Omega} |f| \, d\mu < \infty,
                \end{align*}
            \item
                \begin{align*}
                    \int_{\Omega} f \, d\mu = \lim_{n \to \infty} \int_{\Omega} f \, d\mu.
                \end{align*}
        \end{enumerate}

\end{theorem}

\begin{proposition}{}{Restricted Expectation}

    Let $X$ be a random variable and $A \in \CalF$, then:

        \begin{align*}
            E(X \Indicator{A}) = \int_A X \,dP.
        \end{align*}

\end{proposition}

\begin{theorem}{3.17}{Integration Over The Sample Space}

    Let $f: \mathbb{R} \to \mathbb{R}$ be a Borel function and $X$ a \emph{finite} random variable, then:

        \begin{align*}
            Ef(X) = \int_{\mathbb{R}} f Q_X(dx).
        \end{align*}

\end{theorem}

\begin{proposition}{3.18}{Markov-Chebyshev's Inequality}

    Let $X$ be a \emph{non-negative} R.V., then

        \begin{align*}
            P(X \geq \lambda) \leq \lambda^{-\alpha} E(X^{\alpha}) \quad \forall \lambda > 0, \alpha > 0.
        \end{align*}

    \Hint $E(X^{\alpha}) \geq E(\Indicator{X \geq \lambda} X^{\alpha}) \geq E(\Indicator{X \geq \lambda} \lambda^{\alpha}) = \lambda^{\alpha} E(\Indicator{X \geq \lambda}) = \lambda^{\alpha} P(X \geq \lambda)$.

\end{proposition}

\begin{proposition}{3.18}{Markov-Chebyshev's Inequality (General)}

    Let $f: \Omega \to \overline{\mathbb{R}}$ be a \emph{non-negative}, measurable function, then

        \begin{align*}
            \mu(f \geq \lambda) \leq \lambda^{-\alpha} \int_{\Omega} f^{\alpha} \, d\mu \quad \forall \lambda > 0, \alpha > 0.
        \end{align*}

\end{proposition}


\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%        VECTOR ANALYSIS APPENDIX       %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Vector Analysis}
\index{Vector Analysis}
\label{app:vec-analysis}
Here we review various techniques within vector calculus and analysis.

% Vector Algebra
\section{Vector Algebra}
Vectors, by definition, are members of a vector space. It is therefore prudent to summarize the properties of vector fields. A vector space over a field $K$ is a set $V$ equipped with vector addition $+: V\times V \to V$ and scalar multiplication $*: K\times V \to V$, where $+$ is an additive group and $\cdot$ is distributive over $+$ and the underlying field addition $+_K$. There are two additional structures of note. First, a bilinear form $(\cdot,\cdot): V\times V\to K$ that, when positive definite and nondegenerate, forms the familiar inner product $v\cdot w \to k\in K$. Second, a bilinear operator $\times : V\times V \to V$, called the \textit{cross product}, which gives the vector space an algebra structure.

\subsection{Inner Product}
The vector inner product $\cdot$ takes two vectors and produces a scalar. It can be used to measure the angle between two vectors.
\eqn{\vec{A}\cdot\vec{B}=AB\cos\theta=A_x B_x + A_y B_y + A_z B_z}
The inner product is also used to compute magnitudes or the \textit{norm} of a vector.
\eqn{\norm{\vec{A}}=\sqrt{\vec{A}\cdot\vec{A}}=\sqrt{A_x^2 + A_y^2 + A_z^2}}

\subsection{Outer (Cross) Product}
The cross product takes two vectors and produces a vector whose direction corresponds to the right-hand rule.
\eqn{\vec{A}\times\vec{B} =
	\wrap{A_y B_z - A_z B_y}\vec{\hat{x}}
	+ \wrap{A_z B_x - A_x B_z}\vec{\hat{y}}
	+ \wrap{A_z B_x - A_x B_z}\vec{\hat{z}}}
\eqn{\vec{A}\times\vec{B} =
	\begin{vmatrix}
		\hat{i} & \hat{j} & \hat{k}\\
		A_x & A_y & A_z \\
		B_x & B_y & B_z
	\end{vmatrix}}

\subsection{Triple Product}
The scalar triple product represents the volume of the parallelepiped generated by $\vec{A}$, $\vec{B}$, and $\vec{C}$.
\eqn{\vec{A}\cdot\wrap{\vec{B}\times\vec{C}} = \begin{vmatrix}
	A_x & A_y & A_z \\
	B_x & B_y & B_z \\
	C_x & C_y & C_z
\end{vmatrix}}
The vector triple product can be simplified
\eqn{\vec{A} \times \wrap{\vec{B} \times \vec{C}} = \vec{B}\wrap{\vec{A}\cdot\vec{C}} - \vec{C}\wrap{\vec{A}\cdot\vec{B}}}

% Differential Calculus
\section{Differential Calculus}
\subsection{Gradient}
The gradient of a function $f$, denoted $\vec{\del}f$, is a vector that represents the direction in which $f$ changes the greatest.
\eqn{\vec{\del} = \pd{}{x}\vec{\hat{x}} + \pd{}{y}\vec{\hat{y}} + \pd{}{z}\vec{\hat{z}}}

\subsection{Divergence}
The divergence of a vector field measures how much the vector spreads out (or diverges) from the point in question.
\eqn{\vec{\del}\cdot \vec{v} = \pd{v_x}{x} + \pd{v_y}{y} + \pd{v_z}{z}}

\subsection{Curl}
The curl of a vector field measures how much the vector swirls around the point in question.
\eqn{\vec{\del}\times\vec{v} = \begin{vmatrix}
	\hat{i} & \hat{j} & \hat{k} \\
	\p/\p x & \p / \p y & \p/\p z \\
	v_x & v_y & v_z
\end{vmatrix}}

\subsection{Second Derivatives}
The Laplacian $\del^2$ represents a second-order change in a vector. There are also a few useful identities related to closed and exact differential forms.
\eqn{\del^2\vec{v} = \pd[2]{v_x}{x} + \pd[2]{v_y}{y} + \pd[2]{v_z}{z}}
\eqn{\vec{\del} \times \wrap{\vec{\del}f}=0}
\eqn{\vec{\del}\cdot\wrap{\vec{\del}\times \vec{v}}=0}

% Integral Calculus
\section{Integral Calculus}
\subsection{Fundamental Theorem of Calculus}
\eqn{\int_a^b \wrap{\frac{df}{dx}}dx=f(b) - f(a)}

\subsection{Stokes-Cartan Theorem}
The Stokes-Cartan theorem related the integral of a form over a the boundary of a manifold to the integral of the exterior derivative of the form over the whole manifold. The Divergence theorem and Green's theorem are both instances of the more general stokes theorem.
\eqn{\int_{\p\Omega}\omega=\int_\Omega d\omega}
\eqn{\oint_{\Gamma} \vec{v} \cdot d \vec{\Gamma} = \iint_S\vec{v}\cdot d\vec{S} \where \Gamma = \p S}
\eqn{\oiint_S \vec{v}\cdot d\vec{S}  = \iiint_V \wrap{\vec{\del}\cdot v} dV \where S = \p V}

\subsection{Integration by Parts}
Another useful identity is integration by parts, which can help solve integrals that produce iterative expressions, like integrals of the form $\int x^p e^{-x} dx$.
\eqn{\int_a^b f\wrap{\frac{dg}{dx}}dx = fg \Big\rvert_a^b - \int_a^b g \wrap{\frac{df}{dx}}dx}

% Curvilinear Coordinates
\section{Curvilinear Coordinates}
\subsection{Spherical Coordinates}
We adopt the mathematical convention for spherical coordinates $(r, \theta, \phi)$ whereby $\theta$ is the \textit{azimuthal} angle in the $x$-$y$ plane and $\phi$ is the \textit{polar} angle between the radial vector and the $z$-axis. This has the enormous advantage of sharing $\theta$ with polar coordinates in the $x$-$y$ plane. This can be problematic because Griffiths uses the opposite convention \cite{griffithsIntroductionElectrodynamics2018}.
\eqn{x = r\sin\phi\cos\theta \quad\quad y = r\sin\phi\sin\theta \quad\quad z = r \cos\phi}
\begin{align}
	\vec{\hat{r}} &= \sin\phi\cos\theta\ \vec{\hat{x}} + \sin\phi\sin\theta\ \vec{\hat{y}} + \cos\phi\ \vec{\hat{z}} \\
	\vec{\hat{\theta}} &= \cos\phi\cos\theta\ \vec{\hat{x}} + \cos\phi\sin\theta\ \vec{\hat{y}} - \sin\theta\ \vec{\hat{z}} \\
	\vec{\hat{\phi}} &= - \sin\theta\ \vec{\hat{x}} + \cos\theta\ \vec{\hat{y}}
\end{align}
The Gradient and Laplacian can also be written in spherical coordinates, which are often useful:
\eqn{\del = \pd{}{r}\vec{\hat{r}} + \frac{1}{r \sin \phi}\pd{}{\theta} \vec{\hat{\theta}} + \frac{1}{r}\pd{}{\phi}\vec{\hat{\phi}}}
\eqn{\del^2 = \frac{1}{r^2}\pd{}{r}\wrap{r^2 \pd{}{r}} + \frac{1}{r^2 \sin^2\phi}\pd{{}^2}{\theta^2} + \frac{1}{r \sin \phi}\pd{}{\phi}\wrap{\sin\phi \pd{}{\phi}}}

% Dirac Delta
\section{Dirac Delta}
The \textit{Dirac delta} is the first example of a \textit{functional}, or \textit{distribution}, encountered in a traditional physics undergraduate sequence, and is central to the entire theory of electrodynamics. A common example is given by the divergence of $\vec{v} = \frac{1}{r^2} \vec{\hat{r}}$. A naive computation yields $\del \cdot \vec{v} = 0$, which seems counterintuitive. However, $\vec{v}$ attains an infinite value at $0$, and any integral over a spherical region centered at $0$, we obtain a constant $4\pi$. As we will see, this is a Dirac delta function.
% One-Dimensional Dirac Delta
\subsection{One-Dimensional Dirac Delta}
The one-dimensional Dirac delta function is a function that is infinite at one point, and zero elsewhere, that is integrable to a constant value:
\eqn{\delta(x) =
	\begin{cases}
		0, & x \neq 0 \\
		\infty, & x = 0
	\end{cases}
\quad\quad \text{such that}\quad\quad \int_{-\infty}^{\infty}\delta(x)\ dx = 1}
By far the most important fact about the delta function, is that, when paired with a general function $f(x)$, the delta function "selects" the value of $f$ at $x=0$, such that $f(x)\delta(x) = f(0)\delta(x)$. This is useful for simplifying integrals.
\eqn{\int_{-\infty}^{\infty}f(x)\delta(x)dx = f(0)\int_{-\infty}^{\infty}\delta(x)dx = f(0) \quad \implies \quad \int_{-\infty}^{\infty}f(x)\delta(x - a)dx = f(a)}
% Three-Dimensional Dirac Delta
\subsection{Three-Dimensional Dirac Delta}
We can generalize the delta function to three-dimensional space: $\delta^3(\vec{r})=\delta(x)\delta(y)\delta(z)$. Integrating this function over all space exhibits the same "selection" behavior as in the one-dimensional case:
\eqn{\int_{\infty^3}f(\vec{r})\delta^3(\vec{r} - \vec{a})dx^3 = f(\vec{a})}
We can now restate the divergence of the function $f(\vec{r})=\frac{1}{r^2}\vec{\hat{r}}$ in terms of the delta function.
\eqn{\boxed{\vec{\del} \cdot \wrap{\frac{\vec{\hat{r}}}{r^2}} = 4\pi \delta^3(\vec{r})}}
This can be used to simplify many integrals appearing in electrodynamics and other three-dimensional cases, such as spherical harmonics. For example:
\eqn{\int_{\mathcal{V}}\wrap{r^2 + 2} \vec{\del}\cdot\wrap{\frac{\vec{\hat{r}}}{r^2}} dV = \int_{\mathcal{V}}\wrap{r^2 + 2} 4\pi \delta^3(\vec{r}) dV = 4\pi \wrap{0 + 2} = 8\pi}

% Vector Fields
\newpage
\section{Vector Fields}
\label{app:vector-analysis:potentials}
\subsection{Potentials (Closed and Exact Forms)}
Given a differential form $a\in \Omega^p$, it is said to be \textit{closed} if it's exterior derivative is zero, $da=0$, and is said to be \textit{exact} if it is the exterior derivative of another form $b \in \Omega^{p-1}$, $a = db$. In the latter case, $b$ would be called the \textit{potential function} of $a$. Since $d^2=0$, all exact forms are closed. Whether or not closed forms are exact depends on the topology, which is out of scope here.

In the case of a vector field $\vec{F}$ on $\RThree$, the property of having no curl $\vec{\del} \times \vec{F} = 0$ is equivalent to saying $\vec{F}$ is closed. Consequently, there must exist a scalar potential $V$ such that $\vec{F} = -\del V$. To state it more formally:
\eqn{\vec{\del}\times\vec{F} = 0 \implies \vec{F} = -\del V}

\subsection{The Helmholtz Theorem}
The Helmholtz Theorem states that a vector field $\vec{F}$ is uniquely specified by its divergence $\vec{\del}\cdot \vec{F}$ and its curl $\vec{\del}\times \vec{F}$, assuming the boundary condition $\lim_{r\to \infty} \vec{F}(\vec{r})=0$ is strong enough. If this is the case, then $F$ can be constructed from the divergence and curl:
\eqn{\vec{F} = -\del U + \vec{\del}\times \vec{W} \quad\quad \text{where} \quad\quad
\begin{aligned}
	U(\vec{r}) &= \frac{1}{4\pi}\int \frac{\vec{\del}\cdot \vec{F}}{\norm{\vec{r} - \vec{r}'}} dr'^3 \\
	\ & \ \\
	\vec{W}(\vec{r}) &= \frac{1}{4\pi} \int\frac{\vec{\del}\times \vec{F}}{\norm{\vec{r} - \vec{r}'}}dr'^3 \\
\end{aligned}}

% Substitutions
\section{Substitutions}
%Various integrals can be simplified via substitution techniques. This section details two such substitution methods, both of which have been useful in undergraduate physics.
\subsection{Trigonometric Substitution}
Substituting trigonometric functions for more complicated expressions can simplify an integral. The below cases highlight the most common trig substitutions:
\eqn{\begin{array}{ccc}
	\textbf{Case} & \textbf{Integrand Contains} & \textbf{Substitution} \\
	\text{I}   & a^2 - x^2 & x = a\sin\theta \\
	\text{II}  & a^2 + x^2 & x = a\tan\theta \\
	\text{III} & x^2 - a^2 & x = a\sec\theta \\
\end{array}}

\subsection{Weierstrass Substitution}
The Weierstrass substitution transforms a function that depends on basic trigonometric functions of an angle into a function that depends on algebraic variables. Specifically, the substitution is:
\eqn{t = \tan\wrap{\frac{\theta}{2}} \implies \left\{
\begin{aligned}
	\sin(\theta) &= \frac{2t}{1 + t^2} \\
	\cos(\theta) &= \frac{1 - t^2}{1 + t^2} \\
\end{aligned}\right.}
\eqn{f\wrap{\sin \theta, \cos \theta} \to f\wrap{\frac{2t}{1 + t^2}, \frac{1 - t^2}{1 + t^2}}}


\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{xfrac}
\usepackage{todonotes}

\title{Neural Network Implementation Draft}
\date{October 2019}

\begin{document}
\maketitle

\section{Matrix Calculus}

\begin{table}
    \centering
    \begin{tabular}{cc}
        \toprule
            $y$ & $\frac{\partial}{\partial x} y$ \\
        \midrule
            $Ax$     & $A^T$ \\
            $x^T A$  & $A$   \\
            $x^T x$  & $2x$  \\
            $x^T Ax$ & $Ax + A^Tx$  \\
        \bottomrule
    \end{tabular}
    \caption{Useful derivatives equations}
\end{table}

\subsection{Chain Rule for Matrix Calculus}

The chain rule for a vectors is similar to the chain rule for scalars. Except the order is important. For $\mathbf{z} = f(\mathbf{y})$ and $\mathbf{y} = g(\mathbf{x}) $ the chain rule is:
\begin{equation}
    \frac{\partial \mathbf{z}}{\partial \mathbf{x}} = \frac{\partial \mathbf{y}}{\partial \mathbf{x}}     \frac{\partial \mathbf{z}}{\partial \mathbf{y}}
\end{equation}

\section{Example: 3 Layer Fully Connected Neural Network}

For the input $x$ the neural network which is described by its weights $W$, its biases $b$ and the activation functions $g(t)$. The network has $L_1$ neurons in the first layer, $L_2$ neurons in the second layer and $L_3$ neurons in the final layer.

\begin{table}
    \centering
    \begin{tabular}{lcc}
        \toprule
            Layer & Weights & Bias \\
        \midrule
            1     & [L1 nx] & [L1 1] \\
            2     & [L2 L1] & [L2 1] \\
            3     & [ny L2] & [ny 1] \\
        \bottomrule
    \end{tabular}
    \caption{Dimensions of the weight and bias matrices}
\end{table}

\begin{align*}
    z_1 &= W_1 x + b_1 \\
    a_1 &= f(z_1)      \\
    z_2 &= W_2 a_1 + b_2 \\
    a_2 &= f(z_2)      \\
    z_3 &= W_3 a_2 + b_3 \\
    h   &= z_3         \\
\end{align*}

\begin{equation}
    J = \frac{1}{N} \sum_i^N (h(x_i;W,b)-y_i)^2
\end{equation}

\subsection{Backpropagation}

The update rule for gradient descent is
\begin{equation}
    p_{i+1} = p_i + \mu \frac{\partial J}{\partial p_i}  \quad \forall p \in \{ W, b \}
\end{equation}

The main difficulty here is to calculate the gradient for each parameter in the network, which can easily be several thousands or even million parameters. Here back-propagation is used to efficiently calculate those derivatives. The first step is to differentiate the cost function with respect to an parameter $p$ which can describe an weight or a bias

\begin{equation}
    \frac{\partial J}{\partial p_i}  = \frac{2}{N} \sum_i^N (h(x_i;W,b)-y_i) \frac{\partial h}{\partial p_i}
\end{equation}

The total list of needed derivatives are: \todo{The order is not correct yet}

\begin{align*}
    \frac{\partial h}{\partial W_3} &=  \frac{\partial h}{\partial z_3} \frac{\partial z_3}{\partial W_3} = a_2 \\
    \frac{\partial h}{\partial W_2} &= \frac{\partial h}{\partial a_2}     \frac{\partial a_2}{\partial z_2}     \frac{\partial z_2}{\partial W_2} = W_3^T f'(z_2) a_2 \\
    \frac{\partial h}{\partial W_1} &= \frac{\partial h}{\partial a_2} \frac{\partial a_2}{\partial z_2} \frac{\partial z_2}{\partial a_1} \frac{\partial a_1}{\partial z_1} \frac{\partial z_1}{\partial W_1} = W_2^T W_3^T f'(z_2) f'(z_1)   x  \\
    %
    \frac{\partial h}{\partial b_3} &=  \frac{\partial h}{\partial z_3} \frac{\partial z_3}{\partial b_3}  = 1 \\
    \frac{\partial h}{\partial b_2} &= \frac{\partial h}{\partial a_2}     \frac{\partial a_2}{\partial z_2}     \frac{\partial z_2}{\partial b_2} = W_3 f'(z_2)   \\
    \frac{\partial h}{\partial b_1} &= \frac{\partial h}{\partial a_2} \frac{\partial a_2}{\partial z_2} \frac{\partial z_2}{\partial a_1} \frac{\partial a_1}{\partial z_1} \frac{\partial a_1}{\partial b_1} =  W_2^T W_3^T  f'(z_2) f'(z_1)  \\
\end{align*}

Note: Here the equations are used as scalars. Because those are vectors all equations need to be transposed.

Calculations of the delta terms is as follows
\begin{equation}
    \delta^{(4)} = h - y
\end{equation}
then the next delta value is computed using the update equation
\begin{equation}
    \delta^{(l-1)} = W_i^T \delta^{(l)} .* f'(z)
\end{equation}
and then the total update term is calculated:
\begin{equation}
    \Delta^{(l)} = \delta^{(l)} a^{(l)}
\end{equation}

It can be seen that the same derivatives are used more often

\begin{table}
    \centering
    \begin{tabular}{lc}
        \toprule
            Derivative & Result  \\
        \midrule
            $\sfrac{\partial z_1}{\partial W_1}$     & $ x $ \\
            $\sfrac{\partial z_1}{\partial b_1}$     & $ 1 $ \\
            $\sfrac{\partial a_1}{\partial z_1}$     & $ f'(z_1) $ \\
        \midrule
            $\sfrac{\partial z_2}{\partial a_1}$     & $ W_2 $ \\
            $\sfrac{\partial z_2}{\partial W_2}$     & $ a_1 $ \\
            $\sfrac{\partial z_2}{\partial b_2}$     & $ 1 $ \\
            $\sfrac{\partial a_2}{\partial z_2}$     & $ f'(z_2) $ \\
        \midrule
            $\sfrac{\partial z_3}{\partial a_2}$     & $ W_3 $ \\
            $\sfrac{\partial z_3}{\partial W_3}$     & $ a_2 $ \\
            $\sfrac{\partial z_3}{\partial b_3}$     & $ 1 $ \\
        \bottomrule
    \end{tabular}
    \caption{Calculations of all derivatives of the network}
\end{table}

Here something to note here: $f'(z_i)$ is the derivative of the activation function with respect to $z_i$. As an example, the equation of the ReLU activation function is:
\begin{equation}
    f(t) = \begin{cases}
        t   \quad \text{if} \quad t > 0 \\
        0   \quad \text{else}
    \end{cases}
\end{equation}
The derivative $f'(t)$ is
\begin{equation}
    f'(t) = \begin{cases}
        1   \quad \text{if} \quad t > 0 \\
        0   \quad \text{else}
    \end{cases}
\end{equation}

\section{Example: Convolutional Neural Network}

The two dimensional convolution is defined as:

\begin{equation}
   z(i,j) = (f*g)(i,j) = \sum_{m=-\infty}^{\infty} \sum_{n=-\infty}^{\infty} f(m,n) g(m-i,n-j)
\end{equation}

Compared to the previous example the input data is now not one dimensional but two-dimensional.

Also because the problem tackled here is a classification one, the loss function used is the cross-entropy function:
\begin{equation}
    J = - y .* \log(h) + (1-y) .* \log(1-h)
\end{equation}
where $.*$ is used as the element-wise multiplication. This can be interpreted as:
\begin{equation}
    J = \begin{cases}
        - y_i * \log(h_i)       \quad &\text{if} \quad y_i = 1 \\
        (1 - y_i) * \log(h_i)   \quad &\text{if} \quad y_i = 0
    \end{cases}
\end{equation}

Because each image is exclusively in one class, the vector
\begin{equation}
y = \begin{bmatrix}
    y_1 & y_2 & \cdots & y_n
\end{bmatrix}^T
\end{equation}
consists of all zeros except for a single $1$.

\end{document}