\section{Markov chain Monte Carlo (MCMC)}

In this thesis, Markov chains were used to train the presented \acl{sorn}. Before the specific methods are addressed, some theory about Markov chains is introduced.

%%%%% Markov chains %%%%%
\subsection{Markov chains}

\emph{Markov chains} contain a number of states which are changing with a specific probability. The \emph{markov property} states that the probability of the current state only depends on the immediate previous state. Therefore, the Markov property in the discrete case can be defined as follows:

\begin{definition}[Markov property]
Let $(\Omega, \mathcal{F}, \Pb)$ be a probability space and $(S, \mathcal{S})$ a measurable space, where $S = \{x_1, x_2, ... \}$ is a countable set, called state space. A stochastic process $X = \{ X_t : \Omega \to S\}_{t\in\N}$ is said to possess the Markov property if and only if for each $x_0, ..., x_{t+1}\in S$ and $t\in\N$

\begin{equation}
\label{eq:markov-chain}
\Pb(X_{t+1} = x_{t+1} | X_{t} = x_{t}, X_{t-1} = x_{t-1}, ..., X_0 = x_0) = \Pb(X_{t+1} = x_{t+1} | X_{t} = x_{t}).
\end{equation}
\end{definition}

The Markov property is also called \emph{memorylessness}, since it is sufficient to memorize the probability of the state before, no further information is necessary.

The probability of changing from one state to another is called \emph{transition probability}, which is defined as

\begin{equation}
\label{eq:trans-prob}
p_{ij}(t) = \Pb(X_{t+1} = x_j | X_{t} = x_i),
\end{equation}

where $x_i, x_j \in S$ and $i,j \in \N$. The transition probabilities can be summarized in a \emph{transition matrix}. A transition matrix for a finite Markov chain where $i,j \in \{1, ..., n\}$ with $n \in\N$ has the form

\begin{equation}
\label{eq:trans-matrix}
M(t) = (p_{ij}(t))_{i,j \in \{1,...,n\}} = \begin{bmatrix}
p_{11}(t) & p_{12}(t) & \hdots  & p_{1n}(t)\\
p_{21}(t) & \ddots &  & \vdots \\
\vdots &  & \ddots & \vdots \\
p_{n1}(t) & \hdots & \hdots & p_{nn}(t)
\end{bmatrix}.
\end{equation}

\nomenclature{$M$}{Transition matrix of a Markov chain}

% transition graph
% reference to graph from properties

%%%%% Properties %%%%%
\subsection{Properties of Markov chains}
\label{sec:markov-properties}

\begin{figure}
    \centering
    \begin{subfigure}{\textwidth}
    	\centering
        \includegraphics[width=0.7\textwidth]{sorn_markov/mc-irreducible}
        \vspace{5pt}
        \caption{In irreducible Markov chains every state is reachable in finite time, independent of the present state. The first chain on the left side fulfills this property, while the center and right chain do not. The center chain may start in $A$ or $B$, but if $C$ is reached, $A$ and $B$ are not reachable any more. In the right chain, states $A$ and $B$ are totally separated from $C$ and $D$.}
        \vspace{15pt}
        \label{fig:irreducible}
    \end{subfigure}
    \begin{subfigure}{\textwidth}
    	\centering
        \includegraphics[width=0.7\textwidth]{sorn_markov/mc-aperiodic}
        \vspace{5pt}
        \caption{A state is periodic, if it returns back to the same state periodically, if not, the state is aperiodic. In the chain on the left side every state is periodic, since every state returns after $4$ steps. Therefore, the whole chain is periodic. On the right side, only state $B$ returns periodically every second time, states $A$ and $C$ are not periodic.}
        \vspace{15pt}
        \label{fig:aperiodic}
    \end{subfigure}
    \begin{subfigure}{\textwidth}
    	\centering
        \includegraphics[width=0.7\textwidth]{sorn_markov/mc-recurrent}
        \vspace{5pt}
        \caption{If a state will be reached again almost surely, it is recurrent, otherwise transient. Two simple examples are given on top left. In the first small chain, both states $A$ and $B$ will be reached again all the time. In the second example $A$ directly goes to $B$ and will not be active any more. State $A$ is transient. On the top right, states $A$ and $B$ are transient. If $C$ is activated the first time, the chain will never reach $A$ and $B$ again. On the other hand, $C$ and $D$ are recurrent, both can be reached again all the time. The last example on the bottom shows a null recurrent case. Shown is a Markov chain with infinite states. Since the probability of going back to $A$ is decreasing with a sequence $1/n$ for $n \to\infty$, the time to go back to $A$ is $\E[T_A] = \infty$. Regarding the last chain, more details can be found in a proof in appendix in section \ref{sec:proof-null-recurrent}.}
        \label{fig:recurrent}
    \end{subfigure}
    \caption[Markov chain properties]{Three properties of Markov chains are shown: irreducibility, periodicity and recurrence. If no probability is written at an arrow, the probability is simply $p > 0$.}
    \label{fig:mc-properties}
\end{figure}

\paragraph{a) Homogeneous / Stationary Markov chain}

If the transition probabilities $p_{ij}(t)$ from state $x_i$ to state $x_j$ are independent from time $t$, i.e. $p_{ij}(t) = p_{ij}$ for all $t\in\N$, the Markov chain is called \emph{homogeneous} or stationary Markov chain. It means that

\begin{equation}
\label{eq:markov-homo}
p_{ij} = \Pb(X_{t+1} = x_j | X_t = x_i) =  \Pb(X_t = x_j | X_{t-1} = x_i).
\end{equation}

\paragraph{b) Irreducible Markov chain}

In \emph{irreducible} Markov chains, any state is reachable in finite time, independent of the present state. Formally a Markov chain is irreducible if there exists any $m < \infty$ with

\begin{equation}
\Pb(X_{t+m} = x_j | X_t = x_i) = p_{ij}^{(t+m)} > 0
\end{equation}

for all $i,j$. Examples for reducible and irreducible Markov chains are given in figure \ref{fig:irreducible}.

\paragraph{c) Aperiodic Markov chain}

If a state $x_i$ returns with a multiple of $k$ again to state $x_i$, it is called \emph{periodic}, otherwise \emph{aperiodic}. Formally a \emph{period} is defined as

\begin{equation}
k = \text{gcd} \{ t > 0\,:\,\Pb(X_t = x_i | X_0 = x_i) = p_{ii}^{(t)} > 0 \},
\end{equation}

where $\text{gcd}$ is the greatest common divisor. If the set is empty, the period is not defined. In case that $k=1$ the state $x_i$ is aperiodic and otherwise, if $k>1$, $x_i$ is periodic.

If all states are aperiodic, the Markov chain is called aperiodic. Examples for periodicity and aperiodicity are given in figure \ref{fig:aperiodic}.

\paragraph{d) Recurrent Markov chain}

A state $x_i$ of a Markov chain is called \emph{recurrent} if it  will be reached again almost surely. Therefore, the probability to come back to state $x_i$ is one. Let

\begin{equation}
\label{eq:first-comeback}
T_{x_i} = \inf\{ t \ge 0 \,:\, X_t = x_i | X_0 = x_i \}
\end{equation}

\nomenclature{$T_{x_i}$}{Number of steps until a Markov state is reached again for the first time}

be a random variable, which defines the number of steps until state $x_i$ is reached again for the first time and let $\Pb(T_{x_i} = t)$ the probability to reach $x_i$ again for the first time after exactly $t$ steps. Then state $x_i$ is recurrent, if and only if

\begin{equation}
\label{eq:recurrent}
\Pb(T_{x_i} < \infty) = \sum_{t=1}^\infty \Pb(T_{x_i} = t) = 1,
\end{equation}

If a state is not recurrent, it is called \emph{transient}, which is the case if $\Pb(T_{x_i} < \infty) < 1$. It means that the state will not be reached again almost surely.

Furthermore, if the state is recurrent and the expected value is finite, thus

\begin{equation}
\label{eq:first-comeback-expect}
\E[T_{x_i}] = \sum_{t=1}^\infty t \cdot \Pb(T_{x_i} = t) < \infty,
\end{equation}

the state is called \emph{positive recurrent}. Otherwise, if $\E[T_{x_i}] = \infty$, it is called \emph{null recurrent}.

If all states are (positive) recurrent, the Markov chain is called (positive) recurrent. Examples for recurrent and transient Markov chains are given in figure \ref{fig:recurrent}. A proof for the null recurrent chain in the figure can be found in the appendix in section \ref{sec:proof-null-recurrent}.

%%%%% Stationary %%%%%
\subsection{Stationary distribution}
\label{sec:stat-markov}

%* Definition of stationary distribution\\
%* Theorems about stationary distribution

\begin{definition}[Stationary distribution]

Given a homogeneous Markov chain in discrete time $t \in \N$ on state space $S = \{x_1, x_2, ...\}$, the distribution $\Pb_\pi$ is called stationary distribution if and only if

\begin{equation}
\label{eq:markov-stat}
\bm\pi_{x_j} = \Pb_\pi(X_t = x_j) = \sum_{x_i \in S} \Pb_\pi(X_t = x_i)\,p_{ij} \overset{\eqref{eq:markov-homo}}{=} \sum_{x_i \in S} \Pb_\pi(X_t = x_i)\,\Pb(X_t = x_j | X_{t-1} = x_i)
\end{equation}

for all $x_j \in S$.
\end{definition}

\nomenclature{$\bm\pi$}{Stationary distribution of a Markov chain}

The stationary distribution should not be confused with the stationary Markov chain. While the stationary Markov chain is stationary regarding the transition probabilities, the stationary distribution is stationary regarding the probability of the states. Furthermore, the presence of a stationary Markov chain (homogeneous Markov chain) is a condition for the existence of a stationary distribution.

It is not always given that a stationary distribution exists. Furthermore a stationary distribution is not necessarily unique. In the following, first, the general Perron-Frobenius-Theorem \parencite{seneta2006non, pillai2005perron} is presented. Based on this theorem, a statement about the stationary distribution can be derived. While this theorem is highly useful, it assumes aperiodic Markov chains, which are not given in any case. Hence, afterwards another theorem is introduced, which lowers the assumption. At that point I want to remind, that a \emph{homogeneous} Markov chain is assumed in the following.

\begin{definition}[Primitive matrix]
Given a squared matrix $Q = (q_{ij})_{i,j \in\N}$, it is called a primitive matrix if all items are non-negative, $q_{ij} > 0 \,\forall i,j$, denoted by $Q > 0$ and if a $k \in \N$ exists, such that all items of the matrix $Q^k$ ($k$-th power of $Q$) are positive, denoted by $Q^k > 0$.
\end{definition}

\begin{theorem}[Perron-Frobenius]
\label{eq:perron-frobenius}
Suppose $Q$ is an $n \times n$ non-negative primitive matrix. Then there exists an eigenvalue $r$, called Perron-Frobenius eigenvalue, such that:

\begin{enumerate}
\item $r$ is real and positive, $r>0$.
\item $r$ can be associated with strictly positive left and right eigenvectors.
\item $r > |\lambda|$ for any other eigenvalue $\lambda \neq r$.
\item The eigenvectors associated with $r$ are unique to constant multiples.
\item If $0 \le B \le Q$ and $\beta$ is eigenvalue of $B$, then $|\beta| \le r$. Moreover, $|\beta| = r$ implies $B = Q$.
\item $r$ is a simple root of the characteristic equation of $Q$.
\end{enumerate}
\end{theorem}

A proof is given in \textcite[Theorem 1.1]{seneta2006non}. Using the Perron-Frobenius theorem it is possible to state in which cases a unique solution for the stationary distribution can be found.

\begin{theorem}
\label{th:markov-stat}
Let $M$ be a transition matrix and $\bm 1$ a vector where all entries are one. An irreducible and aperiodic Markov chain has a unique stationary distribution given by the solution $\bm\pi$ of $\bm\pi^T M = \bm\pi^T$, where $\bm\pi^T \mathbf{1} = 1$.
\end{theorem}

This theorem will be proven, but two corollaries are necessary.

\begin{corollary}
\label{co:primitive}
An irreducible and aperiodic matrix is primitive.
\end{corollary}

\begin{corollary}
\label{co:smallest-largest}
Let $Q$ be a positive square matrix. Then the minimal row sum is a lower bound and the maximal row sum is an upper bound of the the largest eigenvalue of $Q$.
\end{corollary}

\begin{proof}[Proof for theorem \ref{th:markov-stat}]
Since any transition matrix $M$ is per definition square and non-negative, the statements of the Perron-Frobenius theorem hold for transition matrix $M$, if the Markov chain is primitive. Since corollary \ref{co:primitive} states that any irreducible and aperiodic matrix is primitive, the assumptions of the theorem are clarified.

Furthermore, it holds that

\begin{equation*}
M\mathbf{1} = \mathbf{1},
\end{equation*}

since every row sums to one by definition of the transition matrix. Hence, $M$ has an eigenvalue of $1$ and an eigenvector of $\mathbf{1}$. With corollary \ref{co:smallest-largest}, the largest eigenvalue is exactly the row sum $1$, since all rows of $M$ have the same sum. Therefore, the eigenvalue $r=1$ is the Perron-Frobenius eigenvalue, since property (3) of the theorem states that all other eigenvalues are smaller in absolute. The corresponding right Perron-Frobenius eigenvector is $\mathbf{1}$.

Assuming that a vector $\bm v^T$ is normed, such that $\bm v^T \mathbf{1} = 1$, the eigenvalue problem for the left eigenvector $\bm v^T$ has the following form:

\begin{equation*}
\bm v^T M = r \bm v^T = \bm v^T
\end{equation*}

From statement (2), we know that the left eigenvector regarding $r$ is strictly positive. Therefore, $\bm v^T = \bm\pi^T$ is a distribution, the stationary distribution.

Finally, statement (4) of the Perron-Frobenius theorem states that $\bm\pi$ is unique.
\end{proof}

As a side note, using theorem \ref{th:markov-stat}, the stationary distribution can be calculated as a right eigenvalue problem, just by transposing both sides $(\bm\pi^T M)^T = (\bm\pi^T)^T$, which results in $M^T \bm\pi = \bm\pi$.

Theorem \ref{th:markov-stat} is limited to aperiodic Markov chains. Another theorem lowers the assumptions.

\begin{theorem}
\label{th:irr-rec}
An irreducible and positive recurrent Markov chain has a unique stationary distribution $\bm\pi$ given by

\begin{equation}
\pi_{x_i} = \frac{1}{\E[T_{x_i}]}
\end{equation}
\end{theorem}

A proof is given in \textcite[Corollary 1.2.29]{bladt2017matrix}. The theorem uses the definition of the random variable $T_{x_i}$ from equation \eqref{eq:first-comeback}, which indicates the number of steps, necessary to come back to state $x_i$ again for the first time. Furthermore $\E[T_{x_i}]$ was defined in equation \eqref{eq:first-comeback-expect}.

\begin{theorem}
An irreducible and finite Markov chain is positive recurrent.
\end{theorem}

This theorem is proven in \textcite[Theorem 3.3]{bremaud2013markov}. It shows, that theorem \ref{th:irr-rec} is highly practical for finite Markov chains. Since, in case of finite Markov chains, only the property of irreducibility needs to be fulfilled.

The Perron-Frobenius theorem has a slight advantage, compared to theorem \ref{th:irr-rec}. Using the Perron-Frobenius theorem, it can be shown that even a \emph{limiting distribution} can be obtained. It is defined by $\pi_i = \lim_{t \to\infty} \Pb(X_t = x_i)$. In case of theorem \ref{th:irr-rec}, this is not necessarily the case. If we assume for example a periodic Markov chain with just two states, which are alternating $(0,1,0,1,0,1, ...)$, the Markov chain has no limiting distribution, but a unique stationary distribution. However, for calculating stationary distributions, in most cases theorem \ref{th:markov-stat} was used.

%%%%% Measures %%%%%
\subsection{Measures}
\label{sec:markov-measures}

To evaluate the stationary distribution many measures can be used, which are common for any distribution, not only for Markov chains. Two types are introduced, which are used in the results. At first, information measures, namely variance and Kullback-Leibler divergence, are defined and afterwards a measure for the concentration of a distribution is presented, where the Lorenz curve and the Gini coefficient are introduced. Note that the Gini coefficient is directly derived from the Lorenz curve.

\paragraph{Information measures}

For simplification, the state space can be defined as $S = \{1, ..., n\}$ where $n \in\N$. A simple measure for the stationary distribution is the deviation of the probabilities from their mean probability. Therefore an empirical variance measure is used by calculating

\begin{equation}
\label{eq:variance-estimate}
\sigma^2_\pi = \frac{1}{n} \sum_{k=1}^n (\pi_k - \bar\pi)^2,
\end{equation}

where $\bar\pi = \frac{1}{n} \sum_{k=1}^n \pi_k$. In the following it is just called \emph{variance}.

% If $\Pi = (\Pi_1, ..., \Pi_n)^T$ denotes a random variable for the stationary distribution, the variance of the stationary distribution is given by

% \begin{equation}
% \label{eq:variance}
% \Var(\Pi) = \E[(\Pi - \E[\Pi])^T (\Pi - \E[\Pi])],
% \end{equation}

% which can be interpreted as the expected squared distances between the probabilities of the stationary distribution and their expectation value.

% Since the random variable $\Pi$ depends on the stochastic transition matrix, with probabilities $\Pb(X_{t+1} = x_i | X_t = x_j)$, where $X_t$ is the random variable of the current state at time $t$, it is not easy to obtain the variance analytically. Therefore, a simple estimator is suggested, defined by

% \begin{equation}
% \label{eq:variance-estimate}
% \widehat{\Var(\Pi)} = \sigma^2_\pi = \sum_{k=1}^n (\pi_k - \bar\pi)^2,
% \end{equation}

% where $\bar\pi = \frac{1}{n} \sum_{k=1}^n \pi_k$.

Another measure uses the Kullback-Leibler divergence. If $\bm\pi' \in \{\pi'_1, ..., \pi'_n\} = \{1/n, ..., 1/n\}$ is a stationary distribution where all states are equally probable, the Kullback-Leibler divergence can be obtained by

\begin{equation}
\label{eq:kullbackleibler}
\DKL = \sum_{i=1}^n \pi_i \ln{\left(\frac{\pi_i}{\pi'_i}\right)}.
\end{equation}

The Kullback-Leibler divergence can be understood as relative entropy. It describes the increase in information which is necessary to describe $\bm\pi$, when $\bm\pi'$ is given. It is important to note that the Kullback-Leibler divergence is an asymmetric measure. The value of the measure depends on the order of the distributions and therefore the divergence does not fulfill the properties of a \emph{metric}. It is in particular different from the \emph{total variation distance} $\delta(\bm\pi, \bm\pi')$. A link between the total variation distance and the Kullback-Leibler divergence is given by \emph{Pinsker's inequality}. However, \emph{Gibb's inequality} states that

\begin{equation}
\label{eq:kullbackleibler-gibbs}
\DKL \ge 0
\end{equation}

and $\DKL = 0$ if and only if $\bm\pi = \bm\pi'$, which means that a divergence of $0$ indicates a match between the distributions $\bm\pi$ and $\bm\pi'$.

\begin{SCfigure}[0.6][!b]
    \includegraphics[width=0.45\textwidth]{sorn_markov/lorenz}
    \caption[Lorenz curve]{Illustration of a Lorenz curve. On the $x$-axis is the cumulative share of the probabilities from the states of an equally distributed stationary distribution. On the $y$-axis is the cumulative share of the probabilities of the actual stationary distribution. If the actual distribution is equally distributed, the Lorenz curve reaches the line of equality. The Gini coefficient is calculated by $G = A / (A+B)$.}
    \label{fig:lorenz-illustration}
\end{SCfigure}

\paragraph{Concentration measures}

Another way of evaluating the properties of a Markov chain is to plot the \emph{Lorenz curve} and calculate the \emph{Gini coefficient}. Traditionally, both is often applied to indicate inequality regarding income, used in social sciences. But the Lorenz curve shows the concentration of a distribution in general, whereas the Gini coefficient is a related measure for the intensity of the concentration of the given distribution.

First, assume that $\bm\pi = (\pi_1, ..., \pi_n)^T$ is sorted in a sense that $\pi_1 \le ... \le \pi_n$ and denote $r_i = 1/n$ as the share of a state with probability $\pi_i$, where $n\in\N$ is the number of states. Furthermore $R_k = \sum_{i=1}^k r_i = \sum_{i=1}^k 1/n = k/n$ is the cumulative share of the states. On the other hand, $P_k = \sum_{i=1}^k \pi_i$ is the cumulated probability of $\bm\pi$. The \emph{Lorenz curve} shows the cumulated share of states $R_k$ at the $x$-axis and the cumulated probability $P_k$ at the $y$-axis. If all probabilities are equal, it will result in a straight diagonal line. The more the distribution $\bm\pi$ is concentrated, the more the line will appear in the right bottom corner. An illustration is given in figure \ref{fig:lorenz-illustration}.

Finally, the \emph{Gini coefficient} $G$ is the ratio of the area that lies between the diagonal line, where all states are equal probable, and the area between the actual Lorenz curve and the diagonal. The more the Lorenz curve tends to the corner and, therefore, the more the stationary distribution is unequal, the more the Gini coefficient will tend to $1$, where $1$ represents a perfect concentrated situation. It means that the whole probability is concentrated at one state, the other states are zero. If the distribution is perfectly equal, the Gini coefficient will reach $0$. It can be shown that $G \in [0,1]$ is half of the \emph{relative mean absolute difference}. Hence, for the discrete case, the Gini coefficient is defined as

\begin{equation}
\label{eq:gini}
G = \frac{\sum_{i=1}^n \sum_{j=1}^n |\pi_i - \pi_j|}{2n\cdot \sum_{i=1}^n \pi_i}.
\end{equation}

%%%%% Monte Carlo %%%%%
\subsection{Markov Chain Monte Carlo (MCMC)}

A \emph{Monte Carlo simulation} is a stochastic procedure where random samples are drawn many times. As a result, numerical solutions for analytically difficult or even impossible problems are obtained. \textcite{thomopoulos2012essentials} described the process as follows:

\begin{quote}
To apply the Monte Carlo method, the analyst constructs a mathematical model that simulates a real system. A large number of random sampling of the model is applied yielding a large number of random samples of output results from the model.
\end{quote}

Specifically, if random samples are drawn from the distribution of a Markov chain, it is called \acfi{mcmc} method which includes a broad class of algorithms. Famous and well known algorithms are the \emph{Metropolis–Hastings algorithm} \parencite{hastings1970monte} or the \emph{Gibbs sampler} \parencite{geman1984stochastic}. These techniques are often used in Bayesian inference to evaluate posterior distributions.

There are many different possibilities to understand the behavior of neural networks under the perspective of \acs{mcmc} sampling. For example, there are applications for Boltzmann machines \parencite{osogami2017boltzmann} or energy-based models in general \parencite{goodfellow2007deep}. In \acs{sorn}, it is possible to view the learned time-dependent input patterns under the perspective of \acs{mcmc} sampling. The approach will be introduced in methods (section \ref{sec:methods}) and many illustrative implementations will be given in results (section \ref{sec:results}).

%%%%% MCMC / ANN %%%%%
%\subsection{MCMC and neural networks}

%An overview to MCMC methods in neural networks is given in \cite{andrieu2003introduction}.

%Boltzmann machines, which are a special kind of a stochastic recurrent neural networks, highly depend on a MCMC approach. They are trained by stochastic gradient, where expected values need to be calculated. In many cases those expected values cannot be calculated analytically. Therefore MCMC methods, especially Gibbs sampling can be used to finally train Boltzmann machines \parencite{osogami2017boltzmann}. The MCMC approach can also be used for the broader class of energy-based models \parencite{goodfellow2007deep}.

%MCMC can also be used for model selection. It can for example applied to estimate the number of neurons in a network (3 citations).
% http://www.cs.ubc.ca/~arnaud/andrieu_defreitas_doucet_jordan_intromontecarlomachinelearning.pdf (Seite 23)

%Sampling Maas / Sampling perspective

%It can be summarized that MCMC plays an important role in specific applications of neural networks. The last application should be kept in mind, since in methods section another perspective is introduced, which is inspired by the approach of (paper Maas).

%Where was it used in literature before? link to neural networks?

%Introduce perspective to SORN networks -> In methods


%!TEX root = TTK4150-Summary.tex
\section{Perturbation theory and averaging}
\begin{equation}\label{eq:perturbed}
	\dot{x} = f(x) + \epsilon g(t,x,\epsilon)
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Periodic perturbation of autonomous systems}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Definition of $P_\epsilon(x)$:}
$\phi(t;t_0,x_0,\epsilon)$ is the solution of \eqref{eq:perturbed} that starts at $(t_0,x_0)$. $P_\epsilon(x)$ is
\begin{equation}
	P_\epsilon(x) = \phi(T;0,x,\epsilon)
\end{equation}

\paragraph{Lemma 10.1}
The system \eqref{eq:perturbed} has a T-periodic solution iff
\begin{equation}
	x = P_\epsilon(x)
\end{equation}
has a solution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Averaging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

TTK4150 Nonlinear control systems/sec-perturbation-theory-and-averaging.tex


\section{Volumes}
\subsection{Volumes from Cross Sections}
Imagine we want to find the volume of some complex object.
One way we could approximate it is by slicing it into narrow cross sections.
The volume of each cross section would roughly be the area of the cross sectional face, times the width of the cross section.

\begin{figure}[H]
	\label{volumes}
	\centering
	\includegraphics[width=0.5\textwidth]{./applications_integrals/volumes.png}
	\caption{\hyperref{https://tutorial.math.lamar.edu/classes/calci/Area\_Volume\_Formulas.aspx}{}{}{Paul's Online Notes - Area and Volume Formulas}}
\end{figure}

Adding the volumes of these cross sections up, we'd get our approximation, which would get better and better the narrower the width of each cross section.
In the limit, this is exactly the definition of an integral.

\begin{definition}
	The volume of a solid with integrable corss sectional area $A(x)$ from $x=a$ to $x=b$ is given by
	\begin{equation*}
		V = \int_{a}^{b}{A(x)\d{x}}.
	\end{equation*}
\end{definition}

Let's start by finding the volume of a solid we already know.
\begin{example}
	Find the volume of a cube with sidelength $a$.
\end{example}
\begin{answer}
	For any slice of a cube, the cross section is a square with sidelength $a$.
	\begin{align*}
		A(x) &= a^2 \\
		V &= \int_{0}^{a}{a^2\d{x}} \\
		&= a^2x\biggr\rvert_0^a \\
		&= a^3.
	\end{align*}
\end{answer}

Now let's find the volume of a slightly more complicated shape.
\begin{example}
	Find the volume of a square pyramid with base sidelength $a$ and height $h$.
\end{example}
\begin{answer}
	Like the cube, any slice of the square pyramid is a square.
	However, the sidelength of the square depends on the height of your slice.
	A slice at the very tip of the pyramid would have sidelength 0, while a slice at the very bottom of the pyramid would have sidelength $a$.
	The sidelength grows linearly from $x=0$ to $x=h$, so it must be $\frac{a}{h}x$.
	\begin{align*}
		A(x) &= \left(\frac{a}{h}x\right)^2 \\
		V &= \int_{0}^{h}{\frac{a^2}{h^2}x^2\d{x}} \\
		&= \frac{a^2}{3h^2}x^3 \biggr\rvert_0^h \\
		&= \frac{a^2h}{3} \\
		&= \frac{1}{3}a^2 h.
	\end{align*}
\end{answer}

\subsection{Solids of Revolution}
You can think of solids of revolution as a special case of volumes from cross sections.
We'll tend to be on the lookout for a function that defines the radius, and we can then use the formula for the area of a circle to get the cross sectional area.
\begin{equation*}
	A(x) = \pi r^2(x).
\end{equation*}

\begin{example}
	Find the volume of the cone formed by rotating the line $y=x/3$ about the $x$-axis for $0 \leq x \leq 6$.
\end{example}
\begin{answer}
	The radius is simply the distance from the $x$-axis, which is just another name for $y$.
	So,
	\begin{equation*}
		A(x) = \pi \left(\frac{x}{3}\right)^2.
	\end{equation*}

	Integrating,
	\begin{align*}
		V &= \int_{0}^{6}{\pi\left(\frac{x}{3}\right)^2\d{x}} \\
		&= \frac{\pi}{9}\int_{0}^{6}{x^2\d{x}} \\
		&= \frac{\pi}{9}\left(\frac{x^3}{3}\biggr\rvert_0^6\right) \\
		&= 8\pi.
	\end{align*}

	This is the volume of a cone with radius 2 and height 6: $\frac{1}{3}\pi(2)^2(6)=8\pi$.
\end{answer}

Let's try a more complicated solid.
\begin{example}
	Find the volume of the solid of revolution bounded by $y=2+x\cos{x}$ from $-\frac{\pi}{2} \leq x \leq \frac{\pi}{2}$.
\end{example}
\begin{answer}
	Again, the radius is simply the distance from the $x$-axis, which is $y$.
	\begin{align*}
		A(x) &= \pi \left(2+x\cos{x}\right)^2 \\
		&= \pi \left(4 + 4x\cos{x} + x^2\cos^2{x}\right) \\
		V &= \int_{-\pi/2}^{\pi/2}{\pi \left(4 + 4x\cos{x} + x^2\cos^2{x}\right)\d{x}} \\
		&= \pi\left(\int_{-\pi/2}^{\pi/2}{4\d{x}}+\int_{-\pi/2}^{\pi/2}{4x\cos{x}\d{x}}+\int_{-\pi/2}^{\pi/2}{x^2\cos^2{x}\d{x}}\right) \\
		&= \pi\left(4\pi + 0 + \frac{1}{24}\pi\left(\pi^2-6\right)\right) \text{ (use integration by parts)}\\
		&= \frac{\pi^4}{24} + \frac{15\pi^2}{4}.
	\end{align*}
\end{answer}

\subsubsection{Washer Method}
Imagine now we want to find the volume of a solid of revolution with a ``hole.''
If we were to take a cross section of such a solid, it'd look like a washer with an outer radius $R(x)$ and inner radius $r(x)$.

\begin{figure}[H]
	\label{washers}
	\centering
	\includegraphics[width=0.33\textwidth]{./applications_integrals/Washer-1.png}
	\caption{\hyperref{https://www.shelovesmath.com/calculus/integral-calculus/applications-integration-area-volume/}{}{}{She Loves Math - Applications of Integration: Area and Volume}}
\end{figure}

We can think of this in a similar way as the area between curves in 2D.
We'll find the volume swept by the outer radius and the subtract the volume swept and removed by the inner radius.
\begin{equation*}
	V = \pi\int_{a}^{b}{(R^2(x)-r^2(x))\d{x}}.
\end{equation*}

\begin{example}
	Find the volume of shape formed by revolving the area enclosed by the $y$-axis, $y=\cos{x}$, and $y=\sin{x}$ around the $x$-axis.
\end{example}
\begin{answer}
	In the first quadrant, $\cos{x} \geq \sin{x}$ for $x \leq \frac{\pi}{4}$, so our bounds are $0 \leq x \leq \frac{\pi}{4}$, $R(x)=\cos{x}$, and $r(x)=\sin{x}$.
	\begin{align*}
		V &= \pi\int_{0}^{\pi/4}{(\cos^2{x}-\sin^2{x})\d{x}} \\
		&= \pi\int_{0}^{\pi/4}{\cos{(2x)}\d{x}} \\
		&= \pi\left(\frac{\sin{2x}}{2}\right)\biggr\rvert_0^{\pi/4} \\
		&= \frac{\pi}{2}.
	\end{align*}
\end{answer}

\subsubsection{Cylindrical Shells Method}
All the previous methods for finding the volumes of solids of rotation have relied on summing the volumes of thin cross sectional slices that are perpendicular to the axis of rotation.
However, we can instead sum the volume of thin cylindrical shells that grow outwards from and parallel to the axis of revolution.

\begin{figure}[H]
	\label{shells}
	\centering
	\includegraphics[width=0.5\textwidth]{./applications_integrals/shells.png}
	\caption{\hyperref{https://en.wikipedia.org/wiki/Shell\_integration}{}{}{Wikipedia - Shell Integration}}
\end{figure}

Each cylindrical shell will have a radius $r(x)$, a height $h(x)$, and a thickness $\d{x}$, meaning a volume of $2\pi r(x)h(x)\d{x}$.
\begin{equation*}
	V = 2\pi\int_{a}^{b}{r(x)h(x)\d{x}}.
\end{equation*}

\begin{example}
	Find the volume of the area bounded by the $y$-axis, $y=4-x^2$, and $y=x$ revolved around the $y$-axis.
\end{example}
\begin{answer}
	Each shell's height is parallel to the axis of rotation.
	In this case that's the distance between the two curves.
	\begin{equation*}
		h(x) = 4-x^2-x \text{ and } r(x)=x.
	\end{equation*}

	The two curves intersect at $x=\frac{-1+\sqrt{17}}{2}$.
	Finding the volume,
	\begin{align*}
		V &= 2\pi\int_{0}^{\frac{-1+\sqrt{17}}{2}}{x(4-x^2-x)\d{x}} \\
		&= 2\pi\left(-\frac{x^4}{4}-\frac{x^3}{3}+2x^2\right)\biggr\rvert_{0}^{\frac{-1+\sqrt{17}}{2}} \\
		&= \frac{\pi}{12}\left(121-17\sqrt{17}\right).
	\end{align*}
\end{answer}

\begin{example}
	Find the volume of the area bounded by the $x$-axis and the curve $y=(x-1)^2(x-2)^2$ rotated about the $y$-axis.
\end{example}
\begin{answer}
	The height of the shell is parallel to the $y$-axis.
	In this case it's exactly equal to the value of the bounding curve.
	\begin{equation*}
		h(x)=(x-1)^2(x-2)^2 \text{ and } r(x)=x.
	\end{equation*}

	Finding the volume,
	\begin{align*}
		V &= 2\pi\int_{1}^{2}{x(x-1)^2(x-2)^2\d{x}} \\
		&= 2\pi\left(\frac{x^6}{6}-\frac{6x^5}{5}+\frac{13x^4}{4}-4x^3+2x^2\right)\biggr\rvert_{1}^{2} \\
		&= \frac{\pi}{10}.
	\end{align*}
\end{answer}

\subsubsection{Other Axes of Rotation}
Although rotating about the $x$ and $y$ axes are the most common, the methods we have here apply to rotating about any axis parallel to the $x$ or $y$ axis.
The includes any lines of the form $y=k$ or $x=k$ where $k$ is some constant. \\

One valid approach is simply to rewrite the equations for your bounding curves by shifting them such that your axis of rotation is the $x$ or $y$ axis.
However, it's often more convenient to not rewrite the equations and simply apply the methods.

\begin{example}
	Find the volume of the solid generated by rotating the region in the first quadrant bounded by $y=x^2$ and $y=2x$ about the line $x=-2$.
\end{example}
\begin{answer}
	We can use the washer method (although shells also works).
	Since we're rotating about a line parallel to the $y$-axis, we'll have to rewrite our equations in the form $x=\ldots$.
	\begin{equation*}
		x = \sqrt{y} \text{ and } x = \frac{y}{2}.
	\end{equation*}

	The inner radius is given by the distance from $x=-2$ to $x=y/2$, and the outer radius is given by the distance from $x=-2$ to $x=\sqrt{y}$.
	The curves intersect at $y=0$ and $y=4$.
	\begin{align*}
		r(x) = 2 + \frac{y}{2} &\text{ and } R(x) = 2 + \sqrt{y} \\
		V &= \pi\int_{0}^{4}{(2+\sqrt{y})^2-\left(2+\frac{y}{2}\right)^2\d{y}} \\
		&= \pi\int_{0}^{4}{\left(4\sqrt{y}-y-\frac{y^2}{4}\right)\d{y}} \\
		&= 8\pi.
	\end{align*}
\end{answer}

\begin{example}
	Find the volume of the solid generated by rotating the region bounded by $y=x^2$ and $y=x+2$ about the line $x=3$.
\end{example}
\begin{answer}
	We can use the shells method.
	Our height is simply the difference in $y$ values between the two curves.
	The radius is the distance between the lines $x=3$ and $x+2$.
	\begin{equation*}
		h(x) = x+2-x^2 \text{ and } r(x) = 3-x.
	\end{equation*}

	The curves intersect at $x=-1$ and $x=2$.
	Finding the volume,
	\begin{align*}
		V &= 2\pi\int_{-1}^{2}{(3-x)(x+2-x^2)\d{x}} \\
		&= 2\pi\left(\frac{x^4}{4}-\frac{4x^3}{3}+\frac{x^2}{2}+6x\right)\biggr\rvert_{-1}^{2} \\
		&= \frac{45\pi}{2}.
	\end{align*}
\end{answer}

\subsubsection{Surface Area}
Can apply the idea behind shell integration to derive a formula for surface area of a solid of rotation.
A cylindrical shell would have height $\d{s}$, which is given to us by the arc length formula, and radius $x$ or $y$, if the axis of rotation is the $y$ or $x$ axis respectively.
\begin{align*}
	S &= \begin{cases}
		2\pi\int{y\d{s}} & \text{Rotation about $x$-axis} \\
		2\pi\int{x\d{x}} & \text{Rotation about the $y$-axis}
	\end{cases} \\
	& \text{where } \\
	\d{s} &= \begin{cases}
		\sqrt{1+\left(\dd{y}{x}\right)^2}\d{x} & y=f(x) \\
		\sqrt{1+\left(\dd{x}{y}\right)^2}\d{y} & x=g(y) \\
	\end{cases}.
\end{align*}

\begin{example}
	Find the surface area of a sphere with radius $r$.
\end{example}
\begin{answer}
	We can obtain a sphere by rotating $y=\sqrt{r^2-x^2}, -r\leq x\leq r$ about the $x$-axis.
	\begin{align*}
		\dd{y}{x} &= \frac{-x}{\sqrt{r^2-x^2}} \\
		\left(\dd{y}{x}\right)^2 &= \frac{x^2}{r^2-x^2} \\
		S &= 2\pi\int_{-r}^{r}{y\sqrt{1+\frac{x^2}{r^2-x^2}}\d{x}} \\
		&= 2\pi\int_{-r}^{r}{\sqrt{r^2-x^2}\frac{r}{\sqrt{r^2-x^2}}\d{x}} \\
		&= 2\pi\int_{-r}^{r}{r\d{x}} \\
		&= 4\pi r^2.
	\end{align*}
\end{answer}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{Proving the Quadratic Formula}
\author{Ethan Xu}
\date{September 2021}

\usepackage{chngcntr}
\usepackage{amssymb}
\usepackage{ragged2e}
\usepackage{amsmath,amsthm}

\renewcommand{\qed}{\hfill\square}

\counterwithin*{equation}{section}
\counterwithin*{equation}{subsection}

\begin{document}

\maketitle

\section{Introduction}

The quadratic formula states that:

\begin{equation}
    x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}
\end{equation}

when given a quadratic equation such as:

\begin{equation}
    ax^2+bx+c=0
\end{equation}

\section{Proof}

\begin{equation}
    ax^2+bx+c=0
\end{equation}

\begin{equation}
    ax^2+bx=-c
\end{equation}

\begin{equation}
    x^2+\frac{b}{a}x=-\frac{c}{a}
\end{equation}

\begin{equation}
    x^2+\frac{b}{a}x+\frac{b^2}{4a^2}=-\frac{c}{a}+\frac{b^2}{4a^2}
\end{equation}

\begin{equation}
    (x+\frac{b}{2a})^2=-\frac{c}{a}+\frac{b^2}{4a^2}
\end{equation}

\begin{equation}
    x+\frac{b}{2a}=\pm\sqrt{-\frac{c}{a}+\frac{b^2}{4a^2}}
\end{equation}

\begin{equation}
    x=-\frac{b}{2a}\pm\sqrt{-\frac{c}{a}+\frac{b^2}{4a^2}}
\end{equation}

\begin{equation}
    x=-\frac{b}{2a}\pm\frac{\sqrt{-\frac{c}{a}\cdot(2a)^2+\frac{b^2}{4a^2}\cdot(2a)^2}}{2a}
\end{equation}

\begin{equation}
    x=-\frac{b}{2a}\pm\frac{\sqrt{-4ac+b^2}}{2a}
\end{equation}

\begin{equation}
    x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}
\end{equation}

$\qed$

\end{document}


In this chapter we explore some consequences of a basic mathematical fact called the \textbf{Pigeonhole Principle}. Although the statement itself is very elementary, the various consequences and results that one can obtain as a consequence of this basic fact are very deep and surprising. In the first section we will explore the pigeonhole principle's application in the finite case and the infinite case.

\section{The Pigeonhole Principle}
\input{Ch9/pigeonhole}

\section{Ramsey's Theorem}

\subsection{Infinite Ramsey Implies Finite Ramsey}
\input{Ch9/infImpliesF.tex}


% based on example 7 in pythontex_gallery
% https://github.com/gpoore/pythontex/

\documentclass[12pt]{mmalatex}
\usepackage{examples}

\begin{document}

\section*{Step-by-step integration}

This is another nice example drawn from the Pythontex gallery, see
\ \url{https://github.com/gpoore/pythontex}.

It shows the step-by-step computations of a simple triple integral.

\vspace{12pt}

\lstset{numbers=left}
\begin{mathematica}
   xmax = 2;  ymax = 3;  zmax = 4;
   xmin = 0;  ymin = 0;  zmin = 0;

   fun = f[x,y,z];

   mytmp = HoldForm[Integrate[#1, {z, #6, #7},
                                  {y, #4, #5},
                                  {x, #2, #3}]] & @@ {fun, xmin, xmax, ymin, ymax, zmin, zmax};   (* mma(lhs.01,mytmp) *)

   fun = x y + y Sin[z] + Cos[x+y];

   myint = HoldForm[Integrate[#1, {z, #6, #7},
                                  {y, #4, #5},
                                  {x, #2, #3}]] & @@ {fun, xmin, xmax, ymin, ymax, zmin, zmax};   (* mma(rhs.01,myint) *)

   myansx =          Integrate[#1, {x, #2, #3}] & @@ {fun, xmin, xmax};
   myintx = HoldForm[Integrate[#1, {z, #4, #5},
                                   {y, #2, #3}]] & @@ {myansx, ymin, ymax, zmin, zmax};           (* mma(rhs.02,myintx) *)

   myansxy =          Integrate[#1, {y, #2, #3}]  & @@ {myansx, ymin, ymax};
   myintxy = HoldForm[Integrate[#1, {z, #2, #3}]] & @@ {myansxy, zmin, zmax};                     (* mma(rhs.03,myintxy) *)

   myansxyz = Integrate[#1, {z, #2, #3}] & @@ {myansxy, zmin, zmax};                              (* mma(rhs.04,myansxyz) *)

   myapprox = N[myansxyz,15];                                                                     (* mma(rhs.05,myapprox) *)
\end{mathematica}

\begin{minipage}[t]{0.65\textwidth}
\begin{align*}
   \mma{lhs.01} &= \mma{rhs.01}\\
                &= \mma{rhs.02}\\
                &= \mma{rhs.03}\\
                &= \mma{rhs.04}\\[8pt]
                &\approx \mma{rhs.05}
\end{align*}
\end{minipage}
\hskip 1cm
\lower16pt\hbox{%
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      \mma{lhs.01} &= \mma{rhs.01}\\
                   &= \mma{rhs.02}\\
                   &= \mma{rhs.03}\\
                   &= \mma{rhs.04}\\[8pt]
                   &\approx \mma{rhs.05}
   \end{align*}
\end{latex}
\end{minipage}}

\end{document}


\section*{Exercises}

\begin{ex} \label{exer-linear-transf}
Consider the following functions which map $\R^{n}$ to $\R^{n}$.

\begin{enumerate}
\item $T$ multiplies the $j\th$ component of $\vect{x}$ by a non-zero
number $b$.

\item $T$ replaces the $i\th$ component of $\vect{x}$ with $b$ times the
$j\th$ component added to the $i\th$ component.

\item $T$ switches the $i\th$ and $j\th$ components.
\end{enumerate}

Show these functions are linear transformations and describe their matrices $A$ such that $T(\vect{x}) = A\vect{x}$.
\begin{sol}
\begin{enumerate}
\item The matrix of $T$ is the elementary matrix which multiplies
the $j\th$ diagonal entry of the identity matrix by $b$.
\item The matrix of $T$ is the
elementary matrix which takes $b$ times the $j\th$ row and adds to the $%
i\th$ row.
\item The matrix of $T$ is the elementary matrix which switches the $%
i\th$ and the $j\th$ rows where the two components are in the $i\th$
and $j\th$ positions.
\end{enumerate}
\end{sol}
\end{ex}

\begin{ex} You are given a linear transformation $T:\R^{n}\rightarrow
\R^{m}$ and you know that
\begin{equation*}
T(A_{i})=B_{i}
\end{equation*}
where $\begin{mymatrix}{ccc}
A_{1} & \cdots & A_{n}
\end{mymatrix} ^{-1}$ exists. Show that the matrix of $T$ is of the form
\begin{equation*}
\begin{mymatrix}{ccc}
B_{1} & \cdots & B_{n}
\end{mymatrix} \begin{mymatrix}{ccc}
A_{1} & \cdots & A_{n}
\end{mymatrix} ^{-1}
\end{equation*}
\begin{sol}
Suppose
\[
\begin{mymatrix}{c}
\vect{c}_{1}^{T} \\
\vdots \\
\vect{c}_{n}^{T}
\end{mymatrix} =\begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix}^{-1}
\]
Thus $\vect{c}_{i}^{T}\vect{a}_{j}=\delta _{ij}$. Therefore,
\begin{eqnarray*}
\begin{mymatrix}{ccc}
\vect{b}_{1} & \cdots & \vect{b}_{n}
\end{mymatrix} \begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} ^{-1}\vect{a}_{i} &=&
\begin{mymatrix}{ccc}
\vect{b}_{1} & \cdots & \vect{b}_{n}
\end{mymatrix} \begin{mymatrix}{c}
\vect{c}_{1}^{T} \\
\vdots \\
\vect{c}_{n}^{T}
\end{mymatrix} \vect{a}_{i} \\
&=&\begin{mymatrix}{ccc}
\vect{b}_{1} & \cdots & \vect{b}_{n}
\end{mymatrix} \vect{e}_{i} \\
&=&\vect{b}_{i}
\end{eqnarray*}
Thus $T\vect{a}_{i}=\begin{mymatrix}{ccc}
\vect{b}_{1} & \cdots & \vect{b}_{n}
\end{mymatrix} \begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} ^{-1}\vect{a}_{i} =  A\vect{a}_{i}$. If $\vect{x}$ is
arbitrary, then since the matrix $\begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} $ is invertible, there exists a unique $\vect{y}$ such that $
\begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} \vect{y}=\vect{x}$ Hence
\[
T\vect{x}=T\paren{\sum_{i=1}^{n}y_{i}\vect{a}_{i}}
=\sum_{i=1}^{n}y_{i}T\vect{a}_{i}=\sum_{i=1}^{n}y_{i}A\vect{a}
_{i}=A\paren{\sum_{i=1}^{n}y_{i}\vect{a}_{i}} =A\vect{x}
\]

\end{sol}
\end{ex}

\begin{ex} Suppose $T$ is a linear transformation such that
\begin{eqnarray*}
T\begin{mymatrix}{r}
1 \\
2 \\
-6
\end{mymatrix} &=&\begin{mymatrix}{r}
5 \\
1 \\
3
\end{mymatrix} \\
T\begin{mymatrix}{r}
-1 \\
-1 \\
5
\end{mymatrix} &=&\begin{mymatrix}{r}
1 \\
1 \\
5
\end{mymatrix} \\
T\begin{mymatrix}{r}
0 \\
-1 \\
2
\end{mymatrix} &=&\begin{mymatrix}{r}
5 \\
3 \\
-2
\end{mymatrix}
\end{eqnarray*}
Find the matrix of $T$. That is find $A$ such that $T(\vect{x})=A\vect{x}$. \vspace{1mm}
\begin{sol}
\[
\begin{mymatrix}{rrr}
5 & 1 & 5 \\
1 & 1 & 3 \\
3 & 5 & -2
\end{mymatrix} \begin{mymatrix}{ccc}
3 & 2 & 1 \\
2 & 2 & 1 \\
4 & 1 & 1
\end{mymatrix} =\begin{mymatrix}{ccc}
37 & 17 & 11 \\
17 & 7 & 5 \\
11 & 14 & 6
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Suppose $T$ is a linear transformation such that
\begin{eqnarray*}
T\begin{mymatrix}{r}
1 \\
1 \\
-8
\end{mymatrix} &=&\begin{mymatrix}{r}
1 \\
3 \\
1
\end{mymatrix} \\
T\begin{mymatrix}{r}
-1 \\
0 \\
6
\end{mymatrix} &=&\begin{mymatrix}{r}
2 \\
4 \\
1
\end{mymatrix} \\
T\begin{mymatrix}{r}
0 \\
-1 \\
3
\end{mymatrix} &=&\begin{mymatrix}{r}
6 \\
1 \\
-1
\end{mymatrix}
\end{eqnarray*}
Find the matrix of $T$. That is find $A$ such that $T(\vect{x})=A\vect{x}$. \vspace{1mm}
\begin{sol}
\[
\begin{mymatrix}{rrr}
1 & 2 & 6 \\
3 & 4 & 1 \\
1 & 1 & -1
\end{mymatrix} \begin{mymatrix}{ccc}
6 & 3 & 1 \\
5 & 3 & 1 \\
6 & 2 & 1
\end{mymatrix} =\begin{mymatrix}{ccc}
52 & 21 & 9 \\
44 & 23 & 8 \\
5 & 4 & 1
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Suppose $T$ is a linear transformation such that
\begin{eqnarray*}
T\begin{mymatrix}{r}
1 \\
3 \\
-7
\end{mymatrix} &=&\begin{mymatrix}{r}
-3 \\
1 \\
3
\end{mymatrix} \\
T\begin{mymatrix}{r}
-1 \\
-2 \\
6
\end{mymatrix} &=&\begin{mymatrix}{r}
1 \\
3 \\
-3
\end{mymatrix} \\
T\begin{mymatrix}{r}
0 \\
-1 \\
2
\end{mymatrix} &=&\begin{mymatrix}{r}
5 \\
3 \\
-3
\end{mymatrix}
\end{eqnarray*}
Find the matrix of $T$. That is find $A$ such that $T(\vect{x})=A\vect{x}$. \vspace{1mm}\vspace{1mm}
\begin{sol}
\[
\begin{mymatrix}{rrr}
-3 & 1 & 5 \\
1 & 3 & 3 \\
3 & -3 & -3
\end{mymatrix} \begin{mymatrix}{ccc}
2 & 2 & 1 \\
1 & 2 & 1 \\
4 & 1 & 1
\end{mymatrix} = \begin{mymatrix}{rrr}
15 & 1 & 3 \\
17 & 11 & 7 \\
-9 & -3 & -3
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Suppose $T$ is a linear transformation such that
\begin{eqnarray*}
T\begin{mymatrix}{r}
1 \\
1 \\
-7
\end{mymatrix} &=&\begin{mymatrix}{r}
3 \\
3 \\
3
\end{mymatrix} \\
T\begin{mymatrix}{r}
-1 \\
0 \\
6
\end{mymatrix} &=&\begin{mymatrix}{r}
1 \\
2 \\
3
\end{mymatrix} \\
T\begin{mymatrix}{r}
0 \\
-1 \\
2
\end{mymatrix} &=&\begin{mymatrix}{r}
1 \\
3 \\
-1
\end{mymatrix}
\end{eqnarray*}
Find the matrix of $T$. That is find $A$ such that $T(\vect{x})=A\vect{x}$. \vspace{1mm}
\begin{sol}
\[
\begin{mymatrix}{rrr}
3 & 1 & 1 \\
3 & 2 & 3 \\
3 & 3 & -1
\end{mymatrix} \begin{mymatrix}{ccc}
6 & 2 & 1 \\
5 & 2 & 1 \\
6 & 1 & 1
\end{mymatrix} =\allowbreak \begin{mymatrix}{ccc}
29 & 9 & 5 \\
46 & 13 & 8 \\
27 & 11 & 5
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Suppose $T$ is a linear transformation such that
\begin{eqnarray*}
T\begin{mymatrix}{r}
1 \\
2 \\
-18
\end{mymatrix} &=&\begin{mymatrix}{r}
5 \\
2 \\
5
\end{mymatrix} \\
T\begin{mymatrix}{r}
-1 \\
-1 \\
15
\end{mymatrix} &=&\begin{mymatrix}{r}
3 \\
3 \\
5
\end{mymatrix} \\
T\begin{mymatrix}{r}
0 \\
-1 \\
4
\end{mymatrix} &=&\begin{mymatrix}{r}
2 \\
5 \\
-2
\end{mymatrix}
\end{eqnarray*}
Find the matrix of $T$. That is find $A$ such that $T(\vect{x})=A\vect{x}$. \vspace{1mm}
\begin{sol}
\[
\begin{mymatrix}{rrr}
5 & 3 & 2 \\
2 & 3 & 5 \\
5 & 5 & -2
\end{mymatrix} \begin{mymatrix}{ccc}
11 & 4 & 1 \\
10 & 4 & 1 \\
12 & 3 & 1
\end{mymatrix} =\begin{mymatrix}{ccc}
109 & 38 & 10 \\
112 & 35 & 10 \\
81 & 34 & 8
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Consider the following functions $T:\R^{3}\rightarrow \R^{2}$.
Show that each is a linear transformation and determine for each the matrix $A$ such that
$T(\vect{x})=A\vect{x}$.

\begin{enumerate}
\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
x+2y+3z \\
2y-3x+z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
7x+2y+z \\
3x-11y+2z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
3x+2y+z \\
x+2y+6z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
2y-5x+z \\
x+y+z
\end{mymatrix} $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Consider the following functions $T:\R^{3}\rightarrow \R^{2}$.
Explain why each of these functions $T$ is not linear.

\begin{enumerate}
\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
x+2y+3z+1 \\
2y-3x+z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
x+2y^{2}+3z \\
2y+3x+z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
\sin x+2y+3z \\
2y+3x+z
\end{mymatrix} $

\item $T\begin{mymatrix}{c}
x \\
y \\
z
\end{mymatrix} =\begin{mymatrix}{c}
x+2y+3z \\
2y+3x-\ln z
\end{mymatrix} $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Suppose
\begin{equation*}
\begin{mymatrix}{ccc}
A_{1} & \cdots & A_{n}
\end{mymatrix} ^{-1}
\end{equation*}
 exists where each $A_{j}\in \R^{n}$ and let
vectors  $\set{B_{1},\ldots,B_{n}} $ in $\R^{m}$ be given.
Show that there \textbf{always }exists a linear
transformation $T$ such that $T(A_{i})=B_{i}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex}  Find the matrix for $T(\vect{w}) = \proj_{\vect{v}}(\vect{w}) $
where $\vect{v}=\begin{mymatrix}{rrr}
1 & -2 & 3
\end{mymatrix} ^{T}$.
\begin{sol}
 Recall that $\proj_{\vect{u}}(\vect{v}) =\frac{\vect{v}\dotprod\vect{u} }{\norm{\vect{u}} ^{2}}\vect{u}$ and so the desired matrix
has $i\th$ column equal to $\proj_{\vect{u}}(\vect{e}_{i})$. Therefore, the matrix desired is
\[
\frac{1}{14}\begin{mymatrix}{rrr}
1 & -2 & 3 \\
-2 & 4 & -6 \\
3 & -6 & 9
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex}  Find the matrix for $T(\vect{w}) = \proj_{\vect{v}}(\vect{w}) $
where $\vect{v}=\begin{mymatrix}{rrr}
1 & 5 & 3
\end{mymatrix} ^{T}$.
\begin{sol}
\[
\frac{1}{35}\begin{mymatrix}{rrr}
1 & 5 & 3 \\
5 & 25 & 15 \\
3 & 15 & 9
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Find the matrix for $T(\vect{w}) = \proj_{\vect{v}}(\vect{w}) $
where $\vect{v}=\begin{mymatrix}{rrr}
1 & 0 & 3
\end{mymatrix} ^{T}$.
\begin{sol}
\[
\frac{1}{10}\begin{mymatrix}{ccc}
1 & 0 & 3 \\
0 & 0 & 0 \\
3 & 0 & 9
\end{mymatrix}
\]
\end{sol}
\end{ex}

\begin{ex} Suppose $A$ is a $3\times 3$ antisymmetric matrix such that $A^{T}=-A$. Show
there exists a vector $\vect{v}$ such that for all $\vect{u}\in
\R^{3}$
\begin{equation*}
A\vect{u}=\vect{v} \times \vect{u}
\end{equation*}
\textbf{Hint: }Explain why, since $A$ is antisymmetric it is of the form
\begin{equation*}
A=\begin{mymatrix}{ccc}
0 & -\omega _{3} & \omega _{2} \\
\omega _{3} & 0 & -\omega _{1} \\
-\omega _{2} & \omega _{1} & 0
\end{mymatrix}
\end{equation*}
where the $\omega _{i}$ are numbers. Then consider $\omega _{1}\vect{i}
+\omega _{2}\vect{j}+\omega _{3}\vect{k}$.
%\begin{sol}
%\end{sol}
\end{ex}


\section{Uniqueness of Solutions to Laplace equation}
Assume 2 solutions $\phi_1$ and $\phi_2$ with
$\phi_2\ne \phi_1$.
$\phi_1,\phi_2$ satisfy some boundary conditions,
Dirichlet or Neumann
$\vec{\nabla}\cdot\hat{n}$.

Consider $U=\phi_2 - \phi_1$.
We know $\nabla^2 U=0$ so
\begin{align}
    \int d^3x\,
    \underbrace{
    U \nabla^2 U
    }_{
    \vec{\nabla}\cdot\left( U\vec{\nabla} U \right)
    -
    \left( \vec{\nabla} U \right)^2
    }
    &=0\\
\end{align}
which implies
\begin{align}
    \int_V d^3x\,
    |\vec{\nabla} U|^2
    &=
    \underbrace{
    \int_V d^3x\, \vec{\nabla}\cdot\left( U\vec{\nabla}U \right)
    }_{
    \oint_S dS\, U\left( \vec{n}\cdot\vec{\nabla}U \right)
    }
\end{align}
hence
\begin{align}
    \int_V d^3x\, |\vec{\nabla}U|^2
    &=
    \oint_S dS\, U\left( \vec{\nabla} U\cdot\hat{n} \right)
\end{align}
For Dirichlet boundary conditions,
value of $\phi$ is specified on boundary $S$.
Then $U=0$ on $S$
and
\begin{align}
    \int d^3x\,
    |\vec{\nabla}U|^2 = 0
\end{align}
and we're integrating a positive quantity to be zero,
which means
\begin{align}
    \vec{\nabla}U = 0
\end{align}
everywhere in $V$
and so $U$ is constant.
But since $U=0$ on boundary,
$U=0$ everywhere.

For Neumann case,
$\hat{n}\cdot\vec{\nabla}\phi$ is specified on the boundary.
This is basically saying the normal component of the electric field is
specified.
Then $\vec{n}\cdot\vec{\nabla}U=0$
on the boundary.

So once again,
\begin{align}
    \int d^3x \left| \vec{\nabla} U\right|^2 = 0
\end{align}
Now by exactly the same logic,
$U$ is a constant everywhere.
But we don't know it's actually zero on the boundary.
And so this is as far as we can go.
So all we can really show is that
$\phi_2 - \phi_1$
is a constant in $V$.
So the solution is unique up to an arbitrary constant.

You can always shift the minimum of the potential,
you can change your potential by a constant everywhere and the electric field
doesn't change.
It's the potential differences that matter,
not the absolute value.

You specify the Neumann boundary conditions,
you're just specifying the electric field of the boundary,
the normal component.
Since you didn't specify the potential anywhere,
you can choose your zero potential anywhere,
so you can shift your potential by a constant and the electric field will still
be the same.

In that sense the solution is unique,
because the physical thing,
which is the electric field,
is unique and doesn't change.

The electric field everywhere is completely specified.

The next thing to consider is electrostatics.

\section{Electrostatic Energy}
Change $q_1$ located at $\vec{x}_1$.
Potential at $\vec{x}$ is due to $\vec{q}_1$.
\begin{align}
    \phi\left(\vec{x}\right)
    &=
    \frac{1}{4\pi\epsilon_0}
    \frac{q_1}{\left|\vec{x} - \vec{x}_1\right|}
\end{align}
Now move another charge $q_2$ from $\infty$
to $\vec{x}_2$.
Potential energy is the work done
\begin{align}
    W &=
    q_2 \phi\left( \vec{x}_2 \right)\\
    &=
    \frac{1}{4\pi\epsilon_0}
    \frac{q_1 q-2}{\left|\vec{x}_2 - \vec{x}_1\right|}
\end{align}
And now if we bring in a third charge $q_3$,
you have to include the potential of the two existing charges.
\begin{align}
    W &=
    \frac{1}{4\pi\epsilon_0_0}
    \left\{
    \frac{q_1 q_2}{\left| \vec{x}_2 - \vec{x}_1 \right|}
    + \frac{q_1 q_3}{\left| \vec{x}_3 - \vec{x}_1 \right|}
    + \frac{q_2 q_3}{\left| \vec{x}_3 - \vec{x}_2 \right|}
    \right\}
\end{align}
This generalizes pretty easily to the case where you have $N$ charges.
You add it all up,
up to when you bring the $n$th charge to one point and consider the potential of
all the other $N-1$ charges.

This generalizes in a simple way.
In the case of $n$ charges,
\begin{align}
    W &=
    \sum_{i=1}^{n}
    \sum_{j<i}
    \frac{1}{4\pi\epsilon_0}
    \frac{q_i q_j}{\left| \vec{x}_i - \vec{x}_j \right|}\\
    &=
    \frac{1}{2}
    \sum_{i=1}^{n}
    \sum_{j=1: i\ne j}^{n}
    \frac{1}{4\pi\epsilon_0}
    \frac{q_i q_j}{\left| \vec{x}_i - \vec{x}_j \right|}
\end{align}
For continuous charge distributions,
\begin{align}
    W &=
    \frac{1}{2}
    \int d^3\vec{x}
    \int d^3\vec{x}\,
    \frac{\rho\left( \vec{x} \right) \rho\left( \vec{x}' \right)}{
    \left| \vec{x} - \vec{x}' \right|
    }
    \frac{1}{4\pi\epsilon_0}
\end{align}
Notice that almost all those factors are there.
We can rewrite this in terms of the potential
$\phi(\vec{x})=\frac{1}{4\pi\epsilon_0} \frac{\rho(\vec{x}')}{
\left|\vec{x} - \vec{x}'\right|}$
to get
\begin{align}
    W &=
    \frac{1}{2}
    \int d^3x\,
    \rho\left( \vec{x} \right)
    \phi\left( \vec{x} \right)
\end{align}
And now what we can do is use Poisson's equation to eliminate this.
Using Poisson's equation,
this becomes
\begin{align}
    W &=\frac{1}{2}\epsilon_0 \int d^3x\,
    \underbrace{\phi\left( \vec{x} \right) \nabla^2 \phi(\vec{x})}_{
    \vec{\nabla}\cdot\left\{
    \phi \vec{\nabla} \phi
    \right\}
    -
    \left( \vec{\nabla} \phi \right)^2
    }\\
    &=
    \frac{1}{2}\epsilon_0 \int d^3x\,
    \left( \vec{\nabla}\phi \right)^2
    -
    \frac{1}{2}\epsilon_0
    \underbrace{
    \int d^3x\,
    \vec{\nabla}\cdot\left\{
    \phi \cdot \vec{\nabla} \phi
    \right\}
    }_{
    \int_S dS \phi\left(
    n\cdot \underbrace{\vec{\nabla}\phi}_{=0}
    \right)
    }
\end{align}
where the surface we are integrating over is at infinity.
So then
\begin{align}
    W &=
    \frac{1}{2} \epsilon_0 \int d^3x\,
    \left( \vec{\nabla}\phi \right)^2\\
    &=
    \frac{1}{2}
    \epsilon_0
    \int d^3x\,
    \left|\vec{E}\right|^2
\end{align}
Note even far apart,
there is self energy of the charges.
On the other hand,
you're calculating the work needed (change in potential energy)
to bring charges from infinity into your configuration.
That is a physical thing,
because it's a potnetial energy difference.
That's a convnetion.

The self-energy has a long and interesting history I could give several lectures
on, but we don't have time for that.
The energy density stored in electrostatic field.
\begin{align}
    w &=
    \frac{1}{2}
    \epsilon_0
    \left| \vec{E} \right|^2
\end{align}
So you can think of the energy density per unit energy stored in the electric
field.

\section{Force on a conducting surface}
There are two ways you can compute this.
\begin{enumerate}
    \item Energy change from a virtual displacement.
    \item Directly from the electric field.
\end{enumerate}

First method.
The energy density is $u = \frac{1}{2} \epsilon_0 \left| \vec{E} \right|^2$.
At the surface of conductor $\vec{E} = \sigma/\epsilon_0$
directed normal to the surface.

Consider a small displacement $\Delta x$ in a direction normal to a surface
element $\Delta A$.

Change in energy
\begin{align}
    \Delta W &=
    -\frac{\epsilon_0}{2}
    |\vec{E}|^2 \Delta A \Delta x\\
    &=
    -\frac{1}{2\epsilon_0} \sigma^2 \Delta A \Delta x
\end{align}
So the force is
\begin{align}
    F &=
    -\frac{\Delta W}{\Delta x}\\
    &=
    \frac{\sigma^2 \delta A}{2\epsilon_0}
\end{align}
directed outward from surface.
Force per unit area is
\begin{align}
    f &=
    \frac{\sigma^2}{2\epsilon_0}
\end{align}
outwards.
To get the total force on conductor integrate $f$ over the surface of conductor
bar.

Second method.
The force on an element of area $\Delta A$
is given by
\begin{align}
    \vec{F} &=
    \left( \sigma \Delta A \right)\vec{E}_{ext}
\end{align}
where $\vec{E}_{ext}$ is the electric field after contribution from $\Delta A$
has been removed.
The force is thus
\begin{align}
    F &=
    \left( \sigma \Delta A \right)
    \frac{\sigma}{2\epsilon_0}\\
    &=
    \frac{\sigma^2}{2\epsilon_0} \Delta A
\end{align}
outwards.
The hard part is finding the charge distribution $\sigma$,
but once you've found it,
it's easy to get the force.

\section{Capacitance}
The capacitance is defined as
charge divided by potential
\begin{align}
    C := \frac{Q}{\phi}
\end{align}
An object with large capacitance means it can hold a large amount of charge
without much potential.

Two conductors carrying equal and opposite charge
$+Q$ and $-Q$,
with potential difference $\Delta \phi$
between them,
the capacitance is by definition
\begin{align}
    C &:=
    \frac{Q}{\Delta \phi}.
\end{align}

\section{Boundary Value Problems}
\subsection{Method of images}
Imagine a charge near a conducting surface.

\begin{example}
    A grounded conducting sphere of radius $a$ is centred at the origin.
    A point of charge $q$ is located at point $\vec{y}$.
    Find the potential in the region outside the sphere.
\end{example}
The potential due to $q$ and $q'$ is
\begin{align}
    \phi(\vec{x})
    &=
    \frac{1}{4\pi\epsilon_0}
    \left\{
    \frac{q}{\left| \vec{x} - \vec{y} \right|}
    \right\}
    +
    \frac{1}{4\pi\epsilon_0}
    \left\{
    \frac{q'}{\left| \vec{x} - \vec{y}' \right|}
    \right\}
\end{align}
Let $\hat{n}$ be the unit vector in direction of $\vec{x}$.
Let $\hat{n}'$ be in the direction of $y$.
Then
\begin{align}
    \hpi\left( \vec{x} \right)
    &=
    \frac{1}{4\pi\epsilon_0}\left\{
    \frac{q}{\left| x\hat{n} - y\hat{n}'\right|}
    + \frac{q'}{\left| x\hat{n} - y\hat{n}'\right|}
    \right\}
\end{align}
where
\begin{align}
    |\vec{x}| &= x\\
    |\vec{y}| &= y\\
    |\vec{y}'| &= y'
\end{align}
We need $\phi\left( |\vec{x}| = a \right) = 0$.
Then
\begin{align}
    \phi(\vec{x}=a)
    &=
    \frac{1}{4\pi\epsilon_0}
    \left\{
    \frac{q}{a\left| \hat{n} - \frac{1}{a} \hat{n}'\right|}
    + \frac{q'}{y'\left| \hat{n}' - \frac{1}{y'} \hat{n}\right|}
    \right\}
\end{align}
If we choose
\begin{align}
    \frac{q}{a}
    &=
    - \frac{q'}{y'}
\end{align}
and
\begin{align}
    \frac{y}{a} &= \frac{a}{y'}
\end{align}
then $\phi\left( |\vec{x}| = a \right) = 0$
for all $\hat{n}$ and $\hat{n}'$.
Then the position and magnitude of the image charge is
\begin{align}
    q' &= -\frac{a}{y}q\\
    y' &= \frac{a^2}{y}
\end{align}


\lab{The Discrete Fourier Transform}{The Discrete Fourier Transform}
\objective{The analysis of periodic functions has many applications in pure and applied mathematics, especially in settings dealing with sound waves.
The Fourier transform provides a way to analyze such periodic functions.
In this lab, we introduce how to work with digital audio signals in Python, implement the discrete Fourier transform, and use the Fourier transform to detect the frequencies present in a given sound wave.
We strongly recommend completing the exercises in a Jupyter Notebook.}

\section*{Digital Audio Signals} % ============================================

Sound waves have two important characteristics: \emph{frequency}, which determines the pitch of the sound, and \emph{intensity} or \emph{amplitude}, which determines the volume of the sound.
Computers use \emph{digital audio signals} to approximate sound waves.
These signals have two key components: \emph{sample rate}, which relates to the frequency of sound waves, and \emph{samples}, which measure the amplitude of sound waves at a specific instant in time.

To see why the sample rate is necessary, consider an array with samples from a sound wave.
The sound wave can be arbitrarily stretched or compressed to make a variety of sounds.
If compressed, the sound becomes shorter and has a higher pitch.
Similarly, the same set of samples with a lower sample rate becomes stretched and has a lower pitch.
% This is shown by Figure \ref{fig:DFT_comp_wave}.

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.425\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/tada.pdf}
    \caption{The plot of \texttt{tada.wav}.}
    \label{fig:DFT_tada}
\end{subfigure}
\qquad
\begin{subfigure}{.425\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/fast_tada.pdf}
    \caption{Compressed plot of \texttt{tada.wav}.}
    \label{fig:DFT_fasttada}
\end{subfigure}
\caption{Plots of the same set of samples from a sound wave with varying sample rates.
The plot on the left is the plot of the samples with the original sample rate.
The sample rate of the plot on the right has been doubled, resulting in a compression of the actual sound when played back.}
\label{fig:DFT_comp_wave}
\end{figure}

Given the rate at which a set of samples is taken, the wave can be reconstructed exactly as it was recorded.
In most applications, this sample rate is measured in \emph{Hertz} (Hz), the number of samples taken per second.
% \footnote{Hertz is ``cycles per second'' and has units $1/s$.}
The standard rate for high quality audio is $44100$ equally spaced samples per second, or $44.1$ kHz.

\subsection*{Wave File Format} % ----------------------------------------------

One of the most common audio file formats across operating systems is the \emph{wave} format, also called \texttt{wav} after its file extension.
SciPy has built-in tools to read and create \texttt{wav} files.
To read a \texttt{wav} file, use \li{scipy.io.wavfile.read()}.
This function returns the signal's sample rate and its samples.

\begin{lstlisting}
# Read from the sound file.
>>> from scipy.io import wavfile
>>> rate, samples = wavfile.read("tada.wav")
\end{lstlisting}

Sound waves can be visualized by plotting time against the amplitude of the sound, as in Figure \ref{fig:DFT_comp_wave}.
% This plots in the \emph{time domain}.
The amplitude of the sound at a given time is just the value of the sample at that time.
Since the sample rate is given in samples per second, the length of the sound wave in seconds is found by dividing the number of samples by the sample rate:
\begin{align}
\label{eq:dft-xaxis-time}
\frac{\mbox{num samples}}{\mbox{sample rate}}
= \frac{\mbox{num samples}}{\mbox{num samples/second}}
= \mbox{second.}
\end{align}

\begin{problem} % SoundWave.__init__(), SoundWave.plot().
Write a \li{SoundWave} class for storing digital audio signals.
\begin{enumerate}
\item The constructor should accept an integer sample rate and an array of samples.
Store each input as an attribute.
\item Write a method that plots the stored sound wave.
Use \eqref{eq:dft-xaxis-time} to correctly label the $x$-axis in terms of seconds, and set the $y$-axis limits to $[-32768,32767]$ (the reason for this is discussed in the next section).
\end{enumerate}
Use SciPy to read \texttt{tada.wav}, then instantiate a corresponding \li{SoundWave} object and display its plot.
Compare your plot to Figure \ref{fig:DFT_tada}.
\label{prob:fft-SoundWave-init}
\end{problem}

\subsection*{Scaling} % -------------------------------------------------------

To write to a \texttt{wav} file, use \li{scipy.io.wavfile.write()}.
This function accepts the name of the file to write to, the sample rate, and the array of samples as parameters.

\begin{lstlisting}
>>> import numpy as np

# Write a 2-second random sound wave sampled at a rate of 44100 Hz.
>>> samples = np.random.randint(-32768, 32767, 88200, dtype=np.int16)
>>> wavfile.write("white_noise.wav", 44100, samples)
\end{lstlisting}

For \li{scipy.io.wavfile.write()} to correctly create a \texttt{wav} file, the samples must be one of four numerical datatypes: 32-bit floating point (\li{np.float32}), 32-bit integers (\li{np.int32}), 16-bit integers (\li{np.int16}), or 8-bit unsigned integers (\li{np.uint8}).
If samples of a different type are passed into the function, it may still write a file, but the sound will likely be distorted in some way.
In this lab, we only work with 16-bit integer samples, unless otherwise specified.

A 16-bit integer is an integer between $-32768$ and $32767$, inclusive.
If the elements of an array of samples are not all within this range, the samples must be scaled before writing to a file: multiply the samples by 32767 (the largest number in the 16-bit range) and divide by the largest sample magnitude.
This ensures the most accurate representation of the sound and sets it to full volume.
\begin{align}
\label{eq:dft-scale-samples}
\texttt{np.int16}\left(\left(\frac{\mbox{original samples}}{\mbox{$\max$(|original samples|)}}\right) \times 32767\right) =\mbox{scaled samples}
\end{align}

Because 16-bit integers can only store numbers within a certain range, it is important to multiply the original samples by the largest number in the 16-bit range \emph{after} dividing by the largest sample magnitude.
Otherwise, the results of the multiplication may be outside the range of integers that can be represented, causing overflow errors.
Also, samples may sometimes contain complex values, especially after some processing.
Make sure to scale and export only the real part (use the \li{real} attribute of the array).

\begin{info} % Embedding sound in a Jupyter Notebook
The IPython API includes a tool for embedding sounds in a Jupyter Notebook.
The function \li{IPython.display.Audio()} accepts either a file name or a sample rate (\li{rate}) and an array of samples (\li{data}); calling the function generates an interactive music player in the Notebook.
\begin{center}
\includegraphics[width=\textwidth]{figures/notebook_audio.pdf}
\end{center}
% See \href{https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html?highlight=display.Audio}{\texttt{https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html}} for more info.
\end{info}

\begin{warn}
\textbf{Turn the volume down before listening to any of the sounds in this lab.}
%, especially if you are using headphones.
\end{warn}

\begin{problem} % SoundWave.export().
Add a method to the \li{SoundWave} class that accepts a file name and a boolean \li{force}.
Write to the specified file using the stored sample rate and the array of samples.
If the array of samples does not have \li{np.int16} as its data type, or if \li{force} is \li{True}, scale the samples as in \eqref{eq:dft-scale-samples} before writing the file.

Use your method to create two new files that contains the same sound as \texttt{tada.wav}: one without scaling, and one with scaling (use \li{force=True}).
Use \li{IPython.display.Audio()} to display \texttt{tada.wav} and the new files.
All three files should sound identical, except the scaled file should be louder than the other two.
\end{problem}

\section*{Generating Sounds} % ================================================

Sinusoidal waves correspond to pure frequencies, like a single note on the piano.
Recall that the function $\sin(x)$ has a period of $2\pi$.
To create a specific tone for $1$ second, we sample from the sinusoid with period $1$,
\[
f(x) = \sin(2\pi xk),
\]
where $k$ is the desired frequency.
According to \eqref{eq:dft-xaxis-time}, generating a sound that lasts for $s$ seconds at a sample rate $r$ requires $rs$ equally spaced samples in the interval $[0, s]$. % (these are the $x$ values).

\begin{problem} % Generate simple tones.
Write a function that accepts floats $k$ and $s$.
Create a \li{SoundWave} instance containing a tone with frequency $k$ that lasts for $s$ seconds.
Use a sample rate of $r = 44100$.

The following table shows some frequencies that correspond to common notes.
Octaves of these notes are obtained by doubling or halving these frequencies.
\begin{center}
\begin{tabular}{c|c}
Note & Frequency (Hz) \\
\hline
A & $440$ \\
B & $493.88$ \\
C & $523.25$ \\
D & $587.33$ \\
E & $659.25$ \\
F & $698.46$ \\
G & $783.99$ \\
A & $880$ \\
\end{tabular}
\end{center}
Use your function to generate an A tone lasting for 2 seconds.
\label{prob:generate_note}
\end{problem}

\begin{problem} % Make a chord and a changing sound.
Digital audio signals can be combined by addition or concatenation.
Adding samples overlays tones so they play simultaneously; concatenated samples plays one set of samples after the other with no overlap.

\begin{enumerate}
\item Implement the \li{__add__()} magic method for the \li{SoundWave} class so that if \li{A} and \li{B} are \li{SoundWave} instances, \li{A + B} creates a new \li{SoundWave} object whose samples are the element-wise sum of the samples from \li{A} and \li{B}.
Raise a \li{ValueError} if the sample arrays from \li{A} and \li{B} are not the same length.

Use your method to generate a three-second A minor chord (A, C, and E together).

\item Implement the \li{__rshift__()} magic method\footnote{The \lstinline[basicstyle=\footnotesize\ttfamily]{>>} operator is a \emph{bitwise shift operator} and is usually reserved for operating on binary numbers.} for the \li{SoundWave} class so that if \li{A} and \li{B} are \li{SoundWave} instances, \li{A >> B} creates a new \li{SoundWave} object whose samples are the concatenation of the samples from \li{A}, then the samples from \li{B}.
Raise a \li{ValueError} if the sample rates from the two objects are not equal.
\\(Hint: \li{np.concatenate()}, \li{np.hstack()}, and/or \li{np.append()} may be useful.)

Use your method to generate the arpeggio A $\rightarrow$ C $\rightarrow$ E, where each pitch lasts one second.

\end{enumerate}
Consider using these two methods to produce elementary versions of some simple tunes.
\label{prob:minor_chord_example}
\end{problem}

\section*{The Discrete Fourier Transform} % ===================================

As with the chords generated above, all sound waves are sums of varying amounts of different frequencies (pitches).
In the case of the discrete samples $\f = \begin{bmatrix}f_0 & f_1 & \cdots & f_{n-1}\end{bmatrix}\trp$ that we have worked with thus far, each $f_i$ gives information about the amplitude of the sound wave at a specific instant in time.
However, sometimes it is useful to find out what frequencies are present in the sound wave and in what amount.

We can write the sound wave sample as a sum
\begin{align}
\f = \sum_{k=0}^{n-1} c_k {\w_n}^{(k)},
\end{align}
where $\{\w_n^{(k)}\}_{k=0}^{n-1}$, called the \emph{discrete Fourier basis}, represents various frequencies.
The coefficients $c_k$ represent the amount of each frequency present in the sound wave.

The \emph{discrete Fourier transform (DFT)} is a linear transformation that takes $\f$ and finds the coefficients  $\c = \begin{bmatrix}c_0 & c_1 & \cdots & c_{n-1}\end{bmatrix}\trp$ needed to write $\f$ in this frequency basis.
Later in the lab, we will convert the index $k$ to a value in Hertz to find out what frequency $c_k$ corresponds to.

Because the sample $\f$ was generated by taking $n$ evenly spaced samples of the sound wave, we  generate the basis $\{\w_n^{(k)}\}_{k=0}^{n-1}$ by taking $n$ evenly spaced samples of the frequencies represented by the oscillating functions $\{e^{-2 \pi i k t  / n}\}_{k = 0}^{n-1}$.
(Note that $i = \sqrt{-1}$, the imaginary unit, is represented as  \li{1j} in Python).
This yields
\begin{align}
\w_n^{(k)} = \begin{bmatrix}\omega_n^0 & \omega_n^{-k} & \cdots & \omega_n^{-(n-1)k}\end{bmatrix}\trp,
\end{align}
where $\omega_n = e^{2 \pi i / n}$.

The DFT is then represented by the change of basis matrix
\begin{align}
F_{n} = \frac{1}{n} \begin{bmatrix}\w_n^0 & \w_n^1 & \w_n^2 & \cdots & \w_n^{n-1} \end{bmatrix}
= \frac{1}{n}\left[\begin{array}{*{5}c}
1 & 1 & 1 & \cdots & 1 \\
1 & \omega_n^{-1} & \omega_n^{-2} & \cdots & \omega_n^{-(n-1)} \\
1 & \omega_n^{-2} & \omega_n^{-4} & \cdots & \omega_n^{-2(n-1)} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
1 & \omega_n^{-(n-1)} & \omega_n^{-2(n-1)} & \cdots & \omega_n^{-(n-1)^2} \\
 \end{array}\right],
 \label{eq:dft-matrix}
\end{align}

and we can take the DFT of $f$ by calculating
\begin{align}
\c = F_n \f.
\label{eq:dft}
\end{align}

Note that the DFT depends on the number of samples $n$, since the discrete Fourier basis we use depends on the number of samples taken.
The larger $n$ is, the closer the frequencies approximated by the DFT will be to the actual frequencies present in the sound wave.

\begin{warn}
There are several different conventions for defining the DFT.
For example, instead of \eqref{eq:dft}, \li{scipy.fftpack.fft()} uses the formula
\begin{align*}
\c = n F_n \f,
\end{align*}
where $F_n$ is as given \eqref{eq:dft-matrix}.
Denoting this version of the DFT as $\hat{F}_n\f = \hat{\c}$, we have $nF_n = \hat{F}_n$ and $n\c = \hat{\c}$.
The conversion is easy, but it is very important to be aware of which convention a particular implementation of the DFT uses.
\end{warn}

\begin{problem} % naive DFT.
Write a function that accepts an array $\f$ of samples.
Use \ref{eq:dft} to calculate the coefficients $\c$ of the DFT of $\f$. Include the $1/n$ scaling in front of the sum.

Test your implementation on small, random arrays against \li{scipy.fftpack.fft()}, scaling your output $\c$ to match SciPy's output $\hat{\c}$.
Once your function is working, try to optimize it so that the entire array of coefficients is calculated in the one line.
\\(Hint: Use array broadcasting.)
\label{prob:dft-naive}
\end{problem}

\subsection*{The Fast Fourier Transform} % ------------------------------------

Calculating the DFT of a vector of $n$ samples using only (\ref{eq:dft}) is at least $O(n^2)$, which is incredibly slow for realistic sound waves.
Fortunately, due to its inherent symmetry, the DFT can be implemented as a recursive algorithm by separating the computation into even and odd indices.
This method of calculating the DFT is called the \emph{fast Fourier transform} (FFT) and runs in $O(n\log{n})$ time.

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{simple\_fft}{$\f$, $N$}
    \Procedure{split}{$\g$}
        \State $n \gets \size{\g}$
        \If{$n \le N$}
            \label{step:fft-base-case-a}
            \State \pseudoli{return} $n F_n \g$
                \Comment{Use the function from Problem \ref{prob:dft-naive} for small enough $\g$.}
                \label{step:fft-base-case-b}
        \Else{}
            \State \li{even} $\gets$ \li{SPLIT}$(\g_{::2})$
                \Comment{Get the DFT of every other entry of $\g$, starting from $0$.}
            \State \li{odd} $\gets$ \li{SPLIT}$(\g_{1::2})$
                \Comment{Get the DFT of every other entry of $\g$, starting from $1$.}
            \State $\z \gets\zeros{n}$
            \For{$k = 0,1,\ldots,n-1$}
                \Comment{Calculate the exponential parts of the sum.}
                \label{step:fft-loop-a}
                \State $z_k \gets e^{-2\pi i k / n}$
                    % \Comment{This part can be done with array broadcasting.}
                    \label{step:fft-loop-b}
            \EndFor
            \State{$m \gets n\,$\li{//}$\,2$}
                \Comment{Get the middle index for $\z$ (\li{//} is integer division).}
            \State \pseudoli{return} $[$\li{even}$\ +\ \z_{:m} \odot $\li{odd}$,\ $\li{even}$\ +\ \z_{m:} \odot $\li{odd}$]$
                \Comment{Concatenate two arrays of length $m$.}
                % \Comment{$\odot$ is component-wise multiplication.}
                \label{step:fft-concatenate}
        \EndIf
    \EndProcedure
    \State \pseudoli{return} \li{SPLIT}$(\f)\,/\,\size{\f}$
\EndProcedure
\end{algorithmic}
\caption{The fast Fourier transform for arrays with $2^a$ entries for some $a \in \mathbb{N}$.}
\label{alg:FFT}
\end{algorithm}
Note that the base case in lines \ref{step:fft-base-case-a}--\ref{step:fft-base-case-b} of Algorithm \ref{alg:FFT} results from setting $n = 1$ in \eqref{eq:dft}, yielding the single coefficient $c_0 = g_0$.
The $\odot$ in line \ref{step:fft-concatenate} indicates the component-wise product
\begin{align*}
\f\odot\g = \begin{bmatrix}f_0g_0 & f_1g_1 & \cdots & f_{n-1}g_{n-1}\end{bmatrix}\trp,
\end{align*}
which is also called the \emph{Hadamard product} of $\f$ and $\g$.

This algorithm performs significantly better than the na\"ive implementation of the DFT, but the simple version described in Algorithm \ref{alg:FFT} only works if the number of original samples is exactly a power of $2$.
SciPy's FFT routines avoid this problem by padding the sample array with zeros until the size is a power of $2$, then executing the remainder of the algorithm from there. % TODO: This might not be true!! See https://stackoverflow.com/questions/6855169/convolution-computations-in-numpy-scipy.
Of course, SciPy also uses various other tricks to further speed up the computation.

\begin{problem} % simple FFT.
Write a function that accepts an array $\f$ of $n$ samples where $n$ is a power of $2$.
Use Algorithm \ref{alg:FFT} to calculate the DFT of $\f$.
\\(Hint: eliminate the loop in lines \ref{step:fft-loop-a}--\ref{step:fft-loop-b} with \li{np.arange()} and array broadcasting, and use \li{np.concatenate()} or \li{np.hstack()} for the concatenation in line \ref{step:fft-concatenate}.)

Test your implementation on random arrays against \li{scipy.fftpack.fft()}, scaling your output $\c$ to match SciPy's output $\hat{\c}$.
Time your function from Problem \ref{prob:dft-naive}, this function, and SciPy's function on an array with $8192$ entries.
\\(Hint: Use \li{<p<\%time>p>} in Jupyter Notebook to time a single line of code.)
\label{prob:fft-simple}
\end{problem}

\section*{Visualizing the DFT} % ==============================================

The graph of the DFT of a sound wave is useful in a variety of applications.
While the graph of the sound in the time domain gives information about the amplitude (volume) of a sound wave at a given time, the graph of the DFT shows which frequencies (pitches) are present in the sound wave.
Plotting a sound's DFT is referred to as plotting in the \emph{frequency domain}.

As a simple example, the single-tone notes generated by the function in Problem \ref{prob:generate_note} contain only one frequency.
For instance, Figure \ref{fig:dft_a} graphs the DFT of an A tone.
However, this plot shows two frequency spikes, despite there being only one frequency present in the actual sound.
This is due to symmetries inherent to the DFT; for frequency detection, the second half of the plot can be ignored as in Figure \ref{fig:dft_a_half}.

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/dft_a.pdf}
    \caption{The DFT of an A tone with symmetries.}
    \label{fig:dft_a}
\end{subfigure}
% \qquad
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/dft_a_half.pdf}
    \caption{The DFT of an A tone without symmetries.}
    \label{fig:dft_a_half}
\end{subfigure}
\caption{Plots of the DFT with and without symmetries.
Notice that the $x$-axis of the symmetrical plot on the left goes up to $44100$ (the sample rate of the sound wave) while the $x$-axis of the non-symmetric plot on the right goes up to only $22050$ (half the sample rate).
Also notice that the spikes occur at $440$ Hz and $43660$ Hz (which is $44100 - 440$).}
\end{figure}

The DFT of a more complicated sound wave has many frequencies, each of which corresponds to a different tone present in the sound wave.
The magnitude of the coefficients indicates a frequency's influence in the sound wave; a greater magnitude means that the frequency is more influential.
% See Figure \ref{fig:dft_tada} for an example.

\begin{figure}[H]
\centering
\includegraphics[width=.7\textwidth]{figures/dft_tada.pdf}
\caption{The discrete Fourier transform of \texttt{tada.wav}.
Each spike in the graph corresponds to a frequency present in the sound wave.
Since the sample rate of \texttt{tada.wav} is $22050$ Hz, the plot of its DFT without symmetries only goes up to $11025$ Hz, half of its sample rate.}
\label{fig:dft_tada}
\end{figure}

\subsection*{Plotting Frequencies} % ------------------------------------------

Since the DFT represents the frequency domain, the $x$-axis of a plot of the DFT should be in terms of Hertz, which has units $1/s$.
In other words, to plot the magnitudes of the Fourier coefficients against the correct frequencies, we must convert the frequency index $k$ of each $c_k$ to Hertz.
This can be done by multiplying the index by the sample rate and dividing by the number of samples:
\begin{align}
\label{eq:fft-cycles-conversion}
\frac{k}{\mbox{num samples}} \times \frac{\mbox{num samples}}{\mbox{second}}
= \frac{k}{\mbox{second}}.
% = k\ \mbox{Hz}
\end{align}
In other words, $kr/n = v$, where $r$ is the sample rate, $n$ is the number of samples, and $v$ is the resulting frequency.

% Plotting the DFT of a sound wave without any other considerations results in an $x$-axis that corresponds to the index of the coefficients in the DFT, not their frequencies.
% To label the $x$-axis with the frequencies measured in Hz, or cycles per second, the units must be converted.
% The sample rate is measured in samples per second, so dividing the frequency index by the number of samples and multiplying by the sample rate results in cycles per second, or Hz.

\begin{problem} % plotting the DFT.
Modify your \li{SoundWave} plotting method from Problem \ref{prob:fft-SoundWave-init} so that it accepts a boolean defaulting to \li{False}.
If the boolean is \li{True}, take the DFT of the stored samples and plot---in a new subplot---the frequencies present on the $x$-axis and the magnitudes of those frequencies (use \li{np.<<abs>>()} to compute the magnitude) on the $y$-axis.
Only display the first half of the plot (as in Figures \ref{fig:dft_a_half} and \ref{fig:dft_a_half}), and use \eqref{eq:fft-cycles-conversion} to adjust the $x$-axis so that it correctly shows the frequencies in Hertz.
Use SciPy to calculate the DFT.

Display the DFT plots of the A tone and the A minor chord from Problem \ref{prob:minor_chord_example}.
Compare your results to Figures \ref{fig:dft_a} and \ref{fig:dft_chord}.
\end{problem}

\begin{figure}[H] % DFT of the minor chord.
\centering
\includegraphics[width=.7\textwidth]{figures/dft_chord.pdf}
\caption{The DFT of the A minor chord.}
\label{fig:dft_chord}
\end{figure}

If the frequencies present in a sound are already known before plotting its DFT, the plot may be interesting, but little new information is actually revealed.
Thus, the main applications of the DFT involve sounds in which the frequencies present are unknown.
One application in particular is sound filtering, which will be explored in greater detail in a subsequent lab.
The first step in filtering a sound is determining the frequencies present in that sound by taking its DFT.

Consider the DFT of the A minor chord in Figure \ref{fig:dft_chord}.
This graph shows that there are three main frequencies present in the sound.
To determine what those frequencies are, find which indices of the array of DFT coefficients have the three largest values, then scale these indices the same way as in \eqref{eq:fft-cycles-conversion} to translate the indices to frequencies in Hertz.

\begin{problem} % Mystery chord.
The file \texttt{mystery\_chord.wav} contains an unknown chord.
Use the DFT and the frequency table in Problem \ref{prob:generate_note} to determine the individual notes that are present in the sound.
\\(Hint: \li{np.argsort()} may be useful.)
\end{problem}

\begin{comment}
\newpage
\section*{Additional Material} % ==============================================

\subsection*{Stereo vs Mono} % ------------------------------------------------

Mono has a single channel so sample arrays are one-dimensional; stereo sounds have channels for left and right speakers, so sample arrays are 2-dimensional: one COLUMN for the left channel, and one COLUMN for the right channel.

\subsection*{Improvements to the SoundWave Class} % ---------------------------

\subsection*{Other Implementations of the FFT} % ------------------------------

FASTEST FOURIER TRANSFORM IN THE WEST!!!
\url{https://en.wikipedia.org/wiki/FFTW}
\url{http://www.fftw.org/}

\end{comment}