\lab{Convolution and Filtering}{Convolution and Filtering}
\objective{The Fourier transform reveals information in the frequency domain about signals and images that might not be apparent in the usual time (sound) or spatial (image) domain.
% It provides an extremely powerful tool to analyze difficult problems simply.
In this lab, we use the discrete Fourier transform to efficiently convolve sound signals and filter out some types of unwanted noise from both sounds and images.
This lab is a continuation of the Discrete Fourier Transform lab and should be completed in the same Jupyter Notebook.}

\section*{Convolution} % ======================================================

Mixing two sounds signals---a common procedure in signal processing and analysis---is usually done through a \emph{discrete convolution}.
% Recall that sound is modeled with discrete samples of a sound wave in rapid succession.
Given two periodic sound sample vectors $\f$ and $\g$ of length $n$, the discrete convolution of $\f$ and $\g$ is a vector of length $n$ where the $k$th component is given by
\begin{align}
(\f \ast \g)_k = \sum_{j=0}^{n-1} f_{k-j}g_j,\qquad k = 0,1,2,\dots,n-1.
\label{eq:naive-convolve}
\end{align}

Since audio needs to be sampled frequently to create smooth playback, a recording of a song can contain tens of millions of samples; even a one-minute signal has $2,646,000$ samples if it is recorded at the standard rate of $44,100$ samples per second ($44,100$ Hz).
The na\"ive method of using the sum in \eqref{eq:naive-convolve} $n$ times is $O(n^2)$, which is often too computationally expensive for convolutions of this size.

Fortunately, the discrete Fourier transform (DFT) can be used compute convolutions efficiently.
The \emph{finite convolution theorem} states that the Fourier transform of a convolution is the element-wise product of Fourier transforms:
\begin{align}
F_n(\f \ast \g) = n(F_n\f)\odot (F_n\g).
\label{eq:convolution-theorem}
\end{align}
In other words, convolution in the time domain is equivalent to component-wise multiplication in the frequency domain.
Here $F_n$ is the DFT on $\mathbb{R}^n$, $\ast$ is discrete convolution, and $\odot$ is component-wise multiplication. % (the \emph{Hadamard product}).
Thus, the convolution of $\f$ and $\g$ can be computed by
\begin{align}
\f \ast \g = nF_n^{-1}((F_n\f)\odot (F_n\g)),
\label{eq:fft-convolution}
\end{align}
where $F_n^{-1}$ is the \emph{inverse discrete Fourier transform} (IDFT).
The fast Fourier transform (FFT) puts the cost of \eqref{eq:fft-convolution} at $O(n\log{n})$, a huge improvement over the na\"ive method.

\begin{info} % Discard imaginaries after IDFT
Although individual samples are real numbers, results of the IDFT may have small complex components due to rounding errors.
These complex components can be safely discarded by taking only the real part of the output of the IDFT.

\begin{lstlisting}
>>> import numpy
>>> from scipy.fftpack import fft, ifft # Fast DFT and IDFT functions.

>>> f = np.random.random(2048)
>>> f_dft_idft = ifft(fft(f)).real      # Keep only the real part.
>>> np.allclose(f, f_dft_idft)          # Check that IDFT(DFT(f)) = f.
<<True>>
\end{lstlisting}
\end{info}

\begin{warn} % SciPy conventions
SciPy uses a different convention to define the DFT and IDFT than this and the previous lab, resulting in a slightly different form of the convolution theorem.
Writing SciPy's DFT as $\hat{F}_n$ and its IDFT as $\hat{F}_n^{-1}$, we have $\hat{F}_n = n F_n$, so \eqref{eq:fft-convolution} becomes
\begin{align}
\f \ast \g = \hat{F}_n^{-1}((\hat{F}_n\f)\odot (\hat{F}_n\g)),
\label{eq:fft-convolution-scipy}
\end{align}
without a factor of $n$.
Use \eqref{eq:fft-convolution-scipy}, not \eqref{eq:fft-convolution}, when using \li{fft()} and \li{ifft()} from \li{scipy.fftpack}.
\end{warn}

\subsection*{Circular Convolution} % ------------------------------------------

The definition \eqref{eq:naive-convolve} and the identity \eqref{eq:fft-convolution} require $\f$ and $\g$ to be periodic vectors.
However, the convolution $\f \ast \g$ can always be computed by simply treating each vector as periodic.
The convolution of two raw sample vectors is therefore called the \emph{periodic} or \emph{circular convolution}.
This strategy mixes sounds from the end of each signal with sounds at the beginning of each signal.

\begin{problem} % Circular convolution.
\item Implement the \li{__mul__()} magic method for the \li{SoundWave} class so that if \li{A} and \li{B} are \li{SoundWave} instances, \li{A * B} creates a new \li{SoundWave} object whose samples are the circular convolution of the samples from \li{A} and \li{B}.
If the samples from \li{A} and \li{B} are not the same length, append zeros to the shorter array to make them the same length before convolving.
Use \li{scipy.fftpack} and \eqref{eq:fft-convolution-scipy} to compute the convolution, and raise a \li{ValueError} if the sample rates from \li{A} and \li{B} are not equal.

A circular convolution creates an interesting effect on a signal when convolved with a segment of white noise: the sound loops seamlessly from the end back to the beginning.
To see this, generate two seconds of white noise (at the same sample rate as \texttt{tada.wav}) with the following code.
\begin{lstlisting}
>>> rate = 22050        # Create 2 seconds of white noise at a given rate.
>>> white_noise = np.random.randint(-32767, 32767, rate*2, dtype=np.int16)
\end{lstlisting}
Next, convolve \texttt{tada.wav} with the white noise.
Finally, use the \li{>>} operator to append the convolution result to itself.
This final signal sounds the same from beginning to end, even though it is the concatenation of two signals.
\end{problem}

\subsection*{Linear Convolution} % --------------------------------------------

Although circular convolutions can give interesting results, most common sound mixtures do not combine sounds at the beginning of one signal with sounds at the end of another.
Whereas circular convolution assumes that the samples represent a full period of a periodic function, \emph{linear convolution} aims to combine non-periodic discrete signals in a way that prevents the beginnings and endings from interacting.
Given two samples with lengths $n$ and $m$, the simplest way to achieve this is to pad both samples with zeros so that they each have length $n+m-1$, compute the convolution of these larger arrays, and take the first $n+m-1$ entries of that convolution.

\begin{problem} % Linear convolution.
\item Implement the \li{__pow__()} magic method for the \li{SoundWave} class so that if \li{A} and \li{B} are \li{SoundWave} instances, \li{A ** B} creates a new \li{SoundWave} object whose samples are the linear convolution of the samples from \li{A} and \li{B}.
Raise a \li{ValueError} if the sample rates from \li{A} and \li{B} are not equal.

Because \li{scipy.fftpack} performs best when the length of the inputs is a power of $2$, start by computing the smallest $2^a$ such that $2^a \ge n + m - 1$, where $a\in\mathbb{N}$ and $n$ and $m$ are the number of samples from \li{A} and \li{B}, respectively.
Append zeros to each sample so that they each have $2^a$ entries, then compute the convolution of these padded samples using \eqref{eq:fft-convolution-scipy}.
Use only the first $n + m - 1$ entries of this convolution as the samples of the returned \li{SoundWave} object.

To test your method, read \texttt{CGC.wav} and \texttt{GCG.wav}.
Time (separately) the convolution of these signals with \li{SoundWave.__pow__()} and with \li{scipy.signal.fftconvolve()}.
Compare the results by listening to the original and convolved signals.
\label{prob:fft-linear-convolution}
\end{problem}

\begin{problem} % Convolve the balloon pop with Chopin.
Clapping in a large room with an echo produces a sound that resonates in the room for up to several seconds.
This echoing sound is referred to as the \emph{impulse response} of the room, and is a way of approximating the acoustics of a room.
When the sound of a single instrument in a carpeted room is convolved with the impulse response from a concert hall, the new signal sounds as if the instrument is being played in the concert hall.

The file \texttt{chopin.wav} contains a short clip of a piano being played in a room with little or no echo, and \texttt{balloon.wav} is a recording of a balloon being popped in a room with a substantial echo (the impulse).
Use your method from Problem \ref{prob:fft-linear-convolution} or \li{scipy.signal.fftconvolve()} to compute the linear convolution of \texttt{chopin.wav} and \texttt{balloon.wav}.
% Listen to the new signal; there should be echo in the piano recording.
\end{problem}

\section*{Filtering Frequencies with the DFT} % ===============================

The DFT also provides a way to clean a signal by altering some of its frequencies.
Consider \texttt{noisy1.wav}, a noisy recording of a short voice clip.
The time-domain plot of the signal only shows that the signal has a lot of static.
On the other hand, the signal's DFT suggests that the static may be the result of some concentrated noise between about $1250$--$2600$ Hz.
Removing these frequencies could result in a much cleaner signal.

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.53\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/noisy1.pdf}
\end{subfigure}%
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/noisy1_dft.pdf}
\end{subfigure}
\caption{The time-domain plot (left) and DFT (right) of \texttt{noisy1.wav}.}
\label{fig:fft-noisy-signal}
\end{figure}

To implement this idea, recall that the $k$th entry of the DFT array $\c = F_n \f$ corresponds to the frequency $v = k r / n$ in Hertz, where $r$ is the sample rate and $n$ is the number of samples.
Hence, the DFT entry $c_k$ corresponding to a given frequency $v$ in Hertz has index $k = v n / r$, rounded to an integer if needed.
In addition, since the DFT is symmetric, $c_{n-k}$ also corresponds to this frequency. %; any changes to $c_k$ must also be done to $c_{n-k}$ in order to alter the signal.
This suggests a strategy for filtering out an unwanted interval of frequencies $[v_\text{low},v_\text{high}]$ from a signal:
\begin{enumerate}
\item Compute the integer indices $k_\text{low}$ and $k_\text{high}$ corresponding to $v_\text{low}$ and $v_\text{high}$, respectively.
\item Set the entries of the signal's DFT from $k_\text{low}$ to $k_\text{high}$ and from $n-k_\text{high}$ to $n-k_{low}$ to zero, effectively removing those frequencies from the signal.
\item Take the IDFT of the modified DFT to obtain the cleaned signal.
\end{enumerate}
Using this strategy to filter \texttt{noisy1.wav} results in a much cleaner signal. %, shown in Figure \ref{fig:fft-noisy-cleaned}.
However, any ``good'' frequencies in the affected range are also removed, which may decrease the overall sound quality.
The goal, then, is to remove only as many frequencies as necessary.

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.53\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/noisy1_clean.pdf}
\end{subfigure}%
\begin{subfigure}{.47\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/noisy1_clean_dft.pdf}
\end{subfigure}
\caption{The time-domain plot (left) and DFT (right) of \texttt{noisy1.wav} after being cleaned.}
\label{fig:fft-noisy-cleaned}
\end{figure}

\begin{problem} % Clean up a noisy signal (The only thing we have to fear...)
\label{prob:fft-filter-frequencies}
Add a method to the \li{SoundWave} class that accepts two frequencies $v_\text{low}$ and $v_\text{high}$ in Hertz.
Compute the DFT of the stored samples and zero out the frequencies in the range $[v_\text{low}, v_\text{high}]$ (remember to account for the symmetry DFT).
Take the IDFT of the altered array and store it as the sample array.
% \\(Hint: be sure to sure the indices of the DFT array are integers.)

Test your method by cleaning \texttt{noisy1.wav}, then clean \texttt{noisy2.wav}, which also has some artificial noise that obscures the intended sound.
\\(Hint: plot the DFT of \texttt{noisy2.wav} to determine which frequencies to eliminate.)
\end{problem}

A digital audio signal made of a single sample vector with is called \emph{monoaural} or \emph{mono}.
When several sample vectors with the same sample rate and number of samples are combined into a matrix, the overall signal is called \emph{stereophonic} or \emph{stereo}.
This allows multiple speakers to each play one \emph{channel}---one of the original sample vectors---simultaneously.
``Stereo'' usually means there are two channels, but there may be any number of channels ($5.1$ surround sound, for instance, has five).

% \begin{warn}
Most stereo sounds are read as $n\times m$ matrices, where $n$ is the number of samples and $m$ is the number of channels (i.e., each column is a channel).
However, some functions, including Jupyter's embedding tool \li{IPython.display.Audio()}, receive stereo signals as $m\times n$ matrices (each row is a channel).
Be aware that both conventions are common.
% \end{warn}

% When a digital audio signal is played on a computer, the signal is sent to a speaker, which vibrates, producing sound waves.
% When multiple speakers are used, they can all produce the same signal or they can produce different signals.
% When there is only one signal, the sound is \emph{monoaural}, or \emph{mono}.  When speakers produce more than one signal, the overall signal is \emph{stereophonic}, or \emph{stereo}.
% Usually stereo means two, but there may be any number of signals ($5.1$ surround sound, for instance, has $5$).

% Most sounds are multi-dimensional, but can be treated similarly to mono sounds.
% The first and second columns of the array sample correspond to the signals for the left and right speaker respectively.

\begin{problem}
\label{prob:fft-vuvuzelas}
During the 2010 World Cup in South Africa, large plastic horns called vuvuzelas were blown excessively throughout the games.
Broadcasting organizations faced difficulties with their programs due to the incessant noise level.
Eventually, audio filtering techniques were used to cancel out the sound of the vuvuzela, which has a frequency of around $200$--$500$ Hz.

The file \li{vuvuzela.wav}\footnote{See \url{https://www.youtube.com/watch?v=g_0NoBKWCT8}.} is a stereo sound with two channels.
Use your function from Problem \ref{prob:fft-filter-frequencies} to clean the sound clip by filtering out the vuvuzela frequencies in each channel.
Recombine the two cleaned samples.
\end{problem}

% \begin{info} % Notch filtering and other filters
% The cleaning strategy of Problems \ref{prob:fft-filter-frequencies} and \ref{prob:fft-vuvuzelas} is called \emph{notch filtering} and is a special case of \emph{band-stop filtering}.
% \end{info}

\subsection*{The Two-dimensional Discrete Fourier Transform} % ----------------

The DFT can be easily extended to any number of dimensions.
Computationally, the problem reduces to performing the usual one-dimensional DFT iteratively along each of the dimensions.
For example, to compute the two-dimensional DFT of an $m \times n$ matrix, calculate the usual DFT of each of the $n$ columns, then take the DFT of each of the $m$ rows of the resulting matrix.
Calculating the two-dimensional IDFT is done in a similar fashion, but in reverse order: first calculate the IDFT of the rows, then the IDFT of the resulting columns.

\begin{lstlisting}
>>> from scipy.fftpack import fft2, ifft2

>>> A = np.random.random((10,10))
>>> A_dft = fft2(A)                 # Calculate the 2d DFT of A.
>>> A_dft_ifft = ifft2(A_dft).real  # Calculate the 2d IDFT.
>>> np.allclose(A, A_dft_ifft)
<<True>>
\end{lstlisting}

% The DFT for matrices is useful for a variety of image applications, including cleaning, compression, enhancement, and edge detection.
Just as the one-dimensional DFT can be used to remove noise in sounds, its two-dimensional counterpart can be used to remove ``noise'' in images.
The procedure is similar to the filtering technique in Problems \ref{prob:fft-filter-frequencies} and \ref{prob:fft-vuvuzelas}: take the two-dimensional DFT of the image matrix, modify certain entries of the DFT matrix to remove unwanted frequencies, then take the IDFT to get a cleaner version of the original image.
This strategy makes the fairly strong assumption that the noise in the image is periodic and corresponds to certain frequencies.
While this may seem like an unlikely scenario, it does actually occur in many digital images---for an example, try taking a picture of a computer screen with a digital camera.

\begin{figure}[H]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/blurry_face.png}
    \caption{The original blurry image.}
    \label{fig:blurry_face}
\end{subfigure}
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/blurry_fft.png}
    \caption{The DFT of the original image.}
    \label{fig:blurry_fft}
\end{subfigure}
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/improved_face.png}
    \caption{The improved image.}
    \label{fig:improved_face}
\end{subfigure}
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/covered_fft.png}
    \caption{The DFT of the improved image.}
    \label{fig:covered_fft}
\end{subfigure}
\caption{To remove noise from an image, take the DFT of the image and replace the abnormalities with values more consistent with the rest of the DFT.
Notice that the new image is less noisy, but only slightly.
This is because only some of the abnormalities in the DFT were changed; in order to further decrease the noise, we would need to further alter the DFT.}
\label{fig:image_fft}
\end{figure}

To begin cleaning an image with the DFT, take the two-dimensional DFT of the image matrix.
Identify \emph{spikes}---abnormally high frequency values that may be causing the noise---in the image DFT by plotting the log of the magnitudes of the Fourier coefficients.
With \li{cmap="gray"}, spikes show up as bright spots.
See Figures \ref{fig:blurry_face}--\ref{fig:blurry_fft}.

\begin{lstlisting}
# Read the image.
>>> import imageio
>>> im = imageio.read("noisy_face.png")

# Plot the log magnitude of the image's DFT.
>>> im_dft = fft2(image)
>>> plt.imshow(np.log(np.<<abs>>(im_dft)), cmap="gray")
>>> plt.show()
\end{lstlisting}

Instead of setting spike frequencies to zero (as was the case for sounds), replace them with values that are similar to those around them.
There are many ways to do this, but one convention is to simply ``patch'' each spike by setting portions of the DFT matrix to some set value, such as the mean of the DFT array.
See Figure \ref{fig:covered_fft}.

Once the spikes have been covered, take the IDFT of the modified DFT to get a (hopefully cleaner) image.
Notice that Figure \ref{fig:improved_face} still has noise present, but it is a slight improvement over the original.
However, it often suffices to remove some of the noise, even if it is not possible to remove it all with this method.

\begin{problem} % Clean an image.
The file \texttt{license\_plate.png} contains a noisy image of a license plate.
The bottom right corner of the plate has is a sticker with information about the month and year that the vehicle registration was renewed.
However, in its current state, the year is not clearly legible.

Use the two-dimensional DFT to clean up the image enough so that the year in the bottom right corner is legible.
This may require a little trial and error.
\end{problem}

\begin{comment}
\newpage
\section*{Additional Material} % ==============================================

\subsection*{Other Convolution Functions} % -----------------------------------

Talk about the options in \li{np.covolve()}, etc. (circular by default).
Use \li{scipy.ndimage.convolve()} to do circular convolution if needed (but the results must be shifted to match what we do).

\subsection*{Other Sound Filters} % -------------------------------------------

What we did is called a notch filter, blah blah blah...

\subsection{Other Image Filters} % --------------------------------------------

Overview of the following comment?

\end{comment}

\begin{comment} % VERY OLD (and bad) image filtering section ==================

\section*{Image Filters}

Recall that a computer stores an image as a 2-D array of pixel values (i.e., a matrix of intensities).
An image filter is a function that transforms an image by operating on it locally.
That is, to compute the $(ij)$th pixel value in the new image, an image filter uses only the pixels in a small neighborhood around the $(ij)$th pixel in the original image.

In this lab, we use a filter derived from the gradient of an image to find edges in an image.

\subsection*{Convolutions}

One example of an image filter is to \emph{convolve} an image with a filter matrix.
A filter matrix is a matrix whose height and width are relatively small odd numbers.
If the filter matrix is
\[
F = \begin{pmatrix}
f_{-1,-1}&f_{-1,0}&f_{-1,1}\\
f_{0,-1}&f_{0,0}&f_{0,1}\\
f_{1,-1}&f_{1,0}&f_{1,1}
\end{pmatrix},
\]
then the convolution of an image $A$ with $F$ is $A \ast F = (C_{ij})$ where
\begin{equation}\label{equ:convolve}
C_{ij} = \sum_{k=-1}^1 \sum_{\ell=-1}^1 f_{k\ell}A_{i+k,j+\ell}.
\end{equation}
Say $A$ is an $m \times n$ matrix. Here, we take $A_{ij}=0$ when $i \not \in \{1, \ldots m\}$ or $j \not \in \{1, \ldots, n\}$.
The value of $C_{ij}$ is a linear combination of the nearby pixel values, with coefficients given by $F$ (see Figure \ref{fig:convolution}).
In fact, $C_{ij}$ equals the Frobenius inner product of $F$ with the $3 \times 3$ submatrix of $A$ centered at $ij$.

\begin{figure}
\centering
\begin{tikzpicture}
\node[draw, minimum width=12cm, minimum height=
    7cm, ultra thick](outer_rec)[]{};
\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, fill=black!10!](inner_rec)[]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square1)at(-4.75,2.25){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick,fill=black!30!](square2)at(-1.75,-.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square3)at(4.75,.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square4)at(3.25,-1.25){\textbullet};
\draw[step=.5, ultra thin, color=black!50!](-6,-3.5)grid(6,3.5);

\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, ][]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-4.75,2.25){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-1.75,-.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(4.75,.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(3.25,-1.25){};
\end{tikzpicture}
\caption{This diagram illustrates how to convolve an image with a filter.
The light grey rectangle represents the original image $A$, and the dark grey squares are the filter $F$.
The larger rectangle is the image padded with zeros; i.e., all pixel values in the outer white band are 0.
To compute the entry of the convolution matrix $C$ located at a black dot, take the inner product of $F$ with the submatrix of the padded image centered at the dot.}
\label{fig:convolution}
\end{figure}

\subsubsection*{Implementation in NumPy}

Let us write a function that convolves an image with a filter.
You can test this function on the image \li{cameraman.jpg}, which appears in Figure \ref{fig:cameraman}.
The following code loads this image and plots it with matplotlib.

\begin{lstlisting}
>>> image = plt.imread('cameraman.jpg')
>>> plt.imshow(image, cmap = 'gray')
>>> plt.show()
\end{lstlisting}

Here is the function definition and some setup.

\begin{lstlisting}
1. def Filter(image, F):
2.     m, n = image.shape
3.     h, k = F.shape
\end{lstlisting}

To convolve \li{image} with the filter \li{F}, we must first \emph{pad} the array \li{image} with zeros around the edges.
This is because in \eqref{equ:convolve}, entries $A_{ij}$ are set to zero when $i$ or $j$ is out of bounds.
We do this by creating a larger array of zeros, and then making the interior part of the array equal to the original image (see Figure \ref{fig:convolution}).

For example, if the filter is a $3 \times 3$ matrix, then the following code pads the matrix with the appropriate number of zeros.

\begin{lstlisting}
 # Create a larger matrix of zeros
image_pad = np.zeros((m+2, n+2))
# Make the interior of image_pad equal to the original image
image_pad[1:1+m, 1:1+n] = image
\end{lstlisting}

We want to do this in general in our function.  Note that the number of zeros we need to pad our array depends on the size of the filter \li{F}.

\begin{lstlisting}
5.    image_pad = # Create an array of zeros of the appropriate size
6.   # Make the interior of image_pad equal to image
\end{lstlisting}

Finally, we iterate through the image to compute each entry of the convolution matrix.

\begin{lstlisting}
7.    C = np.zeros(image.shape)
8.    for i in range(m):
9.        for j in range(n):
10.            C[i,j] = # Compute C[i, j]
\end{lstlisting}

\subsubsection*{Gaussian Blur}

A \emph{Gaussian blur} is an image filter that operates on an image by convolving with the matrix
\[
G = \frac{1}{159}\begin{pmatrix}
2&4&5&4&2\\
4&9&12&9&4\\
5&12&15&12&5\\
4&9&12&9&4\\
2&4&5&4&2
\end{pmatrix}.
\]

Blurring an image can remove ``noise'', or random variation that is the visual analog of static in a radio signal (and equally undesirable).

\begin{problem}\label{prob:filter}
\leavevmode
Finish writing the function \li{Filter} by filling in lines 5, 6, and 10.  Hint: Note in \ref{equ:convolve}, $C_{ij}$ was calculated by summing from -1 to 1.  This is only the case if the filter \li{F} is $3 \times 3$. A slight modification is needed in the general case.  Test your function on the image \li{cameraman.jpg} using the Gaussian Blur. The result is in Figure \ref{fig:cameraman_blur}.
\end{problem}

\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
\centering
% \includegraphics[width=\textwidth]{figures/cameraman.jpg}
\caption{Unfiltered image.}
\label{fig:cameraman}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
% \includegraphics[width=\textwidth]{figures/cameramanBlur.pdf}
\caption{Image after Gaussian blur is applied.}
\label{fig:cameraman_blur}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
% \includegraphics[width=\textwidth]{figures/edges.pdf}
\caption{Image after the Sobel filter is applied.}
\label{fig:cameraman_edges}
\end{subfigure}
\caption{Here is an example of a Gaussian blur and the Sobel filter applied to an image.
This photo, known as ``cameraman,'' is a standard test image in image processing.
A database of such images can be downloaded from \url{http://www.imageprocessingplace.com/root_files_V3/image_databases.htm}.}
\label{fig:cameraman1}
\end{figure}

\subsection*{Edge Detection}

Automatic detection of edges in an image can be used to segment or sharpen the image.
We find edges with the Sobel filter, which computes the gradient of the image at each pixel.
The magnitude of the gradient tells us the rate of change of the pixel values, and so large magnitudes should
correspond to edges within the image.
The Sobel filter is not a convolution, although it does use convolutions.

We can think of an image as a function from a $2 \times 2$ grid of points to $\mathbb{R}$.
The image maps a pixel location to an intensity.
It does not make sense to define the derivative of this function as a limit because the domain is discrete---a step size $h$ cannot take on arbitrarily small values.
Instead, we \emph{define} the derivative to be the centered difference quotient of the previous section.
That is, we define the derivative in the $x$-direction at the $ij$th pixel to be
\[
\frac{1}{2}A_{i+1, j} - \frac{1}{2}A_{i-1, j}.
\]

We can use a convolution to create a matrix $A_x$ whose $ij$th entry is the derivative of $A$ at the $ij$th entry, in the $x$-direction.
In fact, $A_x = A \ast S$, where
\[
S = \frac{1}{8}
\left[\begin{array}{ccc}
-1 & 0 & 1\\
-2 & 0 & 2\\
-1 & 0 & 1
\end{array}\right].
\]

Note that this convolution takes a weighted average of the $x$-derivatives at $(i, j)$, $(i, j+1)$, and $(i, j-1)$.
The derivative at $(i, j)$ is weighted by 2.
Using a weighted average instead of just the derivative at $(i, j)$ makes the derivative less affected by noise.

Now we can define the Sobel filter.
A Sobel filter applied to an image $A$ results in an array $B = (B_{ij})$ of 0's and 1's, where the 1's trace out the edges in the image.
By definition,
\[
B_{ij} = \left\{
     \begin{array}{ll}
       1 & \text{if}\; \;\|\nabla A(ij)\|_2 > M \\
       0 & \text{otherwise}.
     \end{array}
   \right.
\]
Here, $\nabla A(ij) = ((A \ast S)_{ij}, (A\ast S^T)_{ij})$ is the gradient of $A$ at the $ij$th pixel.
The constant $M$ should be ``sufficiently large'' enough to pick out those pixels with the largest gradient (i.e., those pixels that are part of an edge).
A good choice for $M$ is 4 times the average value of $\|\nabla A(ij)\|_2$ over the whole image $A$.

When the Sobel filter is applied to \li{cameraman.jpg}, we get the image in Figure \ref{fig:cameraman_edges}.
Here, the 1's in $B$ were mapped to ``white'' and the 0's were mapped to ``black.''

\begin{problem}
Write a function that accepts an image as input and applies the Sobel filter to the image.  Test your function on \li{cameraman.jpg}.  Hint: If you want to find the average of a matrix \li{A}, use the function \li{A.mean()}.
\end{problem}

\end{comment}


\section{Spans, Linear Independence and Steinitz Exchange Lemma}
In this section, we shall characterise the properties of the dimension and basis of a vector space.
\begin{definition}[Span of a Family of Vectors]
    Let $V$ be a vector space over $F$ and $S\subset V$.
    We define the span of $S$ to be
    $$\langle S\rangle=\operatorname{span}(S)=\left\{\sum_{i=1}^n\lambda_is_i:n\in\mathbb N,\lambda_i\in F,s_i\in S\right\}$$
\end{definition}
That is, $\langle S\rangle$ consists of all possible (finite) linear combination of elements of $S$.
By convention, we say $\langle \varnothing\rangle=\{0\}$.
Note also that the span of $S$ is essentially the minimal subspace of $V$ containing $S$.
\begin{example}
    1. Take $V=\mathbb R^3$ and
    $$S=\left\{\begin{pmatrix}
        1\\
        0\\
        0
    \end{pmatrix},\begin{pmatrix}
        0\\
        1\\
        2\\
    \end{pmatrix},\begin{pmatrix}
        3\\
        -2\\
        -4
    \end{pmatrix}\right\}$$
    then
    $$\langle S\rangle=\left\{\begin{pmatrix}
        a\\
        b\\
        2b
    \end{pmatrix}:a,b\in\mathbb R\right\}$$
    2. Take $V=\mathbb R^n$ and let $e_i$ be the vector in $V$ that only has $1$ at the $i^{th}$ entry and zero elsewhere, then $\langle \{e_i\}_{i=1}^n\rangle=V$.\\
    3. Let $V=\mathbb R^X$ and $S_x:X\to\mathbb R$ be such that $S_x(y)=1_{x=y}$.
    Then $\langle \{S_x\}_{x\in\mathbb R}\rangle$ are the set of functions $f\in\mathbb R^X$ that has finite support.
\end{example}
\begin{definition}
    Let $V$ be a vector space over $F$ and $S\subset V$.
    We say $S$ spans $V$ if $\langle S\rangle =V$.
\end{definition}
\begin{example}
    Take $V=\mathbb R^2$, then any set of two non-parallel vectors would span $V$.
\end{example}
\begin{definition}
    A vector space $V$ over a field $F$ is finite dimensional if there is a finite $S\subset V$ that spans $V$.
\end{definition}
\begin{example}
    The space $V=\mathbb P[x]$ be the set of polynomials in $\mathbb R$ and $V_n=\mathbb P_n[x]$ be the set of real polynomials with degree at most $n$.
    Then $V_n=\langle\{1,x,\ldots,x^n\}\rangle$ is finite dimensional, but $V$ is not finite dimensional as any finite set of polynomials must be contained in $V_n$ where $n$ is the maximal degree of polynomials in that set.
\end{example}
As $\mathbb N$ is well-ordered, there must be a minimum number of vectors that can possibly span $V$.
We then focus on how to capture this minimality.
\begin{definition}[(Linear) Independence]
    Let $V$ be a vector space over $F$.
    We say $\{v_1,\ldots,v_n\}\subset V$ are (linearly) independent (or is a free family) if for any $\lambda_1,\ldots,\lambda_n\in F$
    $$\sum_{i=1}^n\lambda_iv_i=0\implies\forall i,\lambda_i=0$$
    On the other hand, this set is not linearly independent if there exists $\lambda_1,\ldots,\lambda_n\in F$ not all zero such that $\sum_{i=1}^n\lambda_iv_i=0$.
\end{definition}
\begin{example}
    Let $V=\mathbb R^3$ and
    $$v_1=(1,0,0)^\top,v_2=(0,1,0)^\top,v_3=(1,1,0)^\top,v_4=(0,1,1)^\top$$
    Then $\{v_1,v_2\}$ is linearly independent.
    Note that $v_3\in\langle\{v_1,v_2\}\rangle$, so $\{v_1,v_2,v_3\}$ is not linearly independent.
    On the other hand, $v_4\notin\langle\{v_1,v_2\}\rangle$, which as one can verify means that $\{v_1,v_2,v_4\}$ is linearly independent.
\end{example}
\begin{remark}
    If the family $\{v_i\}_{1\le i\le n}$ is linearly independent, then none of $v_i$ is zero.
\end{remark}
\begin{definition}[Basis]
    A subset $S\subset V$ is a basis if it is linearly independent and $\langle S\rangle=V$.
\end{definition}
\begin{remark}
    When $S$ spans $V$, we say that $S$ is a generating family of $V$.
    So a basis is just a linearly independent generating family.
\end{remark}
\begin{example}
    1. Take $V=\mathbb R^n$, then the family $\{e_i\}_{1\le i\le n}$ where $e_i$ is the vector having $1$ at $i^{th}$ entry and zero otherwise is a basis.\\
    2. Take $V=\mathbb C$ over $\mathbb C$, then $\{a\}$ is a basis for any $a\neq 0$.\\
    3. Take also $V=\mathbb C$ but over $\mathbb R$, then $\{1,i\}$ is a basis.\\
    4. Take $V=\mathbb P[x]$ be the set of polynomials in $\mathbb R$ and $S=\{x^n:n\ge 0\}$.
    Then $S$ is a basis.
    Worth noting that $|S|=\infty$ in this case.
\end{example}
\begin{lemma}
    If $V$ is a vector space over $F$, then $\{v_1,\ldots,v_n\}$ is a basis of $V$ if and only if for any vector $v\in V$, there is a unique decomposition
    $$v=\sum_{i=1}^n\lambda_iv_i$$
\end{lemma}
\begin{remark}
    If the conditions are true, then the tuple $(\lambda_1,\ldots,\lambda_n)$ (ordered via the ordering one chose on $v_i$) is called the coordinate of $v$ in the basis $(v_i)$.
\end{remark}
\begin{proof}
    Trivial.
\end{proof}
\begin{lemma}
    If $S$ is a finite set that spans $V$, then a subset of $S$ is a basis of $V$.
\end{lemma}
\begin{proof}
    If $S$ is independent, then we are done.
    Otherwise, there is some $\lambda\neq 0$ and $\lambda_w$ such that there is $v\in S$ with
    $$\lambda v+\sum_{w\in S\setminus\{v\}}\lambda_ww=0\implies v=\frac{1}{\lambda}\sum_{w\in S\setminus\{v\}}\lambda_ww\in\langle S\setminus\{v\}\rangle$$
    Therefore $S\setminus\{v\}$ also spans $V$.
    We can repeat this process and, by the well-ordering of $\mathbb N$, will reach a basis.
\end{proof}
\begin{theorem}[Steinitz Exchange Lemma]\label{steinitz}
    Let $V$ be a finite dimensional vector space over $F$, $\{v_1,\ldots,v_m\}\subset V$ linearly independent, $\{w_1,\ldots,w_n\}\subset V$ a generating set, then:\\
    1. $m\le n$.\\
    2. Up to relabeling, $\{v_1,\ldots,v_m,w_{m+1},\ldots,w_n\}$ spans $V$.
\end{theorem}
\begin{proof}
    Suppose $\{v_1,\ldots,v_l,w_{l+1},\ldots,w_n\}$ spans $V$ for some $l<m$, then
    $$\exists\alpha_i,\beta_i\in F, v_{l+1}=\sum_{i\le l}\alpha_iv_i+\sum_{i>l}\beta_iw_i$$
    But $\{v_i\}$ is linearly independent, so one of the $\beta_i$ is nonzero.
    By relabelling $\beta_{l+1}\neq 0$, then $w_{l+1}\in\langle\{v_1,\ldots,v_l,v_{l+1},w_{l+2}\ldots,w_n\}\rangle$, therefore the set of vectors $\{v_1,\ldots,v_l,v_{l+1},w_{l+2}\ldots,w_n\}$ also spans $V$.
    The theorem is then obvious by induction.
\end{proof}
\begin{corollary}\label{dim_well_defined}
    Let $V$ be a finite dimensional vector space, then any two bases of $V$ have the same cardinality.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
This corollary allows us to give a proper definition of the dimension of a vector space.
Before we step right into that, another corollary of Theorem \ref{steinitz} can help us to capture important properties of a finite dimensional vector space that will come in handy in further discussions of basis.
\begin{corollary}
    Let $V$ be a vector space with $\dim V=n$, then:\\
    1. Any independent set of vectors has size at most $n$.
    The size is exactly $n$ iff this set is a basis.\\
    2. Any spanning set has size at least $n$.
    The size is exactly $n$ iff this set is a basis.
\end{corollary}
\begin{proof}
    Obvious.
\end{proof}


\subsection{Cosine functions as activation function}
Let us use a simple example to motivate the spectral Barron space. Consider a bounded domain $\Omega\subset \mathbb
R^d$ and a real function $u\in L^1(\Omega)$.
Recall the Fourier transform of $u\in L^1(\mathbb{R})$ in Definition~\ref{def:fourier1} and \ref{def:fourier2}.
This gives the following integral representation of $u$ in terms of the cosine function
\begin{equation}
 \label{eq:reint}
u(x)=Re\int_{\mathbb{R}^d} e^{2\pi i\omega\cdot x} \hat u(\omega)d\omega
= \int_{\mathbb{R}^d}\cos (2\pi (\omega\cdot x + b(\omega))) |\hat u(\omega)|d\omega,
\end{equation}
where $ \hat u(\omega)= e^{2\pi ib(\omega)}|\hat u(\omega)|$. Let
\begin{equation}
 \label{eq:2}
g(x, \omega) = \cos(2\pi (\omega\cdot x + b(\omega)))\quad \mbox{ and }\quad
\rho(\omega)= |\hat u(\omega)| .
\end{equation}
Thus,
\begin{equation}
\label{int-rep}
u(x)= \int_{\mathbb{R}^d}g(x,\omega) \rho(\omega)d\omega,
\end{equation}
If
$$
\int_{\mathbb R^d} |\hat u(\omega)|d\omega <\infty,
$$
then $\|\rho\|_{L^1}<\infty$. By applying the Lemma \ref{lem:sample},
there exist $\omega_i\in \mathbb R^d$
such that
\begin{equation}
  \label{eq:3}
\|u-u_N\|_{0,\Omega}\le N^{-1/2}\|\hat u\|_{L^1(\mathbb R^d)}.
\end{equation}
where
\begin{equation}\label{cosfn}
u_N(x) = {\|\hat u\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N \cos (2\pi(\omega_i\cdot x + b(\omega_i)))
\end{equation}
More generally, we consider the approximation property in $H^m$-norm.
By \eqref{eq:reint},
\begin{equation}
\partial^\alpha u(x)= \int_{\mathbb{R}^d} \cos^{|\alpha|}(2\pi (\omega\cdot x + b(\omega)))\omega^\alpha |\hat u(\omega)|d\omega,  \quad \forall\ |\alpha|\le m.
\end{equation}
For any positive integer $m$, let
\begin{equation} \label{eq:gm}
g_m(x,\omega)= {\cos (2\pi (\omega\cdot x + b(\omega)))\over  (1+ \|\omega\|)^m}\quad \mbox{and}\quad \rho_m(\omega)= (1+ \|\omega\|)^m|\hat u(\omega) |,
\end{equation}
where
$$
\| \rho_m\|_{L^1(\mathbb R^d)}=\int_{\mathbb R^d} (1+ \|\omega\|)^m|\hat u(\omega) | d\omega<\infty.
$$
Then, $\displaystyle u(x)=\int_{\mathbb R^d} g_m(x,\omega)\rho_m d\omega = \| \rho_m\|_{L^1(\mathbb R^d)}\mathbb{E}g_m(x,\omega)$. Define
\begin{equation}
u_N(x) = {\|\rho_m\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N g_m(x,\omega_i)
= {\|\rho_m\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N {\cos (2\pi(\omega_i\cdot x + b(\omega_i)))\over (1+\|\omega_i\|)^m}.
\end{equation}
It holds that
$$
\partial^\alpha (u(x) - u_N(x))={\|\rho_m\|_{L^1(\mathbb R^d)}\over N}\sum_{i=1}^N \mathbb{E} \partial^\alpha (g_m(x,\omega) -  g_m(x,\omega_i)).
$$
By Lemma \ref{MC},
\begin{align}
\mathbb{E}_N \sum_{|\alpha|\le m}\|\partial^\alpha (u(x) - u_N(x))\|_{0, \Omega}^2
&\le
\|\rho_m \|_{L^1(\mathbb R^d)}^2\mathbb{E}_N \sum_{|\alpha|\le m}\frac{1}{N^2}\sum_{i=1}^N \left (\mathbb{E} \partial^\alpha (g_m(x,\omega) -  g_m(x,\omega_i))\right )^2
\\
&\le
{\|\rho_m \|_{L^1(\mathbb R^d)}^2\over N}  \sum_{|\alpha|\le m}\mathbb{E} \left (\partial^\alpha g_m(x,\omega)\right)^2
\end{align}
Note that the definitions of $g_m$ and $\rho_m$ in \eqref{eq:gm} guarantee that
$$
|\partial^\alpha g_m(x,\omega)|\le 1.
$$
Thus,
$$
\mathbb{E}_N \sum_{|\alpha|\le m}\|\partial^\alpha (u(x) - u_N(x))\|_{0, \Omega}^2 \lesssim {\|\rho_m \|_{L^1(\mathbb R^d)}^2\over N}.
$$
This implies that there exist
$\omega_i\in \mathbb R^d$ such that
\begin{equation}
\label{cosHm}
\|u-u_N\|_{H^m(\Omega)}\lesssim N^{-1/2}\int_{\mathbb{R}^d} (1+ \|\omega\|)^m|\hat u(\omega) | d\omega.
\end{equation}

Given $v\in L^2(\Omega)$,   consider all the possible extension $v_E:
\mathbb{R}^d \mapsto \mathbb{R}$ with $v_E |_{\Omega} = v$ and define
the spectral  Barron norm for any $s\ge 1$:
	\begin{equation}\label{barron-norm0}
	\|v\|_{B^{s}(\Omega)} = \inf_{v_E |_{\Omega} = v} \int_{\mathbb{R}^d}(1+\|\omega\|)^s|\hat{v}_E(\omega)|d\omega
	\end{equation}
and  spectral  Barron space
\begin{equation}
  \label{Barron}
	B^{s}(\Omega) = \{v\in L^2(\Omega): \|v\|_{B^{s}(\Omega)}<\infty\}.
\end{equation}

In summary, we have
\begin{equation}
\|u-u_N\|_{H^m(\Omega)}\lesssim N^{-1/2}  \|u\|_{B^{m}(\Omega)},
\end{equation}
where $u_N$ is defined in \eqref{cosfn}.

Specifically, we will consider the problem of approximating a function with bounded Barron norm \eqref{barron-norm0} in the Sobolev space $H^m(\Omega)$. Our first step will be to prove a lemma showing that the Sobolev norm is bounded by the Barron norm.

\begin{lemma}\label{smoothness-lemma}
 Let $m \geq 0$ be an integer and $\Omega\subset \mathbb{R}^d$ a bounded domain. Then for any Schwartz function $v$, we have
 \begin{equation}\label{embend}
 \|v\|_{W^{m,\infty}(\Omega)} \lesssim \|v\|_{{B}^m(\Omega)} \lesssim  \|v\|_{H^{m + {d\over 2}+\epsilon}(\Omega)},
 \end{equation}
 where $\epsilon$ is positive.
\end{lemma}
\begin{proof}
Recall the inverse Fourier transform in Definition \ref{def:fourier1}
$$
v(x)=\int \hat{v}(\omega) e^{2 \pi i\omega \cdot x} d \omega.
$$
For any $|\alpha|\le m$,
$$
|\partial^\alpha v|=|(2\pi)^{|\alpha|}\int \hat{v}(\omega) \omega^\alpha e^{2 \pi i\omega \cdot x} d \omega|\le |(2\pi)^{|\alpha|}\int \hat{v}(\omega) \|\omega\|^\alpha e^{2 \pi i\omega \cdot x} d \omega| \lesssim \|v\|_{B^m(\Omega)},
$$
which proves $ \|v\|_{W^{m,\infty}(\Omega)} \lesssim \|v\|_{{B}^m(\Omega)}$.

\iffalse
 Let $\chi$ be a Schwartz function satisfying $\chi(x) = 1$ for $x\in \Omega$. Such a function exists because $\Omega$ is bounded. Let $\alpha$ be any multi-index with $|\alpha|\leq m$. Then we have
 \begin{equation}
  \|D^\alpha v\|_{L^2(\Omega)} \leq \|\chi D^\alpha u\|_{L^2(\mathbb{R}^d)} \leq \|\hat\chi * \widehat{D^\alpha v}\|_{L^2(\mathbb{R}^d)}
 \end{equation}
 Now we use Young's inequality to obtain
 \begin{equation}
  \|\hat\chi * \widehat{D^\alpha v}\|_{L^2(\mathbb{R}^d)} \leq \|\hat\chi\|_{L^2(\mathbb{R}^d)}\|\widehat{D^\alpha v}\|_{L^1(\mathbb{R}^d)} \leq \|\hat\chi\|_{L^2(\mathbb{R}^d)}\|v\|_{\mathcal{B}^m(\Omega)}.
 \end{equation}
 Combining this over all multi-indices $\alpha$, we get
 \begin{equation}
  \|u\|_{H^m(\Omega)} \lesssim \|v\|_{\mathcal{B}^m(\Omega)},
 \end{equation}
 as desired. \fi

 A version of the second inequality
  in \eqref{embend} and its proof
can be found in \cite{barron1993universal}. Below is a proof, by
definition and Cauchy-Schwarz
  inequality,
\begin{align}
\|v\|_{B^m(\Omega)} =& \inf_{v_E |_{\Omega} = v} \left(\int_{\mathbb{R}^d}(1+\|\omega\|)^m|\hat{v}_E(\omega)|d\omega \right)^2
\\
\le &  \int_{\mathbb{R}^d}(1+\|\omega\|)^{-d - 2\epsilon}d\omega  \inf_{v_E |_{\Omega} = v}
\int_{\mathbb{R}^d}(1+\|\omega\|)^{d + 2m + 2\epsilon} |\hat{v}_E(\omega)|^2d\omega
\\
\lesssim &  \inf_{v_E |_{\Omega} = v}
\int_{\mathbb{R}^d}(1+\|\omega\|)^{d + 2m + 2\epsilon} |\hat{v}_E(\omega)|^2d\omega
\lesssim \|v\|_{H^{m + {d\over 2}+\epsilon}(\Omega)}.
\end{align}

\end{proof}


\newpage
\chapter*{Summary of Probability Theory I}
\addcontentsline{toc}{chapter}{Summary of Probability Theory I}

{\scriptsize

\begin{framed}
\begin{tabular}{rcl}
SET SUMMARY\\ \\
$\{a_1,a_2,\dots,a_n\}$& $-$& a set containing the elements, $a_1,a_2,\dots,a_n$.\\
$a\in A$ &$-$& $a$ is an element of the set $A$.\\
$A\subseteq B$ &$-$& the set $A$ is a subset of $B$.\\
$A\cup B$ &$-$& ``union'', meaning the set of all elements which are in $A$ or $B$, \\
& & \ \ or both.\\
$A\cap B$ &$-$& ``intersection'', meaning the set of all elements in both $A$ and $B$.\\
$\{\}$ or $\emptyset$ &$-$& empty set.\\
$\Omega $ &$-$& universal set.  \\
$A^c $ &$-$& the complement of $A$, meaning the set of all elements in $\Omega$,\\
& & \ \ the universal set, which are not in $A$.\\
\end{tabular}
\end{framed}

\begin{framed}
EXPERIMENT SUMMARY\\

\begin{tabular}{rcl}
Experiment &$-$& an activity producing distinct outcomes.\\
$\Omega$ &$-$& set of all outcomes of the experiment.\\
$\omega$ &$-$& an individual outcome in $\Omega$, called a simple event.\\
$A\subseteq \Omega$ &$-$& a subset $A$ of $\Omega$ is an event.\\
Trial &$-$& one performance of an experiment resulting in 1 outcome.\\
\end{tabular}
\end{framed}

\begin{framed}
PROBABILITY SUMMARY

\medskip

Axioms:
\begin{enumerate}
\item If $A\subseteq \Omega$ then $0\leq P(A)\leq 1$ and $P(\Omega)=1$.
\item If $A$, $B$ are disjoint events, then $P(A\cup B)=P(A)+P(B)$.

[This is true only when $A$ and $B$ are disjoint.]
\item If $A_1,A_2,\dots$ are disjoint then $P(A_1\cup A_2
\cup\dots)=P(A_1)+P(A_2)+\dots$
\end{enumerate}
Rules:
$$P(A^c)\;=\;1-P(A)$$
$$P(A\cup B)\;=\;P(A)+P(B)-P(A\cap B) \qquad [\textrm{always true}]$$
\end{framed}

\newpage
\begin{framed}
CONDITIONAL PROBABILITY SUMMARY\\

$P(A|B)$ means the probability that $A$ occurs given that $B$ has
occurred.

$$P(A|B)\;=\;\frac{P(A\cap B)}{P(B)}\;=\;\frac{P(A)P(B|A)}{P(B)}\quad \textrm{if}\quad P(B)\neq0$$

$$P(B|A)\;=\;\frac{P(A\cap B)}{P(A)}\;=\;\frac{P(B)P(A|B)}{P(A)}\quad \textrm{if}\quad P(A)\neq 0$$

Conditional probabilities obey the 4 axioms of probability.
\end{framed}

\begin{framed}
DISCRETE RANDOM VARIABLE SUMMARY\\

Probability mass function $$f(x)=P(X=x_i)$$
Distribution function $$F(x)=\sum_{x_i\leq x}f(x_i)$$

\begin{center}
{\small
{\renewcommand{\arraystretch}{1.25}
\begin{tabular}{|c|c|p{3.0cm}|p{5.0cm}|}
\multicolumn{1}{c}{\bf Random  Variable} & \multicolumn{1}{c}{\bf Possible  Values}
&\multicolumn{1}{c}{\bf Probabilities} &\multicolumn{1}{c}{\bf Modeled situations} \\\hline
Discrete  uniform&$\{x_1,x_2,\dots,x_k\}$&$P(X=x_i)=\frac{1}{k}$&Situations with $k$ equally likely values.  Parameter: $k$.\\\hline
$\bernoulli(\theta)$&$\{0,1\}$&$P(X=0)=1-\theta$\newline$P(X=1)=\theta$&Situations with only 2 outcomes, coded 1 for success and 0 for failure.\newline Parameter:\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Geometric($\theta$)&$\{1,2,3,\dots\}$&$P(X=x)\newline=(1-\theta)^{x-1}\theta$& Situations where you count the number of trials until the first success in a sequence of independent trails with a constant probability of success. \newline Parameter:\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Binomial($n,\theta$)&$\{0,1,2,\dots,n\}$&$P(X=x)$\newline $=\displaystyle\binom{n}{x}\theta^x(1-\theta)^{n-x}$&Situations where you count the number of success in $n$ trials where each trial is independent and there is a constant probability of success.\newline Parameters: $n \in \{1,2,\ldots\}$;\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Poisson($\lambda$)&$\{0,1,2,\dots\}$&$P(X=x)\newline=\displaystyle \frac{\lambda^xe^{-\lambda}}{x!}$&Situations where you count the number of events in a continuum where the events occur one at a time and are independent of one another.\newline Parameter: $\lambda$= rate $\in (0,\infty)$.\\\hline
\end{tabular}}
}
\end{center}
\end{framed}

\begin{framed}
CONTINUOUS RANDOM VARIABLES: NOTATION\\

$f(x)$: Probability density function (PDF)
\begin{itemize}
\item$f(x)\;\geq\;0$
\item Areas underneath $f(x)$ measure probabilities.
\end{itemize}

$F(x)$: Distribution function (DF)
\begin{itemize}
\item $0\;\leq\;\ F(x)\;\leq \;1$
\item $F(x)\;= \;P(X\leq x)$ is a probability
\item $F^{\prime}(x)\;=\;f(x)$ for every $x$ where $f(x)$ is continuous
\item $F(x)\;=\;\displaystyle\int^x_{-\infty}f(v)dv$
\item $P(a<X\leq b)\;=\;F(b)-F(a)\;=\;\displaystyle \int^b_af(v)dv$
\end{itemize}
\end{framed}

\begin{framed}
\textbf{Expectation} of a function $g(X)$ of a random variable $X$ is defined as:
\[
E(g(X))\; =\;
\begin{cases}
\displaystyle \sum_x g(x) f(x) & \text{if $X$ is a discrete RV}\\[12pt]
\displaystyle \int_{-\infty}^{\infty} g(x) f(x) dx & \text{if $X$ is a continuous RV}
\end{cases}
\]
\begin{center}
Some Common Expectations
{\renewcommand{\arraystretch}{1.75}
%\begin{tabular}{|c|c|p{3.0cm}|p{5.0cm}|}
\begin{tabular}{|c|p{6.0cm}|p{6.0cm}|}
\hline
$g(x)$ & definition & also known as \\
\hline
$x$    & $E(X)$     & Expectation, Population Mean or First Moment of $X$\\
%       &            & or First Moment of $X$\\
\hline
$(x-E(X))^2$ & $V(X):=E((X-E(X))^2)$ \newline $\quad =E(X^2)-(E(X))^2$ & Variance or Population Variance of $X$\\
\hline
$e^{\imath t x}$ & $\phi_X(t) := E\left( e^{\imath t X}\right)$ & Characteristic Function (CF) of $X$\\
\hline
$x^k$  & $E(X^k) =\frac{1}{\imath^k} \left[\frac{d^k \phi_X(t)}{dt^k}\right]_{t=0} $  & $k$-th Moment of $X$\\
\hline
\end{tabular}
}
\end{center}
\end{framed}

\begin{table}[htb]
\centering
{\small
\begin{tabular}{| l | l |}
\hline
Symbol & Meaning\\ \hline
$\BB{1}_{A}(x)$ & Indicator or set membership function that returns $1$ if $x \in A$ and $0$ otherwise\\
$\Rz^d := (-\infty,\infty)^d$ & $d$-dimensional Real Space \\
\hline
\end{tabular}
}
\caption{Symbol Table: Probability and Statistics \label{T:SymbTableProbStats}}
\end{table}

%\section*{Summary of Random Variables}\label{S:SummaryRVs}
\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|c|}%|c}
\hline
Model &PDF or PMF&Mean&Variance\\ \hline%&{\tt MGF}\\
%$\pointmass(\theta)$&$\BB{1}_{\{\theta\}}(x)$&$\theta$&$0$\\%&$e^{\theta t}$\\
$\bernoulli(\theta)$ &$\theta^x(1-\theta)^{1-x} \BB{1}_{\{0,1\}}(x)$&$\theta$&$\theta(1-\theta)$\\%&$\theta e^t+(1-\theta)$\\
$\binomial(n,\theta)$&$(^n_{\theta})\theta^x(1-\theta)^{n-x} \BB{1}_{\{0,1,\ldots,n\}}(x)$&$n \theta$&$n \theta(1-\theta)$\\%&$(\theta e^t+(1-\theta))^n$\\
$\geometric(\theta)$ &$\theta(1-\theta)^{x} \BB{1}_{\Zz_+}(x)$&$ \frac{1}{\theta}-1$&$\frac{1-\theta}{\theta^2}$\\%&$\frac{\theta e^t}{1-(1-\theta)e^t}(t<-log(1-\theta))$\\
$\poisson(\lambda)$&$\frac{\lambda^xe^{-\lambda}}{x!} \BB{1}_{\Zz_+}(x)$&$\lambda$&$\lambda$\\%&$e^{\lambda(e^t-1)}$\\
$\uniform(\theta_1,\theta_2)$&$\BB{1}_{[\theta_1,\theta_2]}(x)/(\theta_2-\theta_1)$&$\frac{\theta_1+\theta_2}{2}$&$\frac{(\theta_2-\theta_1)^2}{12}$\\%&$\frac{e^{\theta_2 t}-e^{\theta_1 t}}{(\theta_2-\theta_1)t}$\\
$\exponential(\lambda)$&$\lambda e^{-\lambda x}$&$\lambda^{-1}$&$\lambda^{-2}$\\%&$\frac{1}{1-\frac{1}{\lambda} t}(t<\lambda)$\\
$\normal(\mu,\sigma^2)$&$\frac{1}{\sigma\sqrt{2\pi}}e^{(x-\mu)^2/(2\sigma^2)}$&$\mu$&$\sigma^2$\\%&$exp\{ut+\frac{\sigma^2t^2}{2}\}$\\
%$\gammA(\alpha,\beta)$&$\frac{\beta^{\alpha}}{\Gamma(\alpha)}{x^{\alpha-1}e^{-\beta x}}$&$\alpha/\beta$&$\alpha/\beta^2$\\%&$(\frac{1}{1-\beta t})^{\alpha}(t<1/\beta)$\\
%$\betA(\alpha,\beta)$ & $\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$ & $\frac{\alpha}{\alpha+\beta}$ & $\frac{\alpha\beta}{(\alpha+\beta)^2 (\alpha+\beta+1)} $ \\%& $ 1+\Sigma^\infty_{k+1}(\Pi^{k-1}_{r=0}\frac{\alpha+r}{\alpha+\beta+r})\frac{t^k}{k\!}$\\
%$t_v$ & $\frac{\Gamma((v+1)/2)}{\Gamma(v/2)}  \frac{1}{(1+x^2/v)^{(v+1)/2}$ &0 (if $v>1$ ) & $\frac{v}{v-2}$  (if $v>2$) & does not exist \\
% $\chi^2_p$&$\frac{1}{\Gamma(p/2)2^{p/2}}x^{(p-2)-1}e^{-x/2}$&$p$&$2p$\\ \hline%&$(\frac{1}{1-2t})^{p/2}(t<1/2)$\\
\hline
\end{tabular}
\caption{Random Variables with PDF and PMF (using indicator function), Mean and Variance}
%\label{tab:}
\end{table}

\begin{table}[htb]
\centering
{\small
\begin{tabular}{| l | l |}
\hline
Symbol & Meaning\\ \hline
$A = \{ \star, \circ, \bullet \}$& $A$ is a set containing the elements $\star$, $\circ$ and $\bullet$ \\
$\circ \in A$ & $\circ$ belongs to $A$ or $\circ$ is an element of $A$\\
$A \ni \circ$ & $\circ$ belongs to $A$ or $\circ$ is an element of $A$\\
$\odot \notin A$& $\odot$ does not belong to $A$ \\
$\# A$& Size of the set $A$, for e.g.~$\#\{ \star, \circ, \bullet, \odot \}=4$\\
$\Nz$& The set of natural numbers $\{1,2,3,\ldots\}$\\
$\Zz$&The set of integers $\{\ldots,-3,-2,-1,0,1,2,3,\ldots\}$\\
$\Zz_+$& The set of non-negative integers $\{0,1,2,3,\ldots\}$\\
%$\Rz$& The set of real numbers\\
$\emptyset$&Empty set or the collection of nothing or $\{\}$\\
$A \subset B$ & $A$ is a subset of $B$ or $A$ is contained by $B$, e.g.~$A=\{\circ\}, B=\{\bullet\}$\\
$A \supset B$& $A$ is a superset of $B$ or $A$ contains $B$ e.g.~$A=\{\circ, \star,\bullet\}, B=\{\circ, \bullet\}$ \\
$A=B$ & $A$ equals $B$, i.e.~$A \subset B$ and  $B \subset A$\\
$Q \implies R$ & Statement $Q$ implies statement $R$ or If $Q$ then $R$ \\
$Q \iff R$ & $Q \implies R$ and $R \implies Q$ \\
$\{x: x \text{ satisfies property } R \}$& The set of all $x$ such that $x$ satisfies property $R$\\
$A \cup B$& $A$ union $B$, i.e.~$\{x: x\in A \text{ or } x \in B\}$\\
$A \cap B$& $A$ intersection $B$, i.e.~$\{x: x\in A \text{ and } x \in B\}$\\
$A \setminus B $& $A$ minus $B$, i.e.~$\{x: x\in A \text{ and } x \notin B\}$\\
$A:=B$& $A$ is equal to $B$ by definition\\
$A=:B$& $B$ is equal to $A$ by definition\\
$A^c$& $A$ complement, i.e.~$\{x: x\in U, \text{ the universal set, but } x \notin A\}$\\
$A_1 \times A_2 \times \cdots \times A_m$ & The $m$-product set $\{(a_1,a_2,\ldots,a_{m}): a_1 \in A_1, a_2 \in A_2, \ldots, a_{m} \in A_m \}$ \\
$A^m$ & The $m$-product set $\{(a_1,a_2,\ldots,a_{m}): a_1 \in A, a_2 \in A, \ldots, a_{m} \in A \}$ \\
$f := f(x)=y:\Xz \to \Yz$ & A function $f$ from domain $\Xz$ to range $\Yz$ \\
$f^{[-1]}(y)$ & Inverse image of $y$ \\
$f^{[-1]} := f^{[-1]}(y \in \Yz) = X \subset \Xz$ & Inverse of $f$ \\
%$\Zz:=\{\ldots,-2,-1,0,1,2,\ldots\}$ & Integers \\
$a<b$ or $a \leq b$ & $a$ is less than $b$ or $a$ is less than or equal to $b$ \\
$a>b$ or $a \geq b$ & $a$ is greater than $b$ or $a$ is greater than or equal to $b$ \\
$\Qz$ & Rational numbers \\
$(x,y)$ & the open interval $(x,y)$, i.e.~$\{r: x < r < y\}$ \\
$[x,y]$ & the closed interval $(x,y)$, i.e.~$\{r: x \leq r \leq y\}$ \\
$(x,y]$ & the half-open interval $(x,y]$, i.e.~$\{r: x < r \leq y\}$ \\
$[x,y)$ & the half-open interval $[x,y)$, i.e.~$\{r: x \leq r < y\}$ \\
$\Rz := (-\infty,\infty)$ & Real numbers, i.e.~$\{r: -\infty < r <  \infty \}$ \\
$\Rz_+ := [0,\infty)$ & Real numbers, i.e.~$\{r: 0 \leq r <  \infty \}$ \\
$\Rz_{>0} := (0,\infty)$ & Real numbers, i.e.~$\{r: 0 < r <  \infty \}$ \\
\hline
\end{tabular}
}
\caption{Symbol Table: Sets and Numbers \label{T:SummarySymbTableSets}}
\end{table}

\begin{table}[htb]
\centering
{\small
{\renewcommand{\arraystretch}{1.75}
\begin{tabular}{| p{6.0cm}| p{10.0cm}|}
\hline
Symbol & Meaning\\ \hline
$\BB{1}_{A}(x)$ & Indicator or set membership function that returns $1$ if $x \in A$ and $0$ otherwise\\
$\Rz^d := (-\infty,\infty)^d$ & $d$-dimensional Real Space \\
\rv~ & random vector\\
$F_{X,Y}(x,y)$ & Joint distribution function (JDF) of the \rv~ $(X,Y)$\\
$F_{X,Y}(x,y)$ & Joint cumulative distribution function (JCDF) of the \rv~ $(X,Y)$ --- same as JDF \\
$f_{X,Y}(x,y)$ & Joint probability mass function (JPMF) of the discrete \rv~ $(X,Y)$\\
$\mathcal{S}_{X,Y}$\newline$=\{(x_i,y_j): f_{X,Y}(x_i,x_j)>0\}$ & The support set of the discrete \rv~ $(X,Y)$\\
$f_{X,Y}(x,y)$ & Joint probability density function (JPDF) of the continuous \rv~ $(X,Y)$\\
$f_{X}(x)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)dy$ & Marginal probability density/mass function (MPDF/MPMF) of $X$\\
$f_{Y}(y)$\newline$=\int_{-\infty}^{\infty}f_{X,Y}(x,y)dx$ & Marginal probability density/mass function (MPDF/MPMF) of $Y$\\
$E(g(X,Y))$\newline$=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}g(x,y)f_{X,Y}(x,y)dxdy$ & Expectation of a function $g(x,y)$ for continuous \rv \\
$E(g(X,Y))$\newline$=\sum_{(x,y)\in\mathcal{S}_{X,Y}}g(x,y)f_{X,Y}(x,y)$ & Expectation of a function $g(x,y)$ for discrete \rv \\
$E(X^rY^s)$ & Joint moment \\
$\cv(X,Y) = E(XY)-E(X)E(Y)$ & Covariance of $X$ and $Y$, provided $E(X^2)<\infty$ and $E(Y^2)>\infty$\\
$F_{X,Y}(x,y)=F_X(x)F_Y(y)$,\newline for every $(x,y)$ & if and only if $X$ and $Y$ are said to be independent\\
$f_{X,Y}(x,y)=f_X(x)f_Y(y)$,\newline for every $(x,y)$ & if and only if $X$ and $Y$ are said to be independent\\
$F_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$ & Joint (cumulative) distribution function (JDF/JCDF) of the discrete or continuous \rv~ $(X_1,X_2,\ldots,X_n)$\\
$f_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$ & Joint probability mass/density function (JPMF/JPDF) of the discrete/continuous \rv~ $(X_1,X_2,\ldots,X_n)$\\
$f_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$\newline$=\prod_{i=1}^n f_{X_i}(x_i)$,\newline for every $(x_1,x_2,\ldots,x_n)$ & if and only if $X_1,X_2,\ldots,X_n$ are (mutually/jointly) independent\\
\hline
\end{tabular}
}
}
\caption{Symbol Table: Probability and Statistics \label{T:FullSymbTableProbStats}}
\end{table}
}


\section{Compactness}
Recall that a continuous, real-valued function on a closed bounded interval is bounded and attains its bound.
The question is, for which topological space $X$ is it true that every continuous real functions is bounded.
\begin{example}
    1. For finite $X$, every function $X\to\mathbb R$ is bounded.\\
    2. If for all continuous $f:X\to\mathbb R,\exists n\in\mathbb N,\exists A_1,A_2,\ldots,A_n\subset X$ such that $X=\bigcup_iA_i$ and $f$ is bounded on each $A_i$, then $f$ is bounded on $X$.
\end{example}
Note that given continuous $f:X\to\mathbb R$, for $x\in X$, $U_x=f^{-1}((f(x)-1,f(x)+1))$ is open and $f$ is bounded there.
So if there is some finite subset of $\{U_x:x\in X\}$ that still covers $X$, then $f$ must be bounded.
\begin{definition}
    An open cover of a topological space $X$ is a family of open sets $\mathscr U=\{U_i\}_{i\in I}$ in $X$ such that $X=\bigcup_{i\in I}U_i$.\\
    A subcover of $\mathscr U$ is a subset $\mathscr V\subset \mathscr U$ that is also an open cover of $X$.
    $\mathscr V$ is called a finite subcover if it is finite.\\
    $X$ is compact if every open cover of $X$ has a finite subcover.
\end{definition}
\begin{theorem}
    If $X\neq\varnothing$ is compact and $f:X\to\mathbb R$ is continuous, then $f$ is bounded and attains its bound.
\end{theorem}
\begin{proof}
    By continuity of $f$ and compactness of $X$, there is a finite subset of $\{f^{-1}((f(x)-1,f(x)+1)):x\in X\}$ that covers $X$, which means that $f$ is bounded on any set in a finite family, so $f$ is bounded in the union of that family, which is $X$.
    To show that $f$ attains its bound, let $m=\{f(x):x\in X\}$ which exists since $X\neq\varnothing$ and $f$ is bounded.
    Suppose that there is not an $x$ with $f(x)=m$, so for any $x\in X,f(x)>m$ so $\exists m_x$ such that $f(x)>m_x>m$.
    Let $U_x=f^{-1}((m_x,\infty))$ which is open and contains $x$, and $\inf_{U_x}f\ge m_x>x$.
    Note that the family of all $U_x$ is an open cover of $X$, so there is a finite subcover $\{U_x\}_{x\in F}$, so $\forall y\in X,f(y)\ge \min_{x\in F}m_x>m$, contradiction.
\end{proof}
Note that for a subspace $Y\subset X$, $Y$ is compact iff whenever $\mathscr U$ is a family of open set in $X$ whose union contains $Y$, there is a finite subset $\mathscr V\subset \mathscr U$ such that the union of elements in $\mathscr V$ contains $Y$.
\begin{theorem}\label{01compact}
    $[0,1]$ is compact.
\end{theorem}
\begin{proof}
    Let $\mathscr U$ be a set of open sets in $\mathbb R$ thar contains $[0,1]$, assume that there does not exist finite subcover that contains $[0,1]$, then if $0\le a<b\le 1$ and $[a,b]$ cannot be covered by any finite $\mathscr V\subset\mathscr U$, then let $C=(a+b)/2$, then one of $[a,c],[c,b]$ cannot be covered by finite $\mathscr V\subset\mathscr U$.\\
    Therefore, inductively we can find intervals $I_n=[a_n,b_n]$ such that $$I_0=[0,1],I_{n+1}\subset I_n,|b_n-a_n|=1/2^n$$
    thus $a_n\to x,b_n=a_n+(b_n-a_n)\to x$ for some $x\in [0,1]$.
    Now there is $U\subset \mathscr U$ such that $x\in U$, but then there is some $\epsilon$ with $(x-\epsilon,x+\epsilon)\subset U$, therefore for sufficiently large $n$, $I_n\subset U$, which is a contradiction.
\end{proof}
\begin{proposition}\label{compact_haus_closed}
    Let $X$ be a topological space and $Y\subset X$ a subspace, then\\
    1. If $X$ is compact and $Y$ is closed in $X$, then $Y$ is compact.\\
    2. If $X$ is Hausdorff and $Y$ is compact, then $Y$ is closed.
\end{proposition}
\begin{proof}
    1. Let $\mathscr U$ covers $Y$, then $\mathscr U\cup \{X\setminus Y\}$ is an open cover of $X$.
    Since $X$ is compact, there is a finite subcover $\mathscr V\subset \mathscr U\cup \{X\setminus Y\}$ that covers $X$, hence $\mathscr V\setminus\{X\setminus Y\}\subset \mathscr U$ is finite and covers $Y$.\\
    2. We want to show that its complement is open.
    Indeed, for any $x\in X\setminus Y$, and for any $y\in Y$, there are disjoint $U_y,V_y$ such that $x\in U_y,y\in V_y$, then $\{V_y\}_{y\in Y}$ is an open cover of $Y$, thus there is some finite set $F\subset Y$ such that $Y\subset\bigcup_{y\in F}V_y$, so $U=\cap_{y\in F}U_y$ is open, and by definition it is disjoint from $Y$, hence $x\in U\subset X\setminus Y$, which shows that $X\setminus Y$ is open.
\end{proof}
\begin{proposition}
    If $X$ is compact and $f:X\to Y$ is continuous, then $f(X)$ is compact.
\end{proposition}
\begin{proof}
    For any open $\{U_i\}_{i\in I}$ that covers $f(X)$, $\{f^{-1}(U_i)\}_{i\in I}$ is an open cover of $X$, therefore there is some finite $F\subset I$ such that $\{f^{-1}(U_i)\}_{i\in F}$ covers $X$, hence $\{U_i\}_{i\in F}$ covers $f(X)$.
\end{proof}
\begin{remark}
    1. Compactness is a topological property.\\
    2. Let $f:X\to Y$ and $A\subset X$.
    Suppose $A$ is compact, then $f(A)$ is compact.
\end{remark}
\begin{example}
    For $a<b$, $[a,b]=f([0,1])$ where $f(x)=(b-a)x+a$ which is continuous, thus every closed bounded interval is compact.
\end{example}
\begin{corollary}
    If $X$ is compact and $R$ is an equivalence relation on $X$, then $X/R$ is compact.
\end{corollary}
\begin{proof}
    The quotient map is continuous and surjective.
\end{proof}
\begin{theorem}[Topological Inverse Function Theorem]
    If $f:X\to Y$ is a continuous bijection and $X$ is compact and $Y$ is Hausdorff, then $f$ is a homeomorphism.
\end{theorem}
\begin{proof}
    It suffices to check that $f$ is an open map, which, since $f$ is a bijection, is equivalent to say that $f$ is a closed map.\\
    Fix any closed $V\subset X$, then $V$ is compact since $X$ is compact, thus $f(V)$ is compact since $f$ is continuous, therefore $f(V)$ is closed since $Y$ is hausfdorff.
    The result follows.
\end{proof}
\begin{example}
    Consider $f:\mathbb R\to S^1$ by $f(t)=e^{2\pi it}$ induces a continuous bijection $\tilde{f}:\mathbb R/\mathbb Z\to S^1$.
    Now $\mathbb R/\mathbb Z=q([0,1])$ (where $q$ is the quotient map) is compact and $S^1$ is Hausdorff since it is a metric space, therefore $\tilde{f}$ is a homeomorphism.
\end{example}
\begin{theorem}[Tychonorff's Theorem on Finite Products
    \footnote{It works for arbitrary products, but that case is much much harder}
    ]\label{tycho_finite}
    Finite products of compact spaces are compact.
\end{theorem}
\begin{proof}
    It suffices to show for $2$.\\
    Assume $X,Y$ are compact.
    Fix $x\in X$, $\exists W_y\in\mathscr U$ with $x,y\in W_y$, so there is some $U_y$ open in $X$ and $V_y$ open in $Y$ such that $(x,y)\in U_y\times V_y\subset W_y$, so there is a finite $F_Y\subset Y$ with $\bigcup_{y\in F_Y}V_y=Y$.
    Let $T_x=\bigcup_{y\in F_Y}U_y$ is open and contains $x$ and note that $T_x\times Y\subset \bigcup_{y\in F_Y}W_y$.
    But then $\{T_x\}_{x\in X}$ covers $X$, so there is a finite $F_X\subset X$ such that $\{T_x\}_{x\in F_X}$ covers $X$, hence
    $$X\times Y\subset \bigcup_{x\in F_X}T_x\times Y=\bigcup_{x\in F_X}\bigcup_{y\in F_Y}W_y$$
    The last term is the required finite subcover.
\end{proof}
\begin{theorem}[Heine-Borel Theorem]
    A subset $K\subset\mathbb R^n$ if and only if it is closed and bounded.
\end{theorem}
\begin{proof}
    If $K$ is compact, note that $f:\mathbb R^n\to\mathbb R$ by $x\mapsto \|x\|$, thus it is bounded.
    $K$ is also closed by Proposition \ref{compact_haus_closed}.\\
    Conversely, if $K$ is closed and bounded, there is some $M>0$ such that $K\subset [-M,M]^n$ which is compact by Theorem \ref{01compact} and \ref{tycho_finite}.
    Since $K$ is closed in $[-M,M]^n$, it is compact by Proposition \ref{compact_haus_closed}.
\end{proof}
\begin{definition}
    Given an open set $U\subset\mathbb R^n$, a sequence of functions $f_k:U\to\mathbb R$ converges locally uniformly on $U$ to some function $f:U\to\mathbb R$ if $\forall x\in U, \exists r>0, D_r(x)\subset U$ and $f_k\to f$ uniformly on $D_r(x)$.
\end{definition}
Thus this happens if and only if $f_n\to f$ uniformly on any compact subset of $U$.
\begin{definition}
    A topological space $X$ is called sequentially compact if and only if every sequence in $X$ has a convergent subsequence.
\end{definition}
\begin{example}
    Any closed bounded subset of $\mathbb R^n$ is sequentially compact by Bolzano-Weierstrass.
\end{example}
\begin{definition}
    Fix a metric space $(M,d)$.
    For $\epsilon>0$ and $F\subset M$.
    We say $F$ is an $\epsilon$-net for $M$ if $\forall x\in M,\exists y\in F,d(x,y)\le \epsilon$.
    That is,
    $$M=\bigcup_{y\in F} B_\epsilon(y)$$
    We say $M$ is totally bounded if for any $\epsilon>0$, there is a finite $\epsilon$-net for $M$.
\end{definition}
Note that any compact space is totally bounded, but the converse is not true by taking $[0,1)$, but the only thing missing here is completeness.
\begin{theorem}
    The followings are equivalent:\\
    (1) $M$ is compact.\\
    (2) $M$ is sequentially compact.\\
    (3) $M$ is totally bounded and complete.
\end{theorem}
\begin{proof}
    $1\implies 2$: Let $(x_n)$ be a sequence in $M$, so for $n\in\mathbb N$, let $A_n=\{x_k:k>n\}$.
    We shall show that $\bigcap_{n\in\mathbb N}\bar{A}$ is nonempty.
    Assume not, then
    $$\bigcup_{n\in\mathbb N}M\setminus\bar{A}=M$$
    But $M$ is compact and all $M\setminus\bar{A}$ are closed, there is finite subcover, hence there is some $N\in\mathbb N$ such that $\bigcup_{n\le N}M\setminus\bar{A}=M$.
    Also $A_m\supset A_n,\forall m\le n$, so we have the complement of the closure of $A_{N}$ would be $M$, so that closure is empty, contradiction.\\
    So we can fix $x\in \bigcap_{n\in\mathbb N}\bar{A}$.
    It is then trivial to construct a subsequence of $x_n$ that converges to $x$.\\
    $2\implies 3$: $M$ is complete since a Cauchy sequence with convergent subsequence is convergent.
    To see it is totally bounded, assume it is not, then there is some $\epsilon>0$ such that every $\epsilon$-net is infinite.
    Pick $x_1\in M$, then if we have already picked $x_1,\ldots,x_n$, we can pick $x_{n+1}\notin \bigcup_{k=1}^nB_\epsilon(x_k)$, which we can do since $M$ has no finite $\epsilon$-net.
    But this $(x_n)$ does not have any Cauchy subsequence, so it has no converging subsequence.\\
    $3\implies 1$: Assume $M$ is not compact, so there is an open cover $\mathscr U$ without any finite subcover.
    We say $A\subset M$ is ``bad'' if there is no finite subcover of $A$ in $\mathscr U$.
    So $M$ is bad but $\varnothing$ is not.
    Note if $A=\bigcup_{i=1}^nB_i$ is bad, then there is some $i$ such that $B_i$ is bad.\\
    Next, we want to show that if $A$ is bad and $\epsilon>0$, then $\exists B\subset A$ such that $B$ is bad and $\operatorname{diam}B=\sup_{x,y\in B}d(x,y)<\epsilon$.
    Indeed, since $M$ is bounded, we have a finite $\epsilon/2$-net $F$, that is,
    $$\bigcup_{x\in F}B_{\epsilon/2}(x)=M\implies \bigcup_{x\in F}(B_{\epsilon/2}(x)\cap A)=A$$
    But this would mean that there is some $x\in F$ such that $B_{\epsilon/2}(x)\cap A$ is bad, and by triangle inequality its diameter is less than $\epsilon$.
    Using this we can construct a sequence $M\supset A_1\supset A_2\supset\cdots$ such that $A_n$ is bad for any $n$ and $\operatorname{diam}A<1/n$.
    So we can pick $x_n\in A_n$, then $x_n$ is Cauchy, thus it tends to a limit $x\in M$ by completeness, so there is some $U\in\mathscr U$ such that $x\in U$, so $\exists r>0$ such that $D_r(x)\subset U$, which provides a finite subcover for $A_n$ where $n$ is large enough.
\end{proof}
\begin{remark}
    We have a new proof of Bolzano-Weierstrass now!
    We can also have a new proof of Theorem \ref{tycho_finite} for metric spaces.\\
    However, the equivalence of sequentially compactness and compactness fails in both directions in general topological spaces.
\end{remark}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}

\begin{document}

\section{Formuals}

Permutations, $k$ objects from a sequence of $n$, or $_{n}P_{k}$.

\begin{equation}
   _{n}P_{k} = \frac{n!}{(n - k)!} = n(n-1)(n-2)\ldots\ (n-k+1)
\end{equation}

Permutations with repetition, same as Permutation except you can choose the same object multiple times.

\begin{equation}
   _{n}P^r_{k} = k^n
\end{equation}

Combinations, $k$ objects from a set $n$ to form a new set, or $\binom{n}{k}$ (pronounced $n$ choose $k$).

\begin{equation}
   \binom{n}{k} = \frac{n!}{k!(n-k)!}
\end{equation}

Combinations with repetition, $k$ objects from a set $n$ to form a new \textit{multi}set, or $\left(\binom{n}{k}\right)$.

\begin{equation}
   \left(\binom{n}{k}\right) = \binom{n + k - 1}{k}
\end{equation}

Combination identities,

\begin{gather}
   \binom{n}{k} = \binom{n}{n - k} \\
   \binom{n}{k} = \binom{n - 1}{k} + \binom{n - 1}{k - 1} \\
   \binom{n}{m} \binom{m}{k} = \binom{n}{k} \binom{n - k}{m - k}
\end{gather}

Sums,

\begin{gather}
   \sum^n_{i=1} i = \frac{n(n+1)}{2} \\
   \sum^n_{i=1} i^2 = \frac{n(n+1)(2n+1)}{6} \\
   \sum^n_{i=1} i^3 = \frac{n^2(n+1)^2}{4} \\
   \sum^n_{k=0} \binom{n}{k} = 2^n
\end{gather}

Logarithm identities,

\begin{gather}
   \log_b x = \frac{\log_a x}{\log_a b} \\
   \log x^y = y \log x \\
   \log(xy) = \log x + \log y
\end{gather}

Trigonometry identities,

\begin{gather}
   \tan x = \frac{\sin x}{\cos x} \\
   \sin^2 x + \cos^2 x = 1 \\
   \sin x = \cos (\frac{\pi}{2} - x) \\
   \sin (x \pm y) = \sin x \cos y \pm \cos x \sin y \\
   \pi = 4 \arctan(1) \\
   e^{ix} = \cos x + i \sin x
\end{gather}

\end{document}


\section{Real inner product spaces}

\begin{outcome}
  \begin{enumerate}
  \item Check whether an operation is an inner product.
  \item Give an example of a vector space on which more than one inner
    product can be defined.
  \item Calculate the inner product of vectors in various examples
    of inner product spaces.
  \item Calculate the norm of a vector and the angle between two vectors.
  \item Use the Cauchy-Schwarz inequality and the triangle inequality
    to reason about the size of inner products and norms of vectors.
  \end{enumerate}
\end{outcome}

In this section, we fix $K$ to be the field of real numbers.

\begin{definition}{Real inner product space}{real-inner-product-space}
  A \textbf{(real) inner product space}%
  \index{inner product space!real}%
  \index{real inner product space} is a real vector space $V$ equipped
  with an operation that assigns to any pair of vectors
  $\vect{u},\vect{v}\in V$ a real number $\iprod{\vect{u},\vect{v}}$,
  called the \textbf{inner product}%
  \index{inner product}%
  \index{multiplication!inner product} of $\vect{u}$ and
  $\vect{v}$. This operation must satisfy the following properties:
  \begin{enumerate}
  \item Symmetry:%
    \index{symmetry!of inner product}
    $\iprod{\vect{u},\vect{v}} = \iprod{\vect{v},\vect{u}}$.
  \item Linearity:%
    \index{linearity!of inner product}
    $\iprod{\vect{u},k\vect{v}+\ell\vect{w}}
    = k\iprod{\vect{u},\vect{v}}+\ell\iprod{\vect{u},\vect{w}}$.
  \item The positive definite property:%
    \index{positive definite property}
    $\iprod{\vect{u},\vect{u}} \geq 0$, and moreover,
    $\iprod{\vect{u},\vect{u}}=0$ if and only if
    $\vect{u} = \vect{0}$.
  \end{enumerate}
\end{definition}

\begin{example}{$\R^n$ with the usual dot product}{rn-with-dot-product}
  $V=\R^n$, with the usual dot product
  \begin{equation*}
    \iprod{\vect{u},\vect{v}} = \vect{u}\dotprod\vect{v} =
    u_1v_1 + \ldots + u_nv_n
  \end{equation*}
  is an inner product space.%
  \index{inner product space!Rn@$R^n$}
\end{example}

\begin{proof}
  The relevant properties were shown in
  Proposition~\ref{prop:properties-dot-product}.
\end{proof}

There exist other inner product operations on $\R^n$ besides the dot
product. The following is an example.

\begin{example}{$\R^2$ with a non-standard inner product}{rn-with-inner-product}
  Let $V=\R^2$, and consider the matrix
  $A=\begin{mymatrix}{cc} 1 & 1 \\ 1 & 2 \end{mymatrix}$.
  Define an operation $\iprod{-,-}$ by
  \begin{equation*}
    \iprod{\vect{u},\vect{v}} = \vect{u}^T A \vect{v} =
    \begin{mymatrix}{cc} u_1 & u_2 \end{mymatrix}
    \begin{mymatrix}{cc} 1 & 1 \\ 1 & 2 \end{mymatrix}
    \begin{mymatrix}{c} v_1 \\ v_2 \end{mymatrix}
    = u_1v_1 + u_1v_2 + u_2v_1 + 2u_2v_2.
  \end{equation*}
  Then $\R^2$, with this operation, is an inner product space.
\end{example}

\begin{proof}
  We must show the three properties are satisfied. For symmetry, note
  that a scalar is equal to its own transpose. Therefore
  \begin{equation*}
    \iprod{\vect{u},\vect{v}}
    ~=~ \vect{u}^T A \vect{v}
    ~=~ (\vect{u}^T A \vect{v})^T
    ~=~ \vect{v}^T A^T \vect{u}.
  \end{equation*}
  But this is equal to $\iprod{\vect{v},\vect{u}}$ because $A$ is a
  symmetric matrix, i.e., $A^T=A$.
  For linearity, note that, by properties of matrix multiplication,
  \begin{equation*}
    \iprod{\vect{u},k\vect{v}+\ell\vect{w}}
    ~=~ \vect{u}^T A(k\vect{v}+\ell\vect{w})
    ~=~ k\vect{u}^T A\vect{v} + \ell\vect{u}^T A\vect{w}
    ~=~ k\iprod{\vect{u},\vect{v}} + \ell\iprod{\vect{u},\vect{w}}.
  \end{equation*}
  For the positive definite property, we have
  \begin{equation*}
    \iprod{\vect{u},\vect{u}}
    ~=~ u_1^2 + 2u_1u_2 + 2u_2^2
    ~=~ (u_1^2 + 2u_1u_2 + u_2^2) + u_2^2
    ~=~ (u_1+u_2)^2 + u_2^2
    ~\geq~ 0,
  \end{equation*}
  because the sum of two squares is always non-negative. Moreover, if
  equality holds in the last equation, then we must have $u_1+u_2=0$
  and $u_2=0$, and this is only possible if $\vect{u}=\vect{0}$.
\end{proof}

\begin{example}{Continuous functions on an interval}{continuous-interval}
  Let $a<b$ be real numbers, and let
  \begin{equation*}
    [a,b] = \set{x \mid a \leq x \leq b}
  \end{equation*}
  be the closed interval. Let $V=C[a,b]$%
  \index{Cab@$C[a,b]$} be the vector space of all continuous
  functions%
  \index{continuous function}%
  \index{function!continuous}%
  \index{inner product space!of continuous functions}
  $f:[a,b]\to\R$. Given two functions $f,g\in C[a,b]$, define
  \begin{equation*}
    \iprod{f,g} = \int_{a}^{b} f(x)g(x)\,dx.
  \end{equation*}
  This is an inner product.
\end{example}

\begin{proof}
  This follows from well-known properties of integrals. For symmetry,
  we have
  \begin{equation*}
    \iprod{f,g}
    ~=~ \int_{a}^{b} f(x)g(x)\,dx
    ~=~ \int_{a}^{b} g(x)f(x)\,dx
    ~=~ \iprod{g,f}.
  \end{equation*}
  For linearity, we have
  \begin{equation*}
    \iprod{f,kg+\ell h}
    ~=~ \int_{a}^{b} f(x)(kg(x)+\ell h(x))\,dx
    ~=~ k\int_{a}^{b} f(x)g(x)\,dx+\ell\int_{a}^{b} f(x)h(x)\,dx
    ~=~ k\iprod{f,g}+\ell\iprod{f,h}.
  \end{equation*}
  For the positive definite property, we have
  \begin{equation*}
    \iprod{f,f}
    = \int_{a}^{b} f(x)^2\,dx
    \geq 0.
  \end{equation*}
  This is $\geq 0$ because $a<b$ and the integral over a non-negative
  function is non-negative. Moreover, since $f$ is continuous, we know
  from calculus that the last integral can only be equal to $0$ if $f$
  is the constant zero function.
\end{proof}

\begin{example}{Polynomials}{inner-product-polynomials}
  Let $\Poly$ be the vector space of polynomials (of any degree) with
  real coefficients. Let $a<b$ be real numbers, and consider the inner
  product defined by
  \begin{equation*}
    \iprod{p,q} = \int_{a}^{b} p(x)q(x)\,dx.
  \end{equation*}
  This is an inner product space.%
  \index{inner product space!of polynomials}
\end{example}

\begin{proof}
  The proof is the same as for $C[a,b]$.
\end{proof}

\begin{example}{Real Hilbert space}{hilbert-space}
  Let $\Hilb_{\R}$ be the vector space of all infinite sequences of
  real numbers $a=(a_0,a_1,a_2,\ldots)$ satisfying
  \begin{equation*}
    a_0^2 + a_1^2 + a_2^2 + \ldots < \infty.
  \end{equation*}
  These are called the \textbf{square summable} sequences%
  \index{sequence!square summable}%
  \index{square summable sequence}. (One needs
  to do a little bit of work to show that it is indeed a vector space;
  in particular, to show that the sum of two square summable sequences
  is square summable.) On this space, we can
  define an inner product as follows:
  \begin{equation*}
    \iprod{a,b} = a_0b_0 + a_1b_1 + a_2b_2 + \ldots.
  \end{equation*}
  The details will be worked out in Exercise~\ref{ex:hilbert-space}.
  This inner product space is called \textbf{(real) Hilbert space}%
  \index{Hilbert space}%
  \index{inner product space!Hilbert space}.
\end{example}

We now look at some properties of inner products. The first thing to
note is that while the axioms require linearity in the right
component, symmetry ensures that linearity in the left component also
holds.

\begin{proposition}{Left linearity}{inner-product-left-linearity}
  Let $V$ be an inner product space. Then
  $\iprod{k\vect{u}+\ell\vect{v},\vect{w}}
  = k\iprod{\vect{u},\vect{w}} + \ell\iprod{\vect{v},\vect{w}}$.
\end{proposition}

\begin{proof}
  This follows directly from symmetry. We have
  \begin{equation*}
    \iprod{k\vect{u}+\ell\vect{v},\vect{w}}
    = \iprod{\vect{w},k\vect{u}+\ell\vect{v}}
    = k\iprod{\vect{w},\vect{u}}
    + \ell\iprod{\vect{w},\vect{v}}
    = k\iprod{\vect{u},\vect{w}}
    + \ell\iprod{\vect{v},\vect{w}}.
  \end{equation*}
\end{proof}

Another important property of inner products is the Cauchy-Schwarz
inequality. We have already encountered this in the context of the dot
product in Section~\ref{sec:dot-product}.

\begin{theorem}{Cauchy-Schwarz inequality}{inner-product-cauchy-schwarz}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then%
  \index{Cauchy-Schwarz inequality!in inner product space}%
  \index{inner product space!Cauchy-Schwarz inequality}
  \begin{equation}\label{cauchy-inner}
    \iprod{\vect{u},\vect{v}}^2
    \leq \iprod{\vect{u},\vect{u}} \cdot \iprod{\vect{v},\vect{v}}.
  \end{equation}
\end{theorem}

\begin{proof}
  The proof is almost identical to that of
  Proposition~\ref{prop:cauchy-schwarz-inequality}.  First note that if
  $\vect{u}=\vect{0}$, then both sides of {\eqref{cauchy-inner}} are
  equal to zero, and so there is nothing to show. Therefore, we will
  assume in what follows that $\vect{u}\neq \vect{0}$.  Define a
  function of $t\in \R$ by
  \begin{equation*}
    f(t) = \iprod{t\vect{u}+\vect{v}, t\vect{u}+\vect{v}}.
  \end{equation*}
  Then by the positive definite property, we know that $f(t)\geq 0$
  for all $t\in \R$.  Also from linearity and symmetry, we have
  \begin{eqnarray*}
    f(t) &=& \iprod{t\vect{u}, t\vect{u}+\vect{v}} +
             \iprod{\vect{v}, t\vect{u}+\vect{v}} \\
         &=& t^2\iprod{\vect{u}, \vect{u}}+t\iprod{\vect{u}, \vect{v}} + t\iprod{\vect{v}, \vect{u}}+
             \iprod{\vect{v}, \vect{v}} \\
         &=&t^2\iprod{\vect{u},\vect{u}}+2t\iprod{\vect{u}, \vect{v}}
             + \iprod{\vect{v}, \vect{v}}.
  \end{eqnarray*}
  This means the graph of $y=f(t)$ is a parabola which opens upwards
  and is never negative. It follows that this function has at most one
  root. From the quadratic formula, we know that a quadratic function
  $at^2+bt+c$ has one or zero roots if and only if $b^2-4ac\leq
  0$. Applying this reasoning to the function $f(t)$, we obtain
  \begin{equation*}
    (2\iprod{\vect{u}, \vect{v}})^2
    - 4\iprod{\vect{u},\vect{u}}\iprod{\vect{v},\vect{v}} < 0,
  \end{equation*}
  which is equivalent to
  $\iprod{\vect{u},\vect{v}}^2 \leq \iprod{\vect{u},\vect{u}} \cdot
  \iprod{\vect{v},\vect{v}}$.
\end{proof}

Finally, we can use the inner product to define the norm of a vector.

\begin{definition}{Norm}{inner-product-norm}
  Let $\vect{u}$ be a vector in an inner product space. Then the
  \textbf{norm}%
  \index{norm!in inner product space}%
  \index{vector!norm}%
  \index{inner product space!norm} of $\vect{u}$ is defined as
  \begin{equation*}
    \norm{\vect{u}} = \sqrt{\iprod{\vect{u},\vect{u}}}.
  \end{equation*}
\end{definition}

\begin{example}{Norm in $C[-1,1]$}{norm-in-continuous-functions}
  Calculate the norm of $f(x) = x^2$ in $C[-1,1]$.
\end{example}

\begin{solution}
  In the vector space $C[-1,1]$, the inner product is defined as
  \begin{equation*}
    \iprod{f,g} = \int_{-1}^{1} f(x)g(x)\,dx.
  \end{equation*}
  We therefore have
  \begin{equation*}
    \iprod{f,f}
    = \int_{-1}^{1} f(x)^2\, dx
    = \int_{-1}^{1} x^4\, dx
    = \bigbracket{\frac{1}{5}x^5}_{-1}^{1}
    = \frac{2}{5}.
  \end{equation*}
  Therefore, $\norm{f} = \sqrt{\iprod{f,f}} = \sqrt{\frac{2}{5}}$.
\end{solution}

A vector $\vect{u}$ in an inner product space is called
\textbf{normalized}%
\index{vector!normalized}%
\index{normalized vector} or a \textbf{unit vector}%
\index{vector!unit vector}%
\index{unit vector} if $\norm{\vect{u}} = 1$.
We note that if $\vect{v}$ is any non-zero vector in an inner product
space, then
\begin{equation*}
  \vect{u} = \frac{1}{\norm{\vect{v}}} \vect{v}
\end{equation*}
is normalized.

By the Cauchy-Schwarz inequality, we have the following properties:

\begin{proposition}{Inner product and norm}{inner-product-and-norm}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then
  \begin{equation*}
    \abs{\iprod{\vect{u},\vect{v}}} \leq \norm{\vect{u}}\norm{\vect{v}}.
  \end{equation*}
\end{proposition}

\begin{proof}
  The Cauchy-Schwarz inequality states that
  $\iprod{\vect{u},\vect{v}}^2 \leq
  \norm{\vect{u}}^2\norm{\vect{v}}^2$. The claim follows by taking the
  square root of both sides of the equation.
\end{proof}

\begin{proposition}{Triangle inequality}{inner-product-triangle-inequality}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then
  \begin{equation*}
    \norm{\vect{u}+\vect{v}} \leq \norm{\vect{u}} + \norm{\vect{v}}.
  \end{equation*}
  This is called the \textbf{triangle inequality}%
  \index{triangle inequality!in inner product space}%
  \index{inner product space!triangle inequality}.
\end{proposition}

\begin{proof}
  The proof is essentially the same as that of
  Proposition~\ref{prop:triangle-inequality-dot-product}. We have
  \begin{eqnarray*}
    \norm{\vect{u}+\vect{v}}^2
    &=& \iprod{\vect{u}+\vect{v},\vect{u}+\vect{v}} \\
    &=& \iprod{\vect{u},\vect{u}}+\iprod{\vect{u},\vect{v}}+\iprod{\vect{v},\vect{u}}+\iprod{\vect{v},\vect{v}} \\
    &=&\norm{\vect{u}}^2+2\iprod{\vect{u},\vect{v}}+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\abs{\iprod{\vect{u},\vect{v}}}+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\norm{\vect{u}}\norm{\vect{v}}+\norm{\vect{v}}^2 \\
    &=& (\norm{\vect{u}}+\norm{\vect{v}})^2.
  \end{eqnarray*}
  The triangle inequality follows by taking square roots of both sides.
\end{proof}

Generalizing the situation in $\R^n$, we can define the angle
between any two vectors in an inner product space. Note that by the
Cauchy-Schwarz inequality,
\begin{equation*}
  \frac{\abs{\iprod{\vect{u},\vect{v}}}}{\norm{\vect{u}}\norm{\vect{v}}}
  \leq 1,
\end{equation*}
and therefore,
\begin{equation*}
  -1 \leq \frac{\iprod{\vect{u},\vect{v}}}{\norm{\vect{u}}\norm{\vect{v}}}
  \leq 1.
\end{equation*}
This ensures that the following definition is well-defined.

\begin{definition}{Angle between two vectors}{inner-product-angle}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space.
  The \textbf{angle}%
  \index{angle!in inner product space}%
  \index{angle!between vectors} between $\vect{u}$ and
  $\vect{v}$ is, by definition, the unique $\theta$ such that
  $0\leq\theta\leq\pi$ and
  \begin{equation*}
    \cos\theta = \frac{\iprod{\vect{u},\vect{v}}}{\norm{\vect{u}}\norm{\vect{v}}}.
  \end{equation*}
\end{definition}

\begin{example}{Find the angle between two vectors}{inner-product-angle}
  Find the angle between $1$ and $x^2$ in $C[-1,1]$.
\end{example}

\begin{solution}
  We have
  \begin{eqnarray*}
    \iprod{1,1} &=& \int_{-1}^1 1\cdot 1\,dx ~=~ 2, \\
    \iprod{x^2,x^2} &=& \int_{-1}^1 x^2\cdot x^2\,dx ~=~ \frac{2}{5}, \\
    \iprod{1, x^2} &=& \int_{-1}^1 1\cdot x^2\,dx ~=~ \frac{2}{3}.
  \end{eqnarray*}
  Therefore
  \begin{equation*}
    \cos\theta
    ~=~ \frac{\iprod{1,x^2}}{\norm{1}\norm{x^2}}
    ~=~ \frac{\frac{2}{3}}{\sqrt{2}\sqrt{\frac{2}{5}}}
    ~=~ \frac{\sqrt{5}}{3}.
  \end{equation*}
  The angle $\theta$ is $\cos^{-1}(\frac{\sqrt{5}}{3})$, which is
  approximately $0.7297$ radians or $41.81$ degrees.

\end{solution}


\subsubsection{Magnetic Fields}
\noindent
Gauss's Law for magnetism says that that magnetic flux through a closed surface is 0. Rewritten more formally,
\begin{equation*}
	\oint\limits_{S}\oint{\vec{B} \cdot \mathrm{d}\vec{s}} = 0
\end{equation*}
So,
\begin{equation*}
	\iiint\limits_{V}{\nabla \cdot \vec{B}\mathrm{d}V} = \iiint\limits_{V}{0\mathrm{d}V}
\end{equation*}
If we let $V \to 0$,
\begin{equation*}
	\nabla \cdot \vec{B} = 0
\end{equation*}
This means that there are no lone sources or sinks in magnetic fields. You may also hear this law summarized as "There are no magnetic monopoles." This is the 2nd of Maxwell's Equations.