\subsection{Distance between Two Points and Midpoints}\label{sec:DistanceAndMidpoints}

Another important concept in Geometry is the notion of length.  If we are going to unite Algebra and Geometry using the Cartesian Plane, then we need to develop an algebraic understanding of what distance in the plane means.\\

Given two points $(x_1,y_1)$ and $(x_2,y_2)$, recall that their
horizontal distance from one another is $\Delta x=x_2-x_1$ and their
vertical distance from one another is $\Delta y=y_2-y_1$. Actually,
the word ``distance'' normally denotes ``positive distance''. $\Delta
x$ and $\Delta y$ are {\it signed\/} distances, but this is clear from
context. The (positive) distance from one point to the other
is the length of the hypotenuse of a right triangle with legs $|\Delta
x|$ and $|\Delta y|$, as shown in Figure~\ref{fig:distance between
points}.  The Pythagorean Theorem states that the distance between
the two points is the square root of the sum of the squares of the
horizontal and vertical sides:\\
$$| \, \Delta x \,|^{2} + | \, \Delta y \, |^{2} =  d^{2}$$ \\

\figure[!ht]
\centerline{\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <0.5truein,0.5truein>
\setplotarea x from 0 to 3, y from 0 to 2
\putrule from 0 0 to 3 0
\putrule from 3 0 to 3 2
\plot 0 0 3 2 /
\put {$(x_1,y_1)$} [r] <-5pt,0pt> at 0 0
\put {$(x_2,y_2)$} [l] <5pt,0pt> at 3 2
\put {$\Delta x$} [t] <0pt,-5pt> at 1.5 0
\put {$\Delta y$} [l] <5pt,0pt> at 3 1
\endpicture}}
\caption{Distance between two points (here, $\Delta x$ and $\Delta y$ are positive). \label{fig:distance between points}}
\endfigure

\begin{formulabox}[Distance Formula]
The distance between points $(x_1,y_1)$ and $(x_2,y_2)$ is
$$\hbox{distance}=\sqrt{(\Delta x)^2+(\Delta y)^2}=\sqrt{(x_2-x_1)^2+ (y_2-y_1)^2}$$
\end{formulabox}

\bigskip

\begin{example}{Distance Between Two Points}{DistanceExample}
Find and simplify the distance between $P(-2,3)$ and $Q(1,-3)$.
\end{example}
\begin{solution}
The distance, $d$, between points $P(-2,3)$ and $Q(1,-3)$ is \\
$$\begin{array}{rcl}
d & = & \sqrt{ (x_2-x_1)^2+ (y_2-y_1)^2}\\
 & = & \sqrt{(1-(-2))^2+(-3-(3))^2} \\
 & = & \sqrt{9+36} \\
 & = & 3\sqrt{5} \\
 \end{array}$$
 So the distance  is $3\sqrt{5}$.
\end{solution}

\begin{example}{Point on a Line}
	FFind all of the points with $x$-coordinate $1$ which are $4$ units from the point $(3,2)$.
\end{example}
\begin{solution}
	We shall soon see that the points we wish to find are on the line $x=1$, but for now we'll just view them as points of the form $(1,y)$.  Visually,\\
	$$ \includegraphics[scale=0.4]{images/distance-ex}$$
	We require that the distance from $(3,2)$ to $(1,y)$ be $4$.  The Distance Formula yields

	\[ \begin{array}{rclr}
	d &  = & \sqrt{\left(x_{\mbox{\tiny$2$}}-x_{\mbox{\tiny$1$}}\right)^2+\left(y_{\mbox{\tiny$2$}}-y_{\mbox{\tiny$1$}}\right)^2}  & \\
	4 &  = & \sqrt{(1-3)^2+(y-2)^2} & \\
	4  & = & \sqrt{4+(y-2)^2} & \\
	4^2 & = & \left(\sqrt{4+(y-2)^2}\right)^2 &  \mbox{squaring both sides} \\
	16 & = & 4+(y-2)^2 & \\
	12 & = & (y-2)^2 & \\
	(y-2)^2 & = & 12 &  \\
	y - 2 & = & \pm \sqrt{12} & \mbox{extracting the square root} \\
	y-2 & = & \pm 2 \sqrt{3} & \\
	y & = & 2 \pm 2 \sqrt{3}  &
	\end{array} \]
	We obtain two points:  $(1, 2 + 2 \sqrt{3})$ and $(1, 2-2 \sqrt{3}).$

\end{solution}

As a special case of the distance formula, suppose we want to know the
distance of a point $(x,y)$ to the origin.  According to the distance
formula, this is $$\sqrt{(x-0)^2+(y-0)^2}=\sqrt{x^2+y^2}.$$
A point $(x,y)$ is at a distance $r$ from the origin if and only if
$\sqrt{x^2+y^2}=r$, or, if we square both sides: $x^2+y^2=r^2$.  As illustrated
in the next section, this is the equation of the circle of radius, $r$, centered at the origin.

Furthermore, given two points we can determine the \dfont{midpoint} of the line segment joining the two points.\\

\begin{formulabox}[Midpoint Formula]
The midpoint of the line segment joining two points $(x_1,y_1)$ and
$(x_2,y_2)$ is the point with coordinates:
$$\hbox{midpoint}=\left(\frac{x_1+x_2}{2},\frac{y_1+y_2}{2}\right).$$
\end{formulabox}

\bigskip

\begin{example}{Midpoint of a Line Segment}{Midpoint}
Find the midpoint of the line segment joining the given points: $(1,0)$ and $(5,-2)$.
\end{example}

\begin{solution}
Using the \ifont{midpoint formula} on $(x_1,y_1)=(1,0)$ and $(x_2,y_2)=(5,-2)$ we get:
$$\left(\frac{1+5}{2},\frac{0+(-2)}{2}\right)=(3,-1)$$
Thus, the midpoint of the line segment occurs at $(3,-1)$.
\end{solution}

We close with a more abstract application of the Midpoint Formula.

\begin{example}{Distance between points}
IIf $a \neq b$, prove that the line $y = x$ equally divides the line segment with endpoints $(a,b)$ and $(b,a)$.
\end{example}
\begin{solution}
To prove the claim, we use the Midpoint formula to find the midpoint

	\[ \begin{array}{rcl}

	M & = & \left( \dfrac{a+b}{2},  \dfrac{b+a}{2} \right) \\
	& = & \left( \dfrac{a+b}{2},  \dfrac{a+b}{2} \right)  \\ \end{array} \]

	Since the $x$ and $y$ coordinates of this point are the same, we find that the midpoint lies on the line $y=x$, as required. \
\end{solution}


\subsection{Closed sets}

The complement of any open set is a closed set.

A set can be open, closed, both or neither.


\section{Synthesizing Functions for a Given Encoding} \label{sec:synthesis}
%
As a first step, we tackle the problem of synthesizing
functions over a given encoding.
%
We begin with an overview of the standard encoding for the
data types which we focused our development on.
%
We then explore how to specify the behavior of functions for
their synthesis.

% The first step we took for synthesizing expressions was to synthesize
%   individual functions given an encoding.

\subsection{Church Encodings}
%
We tested and focused our development on three
different data types: \emph{booleans}, \emph{natural
numbers} and \emph{pairs}.
%
We give their standard encodings below.

\paragraph{Booleans}
%
The standard encoding for the boolean values \emph{true} and
\emph{false} is given by:
%
\begin{align*}
  \text{\true} &= \lambda a . \lambda b . a \\
  \text{\false} &= \lambda a . \lambda b . b
\end{align*}
%
Intuitively, boolean logic is interpreted as a choice and
boolean values are represented as functions of two
parameters making this choice: \true chooses its first
parameter and \false chooses its second parameter.

\paragraph{Natural numbers}
%
Church numerals are the standard representation of natural
numbers in the lambda calculus.
%
In this encoding the number $n$ is represented as a
higher-order function that maps a function $f$ to its
$n$-fold composition.
%
The value of the numeral is equivalent to the number of
times the function gets applied: 0 does not apply the
function at all, 1 applies the function once, 2 applies the
function twice, \emph{etc.}
%
\begin{align*}
  0 & = \lambda f . \lambda x . x \\
  1 & = \lambda f . \lambda x . f\,x \\
  2 & = \lambda f . \lambda x . f\,(f\,x) \\
  3 & = \lambda f . \lambda x . f\,(f\,(f\,x)) \\
  \vdots & \\
  n & = \lambda f . \lambda x . f^{\circ n}\,x
\end{align*}

\paragraph{Pairs}
%
A pair is represented as a function that takes two arguments
and returns a higher-order function which when provided with
a function as argument applies it to the two components of
the pair.
%
\begin{align*}
  \pair &= \lambda x. \lambda y. \lambda f.f\,x\,y
\end{align*}

\subsection{Synthesis from Examples} \label{sec:io}
%
We now turn to the question of how the behavior of a
function can be specified.
%
As it has proven useful in many scenarios, we explore the
idea of using input-output examples.

Consider, for instance, the boolean function \f{and}.
We can specify its behavior extensionally as follows:
%
\begin{align*}
\f{?and}\,\true\,\true    &\equiv \true \\
\f{?and}\,\true\,\false   &\equiv \false \\
\f{?and}\,\false\,\true   &\equiv \false \\
\f{?and}\,\false\, \false &\equiv \false \\
\end{align*}
%
By convention, we prefix the functions we wish to synthesize
with \f{?} and by convenience we refer to the boolean values
\true and \false by name, but they should be understood as
if their definition was expanded.
%
With this specification our tool can successfully find the
standard definition of \f{and} for church booleans:
%
\begin{align}
  \label{eq:and-def}
  \f{and} &= \lambda p. \lambda q. p\,q\,p
\end{align}

Similarly, we can apply this idea to numerals to specify the
behavior of the \f{plus} function.
%
In this case, however, we cannot specify the behavior on all
possible inputs but we can select a representative set.
%
\begin{align*}
  \f{?plus}\; 0\; 1 &\equiv 1\\
  \f{?plus}\; 1\; 0 &\equiv 1\\
  \f{?plus}\; 1\; 2 &\equiv 3\\
  \f{?plus}\; 2\; 2 &\equiv 4\\
\end{align*}
%
The above specification is sufficient for our tool to find
the definition:
%
\begin{align}
  \label{eq:plus-def}
  \f{plus} &= \lambda m. \lambda n. \lambda f. \lambda x. m\,f\,(n\,f\,x)
\end{align}

\subsection{Beyond Extensionality}
%
The specifications presented in the previous section only
spell out the extensional behavior of functions.
%
We found that it is also often useful to constraint the way
functions evaluate.
%
The ability to do so proved to be important for the
specification of encodings as we will see in
\cref{sec:co-synthesis}.

Recall the resulting encoding for \f{and} given in
\labelcref{eq:and-def}.
%
This definition uses the first argument as the test.
%
Under normal order evaluation, this corresponds to the usual
short-circuit evaluation where the second argument is only
evaluated when the first one is known to be false.
%
The purely extensional specification given in \cref{sec:io}
cannot distinguish this from a function that first
evaluates the second argument:
%
\begin{align}
  \label{eq:and-def2}
  \f{and} &= \lambda p. \lambda q. q\,p\,q
\end{align}
%
One of our main insights is that this sort of evaluation
behavior can be specified using free variables.
%
For example, the function \f{and} could be specified as:
%
\begin{align*}
  \f{?and}\,\true\,x &\equiv x \\
  \f{?and}\,\false\,x &\equiv \false \\
\end{align*}
%
Here, the variable $x$ is meant to remain free as opposed to
\true and \false which should be expanded to their
definition.
%
This specification effectively rule out
\labelcref{eq:and-def2} from the possible results.

Similarly, the specification for \f{plus} in the previous
section cannot distinguish the definition in
\labelcref{eq:plus-def} from:
%
\begin{align}
  \label{eq:plus-def2}
  \f{plus} &= \lambda m.\lambda n.\lambda f.\lambda x.n\,f\,(m\,f\,x)
\end{align}
%
They differ in the order in which the arguments $m$ and $n$
are applied to get the final result.
%
In this case, we can give an alternative specification based
on the evaluation of \f{plus} seen as a recursively-defined
function:
%
\begin{align*}
  \f{?plus}\;0\;y &\equiv y \\
  \f{?plus}\;(\f{succ}\;x)\;y &\equiv \f{succ}\;(\f{?plus}\;x\;y)
\end{align*}
%
where \f{succ} is the successor function defined as:
%
\begin{align*}
  \f{succ} = \lambda n. \lambda f. \lambda x. f\,(n\,f\,x)
\end{align*}

This specification does not just fix the order in which $m$
and $n$ are evaluated but also rule out various spurious
results which satisfy the incomplete specification given in
\cref{sec:io}.
%
However, checking conformance with this sort of
specifications is in general more challenging because it
requires $\eta$-conversion.
%
To see why consider the evaluation of the term in the first
equivalence when we expand \f{plus} with the definition in
\labelcref{eq:plus-def}.
%
Under normal order, $\f{plus}\;0\;y$ evaluates to $\lambda
f.\lambda x. y\,f\,x$ which is only $\eta$-equivalent to
$y$.

% We will be using input-output examples as our specification, so the
%   behavioral constraints of our synthesis problem are simply the
%   evaluation of our examples for a given term or function.
% For instance, the input-output examples to synthesize \textit{and}
%   will look like $?and \, true \, true = true$,
%   $?and \, false \, true = false$,
%   $?and \, true \, false = false$, and
%   $?and \, false \, false = false$, where the $?$ before \texttt{and}
%   signifies the function we are synthesizing.
% Note that because there is no structural difference in untyped lambda
%   calculus between functions and terms, there is no need to differentiate
%   the two in our synthesizer.
% Additionally, we are using a bottom-up search strategy with some
%   special limitations on the form which are described in detail in
%   section \ref{sec:optimizations}.
% We will generate terms from the untyped lambda calculus grammar, with
%   the additional constraint that they must be in normal form.
% Additionally, we will be generating terms using De Bruijn indices rather
%   than with explicit variables names, in order to make evaluation of the
%   terms easier (largely so we don't have to worry about alpha-renaming).


\lab{Applications}{Filtering and Convolution}{Filtering and Convolution}
\objective{This lesson demonstrates the use of the Fourier Transform for cleaning up noisy signals and finding convolutions of signals.}

\section*{Cleaning up a noisy signal}

Listen to \texttt{Noisysignal1.wav} (see Figure \ref{noisysignal}). This is a mono recording of a (probably familar) voice with some annoying noise over it. Fortunately for us, this noise is ``colored", i.e. it only occurs over a limited range of frequencies. This makes it possible for us to remove it fairly easily without unduly damaging the underlying signal.
\begin{figure}[ht]\caption{Noisy signal}\label{noisysignal}\centering\includegraphics[width=\textwidth]{noisy}\end{figure}
How do we begin? First we examine the DFT of the signal in order to identify which frequencies make up the noise (see Figure \ref{noisyspec}).
\begin{figure}[ht]\caption{Spectrum of noisy signal}\label{noisyspec}\centering\includegraphics[width=\textwidth]{noisyspec}\end{figure}
We notice a large band from 10000 to 20000 units which seems to be intruding in the spectrum. This is our noise. (The fact that it begins and ends so abruptly in the spectrum, instead of tapering off gradually, should be a clue that this noise was produced artificially, by the author of this lab, rather than arising in a natural way in the recording or transmission process.) To remove the noise, we simply zero out this part of the DFT and then take the inverse DFT (using the function \texttt{spfft.ifft}) to get our cleaned-up signal. But we must remember to zero out the corresponding part in the right half of the DFT as well, in order to ensure that the inverse DFT of the result is a real-valued signal.

It is critical that the values zeroed in the right half of the DFT be precisely the mirror images of the zeroed in the left half: i.e., assuming the DFT of the signal is stored in the vector \texttt{fsig}, if we set \texttt{fsig[j]} to zero then we should also set \texttt{fsig[-j]} to zero. If we are off on the index even just by one, then the result will not be a real valued signal and thus will not be playable.

So, we do the whole thing in Python like this:
\begin{lstlisting}
rate,data = wavfile.read('Noisysignal1.wav')
fsig = sp.fft(data,axis = 0)
for j in xrange(10000,20000):
    fsig[j]=0
    fsig[-j]=0

newsig=sp.ifft(fsig)
newsig = sp.real(newsig)
newsig = sp.int16(newsig/sp.absolute(newsig).max() * 32767)
\end{lstlisting}

Now we can either save the resulting cleaned-up signal \texttt{newsig} to a \texttt{.wav} file using \texttt{wavwrite}. If we plot it, the individual syllables are now visible in the waveform (see Figure \ref{cleansignal}).

\begin{figure}[ht]\caption{Cleaned-up \texttt{Noisysignal1.wav} }\label{cleansignal}\centering\includegraphics[width=\textwidth]{Cleanedsignal}\end{figure}

\begin{problem}
Listen to \texttt{Noisysignal2.wav}. You will probably just hear noise. However, there is a signal behind it. Remove the noise using the technique described above. In order to make the cleaned-up signal audible. What does the voice say? Who is the speaker? (If you don't know the answer to this last question, try a quick Google search.)
\end{problem}

\section*{Filtering and Convolution}

We have already seen how the DFT can be used to filter out certain bands of frequencies from a signal. In this section we will look at how the DFT can be used to carry out another type of filtering effect. To begin, suppose we have a recording of musical piece played in, say, a small carpeted room with essentially no acoustics, and suppose we would like to apply an effect to make it sound as if the piece were played in a large concert hall (or some other room). How could we do that?

The first thing we need is a recording of the so-called \emph{impulse response} of the room whose acoustics we are trying to imitate. This is a recording of how the room responds to a short pulse of sound. Effective ways of producing a loud sound approximating a pulse include firing a (blank) gunshot, popping a balloon, or, if neither of those are available, clapping the hands one time. You have probably noticed (or can imagine) that if you pop a balloon in large room, although the sound of the actual pop only lasts a few milliseconds, you can hear the sound echoing about the room for up to several seconds. This echoing sound is called the impulse response of the room. Actually, it is somewhat imprecise to refer to \emph{the} impulse response of the room, since the recorded response can significantly depend both on the location of device producing the pulse and on location of the listener (or recording device) in the room.

Now, the idea is that if we know how the room responds to a pulse, then we can reconstruct how the room will respond to any sound. This is because any sound can be considered to be a series of pulses of varying amplitudes and polarity (i.e., they may be positive or negative). Each sample of sound can be considered as a pulse, even though the ear will not perceive it that way, because the pulses are so close together. Now, we make the assumption that the room's response is linear, meaning that a pulse of $k$ times the amplitude will produce a response of $k$ times the amplitude but otherwise identical; such an assumption is generally reasonably accurate, assuming the pulse is not extremely loud. To calculate how the room responds to a complex sound consisting of millions of samples, we simply add up its response to each sample of the sound, each such response simply being a scaled copy of the impulse response which we recorded.

Thus, in our simulation, each sample of our sound (of which there are usually 44100 per second) triggers a new scaled copy of the room's impulse response. If the impulse response is several seconds long, that means there may be a hundred thousand or more scaled copies of the impulse response sounding at once, which must all be mixed together to produce the final sound. This may be starting to seem computationally infeasible or at least very difficult. The key is to recognize that this process can be described as a convolution: namely, the final sound is simply the convolution of the our original sound with the impulse response. We can calculate convolutions quickly using the convolution theorem which says

\[\mathcal{F}(f \ast g) = (\mathcal{F} f) (\mathcal{F} g)\]

where $\mathcal{F}$ is the Fourier Transform and $\ast$ is convolution.

Thus we calculate the convolution of two arrays by simply taking the fourier transform of each, multiplying them pointwise, and then taking the inverse transform.

\begin{problem}
\emph{This Problem is optional.  If the instructor does not require it then students may use the provided \texttt{balloon.wav} file which contains the sound of a balloon pop in a large room.}

Find a large room or area with good acoustics, and record (an approximation to) its impulse response using a balloon pop. To record the sound, you will want to use at least a decent microphone. You may want to record it using the program Audacity \footnote{Audacity is free software and may be downloaded at http://audacity.sourceforge.net} and a laptop. If you use a unidirectional microphone, be sure the microphone is pointing at the balloon when you pop it, so that the direct sound from the pop is picked up. (If you don't, the result will still be okay. It's just that after we do the convolution, it will probably sound somewhat distant, as if we are at the back of the room, where we can't hear the music directly, only through the reverberation of the room.)
If you've chosen a good room, the response should be audible for at least a full second.

Include a plot of both the waveform and spectrum of the impulse response you recorded.
\end{problem}

\begin{problem}\label{convolution_problem}
Download and listen to the file \texttt{chopinw.wav}. You will hear a piano being played in a dead room with little or no acoustics. Using the Convolution Theorem, take the convolution of this signal with the impulse response recorded in the previous problem. Describe the resulting sound.

In doing this problem, keep in mind that the Convolution Theorem requires both signals to have the same length; therefore you will need to pad the smaller of your two signals (namely, the impulse response signal) with zeros at the end in order to make it the same size as the other signal. But the convolution of the Convolution Theorem is a circular convolution, which will mean that the room's response to a sound near the end of the signal will wrap around back to the beginning of the signal in the result. In order to avoid this undesired effect, you will want to also pad the signals with additional zeros (at least as many zeros as one less than the size of the impulse response signal).
\end{problem}

In doing the preceding problem, keep in mind that the Convolution Theorem requires both signals to have the same length; therefore you will need to pad the smaller of your two signals (namely, the impulse response signal) with zeros at the end in order to make it the same size as the other signal. But the convolution of the Convolution Theorem is a circular convolution, which will mean that the room's response to a sound near the end of the signal will wrap around back to the beginning of the signal in the result. In order to avoid this undesired effect, you will want to also pad the signals with additional zeros (at least as many zeros as one less than the size of the impulse response signal).

In some instances, a circular convolution is actually desirable. For instance, an interesting effect is achieved by taking the circular convolution of a long segment of white noise with some other (shorter) sound. We can create white noise using scipy's \texttt{rand} function:
\begin{lstlisting}
#samplerate = 22050
#noise = sp.int16(sp.random.randint(-32767,32767,samplerate*10)) # Create 10 seconds of mono white noise
\end{lstlisting}

\begin{problem}
Create white noise in this way this and listen to the resulting sound (CAUTION: Turn your volume way down.  It may be very very loud).  This kind of noise is called ``white" because it contains all frequencies (below the Nyquist frequency) with the same strength, or rather, with the same expected strength (since the amplitude of a specific frequency is a matter of chance). In order to see this, plot the spectrum of the noise.
\end{problem}

Now we are going to take the circular convolution of this noise with some other sound. For instance, let's use \texttt{Tada.wav}. The result is in \texttt{tada-conv.wav}. We notice that the original short sound has been sustained to an indefinite length. The result is not a set of static tones, but rather a rich sound which preserves not only the tones, but the texture, of the original sound; you can hear different tones fluctuating randomly in amplitude over time. If you were to play this \texttt{tada-conv.wav} on repeat, you would find that, because we used a circular convolution, the sound loops seamlessly from the end back to the beginning; however, most sound players are not capable of doing this properly, so you will probably hear a break in the sound. To demonstrate the ``seamlessness", we can paste together three copies of the sound consecutively:

\begin{lstlisting}
rate, sig = wavfile.read('tada-conv.wav')
sig = sp.append(sig,sig)
sig = sp.append(sig,sig)
\end{lstlisting}

Listen to the resulting sound, and notice that we are not able to identify where the sound loops back to the beginning, because there is no break or click.

\begin{problem}
Record yourself singing a few notes (or, feel free to produce some other sound another way). Take the circular convolution of white noise with this recording. Now do it again using stereo white noise, which you can create, e.g., by doing \texttt{noise = sp.float16(sp.random.randint(-32767,32767,(samplerate*10,2)))}. It's no problem that your original recording will probably be mono; just make the left and right channels duplicate in the recording (but: be sure to use different left and right channels for the white noise). Can you hear any difference between the mono and stereo versions of the result?
\end{problem}

Feel free to play around with this. The file \texttt{guitar-conv.mp3} is a collage of sounds created using this technique (mostly using guitar samples). You could probably think of other lots of other things you can do with this.

%\begin{problem}
%Using the algorithm described in the lecture notes, implement your own Inverse Fast Fourier Transform (IFFT) in MATLAB from scratch. You may assume that $N$ is a power of 2. Test it on several random vectors and make sure the results match those of MATLAB's \texttt{ifft}. It should have $O(N\log N)$ running time. Now, by adjusting your IFFT, implement the FFT, and redo Problem \ref{convolution_problem} using the FFT and IFFT which you have implemented. Use \texttt{tic} and \texttt{toc} to measure how long it takes to finish, against how long it took using MATLAB's \texttt{fft} and \texttt{ifft}.
%\end{problem}
%
%It is possible to implement your \texttt{fft} and \texttt{ifft} using vector operations, which will speed things up; however, one could hardly expect to beat MATLAB's \texttt{fft} and \texttt{ifft}, which are implemented by calling the extremely well-optimized FFTW library (Fastest Fourier Transform in the West).


\documentclass{article}

\usepackage[letterpaper, margin=1.3cm]{geometry}
\usepackage[utf8]{inputenc}
\usepackage{siunitx}
\usepackage{mathtools}
\usepackage{multicol}

\title{ECE 240 Assignment 8}
\author{Michael Kwok}
\begin{document}

\maketitle
\begin{multicols}{2}
\section*{1}
\begin{align*}
F(s) &= \int_{0^-}^\infty f(t)e^{-st}dt\\
&= \int_{0^-}^\infty \left(e^{2t} -2e^{-t}\right)e^{-st} u(t)dt\\
&= \int_{0}^\infty e^{-(s-2)t}dt - 2\int_{0}^\infty e^{-(1+s)t}dt\\
&= \left[\frac{1}{s-2}e^{-(s-2)t}\right]^\infty_0 + \left[ \frac{1}{1+s}e^{-(s+1)t}\right]^\infty_0
\end{align*}

The first expression is only convergent when $s-2 > 0$. Since $s = \sigma + j\omega$, the region of convergence will be calculated by the following:

\begin{align*}
    \sigma -2 &> 0\\
    \sigma &> 2
\end{align*}

So assuming $\sigma > 2$,

\begin{align*}
    F(s) &= \frac{1}{s-2} \lim_{t \to \infty} \left( e^{-(s-2)t} -1 \right)\\
    &+ \frac{2}{s+1} \lim_{t \to \infty} \left( e^{-(s+1)t} -1 \right)\\
    &= \boxed{\frac{1}{s-2} - \frac{2}{s+1}}
\end{align*}
\section*{2}
\begin{align*}
    f(t) &= \int_0^1 te^{-t}dt + \int_1^\infty e^{-t}dt\\
    &= e^{-1}\left(\frac{1}{2} - 0\right) + \lim_{t\to\infty}\left(-e^{-t} + e^{-1}\right)\\
    &= \frac{1}{2} e^{-1} + e^{-1}\\
    &= \boxed{\frac{3}{2}e^{-1}}
\end{align*}
\section*{3}
\subsection*{a}
\begin{align*}
    f_1(t) &= \sin\left[\omega_0(t-\tau)\right]u(t-\tau)\\
    F_1(s) &= \boxed{\frac{\omega_0}{s^2 + \omega_0^2} e^{-s\tau}}
\end{align*}
\subsection*{b}
\begin{align*}
    f_1(t) &= \sin\left[\omega_0(t-\tau)+\omega_0\tau\right]u(t-\tau)\\
    &= \sin\left[\omega_0(t-\tau)\right]\cos(\omega_0\tau)u(t-\tau)\\
    &+ \cos\left[\omega_0(t-\tau)\right]\sin(\omega_0\tau)u(t-\tau)\\
    F_1(s) &= \cos(\omega_0\tau)\frac{\omega_0}{s^2 + \omega_0^2} e^{-s\tau}\\
    &+ \boxed{\sin(\omega_0\tau)\frac{s}{s^2 + \omega_0^2} e^{-s\tau} }
\end{align*}
\section*{4}
\begin{align*}
    g(t) &= f(t) + f(t-T_0) + f(t-2T_0) + \ldots\\
    G(s) &= F(s) + F(s)e^{-sT_0} + F(s)e^{-s2T_0} + \ldots\\
\end{align*}
Let $x = e^{-sT_0}$
\begin{align*}
    G(s) &= F(s)\left(1+x+x^2+x^3+\ldots\right)\\
    &= \frac{F(s)}{1+x}\\
    &= \boxed{\frac{F(s)}{1+e^{-sT_0}}}
\end{align*}
\section*{5}
\begin{align*}
    f(t) &= t\left[ u(t) - u(t-2) \right] + \left( -2t+6 \right)\left[ u(t-2) - u(t-3) \right]\\
    f(0^{-}) &= 0\\
    f'(t)&= u(t)-u(t-2)-2u(t-2)+2u(t-3)\\
    &+6\delta(t-2)-6\delta(t-3)\\
    \mathcal{L}\{ f'(t)\} &= sF(s)-f(0^-)\\
    sF(s) &= \frac{1}{s} - \frac{1}{s} e^{-2s} - \frac{2}{s} e^{-2s} + \frac{2}{s} e^{-3s} +6e^{-2s} -6e^{-3s}\\
    F(s) &= \boxed{\frac{1}{s^2}-\frac{3}{s^2}e^{-2s}+\frac{2}{s^2}e^{-3s} + \frac{6}{s}e^{-2s} - \frac{6}{s}e^{-3s}}
\end{align*}
\end{multicols}
\end{document}


\section{Finite Sets}

\begin{definition}
  A set is said to be \textbf{finite} if there is a bijective correspondence of
  $A$ with some section of the positive integers. That is, $A$ is finite if it
  is empty or if there is a bijection
  \begin{equation}
    f:A \to \pbrac{1, \dots, n}
  \end{equation}
  for some positive integer $n$. In the former case, we say that $A$ as
  \textbf{cardinality 0}, and in the latter case, we say that $A$ has
  \textbf{cardinality $n$}.
\end{definition}

\section*{Exercises}

\bx{
  \ea{
    \item There are $4 \times 3 \times 2 = 24$ injective mappings,
    \begin{align*}
      (1, 1), (2, 2), (3, 3)\\
      (1, 1), (2, 2), (3, 4)\\
      (1, 1), (2, 3), (3, 2)\\
      (1, 1), (2, 3), (3, 4)\\
      (1, 1), (2, 4), (3, 2)\\
      (1, 1), (2, 4), (3, 3)\\
      (1, 2), (2, 1), (3, 3)\\
      (1, 2), (2, 1), (3, 4)\\
      (1, 2), (2, 3), (3, 1)\\
      (1, 2), (2, 3), (3, 4)\\
      (1, 2), (2, 4), (3, 1)\\
      (1, 2), (2, 4), (3, 3)\\
      (1, 3), (2, 1), (3, 2)\\
      (1, 3), (2, 1), (3, 4)\\
      (1, 3), (2, 2), (3, 1)\\
      (1, 3), (2, 2), (3, 4)\\
      (1, 3), (2, 4), (3, 1)\\
      (1, 3), (2, 4), (3, 2)\\
      (1, 4), (2, 1), (3, 2)\\
      (1, 4), (2, 1), (3, 3)\\
      (1, 4), (2, 2), (3, 1)\\
      (1, 4), (2, 2), (3, 3)\\
      (1, 4), (2, 3), (3, 1)\\
      (1, 4), (2, 3), (3, 2)
    \end{align*}
    \item $10 \cdot 9 \cdot \dots \cdot 3 = 1814400$ \dots, you can tell this is not a fun time.
  }
}

\bx{
  AFSOC $A$ is finite. Then $B$ must be finite since it is a subset of $A$, but this is a contradiction since $b$ is not finite by assumption.
}

\bx{
  Let the propr subset be $X^\omega - {0, 0, \dots}$.
  Let the bijection be
  \begin{align*}
    f(000000\dots) &= 1000000\dots\\
    f(100000\dots) &= 0100000\dots\\
    f(110000\dots) &= 0010000\dots\\
    &\cdots
  \end{align*}
  It’s just binary written in R to L significant digits, + 1 to shift for the missing $000\dots$.
}

\bx{
  \ea{
    \item Induction from empty set as base case. Adding an element gives you two
    cases, either the new element is the largest, or the existing largest
    element remains the largest.
    \item \TODO: What is an order type? I think the idea here is map $A$ into $\mathbb{Z}$ in the order that the elements are in $A$, and then they will have the order type of the $\Zp$.
  }
}

\bx{
  No. $A = \emptyset$ and $B$ could be infinite.
}

\bx{
  \ea{
    \item $X^n$ is basically an $n$-tuple showing whether or not an element of A is included in the set, so you can make that bijection to $\mathcal{P}(A)$.
    \item $\mathcal{P}(A)$ has a bijection with $X^n$, which is finite, so $\mathcal{P}(A)$ is finite as well.
  }

  \label{chap1:sec6:prob6}
}

\bx{
  Consider $C = A \times B$. Any function $f$ is some subset of $C$. Then the
  set of all functions is a subset of $\mathcal{P}(C)$, which from \ref{chap1:sec6:prob6}
  we know is finite since $C$ is finite.
}


\subsection{Robust principal component analysis}

\subsubsection{Robust PCA}

Robust PCA can be used to deal with corrupted data, such as corrupted image data.

Rather than data \(X\) we have \(M=L_0+S_0\) where \(L_0\) is what we want to recover (and is low rank), and \(S_0\) is noise (and sparce).

In video footage, \(L_0\) can correspond to the background, while \(S_0\) corresponds to movement.


\chapter[Residues]{Evaluating Contour Integrals Using Residues}
\section{Motivation}
Let $\mathcal{C}$ be a simple, closed anticlockwise contour.  In the previous section we saw how certain integrals of the form
\[
\int_{\mathcal{C}} f
\]
may be calculated using Cauchy's Integral Formula, when $f$ is a function holomorphic on region $\mathcal{R} \backslash \set{z_0}$ with $z_0$ a point enclosed by $\mathcal{C}$.  However, this does not always work, as the following example illustrates.
\begin{example}
\label{e:gdef}
Cauchy's Integral Formula does not allow us to compute $\int_{\cont} f$, where $\mathcal{C}$ is the anticlockwise unit circle and $f$ is the function defined by
\[
f(z) = \frac{1}{z^2+\frac{1}{4}} = \frac{1}{(z+\frac{i}{2})(z-\frac{i}{2})},
\]
as there are two points enclosed by $\cont$ at which $f$ is not holomorphic.
\begin{blankbox}
\begin{center}
\altgraphics[scale=1]{ch6_shrink1_full}{ch6_shrink1}
\end{center}
Draw the line $L=[-1,1]$ to get two new closed anticlockwise contours $\mathcal{C}_1$ and $\mathcal{C}_2$, with $\mathcal{C}_1$ enclosing $i/4$ and $\mathcal{C}_2$ enclosing $-i/4$.  Then
\[
\int_{\mathcal{C}} f = \int_{\mathcal{C}_1} f + \int_{\mathcal{C}_2} f,
\]
since the integral along $L$ in $\mathcal{C}_1$ and the integral along $\tilde{L}$ in $\mathcal{C}_2$ cancel.  We can then use Cauchy's Integral Formula to evaluate each of $\displaystyle \int_{\mathcal{C}_1} f$ and $\displaystyle \int_{\mathcal{C}_2} f$ separately.
\end{blankbox}
\end{example}
We will assume without proof that the method we have just described will extend to the case when there is any finite number of points enclosed by $\mathcal{C}$ at which $f$ is not holomorphic:

\begin{theorem}[Generalised Shrinking Contour/ Deformation Theorem; proof non-examinable]
\label{t:gdef}
Let $\mathcal{R}$ be a simply connected region, $\mathcal{C}$ a simple, closed anticlockwise contour in $\mathcal{R}$, and $z_1,z_2,\ldots,z_n$ a finite collection of points that are enclosed by $\mathcal{C}$.  If $f$ is holomorphic on $\mathcal{R} \backslash \set{z_1,\ldots,z_n}$, then
\[
\int_{\mathcal{C}} f = \int_{\mathcal{C}_1}f+ \ldots + \int_{\mathcal{C}_n} f,
\]
where each $\mathcal{C}_j$ is a closed simple anticlockwise circular contour enclosing $z_j$ and no other $z_k$.
\end{theorem}

\begin{figure}[H]
\centering
\includegraphics[scale=1]{ch6_residues2}
\caption{A contour $\mathcal{C}$ enclosing points $z_1,z_2,z_3$ and $z_4$ at which $f$ is not holomorphic.  The integral of $f$ along $\mathcal{C}$ is determined by the integrals along the circular contours enclosing these points.}
\end{figure}

 Theorem~\ref{t:gdef} allows us, as least in principle, to evaluate each $\int_{\mathcal{C}_j} f$ separately to obtain $\int_{\mathcal{C}} f$.  Thus we have essentially reduced the problem to that of evaluating integrals where there is one point enclosed by the given contour at which $f$ is not holomorphic.

 This approach will not always work, however.

\begin{example}
Let $f: \C \backslash \set{0} \to \C$ be defined by
\[
f(z) = \frac{\sin(z)}{z^2},
\]
and let $\mathcal{C}$ be any anticlockwise closed circular contour centred at $0$.
We cannot write
\[
\frac{\sin (z)}{z^2} = \frac{g(z)}{(z-z_0)}
\]
for any $z_0$ enclosed by $\mathcal{C}$ and $g$ holomorphic on a simply connected region containing $\mathcal{C}$.  Indeed the obvious choice would be to take $z_0=0$, but then we would have to define $g(z) = \dfrac{\sin(z)}{z}$, which still fails to be holomorphic on a suitable region.
\end{example}

\section{Singulairities of complex functions}
Theorem~\ref{t:gdef} shows that when calculating the integral of $f$ along a simple, anticlockwise, closed contour $\mathcal{C}$, the value $\int_{\mathcal{C}} f$ is in some sense depends only on those points at which $f$ is not holomorphic.  Therefore, we shall study these points in more detail.

\begin{definition}
A function $f$ has an \emph{isolated singularity} at the point $z_0$ if for some $r>0$, $f$ is holomorphic on a punctured disc $D'(z_0,r)$ but not on the (unpunctured) open disc $D(z_0,r)$.
\end{definition}
Note that if $f$ has an isolated singularity at $z_0$, it may be the case that $f$ is not defined at $z_0$, or $f$ is defined at $z_0$ but not differentiable there.

\begin{example}
The function $f$ defined by
\[
f(z) = \frac{1}{z^4+1}
\]
has isolated singularities at the points $e^{i\pi/4},\ e^{3i\pi/4},\ e^{5i\pi/4},\ e^{7i\pi/4}$; in each case, take $r = \frac{1}{4}$ for example.
\begin{center}
\includegraphics[scale=1]{ch6_poles1}
\end{center}
%\vspace*{2cm}

\end{example}
\begin{example}
The function $f$ defined by
\[
f(z) = \frac{\sin (z)}{z^2}
\]
has an isolated singularity at $0$.
\end{example}
\begin{example}
The Principal Logarithm function $\Log : \C \backslash \set{0}$ defined by
\[
\Log (z) = \log(\abs{z}) + i \Arg (z),
\]
is not holomorphic at any point on the negative real axis.  No such point is an isolated singularity of $\Log$
\end{example}
\begin{blankbox}
\begin{center}
\includegraphics[scale=0.5]{ch6_non_iso_full}
\end{center}
Indeed, if $t \leq 0$ is a point on the negative real axis and $r>0$, then $D'(t,r)$ contains points at which $\Log$ is not holomorphic; $z=t-\frac{r}{2}$ for example.
%\vspace*{3cm}
\end{blankbox}
\begin{definition}
Let $f$ be continuous on $\mathcal{R} \backslash \set{z_0}$. We say that $\displaystyle \lim_{z \to z_0} f(z) = \infty$ if given any $M>0$ there is some $r>0$ such that
\[
\abs{f(z)}>M \text{ for all } z \in D'(z_0,r).
\]
\end{definition}
\begin{example}

Consider the function
\[
f(z) = \frac{1}{(z-2i)^3}
\]
which has an isolated singularity at $2i$.  We will show that $\displaystyle \lim_{z \to 2i} f(z) = \infty$.
\end{example}
\begin{solution}

Indeed, given $M>0$, let $r=M^{-1/3}$.  Then
\begin{align*}
0 < \abs{z-2i} < r \Rightarrow \abs{f(z)} &= \abs{\frac{1}{(z-2i)^3}}\\
& = \frac{1}{\abs{z-2i}^3} \\
& > \frac{1}{(M^{-1/3})^3} = M.
\end{align*}
\end{solution}
In most of the examples that we consider, $\displaystyle \lim_{z \to z_0} f(z) = \infty$ will occur whenever evaluating $f$ at $z_0$ would involve division by $0$ (except when $\frac{0}{0}$ would appear).  We shall make this more precise shortly.

\begin{definition}
A function $f$ with an isolated singularity $z_0$ is said to have a \emph{pole at $z_0$} if
\[
\lim_{z \to z_0} f(z) = \infty.
\]
Moreover, for $n \geq 1$, $f$ is said to have a \emph{pole of order $n$} at $z_0$ if for some $r>0$, $f$ can be represented in the form
\[
f(z) = \frac{g(z)}{(z-z_0)^n}\quad\text{ for all } z \in D'(z_0,r)
\]
where $g$ is holomorphic on $D(z_0,r)$ and $g(z_0) \neq 0$.
\end{definition}
Note that the representation
\[
f(z) = \frac{g(z)}{(z-z_0)^n}
\]
need not be valid everywhere on the domain of $f$, only inside the `small' disc $D(z_0,r)$.  The function $g$, unlike $f$, is both defined and differentiable at $z_0$.

\begin{example}
Let us return to the example of
\[
f(z) = \frac{1}{1+z^4}
\]
and investigate the pole $z_1 = e^{i\pi/4}$ of $f$.
\end{example}
\begin{solution}
Denote by $z_2,z_3$ and $z_4$ the other complex $4^{th}$ roots of $-1$, and let $g$ be the function defined by
\[
g(z) = \frac{1}{(z-z_2)(z-z_3)(z-z_4)}.
\]
Then $g$ is holomorphic on $\C \backslash \set{z_2,z_3,z_4}$, and in particular, holomorphic on $D(z_1,\frac{1}{2})$ for example (the precise value of $r>0$ is not important).
\begin{center}
\includegraphics[scale=1]{ch6_poles2}
\end{center}
Moreover, $g(z_1) \neq 0$ and
\[
f(z) = \frac{g(z)}{z-z_1} \quad \text{ for all }\quad z \in D'(z_1,\tfrac{1}{2}).
\]
This shows that $f$ has a pole of order $1$ at $z_1$.
\end{solution}

%\vspace*{5cm}

\begin{example}
\label{e:poles2}
Let us investigate the poles of  the function
\[
f(z) = \frac{1}{(z^2+9)^2}.
\]
\end{example}
\begin{solution}
%\vspace*{12cm}
Using the factorisation
\[
z^2+9 = (z+3i)(z-3i)
\]
we have
\[
f(z)= \frac{1}{(z+3i)^2(z-3i)^2}
\]
so that $f$ has isolated singularities at $\pm 3i$.

If we define the function $g_1$ by
\[
g_1(z) = \frac{1}{(z+3i)^2}
\]
then $g_1$ is holomorphic on $\C \backslash \set{-3i}$ and in particular, holomorphic at $3i$ (that is, holomorphic on some open disc centred at $3i$, for example, $D(3i,1)$).  Since
\[
g_1(3i) = - \frac{1}{36} \neq 0 \quad\text{ and }\quad f(z) = \frac{g_1(z)}{(z-3i)^2} \quad\text{ for } z \in D'(3i,1)
\]
we see that $f$ has a pole of order $2$ at $z=3i$.

Similarly, by considering the function $g_2$ defined by
\[
g_2(z) = \frac{1}{(z-3i)^2}
\]
we see that $g_2$ is holomorphic and nonzero at $z=-3i$ and
\[
f(z) = \frac{g_2(z)}{(z+3i)^2}\quad\text{ for }\quad z \in D'(-3i,1)
\]
so that $f$ has a pole of order $2$ at $z=-3i$ also.
\end{solution}

%\newpage
\section{The Residue Theorem}

\begin{definition}
Let $f$ have an isolated singularity at $z_0$, then we define the \emph{residue of $f$ at $z_0$}, denoted by $\Res (f;z_0)$, to be
\[
\Res (f;z_0) = \frac{1}{2\pi i} \int_{\mathcal{C}} f
\]
where $\mathcal{C}$ is an anticlockwise simple closed contour which contains $z_0$ and no other singularities of $f$, and lies inside a region in which $f$ is holomorphic.
\end{definition}
If $f$ has an isolated singularity at $z_0$, then such a contour $\mathcal{C}$ can always be found.  Indeed, we know that $f$ is holomorphic on the punctured disc $D'(z_0,r)$ for some $r>0$.  Thus we may take $\mathcal{C}$ to be the anticlockwise circle with centre $z_0$ and radius $r/2$.

For example, we have seen before that with $\mathcal{C}$ the anticlockwise unit circle,
\[
\int_{\mathcal{C}} \frac{1}{z}\ dz=2\pi i \quad \text{ and} \quad \int_{\mathcal{C}} \frac{1}{z^2}\ dz = 0.
\]
Thus for the functions $f$ and $g$ defined by $f(z)=\dfrac{1}{z}$ and $g(z) = \dfrac{1}{z^2}$ we have
\[
\Res (f;0)=1 \quad \text{ and } \quad \Res (g;0) = 0.
\]
%\vspace*{7cm}

Theorem~\ref{t:residue} is essentially a reformulation of the Generalised Deformation Theorem (Theorem~\ref{t:gdef}), and generalises Cauchy's Integral Formula.
\begin{theorem}[The Residue Theorem]
\label{t:residue}
Let $\mathcal{R}$ be a simply connected region and let $f$ be a function that is holomorphic on the region $\mathcal{R} \backslash \set{ z_1,z_2,\ldots , z_n}$ and has isolated singularities at the points $z_1,z_2,\ldots,z_n$. If $\mathcal{C}$ is an anticlockwise simple closed contour that lies in $\mathcal{R}\backslash \set{z_1,z_2,\ldots,z_n}$ and encloses the points $z_1,z_2,\ldots,z_n$, then
\[
\int_{\mathcal{C}} f = 2 \pi i \left( \Res (f;z_1)+\Res (f;z_2) + \ldots + \Res (f;z_n ) \right).
\]
\end{theorem}
In other words
\[
\int_{\mathcal{C}} f = 2\pi i \left( \text{ sum of residues of $f$ at isolated singularities enclosed by $\mathcal{C}$ } \right)
\]

\begin{figure}[H]
\centering
\includegraphics[scale=1]{ch6_residues2}
\caption{At each singularity $z_j$, the residue of $f$ at $z_j$ is given by the the integral of $f$ along the small circular contour surrounding $z_j$ (divided by $2\pi i$).  The integral of $f$ along $\mathcal{C}$ is then determined by these residues.}
\end{figure}

We want to use the Residue Theorem to evaluate contour integrals.  The theorem may not look particularly useful yet, since residues themselves are defined to be integrals along circular contours.  However, the following two theorems show us that certain residues may be calculated without performing any integration.

We will need the following Lemma, the proof of which is omitted.
\begin{lemma}[Proof non-examinable]
\label{l:goverh}
Suppose that $z_0 \in \C$, $h$ is holomorphic at the point $z_0$ with $h(z_0)=0$ and $h'(z_0) \neq 0$.  Then there is some $\delta>0$ and a holomophic function $k:D(z_0,\delta) \to \C$ with $k(z) \neq 0$ and $h(z)=(z-z_0)k(z)$ for all $z \in D(z_0,\delta)$.
\end{lemma}
\begin{theorem}[The $g/h$ rule]
\label{t:goverh}
Let $z_0 \in \C,\ r>0$ and let $f$ be holomorphic on $D'(z_0,r)$. Suppose that $f$ can be represented by
\[
f(z) = \frac{g(z)}{h(z)}\quad \text{ for } z \in D'(z_0,r)
\]
where $g$ and $h$ are holomorphic on $D(z_0,r)$, and $g(z_0) \neq 0$, $h(z_0)=0$ and $h'(z_0) \neq 0$. Then
\begin{enumerate}
\item[(i)] The function $f$ has an isolated singularity, and in particular, a  pole of order one at $z_0$.
\item[(iii)] The residue of $f$ at $z_0$ is given by
\[
\Res (f,z_0) = \frac{g(z_0)}{h'(z_0)}.
\]
\end{enumerate}
\end{theorem}
\begin{proof}
By Lemma~\ref{l:goverh} we obtain $\delta>0$ (making $\delta<r$ if necessary) and $k:D(z_0,\delta)\to \C$, with $k$ holomorphic and nonzero on $D(z_0,\delta)$ and
\[
f(z)= \frac{g(z)}{k(z)}\cdot \frac{1}{(z-z_0)} \quad \text{ for all } z \in D'(z_0,\delta).
\]
Since $k(z) \neq 0$ for all $z \in D(z_0,\delta)$ the function $z \mapsto \frac{g(z)}{k(z)}$ is holomorphic on $D(z_0,\delta)$, and since $g(z_0) \neq 0$, $f$ has a pole of order $1$ at $z_0$.

If $\mathcal{C}$ is the anticlockwise circle with centre $z_0$ and radius $r/2$, then Cauchy's Integral Formula gives
\[
\Res (f;z_0) = \frac{1}{2\pi i} \int_{\mathcal{C}} f = \frac{1}{2\pi i} \int_{\mathcal{C}} \frac{g(z)}{k(z)} \cdot \frac{1}{z-z_0}\ dz = \frac{g(z_0)}{k(z_0)}.
\]
But then by the product rule, $h'(z) = k(z)+(z-z_0)k'(z)$ for all $z \in D(z_0,\delta)$, so that $h'(z_0) = k(z_0)$, thus
\[
\Res (f;z_0) = \frac{g(z_0)}{h'(z_0)}.
\]
\end{proof}

\begin{example}
Let us calculate the residues at the poles of the function $f$ defined by
\[
f(z) = \frac{1}{1+z^2}.
\]
\end{example}
\begin{solution}
%\vspace*{12cm}
With $g(z)=1$ and $h(z)=1+z^2$, we have $h(z)=0$ at $z=\pm i$, and $h'(z)=2z \neq 0$ at these points.
Hence $f$ has poles of order $1$ at $z=\pm 1$, with residues
\begin{align*}
\Res (f;i ) & = \frac{g(i)}{h'(i)} = \frac{1}{2i} = - \frac{i}{2} \\
\Res (f;-i) & = \frac{g(-i)}{h'(-i)} = \frac{1}{-2i} = \frac{i}{2}.
\end{align*}

Consider the simple, closed anticlockwise contours $\mathcal{C}_1,\mathcal{C}_2$ and $\mathcal{C}_3$ shown below.
\begin{center}
\begin{tabular}{ccc}
\includegraphics[scale=0.45]{ch6_res1full} & \includegraphics[scale=0.45]{ch6_res2full} & \includegraphics[scale=0.45]{ch6_res3full}
\end{tabular}
\end{center}
By the Residue Theorem, we have
\begin{align*}
\int_{\mathcal{C}_1} f &= 2\pi i \left[ \Res (f;i) \right] = 2\pi i \left[ \frac{-i}{2} \right] = \pi \\
\int_{\mathcal{C}_2} f &= 2\pi i \left[ \Res (f;-i) \right] =2\pi i \left[ \frac{i}{2} \right] = -\pi\\
\int_{\mathcal{C}_3} f &= 2\pi i \left[ \Res (f;i)+\Res(f;-i) \right] = 2\pi i \left[ -\frac{i}{2} + \frac{i}{2} \right] = 0,
\end{align*}
since $\mathcal{C}_1$ encloses the singularity $z=i$ and no others, $\mathcal{C}_2$ encloses the singularity $z=-i$ and no others, and $\mathcal{C}_3$ encloses the singularities $z=i$ and $z=-i$.
\end{solution}
\begin{example}
Let us do the same for the function
\[
f(z) = \frac{-i}{6z^2+13z+6}
\]
\end{example}
\begin{solution}
Let $g(z)=-i$ and
\[
h(z) = 6z^2+13z+6 = (3z+2)(2z+3)
\]
and note that $h(z)=0$ for $z=-\frac{2}{3},-\frac{3}{2}$.  Since $h'(z)=12z+13$, we have
\[
h'(-2/3)  = 5 \quad\text{ and}\quad h'(-3/2) = -5,
\]
both of which are nonzero, so that $f$ has poles of order $1$ at $-\frac{2}{3}$ and $-\frac{3}{2}$ by Theorem~\ref{t:goverh}.  Moreover, by the same result,
\begin{align*}
\Res(f;-2/3) & = \frac{g(-2/3)}{h'(-2/3)} = - \frac{i}{5} \\
\Res (f;-3/2) & = \frac{g(-3/2)}{h'(-3/2)} = \frac{i}{5}.
\end{align*}
\end{solution}
%\vspace*{20cm}
%\begin{absolutelynopagebreak}
\begin{theorem}[The Residue at a pole of order $n$]
\label{t:polen}
Let $f$ have an isolated singularity at $z_0$ which is a pole of order $n$, so that for some $r>0$
\[
f(z) = \frac{g(z)}{(z-z_0)^n} \quad \text{ for } z \in D'(z_0,r),
\]
with $g$ holomorphic on $D(z_0,r)$ and $g(z_0) \neq 0$.  Then
\[
\Res (f,z_0) = \frac{g^{(n-1)}(z_0)}{(n-1)!}.
\]
\end{theorem}
%\end{absolutelynopagebreak}
%\noindent\textit{Proof:}
\begin{proof}
By definition,
\[
\Res (f;z_0) = \frac{1}{2\pi i} \int_{\mathcal{C}} \frac{g(z)}{(z-z_0)^n}\ dz,
\]
where $\mathcal{C}$ is the anticlockwise circle with centre $z_0$ and radius $r/2$.  By Theorem~\ref{t:cauchyd} (Cauchy's Integral Formula for Derivatives),
\[
g^{(n-1)} (z_0) = \frac{(n-1)!}{2\pi i} \int_{\mathcal{C}} \frac{g(z)}{(z-z_0)^n}\ dz,
\]
so that
\[
g^{(n-1)} (z_0) = (n-1)!\ \Res (f;z_0),
\]
or in other words,
\[
\Res (f;z_0) = \frac{g^{(n-1)}(z_0)}{(n-1)!}.
\]
\end{proof}
%\vspace*{10cm}

\begin{example}
\label{e:res3}
%\begin{absolutelynopagebreak}
Let us consider the function $f$ defined by
\[
f(z) = \frac{1}{(z^2+9)^2}.
\]
and calculate the residues at the poles of $f$.
\end{example}
\begin{solution}
%\end{absolutelynopagebreak}
Following Example~\ref{e:poles2}, we define $g_1$ and $g_2$ by
\[
g_1(z) = \frac{1}{(z+3i)^2} \quad\text{ and }\quad g_2(z) = \frac{1}{(z-3i)^2}
\]
so that
\[
f(z) = \frac{g_1(z)}{(z-3i)^2} = \frac{g_2(z)}{(z+3i)^2},
\]
and $g_1$ is holomorphic and nonzero at $z=3i$, $g_2$ is holomorphic and nonzero at $z=-3i$.

Since the poles at $z=\pm 3i$ both have order $2$, Theorem~\ref{t:polen} tells us that
\[
\Res (f;3i) =\frac{g_1'(3i)}{1!} \quad\text{ and } \Res (f;-3i) = \frac{g_2'(-3i)}{1!}.
\]
The required derivatives are
\begin{align*}
g_1'(z) = -2(z+3i)^{-3} & \Rightarrow g_1'(3i) = -2(3i+3i)^{-3} = - \frac{i}{108} \\
g_2'(z) = -2(z-3i)^{-3} & \Rightarrow g_2'(-3i) = -2(-3i-3i)^{-3} = \frac{i}{108},
\end{align*}
Hence
\[
\Res (f;3i) = \frac{-i/108}{1!} = - \frac{i}{108} \quad\text{ and }\quad \Res (f;-3i) = \frac{i/108}{1!}= \frac{i}{108}.
\]
%\vspace*{15cm}

\end{solution}

\begin{example}
%\begin{absolutelynopagebreak}
For the function $f$ defined by
\[
f(z) = \frac{\exp(\pi z)}{(z-i)^3}
\]
find $\Res (f,i)$.  Hence evaluate
\[
\int_{\mathcal{C}} \frac{\exp( \pi z)}{(z-i)^3}\ dz,
\]
where $\mathcal{C}$ is the anticlockwise triangular contour with vertices $2$, $2i$ and $-2$.
\end{example}
%\end{absolutelynopagebreak}
%\vspace*{15cm}
\begin{solution}
The only pole of $f$ is a pole of order $3$ at $z=i$.  With $g$ the holomorphic function defined by $g(z) = \exp(\pi z )$, we have $g'(z) = \pi \exp(\pi z)$ and $g''(z) = \pi^2 \exp(\pi z)$.  By Theorem~\ref{t:polen},
\[
\Res(f;i) = \frac{g''(i)}{2!} = \frac{\pi^2 \exp(\pi i)}{2!} = \frac{\pi^2(-1)}{2} = - \frac{\pi^2}{2}.
\]
Since $\mathcal{C}$ is a simple, closed anticlcokwise contour, and $i$ is the only pole of $f$ enclosed by $\mathcal{C}$, the Residue Theorem gives
\[
\int_{\mathcal{C}} f = 2\pi i \Res (f;i) = 2\pi i \left( -\frac{\pi^2}{2} \right) = -i \pi^3.
\]
\end{solution}

%\newpage
\section{Evaluating Real Integrals using Contour Integration, part 2}
Recall how we used contour integration to evaluate
\[
\int_{-\infty}^{+\infty} \frac{1}{1+x^2}\ dx.
\]
We first considered the complex function $f$ where $f(z)=\dfrac{1}{1+z^2}$, and computed the integral of $f$ along the path $\cont_R = L_R+S_R$ consisting of the line segment $[-R,R]$ and the upper semicircle with centre $0$ and radius $R$, from $R$ to $-R$ via $iR$ (where $R>1$).

We did this using Cauchy's Integral Formula, though we could equally have used the Residue Theorem.  We then showed that
\[
\lim_{R \to \infty} \int_{S_R} f = 0
\]
using The Estimation Lemma.  From this we deduced that
\[
\int_{-\infty}^{+\infty} \frac{1}{1+x^2}\ dx  = \lim_{R \to \infty} \int_{L_R} f = \lim_{R \to \infty} \left( \int_{L_R}f + \int_{S_R} f \right)
= \int_{\cont_R} f.
\]
In fact, the exact same method can be used to evaluate many integrals of the form
\[
\int_{-\infty}^{+\infty} f(x)\ dx,\quad\text{ for example}\quad \int_{-\infty}^{+\infty} \frac{p(x)}{q(x)}\ dx
\]
with $p$ and $q$ real polynomials, where $q$ has no real roots and the degree of $q$ is at least the degree of $p$ plus 2.
\begin{example}
\label{e:realint2}
Let us evaluate
\[
\int_{-\infty}^{+\infty} \frac{1}{(x^2+9)^2}\ dx
\]
using contour integration.
\end{example}
\begin{solution}

We shall follow the approach used in Example~\ref{e:realint1}, by considering the complex function $f$ defined via
\[
f(z) = \frac{1}{(z^2+9)^2}
\]
and the contour $\mathcal{C}_R=L_R+S_R$, where $R>3$, $L_R=[-R,R]$ and $S_R$ is the upper semicircle with centre $0$ and radius $R$ from $R$ to $-R$ via $iR$.

Using the result of Example~\ref{e:res3}, the only singularity of $f$ enclosed by $\mathcal{C}_R$ is at $z=3i$, hence by the Residue Theorem
\[
\int_{\mathcal{C}_R} f = 2\pi i \Res (f;3i) = 2\pi i \left( -\frac{i}{108} \right) = \frac{\pi}{54}
\]
for all $R>3$.

If $z \in S_R$, then $\abs{z}=R$ and hence by the reverse triangle inequality
\[
\abs{z^2+9} \geq \abs{\abs{z^2}-9} = \abs{\abs{z}^2-9} = R^2-9,
\]
so that for all such $z$ we have $\abs{z^2+9}^2 \geq \left( R^2-9 \right)^2$.  Thus for all $z \in S_R$,
\[
\abs{\frac{1}{(z^2+9)^2}} = \frac{1}{\abs{z^2+9}^2} \leq \frac{1}{\left(R^2-9 \right)^2}.
\]
The Estimation Lemma gives
\[
\abs{ \int_{\mathcal{C}_R} \frac{1}{(z^2+9)^2}\ dz } \leq \frac{1}{\left(R^2-9 \right)^2} \cdot \pi R,
\]
hence $\displaystyle \int_{\mathcal{C}_R} f \to 0$ as $R \to \infty$ as before.

Parameterising $L_R$ using $\gamma:[-R,R] \to \C$, $\gamma(t)=t$ we have $\gamma'(t)=1$ and so
\[
\int_{L_R} f = \int_{-R}^{R} \frac{1}{(t^2+9)^2}\ dt.
\]
Hence
\begin{align*}
\frac{\pi}{54} & = \int_{\mathcal{C}_R} f  && (\forall\ R>3) \\
& = \lim_{R \to \infty} \int_{\mathcal{C}_R} f && \\
& = \lim_{R \to \infty} \left( \int_{L_R} f \right) + \lim_{R \to \infty} \left( \int_{S_R} f \right) && \\
& = \lim_{R \to \infty} \int_{-R}^{+R} \frac{1}{(t^2+9)^2}\ dt\ +0&& \\
 &= \int_{-\infty}^{+\infty} \frac{1}{(t^2+9)^2}\ dt. &&
\end{align*}
(Note that it does not matter whether we call the variable of integration $x$ or $t$).
\end{solution}
\begin{example}
Use Example~\ref{e:realint2} to deduce the value of
\[
\int_0^{\infty} \frac{1}{(x^2+9)^2}\ dx.
\]
\end{example}
\begin{solution}
Since $\displaystyle f(x) = \frac{1}{x^2+9}$ is an even function (i.e. $f(-x)=f(x)$) it follows that
\[
\int_0^{\infty} \frac{1}{(x^2+9)^2}\ dx  = \frac{1}{2} \int_{-\infty}^{\infty} \frac{1}{(x^2+9)^2}\ dx = \frac{\pi}{108}.
\]
\end{solution}

We showed in example~\ref{e:trig2} that for a rational function $R$ of two real variables
\[
\int_0^{2\pi} R ( \cos(t), \sin(t))\ dt = \int_{\mathcal{C}} R \left( \frac{z+z^{-1}}{2} , \frac{z-z^{-1}}{2i} \right) \cdot \frac{1}{iz}\ dz
\]
where $\mathcal{C}$ is the anticlockwise circle $\set{z \in \C: \abs{z}=1}$.

\begin{example}
Evaluate
\[
\int_0^{2\pi} \frac{\sin (t)}{5-4\sin(t)}\ dt
\]
using contour integration.
\end{example}
\begin{solution}
For $z = \exp (it) \in \mathcal{C}$, define $f$ via
\begin{align*}
f(z) = R \left( (z+z^{-1})/2,(z-z^{-1})/(2i) \right) \cdot \frac{1}{iz}\ & = \frac{[z-z^{-1}]/(2i)}{5-4[z-z^{-1}]/(2i)}\cdot \frac{1}{iz} \\
& = \frac{z-z^{-1}}{10i-4z+4z^{-1}} \cdot \frac{1}{iz} \\
& =  \frac{z^2-1}{-4z^2+10iz+4} \cdot \frac{1}{iz} \\
& =  \frac{i(z^2-1)}{z(4z^2-10iz-4)}\\
& = \frac{1}{2} \cdot \frac{i(z^2-1)}{z(z-2i)(2z-i)}
\end{align*}
so that
\[
\int_0^{2\pi} \frac{\sin (t)}{5-4\cos(t)}\ dt = \int_{\mathcal{C}}  f dz.
\]
The function $f$ has simple poles at $0, i/2$ and $2i$, and  the first two of these are enclosed by $\mathcal{C}$.  Writing
\[
g(z)=i(z^2-1) \quad \text{ and } h(z) = 4z^3-10iz^2-4z
\]
we have $f=g/h$ and $h'(z)=12z^2-20iz-4$, hence by the $g/h$ rule
\begin{align*}
\mathrm{Res} (f;0) & = \frac{g(0)}{h'(0)} = \frac{-i}{-4} = \frac{i}{4} \\
\mathrm{Res} (f;i/2) & = \frac{g(i/2)}{h'(i/2)} = \frac{i((\tfrac{i}{2})^2-1)}{12(\tfrac{i}{2})^2-20i(\tfrac{i}{2})-4} =\frac{-5i}{12}.
\end{align*}
The Residue Theorem gives
\[
\int_{\mathcal{C}}  \frac{(z^2-1)}{2z(z-2i)(2z-i)}\ dz =  2 \pi i \left( \frac{i}{4} - \frac{5i}{12} \right) =2\pi i \left(\frac{-i}{6} \right) = \frac{\pi}{3}.
\]
Hence
\[
\int_0^{2\pi} \frac{\sin (t)}{5-4\sin(t)}\ dt =\frac{\pi}{3}.
\]
\end{solution}


\section{Differentiation}

\subsection{Exercise 1}
We have that
\begin{align*}
        \abs{f(x) - f(y)} &\leq (x - y)^2 = \abs{x - y}^2 \\
        \frac{\abs{f(x) - f(y)}}{\abs{x - y}} &\leq \abs{x - y} \\
                                              &\implies f'(x) = 0 \: \forall x
\end{align*}
Since we can write $\frac{f(t) - f(x)}{t - x} = f'(x) + u(t)$ with $\lim_{t \to x} u(t) \to 0$, we have that
\begin{align*}
        f(t) - f(x) = (t - x) u(t), &\quad f(t) - f(y) = (t - y) v(t) \\
        f(y) - f(x) = (y - x) u(y), &\quad f(x) - f(y) = (x - y) v(x) \implies u(y) = v(x) \: \forall x, y \\
        f(y) - f(x) &= 0 \implies f(x) = f(y) \: \forall x, y
\end{align*}
Whoops, I did this before reading the mean value theorem section - this problem follows immediately
from applying the mean value theorem after showing $f'(x) = 0$.

\subsection{Exercise 2}
Take $x, t \in (a, b)$ with $t > x$. Applying the mean value theorem to $f$ on $[x, t]$, we get
$f(t) - f(x) = (t - x) f'(y)$ for some $y \in (x, t)$. Since $f'(y) > 0 \implies f(t) - f(x) > 0$,
$f$ is strictly increasing on $(a, b)$. We can prove $g = f^{-1}$ is differentiable directly
\begin{align*}
        \lim_{t \to x} \frac{g(f(t)) - g(f(x))}{f(t) - f(x)} &= \lim_{t \to x} \frac{t - x}{f(t) - f(x)} \\
                                                             &= \frac{1}{f'(x)}
\end{align*}

\subsection{Exercise 3}
Suppose (WLOG) that $x_2 > x_1$ but $f(x_2) = f(x_1)$. Then we have that
\begin{align*}
        x_2 + \epsilon g(x_2) &= x_1 + \epsilon g(x_1) \\
        (x_2 - x_1) + \epsilon (g(x_2) - g(x_1)) &= 0 \\
        1 + \epsilon g'(x) &= 0 \quad x \in (x_1, x_2) \\
        1 + \epsilon g'(x) &\geq 1 - \epsilon \abs{g'(x)} \\
                           &> 0 \quad \forall \epsilon < \frac{1}{M}
\end{align*}
Where the penultimate step follows from the mean value theorem.
Thus, we can choose an $\epsilon$ such that $f(x_2) \neq f(x_1)$, which means we can make $f$ injective.

\subsection{Exercise 4}
Let $f(x) = C_0 + C_1 x + ... + C_n x^n$ and $g(x) = C_0 x + \frac{C_1}{2} x^2 + ... + \frac{C_n}{n+1}x^{n+1}$.
Then $g'(x) = f(x)$. Applying the mean value theorem to $g(x)$ on $[0, 1]$ yields that there is an $x$
such that $g'(x) = f(x) = g(1) - g(0) = 0$, so $f$ has a root in $(0, 1)$.

\subsection{Exercise 5}
It looks like the mean value theorem is this chapter's ratio test; you can guess how this will go.
By the mean value theorem, $f'(y) = f(x + 1) - f(x)$ for $y \in [x, x+1]$. Thus we have
$\lim_{x \to \infty} g(x) = \lim_{y \to \infty} f'(y) = 0$.

\subsection{Exercise 6}
Consider $x > y > 0$. By the mean value theorem (surprise), we have that
\begin{align*}
        f(x) - f(y) &= f'(a) (x - y) \quad a \in (x, y) \\
                    &< f'(x) (x - y) \\
        \lim_{y \to 0} \frac{f(x) - f(y)}{x - y} &< \lim_{y \to 0} f'(x) \\
        \frac{f(x)}{x} &< f'(x)
\end{align*}
Differentiating $g$, we get
\begin{align*}
        g'(x) &= \frac{xf'(x) - f(x)}{x^2} \\
        f'(x) > \frac{f(x)}{x} &\implies g'(x) > 0
\end{align*}
So $g$ is monotonically increasing.

\subsection{Exercise 7}
If $f$ and $g$ are real, then the result follows immediately from L'Hopital's, since the existence of
$f'(x), g'(x)$ imply that $f$ and $g$ are continuous (so the requirement that $f(x) \to 0$ and $g(x) \to 0$
is satisfied). More generally, for complex functions, we have
\begin{align*}
        \lim_{t \to x} \frac{f(t)}{g(t)} &= \lim_{t \to x} \frac{f_1(t) + f_2(t) i}{g_1(t) + g_2(t) i} \\
                                         &= \lim_{t \to x} \frac{\frac{f_1(t) + f_2(t)i}{t - x}}{\frac{g_1(t) + g_2(t)i}{t - x}} \\
                                         &= \frac{f'(x)}{g'(x)}
\end{align*}

\subsection{Exercise 8}
Exercise 7 was a brief detour, but we are now back to hammering away with the mean value theorem. We have
\begin{align*}
        \abs{\frac{f(t) - f(x)}{t - x} - f'(x)} &< \epsilon \\
        \abs{f'(y) - f'(x)} &< \epsilon
\end{align*}
Where $y \in (x, t)$ or $y \in (t, x)$. Since $f'$ is continuous, we can choose $\delta$ such that
$\abs{y - x} < \delta$ implies the above inequality, which means we can take $\abs{t - x} < \delta$ to
get the desired result.

This does not seem to have to hold for vector-valued functions, since in that case we only have
$\abs{f(t) - f(x)} \leq (t - x) \abs{f'(y)}$.

\subsection{Exercise 9}
Yes, since we can apply L'Hopital's to $(0, +\infty)$ and $(-\infty, 0)$
\begin{align*}
        f'(0) &= \lim_{t \to 0} \frac{f(t) - f(0)}{t} \\
              &= \lim_{t \to 0} f'(t) = 3
\end{align*}

\subsection{Exercise 10}
We proceed exactly as directed by Rudin
\begin{align*}
        \lim_{x \to 0} \frac{f(x)}{g(x)} &= \lim_{x \to 0} \bigg(\frac{f(x)}{x} - A\bigg) \frac{x}{g(x)} + \frac{Ax}{g(x)} \\
                                         &= \frac{\lim_{x \to 0} \frac{f(x) - Ax}{x}}{\lim_{x \to 0} \frac{g(x)}{x}} + \frac{A}{\lim_{x \to 0} \frac{g(x)}{x}} \\
                                         &= 0 + \frac{A}{B}
\end{align*}
Where the final step follows from breaking up $f, g, A$ into their real and imaginary components and applying
L'Hopital's ($\Re f'(x) - \Re A$ and $\Im f'(x) - \Im A$ both go to 0).

\subsection{Exercise 11}
Double application of L'Hopital's
\begin{align*}
        \lim_{h \to 0} \frac{f(x + h) + f(x - h) - 2f(x)}{h^2} &= \lim_{h \to 0} \frac{f'(x + h) - f'(x - h)}{2h} \\
                                                               &= \lim_{h \to 0} \frac{f''(x + h) + f''(x - h)}{2} \\
                                                               &= f''(x)
\end{align*}
If we let $f(x) = x \sin \frac{1}{\abs{x}}$ when $x \neq 0$ and $f(0) = 0$, then $f''(0)$ does not exist.
However, plugging $f$ into the above limit and setting $x = 0$, we see that the limit is 0.

\subsection{Exercise 12}
\begin{align*}
        &f(x) = \abs{x}^3 =
        \begin{cases}
                -x^3 & x < 0 \\
                0 & x = 0 \\
                x^3 & x > 0
        \end{cases} \implies
        f'(x) =
        \begin{cases}
                -3x^2 & x < 0 \\
                0 & x = 0 \\
                3x^2 & x > 0
        \end{cases} \\
        &\implies f''(x) =
        \begin{cases}
                -6x & x < 0 \\
                0 & x = 0 \\
                6x & x > 0
        \end{cases} \implies
        f^{(3)}(x) =
        \begin{cases}
                -6 & x < 0 \\
                0 & x = 0 \\
                6 & x > 0
        \end{cases}
\end{align*}
So the left and right limits of $f^{(3)}(0)$ do not equal one another.

\subsection{Exercise 14}
Suppose $f$ is a convex differentiable function. Then
\begin{align*}
        f(\lambda x + (1 - \lambda) y) &\leq \lambda f(x) + (1 - \lambda) f(y) \\
        f(\lambda x + (1 - \lambda) y) - f(y) &\leq \lambda (f(x) - f(y)) \\
        \frac{f(y) - f(\lambda x + (1 - \lambda) y)}{\lambda (y - x)} &\geq \frac{f(y) - f(x)}{y - x} \\
        \frac{f(y) - f(\lambda x + (1 - \lambda) y)}{y - (\lambda x + (1 - \lambda)y)} &\geq \frac{f(y) - f(x)}{y - x}
\end{align*}
Since $x$ was arbitrary, the last inequality implies that $f'$ is monotonically increasing (because we can
consider a new interval whose left endpoint is $\lambda x + (1 - \lambda) y$).

For the other direction, we can modify the convexity condition until we get something that we can apply
the mean value theorem to. Letting $z = \lambda x + (1 - \lambda) y$, we have
\begin{align*}
        f(z) &\leq \lambda f(x) + (1 - \lambda) f(y) \\
        \lambda (f(z) - f(x)) &\leq (1 - \lambda) (f(y) - f(z)) \\
        \frac{f(z) - f(x)}{z - x} &\leq \frac{(1 - \lambda) (f(y) - f(z))}{\lambda (z - x)} \\
                                  &\leq \frac{f(y) - f(z)}{y - z}
\end{align*}
The final inequality is true by monotonicity of $f'$ and applying the mean value theorem
to $(x, z)$ and $(z, y)$. The final step follows from the fact that $\lambda (z - x) = (1 - \lambda) (y - z)$.

\subsection{Exercise 15}
Proceeding as directed by the hint, we have
\begin{align*}
        f(x + 2h) &= f(x) + 2hf'(x) + 2h^2 f''(c) \\
        f'(x) &= \frac{1}{2h} (f(x + 2h) - f(x)) - hf''(c) \\
        M_1 &\leq \frac{1}{2h} (\abs{f(x + 2h)} + \abs{f(x)}) + h\abs{f''(c)} \\
            &\leq \frac{M_0}{h} + h M_2 \\
        M_1^2 &\leq \frac{M_0^2}{h^2} + 2 M_0 M_2 + h^2 M_2^2 \\
              &\leq 4 M_0 M_2 \quad \quad h = \sqrt{\frac{M_0}{M_2}}
\end{align*}

\subsection{Exercise 18}
Differentiating $f(t) - f(\beta) = (t - \beta) Q(t)$ and evaluating at $\alpha$, we have
\begin{align*}
        f^{(k)}(\alpha) &= (\alpha - \beta) Q^{(k)} (\alpha) - k Q^{(k - 1)} (\alpha) \\
        \frac{f^{(k)}(\alpha)}{k!} (\beta - \alpha)^k &= -\frac{Q^{(k)}(\alpha)}{k!}(\beta - \alpha)^{k+1} +
        \frac{Q^{(k - 1)}(\alpha)}{(k - 1)!} (\beta - \alpha)^{k} \\
        \sum_{k = 1}^{n - 1}\frac{f^{(k)}(\alpha)}{k!} (\beta - \alpha)^k &=  -\frac{Q^{(n - 1)}(\alpha)}{(n - 1)!}(\beta - \alpha)^{n} + (f(\beta) - f(\alpha)) \\
        P(\beta) + \frac{Q^{(n - 1)}(\alpha)}{(n - 1)!}(\beta - \alpha)^{n} &= f(\beta)
\end{align*}


\section{Artificial Neural Networks}
Artificial neural networks are a computing system inspired by biological neurons. Neural networks are comprised of neurons which are capable of taking any number of numerical inputs and outputs a numerical value.

\input{figures/background/neural-net}

Mathematically, a single neuron is a function.
The function of a neuron $j$ receiving input $x_j$ and producing output $y_j$ is composed of the activation $a_j$, an activation function $f_a$ which returns the activation, and an output function $f_{o}$.
The activation $a_j$ can also be considered the neuron's state.
The activation function $f_a$ calculates $a_j$ given the network input $x_j$ and can be defined as:
\begin{align}
	a_j &= f_a\left(x_j\right)
\end{align}
The output function $f_o$ computes $y_j$ based on $a_j$ and is defined as:
\begin{align}
	y_j &= f_o\left(a_j\right)
\end{align}
Many activation functions exists and are used including the identity function and the rectified linear unit (ReLU), defined as:
\begin{align}
	f_a(x) &=
	\begin{cases}
		0	& \text{for } x \leq 0\\
		x	& \text{for } x > 0
	\end{cases}
\end{align}

Between each neuron in the network are connections which transfer the output of neuron $i$ to neuron $j$. Each of these connections are assigned a weight $w_{ij}$, which is computed by the learning algorithm.
Each neuron also has a bias $w_{0j}$.
These parameters are used to provide a neuron its input $x_j$. This is defined as:
\begin{align}
	x_j &= \sum_{i}y_iw_{ij} + w_{0j}
\end{align}

\figref{fig:background-neuralnet} shows a visualization of the interactions between multiple neurons.

Learning occurs by using an algorithm known as backpropagation to modify the parameters of the neural network.
Further discussion on backpropagation can be found in Section \ref{section:background-backpropagation}.

Deep neural networks are so called deep due to having multiple "hidden" layers between the input and output which are not accessed directly.

\subsection{Convolutional Neural Networks}\label{section:background-cnn}
Convolutional Neural Networks (CNNs) are a specific class of deep neural networks and have been found to perform exceptionally well for image analysis, such as image classification and segmentation.
The inspiration for CNNs come from biological processes to simulate the organization of the visual cortex in animals~\cite{cnnbiology}.
Convolutional layers are the core building blocks of CNNs.
These convolutional layers consist of a set of learnable filters which are convolved across the entire input and compute the dot products between the different entries of the filter.

\input{figures/background/convolution}

By using multiple convolutional layers, higher level features can be extracted, generating a feature map.
The module responsible for extracting the feature map is commonly known as the encoder.

The convolutional layers that make up the CNN are called the feature encoder.
The output of these convolutional layers are then fed into a module with one or more fully convolutional layers to produce a classification of each pixel in a scene.
These convolutional layers work by taking the deep feature maps produced by the encoder and using deconvolutional or upsampling layers to produce a log-likelihood vector for each pixel.
This module is known as the decoder.
The resulting vector produced is a probability of each pixel being a certain class.
Applying an argmax operation on this vector produces a segmentation of the input image.

A bottleneck structure may also be used to increase performance.
A bottleneck structure in a convolutional neural network reduces the number of features by performing a $1 \times 1$ convolution on a tensor before performing a convolution operation.
A final $1 \times 1$ convolution is then performed to expand the tensor to contain a meaningful combination features.
As the input features are correlated, redundancy can be removed by performing the first $1 \times 1$ convolution.

For example, performing a $3 \times 3$ convolution on a $256 \times 256$ input tensor requires more than 589,000 operations.
A bottleneck structure can be used to reduce the dimensionality of the input tensor to only 64 dimensions using a $1 \times 1$ convolution.
The $3 \times 3$ convolution is then performed on this tensor.
Finally, the resulting tensor is re-expanded back to the original 256 dimensions using another $1 \times 1$ convolution.
This would result in slightly over 69,000 operations, significantly less than without the bottleneck structure.

This reduces both the number of operations required as well as the number of parameters in the network.


\section{Two-Stage Least Squares and its Limitations}

\subsection{Basic Model}
Throughout the paper, unless stated otherwise, we consider the following basic model with two equations, following the notation of \cite{angrist1999jackknife}:
\begin{align*}
Y_i &= X_i\beta + \varepsilon_i \\
X_i &= Z_i\pi + \eta_i
\end{align*}
$Y_i$ is scalar. $X_i$ is $1\times L$ row vector of potentially endogenous regressors.
$Z_i$ is $1\times K$ row vector, with $K\geq L$.
$K-L$ is the number of over-identifying restrictions. In matrix notation:\\
\begin{align}
\underset{(N\times 1)}{\mathbf{Y}} = \underset{(N\times L)}{\mathbf{X}}\beta + \underset{(N\times 1)}{\varepsilon} \\
\underset{(N\times L)}{\mathbf{X}} =  \underset{(N\times K)}{\mathbf{Z}}\pi + \underset{(N\times L)}{\eta}
\end{align}
where (2.1) denotes the structural equation, (2.2) is the first stage. $\mathbf{Y}$ and $\varepsilon$ are $N \times 1$ vectors.
$\mathbf{X}$ and $\eta$ are $N \times L$ matrices, and $\mathbf{Z}$ is $N \times K$ matrix. If the vectors of regressors and that of instruments have $M$ common elements, then $M$ columns of the $N \times L$ matrix $\eta$ have their elements as zero.
\par The following assumptions hold for our model:
\begin{enumerate}
\item Conditional on instruments $Z_i$, the error term $\varepsilon_i$ has expectation zero ($E[\varepsilon_i|Z_i]=0$) and variance $\sigma^2$.
\item $E[\eta|\mathbf Z]=0$ and $E[\eta_i^\prime\eta_i|\mathbf Z]=\Sigma_\eta$, with rank $L-M$.
\item $E[\varepsilon_i\eta_i^\prime|\mathbf Z]=\sigma_{\varepsilon\eta}$, where $\sigma_{\varepsilon\eta}$ is an L-dimensional column vector.
\item All observations of ($Y_i, X_i, Z_i$) are independent and identically distributed.
\end{enumerate}

\subsection{Concentration Parameter}
The Concentration Parameter $\mu^2$, given by:
        \begin{equation}
  \mu^2 = \frac{\pi' Z' Z \pi}{\sigma_{\eta}^2}
        \end{equation}
measures the strength of the instruments. It is unitless. An important question to be addressed at this point is that since $\pi$, and thus $\mu^2$, is unknown, how is the researcher supposed to know whether $\mu^2$ value is low enough for the instrument under study to be weak? This is addressed by the fact that the concentration parameter can be interpreted in terms of the first stage F statistic, discussed in Section 3. If the sample size is large, $E(F) \cong \mu^2/K + 1$, and thus $F-1$ can be treated as an estimator of $\mu^2 /K$, which gives us a convenient way to test for weak instruments.

\par The concept of $\mu^2$ is essentially the starting point for understanding the weak instruments literature. \cite{rothenberg1984approximating} showed that the concentration parameter plays the role we commonly associate with the sample size or number of observations: as $\mu^2$ becomes large, the normal distribution becomes a good approximation for the 2SLS distribution. Likewise a small concentration parameter leads to a non-normal distribution of 2SLS, and the estimator becomes biased. Thus $\mu^2$ can be interpreted as the effective sample size.

\subsection{Two-Stage Least Squares Estimator}

It is helpful to characterize the 2SLS and (later on the jackknife estimator as well), as a feasible version of the ideal but infeasible instrumental variables estimator. Adhering to the notation defined in section 2.1, we derive the formula for the 2SLS estimator using this characterization. The derivation is adopted, with modifications to the notation, from \cite{brucehansen2019} and \cite{angrist1999jackknife}.
\par First, we estimate the ideal estimator $\hat\beta_{OPT}$ using the optimal instrument $Z\pi$, by ordinary least-squares:
\begin{equation}
\hat\beta_{OPT}= ((\mathbf{Z}\pi)'\mathbf{X})^{-1}((\mathbf{Z}\pi)'\mathbf{Y})
\end{equation}
$\pi$ is unknown however, and so the optimal estimator $\hat\beta_{OPT}$ is infeasible.

We use an estimate of $\pi$ instead, and denote it  by $\hat\pi$. $\hat\pi$ is estimated from the reduced form regression which yields $\hat\pi = (\mathbf{Z}'\mathbf{Z})^{-1}(\mathbf{Z}'\mathbf{X})$. Substituting $\hat\pi$ into (2.4),
\begin{equation}
\hat\beta_{2SLS}= ((\mathbf{Z}\hat\pi)'(\mathbf{Z}\hat\pi))^{-1}((\mathbf{Z}\hat\pi)'\mathbf{Y})
\end{equation}
\begin{equation}
\hat\beta_{2SLS}= ((\mathbf{Z}(\mathbf{Z}'\mathbf{Z})^{-1}(\mathbf{Z}'\mathbf{X}))'(\mathbf{Z}(\mathbf{Z}'\mathbf{Z})^{-1}(\mathbf{Z}'\mathbf{X}))^{-1}((\mathbf{Z}(\mathbf{Z}'\mathbf{Z})^{-1}(\mathbf{Z}'\mathbf{X}))'\mathbf{Y})
\end{equation}
\begin{equation}
\hat\beta_{2SLS}= ((\mathbf{X}'\mathbf{Z})(\mathbf Z'\mathbf Z)^{-1}\mathbf Z'\mathbf X)^{-1}\mathbf X'\mathbf Z(\mathbf Z'\mathbf Z)^{-1}\mathbf Z'\mathbf Y
\end{equation}
This is the two-stage least squares estimator.
$\hat\beta_{2SLS}$ can be thought of as an estimator with the constructed instrument $(\mathbf{Z}\hat\pi)$ where $\hat\pi = (\mathbf{Z}'\mathbf{Z})^{-1}(\mathbf{Z}'\mathbf{X})$.
\par
Representing (2.7) in terms of the projection matrix  $P_{Z}=\mathbf Z(\mathbf Z'\mathbf Z)^{-1}\mathbf Z'$:
\begin{equation}
\hat{\beta }_{2SLS}=((\mathbf X'P_{Z}\mathbf X)^{-1}\mathbf X'P_{Z}\mathbf Y)
\end{equation}
The two-stage least squares is by far the most widely used estimator for instrumental variables estimation, however, as we briefly discussed in chapter 1, it has certain concerning limitations, which we expand upon in the next section.

\subsection{Limitations of 2SLS: Bias}
In the weak instruments case, the 2SLS estimator fails to provide unbiased, reliable estimates. As shown in Section 2.3, we arrived at the 2SLS estimator using an estimate of $\pi$ since the true value of the first-stage coefficients is unknown, which implies a certain amount of
over-fitting of the first-stage equation, leading to bias in the
direction of the expectation of the OLS estimator of $\beta$.

\par There is considerable  theoretical research noting the poor finite-sample properties of instrumental variables estimator. Results on the magnitude of this bias have been provided by \cite{nagar1959bias}, \cite{richardson1968exact}, \cite{sawa1969exact} and \cite{buse1992bias}. Further, as shown by noteworthy results of \cite{bound1995problems}, with weak instruments particularly, the finite sample behavior worsens and the estimator can become biased as well as inconsistent.

\subsubsection{Bias in Single Instrument case}
To visualize the problem intuitively, first let us consider the simple case of a single (weak) instrument and endogenous regressor:

        $$Y_i = X_i\beta + \varepsilon_i$$
$$X_i = Z_i\pi_1 + \eta_i$$

In this case, the 2SLS estimator is simply given by the ratio of the two covariances:
\begin{equation}
    \hat\beta_{2SLS}=\frac{\sigma_{Y_iZ_i}}{\sigma_{X_iZ_i}}
\end{equation}
However, given the fact that our instrument is very weak,
$\sigma_{X_iZ_i}=0.$
Hence, $\hat\beta_{2SLS}$ does not exist.
\par In the simplest case, the 2SLS estimator is just the ratio of two covariances, and with weak instruments, the 2SLS or general instrumental variables estimator does not exist.

\subsubsection{Approximate Expression of Bias in Multiple-Instruments case}
In this section we provide two different expressions approximating the bias of the two-stage least squares estimator in the general case and show how it centers around ordinary least squares.
\par Defining the `relative bias' of 2SLS to be its bias relative to the
inconsistency of OLS, \cite{buse1992bias} derived an expression for the approximate bias of $\hat\beta_{2SLS}$ using power series approximations. This result holds even when the errors are not normally distributed:
\begin{equation}
  \frac{\sigma_{\varepsilon\eta }(K-2)}{\pi'\mathbf Z'\mathbf Z\pi}
\end{equation}

where N is the sample size and K is the number of excluded instruments.
This expression is approximately inversely proportional to $\mu^2/(K-2)$, as shown:

\begin{equation}
\frac{\sigma_{\varepsilon\eta}{\sigma_\eta^2}(K-2)}{{\sigma_\eta^2}\pi'\mathbf Z'\mathbf Z\pi} = \frac{\sigma_{\varepsilon\eta}(K-2)}{{\sigma_\eta^2}{\mu^2}}
\end{equation}
where $\mu^2$ denotes the concentration parameter (measure of the strength of the instrument as discussed in Section 1.). Note that in the above equation, the term $\frac{\sigma_{\varepsilon\eta }}{\sigma_{\eta }^{2}}$ approximately equals the asymptotic bias of the OLS estimator when the instrument explains little of the variation of $\mathbf X$. Hence, from equation (2.10) it is clearly seen that for $K>2$, the bias of the 2SLS estimator relative to OLS is inversely proportional to the concentration parameter, hence, weaker the instrument(s), lower is the concentration parameter, and higher is the bias of 2SLS towards the OLS estimate.

\par It can be seen from the above formulation that increasing the number of instruments with the explanatory power remaining constant, causes the relative bias of the 2SLS to only increase.

\par Before moving on to the second expression, it is important to briefly introduce alternate asymptotic representations that are generally used for the weak instruments case.
As discussed by \cite{stock2002survey}, for weak instruments, conventional asymptotic approximations to finite-sample distributions are quite poor. Two alternate asymptotic methods commonly employed are \textbf {weak-instrument asymptotics} (involving a sequence of models chosen to keep $\mu^2/K$ constant as sample size $N \rightarrow \infty$) pioneered by \cite{staiger1997stock} and \textbf {many-instrument or group asymptotics} (involving sequence of models with fixed instruments and normal errors, where $K$ is proportional to $N$ and $\mu^2/K$ converges to a constant finite limit), first proposed by \cite{bekker1994alternative}. In the 1995 working paper version of \cite{angrist1999jackknife}, it was referred to as group asymptotics.

\par We present an expression for the approximate bias of the 2SLS estimator using group asymptotics, wherein we let
the number of instruments grow proportional to rate of the sample size.
This keeps the instruments weak. The detailed derivation of this expression is provided in the appendix A.1.
\begin{equation}
    E[\hat{\beta }_{2SLS}-\beta]\approx \frac{\sigma_{\varepsilon\eta }}{\sigma_{\eta }^{2}}\frac{1}{F+1}
\end{equation}
where F is the population analog of the F-statistic for the joint significance of the instruments in the first-stage regression. $\frac{\sigma_{\varepsilon\eta }}{\sigma_{\eta }^{2}}$ approximately equals the asymptotic bias of the OLS estimator. Given a weak first-stage (weak instruments case), $F\rightarrow0$, and we can see that the bias approaches $ \frac{\sigma_{\varepsilon\eta }}{\sigma_{\eta }^{2}}$. With a strong first-stage, $F\rightarrow \infty$ and then the 2SLS bias goes to 0.

\subsubsection{Inconsistency of the 2SLS estimator}
In this section we relate the two conditions for instrumental variables estimation to the consistency property of the 2SLS estimator. If the weak correlation (between the instrument(s) and the endogenous variable) is coupled together with even a small violation of the second condition of instrumental variables estimation (which is the instrument exogeneity condition) then we have an inconsistent estimator. This insight was first discussed by \cite{bound1995problems}.

To represent the problem discussed above, let us consider the probability limit of the 2SLS estimator:

\begin{equation}
plim \hat{\beta} _{2SLS}=\beta + \frac{\sigma _{\mathbf {\hat X},\varepsilon }}{\sigma _{\mathbf {\hat X}}^{2}}
\end{equation}

where $\mathbf {\hat X}$ is the projection of $\mathbf X$ onto $\mathbf Z$, and ${\sigma _{\mathbf {\hat X},\varepsilon }}$ is the covariance between $\mathbf {\hat X}$ and $\varepsilon$.

From equation (2.13) we can intuitively understand that if $\sigma _{\mathbf {\hat X}}^{2}$ is small, which implies that we have a weak instrument, then as long as ${\sigma _{\mathbf {\hat X},\varepsilon }}$ is zero the estimator will be consistent. However, suppose we have a weak instrument  and also, ${\sigma _{\mathbf {\hat X},\varepsilon }}$ is small but non-zero, then even that small correlation between $\mathbf Z$ (and thus, $\mathbf {\hat X}$) and the structural error term $\varepsilon$ can lead to a large inconsistency.

Hence, with weak instruments, even moderate correlation between instrument and structural error term can magnify the inconsistency of IV estimator.


%
%% Author: Jeffrey Leung
%% Last edited: 2015-07-05
%%
%% This document contains section 1 (encoding) of a course overview of CMPT 150.
%

\section{Encoding}

\subsection{Introduction}
\begin{easylist}[itemize]

& \emph{Alphabet:} Finite set of symbols
	&& E.g. $\{0, 1\}, \{T, F\}, \{0, 1, 2, ... , 9\}, \{A, B, C, ... , Z\}$

& \emph{Message:} Meaningful sequence of symbols from an alphabet
	&& E.g. "CMPT 150", "XYZ-AB2", 0110101 (a binary sequence)

& \emph{Encoding:} Representation of the symbols of one alphabet by sequences of symbols from a second alphabet
	&& \emph{Codeword:} Meaningful sequence of symbols which translates into a sequence of symbols of another alphabet
		&&& \emph{Uniquely decipherable:} Each codeword has only one meaning
		&&& \emph{Fixed-length:} Each codeword has the same length
			&&&& Number of different codewords of length $k$ where there are $j$ elements in the alphabet is $j^k$ (e.g. 4-bit binary has $2^4 = 16$ different possibilities)

\end{easylist}
\subsection{Binary, Hexadecimal, and BCD}
\begin{easylist}[itemize]

& Codewords: See \hyperref[bin_hex_encoding_scheme]{Table~\ref*{bin_hex_encoding_scheme}}

& Binary:
	&& Computers interpret voltage levels of cells as 0 or 1
	&& \emph{Little Endian notation:} The rightmost bit is numbered as the least significant bit (0) and the leftmost bit is numbered as the most significant bit (n-1 where n = length of sequence)
	&& Integer field: Digits to the left of the decimal point
	&& Fractional field: Digits to the right of the decimal point
& Hexadecimal:
	&& Add additional leading 0s if necessary for conversion

	&& E.g. Base 2 to base 16:
	\Deactivate
	\begin{center}
		$001 1111 0010 1011_{2} = x_{16}$

		\bigskip
		\centering
		\begin{tabular}{ c | c | c | c }
			0001 & 1111 & 0010 & 1011 \\
			1 & F & 2 & B \\
		\end{tabular}
		\bigskip \\
		$001 1111 0010 1011_{2} =$ 1F2B$_{16}$
	\end{center}
	\Activate

\pagebreak

\Deactivate
\begin{table}[!htp]
	\caption{Binary/Decimal/Hexadecimal encoding scheme}
	\label{bin_hex_encoding_scheme}
	\centering
	\begin{tabular}{ c | c | c }
		Binary: & Decimal: & Hexadecimal: \\
		\hline
		0000 &  0 & 0 \\
		0001 &  1 & 1 \\
		0010 &  2 & 2 \\
		0011 &  3 & 3 \\
		0100 &  4 & 4 \\
		0101 &  5 & 5 \\
		0110 &  6 & 6 \\
		0111 &  7 & 7 \\
		1000 &  8 & 8 \\
		1001 &  9 & 9 \\
		1010 & 10 & A \\
		1011 & 11 & B \\
		1100 & 12 & C \\
		1101 & 13 & D \\
		1110 & 14 & E \\
		1111 & 15 & F \\
	\end{tabular}
\end{table}
\Activate

\bigskip
& \emph{Binary Coded Decimal (BCD):} Encoding scheme where each decimal digit is encoded as 4 binary bits
%&& Codewords: See \hyperref[bcd_encoding_scheme]{Table~\ref*{bcd_encoding_scheme}}

\Deactivate
\begin{table}[!htb]
	\caption{BCD encoding scheme}
	\label{bcd_encoding_scheme}
	\centering
	\begin{tabular}{ c | c }
		Decimal: & BCD: \\
		\hline
		0 & 0000 \\
		1 & 0001 \\
		2 & 0010 \\
		3 & 0011 \\
		4 & 0100 \\
		5 & 0101 \\
		6 & 0110 \\
		7 & 0111 \\
		8 & 1000 \\
		9 & 1001 \\
	\end{tabular}
\end{table}
\Activate

\pagebreak

	&& E.g. BCD encoding:
	\Deactivate
	\begin{center}
		$169_{10} = x_{BCD}$
		\begin{table}[!htb]
			\centering
			\begin{tabular}{ c | c | c }
				1 & 6 & 9 \\
				0001 & 0110 & 1001 \\
			\end{tabular}
		\end{table}

		$169_{10} = 0001 \ 0110 \ 1001_{BCD}$
	\end{center}
	\Activate

	&& E.g. BCD decoding:
	\Deactivate
	\begin{center}
		$1010 \ 0001 \ 0110_{BCD} = x_{10}$
		\begin{table}[!htb]
			\centering
			\begin{tabular}{ c | c | c }
				1010 & 0001 & 0110 \\
				? & 1 & 6 \\
			\end{tabular}
		\end{table}

		$1010 \ 0001 \ 0110_{BCD}$ is meaningless.
	\end{center}
	\Activate

\end{easylist}
\subsection{Positional Number System Conversions}
\subsubsection{Base 10 to Base X}
\begin{easylist}[itemize]

& Integer: Divide the base 10 number by $x$ and write the remainder to the right. The number in base $x$ is the sequence of remainders from bottom to top.

	&& E.g. Base 10 to base 2:
	\Deactivate % Deactivates easylist; allows usage of &
	\begin{center}
		$13_{10} = x_{2}$

		\begin{table}[!hb]
			\centering
			\begin{tabular}{ r | r l }
				            2 & 13 &   \\
				\cline{2-2} 2 &  6 & 1 \\
				\cline{2-2} 2 &  3 & 0 \\
				\cline{2-2} 2 &  1 & 1 \\
				\cline{2-2} \multicolumn{2}{r}{0} & 1 \\
			\end{tabular}
		\end{table}

		$13_{10} = 1101_{2}$
	\end{center}
	\Activate

	&& E.g. Base 10 to base 16:
	\Deactivate
	\begin{center}
		$38_{10} = x_{16}$

		\begin{table}[!hb]
			\centering
			\begin{tabular}{ r | r l }
				            16 & 38 &   \\
				\cline{2-2} 16 &  2 & 6 \\
				\cline{2-2} \multicolumn{2}{r}{0} & 2 \\
			\end{tabular}
		\end{table}

		$38_{10} = 26_{16}$
	\end{center}
	\Activate

& Fractional: Draw a line down from the decimal point. While the right side is greater than 0, multiply the right side by 2 and write the result below. The fraction in binary is the sequence of 0s and 1s on the left side from top to bottom.

	&& E.g. Base 10 to base 2:
	\Deactivate
	\begin{center}
		$0.625_{10} = x_{2}$

		\begin{table}[!htb]
			\centering
			\begin{tabular}{ r : l } % ':' creates a dashed vertical line (from package arydshln)
				. & 625 \\
				1 & 25 \\
				0 & 5 \\
				1 & 0 \\
			\end{tabular}
		\end{table}

		$0.625_{10} = 0.101_{2}$
	\end{center}
	\Activate

& E.g. Base 10 to base 2:
\Deactivate
\begin{center}
	$18.375_{10} = x_{2}$

	\begin{table}[!htb]
		\begin{minipage}{.5\linewidth}
			\centering
			\begin{tabular}{ r | r l }
				            2 & 18 &   \\
				\cline{2-2} 2 &  9 & 0 \\
				\cline{2-2} 2 &  4 & 1 \\
				\cline{2-2} 2 &  2 & 0 \\
				\cline{2-2} 2 &  1 & 0 \\
				\cline{2-2} \multicolumn{2}{r}{0} & 2 \\
			\end{tabular}
		\end{minipage}%
		\begin{minipage}{.5\linewidth}
			\centering
			\begin{tabular}{ r : l }
				. & 375 \\
				0 & 75 \\
				1 & 5 \\
				1 & 0 \\
			\end{tabular}
		\end{minipage}
	\end{table}

	$18.375_{10} = 1 \ 0010.011_{2}$
\end{center}
\Activate

\end{easylist}
\subsubsection{Base X to Base 10}
\begin{easylist}[itemize]

& Write the position values underneath each digit, then add the position values of all digits with 1s.

& E.g. Base 2 to base 10:
\Deactivate
\begin{center}
	$10 \ 1010 \ 0111_{2} = x_{10}$

	\begin{table}[!htb]
		\centering
		\begin{tabular}{ c | c | c | c | c | c | c | c | c | c }
			   1  &    0  &    1  &    0  &    1  &    0  &    0  &    1  &    1  &    1  \\
			$2^9$ & $2^8$ & $2^7$ & $2^6$ & $2^5$ & $2^4$ & $2^3$ & $2^2$ & $2^1$ & $2^0$ \\
		\end{tabular}
	\end{table}

	$2^9 + 2^7 + 2^5 + 2^2 + 2^1 + 2^0 = 679$ \\
	\medskip
	$10 \ 1010 \ 0111_{2} = 679_{10}$
\end{center}
\Activate

\pagebreak

& E.g. Base 16 to base 10:
\Deactivate
\begin{center}
	$26_{16} = x_{10}$

	\begin{table}[!htb]
		\centering
		\begin{tabular}{ c | c }
			    2  &     6  \\
			$16^1$ & $16^0$ \\
		\end{tabular}
	\end{table}

	$(2 \times 16^1) + (6 \times 16^0) = 32 + 6 = 38$ \\
	\medskip
	$26_{16} = 38_{10}$
\end{center}
\Activate

& E.g. Base 2 to base 10:
\Deactivate
\begin{center}
	$0.101_{2} = x_{10}$

	\begin{table}[!htb]
		\centering
		\begin{tabular}{ c c c | c | c }
			   0  & . &       1  &       0  &      1  \\
			$2^0$ &   & $2^{-1}$ & $2^{-2}$ & $2^{-3}$ \\
		\end{tabular}
	\end{table}

	$2^{-1} + 2^{-3} = 0.5 + 0.125 + 0.625$ \\
	\medskip
	$0.101_{2} = 0.625_{10}$
\end{center}
\Activate

\subsection{Signed Arithmetic}

& \emph{Signed arithmetic:} Binary encoding which represents both positive and negative numbers
	&& Codewords: See \hyperref[binary_representations]{Table~\ref*{binary_representations}}
	&& Rules:
		&&& 0 is always represented
		&&& For any positive number which is represented, its corresponding negative number must also be represented
	&& $2^{k}-1$ codewords where $k$ is the number of bits
		&&& Greatest number which can be represented: $\frac{2^{k} - 1}{2} = 2^{k-1}-1$
		&&& Least number which can be represented: $-2^{k-1}-1$
		&&& E.g. 4 bits can represent:
			&&&& In signed magnitude encoding: \\ $\{-7, -6, \dots , -1, 0, 1, \dots , 6, 7\}$ = 15 numbers
			&&&& In 2's complement encoding: \\ $\{-8, -7, -6, \dots , -1, 0, 1, \dots , 6, 7\}$ = 16 numbers

\Deactivate
\begin{table}[!hbp]
	\caption{Binary representations}
	\label{binary_representations}
	\centering
	\begin{tabular}{ | c | c | c | }
		\hline
		Codeword & Sign-magnitude decoding & 2's complement decoding \\
		\hline
		0000 &  0 &  0 \\
		0001 &  1 &  1 \\
		0010 &  2 &  2 \\
		0011 &  3 &  3 \\
		0100 &  4 &  4 \\
		0101 &  5 &  5 \\
		0110 &  6 &  6 \\
		0111 &  7 &  7 \\
		\hline
		1000 & -0 & -8 \\
		1001 & -1 & -7 \\
		1010 & -2 & -6 \\
		1011 & -3 & -5 \\
		1100 & -4 & -4 \\
		1101 & -5 & -3 \\
		1110 & -6 & -2 \\
		1111 & -7 & -1 \\
		\hline
	\end{tabular}
\end{table}
\Activate

\pagebreak

& \emph{Signed magnitude:} Binary encoding where the most significant bit represents whether the number is positive/negative (0 for positive, 1 for negative) and the other bits represent the value
	&& Conversion: Interpret the sign and value separately, then combine them
		&&& E.g. $-13_{10} = x_{2\ \text{(signed magnitude)}}$
		\medskip \\
		Sign = - = 1 \\
		Magnitude = $13_{10} = 1101_{2}$
		\medskip \\
		$\therefore -13_{10} = 1\ 1101_{2\ \text{(signed magnitude)}}$

		\bigskip

		&&& E.g. $100\ 1001_{2\ \text{(signed magnitude)}} = x_{10}$
		\medskip \\
		Sign = 1 = - \\
		Magnitude = $1001_{2} = 9_{10}$
		\medskip \\
		$\therefore 100\ 1001_{2\ \text{(signed magnitude)}} = -9_{10}$

		\bigskip

		&&& E.g. $1000_{2\ \text{(signed magnitude)}} = x_{10}$
		\medskip \\
		Sign = 1 = - \\
		Magnitude = $0$
		\medskip \\
		$\therefore 1000_{2\ \text{(signed magnitude)}} = -0_{10}$

%TODO CMPT 150 lecture notes, page 5

\end{easylist}


\section{The Field of Complex Numbers}
\subsection{Basic Definitions}
We construct the complex numbers in the following way:
\begin{definition}
    Consider the plane $\mathbb R^2$, we equip it with the \textit{complex multiplication} $\cdot:\mathbb R^2\times\mathbb R^2\to\mathbb R^2$ in a way that:
    $$(a,b)\cdot(c,d)=(ac-bd,ad+bc)$$
    If we denote $(x,y)$ by $x+iy$, the resulting field $(\mathbb R^2, +, \cdot, 1, 0)$ is called the complex numebrs, and is denotes by $\mathbb C$.
\end{definition}
\begin{proposition}
    $\mathbb C$ is indeed a field.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
Note that $i^2=-1$
\begin{definition}
    The conjugate $\bar z$ or $z^*$ of $z=x+iy\in\mathbb C$ is defined as $x-iy$.
    The modulus $|z|$ of $z$ is defined as $\sqrt{z\bar z}=\sqrt{x^2+y^2}$.
    The argument $\arg z$ of $z$ is the angle $\theta$ such that $z=|z|(\cos\theta+i\sin\theta)$, taken mod $2\pi$.
\end{definition}
The last expression is called the polar form of a complex number.
It is obvious that $|zw|=|z||w|$ and that $|z|=|\bar z|$.
\begin{claim}
    The argument of any complex number $z$ is defined.
\end{claim}
\begin{proof}
    Trivial.
\end{proof}
It is worth to note that $\tan\theta = y/x$.
Although there are infinitely many angles that can make the equality in the polar form, we often take the principal value, i.e. within $(-\pi,\pi]$.
But it is the most convenient to take it as a value in $\mathbb R/2\pi\mathbb Z$.\\
Of course, the complex numbers inherits the geometrical meanings of the plane $\mathbb R^2$. The geometric representation of it is called the Argand diagram.
On the Argand diagram, the argument is essentially the (anticlockwise) angle between the vector representating the complex number and the positive real axis.
The modulus, at the same time, is the length of that vector.
The addition and substraction of the complex numbers are the same as what we did it with 2D vectors (i.e. parallelogram law).\\
There is an important theorem associated with the polynomial in the complex numbers.
\begin{theorem}[Fundamental Theorem of Algebra]
    Any nonconstant polynomial in $\mathbb C$ has a root.
\end{theorem}
One should note easily that it is equivalent to say that a nonconstant complex polynomial of order $n$ has exactly $n$ roots.
The theorem means that the process of field extensions ends at $\mathbb C$.
\begin{proof}
    Later.
\end{proof}
\begin{proposition}[Triangle Inequality]
    $|z+w|\le |z|+|w|$
\end{proposition}
\begin{proof}
    Since both sides are positive, it is equivalent to its squared form:
    $$(z+w)(\bar z+\bar w)\le z\bar z+w\bar w+2|z||w|\iff \frac{1}{2}(z\bar w+\bar zw)\le |z\bar w|$$
    But this is just to say that $\Re (z\bar w)\le |z\bar w|$, which is true.
\end{proof}
\begin{corollary}
    Replacing $w$ by $w-z$ gives $|w-z|\ge |w|-|z|$.
    By symmetry $|w-z|\ge |z|-|w|$, so we have the general form
    $$|w-z|\ge||z|-|w||$$
\end{corollary}
\begin{proposition}
    Let $z_1=r_1(\cos \theta_1+i\sin\theta_1)$ and $z_2=r_2(\cos\theta_2+i\sin\theta_2)$, then
    $$z_1z_2=r_1r_2(\cos(\theta_1+\theta_2)+i\sin(\theta_1+\theta_2))$$
    That is, $\arg z_1+\arg z_2=\arg z_1z_2\pmod{2\pi}$
\end{proposition}
\begin{proof}
    Just compound angle formula suffices.
    It is known to be the De Movrie's Theorem.
\end{proof}
\begin{corollary}
    $(\cos\theta+i\sin\theta)^n=\cos(n\theta)+i\sin(n\theta)$
\end{corollary}
\begin{proof}
    Induction shows the case $n\in\mathbb N$, for negative $n=-m$, we have
    $$\text{LHS}=(\cos(m\theta)+i\sin(m\theta))^{-1}=\cos(m\theta)-i\sin(m\theta)=\text{RHS}$$
    That establishes it
\end{proof}
\subsection{Exponential and Trigonometric Functions}
\begin{definition}
    $$e^z:=\sum_{n=0}^\infty \frac{z^n}{n!}$$
\end{definition}
The series converges for all $z$ since it absolutely converges.
Also due to absolute convergence, we can multiply and arrange the series, which gives
$$e^ze^w=e^{z+w}$$
Note as well that $e^0=1$ and $(e^z)^n=e^{nz}$ for $n\in\mathbb Z$.
\begin{definition}
    $\cos(z)=(e^{iz}+e^{-iz})/2$, which gives the series
    $$\sum_{n=0}^\infty(-1)^n\frac{z^{2n}}{(2n)!}$$
    Similarly $\sin(z)=(e^{iz}-e^{-iz})/(2i)$, so its series expansion is
    $$\sum_{n=0}^\infty(-1)^n\frac{z^{2n+1}}{(2n+1)!}$$
\end{definition}
By differentiating the series term by term,
$$(\sin z)^\prime=\cos z, (\cos z)^\prime =-\sin z, (e^z)^\prime=e^z$$
\begin{theorem}
    $$e^{iz}=\cos z+i\sin z$$
\end{theorem}
Note that $\cos z$ is not necessarily real, same for $\sin z$.
But if $z$ is real, they are.
\begin{lemma}
    $e^z=1\iff z=2in\pi$ for some $n\in\mathbb Z$.
\end{lemma}
\begin{proof}
    Write $z=x+iy$, then we have
    $$e^{x+iy}=e^xe^{iy}=e^x(\cos y+i\sin y),x,y\in\mathbb R$$
    So $e^x\cos y=1$ and $e^x\sin y=0$.
    Solving it gives $x=0, y=0\pmod{2\pi i}$
\end{proof}
We have the following general form of complex number
$$z=|z|e^{i\arg z}$$
\subsection{Roots of Unity}
\begin{definition}
    $z\in\mathbb C$ is called an $n^{th}$ root of unity if $z^n=1$
\end{definition}
To find all solutions to $z^n=1$, we write
$$z=re^{i\theta}$$
so $r^n=1$ and $iN\theta=2\pi in$ for some $n\in\mathbb Z$.
This gives $n$ distinct solutions:
$$z=e^{2\pi in/N}$$
where $n\in \{0,1,2,\ldots ,n-1\}$
The roots of unity lie on the unit circle on the Argand diagram.
They are the vertices of a regular $n$-gon.
\subsection{Logarithms and Complex Powers}
\begin{definition}
    Define $w=\log z$ by $e^w=z$ since we want $\log$ to be the inverse of $\exp$, which is not injective.
    So $\log$ is multi-valued.
    $$\log z=\log|z|+i\arg z\pmod{2\pi i}$$
    We can, of course, make it single-valued by restricting $\arg z$ to the principal branch $(-\pi,\pi]$ or $[0,2\pi)$.
    But in this case, we do not have $\log(ab)=\log a+\log b$
\end{definition}
In fact, one can prove that it is impossible to choose a $\log$ in the complex plane that lives up to every one of our expectations.
\begin{example}
    If $z=-1$, then $\log z=i\pi\pmod{2\pi i}$.
\end{example}
\begin{definition}
    We define $z^\alpha=e^{\alpha\log z}$ for any $\alpha, z\in\mathbb C$ where $z\neq 0$.
    Note that since $\log$ is multi-valued, the complex powers are multi-valued in general.
    They differ by a multiplicative factor in the form $e^{2n\pi i\alpha},n\in\mathbb Z$.
\end{definition}
If $\alpha\in\mathbb Z$, then the power is single-valued.
If $\alpha\in\mathbb Q$, it is finite-valued.
But in general, a complex power admits infinitely many values.
\begin{example}
    1. We want to calculate $i^i$. $\log i=\pi/2+2\pi ni$, so
    $$i^i=e^{i\log i}=e^{-\pi/2+2\pi n}$$
    where $n\in\mathbb Z$.\\
    2. We want to calculate $(1+i)^{1/2}$, so it equals
    $$e^{1/2\log\sqrt 2+i(\pi/4+2n\pi i)}=2^{1/4}e^{i\pi/8}/2$$
\end{example}
\subsection{Lines and Circles}
For a fixed $w\in\mathbb C$ such that $w\neq 0$, the set of points $z=\lambda w$ is a line through the origin in the direction of $w$.\\
By shifting $z=z_0+\lambda w$ is a line parallel to $z=\lambda w$ though $z_0$.\\
To write this in the form without the real parameter $\lambda$, we take the conjugate
$$\bar z=\bar z_0+\lambda\bar w$$
Conbining the two equations and eleminate $\lambda$, we have $\bar wz-w\bar z=\bar wz_0-w\bar z_0$.
This is one standard form of the equation of a line (there are others though).\\
The equation of a circle centered at $c\in\mathbb C$ with radius $\rho$ is given by $|z-c|=\rho$, which is to say
$$(z-c)(\bar z-\bar c)=\rho^2\iff z\bar z-c\bar z-\bar cz=\rho^2-c\bar c$$
The general form of a point on the circle is $c+\rho e^{i\theta}$ where $\theta$ is a real parameter.\\
Note that in the geometrical viewpoint of $\mathbb C$, $z\mapsto z+z_0$ is a translation, $z\mapsto\lambda z$ where $\lambda\in\mathbb R$ is a scaling, $z\mapsto ze^{i\theta}$ where $\theta\in\mathbb R$ is a rotation, $z\mapsto\bar z$ is a translation, $z\mapsto 1/z$ is an inversion.
Comparing with groups, translation, rotation, scaling, inversion generates the Mobius group which normally acts on $\mathbb C_\infty=\mathbb C\cup\{\infty\}\cong S^2$ by stereographic projection.
On $\mathbb C_\infty$, lines in $\mathbb C$ are circles.