\chapter{External Interface Requirements}

\section{User Interfaces}

The \acrshort{csaf}  package exists as middleware. It exists as a library, and no CLI or GUI tools are planned.

\section{Software Interfaces}

\subsection{Mathematical Representations}
A component is required to be a dynamical system in some form, a mathematical entity that has the ability to
evolve in time. This means that a system assumes a function in a set $\left\{ f^t \right\}_{t \in \mathbb F}$,
where $\mathbb F$ is some set for time. For continuous time systems, this set can be the real numbers $
\mathbb R$; for discrete, the integers $\mathbb Z$ suffices. For systems that use both discrete and
continuous time, a relation between the two fields is necessary. A sampling frequency and sampling phase is
used if the discrete time system is uniform time step. This general form, a system as a set of functions, is not
conducive towards using in a software system, and another representation is desired. \\

Commonly, this set of functions can be characterized by a system of ordinary differential equations
(\acrshort{ode}s) for continuous time or ordinary difference equations for discrete time. These systems of
equations can be put into a form that permit a simple representation. For continuous time,

\begin{equation}
\begin{cases}
\dot x = g(t, x, u; p) \\
f_{x_0}^t = h(., x(.), u; r) \\
\end{cases}
\end{equation}

This form separates the solution into two steps with respect to an initial condition $x_0$. First, all variables
where the \acrshort{ode}s involve rates of change are grouped in a vector $x \in \mathbb R^N$, where it
assumes an element in a state space. $u \in \mathbb R^M$ is an additional time varying vector, representing
control input to the system. $p$ and $r$ are time independent parameters involved with the two functions.
Then, another function $h$ takes the space and returns the system configuration, evaluating one of the
system functions. Similarly, a construction can be made for a discrete system,

\begin{equation}
\begin{cases}
x_{n} = g(t, n-1, u; p)\\
f^{n}_{x_0} = h(., x(.), u; r) \\
\end{cases},
\end{equation}

for the discrete time system. The evolution of the state $x$ can be computed directly rather than integrating
the system of equations defining the state differential. Further, time itself need not be directly expressed, an
essential parameter for non-autonomous systems. The equivalence holds

\begin{equation} \label{equ:cequ}
\begin{bmatrix}
\dot x \\
t
\end{bmatrix} =
\begin{bmatrix}
g(t, x, u; p) \\
1
\end{bmatrix} \iff \dot x = g(t, x, u;p) .
\end{equation}

However, for the sake of implementation, it is advantageous to express the time parameter directly. To use
this form, all \acrshort{csaf}  components require two functions,

\begin{equation} \label{equ:dequ}
\begin{aligned}
g(.,.,.;p):& \mathbb F \times \mathbb R^N \times \mathbb R^M \rightarrow \mathbb R^N \\
h(.,.,.;r):& \mathbb F \times \mathbb R^N \times \mathbb R^M \rightarrow \mathbb R^D
\end{aligned}.
\end{equation}

For continuous time solution of a single component, the function $g(.,.,u(.);p)$ is of a form that common
\acrshort{ode} solvers can evaluate. The parameters $r$ and $p$ don't necessarily need to be elements of a
vector space, and can be described using other structures.

\subsubsection{Linear System Representation}

A linear system can take the form,

\begin{equation} \label{equ:linsys}
\begin{cases}
\dot{x} = A x + B u \\
y =C x + D u \\
\end{cases},
\end{equation}

where $x$ is in the state space and $u$ is in the input space. $A$, $B$, $C$, and $D$ are matrices
representing linear transformations.   \\

For a controlled system, the forcing input $u$ can be the result of a control policy. A control policy itself can
be dependent of the state vector $x$, as is the case for a state space controller. A linear controller follows a
linear control policy,

\begin{equation}
u = K (x_d -x ) = -Ke,
\end{equation}

where $K$ is some linear operator and $e=x -x_d$ represents an error signal from a desired state $x_d$. This
representation is commonly used, and controller design methods like \acrshort{lqr} and \acrshort{hinf}
methods, produce an operator $K$ that optimizes some notion of cost. From this view, it is clear that the
controller is itself a linear system, possessing no state and an input vector $e$. To match the linear
representation in Equation \ref{equ:linsys},

\begin{equation}
\begin{cases}
\dot{w} = \mathbf{0} w + e \\
u = -K e + G w \\
\end{cases}
\end{equation}

The effect of $w$ is to integrate $e$ with respect to time. This system can represent controllers with tracking
properties, like a \acrshort{pid} controller.

\subsubsection{Neural Controller Representation}\label{sec:nnc}

Neural networks can be used to produce the functions $g$ and $h$ of a controller system, being trained
beforehand and operating feedforward during use. In this, no distinction is made is made in the representation. \\

\subsubsection{Fuzzy Logic Controller Representation}\label{sec:flc}

A fuzzy logic controller is a controller that uses a \acrlong{fls} to determine its output. Generally, this involves
taking the input and transforming them to linguistic variables $(x_i, x_2, ..., x_{n_i}) \in \mathbb U^{n_i}$. Then,
an inference rule $\mathbb R^j$ can be applied from a rules base to produce a linguistic variable describing
the controller output, being a non-linear mapping between two fuzzy sets $\mathbb U \rightarrow \mathbb R$.
\begin{equation}
\mathbb{R}^{(j)}:\text{ IF }x_{1}\text{ is }A_{1}^{j}\text{ and }\ldots\text{ and }x_{n i}\text{ is }A_{n i}^{j}\text{ THEN }y\text{ is }B^{j}, j=1, \ldots, n_{j},
\end{equation}
where $A_i^j$, $B^j$ are fuzzy sets in $\mathbb U$ and $\mathbb R$. This set can be ``defuzzified'' into a crisp value $y \in \mathbb R$ to produce a control signal. \\

A controller adhering to this paradigm is stateless, and need only implement the output mapping $h$
mentioned previously that performs the fuzzification, inference, and defuzzification. Besides this, the fuzzy
sets that the input and output assume, the fuzzication/defuzzification startegies, and the inference rules employed, are necessary for the fuzzy logic controller representation.

\subsection{Model Interfaces}

A concept of a model is derived from the representations in \ref{equ:cequ} and \ref{equ:dequ}. Figure \ref{fig:mio} presents the IO relationships of a \acrshort{csaf} model. The function arguments serve as the input, and evaluations of $g$ and $h$ can be the output.  A third output, labeled ``info'' is available to pass specific information about the model representation (such as the ones seen in \ref{sec:flc} and \ref{sec:nnc}). Note that the model does not use any time varying parameter as a state, requiring all such quantities having to be passed manually. The state of a dynamical system does not serve as a state represented by the programmatic object model. Time invariant parameters are states in the object and settable.

\begin{figure}
\centering
\includegraphics[width=1.0\textwidth]{model.pdf}
\caption{\acrshort{csaf} Model IO. A model requires time, state, and input to be passed in order to collect output. A model can have associated parameters that can be passed in once and set.}
\label{fig:mio}
\end{figure}

\subsection{Component Interfaces}

A component is an object that presents a model as an element of a publish/subscribe architecture, with it IO diagram presented in Figure \ref{fig:cio}. It subscribes to topics received by its input sockets. For normal inputs, the messages received are deserialized into a control signal that is stored into an input buffer. This buffer stores the necessary members to update a system represented by a \acrshort{csaf} model object. The model output is serialized into a collection of output messages, and then send over a single output socket.\\

An events socket is also required (input c); messages can be sent in order to configure the component internals. These actions are

\begin{enumerate}
\item reset the buffers to default
\item configure buffer defaults
\item enable debugging
\item pause/resume the component
\item update the component
\item delete the component, unbinding/disconnecting any ports
\end{enumerate}

\begin{figure}
\centering
\includegraphics[width=1.0\textwidth]{componentio.pdf}
\caption{\acrshort{csaf}  Component IO. For input, sockets receive a message with specific topics. The output is a single socket, producing messages with different topics.}
\label{fig:cio}
\end{figure}

\subsection{System Description/Initialization}

Once valid components are made, they can be composed together to make systems. This composition can be described in a TOML file (Code Block \ref{fig:sysset}).

\begin{figure}
\begin{lstlisting}
name = "inverted-pendulum"

# directory setup
codec_dir = "codec"
output_dir = "output"

# order to construct/evaluate components
evaluation_order = ["maneuver", "controller", "plant"]

# log setup
log_file = "inv_pendulum.log"
log_level = "info"

# port to broadcast events to the components
events_port = 6001

[components]

  [components.controller]
    # environment to run under
    run_command = "python3"

    # path to model executable
    process = "ipcontroller.py"

    # whether to print debug diagnostics
    debug = false

    # subscribe to topic of a component (component name, topic name)
    sub = [["plant", "states"], ["maneuver", "outputs"]]

    # port to publish
    pub = 5502

  [components.plant]
    run_command = "python3"
    process = "ipplant.py"
    debug = false
    sub = [["controller", "outputs"]]
    pub = 5501

  [components.maneuver]
    run_command = "python3"
    process = "ipmaneuver.py"
    debug = false
    sub = []
    pub = 5503
\end{lstlisting}
\caption{Example TOML File Describing Components and System Compositions of Components}
\label{fig:sysset}
\end{figure}

\section{Communications Interfaces}

Analyzers and simulators will interact with the system components via \acrshort{0mq}, using sockets. \acrshort{0mq}
provides \acrshort{csaf}  with a variety of levels to communicate with components, whether in-process,
inter-process or across TCP and multi-cast. The component does hold state, and the system needs to be
properly initialized to avoid staleness.

\subsection{Message Contents}

The temporal messages are sent by every \acrshort{csaf}  component that represent vectors. The message type is derived from the \acrshort{ros}msgs serialization format. First, the the
\acrshort{csaf}  version is transmitted to avoid incompatibility between components made with different
versions. Second, system time is included to ensure that components evaluate
correctly. Next is the vector. These vectors are enumerated under a header, rather than transmitted as a
contiguous array. Figure \ref{fig:cmsg} shows an example message. \\

Non-temporal aspects of the system need to need to be accessed, which is described in a configuration file associated with a component. The system name, parameters, representation identifier and solver name is included. Two booleans are used to determine if the system type---continuous, discrete or hybrid.  Fields specific to the representation of the system is  accessible. For example, if the system is a fuzzy controller, the inference table can be visible. As the  components have a varying degree of transparency (``black box''), no headers are required and parameters are allowed to unchangeable. \\

A controller's representation and purpose influences its requirements. As such, much
variability can exist in what is contained in the message. New headers and variables are allowed to appear in a
component message, but the structure has to remain constant over time. \\

\begin{figure}
\begin{lstlisting}
uint32 version_major
uint32 version_minor

string topic

float64 time
\end{lstlisting}
\caption{Required \acrshort{ros}msg \acrshort{csaf}  Component Temporal Message Contents (Other Fields are Permitted)}
\label{fig:cmsg}
\end{figure}

\begin{figure}
\begin{lstlisting}
system_name = "Inverted Pendulum Plant"
system_representation = "black box"
system_solver = "Euler"

sampling_frequency = 100

is_discrete = false
is_hybrid = false

[parameters]
  mm = 0.5        # Mass of the cart
  m = 0.2         # Mass of the pendulum
  b = 0.1         # coefficient of friction on cart
  ii = 0.006      # Mass Moment of Inertia on Pendulum
  g = 9.8         # Gravitational acceleration
  length = 0.3    # length of pendulum

[inputs]
  msgs = ["ipcontroller_output.msg"]

[topics]

  [topics.states]
    msg = "ipplant_state.msg"
    initial = [0.0, 0.01, 0.52 , -0.01]
\end{lstlisting}
\caption{ \acrshort{csaf}  Component Configuration Information)}
\label{fig:cmsg}
\end{figure}


\subsection{Arithmetic Logic Units (ALUs)}

\subsubsection{Half adder}

We first introduce the half adder

Take two inputs \(A\) and \(B\) and put both inputs to two binary operators.

The operators are an XOR gate and an AND gate.

The AND gate only returns \(1\) if both inputs are \(1\). The XOR gate returns \(1\) if only one input is \(1\).

The XOR gate returns the second “digit” while the AND date returns the first.

\subsubsection{Full adder}

This can be used to add \(0\) and \(1\), but not larger numbers. To do this we introduce the full adder. This adds two numbers, as before, and allows any remainder from the previous addition to be carried forward.

Full adders can be stringed together to add larger numbers.

\subsubsection{Subtraction}

Subtraction works similarly to addition. The equivalent “half adder” returns different values, and the carry forward is not used for addition, but for subtraction.

\subsection{Multiplication and division of natural numbers}

mult x2 is bitshiftleft, /2 is bitshift right) (how to divide when not divisible)

\subsection{Mod and remainders}

\subsection{Powers, logarithms and exponentials}


\chapter{Nonparametric Density Estimation}
This chapter is under \work.
\section{Histogram}
Let $x_1,\ldots,x_n$ be independent and identically distributed samples from a univariate continuous RV $X$ with density $f$. The simplest nonparametric estimator for $f$ is the histogram.  It has some serious drawbacks.  We have encountered the histogram earlier in \ref{S:ExploringData}.  During that encounter we chose the number of bins in an {\em ad hoc} fashion, often resorting to the default number of bins of $10$ in {\sc Matlab}'s {\tt hist} function.

A bin width that is too small when the number of bins is too large will give a histogram with many empty bins and many bins containing only a single data point.  This is referred as {\em under-smoothing}.  At the other extreme, a bin width that is too large due to a small number of bins results in a histogram that lacks details and results in {\em over-smoothing}.  In both of these situations, the histograms do not represent the underlying density well.

\begin{definition}[Density Histogram]
Let $x_1,\ldots,x_n$ be a random univariate sample with density $f$ and let $S_f$ be the support of $f$. Let $S_x\subset S_f$ be a connected interval containing $x_1,\ldots,x_n$ and let $\{B_1,\ldots,B_m\}$ be a finite contiguous partition of $S_x$ into $m$ bins of equal width $b$. For $x\in S_x$ and $B\in \{B_1,\ldots,B_m\}$, let $B(x)$ denote the bin that contains $x$. The {\it density histogram} for $x_1,\ldots,x_n$ with {\it bin width} $b$ is:
\begin{equation}
\widehat{f}_n(x,b)=\frac{1}{nb}\sum^n_{i=1}I_{B(x)}(x_i).
\end{equation}
\end{definition}

Hence, the density at $x$ is estimated by counting the number of sample points in the bin containing $x$, and then appropriately normalising this number so that the area covered by the histogram is 1.

%Consider the $a$-parametric family of the stretched oscillating exponential density with $\lambda = 9/20$, $\epsilon=1/2$:
%\[
%f_a(x) = a^{ {\lambda}^{-1} }{\Gamma (1+ {\lambda}^{-1})} \, \exp \{- a x^{\lambda} \}
%\left( 1 + \epsilon \, \sin ( a \, x^{\lambda} \, \tan( \lambda \pi  ) ) \right)
%\]

Now, let us consider the problem of estimating a histogram from $1500$ samples simulated from the equi-weighted mixture of $\normal(0,5^2)$, $\normal(10,1)$ and $\normal(-10,1)$ using the following code:
\begin{VrbM}
rand('twister',6898962)
randn('state',23121);
A=ceil(3*rand(1,2000));% Mixture Label vector
MuSs=[0 5; 10 2; -10 2]%; 5 2; -5 2]
x=arrayfun(@(i)(MuSs(i,1)+MuSs(i,2)*randn),A);
xgrid=-20:1:20;
pdf=NormalPdf(xgrid,MuSs(1,1),(MuSs(1,2))^2)/3 + NormalPdf(xgrid,MuSs(2,1),(MuSs(2,2))^2)/3 ...
    + NormalPdf(xgrid,MuSs(3,1),(MuSs(3,2))^2)/3;
\end{VrbM}

\begin{figure}[htpb]
\caption{Histogram estimates for the  with nine different bin-widths of $\{2,4,6,8,11,14,17,20,35\}$.  The fifth histogram with a bin-width of $11$ is the optimal one with the right amount of smoothing.  The ones with smaller bin-widths are under-smoothed and those with larger bin-widths are over-smoothed.\label{F:Gauss3MixHistsCV}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/Gauss3MixHistsCV}}
\end{figure}

\subsubsection{Drawbacks of the histogram}
\begin{asparaenum}[(a)]
\item While densities of continuous random variables are continuous, the histogram is a step function with discontinuities.

\item The histogram does not use data efficiently\footnote{With an optimally chosen bin width, the mean integrated squared error, $E[\int [\widehat{f}_n(x,b)-f(x)]^2dx]$, converges to 0 at a rate of $n^{-2/3}$.}.

\item Although only one parameter, the bin width, appears explicitly in the definition of the histogram, the use of the histogram requires the specification of a second parameter. This is the placement of the leftmost bin edge, which can strongly affect the resulting histogram.
\end{asparaenum}

Because of these drawbacks, the histogram should only be used as a graphical tool for exploratory data analysis.

\subsubsection{Selection of histogram bin width}
\begin{asparaenum}[(a)]
\item	{\it Sturges' rule}: The number of bins is given by:
\begin{equation}
m=ceil(1+\log_2n).
\end{equation}
In practice, the bin width is usually obtained by:
\begin{equation}
b=\frac{x_{(n)}-x_{(l)}}{m}.
\end{equation}
\item	{\it Scott's normal reference rule}:
\begin{equation}
b=3.5\widehat{\sigma}n_{-1/3},
\end{equation}

where $\widehat{\sigma}$ is the sample standard deviation.

\item	{\it Freedman-Diaconis' rule}:
\begin{equation}
b=2(\widehat{q}_{0.75}-\widehat{q}_{0.25})n^{-1/3},
\end{equation}
where $\widehat{q}_p$ is the sample $p$-quantile.\\
With Scott's bin width or Freedman-Diaconis' bin width, the number of bins can be obtained by:
\begin{equation}
m= \lceil \left(\frac{x_{(n)}-x_{(1)}}{b}\right) \rceil.
\end{equation}
\end{asparaenum}

In practice, these methods for determining bin width should be used as starting points for trying several possible bin widths, with the final bin width chosen by visual inspection of the resulting histograms.

A suggestion for the placement of the leftmost bin edge is to use a {\em centred histogram}. For a histogram with $m$ bins and bin width $b$, $mb\geq x_{(n)}-x_{(1)}$, and so let:
\begin{equation}
\delta=mb-(x_{(n)}-x_{(1)}).
\end{equation}

To get the centred histogram, place the leftmost bin edge at $x_{(1)}-\delta/2$, so that the histogram extends equally by $\delta/2$ beyond the smallest and largest data values.
\subsubsection{{\sc Matlab} function for density histogram}

\VrbMf[label=histogram.m]{scripts/histogram.m}
%\begin{VrbM}
%% histogram.m
%% Plots density histogram for data in X.
%%
%% Usage: histogram(X, plot data, bounds, colour, bwmethod, bw);
%%
%% Input: X = row vector of data,
%%        plotdata (binary) = plot data points?
%%        bounds = [lower bound , upper bound] for possible X values,
%%        colour (single-character string) = colour of histogram (default =
%%        'y' for yellow),
%%        bwmethod (optional, default = 2) = method of computing bin width:
%%                 0 = Scott's normal reference rule,
%%                 1 = Wand's one-stage rule,
%%                 2 = Wand's two-stage rule,
%%                 3 = manual,
%%        bw = manual bin width if bwmethod = 3.
%%
%% Remark: Bin origin determined by centring the histogram, i.e. so that
%% left and right bin edges extend beyond min(X) and max(X) respectively
%% by equal amounts.
%%
%% Reference: Wand M.P. (1997), "Data-based choice of histogram bin width",
%% American Statistician 51, 59-64.
%\end{VrbM}

%\begin{example}[Shuttle Data] Joint temperatures of the O-rings for each test firing or actual launch of the space shuttle rocket motor are shown in the table below (from {\it Presidential Commission on the Space Shuttle Challenger Accident}).
%\begin{center}
%\begin{tabular}{|cccccccccc|}\hline
%\multicolumn{10}{|c|}{ O-ring temperatures ($^o$F)}\\\hline
%31   &  40&     45   &  49&     52   &  53&     57  &   58&     58  &   60\\
%61   &  61 &    63  &   66 &    67  &   67 &    67 &    67 &    68 &    69\\
%70    & 70  &   70 &    70  &   72 &    73  &   75&     75  &   76&     76\\
%78     &79   &  80&     81   &  83&     84&&&&\\\hline
%\end{tabular}
%\end{center}
%%\includegraphics
%\end{example}

%%% Dominic's material here
%\chapter{Density estimation}
%%\section{Histogram}
%WORK - fuse with previous material
Let $x_1,\ldots,x_n$ be a random univariate sample from a continuous distribution with density $f$. The most common nonparametric estimator for $f$ is the histogram, which has some serious drawbacks.
\subsection{\define}Let $x_1,\ldots,x_n$ be a random univariate sample with density $f$ and let $S_f$ be the support of $f$. Let $S_x\subset S_f$ be a connected interval containing $x_1,\ldots,x_n$ and let $\{B_1,\ldots,B_m\}$ be a finite contiguous partition of $S_x$ into $m$ bins of equal width $b$. For $x\in S_x$ and $B\in \{B_1,\ldots,B_m\}$, let $B(x)$ denote the bin that contains $x$. The {\it density histogram} for $x_1,\ldots,x_n$ with {\it bin width} $b$ is:
\begin{equation}
\hat{f}_H(x,b)=\frac{1}{nb}\sum^n_{i=1}I_{B(x)}(x_i).
\end{equation}
%\begin{flushright}   $\boxbox$ \end{flushright}

Hence, the density at $x$ is estimated by counting the number of sample points in the bin containing $x$, and then appropriately normalising this number so that the area of the histogram equals 1.

\begin{example}
Suppose $x_1,\ldots,x_{500}$ is a random sample drawn from the standard normal distribution. A histogram for $x_1,\ldots,x_{500}$ is shown in the figure:
%\includegraphics
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{example}

\subsection{Drawbacks of the histogram}
\begin{asparaenum}[(a)]
\item While densities of continuous random variables are continuous, the histogram is a step function with discontinuities.

\item The histogram does not use data efficiently\footnote{With an optimally chosen bin width, the mean integrated squared error, $E[\int [\hat{f}_H(x,b)-f(x)]^2dx]$, converges to 0 at a rate of $n^{-2/3}$.}.

\item Although only one parameter, the bin width, appears explicitly in the definition of the histogram, the use of the histogram requires the specification of a second parameter. This is the placement of the leftmost bin edge, which can strongly affect the resulting histogram.
\end{asparaenum}
Because of these drawbacks, the histogram should only be used as a graphical tool for exploratory analysis.
%\begin{flushright}   $\boxbox$ \end{flushright}

\subsection{Selection of histogram bin width}
\begin{asparaenum}[(a)]
\item	{\it Sturges' rule}: The number of bins is given by:
\begin{equation}
m=ceil(1+\log_2n).
\end{equation}
In practice, the bin width is usually obtained by:
\begin{equation}
b=\frac{x_{(n)}-x_{(l)}}{m}.
\end{equation}
\item	{\it Scott's normal reference rule}:
\begin{equation}
b=3.5\hat{\sigma}n^{-1/3},
\end{equation}

where $\hat{\sigma}$ is the sample standard deviation.

\item	{\it Freedman-Diaconis' rule}:
\begin{equation}
b=2(\hat{q}_{0.75}-\hat{q}_{0.25})n^{-1/3},
\end{equation}
where $\hat{q}_p$ is the sample $p$-quantile.\\
With Scott's bin width or Freedman-Diaconis' bin width, the number of bins can be obtained by:
\begin{equation}
m=ceil\left(\frac{x_{(n)}-x_{(l)}}{b}\right).
\end{equation}
\end{asparaenum}
%\begin{flushright}   $\boxbox$ \end{flushright}

In practice, these methods for determining bin width should be used as starting points for trying several possible bin widths, with the final bin width chosen by visual inspection of the resulting histograms. A bin width that is too small will give a histogram with many empty bins and many bins containing only a single data point. At the other extreme, a bin width that is too large results in a histogram that lacks details. In both of these situations, the histograms do not represent the underlying density well.

\begin{example}[Shuttle data]
Joint temperatures of the O-rings for each test firing or actual launch of the space shuttle rocket motor are shown in the table below (from {\it Presidential Commission on the Space Shuttle Challenger Accident}).
\begin{center}
\begin{tabular}{|cccccccccc|}\hline
\multicolumn{10}{|c|}{ O-ring temperatures ($^o$F)}\\\hline
31   &  40&     45   &  49&     52   &  53&     57  &   58&     58  &   60\\
61   &  61 &    63  &   66 &    67  &   67 &    67 &    67 &    68 &    69\\
70    & 70  &   70 &    70  &   72 &    73  &   75&     75  &   76&     76\\
78     &79   &  80&     81   &  83&     84&&&&\\\hline
\end{tabular}
\end{center}
%\includegraphics
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{example}

A suggestion for the placement of the leftmost bin edge is to use a \textquotedblleft centred histogram". For a histogram with $m$ bins and bin width $b$, $mb\geq x_{(n)}-x_{(l)}$, and so let:
\begin{equation}
\delta=mb-(x_{(n)}-x_{(l)}).
\end{equation}

To get the centred histogram, place the leftmost bin edge at $x_{(l)}-\delta/2$, so that the histogram extends equally by $\delta/2$ beyond the smallest and largest data values.
%\subsection{\Matlab {\it function for density histogram.}}
%\begin{VrbM}
%% histogram.m
%% Plots density histogram for data in X.
%%
%% Usage: histogram(X, plot data, bounds, colour, bwmethod, bw);
%%
%% Input: X = row vector of data,
%%        plotdata (binary) = plot data points?
%%        bounds = [lower bound , upper bound] for possible X values,
%%        colour (single-character string) = colour of histogram (default =
%%        'y' for yellow),
%%        bwmethod (optional, default = 2) = method of computing bin width:
%%                 0 = Scott's normal reference rule,
%%                 1 = Wand's one-stage rule,
%%                 2 = Wand's two-stage rule,
%%                 3 = manual,
%%        bw = manual bin width if bwmethod = 3.
%%
%% Remark: Bin origin determined by centring the histogram, i.e. so that
%% left and right bin edges extend beyond min(X) and max(X) respectively
%% by equal amounts.
%%
%% Reference: Wand M.P. (1997), "Data-based choice of histogram bin width",
%% American Statistician 51, 59-64.
%\end{VrbM}

\section{Kernel density estimation}
The {\it kernel density estimator} is a nonparametric estimator for $f$ that is continuous, uses data more efficiently than the histogram, and has only one parameter.

\begin{definition}
Let $x_1,\ldots,x_n$ be a random univariate sample with density $f$, and let $K$ be a function satisfying:
\begin{equation}
\int K(x)dx=1.
\end{equation}

The {\it kernel density estimator} for $f$ with bandwidth $h > 0$ and kernel $K$ is:
\begin{equation}
\hat{f}_K(x,h)=\frac{1}{nh}\sum^n_{i=1}K\left(\frac{x-x_i}{h}\right).
\end{equation}

We shall consider kernel functions that are densities and define:
\begin{equation}
K_h(x)=\frac{1}{h}K\left(\frac{x}{h}\right),
\end{equation}
and so the kernel density estimator can be expressed as:
\begin{equation}
\hat{f}_K(x,h)=\frac{1}{n}\sum^n_{i=1}K_h(x-x_i),
\end{equation}
which is a proper density.
\end{definition}

%\begin{flushright}   $\boxbox$ \end{flushright}
\subsection{Examples of kernel functions}

\begin{table}[tbh]
\centering
$$\begin{array}{|c|c|c|c|}\hline
\textrm{Name}&	K(x)	&\textrm{Efficiency}&\sigma_K\\
Epanechnikov	&\frac{3}{4}(1-x^2)I_{[-1,1]}(x)&1&\frac{1}{\sqrt{5}}	 \\
Biweight	 &\frac{15}{16}(1-x^2)^2I_{[-1,1]}(x)&0.994	 &\frac{1}{\sqrt{7}}\\
Triweight	 &\frac{35}{32}(1-x^2)^3I_{[-1,1]}(x)&0.987	 &\frac{1}{3}\\
Triangle&(1-|x|)I_{[-1,1]}(x)&0.986	 &\frac{1}{\sqrt{6}}\\
Gaussian&\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{x^2}{2}\right)	 &0.951&	1\\
Uniform&\frac{1}{2}I_{[-1,1]}(x)&0.930&\frac{1}{\sqrt{3}}	 \\\hline
\end{array}$$
\end{table}
%\includegraphics
%\begin{flushright}   $\boxbox$ \end{flushright}
It turns out that the choice of the kernel has little impact on the performance of the kernel density estimator, and so the convenient Gaussian kernel is a popular choice. The choice of bandwidth, on the other hand, is critical.

\begin{prop}
Let $\sigma^2_K$ be the variance of kernel function $K$. The variance of $K_h(x)$ is $h^2\sigma^2_K$.

\begin{proof}
Since $K_h(x)$ has mean 0, its variance is:
$$ \sigma_K^2=\int x^2K_h(x)dx=\frac{1}{h}\int x^2K(x/h)dx=h^2\int (x/h)^2K(x/h)d(x/h)=h^2\sigma^2_K.$$
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{proof}
\end{prop}

Note that for the Gaussian kernel, $\sigma_K=1$, and therefore $\sigma_K=h$; i.e. the standard deviation of $K_h$ is equal to its bandwidth.

\begin{labwork}
Using the same standard normal sample from Example 5.1.2, the density estimate obtained from a Gaussian kernel density estimator (with bandwidth 0.3) is shown in the figure, along with the histogram obtained previously.
%\includegraphics
%\Matlab code:
\begin{VrbM}
n = 500;
x = randn(1,n);
bw = 0.3;
s = [-3:0.01:3]; % support points to compute density
f = normpdf(s,0,1); % true N(0,1) density
ns = length(s); % number of support points
fker = zeros(1,ns); % storage for kernel density
for i = 1:ns
    fker(i) = mean(normpdf(s(i),x,bw));
end
histogram(x,0,[-inf inf],'g'); hold on
plot(s,f,':r',s,fker,'-r')
\end{VrbM}
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

We can think of the kernel as spreading a probability mass of $1/n$ associated with each sample point around its neighbourhood. To see what the Gaussian kernel density estimator does, suppose we have only five sample points, $x_1,\ldots,x_5$. The estimator puts a normal density with variance $h^2$ and a probability mass of $1/n$ at each sample point, and estimates the density at support point $x$ by summing the contributions from these normal densities at $x$. This is illustrated in the figure, where the sample points are marked by \textquoteleft x' on the support axis, the dashed curves are the normal densities whose probability masses are scaled to $1/n$, and the solid curve is the estimated density.
%\includegraphics

If f is continuous at $x$ and $h\rightarrow 0$ and $nh\rightarrow\infty$ as $n\rightarrow\infty$, then for any $\epsilon > 0$:
\begin{equation}
\lim_{n\rightarrow\infty}P(|\hat{f}_K(x,h)-f(x)|\leq\epsilon)=1.
\end{equation}
%\begin{flushright}   $\boxbox$ \end{flushright}

\subsection{Bandwidth selection}
The use of the kernel density estimator requires the specification of the bandwidth, which plays a similar role as the histogram's bin width. A practical and easy-to-use bandwidth choice for the Gaussian kernel is {\it Scott's bandwidth}:
\begin{equation}
h=\hat{s}n^{-1/5},
\end{equation}

where $\hat{s}=\min\{\hat{\sigma},(\hat{q}_{0.75}-\hat{q}_{0.25})/1.348\}$. A slightly different version\footnote{This version is also known as the {\it normal reference rule.}} of Scott's bandwidth is given by:
\begin{equation}
h=1.06\hat{s}n^{-1/5}.
\end{equation}

Since the multiplicative constant of 1.06 is very close to 1, the two bandwidths are very close and we can use either of them.

A more sophisticated bandwidth can be obtained by considering the minimisation of {\it asymptotic mean integrated squared error}:
\begin{equation}
AMISE=\lim_{n\rightarrow\infty}E[\int[\hat{f}_K(x,h)-f(x)]^2dx].
\end{equation}

Using $R(g)$ to denote $\int g(x)^2dx$, the resulting optimal bandwidth is given by:
\begin{equation}
h_{AMISE}=\left(\frac{R(K)}{n\sigma^4_KR(f^{\prime\prime})}\right)^{1/5},
\end{equation}

which depends on the unknown density $f$ through $R(f^{\prime\prime})$. An approximate bandwidth is obtained by plugging in an appropriate estimate of $R(f^{\prime\prime})$\footnote{Sheather, S. J. and Jones, M. C. (1991), \textquotedblleft A reliable data-based bandwidth selection method for kernel density estimation", {\it Journal of the Royal Statistical Society, Series B, 53, 683-690}.}.

With an optimally chosen bandwidth, the mean integrated squared error of the kernel density estimator converges to 0 at rate $n^{4/5}$.
%\begin{flushright}   $\boxbox$ \end{flushright}

%\subsection{\result}
Let $K$ and $L$ be two kernel functions with standard deviations $\sigma_K$ and $\sigma_L$ respectively. Then for the resulting kernel densities to be approximately equal, their bandwidths, $h_K$ and $h_L$, must satisfy:
\begin{equation}
h_K\sigma_K\approx h_L\sigma_L,
\end{equation}
i.e. the standard deviations of $K_h$ and $L_h$ must be approximately equal.
%\begin{flushright}   $\boxbox$ \end{flushright}

Recall that the standard deviation of the Gaussian kernel function is 1, so we can use this result to obtain the bandwidth for some other kernel, e.g. kernel function $K$, from the bandwidth of the Gaussian kernel:
\begin{equation}
h_K\approx \frac{h_{gaussian}}{\sigma_K}.
\end{equation}

For example, with the Epanechnikov kernel, $\sigma_K=1/\sqrt{5}$, and so Scott's bandwidth for the Epanechnikov kernel is:$$h_K\approx\sqrt{5}\hat{s}n^{-1/5}.$$

\begin{labwork}
The data in {\tt geyser.txt} are 107 durations (in minutes) of the eruptions of the Old Faithful geyser. Compare the kernel density estimates for eruption duration using the Gaussian and Epanechnikov kernels with Scott's bandwidth.

%\Matlab code:
\begin{VrbM}
x = load('geyser.txt');
n = length(x);
n5 = n^(-0.2);
hscottgauss = min(std(x),iqr(x)/1.348) * n5 % Scott's bandwidth for Gaussian
			% kernel
hscottepan = sqrt(5) * hscottgauss % Scott's bandwidth for Epanechnikov kernel
\end{VrbM}
Results:
\begin{VrbM}
hscottgauss = 0.4087
hscottepan = 0.9139
\end{VrbM}
%\includegraphics
%\caption{Kernel densities with Scott's bandwidth (solid curve) and optimal bandwidth (dashed curve).}
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

\subsection{Adjustment at boundaries}
Suppose we need to estimate a density $f$ whose support $S_f$ is bounded at one or both ends; frequently, $S_f=(0,\infty)$ or $S_f=(a,b)$. The presence of a boundary or boundaries may cause some of the kernels in the kernel density estimator to be truncated. Therefore, the estimator must be adjusted accordingly to ensure that the resulting estimate remains a density:
\begin{equation}
\hat{f}_K(x,h)=\frac{1}{n}\sum^n_{i=1}\frac{K_h(x-x_i)}{p_i},
\end{equation}
where:
\begin{equation}
p_i=\int_SK_h(x-x_i)dx.
\end{equation}

Thus, $0<p_i\leq 1$, and $p_i  = 1$ if the corresponding kernel is not truncated.

If the Gaussian kernel is used and letting $\Phi(t;\mu,\sigma^2)$ denote the cumulative probability at $t$ of the normal distribution with mean $\mu$ and variance $\sigma^2$, then:
\begin{equation}
p_i=1-\Phi(0;x_i,h^2),
\end{equation}
when $S_f=(0,\infty)$, and:
\begin{equation}
p_i=\Phi(b;x_i,h^2)-\Phi(a;x_i,h^2),
\end{equation}
when $S_f=(a,b)$.
%\begin{flushright}   $\boxbox$ \end{flushright}

\begin{labwork}
Let $x_1,\ldots,x_{100}$ be a sample from an exponential distribution with parameter 1. Recall the the support of the exponential distribution is $x \geq 0$.
%\includegraphics
%\Matlab code:
\begin{VrbM}
n = 100;
x = exprnd(1,1,n); % n exponential(1) random variables
n5 = n^(-0.2);
h = min(std(x),iqr(x)/1.348) * n5; % Scott's bandwidth for Gaussian kernel
s = [0:0.01:4]; % support points to compute density
f = exppdf(s,1); % true exponential(1) density
ns = length(s); % number of support points
fker1 = zeros(1,ns); % storage for kernel density without boundary correction
fker2 = zeros(1,ns); % storage for kernel density with boundary correction
p = 1 - normcdf(0,x,h); % boundary correction factors
for i = 1:ns
    fker1(i) = mean(normpdf(s(i),x,h)); % kernel density without boundary
		% correction
    fker2(i) = mean(normpdf(s(i),x,h)./p); % kernel density with boundary
		% correction
end
plot(s,f,'-b',s,fker2,'--b',s,fker1,':b')
legend('true density','kernel density with correction','kernel density without correction')
\end{VrbM}
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

\section{Extension to multivariate data}
The extension of the kernel density estimator to multivariate data is straightforward.

%\begin{definition}
Let $x_1,\ldots,x_n$ be independent and identically distributed random $d$-vectors with $d$-dimensional density $f$. The kernel density estimator for $f$ is:
\begin{equation}
\hat{f}_K(x,H)=\frac{1}{n|H|}\sum^n_{i=1}K(H^{-1}(x-x_i)),
\end{equation}
where $H$ is a $d \times d$ nonsingular matrix, called the {\it bandwidth matrix}, and the kernel function $K$ is a $d$-dimensional density. Let:
\begin{equation}
K_H(x)=\frac{K(H^{-1}x)}{|H|},
\end{equation}

so that:
\begin{equation}
\hat{f}_K(x,H)=\frac{1}{n}\sum^n_{i=1}K_H(x-x_i).
\end{equation}

The kernel can be simplified by assuming that its components are independent (note that this does not make the components of $x$ independent). This simplifies the kernel matrix to a diagonal matrix with diagonal elements, $h_1,\ldots,h_d$. The kernel can then be expressed as a product of univariate kernels:
\begin{equation}
K_H(x)=K_{h_1,\ldots,h_d}(x)=\prod^d_{j=1}K_{h_j}[x(j)],
\end{equation}
where $x(j)$ denotes the $j^{\textrm{th}}$ component of $x$. This gives the {\it product kernel density estimator}:
\begin{equation}
\hat{f}_K(x,h_1,\ldots,h_d)=\frac{1}{n}\sum^n_{j=1}[\prod^d_{j=1}K_{h_j}[x(j)-x_i(j)]].
\end{equation}
%\begin{flushright}   $\boxbox$ \end{flushright}
%\end{definition}

\subsection{Bandwidth selection}
For a product kernel density estimator with the Gaussian kernel, {\it Scott's rule in d dimensions} is:
\begin{equation}
h_j=\hat{s}_jn^{-1/4+d},
\end{equation}
where $\hat{s}_j=\min(\hat{\sigma}_j,(\hat{q}_{0.75j}-\hat{q}_{0.25j})/1.348)$ is the estimate of scale for the $j^{\textrm{th}}$ component that has been computed from $x_1(j),\ldots,x_n(j)$. For some other kernel, the required bandwidth may be obtained by dividing $h_j$ by the standard deviation of that kernel function, as in the univariate case.

With an optimally chosen bandwidth, the mean integrated squared error of the multivariate kernel density estimator converges to 0 at rate $n^{-4/(4+d)}$. Thus, its efficiency decreases rapidly with increasing dimension.
%\begin{flushright}   $\boxbox$ \end{flushright}

\begin{labwork}
The file {\tt nrl.txt} contains data from 42 rugby league matches.  The first column contains the length of game time, in seconds, until the first points are scored by a kick between the posts (penalty, drop goal or conversion); the second column contains the game time (in seconds) until the first try is scored. Denoting the log of the bivariate sample points by $(x_1,y_1),\ldots,(x_42,y_42)$, the product Gaussian kernel density estimator for the bivariate density is:
$$\hat{f}_\phi(x,y,h_x,h_y)=\frac{1}{42}\sum^{42}_{i=1}\phi(x,x_i,h^2_x)\phi(y,y_i,h_y^2).$$

Using Scott's rule:
$$h_x=\hat{s}_x\times 42^{-1/6} \textrm{  and   }h_y=\hat{s}_y\times 42^{-1/6}.$$
%\includegraphics
%\Matlab code:
\begin{VrbM}
data = load('nrl.txt');
x = log(data(:,1))';
y = log(data(:,2))';
n = length(x);

% scatter plot:
plot(x,y,'.r')
xlabel('x'), ylabel('y')
title('Scatter plot')
axis([3 9 3 9]), axis('square')
drawnow

n6 = n^(-1/6);
hx = min(std(x),iqr(x)/1.348) * n6;
hy = min(std(y),iqr(y)/1.348) * n6;
s = 3:.01:9;
ns = length(s);
phix = zeros(n,ns);
phiy = zeros(n,ns);
for i = 1:n
    phix(i,:) = normpdf(s,x(i),hx);
    phiy(i,:) = normpdf(s,y(i),hy);
end
fker = zeros(ns,ns);
for j = 1:ns
    for i = 1:ns
        fker(j,i) = phiy(:,j)' * phix(:,i) / n;
    end
end

% 3-D surface plot:
figure, mesh(s,s,fker)
xlabel('x'), ylabel('y'), zlabel('density')
colorbar, drawnow

% contour plot:
figure, contourf(s,s,fker,20)
xlabel('x'), ylabel('y')
title('Contour plot')
colorbar, axis('square')
\end{VrbM}
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

\section{Smoothed bootstrap}
The {\it smoothed bootstrap} is a variation of the bootstrap idea that is used for continuous data. Instead of estimating the distribution function by the EDF and obtaining bootstrap samples by random sampling with replacement from the data values, the smoothed bootstrap estimates the density by a kernel density and generates bootstrap samples from it. After getting the bootstrap samples, the MSE, variance, bias and confidence intervals can be computed for an estimator of interest, in the same way as in the nonparametric bootstrap. If the kernel density is a good estimate of the data density, then the smoothed bootstrap can give a small improvement over the nonparametric bootstrap for continuous data.

\subsection{Generating from a kernel density}
Let:$$\hat{f}_K(x,h)=\frac{1}{n}\sum^{n}_{i=1}K_h(x-x_i),$$
 be the kernel density.  To generate a new sample point from the kernel density, randomly pick one of the original data values, $x_1,\ldots,x_n$. Suppose   is picked, generate the new sample point from the kernel centred on $x_i$, i.e. from $K_h(x-x_i)$.
%\begin{flushright}   $\boxbox$ \end{flushright}

\begin{labwork}
Generate 1000 random values from the Gaussian kernel density, with Scott's bandwidth, for the geyser data.
%\Matlab code:
\begin{VrbM}
m = 1000; % number of sample points to generate
x = load('geyser.txt')';
n = length(x);
n5 = n^(-0.2);
h = min(std(x),iqr(x)/1.348) * n5; % Scott's bandwidth for Gaussian kernel
xcen = randsample(x,m,true); % randomly resample kernel centres from x
y = normrnd(xcen,h); % generate from Gaussian kernels centred at xcen
histogram(y,0,[0 inf],'r');
\end{VrbM}
%\includegraphics
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

\subsection{Smoothed bootstrap procedure for generating bootstrap samples}
Let $x_1,\ldots,x_n$ be IID random variables with unknown density $f(x)$.
\begin{asparaenum}[(a)]
\item Estimate the data density by a kernel density $\hat{f}_K(x,h)$.

\item	Obtain $N$ bootstrap samples, each of size $n$, by generating from $\hat{f}_K(x,h)$:
$$\begin{array}{c}
\{x_{1,1},\ldots,x_{1,n}\}\sim\hat{f}_K(x,h)\\
\textrm{M}\\
\{x_{N,1},\ldots,x_{N,n}\}\sim\hat{f}_K(x,h)\\
\end{array}$$
\end{asparaenum}
%\begin{flushright}   $\boxbox$ \end{flushright}

\begin{labwork}
Let us revisit Example 4.2.5 to obtain a 0.95 percentile interval for the median amount of sodium using the smoothed bootstrap with a Gaussian kernel density. Recall that the percentile interval provided by the nonparametric bootstrap was $(75.05, 77)$.

%\Matlab code:
\begin{VrbM}
x = load('sodium.txt'); % load data from text file and store in x
N = 100000; % number of bootstrap samples
n = length(x); % determine number of data values
nN = n * N;
alpha = 0.05;
alpha2 = alpha / 2;
alpha21 = 1 - alpha2;
n5 = n^(-0.2);
h = min(std(x),iqr(x)/1.348) * n5; % Scott's bandwidth for Gaussian kernel
xcen = randsample(x,nN,true); % randomly resample kernel centres from x
xboot = normrnd(xcen,h); % generate from Gaussian kernels centred at xcen
xboot = reshape(xboot,n,N); % organise resampled values into N columns of n
                            		% values each so that each column is a bootstrap
                            		% sample of size n
xbootmed = median(xboot); % medians of bootstrap samples
xbootmedsort = sort(xbootmed); % sort medians in increasing order

% (1-alpha) percentile interval:
[xbootmedsort(ceil(N*alpha2)) xbootmedsort(N*alpha21)]
\end{VrbM}
Results:
\begin{VrbM}
ans = 74.8616   76.9451
\end{VrbM}
Therefore, a 0.95 percentile interval for the median amount of sodium, using the smoothed bootstrap with a Gaussian kernel density, is $(74.86, 76.95)$.
%\begin{flushright}   $\boxbox$ \end{flushright}
\end{labwork}

Point-wise confidence band for density using the smoothed bootstrap (see Scott\footnote{Scott, D.W. (1992), {\it Multivariate Density Estimation: Theory, Practice and Visualization, Wiley.}} pp. 259-260; Hardle et al.\footnote{Hardle, W., Muller, M., Sperlich, S. and Werwatz, A. (2004), {\it Nonparametric and Semiparametric Models,} e-book at www.quantlet.com/mdstat/scripts/spm/html/spmhtmltoc.html}  section 3.5).

\section{Exercises}

\begin{exercise}
Consider the two-component normal mixture density given by:
$$f(x)=0.5\phi(x,4,1)+0.5\phi(x,9,4),$$

where $\phi(x,\mu,\sigma^2)$ denotes the normal density with mean $\mu$ and variance $\sigma$, evaluated at $x$.
\begin{asparaenum}[(a)]
\item A sample point, $x$, can be generated from the mixture density by the following algorithm:
\begin{tabbing}
Gene\=rate $u \sim U[0, 1]$.\\
If $u\leq 0.5$\\
	\>Generate $x \sim N(4, 1)$\\
Else\\
	\>Generate $x \sim N(9, 4)$.\\
\end{tabbing}

Implement a \Matlab function to generate from the mixture density and use it to get 100 sample points.

\item Using the 100 sample points from part (a), estimate the underlying density using a histogram with appropriate bin width. Explain clearly how you arrived at your choice of bin width.

\item Using the 100 sample points from part (a), estimate the underlying density using a Gaussian kernel density estimator. Explore bandwidths ranging from 0.3 to 1.9 to decide on an appropriate one.
\end{asparaenum}
\end{exercise}

\begin{exercise}
The Bart Simpson density is given by:
$$f(x)=0.5\phi(x,0,1)+0.1\sum^4_{j=0}\phi(x,0.5j-1,0.01).$$

To see why it is called the Bart Simpson density, plot the density for a sequence of x values from $-3$ to 3 (in steps of 0.01, for example).
\begin{asparaenum}[(a)]
\item Write a \Matlab function to generate sample points from the Bart Simpson density and use it to obtain 1000 sample points.
\item Using the 1000 sample points from part (a), estimate the underlying density using a histogram with appropriate bin width. Explain clearly how you arrived at your choice of bin width.
\item Using the 1000 sample points from part (a), estimate the underlying density using a Gaussian kernel density estimator with bandwidths of 0.005, 0.05 and 0.5. Comment on the performance of each of these bandwidths.
\end{asparaenum}
\end{exercise}

\begin{exercise}
The data in {\tt ceo.txt} contain the ages (column 1) and salaries (column 2, in thousands of dollars) of the CEOs of 59 companies.
\begin{asparaenum}[(a)]
\item Obtain a Gaussian kernel density estimate for the salaries, using Scott's bandwidth. Compare the transformation method and boundary correction method for handling the boundary at zero.

\item Use the nonparametric bootstrap to obtain a point-wise 0.95 BCA confidence band for the density of the salaries.

\end{asparaenum}
\end{exercise}

\begin{exercise}
The second column of {\tt glass.txt} contains measurements of the refractive index of 214 glass specimens collected in forensic work.
\begin{asparaenum}[(a)]
\item Obtain a Gaussian kernel density estimate for refractive index, using Scott's bandwidth.

\item Use the nonparametric bootstrap to obtain a point-wise 0.95 BCA confidence band for the density of refractive index.
\end{asparaenum}

\end{exercise}

\begin{exercise}
The data in {\tt whale.txt} are the times of 121 bowhead whale calf sightings during the 2001 spring migration. The time of each sighting is expressed as the number of hours since midnight of April 5, when the first adult whale was sighted.
\begin{asparaenum}[(a)]
\item Plot a density histogram for the data and superimpose onto it a Gaussian kernel density estimate. Explain your choice of histogram bin width and kernel density bandwidth. For the kernel density estimate, compare the transformation method and the boundary correction method for handling the boundary at zero.

\item Obtain a 0.95 BCA interval for the median sighting time, using the nonparametric bootstrap with 10,000 bootstrap samples.

\item Repeat Part (b) using the smoothed bootstrap with 10,000 bootstrap samples.
\end{asparaenum}

\end{exercise}

\begin{exercise}
The data in {\tt density.txt} are 29 measurements of the density of the earth, which were made by Henry Cavendish in 1789.
\begin{asparaenum}[(a)]
\item Plot a density histogram for the data and superimpose onto it a Gaussian kernel density estimate. Explain your choice of histogram bin width and kernel density bandwidth. For the kernel density estimate, compare the transformation method and the boundary correction method for handling the boundary at zero.
\item Obtain a 0.95 BCA interval for the mean density of the earth, using the nonparametric bootstrap with 10,000 bootstrap samples.
\item Repeat Part (b) using the smoothed bootstrap with 10000 bootstrap samples.
\end{asparaenum}

\end{exercise}

\begin{exercise}
Work through Example 5.3.3 for the NRL data.
\end{exercise}

\begin{exercise}
The data in {\tt infrared.txt} are measurements of infrared emissions from 628 objects beyond our galaxy. The two columns contain total flux measurements for two different wavelength bands - the first column shows the 12 micrometre band and the second column shows the 100 micrometre band. Estimate the bivariate density for the log of the data using a product Gaussian kernel density estimator with Scott's bandwidth. Plot the estimated density using a three-dimensional surface plot and a two-dimensional contour plot.
\end{exercise}


% ***********************************************************************************
% Pure LaTeX part to be inserted in a document (be careful of depencies of packages & commands
% Prepared by Qingan Zhao under the supervision of Arnaud de La Fortelle
% Fall 2017
% Example of solving the 2D heat diffusion subsection of the modeling part part
% ***********************************************************************************

\subgroup{2}{Qingan Zhao}

\paragraph{Description}
Now consider a retangular plane (length: a; width: b) with 2 boundaries that have no heat transfer (i.e., $T(x, 0)=f(x);\ T(x, b)=g(x)$). The other two boundaries have fixed temperature ($T(0, y)=T(a, y)=0$). We would like to solve the PDE under the stationary state (i.e., $\ud T/\ud t=0$). The diagram of the system is shown in Figure~\ref{heatSolSystem.fig}.
\begin{figure}[htb]
	\centering
	\includegraphics[width=10cm]{2dHeatSolDescription.pdf}
	\caption{System description of the example}\label{heatSolSystem.fig}
\end{figure}

\paragraph{PDE and boundary conditions}
From Section 2.2.2 we have already known that the 2D heat equation is:
\begin{equation}
\frac{\partial T}{\partial t}=a^2\left(\frac{\partial ^2 T}{\partial x^2}+\frac{\partial ^2 T}{\partial y^2}\right)\ \ \ \ where\ a=\sqrt{\frac{k}{c\rho}}
\end{equation}

The boundary conditions of this problem are as follows:

\begin{equation}\label{2DHeatBound1.eq}
T(x,0,t)=f(x)
\end{equation}
\begin{equation}\label{2DHeatBound2.eq}
T(x,b,t)=g(x)
\end{equation}
\begin{equation}\label{2DHeatBound3.eq}
T(0, y, t)=0
\end{equation}
\begin{equation}\label{2DHeatBound4.eq}
T(a, y, t)=0
\end{equation}
\begin{equation}
\frac{\ud T}{\ud t}=0
\end{equation}

\paragraph{Solution}
Since the system is under the stationary state ($\ud T/\ud t=0$), $T(x,y,t)$ can be expressed as $T(x,y)$, and the PDE can be written as:
\begin{equation}
\frac{\partial^2T}{\partial x^2}+\frac{\partial^2T}{\partial y^2}=0
\end{equation}

We could separate the variables by express $T(x,y)$ as $X(x)Y(y)$. Then the PDE can be written as:
\begin{equation}
X''(x)Y(y)+X(x)Y''(y)=0
\end{equation}

Since $X(x)Y(y)$ is not identically zero, divide both sides by $X(x)Y(y)$ gives:
\begin{equation}
\frac{X"(x)}{X(x)}+\frac{Y"(y)}{Y(y)}=0
\end{equation}

Since the two parts of the left side are independent, the following equations can be drawn ($\lambda$ is a constant):
\begin{align}
X"(x)+\lambda X(x)=0\\
Y"(y)-\lambda Y(y)=0
\end{align}

Moreover, the boundary conditions Equations~(\ref{2DHeatBound3.eq})-(\ref{2DHeatBound4.eq}) give:
\begin{align}
X(0)Y(y)=0\\
X(a)Y(y)=0
\end{align}

Since $Y(y)$ is not identically zero, we can obtain the following:
\begin{align}
X(0)=0\\
X(a)=0
\end{align}

Hence, the problem becomes solving two indenpendent ordinary differential equations. We know that only when $\lambda=\frac{n^2\pi^2}{a^2}\ (n=1,2,3,...)$, the problem has the following solution:
\begin{equation}
X_n(x)=sin\frac{n\pi x}{a}
\end{equation}
\begin{equation}
Y_n(y)=a_ne^{\frac{n\pi}{a}y}+b_ne^{-\frac{n\pi}{a}y}
\end{equation}

Hence, the solution can be expressed as:
\begin{equation}
T(x,y)=\sum_{n=1}^{\infty}(a_ne^{\frac{n\pi}{a}y}+b_ne^{-\frac{n\pi}{a}y})sin\frac{n\pi x}{a}
\end{equation}

Given the other two boundary conditions Equations~(\ref{2DHeatBound1.eq})-(\ref{2DHeatBound2.eq}), $a_n$ and $b_n$ can be solved using the following 2 equations:
\begin{equation}
a_n+b_n=\int_{0}^{a}f(x)sin\frac{n\pi}{a}xdx
\end{equation}
\begin{equation}
a_ne^{\frac{n\pi}{a}b}+b_ne^{-\frac{n\pi}{a}b}=\int_{0}^{a}g(x)sin\frac{n\pi}{a}xdx
\end{equation}

Thus, the analytical solution of this problem is obtained.


\title{KL(q||p) Minimization}

\subsection{$\text{KL}(q\|p)$ Minimization}

One form of variational inference minimizes the Kullback-Leibler
divergence \textbf{from} $q(\mathbf{z}\;;\;\lambda)$ \textbf{to}
$p(\mathbf{z} \mid \mathbf{x})$,
\begin{align*}
  \lambda^*
  &=
  \arg\min_\lambda \text{KL}(
  q(\mathbf{z}\;;\;\lambda)
  \;\|\;
  p(\mathbf{z} \mid \mathbf{x})
  )\\
  &=
  \arg\min_\lambda\;
  \mathbb{E}_{q(\mathbf{z}\;;\;\lambda)}
  \big[
  \log q(\mathbf{z}\;;\;\lambda)
  -
  \log p(\mathbf{z} \mid \mathbf{x})
  \big].
\end{align*}
The KL divergence is a non-symmetric, information theoretic measure of
similarity between two probability distributions
\citep{hinton1993keeping,waterhouse1996bayesian,jordan1999introduction}.

\subsubsection{The Evidence Lower Bound}

The above optimization problem is intractable because it directly
depends on the posterior $p(\mathbf{z} \mid \mathbf{x})$. To tackle
this, consider the property
\begin{align*}
  \log p(\mathbf{x})
  &=
  \text{KL}(
  q(\mathbf{z}\;;\;\lambda)
  \;\|\;
  p(\mathbf{z} \mid \mathbf{x})
  )\\
  &\quad+\;
  \mathbb{E}_{q(\mathbf{z}\;;\;\lambda)}
  \big[
  \log p(\mathbf{x}, \mathbf{z})
  -
  \log q(\mathbf{z}\;;\;\lambda)
  \big]
\end{align*}
where the left hand side is the logarithm of the marginal likelihood
$p(\mathbf{x}) = \int p(\mathbf{x}, \mathbf{z}) \text{d}\mathbf{z}$,
also known as the model evidence. (Try deriving this using Bayes'
rule!)

The evidence is a constant with respect to the variational parameters
$\lambda$, so we can minimize $\text{KL}(q\|p)$ by instead maximizing
the Evidence Lower BOund,
\begin{align*}
  \text{ELBO}(\lambda)
  &=\;
  \mathbb{E}_{q(\mathbf{z}\;;\;\lambda)}
  \big[
  \log p(\mathbf{x}, \mathbf{z})
  -
  \log q(\mathbf{z}\;;\;\lambda)
  \big].
\end{align*}
In the ELBO, both $p(\mathbf{x}, \mathbf{z})$ and
$q(\mathbf{z}\;;\;\lambda)$ are tractable. The optimization problem we
seek to solve becomes
\begin{align*}
  \lambda^*
  &=
  \arg \max_\lambda \text{ELBO}(\lambda).
\end{align*}
As per its name, the ELBO is a lower bound on the evidence, and
optimizing it tries to maximize the probability of observing the data.
What does maximizing the ELBO do? Splitting the ELBO reveals a trade-off
\begin{align*}
  \text{ELBO}(\lambda)
  &=\;
  \mathbb{E}_{q(\mathbf{z} \;;\; \lambda)}[\log p(\mathbf{x}, \mathbf{z})]
  - \mathbb{E}_{q(\mathbf{z} \;;\; \lambda)}[\log q(\mathbf{z}\;;\;\lambda)],
\end{align*}
where the first term represents an energy and the second term
(including the minus sign) represents the entropy of $q$.
The energy encourages $q$ to focus probability mass where the
model puts high probability, $p(\mathbf{x}, \mathbf{z})$.
The entropy encourages $q$ to spread probability mass to avoid
concentrating to one location.

Edward uses two generic strategies to obtain gradients for
optimization.

\begin{itemize}
  \item Score function gradient;
  \item Reparameterization gradient.
\end{itemize}

\subsection{Score function gradient}

Gradient descent is a standard approach for optimizing complicated
objectives like the ELBO. The idea is to calculate its gradient
\begin{align*}
  \nabla_\lambda\;
  \text{ELBO}(\lambda)
  &=
  \nabla_\lambda\;
  \mathbb{E}_{q(\mathbf{z}\;;\;\lambda)}
  \big[
  \log p(\mathbf{x}, \mathbf{z})
  -
  \log q(\mathbf{z}\;;\;\lambda)
  \big],
\end{align*}
and update the current set of parameters proportional to the gradient.

The score function gradient estimator leverages a property of
logarithms to write the gradient as
\begin{align*}
  \nabla_\lambda\;
  \text{ELBO}(\lambda)
  &=\;
  \mathbb{E}_{q(\mathbf{z}\;;\;\lambda)}
  \big[
  \nabla_\lambda \log q(\mathbf{z}\;;\;\lambda)
  \:
  \big(
  \log p(\mathbf{x}, \mathbf{z})
  -
  \log q(\mathbf{z}\;;\;\lambda)
  \big)
  \big].
\end{align*}
The gradient of the ELBO is an expectation over the variational
model $q(\mathbf{z}\;;\;\lambda)$; the only new ingredient it requires is the
\emph{score function} $\nabla_\lambda \log q(\mathbf{z}\;;\;\lambda)$
\citep{paisley2012variational,ranganath2014black}.

We can use Monte Carlo integration to obtain noisy estimates of both the ELBO
and its gradient. The basic procedure follows these steps:
\begin{enumerate}
  \item draw $S$ samples $\{\mathbf{z}_s\}_1^S \sim q(\mathbf{z}\;;\;\lambda)$,
  \item evaluate the argument of the expectation using $\{\mathbf{z}_s\}_1^S$, and
  \item compute the empirical mean of the evaluated quantities.
\end{enumerate}

A Monte Carlo estimate of the gradient is then
\begin{align*}
  \nabla_\lambda\;
  \text{ELBO}(\lambda)
  &\approx\;
  \frac{1}{S}
  \sum_{s=1}^{S}
  \big[
  \big(
  \log p(\mathbf{x}, \mathbf{z}_s)
  -
  \log q(\mathbf{z}_s\;;\;\lambda)
  \big)
  \:
  \nabla_\lambda \log q(\mathbf{z}_s\;;\;\lambda)
  \big].
\end{align*}
This is an unbiased estimate of the actual gradient of the ELBO.

\subsection{Reparameterization gradient}

If the model has differentiable latent variables, then it is generally
advantageous to leverage gradient information from the model in order to
better traverse the optimization space. One approach to doing this is
the reparameterization gradient
\citep{kingma2014auto,rezende2014stochastic}.

Some variational distributions $q(\mathbf{z}\;;\;\lambda)$ admit useful
reparameterizations. For example, we can reparameterize a normal distribution
$\mathbf{z} \sim \text{Normal}(\mu, \Sigma)$ as
$\mathbf{z} \sim \mu + L \text{Normal}(0, I)$ where $\Sigma = LL^\top$. In general, write
this as
\begin{align*}
  \epsilon &\sim q(\epsilon)\\
  \mathbf{z} &= \mathbf{z}(\epsilon \;;\; \lambda),
\end{align*}
where $\epsilon$ is a random variable that does \textbf{not} depend on the
variational parameters $\lambda$. The deterministic function
$\mathbf{z}(\cdot;\lambda)$ encapsulates the variational parameters instead,
and following the process is equivalent to directly drawing $\mathbf{z}$ from
the original distribution.

The reparameterization gradient leverages this property of the
variational distribution to write the gradient as
\begin{align*}
  \nabla_\lambda\;
  \text{ELBO}(\lambda)
  &=\;
  \mathbb{E}_{q(\epsilon)}
  \big[
  \nabla_\lambda
  \big(
  \log p(\mathbf{x}, \mathbf{z}(\epsilon \;;\; \lambda))
  -
  \log q(\mathbf{z}(\epsilon \;;\; \lambda) \;;\;\lambda)
  \big)
  \big].
\end{align*}
The gradient of the ELBO is an expectation over the base
distribution $q(\epsilon)$, and the gradient can be applied directly
to the inner expression.

We can use Monte Carlo integration to obtain noisy estimates of both the ELBO
and its gradient. The basic procedure follows these steps:
\begin{enumerate}
  \item draw $S$ samples $\{\epsilon_s\}_1^S \sim q(\epsilon)$,
  \item evaluate the argument of the expectation using $\{\epsilon_s\}_1^S$, and
  \item compute the empirical mean of the evaluated quantities.
\end{enumerate}

A Monte Carlo estimate of the gradient is then
\begin{align*}
  \nabla_\lambda\;
  \text{ELBO}(\lambda)
  &\approx\;
  \frac{1}{S}
  \sum_{s=1}^{S}
  \big[
  \nabla_\lambda
  \big(
  \log p(\mathbf{x}, \mathbf{z}(\epsilon_s \;;\; \lambda))
  -
  \log q(\mathbf{z}(\epsilon_s \;;\; \lambda) \;;\;\lambda)
  \big)
  \big].
\end{align*}
This is an unbiased estimate of the actual gradient of the ELBO. Empirically, it
exhibits lower variance than the
score function gradient, leading to
faster convergence in a large set of problems.

For more details, see the \href{/api/}{API} as well as its
implementation in Edward's code base.

\subsubsection{References}\label{references}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{color,amsmath}

\title{Microphone Calibration}
\author{Dean Richert}

\begin{document}

\maketitle

\section{Acoustic sensing basics}

\subsection{Octave bands:} Let $n$ be the band number. Then the center of octave band $n$ (in Hz) is,
\begin{align*}
    f_c(N) = 2^{n-5} \times 10^3.
\end{align*}
The band limits, in Hz, are given by
\begin{align*}
    f_{\text{upper}}(n) &= \sqrt{2} f_c(n)
    \\
    f_{\text{lower}}(n) &= \frac{1}{\sqrt{2}} f_c(n)
\end{align*}

\subsection{Equivalent sound pressure level:} This is a measurement of average sound pressure. Let $p(t)$ be the instantaneous sound pressure at time $t$ and let $T$ be the time duration of a measurement. Then,
\begin{align*}
    L_{eq}(T) = 10 \log_{10}\bigg[\frac{1}{T} \int_0^T\bigg(\frac{p(t)}{p_0}\bigg)^2 dt\bigg]
\end{align*}
where $p_0$ is the reference sound pressure (usually $20\mu$Pa). For discrete time measurements,
\begin{align*}
    L_{eq}(N) = 10 \log_{10}\bigg[\frac{1}{N} \sum_{k=0}^{N-1}\bigg(\frac{p(k)}{p_0}\bigg)^2 \bigg]
\end{align*}
where $N$ is the number of measurements.
\\
We can also compute the equivalent sound pressure level in an octave band using a band pass filter, $H_n$, which pass frequencies in octave band $n$. Then the equivalent sound pressure level in octave band $n$ is
\begin{align*}
    L_{eq}^n(N) = 10 \log_{10}\bigg[\frac{1}{N} \sum_{t=0}^{N-1}\bigg(\frac{H_n[p(t)]}{p_0}\bigg)^2 \bigg]
\end{align*}
Here we assume that $H_n$ has unit gain ($0$dB) at $f_c(n)$. I use $6^{th}$ order Butterworth filters for $H_n$.

To get the total sound pressure level across a set $\mathcal{N}$ of octave bands, the following calculation is performed:
\begin{align*}
    L_{eq}(N) = 10 \log_{10}\bigg( \sum_{n \in \mathcal{N}} 10^{\frac{L_{eq}^n(N)}{10}} \bigg)
\end{align*}

On the other hand, the average $L_{eq}$ of a set of sound pressure measurements taken over distinct time intervals, $\{L_{eq,i}\}_{i = 0}^{M-1}$, is given by
\begin{align*}
    L_{eq} = 10 \log_{10}\bigg( \frac{1}{M} \sum_{i = 0}^{M-1} 10^{\frac{L_{eq,i}}{10}} \bigg)
\end{align*}

\subsection{A-weighted $L_{eq}$:} The human ear is less sensitive to low frequencies; the perceived loudness of a low-frequency signal is less than the perceived loudness of a high-frequency signal of equal $L_{eq}$. The A-weighting of a sound pressure level accounts for this by weighting high-frequency components in a signal more than the low-frequency components. The A-weighted $L_{eq}$ in octave band $n$ is defined as
\begin{align*}
    LA_{eq}^n(N) = L_{eq}^n(N) + A(f_c(n))
\end{align*}
where
\begin{align*}
    A(f) &= 20 \log_{10} R_A(f) + 2 \\
    R_A(f) &= \frac{12194^2 f^4}{(f^2 + 20.6^2)\sqrt{(f^2+107.7^2)(f^2+737.9^2)}(f^2+12194^2)}
\end{align*}
The units of $LA_{eq}$ are dBA. The total A-weighted sound pressure level for a signal can be computed as
\begin{align*}
    LA_{eq}(N) = 10 \log_{10}\bigg( \sum_{n \in \mathcal{N}} 10^{\frac{LA_{eq}^n(N)}{10}} \bigg)
\end{align*}

\section{Calibration}

For a sound pressure signal $p(k)$ in octave band $n$, we assume that the microphone measurement is given by
\begin{align*}
    y(k) = K(n) p(k) + y_{DC}
\end{align*}
Note that there is a frequency dependence on the sensor gain $K(n)$ which is consistent with the data sheet.

\subsection{DC bias:} To compute the DC bias we take advantage of the zero-mean property of $p(k)$. Taking the average over many measurements,
\begin{align*}
    \frac{1}{N} \sum_{k=0}^{N-1} y(k) &= \frac{K(n)}{N} \sum_{k=0}^{N-1} p(k) + y_{DC}
    \\
    y_{DC} &= \frac{1}{N} \sum_{k=0}^{N-1} y(k)
\end{align*}

\subsection{Sensor gain:} Suppose that $p(k)$ is a calibration tone at frequency $f_c(n)$ at $x$ dB. Then,
\begin{align*}
    x &= 10 \log_{10} \bigg[ \frac{1}{N} \sum_{k=0}^{N-1}\bigg(\frac{H_n(p(k))}{p_0} \bigg)^2 \bigg]
    \\
    x &= 10 \log_{10} \bigg[ \frac{1}{N} \sum_{k=0}^{N-1}\bigg(\frac{H_n((y(k)-y_{DC})/(K(n))}{p_0} \bigg)^2 \bigg]
    \\
    x &= 10 \log_{10} \bigg[ \frac{1}{(K(n)p_0)^2} \frac{1}{N} \sum_{k=0}^{N-1}\bigg(H_n(y(k)-y_{DC}) \bigg)^2 \bigg]
    \\
    K_{cal}(n) :=  \frac{1}{(K(n)p_0)^2} &= \frac{10^{x/10}}{\frac{1}{N} \sum_{k=0}^{N-1}\big(H_n(y(k)-y_{DC})\big)^2}
\end{align*}
The calibration routine is then given by:
\begin{enumerate}
    \item Generate a sound signal at $f_c(n)$ at known $x$ dB,
    \item For each new measurement:
        \begin{itemize}
            \item Subtract the computed $y_{DC}$ from the raw measurement
            \item Update the filter states with input $y(k) - y_{DC}$
            \item Square the filter output and add it to the previous squared output
        \end{itemize}
    \item After taking sufficient measurements, compute $K_{cal}(n)$
\end{enumerate}
Note that we do not solve for $K(n)$ explicitly since $K_{cal}(n)$ is sufficient to compute the equivalent sound pressure level:
\begin{align*}
    L_{eq}^n(N) &= 10 \log_{10} \bigg[ \frac{K_{cal}(n)}{N} \sum_{k=0}^{N-1}\big(H_n(y(k) - y_{DC})\big)^2 \bigg]
\end{align*}
Note that if $H_n$ does not have unit gain at $f_c(n)$ this is absorbed into $K_{cal}(n)$. Thus, we do not actually design the band pass filters to have unit gain at the octave frequency and rather compensate for this through calibration.

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{Weierstrass normal form}
\author{Dave Neary}
\date{May 2021}

\begin{document}

\maketitle

\section{Weierstrass normal form for cubic equations}

Given a general cubic equation
\[ ax^3 +bx^2y +cxy^2 + dy^3 +ex^2 + fxy +gy^2 + hx + iy +j = 0 \]
we can perform a set of substitutions to convert the equation to a form called Weierstrass normal form:
\[ Y^2 = X^3 + AX^2 + BX + C \]
or, equivalently:
\[ Y^2 = X^3 + \alpha X + \beta \]

In particular, for the cubic equation $C$:
\[u^3+v^3=\alpha \]
we can use the substitutions
\[x = \frac{12\alpha}{u+v}, y = 36\alpha\frac{u-v}{u+v} \]
to convert this equation to the form $C'$:
\[ y^2 = x^3 - 432 \alpha \]

This process can be inverted, so from any point $(x,y) \in C'$ we can generate a point $(u,v) \in C$ by the transform:
\[u = \frac{36\alpha +y}{6x}, v = \frac{36\alpha - y}{6x} \]

By this means, if we find rational points on either curve, we can generate a rational point on the other by this bijection.

This transformation might seem like magic, but we can work through this process if we have a rational point on the homogeneous projective form of original curve. To get to this form, we replace $u = \frac{U}{W}, v = \frac{V}{W}$ and multiply across by $W^3$ to give:
\[ U^3 + V^3 - \alpha W^3 = 0 \]
The point in $\mathbb{P}^2$ $P = [-1; 1; 0]$ is on the curve $C$. Since $W=0$ this corresponds to a point at infinity - a tangent point of the curve. The tangent at the point $P$ is given by the line:
\[ U \frac{\partial C}{\partial U} (P) + V \frac{\partial C}{\partial V} (P) + W \frac{\partial C}{\partial W} (P) = 0 \]

\[ \frac{\partial C}{\partial V} = 3V^2, \frac{\partial C}{\partial U} = 3U^2, \frac{\partial C}{\partial W} = 3\alpha W^2 \]

So at the point $P = [-1;1;0]$ the tangent line in $\mathbb{P}^2$ is $3U + 3V = 0$ (and we can divide out the common factor of 3).

Examining the curve $u^3+v^3=\alpha$ we can see that this does not intersect at all in the
real numbers, and has a triple root at the point at infinity. We will use this line as our
$Z=0$ axis after our transformation.

We will take $U-V=0$ as our $X$ axis, motivated by the fact that this is a line of symmetry
of our curve, and finally we will take $U+V-W = 0$ as our $Y$ axis, motivated by the fact
that it intersects $Z=0$ at $P$, and is helpfully orthogonal to the $X$ axis, so we should
avoid any pesky $XY$ terms after transformation.

Putting this together, our transformation from $(U,V,W)$ space to $(X,Y,Z)$ space will be:
\[
\begin{pmatrix} 1 & -1 & 0 \\ 1 & 1 & -1 \\ 1 & 1 & 0 \end{pmatrix}
\begin{pmatrix} U \\ V \\ W \end{pmatrix} =
\begin{pmatrix} X \\ Y \\ Z \end{pmatrix} \]

This matrix is, by design, invertible. Its inverse is:

\[ A^{-1} = \frac{1}{2} \begin{pmatrix} 1 & 0 & 1 \\ -1 & 0 & 1 \\ 0 & -2 & 2 \end{pmatrix} \]
This gives us an affine transformation which allows us to go from $(X,Y,Z)$ space to $(U,V,W)$ space as follows:
\begin{equation*}
\begin{split}
    U &= \frac{1}{2}(X + Z) \\
    V &= \frac{1}{2}(-X + Z) \\
    W  &= -Y + Z
\end{split}
\end{equation*}

By substituting these equations back into our homogeneous version of $C$, we get:
\[ \left(\frac{1}{2}(X + Z)\right)^3 + \left(\frac{1}{2}(-X + Z)\right)^3
- \alpha\left(-Y + Z\right)^3 = 0 \]

Expanding and cancelling terms, we simplify our original equation to:
\begin{equation*}
\begin{split}
     \frac{1}{8}(6X^2Z + 2Z^3) - \alpha (-Y^3 + 3Y^2Z - 3YZ^2 + Z^3) &= 0 \\
     3X^2Z + Z^3 + 4 \alpha Y^3 - 12 \alpha Y^2Z + 12\alpha YZ^2 - 4\alpha Z^3 &= 0
\end{split}
\end{equation*}

Replacing $x=\frac{X}{Z}, y=\frac{Y}{Z}$ to dehomogenize, we get:
\[  3x^2 + 1 + 4 \alpha y^3 - 12 \alpha y^2 + 12\alpha y - 4\alpha = 0 \]
\[ 3x^2 = -4\alpha y^3 + 12\alpha y^2 -12 \alpha y +4 \alpha -1 \]

We want a perfect square term on the left, and a leading perfect cube on the right. We can achieve
this by multiplying both sides by $3^3\cdot4^2\cdot \alpha^2 = 432\alpha^2$:

\[ 3^4 4^2 \alpha^2 x^2 = -3^3 4^3 \alpha^3 y^3 + 3^3 4^3 \alpha^3 (3y^2) - 3^3 4^3 \alpha^3(3 y) +3^3 4^3 \alpha^3 - 432\alpha^2 \]

Substituting $a=36\alpha x, b=-12\alpha(y-1)$ we get:
\[ a^2 = b^3 - 432\alpha^2 \]

The transforms from $(a,b)$ to $(u,v)$ are now straightforward to derive:
\begin{equation*}
\begin{split}
         a & = 36\alpha x \\
         & = 36 \alpha \frac{X}{Z} \\
         & = 36 \alpha \frac{U-V}{U+V} \\
         & = 36 \alpha \frac{u-v}{u+v} \\
         b & = -12 \alpha (y-1) \\
         & = -12 \alpha \left(\frac{Y}{Z} - 1 \right) \\
         & = -12 \alpha \left(\frac{U+V-W}{U+V} - 1 \right) \\
         & = -12 \alpha \left(\frac{-W}{U+V}\right) \\
         & = \frac{12 \alpha}{u+v} \\
\end{split}
\end{equation*}

where we replace $u=\frac{U}{W}, v=\frac{V}{W}$ to dehomogenize.

The transformation from $(u,v)$ to $(a,b)$ is similar:
\begin{equation*}
\begin{split}
         u & = \frac{U}{W} \\
         & = \frac{\frac{1}{2}(X + Z)}{-Y + Z} \\
         & = \frac{\frac{X}{Z} + 1}{-2\frac{Y}{Z} + 2} \\
         & = \frac{(x + 1)}{-2(y - 1)} \\
         & = \frac{(\frac{a}{36\alpha} + 1)}{-2\left(\frac{-b}{12\alpha} \right)} \\
         & = \frac{a + 36\alpha}{6b} \\
         v & =  \frac{V}{W} \\
         & = \frac{\frac{1}{2}(-X + Z)}{-Y + Z} \\
         & = \frac{-X + Z}{-2Y + 2Z} \\
         & = \frac{-x + 1}{-2(y -1)} \\
         & = \frac{-\frac{a}{36\alpha} + 1}{-2(\frac{-b}{12\alpha})} \\
         & = \frac{-a + 36\alpha}{6b}
\end{split}
\end{equation*}

\end{document}


\section{Spherical Coordinates}
\noindent
Unlike how cylindrical coordinates extend polar coordinates into 3D by adding a Cartesian term, spherical coordinates add an angular term, $\phi$, the azimuthal angle. All spherical coordinates have the form $(\rho, \theta, \phi)$ where $\rho$ is the distance from the origin, $\theta$ is the polar angle in the xy-plane, and $\phi$ is the azimuthal angle from the $+z$-axis. Shapes with spherical symmetry have a constant bounds of integration in spherical coordinates.

[INSERT IMAGE]

\input{./curvilinearCoordinates/conversions_spherical}
\input{./curvilinearCoordinates/integration_spherical}


\subsection{Times Table} % (fold)
\label{sub:times_table}

This program prints out the times table for a number entered by the user, displaying from 1 x n to 10 x n. The description of the program is in Table \ref{tbl:data-times-table}, the pseudocode in Listing \ref{lst:data-times-pseudo}, the C code in Listing \ref{lst:data-times-c}, and the Pascal code in Listing \ref{lst:data-times-pas}.

\begin{table}[h]
\centering
\begin{tabular}{l|p{10cm}}
  \hline
  \multicolumn{2}{c}{\textbf{Program Description}} \\
  \hline
  \textbf{Name} & \emph{Times Table} \\
  \\
  \textbf{Description} & Displays the Times Table from 1 x n to 10 x n. \\
  \hline
\end{tabular}
\caption{Description of the Times Table program}
\label{tbl:data-times-table}
\end{table}

\pseudocode{lst:data-times-pseudo}{Pseudocode for Times Table program.}{./topics/storing-using-data/examples/times-table.txt}

\mynote{
This is an updated version of the Seven Times Table Program. See Section \ref{sub:seven_times_table} \nameref{sub:seven_times_table}.
}

\clearpage

\csection{\ccode{lst:data-times-c}{C Times Table}{topics/storing-using-data/examples/times_table.c}}

\passection{\pascode{lst:data-times-pas}{Pascal Times Table}{topics/storing-using-data/examples/TimesTable.pas}}

% subsection times_table (end)

\clearpage
\subsection{Circle Area} % (fold)
\label{sub:circle_area_data}

This program prints out the area of a circle. The description of the program is in Table \ref{tbl:data-circle-area}, the pseudocode in Listing \ref{lst:data-circle-areas-pseudo}, the C code in Listing \ref{lst:data-circle-areas-c}, and the Pascal code in Listing \ref{lst:data-circle-areas-pas}.

\begin{table}[h]
\centering
\begin{tabular}{l|p{10cm}}
  \hline
  \multicolumn{2}{c}{\textbf{Program Description}} \\
  \hline
  \textbf{Name} & \emph{Circle Areas} \\
  \\
  \textbf{Description} & Displays the Circle Areas for circles with radius from 1.0 to 5.0 with increments of 0.5. \\
  \hline
\end{tabular}
\caption{Description of the Circle Areas program}
\label{tbl:data-circle-area}
\end{table}

\pseudocode{lst:data-circle-areas-pseudo}{Pseudocode for Circle Areas program.}{./topics/storing-using-data/examples/circle_areas.txt}

\mynote{
This is an updated version of the Circle Areas Program. See Section \ref{sub:circle_area} \nameref{sub:circle_area}.
}

\clearpage

\csection{\ccode{lst:data-circle-areas-c}{C Circle Areas}{topics/storing-using-data/examples/circle_areas.c}}

\passection{\pascode{lst:data-circle-areas-pas}{Pascal Circle Areas}{topics/storing-using-data/examples/CircleAreas.pas}}

% subsection circle_area (end)
\clearpage
\subsubsection{Water Tank} % (fold)
\label{ssub:water_tank}

The \emph{Water Tank} program draws four water tanks to the terminal. Each water tank is drawn as a cylinder that fills a given area on the screen, and shows its current water level. An example execution is shown in \fref{fig:water-tank-img}.

\begin{table}[h]
\centering
\begin{tabular}{l|p{10cm}}
  \hline
  \multicolumn{2}{c}{\textbf{Program Description}} \\
  \hline
  \textbf{Name} & \emph{Water Tank} \\
  \\
  \textbf{Description} & Displays calculates and displays `Water Tanks'. Each tank has a position on the screen, a width, height, and a percent full.\\
  \hline
\end{tabular}
\caption{Description of the Water Tanks program}
\label{tbl:data-water-tanks}
\end{table}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.7\textwidth]{./topics/storing-using-data/examples/WaterTank.png}
   \caption{Example execution of the Water Tank program}
   \label{fig:water-tank-img}
\end{figure}

\clearpage

\csection{\ccode{clst:water-tank}{C Water Tank drawing code}{topics/storing-using-data/examples/water-tank.c}}

\csection{\ccode{clst:water-tank1}{C Water Tank drawing code (continued from \lref{clst:water-tank})}{topics/storing-using-data/examples/water-tank1.c}}

\passection{\pascode{lst:data-water-tank-pas}{Pascal Water Tank}{topics/storing-using-data/examples/WaterTank.pas}}

\passection{\pascode{lst:data-water-tank-1-pas}{Pascal Water Tank (continued from \lref{lst:data-water-tank-pas})}{topics/storing-using-data/examples/WaterTank1.pas}}

% subsubsection water_tank (end)

\clearpage
\subsection{Bicycle Race} % (fold)
\label{sub:bicycle_race}

The Bicycle Race program will simulate a thirty second sprint race between a number of bicycles. The race has a standing start, and then each racer accelerates as fast as they can for thirty seconds. The winner is the racer who makes it the furthest.

\begin{table}[h]
\centering
\begin{tabular}{l|p{10cm}}
  \hline
  \multicolumn{2}{c}{\textbf{Program Description}} \\
  \hline
  \textbf{Name} & \emph{Bike Race} \\
  \\
  \textbf{Description} & Calculates the position of seven bikes at the end of a timed race, drawing the final positions. Each bike's position is calculated based on a random acceleration over the duration of the race. \\
  \hline
\end{tabular}
\caption{Description of the Bike Race program}
\label{tbl:data-bike-race}
\end{table}

\begin{itemize}
  \item You can calculate distance the racers cover using Equation~\ref{eq:acceleration}.
  \begin{itemize}
    \item \emph{s} is the distance covered.
    \item \emph{u} is the starting speed
    \item \emph{t} is time.
    \item \emph{a} is acceleration.
  \end{itemize}
  \begin{equation}
    s = ut + \frac{a t^2}{2}
    \label{eq:acceleration}
  \end{equation}
  \item This race has a standing start, so the initial speed of each racer will be 0.
  \item The time for the race is 30 seconds, this is constant.
  \item Each racer will have a randomly determined acceleration, with a maximum acceleration of 10 $pixels/second^2$
\end{itemize}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.6\textwidth]{./topics/storing-using-data/examples/BikeRace.png}
   \caption{Example execution of the Bike Race program}
   \label{fig:bike-race-img}
\end{figure}

\csection{\ccode{clst:bike-race}{C Bicycle Race, continued in \lref{clst:bike-race1}}{topics/storing-using-data/examples/bike-race.c}}

\csection{\ccode{clst:bike-race1}{C Bicycle Race}{topics/storing-using-data/examples/bike-race1.c}}

\passection{\pascode{lst:bike-race-pas}{Pascal Bike Race)}{topics/storing-using-data/examples/BikeRace.pas}}

\passection{\pascode{lst:bike-race-1-pas}{Pascal Bike Race (continued from \lref{lst:bike-race-pas})}{topics/storing-using-data/examples/BikeRace1.pas}}

% subsection bicycle_race (end)

% \clearpage
% \subsection{Comet Orbit} % (fold)
% \label{sub:comet_orbit}
%
% This program uses SwinGame to draw the orbit of the Hale-Bopp comet around the sun. The Hale-Bopp comet performs an elliptical orbit of the sun that can be plotted using Equation~ \ref{eq:orbit}. This equation calculates the radius (r) of the comet's position based on the \emph{angle} between the comet and the sun. Where \emph{e} is the Eccentricity value with a constant value of 0.995, and \emph{d} is the distance between the pole and directrix with a constant value of 1.828.
%
% \begin{equation}
%   r = \frac{ed}{1 + e sin(angle)}
%   \label{eq:orbit}
% \end{equation}
%
% \begin{table}[h]
% \centering
% \begin{tabular}{l|p{10cm}}
%   \hline
%   \multicolumn{2}{c}{\textbf{Program Description}} \\
%   \hline
%   \textbf{Name} & \emph{Comet Orbit} \\
%   \\
%   \textbf{Description} & Calculates and plots the position of the Hale-Bopp comet, based on an equation of its elliptical orbit of the sun. \\
%   \hline
% \end{tabular}
% \caption{Description of the Comet Orbit program}
% \label{tbl:data-comet-orbit}
% \end{table}
%
% This will require functions and procedures to do the following:
% \begin{itemize}
%   \item A function to \textbf{calculate} the \textbf{r} value for the comet based on an \emph{angle}.
%   \item Functions to \textbf{convert} the \textbf{x} and \textbf{y} positions of the comet from AU (Astronomical Units) to pixel coordinates so that the comets position can be plotted on the screen.
%   \item A procedure to \textbf{Draw} the \textbf{comet} to the screen, based on its current angle.
%   \item A procedure to \textbf{Draw} the \textbf{sun}.
%   \item A procedure to \textbf{draw} the entire \textbf{system}, including the sun and the comet at a given angle.
%   \item The main procedure to coordinate actions (calling, draw system with different angle values).
% \end{itemize}
%
% \clearpage
%
% \csection{\ccode{clst:comet-orbit}{Comet Orbit, continued in \lref{clst:comet-orbit1}}{topics/storing-using-data/examples/comet-orbit.cpp}}
%
% \begin{figure}[p]
% \csection{\ccode{clst:comet-orbit1}{Comet Orbit, continued in \lref{clst:comet-orbit2}}{topics/storing-using-data/examples/comet-orbit1.cpp}}
% \end{figure}
%
% \begin{figure}[p]
% \csection{\ccode{clst:comet-orbit2}{Comet Orbit}{topics/storing-using-data/examples/comet-orbit2.cpp}}
% \end{figure}
%
% % subsection comet_orbit (end)


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{enumitem}
\usepackage{physics}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}
\relpenalty=10000
\binoppenalty=10000
\renewcommand{\thesubsection}{\arabic{subsection}}
\renewcommand*{\theenumi}{\thesubsection.\arabic{enumi}}

\begin{document}
\section*{Functional equations}

\subsection{Injective and surjective functions}
	\begin{itemize}
		\item
		If $f(x)=f(y) \Rightarrow x=y$, then $f$ is injective
		\item
		If for each element $y$ in function codomain, there exists $x$ for which $f(x)=y$, then $f$ is surjective.
		\item
		If $f$ is both injective and surjective then $f$ is bijective.
	\end{itemize}

	\subsubsection*{Problems}
		\begin{enumerate}
			\item
			Let $f: X \to Y$ and $g: Y \to X$ and $g(f(x))=x$. Prove that $f$ is injective and $g$ is surjective.
			\item
			Prove that for any function $f: X \to Y$, there exists a set $Z$ and functions $g: X \to Z$ and $h: Z\to Y$, such that $g$ is injective and $h$ is surjective.
			\item
			Find all strictly monotonic functions $f: \mathbb{R} \to \mathbb{R}$ which satisfy
			$$f(x + f(y)) = f(x) + y$$
		\end{enumerate}

	\subsection{Cauchy functional equations} % Most from Engel book
		\begin{enumerate}
			\item
			Find all functions $f: \mathbb{Q} \to \mathbb{Q}$ for which $f(x)+f(y)=f(x+y)$.
			\item \label{conditions}
			If the above functional equation satisfies any one of the following conditions, find the solutions to the functional equation if $f: \mathbb{R} \to \mathbb{R}$.
			\begin{itemize}
				\item The function is continuous at one point,
				\item The function is monotonic on any interval,
				\item The function is bounded on any interval.
			\end{itemize}
			\item \label{eq:pow}
			Find all functions $f: \mathbb{R^+} \to \mathbb{R^+}$ for which $f(x)f(y)=f(xy)$, given it satisfies any of the constraints defined in question \ref{conditions}.

			\item \label{eq:log}
			Find all functions $f: \mathbb{R^+} \to \mathbb{R}$ for which $f(x)+f(y)=f(xy)$, given it satisfies any of the constraints defined in question \ref{conditions}.

			\item \label{eq:exp}
			Find all functions $f: \mathbb{R} \to \mathbb{R^+}$ for which $f(x)f(y)=f(x+y)$, given it satisfies any of the constraints defined in question \ref{conditions}.

			\item  In what domains are the solutions of the functional equations defined in questions \ref{eq:pow}-\ref{eq:exp} the only solutions if they are not subject to any of the constraints defined in \ref{conditions}?

			\item Find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ for which
			$$f(x+y) + f(x-y) = 2f(x) + 2f(y) $$

			\item Find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ for which
			$$f(x+y) + f(x-y) = 2f(y) $$

			\item Find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ for which
			$$f(x+y) + f(x-y) = 2f(x) $$

			\item Find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ for which
			$$f(x+y) + f(x-y) = 2f(x) f(y) $$

	\end{enumerate}

	\subsection{Recurrence relations} % https://artofproblemsolving.com/wiki/index.php/Characteristic_polynomial
		\begin{itemize}
			\item
			A recurrence relation is a relation that determines the elements of a sequence $x_n, n \in \mathbb{N}_0$, as a function of previous elements. A recurrence relation of the form
			$$(\forall n \geq k) \hspace{2em} x_n + a_1x_{n-1} + \hdots + a_kx_{n-k} = 0$$
			for constants $a_1, \hdots, a_k$ is called a linear homogeneous recurrence relation of order $k$.
			\item
			We define the characteristic polynomial of the relation as
			$$P(x) = x^k + a_1x^{k-1} + \hdots + a_k$$
			\item
			Let $P(x)$	factorize as
			$$P(x) = (x-\alpha_1)^{k_1}(x-\alpha_2)^{k_2} \hdots (x-\alpha_r)^{k_r}$$
			where $\alpha_1, \hdots , \alpha_r$ are distinct complex numbers and $k_1, \hdots, k_r$ are positive integers.
			\item
			The general solution of this recurrence relation is in this case given by
			$$x_n = p_1(n)\alpha_1^n + p_2(n)\alpha_2^n + \hdots + p_r(n)\alpha_r^n$$
			where $p_i$ is a polynomial of degree less than $k_i$.
			\item
			In particular, if $P(x)$ has $k$
			distinct roots, then all $p_i$ are constant.
			\item
			If $x_0, \hdots, x_{k-1}$ are set, then the coefficients of the polynomials are uniquely determined.
		\end{itemize}

			\subsubsection*{Problems}
			\begin{enumerate}
				\item
				Find the closed form expression for $n$-th term of the Fibonacci sequence.
				\item
				Let $S_n$ denote the number of ternary sequences (consisting of $0$,$1$, and $2$s) of length $n$, such that they do not contain a substring of "10", "01", or "11". Find a closed form expression for $S_n$.
				\item
				Let $r$ be a real number, and let $x_n$ be a sequence such that $x_0 = 0, x_1 = 1$, and $x_{n+2} = rx_{n+1} - x_n$ for $n \ge 0$. For which values of $r$ does $x_1 + x_3 + \cdots + x_{2m-1} = x_m^2$ for all positive integers $m$?
				\item
				Let $a_{n}$, $b_{n}$, and $c_{n}$ be geometric sequences with different common ratios and let $a_{n}+b_{n}+c_{n}=d_{n}$ for all integers $n$. If $d_{1}=1$, $d_{2}=2$, $d_{3}=3$, $d_{4}=-7$, $d_{5}=13$, and $d_{6}=-16$, find $d_{7}$.
			\end{enumerate}
\end{document}


\lab{Solitons}{Solitons}
\label{lab:solitons}

\objective{
We study traveling wave solutions of the Korteweg-de Vries (KdV) equation, using a pseudospectral discretization in space and a Runge-Kutta integration scheme in time.  }

Here we consider soliton solutions of the Korteweg-de Vries (KdV) equation. This equation is given by
\[  \frac{\partial u }{\partial t} + u \frac{\partial u}{\partial x} + \frac{\partial^3 u}{\partial x^3} = 0.
\]
The KdV equation is a canonical equation that describes shallow water waves.

The KdV equation possesses traveling wave solutions called solitons.
These traveling waves have the form
\[ u(x,t) = 3s \sech^2\left(\frac{\sqrt{s}}{2}(x - st - a)\right),
\]
where $s$ is the speed of the wave.
Solitons were first studied by John Scott Russell in 1834, in the Union Canal in Scotland.
When a canal boat suddenly stopped, the water piled up in front of the boat continued moving down the canal in the shape of a pulse.

Note that there is a soliton solution for each wave speed $s$, and that the amplitude of the soliton depends on the speed of the wave.
Solitons are traveling waves in the shape of a pulse, they are nonlinearly stable (bumped waves return to their previous shape), and they maintain their energy as they travel.
They also enjoy an additional stability property: they play well with others.
Two interacting solitons will maintain their shape after crossing paths.

\section*{Numerical solution}
Consider the KdV equation on $[-\pi,\pi]$, together with an appropriate initial condition:
\begin{align*}
	 &{ }u_t = -\left(\frac{u^2}{2} \right)_x - u_{xxx},\\
     &{ }u(x,0) = u_0(x).
\end{align*}
We will use initial data that is zero at the endpoints.
This will allow us to use a pseudospectral method for periodic initial data to find a numerical approximation for the solution $u(x,t)$.

If we use $N$ subintervals in space, we then obtain the spatial step $h = 2\pi/N$ and the grid points $\{x_j\}_{j=1} = \{-\pi,-\pi + h,\ldots,\pi-h\}$.
Let $\mathcal{F}(u)(t) = \hat{u}(t)$ denote the Fourier transform of $u(x,t)$ (in space), so that
\[
\mathcal{F}(u) = \hat{u}(k,t), \quad k=-N/2+1, \ldots, N/2.
\]
Similarly we let $\mathcal{F}^{-1}$ represent the discrete inverse Fourier transform.
Recall that $k$ represents the wave numbers in Fourier space; our code defines it by
\begin{lstlisting}
# Array of wave numbers.  This array is reordered in Python to
# accomodate the ordering inside the fft function in scipy.
k = np.concatenate(( np.arange(0,N/2) ,
					 np.array([0])	,
					 np.arange(-N/2+1,0,1)	)).reshape(N,)
\end{lstlisting}

We now apply the Fourier transform to the KdV equation.
In Fourier space, we obtain
\begin{align*}
	\mathcal{F}(u)_t &= -\frac{ik}{2}\mathcal{F}(u^2)- (ik)^3\mathcal{F}(u).
\end{align*}
Let $U(t)$ be the vector valued function given by $U(t) = (u(x_j,t))_{j=1}^N$.
Let $\mathcal{F}(U)(t)$ denote the discrete Fourier transform of $u(x,t)$ (in space), so that
\[
\mathcal{F}(U)(t) = (\mathcal{F}(u)(k,t))_{k=-N/2+1}^{N/2}.
\]
Similarly we let $\mathcal{F}^{-1}$ represent the discrete inverse Fourier transform.
Using the pseudospectral approximation in space leads to the system of ODEs
\begin{align}
	\mathcal{F}(U)_t =  -\frac{i}{2} \vec{k}\mathcal{F}\left( \mathcal{F}^{-1}(\mathcal{F}(U))^2\right) + i\vec{k}^3\mathcal{F}(U)
\end{align}
where $\vec{k}$ is a vector, and multiplication is done element-wise. In terms of $Y = \mathcal{F}(U)$, this simplifies to
\begin{align}
	Y_t =  -\frac{i}{2} \vec{k}\mathcal{F}\left( \mathcal{F}^{-1}(Y)^2\right) + i\vec{k}^3Y
	\label{lab:solitons:pseudospectral}
\end{align}
and is implemented below.
\begin{lstlisting}
# Defines the left hand side of the ODE y' = G(t,y)
# defined above.
ik3 = 1j*k**3.
def G_unscaled(t,y):
	out = -.5*1j*k*fft(ifft(y,axis=0)**2.,axis=0)  + ik3*y
	return out
\end{lstlisting}

Equation \eqref{lab:solitons:pseudospectral} is solved below, using a soliton as initial data for the KdV equation.
Note that the Fourier transform must be applied to the soliton before solving, and that the final numerical solution must be transformed back from Fourier space before plotting.
\begin{lstlisting}
N = 256
x = (2.*np.pi/N)*np.arange(-N/2,N/2).reshape(N,1)   # Space discretization
s, shift = 25.**2., 2.  							# Initial data is a soliton
y0 = (3.*s*np.cosh(.5*(sqrt(s)*(x+shift)))**(-2.)).reshape(N,)

# Solves the ODE.
max_t = .0075
dt = .2*N**(-2.)
max_tsteps = int(round(max_t/dt))
y0 = fft(y0,axis=0)
T,Y = RK4(G_unscaled, y0, t0=0, t1=max_t, n=max_tsteps)

# Using the variable stride, we step through the data,
# applying the inverse fourier transform to obtain u.
# These values will be plotted.
stride = int(np.floor((max_t/25.)/dt))
uvalues, tvalues = np.real(ifft(y0,axis=0)).reshape(N,1), np.array(0.).reshape(1,1)
for n in range(1,max_tsteps+1):
	if np.mod(n,stride) == 0:
		t = n*dt
		u = np.real( ifft(Y[n], axis=0) ).reshape(N,1)
		uvalues = np.concatenate((uvalues,np.nan_to_num(u)),axis=1)
		tvalues = np.concatenate((tvalues,np.array(t).reshape(1,1)),axis=1)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.view_init(elev=45., azim=150)
tv, xv = np.meshgrid(tvalues,x,indexing='ij')
surf = ax.plot_surface(tv,xv, uvalues.T, rstride=1, cstride=1, cmap=cm.coolwarm,
						linewidth=0, antialiased=False)
tvalues = tvalues[0]; ax.set_xlim(tvalues[0], tvalues[-1])
ax.set_ylim(-pi, pi); ax.invert_yaxis()
ax.set_zlim(0., 4000.)
ax.set_xlabel('T'); ax.set_ylabel('X'); ax.set_zlabel('Z')
plt.show()
\end{lstlisting}

The method we have used requires the use of an algorithm for (ODE) initial value problems, such as the RK4 algorithm.
The RK4 method is implemented below.
\begin{lstlisting}
def initialize_all(y0, t0, t1, n):
	""" An initialization routine for the different ODE solving
	methods in the lab. This initializes Y, T, and h. """

	if isinstance(y0, np.ndarray):
		Y = np.empty((n, y0.size),dtype=complex).squeeze()
	else:
		Y = np.empty(n,dtype=complex)
	Y[0] = y0
	T = np.linspace(t0, t1, n)
	h = float(t1 - t0) / (n - 1)
	return Y, T, h

def RK4(f, y0, t0, t1, n):
	""" Use the RK4 method to compute an approximate solution
	to the ODE y' = f(t, y) at n equispaced parameter values from t0 to t
	with initial conditions y(t0) = y0.

	'y0' is assumed to be either a constant or a one-dimensional numpy array.
	't0' and 't1' are assumed to be constants.
	'f' is assumed to accept two arguments.
	The first is a constant giving the current value of t.
	The second is a one-dimensional numpy array of the same size as y.

	This function returns an array Y of shape (n,) if
	y is a constant or an array of size 1.
	It returns an array of shape (n, y.size) otherwise.
	In either case, Y[i] is the approximate value of y at
	the i'th value of np.linspace(t0, t, n).
	"""
	Y, T, h = initialize_all(y0, t0, t1, n)
	for i in xrange(1, n):
		K1 = f(T[i-1], Y[i-1])
		tplus = (T[i] + T[i-1]) * .5
		K2 = f(tplus, Y[i-1] + .5 * h * K1)
		K3 = f(tplus, Y[i-1] + .5 * h * K2)
		K4 = f(T[i], Y[i-1] + h * K3)
		Y[i] = Y[i-1] + (h / 6.) * (K1 + 2 * K2 + 2 * K3 + K4)
	return T, Y
\end{lstlisting}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{soliton.png}
\caption{The solution to Problem \ref{problem:solitons:single}.}
\label{fig:solitons:single}
\end{figure}

\begin{problem}
Run the code above to numerically solve the KdV equation on $[-\pi,\pi]$ with initial conditions
\[
u(x,t=0) = 3s\sech^2\left(\frac{\sqrt{s}}{2}(x+a)\right),
\]
where $s = 25^2,$ $a = 2$. Solve on the time domain $[0,.0075]$.
The solution is shown in Figure \ref{fig:solitons:single}.
\label{problem:solitons:single}
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{interacting_solitons.png}
\caption{The solution to Problem \ref{problem:solitons:interacting}.}
\label{fig:solitons:interacting}
\end{figure}

\begin{problem}
Numerically solve the KdV equation on $[-\pi,\pi]$.
This time we define the initial condition
to be the superposition of two solitons,
\[
u(x,t=0) = 3s_1\sech^2\left(\frac{\sqrt{s_1}}{2}(x+a_1)\right) + 3s_2\sech^2\left(\frac{\sqrt{s_2}}{2}(x+a_2)\right),
\]
where $s_1 = 25^2,$ $a_1 = 2$, and $s_2 = 16^2,$ $a_1 = 1$.\footnote{This problem is solved in \textit{Spectral Methods in MATLAB}, by Trefethen.}
Solve on the time domain $[0,.0075]$.
The solution is shown in Figure \ref{fig:solitons:interacting}.
\label{problem:solitons:interacting}
\end{problem}

\begin{problem}
Consider again equation \eqref{lab:solitons:pseudospectral}.
The linear term in this equation is $i\vec{k}^3Y$.
This term contributes much of the exponential growth in the ODE, and responsible for how short the time step must be to ensure numerical stability.
Make the substitution $Z = e^{-ik^3t}Y$ and find a similar ODE for $Z$.
This essentially allows the exponential growth to be scaled out (it's solved for analytically).
Use the resulting equation to solve the previous problem.
\end{problem}