\chapter[Partial Differential Equations]{Finite difference methods for Partial Differential Equations}

\section[Revision]{Revision of Partial Differential Equations}

A {\em partial differential equation} is a relation between the
partial derivatives of an unknown function and the independent
variables.  The order of the highest derivative is called the {\em
  order} of the equation.

Just as in the case of an ordinary differential equation, we say that
a partial differential equation is {\em linear} if it is of the first
degree in the dependent variable (the unknown function) and its
partial derivatives.  If each term of such equation contains either
the dependent variable or one of its derivatives, the equation is
called {\em homogeneous}; otherwise it is said to be {\em non
  homogeneous}.

\smallskip

\noindent
{\bf Example} - The equations
%
\begin{align*}
  (a) & & \pdv{\phi}{t} - \pdv[2]{\phi}{x} &= 0, \\
  (b) & & \phi(x,t) \pdv[2]{\phi}{t} + \pdv{\phi}{x} &= f(x,t), \\
  (c) & & \pdv[4]{\phi}{x} + \pdv[4]{\phi}{y} &= g(x,y) ,
\end{align*}
%
are, respectively, (a) Linear, homogeneous, second order, (b)
nonlinear, non-homogeneous, second order, (c) linear, non-homogeneous,
fourth order.

\smallskip

In this unit we consider only second order linear partial differential
equations with constant coefficients:
%
\begin{equation*}
  A \pdv[2]{u}{x} + 2 B \pdv{u}{x}{y} +
  C \pdv[2]{u}{y} + D \pdv{u}{x} + E \pdv{u}{y} = f(x,y) .
\end{equation*}
%
These equations are classified in three groups\footnote{Note that the
  coefficients $A$, $B$, etc. need not be constant}:

\medskip

\begin{center}
  \begin{tabular}{|l|l|l|l|} \hline
    \multicolumn{1}{|c|}{ Type} &
    \multicolumn{1}{c|}{Coefficients} &
    \multicolumn{1}{c|} {Typical form} &
    \multicolumn{1}{c|} {Name} \\ \hline \hline
    Hyperbolic & $B^2 - 4 A C > 0 $ &
    $u_{t t} - c^2 u_{x x} = 0 $ & Wave eq. \\ \hline
    Parabolic & $B^2 - 4 A C = 0 $ &
    $u_{t} - c^2 u_{x x} = 0 $ & Heat eq. \\ \hline
    Elliptic & $B^2 - 4 A C < 0 $ &
    $u_{x x} + u_{y y} = 0 $ & Laplace eq. \\ \hline
  \end{tabular}
\end{center}

\medskip

\noindent
The names arise by analogy with the curve
%
\begin{equation*}
  a x^2 + 2 b x y + c y^2 = f,
\end{equation*}
%
which represents an hyperbola, parabola and ellipse according as $b^2
- 4 a c$ is positive, zero or negative, respectively.

The classification of the equations is very important for the study of
their solutions.  A {\em solution} of a partial differential equation
in some region $R$ of the space of the independent variables is a
function that has all the partial derivatives that appear in the
equation and that satisfies the equation everywhere in $R$.  Like
ordinary differential equations, Partial differential equation have
many solutions.  For example, the functions
%
\begin{equation*}
  u(x,y) = x^2 - y^2, \quad u = e^x \cos(y), \quad u=\ln(x^2+y^2),
\end{equation*}
%
are all solutions of the equation
%
\begin{equation*}
  \pdv[2]{u}{x} + \pdv[2]{u}{y} = 0 .
\end{equation*}
%
Exactly like for ordinary differential equations we must specify
something more if we wish to obtain a unique solution: for example we
must specify the value of the function on the boundary of the region
$R$ ({\em boundary conditions}), and/or we must specify the value of
the function at the start ({\em initial conditions}).  What type of
boundary or initial condition to use depends on the type of Partial
differential equation.

\medskip

\centerline{\bf Hyperbolic}

\noindent \underline{Example}: $u_{t t} - c^2 u_{xx} = 0$.

\noindent \underline{Physical meaning}: Wave motion

\noindent \underline{Initial conditions}: The value of $u$ and its
time derivative at $t=0$: $u(x,0)=f(x)$ and $u_t(x,0)=g(x)$.

\noindent \underline{Boundary conditions}: Either $u$ or its normal
derivative at the boundary of the integration region: e.g.\ $u(0,t) =
\phi_1(t)$ and $u(L,t)= \phi_2(t)$.  The former type of boundary
condition is called a \textit{Dirichlet} boundary condition; the
latter is called a \textit{Neumann} boundary condition.  It is also
possible to mix the two types of boundary conditions (\textit{mixed
  boundary conditions}).

\bigskip

\centerline{\bf Parabolic}

\noindent \underline{Example}: $u_{t} - c^2 u_{xx} = 0$.

\noindent \underline{Physical meaning}: Heat diffusion

\noindent \underline{Initial conditions}: The value of $u$ at $t=0$:
$u(x,0)=f(x)$.

\noindent \underline{Boundary conditions}: Either $u$ or its normal
derivative at the boundary of the integration region: e.g.\ $u(0,t) =
\phi_1(t)$ and $u(L,t)= \phi_2(t)$.

\bigskip

\centerline{\bf Elliptic}

\noindent \underline{Example}: $u_{xx} + u_{y y} = 0$ (Laplace's
equation).

\noindent \underline{Physical meaning}: Stationary profile of a
membrane, stationary temperature profile of a metal plate,
electrostatic potential in the absence of charges.

\noindent \underline{Initial conditions}: They do not apply to these
equations.

\noindent \underline{Boundary conditions}: Either $u$ or its normal
derivative at the boundary of the integration region: e.g.\ if the
region is the rectangle $0 \le x \le a$, $0 \le y \le b$: $u(0,y) =
\phi_1(y)$, $u(a,y)= \phi_2(y)$, $u(x,0) = g_1(x)$ and, finally,
$u(x,b)=g_2(x)$.

\begin{figure}
  \centerline{\includegraphics[width=120mm]{figures/pde_summary}}
  \caption{\label{fig:pde_summary} \it Types of partial differential
    equations and their boundary and initial conditions.}
\end{figure}

\section{Numerical methods for PDEs}

There are many classes of methods to solve numerically partial
differential equations.  Three main groups are:

\begin{enumerate}
  %
\item \textit{Finite difference methods}

  The differential operators are approximated using their finite
  difference representation on a given grid.  In this way the Partial
  Differential Equation is transformed to a (nonlinear) algebraic
  equation for the values of the solution on the grid points.  These
  are the only methods that we discuss at any length in this unit.
  %
\item \textit{Finite element methods}

  The domain of the solution is divided into cells.  The solution is
  represented as a simple function (e.g.\ a linear function) on each
  cell and the partial differential equation is transformed to an
  algebraic problem for the matching conditions of the simple
  solutions at the boundaries of the cells.
  %
\item \textit{Spectral methods}

  The solution is represented by a superposition of known functions
  (e.g.\ trigonometric functions or special polynomials).  The partial
  differential equation is transformed to a set of algebraic equations
  or ordinary differential equations for the amplitudes of the
  component functions.  A subclass of these methods are the
  \textit{collocation methods}: the solution is represented on a grid
  and the decomposition of the solution in known functions is used to
  estimate to a high degree of accuracy the partial derivatives of the
  solution on the grid points.  These methods are highly accurate and
  fast, but in general require the domain of the solution to be fairly
  simple, e.g.\ a rectangle.
  %
\end{enumerate}

\section[Elliptic Equations]{Elliptic Equations}

\subsection{Introduction}

A ``standard form'' of elliptic partial differential equation with
Dirichlet boundary conditions is
%
\begin{equation}
  \left\{
    \begin{aligned}
      u_{xx} + u_{y y} & = f(x,y), & & {\rm in}~~\Omega,\\
      u\big |_{\partial\Omega} & = \phi(x,y), & & {\rm on}~~\partial\Omega.
    \end{aligned}
  \right.\label{dp}
\end{equation}
%
This problem has a unique solution if the domain $\Omega$ has a smooth
boundary and $f$ and $\phi$ are continuous functions.

\subsection{A simple finite difference method}

We are going to solve equation~(\ref{dp}) numerically using the method
of finite differences.  At the heart of this method is the finite
difference approximation of the second derivative of a function
$y(x)$:
%
\begin{equation}
  y''(x) = \frac{1}{h^2}[y(x+h)+y(x-h)-2y(x)]-
  \frac{h^2}{12}y^{(4)}(\xi), \label{fd}
\end{equation}
%
where $h$ is a fixed step size.  We illustrate this method using a
simple case of equation~(\ref{dp}), namely the case of a Dirichlet
problem on a rectangle $0<x<a$, $0<y<b$.  First, a network of grid
points is established on the rectangle:
%
\begin{equation*}
  (x_i,y_j)=(i h_x,j h_y), \qquad 0\le i\le n+1, \quad 0\le j\le m+1,
\end{equation*}
%
where the step sizes in the $x$ and $y$ directions are given by
\begin{equation*}
  h_x=\frac{a}{n+1}, \qquad h_y=\frac{b}{m+1}.
\end{equation*}
%
Next, the differential equation in (\ref{dp}) at the mesh point
$(x_i,y_j)$ is replaced by its finite-difference analogue at that
point, which for $1 \le i \le n$ and $1 \le j \le m$ is:
%
\begin{align}
  \label{eq:ellfd}
  & & \frac{1}{h_x^2}[u_{i-1,j}+u_{i+1,j}-2u_{i,j}] +
  \frac{1}{h_y^2}[u_{i,j-1}+u_{i,j+1}-2u_{i,j}] & = f_{i j},
  \nonumber \\
  \implies & & u_{i-1,j}+u_{i+1,j}-2u_{i,j} + \alpha
  [u_{i,j-1}+u_{i,j+1}-2u_{i,j}] & = h^2f_{i,j},
\end{align}
%
where $u_{i,j} = u(x_i,x_j)$, $\alpha=(h_x/h_y)^2$ and $h=h_x$.  The
values of $u_{i,j}$ are known when $i=0$ or $n+1$ and when $j=0$ or
$m+1$, since these are the prescribed boundary values in the problem:
%
\begin{align*}
  u_{0,j} & = \phi(x_0,y_j)  = \phi(0,y_j) , &
  u_{i,0} & = \phi(x_i,y_0)  = \phi(x_i,0), \\
  u_{n+1,j} & = \phi(x_{n+1},y_j)  = \phi(a,y_j), &
  u_{i,m+1} & = \phi(x_i,y_{m+1})  = \phi(x_i,b).
\end{align*}
%
This means that equation~(\ref{dp}) has been transformed to a
non-homogeneous system of linear equations, given by~(\ref{eq:ellfd}),
with unknowns $u_{i,j}$ and $1\le i\le n$, $1\le j\le m$.

As an example, consider the simple case $n=2$ and $m=3$.  The
approximate solution of equation~(\ref{dp}) at the grid points are the
solution of the $6\times 6$ system
%
\begin{align*}
  [u_{01}-2u_{11}+u_{21}]+\alpha[u_{10}-2u_{11}+u_{12}] &= h^2 f_{11},\\
  [u_{02}-2u_{12}+u_{22}]+\alpha[u_{11}-2u_{12}+u_{13}] &= h^2 f_{12},\\
  [u_{03}-2u_{13}+u_{23}]+\alpha[u_{12}-2u_{13}+u_{14}] &= h^2 f_{13},\\
  [u_{11}-2u_{21}+u_{31}]+\alpha[u_{20}-2u_{21}+u_{22}] &= h^2 f_{21},\\
  [u_{12}-2u_{22}+u_{32}]+\alpha[u_{21}-2u_{22}+u_{23}] &= h^2 f_{22},\\
  [u_{13}-2u_{23}+u_{33}]+\alpha[u_{22}-2u_{23}+u_{24}] &= h^2 f_{23}.
\end{align*}
%
The unknown quantities in this problem can be ordered in many ways. We
select the one known as the natural ordering
%
\begin{equation*}
  u=[u_{11},u_{12},u_{13},u_{21},u_{22},u_{23}]^T.
\end{equation*}
%
The system has the form $A \bu = \bF$ with
%
\begin{equation}
  A = \begin{pmatrix}
      -2 (1+\alpha) & \alpha & 0 & 1 & 0 & 0 \\
      \alpha & -2(1+\alpha) & \alpha & 0 & 1 & 0 \\
      0 & \alpha & -2(1+\alpha) & 0 & 0 & 1 \\
      1 & 0 & 0 & -2(1+\alpha) & \alpha & 0 \\
      0 & 1 & 0 & \alpha & -2(1+\alpha) & \alpha \\
      0 & 0 & 1 & 0 & \alpha & -2(1+\alpha) \\
    \end{pmatrix} \label{eq:PDE:A}
\end{equation}
%
and
%
\begin{equation*}
  \bF =
  \begin{pmatrix}
    h^2f_{11}- u_{01}-\alpha u_{10} \\
    h^2f_{12}- u_{02} \\
    h^2f_{13}- u_{03}-\alpha u_{14} \\
    h^2f_{21}- u_{31}-\alpha u_{20} \\
    h^2f_{22}- u_{32} \\
    h^2f_{23}- u_{33}-\alpha u_{24}
  \end{pmatrix},
\end{equation*}
%
where in the expressions of the components of the vector $\bF$ only
the boundary values of $u_{i,j}$ are present.

In general, the $n\times m$ system is sparse because each equation
contains at most five unknowns.  Iterative procedures such as the
Gauss-Seidel iterative method can be quite effective in this
situation.  Moreover, in setting up such a procedure there is no need
to store the matrix $A$ of the coefficients defined in
equation~(\ref{eq:PDE:A}).  In fact, we can rewrite
equation~(\ref{eq:ellfd}) as
%
\begin{align}
  u_{i,j} &= \frac{1}{2(1+\alpha)} \left [ u_{i-1,j} +u_{i+1,j} +
    \alpha (u_{i,j-1}+u_{i,j+1}) - h^2 f_{i,j} \right ],
    \label{PDE:eq:Ell:Updateuij} \\ & \notag \qquad
  1 \le i \le n, \, 1 \le j \le m .
\end{align}
%
This equation is a Gauss-Seidel formula to update $u_{i,j}$. When this
equation is used, the value obtained from the right-hand side replaces
the old value of $u_{i,j}$.  It can also be seen as an implementation
of Jacobi's method, if we assume that the variable on the left-hand
side are stored separately from those on the right hand side.  As
initial guess of the solution we can assume that $u_{i,j} \equiv 0$
(in general we should use a guess that satisfies the boundary
conditions).  Of course an initial guess close to the solution will
reduce the number of iterations needed for convergence.

\subsection{Error analysis}

How accurate is the solution obtained using~(\ref{eq:ellfd}) or,
equivalently~(\ref{PDE:eq:Ell:Updateuij})? To answer this question
we introduce the error
%
\begin{equation*}
  e_{i,j}=u_{i,j}-\hat u_{i,j},
\end{equation*}
%
where $\hat u_{i,j}=u(x_i,y_j)$ are the values of the exact solution
of equation~(\ref{dp}). Substituting
%
\begin{equation*}
  u_{i,j}=e_{i,j}+\hat u_{i,j}
\end{equation*}
%
into the difference equation~(\ref{eq:ellfd}), we obtain
%
\begin{align*}
 \frac{1}{h_x^2}[e_{i-1,j}+e_{i+1,j}-2e_{i,j}] +
 \frac{1}{h_y^2}[e_{i,j-1}+e_{i,j+1}-2e_{i,j}] &= f_{i,j} - \\
 & \qquad \frac{1}{h_x^2}[\hat u_{i-1,j}+\hat u_{i+1,j}-2\hat u_{i,j}] - \\
 & \qquad \frac{1}{h_y^2}[\hat u_{i,j-1}+\hat u_{i,j+1}-2\hat u_{i,j}].
\end{align*}
%
Using the finite-difference approximation formula~(\ref{fd}) to
eliminate the finite differences we obtain
%
\begin{align*}
  \frac{1}{h_x^2}[e_{i-1,j}+e_{i+1,j}-2e_{i,j}] +
     \frac{1}{h_y^2}[e_{i,j-1}+e_{i,j+1}-2e_{i,j}] & =
     f_{i,j} - \\
     & \quad [u_{xx}(x_i,y_j) + \frac{h_x^2}{12}u_{xxxx}(\xi_i,y_i)] -
     \\
     & \quad [u_{y y}(x_i,y_j)+\frac{h_2^2}{12}u_{yyyy}(x_i,\zeta_j)] \\
     & = - \frac{h_x^2}{12}u_{xxxx}(\xi_i,y_i) -
     \frac{h_y^2}{12}u_{yyyy}(x_i,\zeta_j)
\end{align*}
%
where we have used
%
\begin{equation*}
  u_{xx}(x_i,y_j)+u_{y y}(x_i,y_j)=f_{i.j}.
\end{equation*}
%
Hence, the error satisfies the difference equation
%
\begin{equation*}
  e_{i-1,j}+e_{i+1,j}-2e_{i,j}+\alpha[e_{i,j-1}+e_{i,j+1}-2e_{i,j}]=
  -\frac{h_x^4}{12}u_{xxxx}(\xi_i,y_i)-
  \frac{h_x^2 h_y^2}{12}u_{yyyy}(x_i,\zeta_j),
\end{equation*}
%
and zero boundary conditions. Analysing this equality, it is possible
to prove that
%
\begin{equation*}
  ||e_{i,j}|| \simeq \order{h_x h_y}.
\end{equation*}
%
This means that in the limit $n\to +\infty$ and $m\to +\infty$ the
norm of the error tends to zero.

\section{Parabolic Equations}

\subsection{Introduction}

Throughout this section we assume that the parabolic equation to be
solved is
%
\begin{equation}
  \pdv{u}{t} = \pdv[2]{u}{x} , \qquad 0 \le x \le 1, \, 0 \le t
  \label{PDE:eq:Para:template}
\end{equation}
%
with boundary conditions
%
\begin{equation*}
  u(0,t) = 0, \qquad u(1,t) = 0,
\end{equation*}
%
and initial condition $u(x,0) = g(x)$.  There are many finite
difference methods to solve equation~(\ref{PDE:eq:Para:template}).
Here we consider only a few that are typical examples of their
respective classes.  Moreover, it should be noted that
equation~(\ref{PDE:eq:Para:template}) is linear.  The methods that we
discuss become much harder to implement in the case of nonlinear
equations.

\subsection[Forward-Time, Centred-Space (FTCS)]{An explicit method - Forward-Time, Centred-Space (FTCS)}

\subsubsection{The method}

We fix a time discretisation step $\delta$ and a space discretisation
step $h$ so that the solution of equation~(\ref{PDE:eq:Para:template})
is represented on the grid
%
\begin{equation*}
  (x_i,t^n) = (i h, n \delta), \qquad 0 \le i \le N + 1, \, n \ge 0
\end{equation*}
%
where the number of grid points between $x=0$ and $x=1$ is $N+2$.  We
use the notation $u_i^n$ to indicate $u(x_i,t^n)$.  We discretise the
time derivative in equation~(\ref{PDE:eq:Para:template}) using a
forward difference approximation
%
\begin{equation}
  \pdv{u}{t} = \frac{u_i^{n+1}-u_i^{n}}{\delta} + \order{\delta}
  \label{PDE:eq:ut}
\end{equation}
%
and the spatial derivative using
%
\begin{equation}
  \pdv[2]{u}{x} = \frac{u_{i+1}^{n}+u_{i-1}^{n}-2u_{i}^{n}}{h^2} + \order{h^2}
  \label{PDE:eq:uxx}
\end{equation}
%
so that equation~(\ref{PDE:eq:Para:template}) is represented by the
set of (linear) algebraic equations
%
\begin{equation*}
  u_{i}^{n+1} = u_{i}^{n} +
  \frac{\delta}{h^2} (u_{i+1}^{n}+u_{i-1}^{n}-2u_{i}^{n})
\end{equation*}
%
or, equivalently,
%
\begin{equation}
  u_{i}^{n+1} = (1-2s) u_{i}^{n} + s (u_{i+1}^{n}+u_{i-1}^{n})
  \label{PDE:eq:Para:expl}
\end{equation}
%
where $s = \delta/h^2$.  Equation~(\ref{PDE:eq:Para:expl}) is a finite
difference representation of Equation~(\ref{PDE:eq:Para:template}):
since this equation gives the new values of $u_{i}^{n+1}$ explicitly
in terms of previous values of $u_{i+1}^{n}$, $u_{i}^{n}$ and
$u_{i-1}^{n}$ the method based on this equation is called an
\textit{explicit method}.  As it involves a forward difference
approximation of the time derivative and a centred approximation of
the spatial derivative it is called a \textit{forward-time
  centred-space} (FTCS) method.

Equation~(\ref{PDE:eq:Para:expl}) must be complemented by the
discretized version of the initial and boundary conditions, namely
%
\begin{equation*}
  u_{0}^{n} = u_{N+1}^{n} = 0 \qand u_{i}^{0} = g(x_i) .
\end{equation*}

\subsubsection{Consistency, Stability and Convergence}

We have claimed that equation~(\ref{PDE:eq:Para:expl}) is a finite
difference representation of equation~(\ref{PDE:eq:Para:template}) and
that, therefore, can be used to find an accurate numerical solution to
this equation.  Given any algorithm to solve numerically a partial
differential equation we must clearly determine if it is ``good'', in
the sense that it can be used to obtain efficiently an accurate
solution of the problem we aim to solve.  In order to do this we must
define clearly what we mean by a ``good method''.

\begin{enumerate}
  %
\item A finite difference equation is \textbf{consistent} with a
  partial differential equation if the difference between the Finite
  Difference Equation (FDE) and the PDE (i.e.\ the truncation error)
  vanishes as the sizes of the grid spacings go to zero independently.

  When the truncation error of the finite difference approximations of
  the individual exact partial derivatives are known, proof of
  consistency is straightforward. When the truncation errors of the
  individual finite difference approximations are not known, the
  complete finite difference equation must be analysed for
  consistency.  That is accomplished by expressing each term in the
  finite difference equation by a Taylor series about a particular
  grid point.  The resulting equation, which is called the modified
  differential equation (MDE), can be simplified to yield the exact
  form of the truncation error of the complete finite difference
  equation.  We will not develop this concept further in this unit.
  %
\item The \textbf{order} of a finite difference approximation of a
  partial differential equation is the rate at which the global error
  of the finite difference solution approaches zero as the size of the
  grid spacings approach zero.

  The global error of a finite difference equation is the order of the
  truncation error terms in the finite difference approximations of
  the individual exact partial derivatives in the Partial Differential
  Equation.
  %
\item When applied to a partial differential equation that has a
  bounded solution, a finite difference equation is \textbf{stable} if
  it produces a bounded solution and is unstable otherwise.

  If the solution of the FDE is bounded for all values of the grid
  spacings, then the FDE is \textit{unconditionally stable}.  If the
  solution of the FDE is bounded only for certain values of the grid
  spacings, then the FDE is \textit{conditionally stable}.  If the
  solution of the FDE is unbounded for all values of the grid
  spacings, then the FDE is \textit{unconditionally unstable}.  The
  most used method to prove the stability of a (linear) finite
  difference scheme is the Von Neumann method (see below).
  %
\item A finite difference method is \textbf{convergent} if the
  solution of the finite difference equation approaches the exact
  solution of the partial differential equation as the sizes of the
  grid spacings go to zero.

  Convergence is the most desirable property of an integration scheme.
  However it is quite hard to prove directly that a method is
  convergent.
%
\end{enumerate}

Consistency, stability and convergence are not independent concepts.
They are related by the following theorem due to Lax (1954):

\begin{quote}
  %
  Given a properly posed linear initial-value problem and a finite
  difference approximation to it that is consistent, stability is the
  necessary and sufficient condition for convergence.
  %
\end{quote}

Thus, the question of convergence of a finite difference method is
answered by a study of the consistency and stability of the finite
difference equation.  If the finite difference equation is consistent
and stable, then the finite difference method is convergent.

The Lax equivalence theorem applies to well-posed linear initial-value
problems.  May problems in engineering and science are not linear and
nearly all problems involve boundary conditions in addition to initial
conditions.  There is no equivalence theorem for such problems.
Nonlinear PDEs must be linearised locally and the FDE that
approximates the linearised PDE is analysed for stability.  Experience
has shown that the stability criteria obtained of the linearised FDE
also apply to the nonlinear FDE and that FDEs that are consistent and
whose linearised equivalent is stable generally converge even for
nonlinear initial-boundary-value problems.

\subsubsection{Consistency, stability and convergence of a FTCS
method}

Equation~(\ref{PDE:eq:Para:expl}) is first order in time and second
order in space: in fact the error involved in the discretisation of
the time derivative, equation~(\ref{PDE:eq:ut}), is $\order{\delta}$, while
the error in the discretisation of the space derivative,
equation~(\ref{PDE:eq:uxx}), is $\order{h^2}$.

Equation~(\ref{PDE:eq:Para:expl}) is clearly consistent.  As $\delta
\to 0$ and $h \to 0$ the finite difference approximations of the time
and space derivatives, equations~(\ref{PDE:eq:ut},~\ref{PDE:eq:uxx}),
converge to the respective derivatives.

To verify whether the method given by
equation~(\ref{PDE:eq:Para:expl}) is stable, we start by observing
that the solution of equation~(\ref{PDE:eq:Para:template}) with
boundary conditions $u(0,t)=u(1,t)=0$ tends to zero in the long time
limit whatever the initial condition $g(x)$.  Therefore, also the
solution of the finite difference approximation,
equation~(\ref{PDE:eq:Para:expl}), must tend to zero as the time
discretisation index $n$ tends to infinity.  To verify whether this is
the case we use the \textit{Von Neumann method}.  The solution of
equation~(\ref{PDE:eq:Para:expl}) is of the form
%
\begin{equation}
  u_{\ell}^{k} = e^{\tj \alpha \ell h} q^k
  \label{PDE:eq:uellk}
\end{equation}
%
where $\tj = \sqrt{-1}$ while $\alpha$ and $q$ are real parameters
that have to be determined by requiring that~(\ref{PDE:eq:uellk})
satisfies~(\ref{PDE:eq:Para:expl}).  Substituting~(\ref{PDE:eq:uellk})
into~(\ref{PDE:eq:Para:expl}) we obtain that $q$ is given by
%
\begin{equation*}
  q = s \left ( e^{\tj \alpha h} + e^{-\tj \alpha h} \right ) + (1 - 2s) =
  1 - 4 s \sin^2 \left ( \frac{\alpha h}{2} \right ) .
\end{equation*}
%
In order for $|u_{\ell}^{k}|$ to decrease to zero as $k$ tends to
infinity we must have
%
\begin{equation}
  |q| < 1 \implies s < \frac{1}{2} \implies
  \frac{\delta}{h^2} < \frac{1}{2} .
  \label{PDE:eq:FTCSConstr}
\end{equation}
%
In other words, the FTCS method~(\ref{PDE:eq:Para:expl}) is only
conditionally stable and the time and space discretisation steps,
$\delta$ and $h$ respectively, must satisfy the
constraint~(\ref{PDE:eq:FTCSConstr}).  Using the Lax theorem we can
therefore conclude that, under these conditions, the FTCS method is
also convergent.  However, equation~(\ref{PDE:eq:FTCSConstr}) is a
rather stringent constraint as the time step must be reduce by a
factor of four if the space step is halved.  Therefore, methods
like~(\ref{PDE:eq:Para:expl}) tend to be rather slow.

\subsection[Backward-Time, Centred-Space (BTCS)]{An Implicit method - Backward-Time, Centred-Space (BTCS)}

\subsubsection{The method}

In the explicit finite difference approximation of the diffusion
equation the right hand side of equation~(\ref{PDE:eq:Para:template})
is computed at the current time step and used to compute the value of
the solution at the next time step.  In the implicit finite difference
approximation the right hand side is ``computed'' at the next time
step, so that the finite difference scheme is an implicit equation for
the new value of the solution.  In the case of
equation~(\ref{PDE:eq:Para:template}) this is accomplished by using a
backward difference approximation of the time derivative at
$t^{n+1}=t+\delta$ so that equation~(\ref{PDE:eq:Para:template}) is
discretised as
%
\begin{align}
 && \frac{u_{i}^{n+1} - u_{i}^{n}}{\delta} & =
       \frac{u_{i+1}^{n+1}+u_{i-1}^{n+1}-2u_{i}^{n+1}}{h^2} \nonumber \\
 \implies && (1+2s) u_{i}^{n+1} & = u_{i}^{n} + s
 (u_{i+1}^{n+1}+u_{i-1}^{n+1})
 \label{PDE:eq:Para:BTCS}
\end{align}
%
where $s = \delta/h^2$ and the unknowns are all the $u_{i}^{n+1}$.
Equation~(\ref{PDE:eq:Para:BTCS}) constitutes of a tri-diagonal linear
system for the unknowns $u_{i}^{n}$ with row
%
\begin{equation*}
  -s u_{i-1}^{n+1} + (1+2s) u_{i}^{n+1} - s u_{i+1}^{n+1} = u_{i}^{n},
  \qquad 1 \le i \le N .
\end{equation*}
%
This system can be solved by Gaussian elimination.  Note, however,
that if the partial differential equation is nonlinear then
equation~(\ref{PDE:eq:Para:BTCS}) becomes a nonlinear algebraic
equation for the unknowns $u_{i}^{n+1}$ and is, as a consequence, very
hard to solve.

\subsubsection{Consistency, stability and convergence}

Like the FTCS method, the BTCS is first order in time and second order
in space.  Moreover, it is clearly consistent.

The stability analysis is very similar to that of the explicit method.
We look for a solution of equation~(\ref{PDE:eq:Para:BTCS}) of the
form (\ref{PDE:eq:uellk}) and substitute it in
equation~(\ref{PDE:eq:Para:BTCS}).  We obtain that $q$ must satisfy
%
\begin{equation*}
  q = 1 + 2 s \left ( e^{\tj \alpha h} + e^{-\tj \alpha h} - 2 \right ) q \implies
  q = \frac{1}{1 + 4 s \sin^2 \left ( \frac{\alpha h}{2} \right )} \, .
\end{equation*}
%
In other words $|q| < 1$ for all values of $s$ and the backward-time,
centred space method is unconditionally stable and, hence,
unconditionally convergent.

\section{Hyperbolic Equations}

\subsection{Introduction}

The theory of hyperbolic equations and their solutions is quite
involved and is not studied in this unit.  However, it is essential to
know it and understand it if any serious work with hyperbolic partial
differential equations is to be attempted.  Here we consider a very
simple case and use it to discuss a few methods to integrate
hyperbolic partial differential equations.  The methods discussed
apply also to more complex cases, but care should be taken to ensure
that the solutions obtained are acceptable.

As an example of a hyperbolic equation we consider the advection
equation
%
\begin{equation}
  \pdv{u}{t} + v \pdv{u}{x} = 0 , \qquad a \le x, \quad v > 0 .
  \label{PDE:eq:advect}
\end{equation}
%
This equation represents a function propagating in the positive
$x$-direction with speed $v$.  In order for the equation to have a
unique solution we must specify an initial condition, $u(x,0) = F(x)$
and a boundary condition at $x=a$, $u(a,t) = G(t)$.  Note that the
domain is unbounded in the positive $x$-direction: numerically this
implies that the numerical solution of equation~(\ref{PDE:eq:advect})
is acceptable only if the effects of the right boundary are
negligible, i.e.\ only for the time taken for the signal to propagate
across the integration region and reach the right boundary.

\subsection{The forward-time centred-space method}

The most straightforward finite difference method for solving
hyperbolic partial differential equations would appear to be the
forward-time centred-space (FTCS) method.  Applied to the diffusion
equation this method is conditionally stable; however, when applied to
the convection equation this method is unconditionally unstable.  The
algorithm consists in replacing the time derivative in
equation~(\ref{PDE:eq:advect}) with its forward difference
approximation and the space derivative with the centred-difference
approximation.  We obtain:
%
\begin{equation*}
  \frac{u_{i}^{n+1}-u_{i}^{n}}{\delta} +
  v \frac{u_{i+1}^{n}-u_{i-1}^{n}}{2 h} = 0
\end{equation*}
%
where $\delta$ is the time step, $h$ is the space step, the spatial
grid is $x_i = i h$, with $i=0,1,\ldots,N+1$, the time grid is $t^n =
n \delta$ and $u_{i}^{n} = u(x_i,t^n)$.  Solving for $u_{i}^{n}$ we
obtain
%
\begin{equation}
  u_{i}^{n+1} = u_{i}^{n} - \frac{c}{2} (u_{i+1}^{n}-u_{i-1}^{n}) \, ,
  \label{PDE:eq:hypFTCS}
\end{equation}
%
where $c = u \delta/h$ is the \textit{convection number}.  The
stability analysis of this equation shows that the solution
of~(\ref{PDE:eq:hypFTCS}) is $u_{\ell}^{k} = \exp(\tj \alpha \ell h)
q^k$, with
%
\begin{equation*}
  q = 1 - \tj c \sin(\alpha h) ,
\end{equation*}
%
where $\tj = \sqrt{-1}$.  From this we obtain that
%
\begin{equation*}
  |q| = \sqrt{1 + c^2 \sin^2(\alpha h)} > 1 \, \, \forall c \in \bR .
\end{equation*}
%
Hence the FTCS method is unconditionally unstable.

\subsection{The Lax method}

Lax (1954) proposed a modification to the FTCS method for the
convection equation that yields a conditionally stable method.  In
that modification, the value $u_{i}^{n}$ in the finite difference
approximation of the time derivative is replaced by the average of its
values at the neighbouring points:
%
\begin{equation}
  u_{i}^{n+1} = \frac{1}{2} (u_{i+1}^{n} + u_{i-1}^{n}) -
  \frac{c}{2} (u_{i+1}^{n}-u_{i-1}^{n}) \, ,
  \label{PDE:eq:hypLax}
\end{equation}
%
This method is not consistent in general.  It can be shown that the
partial differential equation that is represented by
equation~(\ref{PDE:eq:hypLax}) is
%
\begin{equation}
  \pdv{u}{t} + v \pdv{u}{x} = \frac{1}{2}
  \left ( \frac{h^2}{\delta} - v^2 \delta \right )
  \pdv[2]{u}{x} +
  \frac{1}{3} \left ( v h^2 - v^3 \delta^2 \right ) \pdv[3]{u}{x} +
  \ldots \, ,
  \label{PDE:eq:LaxEquiv}
\end{equation}
%
where $h$ and $\delta$ are the space and time discretisation steps.
From this equation we can see that as $h$ and $\delta$ tend to zero
the first term on the right hand side does not vanish: on the
contrary, it is undetermined.  However, if the two discretisation
steps tend to zero so that their ratio is constant, for example at the
value $\beta = h / \delta$, then equation~(\ref{PDE:eq:hypLax})
approaches the equation
%
\begin{equation}
  \pdv{u}{t} + v \pdv{u}{x} = \frac{1}{2}
  \left ( \beta h - v^2 \delta \right )
  \pdv[2]{u}{x}
  \label{eq:PDE:LaxMDE}
\end{equation}
%
as $h$ and $\delta$ tend to zero.  The equation~(\ref{eq:PDE:LaxMDE})
is a parabolic convection-diffusion equation.  Substituting the
convection number $c=v \delta/h$ into equation~(\ref{eq:PDE:LaxMDE})
we obtain
%
\begin{equation}
  \pdv{u}{t} + v \pdv{u}{x} = \frac{1}{2} v h
  \left ( \frac{1}{c} - c \right ) \pdv[2]{u}{x}
  \label{eq:PDE:LaxMDE1}
\end{equation}
%
In other words if $c \neq 1$ the Lax approximation introduces some
numerical diffusion in the model that is to be integrated.  This
effect is common to most finite difference schemes: it is normally not
important if the original equation includes a diffusive term.  It may,
however, give completely false results if no diffusion terms are
present in the original equation.  In the case of the advection
equation the effect of the numerical diffusion is to smooth the
signals as they propagate and decrease their amplitude.

Another effect of the numerical diffusion of the Lax algorithm is to
make it conditionally stable.  The Von Neumann stability analysis
shows that the method is stable if
%
\begin{equation}
  c = \frac{v \delta}{h} \le 1 .
  \label{eq:PDE:hyp:CFL}
\end{equation}
%
Comparing this result with equation~(\ref{eq:PDE:LaxMDE1}) we see that
the scheme is stable if numerical diffusion is present.
Equation~(\ref{eq:PDE:hyp:CFL}) is called the Courant-Friedrichs-Lewy
stability criterion.  It states that the numerical speed of
propagation $v_{num} = h/ \delta$ must be greater than or equal to the
physical speed of propagation $v$.

\subsubsection{Derivation of Equation~(\ref{PDE:eq:LaxEquiv})}

To obtain equation~(\ref{PDE:eq:LaxEquiv}) we Taylor expand
$u_{i}^{n+1}$ and $u_{i\pm 1}^{n}$ around $u_{i}^{n}$ (indicated with
$u$ in what follows):
%
\begin{align*}
 u_{i}^{n+1} &= u + \delta \partial_t u +
                 \frac{\delta^2}{2}\partial_{t t} u +
                 \frac{\delta^3}{3!} \partial_{t t t} u + \order{\delta^4}, \\
 u_{i\pm 1}^{n} &= u + h \partial_x u \pm
                 \frac{h^2}{2}\partial_{x x} u \pm
                 \frac{h^3}{3!} \partial_{x x x} u + \order{h^4} .
\end{align*}
%
Substituting into equation~(\ref{PDE:eq:hypLax}) and dividing through
by $\delta$ we obtain
%
\begin{equation}
  \partial_t u + v \partial_x u =
  -\frac{\delta}{2} \partial_{t t} u
  -\frac{\delta^2}{6} \partial_{t t t} u
  + \frac{h^2}{2 \delta} \partial_{x x} u
  - \frac{v h^2}{6} \partial_{x x x}u + \order{\delta^3} + \order{h^3}
  \label{eq:PDE:14}
\end{equation}
%
We now wish to replace the derivatives with respect to time with
derivatives respect to space using this relation.  At first order in
$\delta$ we have that [we indicate with $\order{\delta^n}$ terms that are
of the order of $n$ in either $\delta$ or $h$]:
%
\begin{equation}
  \partial_t u = - v \partial_x u + \order{\delta}  \label{PDE:eq:16}
\end{equation}
%
so that
%
\begin{equation}
  \partial_{t t} u = - v \partial_x \partial_t u + \order{\delta} =
  v^2 \partial_{x x} u + \order{\delta}
  \label{PDE:eq:15}
\end{equation}
%
This is not accurate enough to be replaced into
equation~(\ref{eq:PDE:14}): we need to obtain an expression correct up
to second order in the space and time steps.  Starting once again from
equation~(\ref{eq:PDE:14}) we have
%
\begin{equation*}
  \partial_t u = -v \partial_x u
  - \frac{\delta}{2} \partial_{t t} u + \order{\delta^2} .
\end{equation*}
%
Differentiating with respect to time on both sides and making use
of~(\ref{PDE:eq:16}) and~(\ref{PDE:eq:15}) we obtain:
%
\begin{align*}
 \partial_{t t} u &=  -v \partial_x \partial_t u
   - \frac{\delta}{2} \partial_t \partial_{t t} u + \order{\delta^2} \\
  &=  -v \partial_x \left ( -v \partial_x u
   - \frac{\delta}{2} \partial_{t t} u \right )
   - \frac{\delta}{2} \partial_t v^2 \partial_{x x} u + \order{\delta^2} \\
  &=  v^2 \partial_{x x} u + v \partial_x
        \left ( \frac{\delta}{2} v^2 \partial_{x x} u \right ) +
      v^3 \frac{\delta}{2} \partial_{x x x} u + \order{\delta^2} \\
  &=  v^2 \partial_{x x} u + \delta v^3 \partial_{x x x} u +
      \order{\delta^2}.
\end{align*}
%
Differentiating equation~(\ref{PDE:eq:15}) with respect to time we
obtain
%
\begin{equation*}
  \partial_{t t t} u = -v^3 \partial_{x x x} u + \order{\delta} ,
\end{equation*}
%
and substituing both these expression into equation~(\ref{eq:PDE:14})
we (finally) obtain equation~(\ref{PDE:eq:LaxEquiv}).

\subsection{Upwind methods}

A salient feature of hyperbolic equations is that they describe the
propagation of information.  In the case of the advection
equation~(\ref{PDE:eq:advect}) the information propagates from
negative to positive $x$ with speed $v$.  This type of information
propagation is referred to as \textit{upwind} propagation, since the
information comes from the direction from which the convection
velocity comes, that is, the upwind direction.  Finite difference
methods that account for the upwind influence are called
\textit{upwind} methods.

The simplest procedure for developing an upwind finite difference
equation is to replace the time derivative by the first-order
forward-difference approximation and the space derivative by the
first-order one-sided-difference approximation in the upwind
direction.  For $v > 0$ the finite difference upwind approximation of
equation~(\ref{PDE:eq:advect}) is
%
\begin{align}
  && \frac{u_{i}^{n+1}-u_{i}^{n}}{\delta} + v
  \frac{u_{i}^{n} - u_{i-1}^{n}}{h} &= 0 \\ \implies &&
  u_{i}^{n+1} &= u_{i}^{n} - c ( u_{i}^{n} - u_{i-1}^{n} ) ,
  \label{eq:PDE:Upw}
\end{align}
%
where $c=v \delta/h$ is the convection number.  The upwind method is
consistent, first order in time and space and stable if $c \le 1$,
i.e.\ if the convection number satisfies the Courant-Friedrichs-Lewy
condition.  As in the case of the Lax scheme numerical diffusion (and
dispersion) is present unless $c=1$.

\section*{Further reading}

Topics covered here are also covered in
\begin{itemize}
\item Chapters 12 and 13 of Linz \& Wang, \textit{Exploring Numerical
    Methods} (QA297 LIN),
\item Chapter 9 of Kincaid \& Cheney, \textit{Numerical Analysis}
  (QA297 KIN),
\item Parts II and III (especially chapters 13 and 14) of Iserles,
  \textit{A First Course in the Numerical Analysis of Differential
    Equations} (QA297 ISE).
\end{itemize}


%!TEX root = main.tex

\appendix
\onecolumn
\section*{Overview}
\begin{table}[H]
    \centering
    \hspace*{-1cm}\begin{tabular}{lllll}
    \toprule
    Name                     & Function $\varphi(x)$ & Range of Values & $\varphi'(x)$ & Used by \\\midrule %
    Sign function$^\dagger$  & $\begin{cases}+1 &\text{if } x \geq 0\\-1 &\text{if } x < 0\end{cases}$ & $\Set{-1,1}$                              & $0$                    & \cite{971754} \\
    \parbox[t]{2.6cm}{Heaviside\\step function$^\dagger$} & $\begin{cases}+1 &\text{if } x > 0\\0 &\text{if } x < 0\end{cases}$ & $\Set{0, 1}$  & $0$                       & \cite{mcculloch1943logical}\\
    Logistic function        & $\frac{1}{1+e^{-x}}$                           & $[0, 1]$                                                        & $\frac{e^x}{(e^x +1)^2}$  & \cite{duch1999survey} \\
    Tanh                     & $\frac{e^x - e^{-x}}{e^x + e^{-x}} = \tanh(x)$ & $[-1, 1]$                                                       & $\sech^2(x)$              & \cite{LeNet-5,Thoma:2014}\\
    \gls{ReLU}$^\dagger$           & $\max(0, x)$                                   & $[0, +\infty)$                                                  & $\begin{cases}1 &\text{if } x > 0\\0 &\text{if } x < 0\end{cases}$      & \cite{AlexNet-2012}\\
    \parbox[t]{2.6cm}{\gls{LReLU}$^\dagger$\footnotemark\\(\gls{PReLU})} & $\varphi(x) = \max(\alpha x, x)$                        & $(-\infty, +\infty)$                                             & $\begin{cases}1 &\text{if } x > 0\\\alpha &\text{if } x < 0\end{cases}$ & \cite{maas2013rectifier,he2015delving} \\
    Softplus                 & $\log(e^x + 1)$                                & $(0, +\infty)$                                       & $\frac{e^x}{e^x + 1}$    & \cite{dugas2001incorporating,glorot2011deep} \\
    \gls{ELU}                & $\begin{cases}x &\text{if } x > 0\\\alpha (e^x - 1) &\text{if } x \leq 0\end{cases}$ & $(-\infty, +\infty)$ & $\begin{cases}1 &\text{if } x > 0\\\alpha e^x &\text{otherwise}\end{cases}$ & \cite{clevert2015fast} \\
    Softmax$^\ddagger$       & $o(\mathbf{x})_j = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}$    & $[0, 1]^K$                                           & $o(\mathbf{x})_j \cdot \frac{\sum_{k=1}^K e^{x_k} - e^{x_j}}{\sum_{k=1}^K e^{x_k}}$         & \cite{AlexNet-2012,Thoma:2014}\\
    Maxout$^\ddagger$        & $o(\mathbf{x}) = \max_{x \in \mathbf{x}} x$                 & $(-\infty, +\infty)$                                 & $\begin{cases}1 &\text{if } x_i = \max \mathbf{x}\\0 &\text{otherwise}\end{cases}$          & \cite{goodfellow2013maxout}       \\
    \bottomrule
    \end{tabular}
    \caption[Activation functions]{Overview of activation functions. Functions
             marked with $\dagger$ are not differentiable at 0 and functions
             marked with $\ddagger$ operate on all elements of a layer
             simultaneously. The hyperparameters $\alpha \in (0, 1)$ of Leaky
             ReLU and ELU are typically $\alpha = 0.01$. Other activation
             function like randomized leaky ReLUs exist~\cite{xu2015empirical},
             but are far less commonly used.\\
             Some functions are smoothed versions of others, like the logistic
             function for the Heaviside step function, tanh for the sign
             function, softplus for ReLU.\\
             Softmax is the standard activation function for the last layer of
             a classification network as it produces a probability
             distribution. See \Cref{fig:activation-functions-plot} for a plot
             of some of them.}
    \label{table:activation-functions-overview}
\end{table}
\footnotetext{$\alpha$ is a hyperparameter in leaky ReLU, but a learnable parameter in the parametric ReLU function.}

\section*{Evaluation Results}
\glsunset{LReLU}
\begin{table}[H]
    \centering
    \begin{tabular}{@{\extracolsep{4pt}}lcccccc@{}}
    \toprule
    \multirow{2}{*}{Function} & \multicolumn{4}{c}{Single model}                                                    & \multicolumn{2}{c}{Ensemble of 10} \\\cline{2-3}\cline{4-5}\cline{6-7}
                   & \multicolumn{2}{c}{Training set}     &\multicolumn{2}{c}{Test set}                  & Training set         & Test set \\\midrule
    Identity       & \SI{66.25}{\percent} & $\boldsymbol{\sigma=0.77}$ &\SI{56.74}{\percent} & \textbf{$\sigma=0.51$} & \SI{68.77}{\percent} & \SI{58.78}{\percent}\\
    Logistic       & \SI{51.87}{\percent} & $\sigma=3.64$ &\SI{46.54}{\percent} & $\sigma=3.22$          & \SI{61.19}{\percent} & \SI{54.58}{\percent}\\
    Logistic$^-$   & \SI{66.49}{\percent} & $\sigma=1.99$ &\SI{57.84}{\percent} & $\sigma=1.15$          & \SI{69.04}{\percent} & \SI{60.10}{\percent}\\
    Softmax        & \SI{75.22}{\percent} & $\sigma=2.41$ &\SI{59.49}{\percent} & $\sigma=1.25$          & \SI{78.87}{\percent} & \SI{63.06}{\percent}\\
    Tanh           & \SI{67.27}{\percent} & $\sigma=2.38$ &\SI{55.70}{\percent} & $\sigma=1.44$          & \SI{70.21}{\percent} & \SI{58.10}{\percent}\\
    Softsign       & \SI{66.43}{\percent} & $\sigma=1.74$ &\SI{55.75}{\percent} & $\sigma=0.93$          & \SI{69.78}{\percent} & \SI{58.40}{\percent}\\
    \gls{ReLU}     & \SI{78.62}{\percent} & $\sigma=2.15$ &\SI{62.18}{\percent} & $\sigma=0.99$          & \SI{81.81}{\percent} & \SI{64.57}{\percent}\\
    \gls{ReLU}$^-$ & \SI{76.01}{\percent} & $\sigma=2.31$ &\SI{62.87}{\percent} & $\sigma=1.08$          & \SI{78.18}{\percent} & \SI{64.81}{\percent}\\
    Softplus       & \SI{66.75}{\percent} & $\sigma=2.45$ &\SI{56.68}{\percent} & $\sigma=1.32$          & \SI{71.27}{\percent} & \SI{60.26}{\percent}\\
    S2ReLU         & \SI{63.32}{\percent} & $\sigma=1.69$ &\SI{56.99}{\percent} & $\sigma=1.14$          & \SI{65.80}{\percent} & \SI{59.20}{\percent}\\
    \gls{LReLU}    & \SI{74.92}{\percent} & $\sigma=2.49$ &\SI{61.86}{\percent} & $\sigma=1.23$          & \SI{77.67}{\percent} & \SI{64.01}{\percent}\\
    \gls{PReLU}    & \textbf{\SI{80.01}{\percent}} & $\sigma=2.03$ &\SI{62.16}{\percent} & $\sigma=0.73$ & \textbf{\SI{83.50}{\percent}} & \textbf{\SI{64.79}{\percent}}\\
    \gls{ELU}      & \SI{76.64}{\percent} & $\sigma=1.48$ &\textbf{\SI{63.38}{\percent}} & $\sigma=0.55$ & \SI{78.30}{\percent} & \SI{64.70}{\percent}\\
    \bottomrule
    \end{tabular}
    \caption[Activation function evaluation results on CIFAR-100]{Training and
             test accuracy of adjusted baseline models trained with different
             activation functions on CIFAR-100. For \gls{LReLU}, $\alpha = 0.3$ was
             chosen.}
    \label{table:CIFAR-100-accuracies-activation-functions}
\end{table}

\begin{table}[H]
    \centering
    \setlength\tabcolsep{1.5pt}
    \begin{tabular}{@{\extracolsep{4pt}}lcccccccr@{}}
    \toprule
    \multirow{2}{*}{Function} & \multicolumn{4}{c}{Single model}              & \multicolumn{2}{c}{Ensemble of 10} & \multicolumn{2}{c}{Epochs}\\\cline{2-5}\cline{6-7}\cline{8-9}
                              & \multicolumn{2}{c}{Training set}     &\multicolumn{2}{c}{Test set}                  & Train                & Test                 & Range     & \multicolumn{1}{c}{Mean} \\\midrule
    Identity                  & \SI{87.92}{\percent} & $\sigma=0.40$ & \SI{84.69}{\percent} & $\sigma=0.08$         & \SI{88.59}{\percent} & \SI{85.43}{\percent} & \hphantom{0}92 -- 140 & 114.5\\%TODO: Really?
    Logistic                  & \SI{81.46}{\percent} & $\sigma=5.08$ & \SI{79.67}{\percent} & $\sigma=4.85$         & \SI{86.38}{\percent} & \SI{84.60}{\percent} & \hphantom{0}\textbf{58} -- \hphantom{0}\textbf{91}  & \textbf{77.3}\\
    Softmax                   & \SI{88.19}{\percent} & $\sigma=0.31$ & \SI{84.70}{\percent} & $\sigma=0.15$         & \SI{88.69}{\percent} & \SI{85.43}{\percent} & 124 -- 171& 145.8\\
    Tanh                      & \SI{88.41}{\percent} & $\sigma=0.36$ & \SI{84.46}{\percent} & $\sigma=0.27$         & \SI{89.24}{\percent} & \SI{85.45}{\percent} & \hphantom{0}89 -- 123 & 108.7\\
    Softsign                  & \SI{88.00}{\percent} & $\sigma=0.47$ & \SI{84.46}{\percent} & $\sigma=0.23$         & \SI{88.77}{\percent} & \SI{85.33}{\percent} & \hphantom{0}77 -- 119 & 104.1\\
    \gls{ReLU}                & \SI{88.93}{\percent} & $\sigma=0.46$ & \textbf{\SI{85.35}{\percent}} & $\sigma=0.21$         & \SI{89.35}{\percent} & \SI{85.95}{\percent} & \hphantom{0}96 -- 132 & 102.8\\
    Softplus                  & \SI{88.42}{\percent} & $\boldsymbol{\sigma=0.29}$ & \SI{85.16}{\percent} & $\sigma=0.15$         & \SI{88.90}{\percent} & \SI{85.73}{\percent} &            108 -- 143 & 121.0\\
    \gls{LReLU}               & \SI{88.61}{\percent} & $\sigma=0.41$ & \SI{85.21}{\percent} & $\boldsymbol{\sigma=0.05}$         & \SI{89.07}{\percent} & \SI{85.83}{\percent} & \hphantom{0}87 -- 117 & 104.5\\
    \gls{PReLU}               & \textbf{\SI{89.62}{\percent}} & $\sigma=0.41$ & \textbf{\SI{85.35}{\percent}} & $\sigma=0.17$& \textbf{\SI{90.10}{\percent}} & \SI{86.01}{\percent} & \hphantom{0}85 -- 111 & 100.5\\
    \gls{ELU}                 & \SI{89.49}{\percent} & $\sigma=0.42$ & \textbf{\SI{85.35}{\percent}} & $\sigma=0.10$         & \SI{89.94}{\percent} & \textbf{\SI{86.03}{\percent}} & \hphantom{0}73 -- 113 &  92.4\\
    \bottomrule
    \end{tabular}
    \caption[Activation function evaluation results on HASYv2]{Test accuracy of
             adjusted baseline models trained with different activation
             functions on HASYv2. For \gls{LReLU}, $\alpha = 0.3$ was chosen.}
    \label{table:HASYv2-accuracies-activation-functions}
\end{table}

\begin{table}[H]
    \centering
    \setlength\tabcolsep{1.5pt}
    \begin{tabular}{@{\extracolsep{4pt}}lcccccccr@{}}
    \toprule
    \multirow{2}{*}{Function} & \multicolumn{4}{c}{Single model}              & \multicolumn{2}{c}{Ensemble of 10} & \multicolumn{2}{c}{Epochs}\\\cline{2-5}\cline{6-7}\cline{8-9}
                              & \multicolumn{2}{c}{Training set}     &\multicolumn{2}{c}{Test set}                  & Train                & Test                 & Range     & \multicolumn{1}{c}{Mean} \\\midrule
    Identity                  & \SI{87.49}{\percent} & $\sigma=2.50$ & \SI{69.86}{\percent} & $\sigma=1.41$         & \SI{89.78}{\percent} & \SI{71.90}{\percent} & \hphantom{0}51 -- \hphantom{0}65  &  53.4\\
    Logistic                  & \SI{45.32}{\percent} & $\sigma=14.88$& \SI{40.85}{\percent} & $\sigma=12.56$        & \SI{51.06}{\percent} & \SI{45.49}{\percent} & \hphantom{0}38 -- \hphantom{0}93  &  74.6\\
    Softmax                   & \SI{87.90}{\percent} & $\sigma=3.58$ & \SI{67.91}{\percent} & $\sigma=2.32$         & \SI{91.51}{\percent} & \SI{70.96}{\percent} & 108 -- 150           & 127.5\\
    Tanh                      & \SI{85.38}{\percent} & $\sigma=4.04$ & \SI{67.65}{\percent} & $\sigma=2.01$         & \SI{90.47}{\percent} & \SI{71.29}{\percent} & 48 -- \hphantom{0}92 & 65.2\\
    Softsign                  & \SI{88.57}{\percent} & $\sigma=4.00$ & \SI{69.32}{\percent} & $\sigma=1.68$         & \SI{93.04}{\percent} & \SI{72.40}{\percent} & 55 -- 117            & 83.2\\
    \gls{ReLU}                & \SI{94.35}{\percent} & $\sigma=3.38$ & \SI{71.01}{\percent} & $\sigma=1.63$         & \SI{98.20}{\percent} & \SI{74.85}{\percent} & 52 -- \hphantom{0}98 & 75.5\\
    Softplus                  & \SI{83.03}{\percent} & $\sigma=2.07$ & \SI{68.28}{\percent} & $\sigma=1.74$         & \SI{93.04}{\percent} & \SI{75.99}{\percent} & 56 -- \hphantom{0}89 & 68.9\\
    \gls{LReLU}               & \SI{93.83}{\percent} & $\sigma=3.89$ & \SI{74.66}{\percent} & $\sigma=2.11$         & \SI{97.56}{\percent} & \SI{78.08}{\percent} & 52 -- 120 & 80.1\\
    \gls{PReLU}               & \SI{95.53}{\percent} & $\sigma=1.92$ & \SI{71.69}{\percent} & $\sigma=1.37$         & \SI{98.17}{\percent} & \SI{74.69}{\percent} & 59 -- 101 & 78.8\\
    \gls{ELU}                 & \SI{95.42}{\percent} & $\sigma=3.57$ & \SI{75.09}{\percent} & $\sigma=2.39$         & \SI{98.54}{\percent} & \SI{78.66}{\percent} & 66 -- \hphantom{0}72 & 67.2\\
    \bottomrule
    \end{tabular}
    \caption[Activation function evaluation results on STL-10]{Test accuracy of
             adjusted baseline models trained with different activation
             functions on STL-10. For \gls{LReLU}, $\alpha = 0.3$ was chosen.}
    \label{table:STL-10-accuracies-activation-functions}
\end{table}

\begin{table}[H]
    \centering
    \hspace*{-1cm}\begin{tabular}{lllll}
    \toprule
    Name                     & Function $\varphi(x)$ & Range of Values & $\varphi'(x)$ \\\midrule % & Used by
    Sign function$^\dagger$  & $\begin{cases}+1 &\text{if } x \geq 0\\-1 &\text{if } x < 0\end{cases}$ & $\Set{-1,1}$                              & $0$                    \\%& \cite{971754} \\
    \parbox[t]{2.6cm}{Heaviside\\step function$^\dagger$} & $\begin{cases}+1 &\text{if } x > 0\\0 &\text{if } x < 0\end{cases}$ & $\Set{0, 1}$  & $0$                       \\%& \cite{mcculloch1943logical}\\
    Logistic function        & $\frac{1}{1+e^{-x}}$                           & $[0, 1]$                                                        & $\frac{e^x}{(e^x +1)^2}$  \\%& \cite{duch1999survey} \\
    Tanh                     & $\frac{e^x - e^{-x}}{e^x + e^{-x}} = \tanh(x)$ & $[-1, 1]$                                                       & $\sech^2(x)$              \\%& \cite{LeNet-5,Thoma:2014}\\
    \gls{ReLU}$^\dagger$           & $\max(0, x)$                                   & $[0, +\infty)$                                                  & $\begin{cases}1 &\text{if } x > 0\\0 &\text{if } x < 0\end{cases}$      \\%& \cite{AlexNet-2012}\\
    \parbox[t]{2.6cm}{\gls{LReLU}$^\dagger$\footnotemark\\(\gls{PReLU})} & $\varphi(x) = \max(\alpha x, x)$                        & $(-\infty, +\infty)$                                             & $\begin{cases}1 &\text{if } x > 0\\\alpha &\text{if } x < 0\end{cases}$ \\%& \cite{maas2013rectifier,he2015delving} \\
    Softplus                 & $\log(e^x + 1)$                                & $(0, +\infty)$                                       & $\frac{e^x}{e^x + 1}$    \\%& \cite{dugas2001incorporating,glorot2011deep} \\
    \gls{ELU}                & $\begin{cases}x &\text{if } x > 0\\\alpha (e^x - 1) &\text{if } x \leq 0\end{cases}$ & $(-\infty, +\infty)$ & $\begin{cases}1 &\text{if } x > 0\\\alpha e^x &\text{otherwise}\end{cases}$ \\%& \cite{clevert2015fast} \\
    Softmax$^\ddagger$       & $o(\mathbf{x})_j = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}$    & $[0, 1]^K$                                           & $o(\mathbf{x})_j \cdot \frac{\sum_{k=1}^K e^{x_k} - e^{x_j}}{\sum_{k=1}^K e^{x_k}}$          \\%& \cite{AlexNet-2012,Thoma:2014}\\
    Maxout$^\ddagger$        & $o(\mathbf{x}) = \max_{x \in \mathbf{x}} x$                 & $(-\infty, +\infty)$                                 & $\begin{cases}1 &\text{if } x_i = \max \mathbf{x}\\0 &\text{otherwise}\end{cases}$          \\%& \cite{goodfellow2013maxout}       \\
    \bottomrule
    \end{tabular}
    \caption[Activation functions]{Overview of activation functions. Functions
             marked with $\dagger$ are not differentiable at 0 and functions
             marked with $\ddagger$ operate on all elements of a layer
             simultaneously. The hyperparameters $\alpha \in (0, 1)$ of Leaky
             ReLU and ELU are typically $\alpha = 0.01$. Other activation
             function like randomized leaky ReLUs exist~\cite{xu2015empirical},
             but are far less commonly used.\\
             Some functions are smoothed versions of others, like the logistic
             function for the Heaviside step function, tanh for the sign
             function, softplus for ReLU.\\
             Softmax is the standard activation function for the last layer of
             a classification network as it produces a probability
             distribution. See \Cref{fig:activation-functions-plot} for a plot
             of some of them.}
    \label{table:activation-functions-overview}
\end{table}
\footnotetext{$\alpha$ is a hyperparameter in leaky ReLU, but a learnable parameter in the parametric ReLU function.}

\begin{figure}[ht]
    \centering
    \begin{tikzpicture}
        \definecolor{color1}{HTML}{E66101}
        \definecolor{color2}{HTML}{FDB863}
        \definecolor{color3}{HTML}{B2ABD2}
        \definecolor{color4}{HTML}{5E3C99}
        \begin{axis}[
            legend pos=north west,
            legend cell align={left},
            axis x line=middle,
            axis y line=middle,
            x tick label style={/pgf/number format/fixed,
                                /pgf/number format/fixed zerofill,
                                /pgf/number format/precision=1},
            y tick label style={/pgf/number format/fixed,
                                /pgf/number format/fixed zerofill,
                                /pgf/number format/precision=1},
            grid = major,
            width=16cm,
            height=8cm,
            grid style={dashed, gray!30},
            xmin=-2,     % start the diagram at this x-coordinate
            xmax= 2,     % end   the diagram at this x-coordinate
            ymin=-1,     % start the diagram at this y-coordinate
            ymax= 2,     % end   the diagram at this y-coordinate
            xlabel=x,
            ylabel=y,
            tick align=outside,
            enlargelimits=false]
          \addplot[domain=-2:2, color1, ultra thick,samples=500] {1/(1+exp(-x))};
          \addplot[domain=-2:2, color2, ultra thick,samples=500] {tanh(x)};
          \addplot[domain=-2:2, color4, ultra thick,samples=500] {max(0, x)};
          \addplot[domain=-2:2, color4, ultra thick,samples=500, dashed] {ln(exp(x) + 1)};
          \addplot[domain=-2:2, color3, ultra thick,samples=500, dotted] {max(x, exp(x) - 1)};
          \addlegendentry{$\varphi_1(x)=\frac{1}{1+e^{-x}}$}
          \addlegendentry{$\varphi_2(x)=\tanh(x)$}
          \addlegendentry{$\varphi_3(x)=\max(0, x)$}
          \addlegendentry{$\varphi_4(x)=\log(e^x + 1)$}
          \addlegendentry{$\varphi_5(x)=\max(x, e^x - 1)$}
        \end{axis}
    \end{tikzpicture}
    \caption[Activation functions]{Activation functions plotted in $[-2, +2]$.
             $\tanh$ and ELU are able to produce negative numbers. The image of
             ELU, ReLU and Softplus is not bound on the positive side, whereas
             $\tanh$ and the logistic function are always below~1.}
    \label{fig:activation-functions-plot}
\end{figure}

\glsreset{LReLU}
\twocolumn


\subsection{Tangent space and tangent vectors}

Take a topological space: can all subsets in the toplogy be mapped to \(n\) dimensional space? if so, manifold

For this we need openness: a graph for example isn't open and so isn't a manifold

We also need the same number of dimensions at each point

Isn't always the case. eg two circles conneceted by a line is not a manifold. it's 2d in circles, 1d on line (and 3d at connections)

We have a homeomorphism from each point in the toplogy to an n dimensional coordinate system

We also have homeomorphisms of transformation maps, between different points on the topology

The vector space from the homeomorphism is tangent to the manifold at that point. the set of all tangents forms a tangent space

Interior: \(M\); boundry \(\delta M\)
Tangent on a manifold:

The tangent space of manifold \(M\) at point \(p\) is denoted \(TM_p\).

If we have a normal field

\(v=v^ie_i\)

Then we can differentiate wrt a direction \(x\).

\(\dfrac{\delta }{\delta x}v=\dfrac{\delta }{\delta x}v^ie_i\)

\(\dfrac{\delta }{\delta x}v=e_i\dfrac{\delta v^i}{\delta x}\)

Because the basis does not change.

If the basis does change we instead have:

\(\dfrac{\delta }{\delta x}v=\dfrac{\delta }{\delta x}v^ie_i\)

\(\dfrac{\delta }{\delta x}v=e_i\dfrac{\delta v^i}{\delta x}+v^i\dfrac{\delta e_i }{\delta x}\)

General point. basis can vary across manifold

After this basis diff

\subsubsection{Tangent space as vector bundle}

\subsubsection{Christoffel symbols (page)}

Christoffel symbols are connections.

\subsubsection{The torsion tensor (own page)}

Torsion tensor is

\(T_{jk}^i=\Gamma^i_{jk}-\Gamma^i_{kj}\)

If torsion is \(0\), then the connection is symmetric.

\subsubsection{Basis of tangent space}

We can use as the basis for tangent space:

\(\{\dfrac{(\delta }{\delta x^1})_p,(\dfrac{\delta }{\delta x^2})_p,...\}\)

This means we can write a tangent vector as:

\(u=u^i(\dfrac{\delta }{\delta x^i})_p\)

\subsubsection{Basis of contant space}

We can use as the basis for the contangent space:

\(\{dx^1,dx^2,...\}\)

\subsubsection{Metric on the tangent space (to Riemann)}

\subsubsection{Basis of metric (to Riemann)}

The metric depends on the basis too:

\(g_{ij}(p)=g((\dfrac{\delta }{\delta x^i})_p,(\dfrac{\delta }{\delta x^j})_p)\)

The metric on two tangent vectors is defined on the components.

\(g=g_{ij}(p)u^iv^j\)


\section{Calculer (2 points)}

\begin{questions}
	\question[1] $\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2$
	\begin{solution}
		\begin{eqnarray*}
			\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2 &=& \dfrac{1}{4} + \dfrac{5\times 2}{2 \times 2} - \dfrac{3}{4} + \dfrac{2 \times 4}{1 \times 4} \\
			\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2 &=& \dfrac{1}{4} + \dfrac{10}{4} - \dfrac{3}{4} + \dfrac{8}{4} \\
			\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2 &=& \dfrac{1 + 10 - 3 + 8}{4} \\
			\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2 &=& \dfrac{16}{4} \\
			\dfrac{1}{4} + \dfrac{5}{2} - \dfrac{3}{4} + 2 &=& 4 \\
		\end{eqnarray*}
	\end{solution}

	\question[1] $4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12}$
	\begin{solution}
		\begin{eqnarray*}
			4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12} &=& \dfrac{4 \times 12}{1 \times 12} - \dfrac{7\times 4}{3 \times 4} - \dfrac{3 \times 3}{4 \times 3} + \dfrac{7}{12} \\
			4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12} &=& \dfrac{48}{12} - \dfrac{28}{12} - \dfrac{9}{12} + \dfrac{7}{12} \\
			4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12} &=& \dfrac{48 - 28 - 9 + 7}{12} \\
			4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12} &=& \dfrac{18}{12} \\
			4 - \dfrac{7}{3} - \dfrac{3}{4} + \dfrac{7}{12} &=& \dfrac{3}{2} \\
		\end{eqnarray*}
	\end{solution}
\end{questions}


\chapter{Additional Proofs}

In this appendix we provide proofs of incidental statements made throughout this thesis.
These proofs are provided in the interest of completeness.

In the introduction, it was stated that the word problem for any finitely presented group is recursively enumerable.
This is a well-known fact which we prove as follows.

\begin{proposition}\label{prop:appendix/wp-fp}
	Finitely presented groups have recursively enumerable word problems.
\end{proposition}

\begin{proof}
	Let $G$ be a group with presentation $\left\langle X \mid R \right\rangle$ where $X$ and $R$ are both finite.
	We then see that a word $w \in X^*$ is in the word problem if and only if we have
	\[
		w
		=_{F_X}
		\prod_{i=0}^n  u_i r_i^{\delta_i} u_i^{-1}
	\]
	for some $n \in \mathbb{N}$ where each $u_i \in X^*$, $\delta_j \in \{-1,1\}$ and $r_i \in R$.
	Notice here that `$=_{F_X}$' denotes that the left and right-hand sides are the same word after free-reduction is performed, that is, they are equivalent if viewed as elements of the free group.

	We may then construct a Turing machine $M$ which takes a word $w \in X^*$ as input, then iterates through the set of all finite products $\prod_{i=0}^n  u_i r_i^{\delta_i} u_i^{-1}$.
	At each iteration, the machine should compare the word $w$ and the result of the product.
	The machine then terminate and accepts only if the two words are equal.

	We see that the machine $M$ accepts a word $w \in X^*$ if and only if it lies within the word problem $\WP_X$, that is, membership to the word problem $\WP_X$ is \emph{semi-decidable}.

	It is well known that a problem is semi-decidable if and only if the set of all accepted words (in this case the word problem) is recursively enumerable.
	This can be proven by constructing a machine which checks all words in parallel using a technique known as \emph{dovetailing} (see Theorem 20.8 on p.~441 of \cite{rich2007} for a proof of this fact).
\end{proof}

In \cref{sec:linear-constraints} we gave an example of a linearly constrained language, and provided its generating function.
In the following we show that this generating function is holonomic by explicitly constructing a system of linear differential equations which it satisfies.

\begin{proposition}\label{prop:appendix/holonomic-function}
	The multivariate power series
	\[
		f(x,y,z) =
		\sum_{n \in \mathbb{N}}
		\frac{(3n)!}{(n!)^3}
		x^n y^n z^n
	\]
	satisfies the differential equations
	\begin{equation}\label{eq:appendix/holonomic-differential-eq}
	\left.
	\begin{aligned}
		(x^2 - 27 x^3 y z) \partial_x^2 f(x,y,z)
		+ (x - 54 x^2 y z) \partial_x f(x,y,z)
		- 6 x y z \, f(x,y,z)
		&= 0
		\\
		(y^2 - 27 x y^3 z) \partial_y^2 f(x,y,z)
		+ (y - 54 x y^2 z) \partial_y f(x,y,z)
		- 6 x y z \, f(x,y,z)
		&= 0
		\\
		(z^2 - 27 x y z^3) \partial_z^2 f(x,y,z)
		+ (z - 54 x y z^2) \partial_z f(x,y,z)
		- 6 x y z \, f(x,y,z)
		&= 0.
	\end{aligned}
	\right\}
	\end{equation}
	Thus, $f(x,y,z)$ is holonomic.
\end{proposition}

\begin{proof}
	Notice that the system of differential equations in \eqref{eq:appendix/holonomic-differential-eq} is equivalent to
	\begin{equation}\label{eq:appendix/holonomic-differential-eq2}
	\left.
	\begin{aligned}
		x^2 \partial_x^2 f(x,y,z)
		+ x \partial_x f(x,y,z)
		-27 x^2 \partial_x^2 (x y z \, f(x,y,z))
		- 6 x y z \, f(x,y,z)
		&= 0
		\\
		y^2 \partial_y^2 f(x,y,z)
		+ y \partial_y f(x,y,z)
		-27 y^2 \partial_y^2 (x y z \, f(x,y,z))
		- 6 x y z \, f(x,y,z)
		&= 0
		\\
		z^2 \partial_z^2 f(x,y,z)
		+ z \partial_z f(x,y,z)
		-27 z^2 \partial_z^2 (x y z \, f(x,y,z))
		- 6 x y z \, f(x,y,z)
		&= 0.
	\end{aligned}
	\right\}
	\end{equation}
	This can be shown using the \emph{product rule} of differentiation.

	Let
	\[
		f(x,y,z) =
		\sum_{n \in \mathbb{N}}
		\frac{(3n)!}{(n!)^3}
		x^n y^n z^n,
	\]
	then
	\begin{align*}
			x \partial_x f(x,y,z)
			&=
			\sum_{n=1}^\infty
			\frac{(3n)!}{(n!)^3}
			\,n\,
			x^n y^n z^n,
		\\
			x^2 \partial_x^2 f(x,y,z)
			&=
			\sum_{n=1}^\infty
			\frac{(3n)!}{(n!)^3}
			\,n(n-1)\,
			x^n y^n z^n,
		\\
			xyz \, f(x,y,z)
			&=
			\sum_{n=1}^\infty
			\frac{(3(n-1))!}{((n-1)!)^3}
			\,
			x^n y^n z^n,\text{ and}
		\\
			x^2 \partial_x^2 (xyz \, f(x,y,z))
			&=
			\sum_{n=1}^\infty
			\frac{(3(n-1))!}{((n-1)!)^3}
			\,n(n-1)\,
			x^n y^n z^n.
	\end{align*}

	We then see that
	\[
		x^2 \partial_x^2 f(x,y,z)
		+ x \partial_x f(x,y,z)
		-27 x^2 \partial_x^2 (x y z \, f(x,y,z))
		- 6 x y z \, f(x,y,z)
		=
		\sum_{n=1}^\infty c_n x^n y^n z^n
	\]
	where each
	\begin{align*}
		c_n
		&=
		n^2
		\frac{(3n)!}{(n!)^3}
		-
		(27n^2 - 27n + 6)
		\frac{(3(n-1))!}{((n-1)!)^3}
		\\
		&=
		\frac{1}{n}
		\left[
			n^3
			\frac{(3n)!}{(n!)^3}
			-
			3n(3n-1)(3n-2)
			\frac{(3(n-1))!}{((n-1)!)^3}
		\right].
	\end{align*}
	Moreover, we see that each $c_n = 0$ as
	\[
		\frac{(3n)!}{(n!)^3}
		=
		\frac{3n(3n-1)(3n-2)}{n^3}
		\cdot
		\frac{(3(n-1))!}{((n-1)!)^3}.
	\]
	Thus, we have
	\[
		x^2 \partial_x^2 f(x,y,z)
		+ x \partial_x f(x,y,z)
		-27 x^2 \partial_x^2 (x y z \, f(x,y,z))
		- 6 x y z \, f(x,y,z)
		=
		0.
	\]
	The proofs of the other two differential equations in \eqref{eq:appendix/holonomic-differential-eq2} are the same.
\end{proof}


\subsection{Power law}

\(P(X)=\dfrac{\alpha -1}{a}(\dfrac{x}{a})^{-\alpha }\)

Where \(a\) is the lower bound.

\(P(X)=0\) for \(X<a\).

\subsubsection{Moments of the power law}

\(E[X^m]=\dfrac{\alpha - 1}{\alpha -1 -m }a\)

If \(m\ge \alpha -1 \) then this is not well defined.

Higher order moments, such that the variance, cannot be identified.


\section{First Order Unification}

 Unification is widely used in automated reasoning, logic programming
 and programming language type system implementation.

\subsection{First-order terms}

Given a set of \textit{variable symbols} $X = \{x,y,z,\cdots \}$ , a
set of distinct \textit{constant symbols} $ C = \{ a,b,c,\cdots \}$ a
set of distinct \textit{function symbols} $F = \{f,g,h, \cdots
\}$. Term is defined as any expression that can be generated by a
finite number of applications of the following rules:

\begin{enumerate}
\item Basis: any variable , and also any constant  is a term
\item Induction: if $t_1, \cdots, t_k$ are terms then $f(t_1,\cdots,
  t_k)$ is term for finite $k, 0 < k$.
\end{enumerate}

For brevity, the constant symbols are regarded as function symbols
taking zero arguments, the induction rule is relaxed to allow terms
with zero arguments , and $a()$ is regarded as syntactically equal to
$a$ . Mathematicians fix the arity of a function symbol while
typically in syntactic unification problems, a function symbol may
have any number of arguments, and possibly may have different number
of arguments in different arguments.

First order unification is the syntactic unification of first-order
terms, while higher order unification is the unification of
higher-order terms. First-order unification is especially widely used
in logic programming, programming language type system design,
especially in type inferencing algorithms based on the Hindley-Milner
type system, and automated reasoning. Higher-order unification is also
widely used in proof assistants, for example Isabelle and Twelf, and
restricted forms of higher-order unification (higher-order pattern
unification) are used in some programming language implementations,
such as lambdaProlog, as higher-order patterns are expressive, yet
their associated unification procedure retains theoretical properties
closer to first-order unification. Semantic unification is widely used
in SMT solvers and term rewriting algorithms.

It is well known that if two terms have a unfier, they also have a
\textit{most general unifier}
\subsection{Substitution}

A substitution is defined as a finite set of mappings from variables
to terms where each mapping must be \textit{unique}, because mapping
the same variable to two different terms would be ambiguous. A
substitution may be applied to a term u and is written $u\{ x_0
\rightarrow t_0, \cdots, x_k \rightarrow t_k \}$, which means
\textit{simultaneously} replace every occurrence of each variable
$x_i$ in the term $u$ with the term $t_i$ for $0 \le i \le k $. E.g.
$f(x,a,g(z),y)\{x \rightarrow h(a,y), z \rightarrow b \} = f(h(a,y), a
, g(b),y)$. A unifier U is called a \textit{most general unifer} for
$L$, if $\forall$ $U'$ of $L$, $\exists$ substitution $s$, $subst(U',L) =
subst(s, subst(U,L))$ .

\subsection{Unification in Various areas}

Unification in Prolog
\begin{itemize}
\item A variable which is uninstantiated-i.e. no previous unifications
  were performed on it-can be unified with an atom, a term, or another
  uninstantiated variable, thus effectively becoming its alias. In
  many modern Prolog dialects and in first-order logic, a variable
  cannot be unified with a term that contains it; this is the so
  called occurs check.

\item Two atoms can only be unified if they are identical.

\item Similarly, a term can be unified with another term if the top
  function symbols and arities of the terms are identical and if the
  parameters can be unified simultaneously. Note that this is a
  recursive behavior.

\end{itemize}
Unification in HM type inference

\begin{itemize}
\item Any type variable unifies with any type expression, and is
  instantiated to that expression. A specific theory might restrict
  this rule with an occurs check.
\item Two type constants unify only if they are the same type
\item Two type constructions unify only if they are applications of
  the same type constructor and all of their component types
  recursively unify.
\end{itemize}

Due to its declarative nature, the order in a sequence of unifications
is (usually) unimportant.

\subsection{Occurs check}

If there is ever an attempt to unify a variable $x$ with a term with a
function symbol containing $x$ as a strict subterm $x=f(\cdots,
x,\cdots)$, $x$ would have to be an infinite term, which contradicts
the strict definition of terms that requires a \textit{finite} number
of applications of the induction rule. e.g $x=f(x)$ does not represent
a strictly valid term.

\subsection{Unification Examples}

\begin{tabular}{|c|c|c|}
\hline
Prolog Notation & Unify Substitution & Explaination \\
\hline

f(X)=f(Y) & X $\rightarrow$ Y & X and Y are aliased \\
f(g(X),X) = f(Y,a) & X $\rightarrow$ a, Y $\rightarrow$ g(a) & \\
X = f(X) & should fail & enforced by \textit{occurs check} \\
X=Y,Y=a & X $\rightarrow$ a , Y $\rightarrow$ a & \\
\hline
\end{tabular}

\subsection{Algorithm}
Unification algorithms can either perform occurs checks as soon as a
variable has to be unified with a non-variable term or postpone all
occurs checks to the end. The first kind check is \textit{inline} and
the second is called \textit{post} occurs checks. The second performs well.
\begin{eqnarray}
G \cup  \{t  \sim t\}  \Rightarrow G \\
G \cup \{ f(s_0, \cdots ,s_k )  \sim f(t_0, \cdots, t_k)\}  \Rightarrow
G \cup \{s_0 \sim t_0, \cdots, s_k \sim t_k \}\\
G \cup \{f(s_0, \cdots, s_k) \sim g(t_0,\cdots, t_m ) \} \Rightarrow
\perp if f \neq g \vee  k \neq m\\
G \cup \{ x \sim t \} \Rightarrow
G{x\rightarrow t} \cup {x \sim t} if x \in Vars(G) \wedge x \notin Vars(t) \\
G \cup \{ x \sim f(s_0, \cdots, s_k) \} \Rightarrow \perp if x \in
Vars(f(s_0, \cdots, s_k ) )
\end{eqnarray}
The set of variables in a term $t$ is written as $Vars(t)$, and the et
of variables in all terms on \verb|LHS| or \verb|RHS| of potential
equations in a problem \verb|G| is written as \verb|Vars(G)|. For
brevity, constant symbols are regarded as function symbols having zero
arguments.

Robinson Algorithm
\begin{verbatim}
Function robOccursCheck(x,t,delta)
INPUT:
  Variable x, term t, substitution delta
OUTPUT:
  false or true
BEGIN
  let S be a stack, initially containing t
  while (S is non-empty) do
    t := pop(S);
    foreach variable y in t do
      if x = y then
        return false
      if y is bound in delta  then
        push y <* delta onto S
    od
  od
  return true
END

FUNCTION ROB(s,t)
INPUT:
  Term s and t
OUTPUT:
  substitution or failure
BEGIN
  let S be an empty stack of pairs of terms, initially containing
(s,t)
  let delta be the empty substitution
  while (S is non-empty) do
     (s,t) := pop (S);
     while (s is a variable bound by delta) s := s <* delta
     while (t is a variable bound by detal) t := t <* delta
     if s <> t then
       case(s,t) of
         (x,y) => add x -> y to delta
         (x,u) =>
               if robOccursCheck(x,u,delta)
               then
                 add x -> u to delta
                 apply x -> u to each term in delta
               else failure
         (u,x) =>
               if robOccursCheck(x,u,delta)
               then
                 add x -> u to delta
                 apply x -> u to each term in data
               else failure

         (f(s1,..,sn),f(t1,...,tn)) =>
               push (s1,t1), ... , (sn,tn) onto S
      end
  od
  return delta
END
\end{verbatim}


\chapter{Naive Set Theory}
The concept of \href{https://en.wikipedia.org/wiki/set_theory}{set theory} has arisen towards the end of the 19th century
from an effort to put mathematics on a solid foundation.  The creation of a solid foundation was considered necessary as
the concept of infinity increasingly worried mathematicians.

The essential parts of set theory have been defined by \href{https://de.wikipedia.org/wiki/Georg_Cantor}{Georg
  Cantor} (1845 -- 1918). The first definition of the concept of a set was approximately as follows
\cite{cantor:1895}:

\begin{center}
\colorbox{red}{\framebox{\colorbox{yellow}{
\begin{minipage}{0.85\linewidth}
  A ``set'' is a \blue{well-defined} collection $M$ of certain objects $x$ of our perception or our thinking.
\end{minipage}}}}
\end{center}
\vspace*{0.2cm}

\noindent
Here, the attribute ``\blue{well-defined}'' expresses the fact that for a given quantity $M$ and an object $x$ we have
to be able to decide whether the object $x$ belongs to the set $M$.  If $x$ belongs to $M$, then $x$ is called an
\blue{element} of the set $M$ and we write this as
\\[0.2cm]
\hspace*{1.3cm}
$x \in M$.
\\[0.2cm]
The symbol ``$\in$'' is therefore used in set theory as a binary predicate symbol.  We use infix notation when
using this symbol, that is we write $x \in M$ instead of ${\in}(x, M)$.
Slightly abbreviated we can define the notion of a set as follows:
\\[0.2cm]
\hspace*{1.3cm}
\textsl{A set is a \blue{well-defined} collection of elements}.
\\[0.2cm]
To mathematically understand the concept of a \blue{well-defined collection of elements},
Cantor introduced the so-called \blue{axiom of comprehension}.
We can formalize this axiom as follows:  If $p(x)$ a \blue{property} that
an object $x$ can have, we can define the set $M$ of all objects that have this
property.  Therefore, the set $M$ can be defined as
\\[0.2cm]
\hspace*{1.3cm}
$M := \{ x \;|\; p(x) \}$
\\[0.2cm]
and we read this definition as ``$M$ is the set of all $x$ such that $p(x)$ holds''.
Here, a property $p(x)$ is just a formula in which the variable $x$ happens to appear.
We illustrate the axiom of comprehension by an example: If $\mathbb{N}$ is
the set of natural numbers, then we can define the set of all even numbers
via the property \\[0.2cm]
\hspace*{1.3cm} $p(x) \;:=\; (\exists y\in \mathbb{N}: x = 2 \cdot y)$. \\[0.2cm]
Using this property, the set of even numbers can be defined as \\[0.2cm]
\hspace*{1.3cm} $\{ x \;|\; \exists y\in \mathbb{N}: x = 2 \cdot y \}$.

Unfortunately, the unrestricted use of the axiom of comprehension leads to serious problems.  To give an
example, let us consider the property of a set to \underline{not} contain itself.  Therefore, we define
\\[0.2cm]
\hspace*{1.3cm}
 $p(x) := \neg(x \in x)$
\\[0.2cm]
and further define the set $R$ as follows:
\\[0.2cm]
\hspace*{1.3cm}
$R := \{ x \;|\; \neg (x \in x) \}$.
\\[0.2cm]
Intuitively, we might expect that no set can contain itself.  However, things turn out to be more complicated.
Let us try to check whether the set $R$ contains itself.  We have
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}{cl}
                  & R \in R \\[0.2cm]
  \Leftrightarrow & R \in \bigl\{ x \;|\; \neg (x \in x) \bigr\} \\[0.2cm]
  \Leftrightarrow & \neg (R \in R).
\end{array}
$
\\[0.2cm]
So we have shown that
\\[0.2cm]
\hspace*{1.3cm}
$R \in R \;\Leftrightarrow\; \neg(R \in R)$
\\[0.2cm]
holds.  Obviously, this is a contradiction.  As a way out, we can only conclude that the expression \\[0.2cm]
\hspace*{1.3cm} $\{ x \mid \neg (x \in x) \}$ \\[0.2cm]
does not define a set.  This shows that the axiom of comprehension is too general:  Not every expression of the form
\\[0.2cm]
\hspace*{1.3cm}
$M := \{ x \mid p(x) \}$
\\[0.2cm]
defines a set.  The expression
\\[0.2cm]
\hspace*{1.3cm}
$\bigl\{x \mid \neg(x \in x)\bigr\}$
\\[0.2cm]
has been found by the British logician and philosopher
\href{http://de.wikipedia.org/wiki/Bertrand_Russell}{Bertrand Russell} (1872 -- 1970).  It is known as
\href{http://de.wikipedia.org/wiki/Russellsche_Antinomy}{Russell's Antinomy}.

In order to avoid paradoxes such as Russell's antinomy, it is necessary to be more careful when sets are
constructed.  In the following, we will present methods to construct sets that are weaker than the
axiom of comprehension, but, nevertheless, these methods will be sufficient for our purposes.  We will use the
notation underlying the comprehension axiom and write set definitions in the form
\\[0.2cm]
\hspace*{1.3cm}
$M = \{ x \mid p(x) \}$.
\\[0.2cm]
However, we won't be allowed to use arbitrary formulas $p(x)$ here.  Instead, the formulas we are going to use
for $p(x)$ have to satisfy some restrictions.  These restrictions will prevent the construction of
self-contradictory sets.

\section{Defining Sets by Listing their Elements}
The simplest way to define a set is to list of all of its elements. These elements are enclosed in the
curly braces  ``\texttt{\{}'' and ``\texttt{\}}'' and are separated by commas.
For example, when we define \\[0.2cm]
\hspace*{1.3cm} $M := \{ 1, 2, 3 \}$, \\[0.2cm]
then the set $M$ contains the elements $1$, $2$ and $3$.
Using  the notation of the axiom of comprehension we could write this set as \\[0.2cm]
\hspace*{1.3cm}
$M = \{ x \mid x = 1 \vee x = 2 \vee x = 3 \}$.
\\[0.2cm]
Another example of a set that can be created by explicitly enumerating its elements
is the set of all lower case Latin letters.  This set is given as
define: \\[0.2cm]
\hspace*{1.3cm}
$\{\mathtt{a}, \mathtt{b}, \mathtt{c}, \mathtt{d}, \mathtt{e},
 \mathtt{f}, \mathtt{g}, \mathtt{h}, \mathtt{i}, \mathtt{j}, \mathtt{k}, \mathtt{l},
 \mathtt{m}, \mathtt{n}, \mathtt{o}, \mathtt{p}, \mathtt{q}, \mathtt{r}, \mathtt{s},
 \mathtt{t}, \mathtt{u}, \mathtt{v}, \mathtt{w}, \mathtt{x}, \mathtt{y}, \mathtt{z}\}$.
 \\[0.2cm]
Occasionally, we will use \blue{dot notation} to define a set.  Using dot notation, the set of all lower case
elements is written as
\\[0.2cm]
\hspace*{1.3cm}
$\{ \mathtt{a}, \mathtt{b}, \mathtt{c}, \cdots, \mathtt{x}, \mathtt{y}, \mathtt{z}\} $.
\\[0.2cm]
Of course, if we use dot notation the interpretation of the dots ``$\cdots$'' must always be obvious from the
context of the definition.

As a last example, we consider the \blue{empty set} $\emptyset$, which is defined as
\\[0.2cm]
\hspace*{1.3cm}
$\emptyset := \{\}$.
\\[0.2cm]
Therefore, the empty set does not contain any element at all.  This set plays an important role in set theory.  This
role is similar to the role played by the number $0$ in algebra.

If a set is defined by listing all of its elements, the order in which the
elements are listed is not important.  For example, we have
\\[0.2cm]
\hspace*{1.3cm}
$\{1,2,3\} = \{3,1,2\}$,
\\[0.2cm]
since both sets contain the same elements.

\section{Predefined Infinite Sets of Numbers}
All sets that are defined by explicitly listing their elements can only have finitely many elements.
In mathematics there are a number of sets that have an \blue{infinite} number of
elements.  One example is the
\href{http://en.wikipedia.org/wiki/Natural_number}{set of natural numbers}, which is usually denoted by the symbol $\mathbb{N}$.
Unlike some other authors, I regard the number zero as a natural number.  This is consistent with the
\href{https://en.wikipedia.org/wiki/ISO_31-11}{\textsc{Iso}-standard 31-11}.\footnote{
  The \textsc{Iso} standard 31-11 has been replaced by the
  \href{https://en.wikipedia.org/wiki/ISO_80000-2}{\textsc{Iso}-standard 80000-2},
  but the definition of the set $\mathbb{N}$ has not changed.  In the text, I did not cite \textsc{Iso} 80000-2 because
  the content of this standard is not freely available, at least not legally.
}
Given the concepts discussed so far, the quantity $\mathbb{N}$ cannot be defined.
We must therefore demand the existence of this set as an \blue{axiom}.  More precisely, we postulate that there is a
set $\mathbb{N}$ which has the following three properties:
\begin{enumerate}
\item $0 \in \mathbb{N}$.
\item If we have a number $n$ such that $n \in \mathbb{N}$, then we also have $n+1 \in \mathbb{N}$.
\item The set $\mathbb{N}$ is the smallest set satisfying the first two conditions.
\end{enumerate}
We write \\[0.2cm]
\hspace*{1.3cm} $\mathbb{N} := \{ 0, 1, 2, 3, \cdots \}$. \\[0.2cm]
Along with the set $\mathbb{N}$ of natural numbers we will use the following sets of numbers:
\begin{enumerate}
\item $\mathbb{N}^*$ is the set of \blue{positive natural numbers}, so we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $\mathbb{N}^* := \{ n \mid n \in \mathbb{N} \wedge n > 0 \}$.
\item $\mathbb{Z}$ is the set of \blue{integers}, we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $\mathbb{Z} \ = \{ 0, 1, -1, 2, -2, 3, -3, \cdots \}$

\item $\mathbb{Q}$ is the set of \blue{rational numbers}, we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $\Bigl\{ \ds\frac{p}{q} \Bigm| p \in \mathbb{Z} \wedge q \in \mathbb{N}^* \Bigr\}$.
\item $\mathbb{R}$ is the set of \blue{real numbers}.

      A clean  mathematically definition of the notion of a \href{https://de.wikipedia.org/wiki/Reelle_number}{real number}
      requires a lot of effort and is out of the scope of this lecture.  If you are interested, a detailed
      description of the construction of real numbers is given in my lecture notes on
      \href{https://github.com/karlstroetmann/Analysis/blob/master/Skript/analysis.pdf}{Analysis}.
\end{enumerate}

\section{The Axiom of Specification}
The \blue{axiom of specification}, also known as the
\href{https://en.wikipedia.org/wiki/Axiom_schema_of_specification}{axiom of restricted comprehension},
is a weakening of the comprehension axiom.  The idea behind the axiom of specification
is to use a property $p$ \blue{to select from an existing set $M$ a subset $N$ of those elements
that have the property $p(x)$}:
\\[0.2cm]
\hspace*{1.3cm}
$N := \{ x\in M \;|\; p(x) \}$
\\[0.2cm]
In the notation of the axiom of comprehension this set is written as
\\[0.2cm]
\hspace*{1.3cm}
$N := \{ x \mid x \in M \wedge p(x) \}$.
\\[0.2cm]
This is a \blue{restricted} form of the axiom of comprehension, because the condition ``$p(x)$'' that was used in the
axiom of comprehension is now strengthened to the condition ``$x \in M \wedge p(x)$''.

\exampleEng
Using the axiom of restricted comprehension, the set of even numbers can be defined as
\\[0.2cm]
\hspace*{1.3cm}
 $\{ x \in \mathbb{N} \;|\; \exists y\in \mathbb{N}: x = 2 \cdot y \}$.

\section{Power Sets}
In order to introduce the notion of a \blue{power set} we first have to define the notion of a \blue{subset}.
If $M$ and $N$ are sets, then $M$ is a \blue{subset} of $N$ if and only if each element of the
set $M$ is also an element of the set $N$.  In that case, we write $M \subseteq N$.  Formally, we define
 \\[0.2cm]
\hspace*{1.3cm}
$M \subseteq N \;\stackrel{\mathrm{def}}{\Longleftrightarrow}\; \forall x: (x \in M \rightarrow x \in N)$.

\exampleEng
We have
\\[0.2cm]
\hspace*{1.3cm}
$\{ 1, 3, 5\} \subseteq \{ 1, 2, 3, 4, 5 \}$.
\\[0.2cm]
Furthermore, for any set $M$ we have that
\\[0.2cm]
\hspace*{1.3cm}
$\emptyset \subseteq M$. \eox
\vspace*{0.2cm}

The  \blue{power set} of a set $M$ is now defined as the set of all subsets
of $M$.  We write $2^M$ for the power set of $M$.  Therefore we have
\\[0.2cm]
\hspace*{1.3cm}
$2^M := \{ x \;|\; x \subseteq M \}$.

\exampleEng
Let us compute the power set of the set $\{1,2,3\}$.  We have \\[0.2cm]
\hspace*{1.3cm}
 $2^{\{1,2,3\}} = \bigl\{ \{\},\, \{1\}, \, \{2\},\, \{3\},\, \{1,2\}, \, \{1,3\}, \{2,3\},\, \, \{2,3\},\,\{1,2,3\}\bigr\}$.
\\[0.2cm]
This set has $8 = 2^3$ elements.
\eox

In general, if the set $M$ has $m$ different elements, then it can be shown
that the power set $2^M$ has $2^m$ different elements.
More formally, let us designate the number of elements of a finite set $M$ as
$\textsl{card}(M)$.  Then we have
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{card}\left(2^M\right) = 2^{\textsl{\scriptsize card}(M)}$.
\\[0.2cm]
This explains the notation $2^M$ to denote the power set of $M$.

\section{The Union of Sets}
If two sets $M$ and $N$ are given, the union
of $M$ and $N$ is the set of all elements that are either in the set $M$ or in the set $N$ or in both $M$ and
in $N$.  This set is written as $M \cup N$.
Formally, this set is defined as
\\[0.2cm]
\hspace*{1.3cm} $M \cup N := \{ x \;|\; x \in M \vee x \in N \}$.

\exampleEng
If $M = \{1,2,3\}$ and $N = \{2,5\}$, we have
\\[0.2cm]
\hspace*{1.3cm}
$\{1,2,3\} \cup \{2,5\} = \{1,2,3,5\}$.  \eox
\vspace*{0.2cm}

The concept of the union of two sets can be generalized.  Consider
a set $X$ such that the elements of $X$ are sets themselves. For example, the
\blue{power set} of a set $M$ is a set whose elements are sets themselves.  We can form the union of all the
sets that are elements of the set $X$.  We write this set as $\bigcup X$.  Formally,
we have
\\[0.2cm]
\hspace*{1.3cm} $\bigcup X := \{ y \;|\; \exists x \in X: y \in x \}$.

\exampleEng
If we have \\[0.2cm]
\hspace*{1.3cm}
 $X = \big\{\, \{\},\, \{1,2\}, \, \{1,3,5\}, \, \{7,4\}\,\big\}$, \\[0.2cm]
then \\[0.2cm]
\hspace*{1.3cm}
 $\bigcup X = \{ 1, 2, 3, 4, 5, 7 \}$. \eox
\vspace*{0.2cm}

\exerciseEng
Assume that $M$ is a subset of $\mathbb{N}$.  Compute the set $\bigcup 2^M$.
\eox

\section{The Intersection of Sets}
If two sets $M$ and $N$ are given, we define the \blue{intersection} of $M$ and $N$ as a set of all objects that are
elements of both $M$ and  $N$.  We write that set as the average $M \cap N$.
Formally, we define
\\[0.2cm]
\hspace*{1.3cm} $M \cap N := \{ x \mid x \in M \wedge x \in N \}$.

\exampleEng
We calculate the intersection of the sets $M = \{ 1, 3, 5 \}$ and $N = \{ 2, 3, 5, 6 \}$.  We have
\\[0.2cm]
\hspace*{1.3cm} $M \cap N = \{ 3, 5 \}$.
\eox
\vspace*{0.2cm}

The concept of the intersection of two sets can be generalized.  Consider
a set $X$ such that the elements of $X$ are sets themselves.
We can form the intersection of all the
sets that are elements of the set $X$.  We write this set as $\bigcap X$.  Formally,
we have
\\[0.2cm]
\hspace*{1.3cm}
$\bigcap X := \{ y \;|\; \forall x \in X: y \in x \}$.
\vspace*{0.2cm}

\exerciseEng
Assume that $M$ is a subset of $\mathbb{N}$.  Compute the set $\bigcap 2^M$.
\eox

\section{The Difference of Sets}
If $M$ and $N$ are sets, we define the \blue{difference}
of $M$ and $N$ as the set of all objects from $M$ that are not elements of $N$.  The difference of the sets $M$
and $N$ is written as $M\backslash N$ and is formally defined as
 \\[0.2cm]
\hspace*{1.3cm} $M \backslash N := \{ x \mid x \in M \wedge x \not\in N \}$.

\exampleEng
We compute the difference of the sets $M = \{ 1, 3, 5, 7 \}$ and $N = \{ 2, 3, 5, 6 \}$.  We have
\\[0.2cm]
\hspace*{1.3cm} $M \backslash N = \{ 1, 7 \}$. \eox

\section{Image Sets}
If $M$ is a set and $f$ is a function defined for all $x$ of $M$, then the \blue{image of $M$ under $f$}
is defined as follows:
\\[0.2cm]
\hspace*{1.3cm}
$f(M) := \{ y \;|\; \exists x \in M: y = f(x) \}$.
\\[0.2cm]
This set is also written as
\\[0.2cm]
\hspace*{1.3cm}
$f(M) := \bigl\{ f(x) \;|\; x \in M \}$.

\exampleEng
The set $Q$ of all square numbers can be defined as
\\[0.2cm]
\hspace*{1.3cm}
$Q := \{ y \mid \exists x \in \mathbb{N}: y = x^2\}$.
\\[0.2cm]
Alternatively, we can define this set as
\\[0.2cm]
\hspace*{1.3cm}
$Q := \bigl\{ x^2 \mid x \in \mathbb{N} \bigr\}$.
\eox

\section{Cartesian Products}
In order to be able to present the notion of a \href{https://en.wikipedia.org/wiki/Cartesian_product}{Cartesian product},
we first have to introduce the notion of an \href{https://en.wikipedia.org/wiki/Ordered_pair}{ordered pair} of two objects
$x$ and $y$.  The \blue{ordered pair} of $x$ and $y$ is written as
\\[0.2cm]
\hspace*{1.3cm}
$\langle x, y \rangle$.
\\[0.2cm]
In the literature, the ordered pair of $x$ and $y$ is sometimes written as $(x,y)$, but I prefer the notation
with angle brackets.  The  \blue{first component} of the pair $\langle x, y \rangle$ is $x$, while $y$ is
\blue{the second component}.  Two  ordered pairs $\langle x_1, y_1 \rangle$ and $\langle x_2, y_2 \rangle$ are
equal if and only if they have the same first and second component, i.e.~we have
\\[0.2cm]
\hspace*{1.3cm}
$\langle x_1, y_1 \rangle \,=\,\langle x_2, y_2 \rangle  \;\Leftrightarrow\; x_1 = x_2 \wedge y_1 = y_2$.
\\[0.2cm]
The \blue{Cartesian product} of two sets $M$ and $N$ is now defined as the set of all ordered pairs such
that the first component is an element of  $M$ and the second component is an element of $N$.
Formally, we define the cartesian product $M \times N$ of the sets $M$ and $N$ as follows:
\\[0.2cm]
\hspace*{1.3cm}
$M \times N := \big\{ z \mid \exists x\colon \exists y\colon\bigl(z = \langle x,y\rangle \wedge x\in M \wedge y \in N\bigr) \bigr\}$.
\\[0.2cm]
To be more concise we usually write this as
\\[0.2cm]
\hspace*{1.3cm}
$M \times N := \big\{ \langle x,y\rangle \mid  x\in M \wedge y \in N \}$.

\exampleEng
If $M = \{ 1, 2, 3 \}$ and $N = \{ 5, 7 \}$ we have
\\[0.2cm]
\hspace*{1.3cm}
$M \times N = \bigl\{ \pair(1,5),\pair(2,5),\pair(3,5),\pair(1,7),\pair(2,7),\pair(3,7)\bigr\}$.
\eox
\vspace*{0.2cm}

\noindent
The notion of an ordered pair can be generalized to the notion of an
\blue{$n$-tuple} where $n$ is a natural number: An $n$-tuple has the form
\\[0.2cm]
\hspace*{1.3cm} $\langle x_1, x_2, \cdots, x_n \rangle$.
\\[0.2cm]
In a similar way, we can generalize the notion of a Cartesian product of two sets to the Cartesian product of
$n$ sets.  The \blue{general Cartesian product} of $n$ sets  $M_1$, $\cdots$, $M_n$ is defined as follows: \\[0.2cm]
\hspace*{1.3cm}
$M_1 \times \cdots \times M_n =
  \big\{ \langle x_1,x_2,\cdots,x_n \rangle \bigm| x_1\in M_1 \wedge \cdots \wedge x_n \in M_n \big\}
$.
\\[0.2cm]
Sometimes,  $n$-tuples are called lists.  In this case they are written with the square brackets ``\texttt{[}''
and ``\texttt{]}'' instead of the angle brackets ``$\langle$'' and ``$\rangle$'' that we are using.

\exerciseEng
Assume that $M$ and $N$ are finite sets.  How can the expression $\textsl{card}(M \times N)$ be reduced to an
expression containing the expressions $\textsl{card}(M)$ and $\textsl{card}(N)$?
\eox

\section{Equality of Sets}
We have now presented all the methods that we will use in this lecture in order to construct sets.
Next, we discuss the notion of \blue{equality} of two sets.  As a set is solely defined by its members,
the question of the equality of two sets is governed by the
\href{https://en.wikipedia.org/wiki/Axiom_of_extensionality}{axiom of extensionality}:
\vspace*{0.2cm}

\begin{center}
\colorbox{red}{\framebox{\colorbox{yellow}{
\begin{minipage}{0.57\linewidth}
{\sl Two sets are equal if and only if they have the same elements. }
\end{minipage}}}}
\end{center}
\vspace{0.2cm}

\noindent
Mathematically, we can capture the axiom of extensionality through the formula
\\[0.2cm]
\hspace*{1.3cm} $M = N \;\leftrightarrow\; \forall x: (x \in M \leftrightarrow x \in N)$
\\[0.2cm]
An important consequence of this axiom is the fact that the order in which the
elements are listed in a set does not matter.  For example, we have
\\[0.2cm]
\hspace*{1.3cm} $\{1,2,3\} = \{3,2,1\}$,
\\[0.2cm]
because both sets contain the same elements.  Similarly, we have
\\[0.2cm]
\hspace*{1.3cm}
$\{1,2,2,3\} = \{1,1,2,3,3\}$,
\\[0.2cm]
because both these sets contain the elements $1$, $2$, and $3$.  It does not matter how often we list these
elements when defining a set:  An object $x$ either is or is not an element of a given set $M$.  It does not
make sense to say something like ``$M$ contains the object $x$ $n$ times''.\footnote{In the literature, you will find
the concept of a \href{https://en.wikipedia.org/wiki/Multiset}{multiset}.  A \blue{multiset} does not abstract
from the number of occurrences of its elements.  In this lecture, we will not use multisets.}

If two sets are defined by explicitly enumerating their elements, the question whether
these sets are equal is trivial to decide.  However, if a set is defined using the axiom of specification, then
it can be very difficult to decide whether this set is equal to another set.  For
example, it has been shown that \\[0.2cm]
\hspace*{1.3cm}
$\{ n \in \mathbb{N}^* \mid \exists x, y, z\in\mathbb{N}^*: x^n + y^n = z^n \} = \{1, 2\}$.
\\[0.2cm]
However, the proof of this equation is very difficult because this equation
is equivalent to \href{https://en.wikipedia.org/wiki/Fermat%27s_Last_Theorem}{Fermat's conjecture}.
This conjecture was formulated in 1637 by \href{https://de.wikipedia.org/wiki/Pierre_de_Fermat}{Pierre de Fermat}.
It took mathematicians more than three centuries to come up with a rigorous proof that validates this conjecture:
In 1994 \href{https://de.wikipedia.org/wiki/Andrew_Wiles}{Andrew Wiles}
and \href{https://de.wikipedia.org/wiki/Richard_Taylor_(Mathematician)}{Richard Taylor} were able to do this.
There are some similar conjectures concerning the equality of sets that are still open mathematical problems.

\section{Chapter Review}
\begin{enumerate}
\item What is a set?
\item How is the axiom of comprehension defined?  Why can't we use this axiom to define sets?
\item What is the axiom of restricted comprehension?
\item Lists all the methods that have been introduced to define sets.
\item What is the axiom of extensionality?
\end{enumerate}
If you want to develop a deeper understand of set theory, I can highly recommend the book
\emph{Set Theory and Related Topics} by Seymour Lipschutz \cite{lipschutz:1998}.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "logic"
%%% End: