\chapter{Introduction to Finite-Difference Time-Domain Methods}\label{ch:FDTD}

\begin{flushright}{\it
``Since Newton, mankind has come to realize that the laws of physics\\
are always expressed in the language of differential equations.''\\
- Steven Strogatz} %\\
%\SWcomment[(https://youtu.be/O85OWBJ2ayo?t=44)]
\end{flushright}
%
\vspace{2em}
This chapter introduces some important concepts needed to understand finite-difference time-domain (FDTD) methods. These techniques are what the implementation of the physical models presented later on in this document are based on.
By means of a simple mass-spring system and the 1D wave equation, the notation and terminology used throughout this document will be explained.
Unless denoted otherwise, the theory presented in this chapter and the notation have been taken from \cite{theBible}. %Before diving into the mathematics, let us go over some useful terminology.

\section{Differential equations}\label{sec:differentialEquations}
Differential equations are used to describe the motion of dynamic systems, including vibrations in musical instruments. In this work, these equations are used to describe, among others, the movement of a string, an instrument body and the air pressure in an acoustic tube.

A characteristic feature of these equations is that, rather than an absolute value or \textit{state} of a system, such as displacement from the equilibrium of a string, or the pressure in a tube, the time derivative of its state -- its velocity -- or the second time derivative -- its acceleration -- is described. From this, the absolute state of the system can then be computed.
%
This state is usually described by the variable $u$ which is always a function of time, i.e., $u=u(t)$. If the system is distributed in space, $u$ also becomes a function of space, i.e., $u = u(x,t)$, or with two spatial dimensions, $u = u(x,y,t)$, etc. Though this work only describes systems of up to two spatial dimensions, one can easily extend to three dimensions \cite{Hamilton2016} and potentially higher-dimensional systems% \cite{Bustamante2017}\todo{maybe not such a relevant reference..}
. See Section \ref{sec:dimensions} for more information on dimensions.
% one could potentially extend to systems of infinite spatial dimensions evolving over time!

If $u$ is univariate, and only a function of time, the differential equation that describes the motion of the system is called an \textit{ordinary differential equation} (ODE). Various ways to describe the second derivative in time of $u$, or the acceleration of $u$ are
%
\begin{equation}\nonumber
    \begin{aligned}
        \frac{d^2 u}{d t^2} & \quad \text{(Leibniz's notation)},\\
        \ddot u\ \  &\quad \text{(Newton's notation)},\\[3pt]
        D_t^2 u& \quad \text{(Euler's notation)}.
    \end{aligned}
\end{equation}
%
Leibniz's notation could be considered the most standard notation but is not necessarily compact. Newton's notation on the other hand allows for an ultra compact notation using dots above the function to denote time-derivatives. For this reason, Newton's notation will be used for ODEs in isolation. The drawback of this notation is that it can only be used for univariate functions. Finally, Euler's notation indicates a derivative using an operator which can be applied to a function.

If $u$ is also a function of at least one spatial dimension, the equation of motion is a called a \textit{partial differential equation} (PDE).
The literature uses different types of notation for taking (continuous-time) partial derivatives. Applied to a state variable $u$ these can look like
\begin{equation}\nonumber
    \begin{aligned}
        \frac{\partial^2 u}{\partial t^2} & \quad \text{(Leibniz's notation)},\\
        u_{tt}\:\,& \quad \text{(subscript notation)},\\[3pt]
        \ptt u\: & \quad \text{(Euler's notation)},
    \end{aligned}
\end{equation}
%
%
% \begin{equation}\nonumber
%     \begin{aligned}
%         \frac{\partial^2 u}{\partial t^2}, \quad
%         u_{tt},\quad
%         \ptt u
%     \end{aligned}
% \end{equation}
%
where the subscript notation could be seen as the partial derivative counterpart to Newton's notation due to its compactness. In the remainder of this document, Euler's notation will be used for PDEs, due to their similarity to operators in discrete time (introduced in Section \ref{sec:FDoperators}). Also, it allows for more compactness when creating bigger operators with multiple (connected) systems (see e.g. Chapter \ref{ch:tromba}). Moreover, state-of-the-art literature in the field of FDTD methods for sound synthesis use this notation as well \cite{Bilbao2018}.

% Often-used partial derivatives and their meanings \todo{maybe not yet as this is super general still} are shown below

% \begin{minipage}[c]{0.49\textwidth}
%     \begin{align*}
%         \ptt u &\quad \text{(acceleration)}\\
%         \pt u &\quad \text{(velocity)}
%     \end{align*}
% \end{minipage}
% \begin{minipage}[c]{0.49\textwidth}
%     \begin{align*}
%     \pxx u &\quad \text{(curvature)}\\
%     \px u &\quad \text{(slope)}
%     \end{align*}
% \end{minipage}

\subsection{Dimensions and degrees of freedom}\label{sec:dimensions}
All objects in the physical world are three-dimensional (3D) as they have a non-zero width, length and depth. Moreover, these objects can move in these three dimensions and thus have three translational \textit{degrees of freedom (DoF)} (the three rotational DoF are ignored here).
To reduce the complexity of the models describing physical systems as well as computational complexity (computational cost), simplifications can be made to reduce both the dimensionality of the spatial distribution of a physical object as well as that of the translational DoF.

Generally, the spatial distribution of an object can be simplified if one (or more) of the dimensions are small, relative to the wavelengths of interest. A guitar string, for instance, has much greater length than its width or depth and can therefore be reduced to a one-dimensional (1D) system. If a 3D description were to be kept, the relative displacement between two locations on one cross-section along the length of the string would be taken into account. One could imagine that this displacement will always be orders of magnitude smaller than the relative displacement of two points along the string length and is thus negligible. Similarly, the thickness of a drum membrane is much smaller than its length and width and can therefore be simplified to a two-dimensional (2D) system.\footnote{In this work, `1D' and `2D' will also be used to abbreviate `one dimension' and `two dimensions'.}

The translational DoF, on the other hand, describe now many ``coordinates'' a state variable includes.
In much of the literature on FDTD methods in the field of musical acoustics, the state variable only has one coordinate. In most string models, for example, only the transverse displacement in one polarisation is considered (see Chapter \ref{ch:stiffString}) and the other polarisation as well as the longitudinal motion of the string (motion along the string length) are ignored. In other words, every point along the string can only move up and down, not side-to-side and not forward and back. Although this greatly simplifies the system at hand and reduces computational complexity, this is not what happens in reality. Nonlinear effects such as pitch glides due to tension modulation caused by high-amplitude string vibration are not present in the simplified model and have not been included in this project.\todo{think about references to this project/work in this chapter that should only be focused on background}

Work has been done on strings with dual (transverse) polarisation by Desvages \cite{Desvages2018} and Desvages and Bilbao \cite{Desvages2016} using FDTD methods. Models including longitudinal string vibration, where the longitudinal and transversal displacements are couples can be found in \cite{theBible,Bilbao2009spring}.
In \cite{Villeneuve2019}, Villeneuve and Leonard present a mass-spring network where the state of every individual mass has three translational DoF. Due to these additional DoF, these networks do capture the aforementioned effects, but greatly increase the computational complexity of the models.

Although the dimensionality reduction ignores some of the physical processes, surprisingly realistic sounding models can be made despite these simplifications. Due to computational considerations, all models used in this work thus only have 1 translational DoF.

\subsubsection{Notation}
When describing the state of a system, the spatial dimensions it is distributed over appear in the argument of the state variable. For example, the state of a 2D system with 1 translational DoF, is written as $u(x,y,t)$.

The translational DoF, on the other hand, determines the number of coordinates that the state variable describes. A 1D system with 3 translational DoF can thus be written as $\mathbf{u}(x,t)$ where $\mathbf{u}$ is a vector containing the coordinates for all three translational DoF.

% What this means for the notation introduced in the previous section is  the number of arguments for the state variable. As all systems (in this context) have one temporal dimension, the state of a 1D system -- such as a string -- is described using $u(x,t)$, a 2D system -- such as a membrane -- is described using $u(x,y,t)$, and the state of a simple zero-dimensional (0D) mass-spring system as $u(t)$.

\subsection{Ranges of definition and domains}\label{sec:domains}
When modelling physical systems, one needs to provide a \textit{range of definition} over which they are defined. For a 1D system $u = u(x,t)$, ranges of definition must be given for $x$ and $t$. Usually, the temporal range $t\geq 0$, meaning that the system is defined for non-negative time.

In space, the range of definition is usually referred to as a (spatial) \textit{domain}, denoted by the symbol $\D$. Using the example above, $x$ may be defined over $\D$, which is written as $x\in \D$. For analysis purposes, infinite domains ($\D = \mathbb{R} = (-\infty, \infty)$) or semi-infinite domains ($\D = \mathbb{R}^+ = [0, \infty)$) may be used, but for implementation purposes, a finite domain needs to be established. For higher dimensional systems, one needs to define higher dimensional domains. A 2D system $u=u(x,y,t)$, for simplicity assumed to be rectangular, may be defined over `horizontal domain' $\D_x$ and `vertical domain' $\D_y$, which are both 1D domains. The system is then defined for $(x,y)\in \D$ where $\D = \D_x\times \D_y$.

\section{Discretisation using FDTD methods}\label{sec:discUsingFDTD}
Differential equations are powerful tools to describe the motion of physical systems. Despite this, only few of these have a closed-form, or analytical, solution. More complex systems require methods that do not perfectly solve, but rather \textit{approximate} the solutions to these equations. FDTD methods are the most straightforward approach to numerically approximate differential equations. These methods are considered to be among the most general and flexible techniques in terms of the systems they can model, and frankly, relatively simple to understand once some familiarity with them is obtained. The main concern with these methods is the numerical stability of the eventual approximation. Conditions for stability can be mathematically derived and will be introduced in Section \ref{sec:stabilityAnalysis}.

% \SWcomment[It is important to note that a discrete FD scheme is an \textit{approximation} to a continuous PDE, not a sampled version of it. This means that the resulting schemes are rarely an exact solution to the original continuous equation.]

FDTD methods essentially subdivide a continuous differential equation into discrete points in time and space, a process called \textit{discretisation}. Once an ODE or PDE is discretised using these methods it is now called a \textit{finite-difference (FD) scheme} which approximates the original differential equation. See Figure \ref{fig:discretisation}. In the following, for generality and ease of explanation, a 1D system will be used, and (again) the theory and notation follows \cite{theBible}.

\begin{figure}[h]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
        thick,main node/.style={circle,draw}]

        \node[] (image) at (0,0) {
        \includegraphics[width=\textwidth]{figures/fdtd/discretisation.eps}
        };

        \draw [-stealth](-0.8,0) -- (0.3,0);
        \draw [-stealth](-5.75,1.2) -- (-5.75,-1.5);
        \node[] (t) at (-6, -1.2) {\small $t$};

        \draw[thin, ->, >=stealth] (5.0,0.88) arc (90:-90:0.25);
        \node[] () at (5.5, 0.6) {\small $\tfrac{1}{\fs}$};

        \draw[thin, ->, >=stealth] (5.0,0.30) arc (90:-90:0.25);
        \node[] () at (5.5, 0.02) {\small $\tfrac{1}{\fs}$};

        \draw[thin, ->, >=stealth] (5.0,-0.28) arc (90:-90:0.25);
        \node[] () at (5.5, -0.56) {\small $\tfrac{1}{\fs}$};

        \draw[thin, ->, >=stealth] (5.0,-0.86) arc (90:-90:0.25);
        \node[] () at (5.5, -1.12) {\small $\tfrac{1}{\fs}$};

      \end{tikzpicture}
      \caption{A continuous PDE is discretised to a FD scheme. In a PDE, time passes continuously, whereas for a FD scheme, time passes in finite increments with a duration of the reciprocal of the sample rate $\fs$. \label{fig:discretisation}}
\end{figure}

\subsection{Grid functions \todo{FULL DOC SWEEP: check capitalisation of headings throughout document (DOING NOW)}} \label{sec:gridFunctions}
The first step to approximate continuous PDEs, is to define a discrete \textit{grid} over time and space. See Figure \ref{fig:gridExp}. A system described by state $u = u(x,t)$ defined over time $t$ and one spatial dimension $x$, can be discretised to a \textit{grid function} $u_l^n$. Here, integers $l$ and $n$ describe the spatial and temporal indices respectively, and arise from the discretisation of the continuous variables $x$ and $t$, according to $x=lh$ and $t=nk$. The spatial step $h$, also called the \textit{grid spacing} describes the distance (in m) between two neighbouring \textit{grid points}, and is closely related to the stability of the FD scheme. The temporal step $k$, or \textit{time step}, is the time (in s) between two consecutive temporal indices and can be calculated $k=1/\fs$ for a sample rate $\fs$ (in Hz). In many audio applications $\fs = 44100$ Hz, which is the sample rate that will be used in this work (unless denoted otherwise).

% \subsubsection{Discrete Domains}
As mentioned in Section \ref{sec:domains}, a 1D system needs to be defined over a temporal range of definition and one spatial domain.
In discrete time, $t \geq 0$ is discretised to $n \in \mathbb{N}^0$.\footnote{In this work, $\mathbb{N}^0$ is used to denote the set of non-negative integers ($\mathbb{N}^0 = 0, 1, 2,
\hdots$).}
The spatial domain $\D$ can be subdivided into $N$ equal sections, or intervals, of length $h$ (see Figure \ref{fig:gridExp}). The grid points describing the state of the system are placed at the edge of each interval, including the end points. The spatial range of interest then becomes $l\in \{0, \hdots, N\}$ and the total number of grid points is $N+1$, which is one more than the number of intervals.

%The grid points interact with their neighbouring grid points according to the model at hand.

% At the ends of each section there needs to be grid point describing the discrete state of the system at each sections needs to be

%Consider a (continuous-time) system, $u = u(x,t)$ defined over time $t\geq 0$ and space $x\in \D$ with domain $\D = [0, L]$. Time can be subdivided in

To summarise, for a 1D system
\begin{equation*}
    u(x,t) \approxeq u_l^n \quad \text{with} \quad x=lh \quad \text{and} \quad t = nk,
\end{equation*}
\begin{equation*}
    l\in\{0, \hdots, N\}\qaq n \in \mathbb{N}^0.
\end{equation*}

\begin{figure}[h]
    \centering
    % \subfloat[If $N=5$ there are 5 sections of length $h$ and 6 grid points describing the state of the system ($l=\{0, \hdots, 5\}$).\label{fig:gridExp1}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/gridExplanation.pdf}}\hspace{0.06\textwidth}
    % \subfloat[If $N$ is large (as is usually the case), The 1D system is divided into $N$ sections of length $h$ and $l=\{0, \hdots, N\}$.\label{fig:gridExp2}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/gridExplanation2.pdf}}
    \includegraphics[width=\textwidth]{figures/fdtd/gridFigure3.pdf}
    \caption{The spatio-temporal grid that appears when a 1D system $u(x,t)$ with $x\in \D$ is discretised to a grid function $\uln$. The spatial domain $\D$ is divided into $N$ intervals of length $h$ and spatial range of interest $l=\{0, \hdots, N\}$. Time is subdivided into time steps of duration $k$ and together with the discretised domain, forms a grid over space and time. Some grid points are labelled with the appropriate grid function. \label{fig:gridExp}}
\end{figure}

\subsection{Finite-difference operators}\label{sec:FDoperators}
Now that the state variable has a discrete counterpart, this leaves the derivatives to be discretised, or approximated. We start by introducing \textit{shift operators} that can be applied to a grid function and `shift' its indexing, either temporally or spatially. These are denoted by $e_s$, where subscript $s$ denotes the type and direction of the shift. Forward and backward shifts in time, together with the identity operation are
%
\begin{equation}\label{eq:temporalShifts}
    e_{t+}\uln = u_l^{n+1},\quad e_{t-}\uln = u_l^{n-1}, \quad \text{and} \quad 1\uln = \uln.
\end{equation}
%
Similarly, forward and backward shifts in space are
%
\begin{equation}\label{eq:spatialShifts}
    e_{x+}\uln = u_{l+1}^n,\quad \text{and}\quad e_{x-}\uln = u_{l-1}^n.
\end{equation}
%
\todo{many figures for shift and FD operators}These shift operators are rarely used in isolation, though they do appear in energy analysis techniques detailed in Section \ref{sec:energyAnalysis}. The operators do, however, form the basis of commonly used \textit{finite-difference (FD) operators}. The first-order derivative in time can be discretised in three different ways. The forward, backward and centred \todo{FULL DOC SWEEP: check centred instead of centered} difference operators are
%
\todo{these spacings are different in overleaf...}
\begin{subnumcases}{\pt \approxeq\label{eq:discFirstTime}}
        \dtp &$\!\!\!\!\triangleq \frac{1}{k}\left(e_{t+} - 1\right),$\label{eq:forwardTimeOperator}\\
        \dtm &$\!\!\!\!\triangleq \frac{1}{k}\left(1 - e_{t-}\right),$\label{eq:backwardTimeOperator}\\
        \dtd &$\!\!\!\!\triangleq \frac{1}{2k}\left(e_{t+} - e_{t-}\right),$\label{eq:centredTimeOperator}
\end{subnumcases}
where ``$\triangleq$'' means ``equal to by definition''. These operators can then be applied to grid function $\uln$ to get
\begin{subnumcases}{\pt u \approxeq\label{eq:discFirstTimeU}}
    \dtp \uln &$\!\!\!\! = \frac{1}{k}\left(u_l^{n+1} - \uln\right),$\label{eq:forwardTimeOperatorU}\\
    \dtm \uln &$\!\!\!\! = \frac{1}{k}\left(\uln - u_l^{n-1}\right),$\label{eq:backwardTimeOperatorU}\\
    \dtd \uln &$\!\!\!\! = \frac{1}{2k}\left(u_l^{n+1} - u_l^{n-1}\right),$\label{eq:centredTimeOperatorU}
\end{subnumcases}
and all approximate the first-order time derivative of $u$. Note that the centred difference has a division by $2k$ as the time difference between $n+1$ and $n-1$ is, indeed, twice the time step.

\todo{figure here visualising operators (with reference to grid figure)}
Similar operators exist for a first-order derivative in space, where the forward, backward and centred difference are
\begin{subnumcases}{\px \approxeq\label{eq:discFirstSpace}}
    \dxp &$\!\!\!\!\triangleq \frac{1}{h}\left(e_{x+} - 1\right),$\label{eq:forwardSpaceOperator}\\
    \dxm &$\!\!\!\!\triangleq \frac{1}{h}\left(1 - e_{x-}\right),$\label{eq:backwardSpaceOperator}\\
    \dxd &$\!\!\!\!\triangleq \frac{1}{2h}\left(e_{x+} - e_{x-}\right),$\label{eq:centredSpaceOperator}
\end{subnumcases}
and when applied to $\uln$ are
\begin{subnumcases}{\px u \approxeq\label{eq:discFirstSpace}}
    \dxp \uln&$\!\!\!\!= \frac{1}{h}\left(u_{l+1}^n- \uln\right),$\\
    \dxm \uln&$\!\!\!\!= \frac{1}{h}\left(\uln - u_{l-1}^n\right),$\\
    \dxd \uln&$\!\!\!\!= \frac{1}{2h}\left(u_{l+1}^n - u_{l-1}^n\right).$\label{eq:centredSpaceOperatorU}
\end{subnumcases}
Higher-order differences can be approximated through a composition of first-order difference operators where their definitions are multiplied.\footnote{Alternatively, one could first apply one operator to a grid function, expand it, and apply the other operator to all individual grid functions in the result of the first expansion thereafter.} The second-order difference in time may be approximated using
\begin{equation}\label{eq:discSecondTime}
    \ptt \approxeq \dtp\dtm = \dtt \triangleq \frac{1}{k^2}\left(e_{t+}-2+e_{t-}\right),
\end{equation}
where ``$2$'' is the identity operator applied twice. This can be done similarly for the second-order difference in space
\begin{equation}\label{eq:discSecondSpace}
    \pxx \approxeq \dxp\dxm = \dxx \triangleq \frac{1}{h^2}\left(e_{x+}-2+e_{x-}\right),
\end{equation}
both of which can be applied to a grid function $\uln$ in a similar fashion. Figure \ref{fig:operators} shows the \textit{stencils} of the operators introduced above. A stencil shows the grid points needed to perform the operation of a FD operator.
\begin{figure}[h]
    \centering
    \subfloat[Temporal operators.\label{fig:temporalOperators}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/operatorsT.pdf}}\hspace{0.06\textwidth}
    \subfloat[Spatial operators.\label{fig:spatialOperators}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/operatorsX.pdf}}
    \caption{The stencils of various FD operators applied to the grid point highlighted with a red square. Black grid points are used in the calculation, and white grid points are not. The averaging operators follow the same pattern.\label{fig:operators}}
\end{figure}

%Further information on combining operators can be found in Section \ref{sec:combiningOperators}.

Also useful \todo{in energy analysis, interleaved grids, etc.} are averaging operators, all of which approximate the identity operation. The temporal forward, backward and centred averaging operators are
\begin{subnumcases}{1 \approxeq \label{eq:averagingTime}}
    \mtp & $\!\!\!\!\triangleq \frac{1}{2}\left(e_{t+} + 1\right),$\label{eq:forwardAvgTime}\\
    \mtm & $\!\!\!\!\triangleq \frac{1}{2}\left(1 + e_{t-}\right),$\label{eq:backwardAvgTime}\\
    \mtd & $\!\!\!\!\triangleq \frac{1}{2}\left(e_{t+} + e_{t-}\right).$\label{eq:centredAvgTime}
\end{subnumcases}
Notice how these definitions are different than the difference operators in \eqref{eq:discFirstTime}: the terms in the parentheses are added rather than subtracted, and rather than a division by the time step $k$ there is a division by $2$. Finally, the centred averaging operator does not have an extra division by $2$ as in \eqref{eq:centredTimeOperator}.
Applied to $\uln$, Eqs. \eqref{eq:averagingTime} become
\begin{subnumcases}{\uln \approxeq \label{eq:averagingTimeU}}
    \mtp \uln & $\!\!\!\!= \frac{1}{2}\left(u_l^{n+1}+ \uln\right),$\label{eq:forwardAvggTimeU}\\
    \mtm \uln & $\!\!\!\!= \frac{1}{2}\left(\uln + u_l^{n-1}\right),$\label{eq:backwardAvggTimeU}\\
    \mtd \uln & $\!\!\!\!= \frac{1}{2}\left(u_l^{n+1} + u_l^{n-1}\right).$\label{eq:centredAvggTimeU}
\end{subnumcases}
%
Similarly, spatial averaging operators are
\begin{subnumcases}{1 \approxeq \label{eq:averagingSpace}}
    \mxp & $\!\!\!\!\triangleq \frac{1}{2}\left(e_{x+} + 1\right),$\label{eq:forwardAvgSpace}\\
    \mxm & $\!\!\!\!\triangleq \frac{1}{2}\left(1 + e_{x-}\right),$\label{eq:backwardAvgSpace}\\
    \mxd & $\!\!\!\!\triangleq \frac{1}{2}\left(e_{x+} + e_{x-}\right),$\label{eq:centredAvgSpace}
\end{subnumcases}
and when applied to $\uln$
\begin{subnumcases}{\uln \approxeq \label{eq:averagingSpaceU}}
    \mxp \uln & $\!\!\!\!= \frac{1}{2}\left(u_{l+1}^n+ \uln\right),$\label{eq:forwardAvgSpaceU}\\
    \mxm \uln & $\!\!\!\!= \frac{1}{2}\left(\uln + u_{l-1}^n\right),$\label{eq:backwardAvgSpaceU}\\
    \mxd \uln & $\!\!\!\!= \frac{1}{2}\left(u_{l+1}^n + u_{l-1}^n\right).$\label{eq:centredAvgSpaceU}
\end{subnumcases}
Finally, using forward and backward averaging operators, second-order temporal and spatial averaging operators can be created according to
\begin{equation}
    1 \approxeq \mtt = \mtp\mtm \triangleq \frac{1}{4}\left(e_{t+}+2+e_{t-}\right),
\end{equation}
and
\begin{equation}
    1 \approxeq \mxx = \mxp\mxm \triangleq \frac{1}{4}\left(e_{x+}+2+e_{x-}\right).
\end{equation}

Operators and derivatives in 2D will be discussed in Chapter \ref{ch:2Dsyst}.

\subsubsection{Accuracy}
As FDTD methods approximate continuous systems, the resulting solution is rarely 100\% accurate. To determine the accuracy of the FD operators above, one can perform a \textit{Taylor series analysis}. The Taylor series is an infinite sum and its expansion of a function $f$ about a point $a$ is defined as
\begin{equation}
    f(x) = \sum_{n=0}^{\infty} \frac{(x-a)^n}{n!}f^{(n)}(a)
\end{equation}
where superscript $(n)$ denotes the $n$\th derivative of $f$ with respect to $x$. The analysis will be performed on the temporal operators in this section, but also applies to the spatial operators presented.

Using continuous function $u=u(t)$ and following Bilbao's ``slight abuse of notation'' in \cite{theBible}, one may apply FD operators to continuous functions according to
\begin{equation}\label{eq:forwardTimeCont}
    \dtp u(t) = \frac{u(t+k) - u(t)}{k}\ .
\end{equation}
%
Assuming that $u$ is infinitely differentiable, $u(t+k)$, i.e., $u$ at the next time step (in continuous time), can be approximated using a Taylor series expansion of $u$ about $t$ according to
\begin{equation}\label{eq:taylorStartForward}
    u(t+k) = u(t) + k \dot u + \frac{k^2}{2} \ddot u + \frac{k^3}{6} \dot{\ddot{u}} + \OO(k^4).
\end{equation}
Here, (following Newton's notation introduced in Section \ref{sec:differentialEquations}) the dot describes a single temporal derivative and $\OO$ includes additional terms in the expansion. The power of $k$ in the argument of $\OO$ describes the order of accuracy, the higher the power of $k$ the more accurate the approximation. Equation \eqref{eq:taylorStartForward} can be rewritten to
\begin{equation*}
    \frac{u(t+k) - u(t)}{k} = \dot u + \frac{k}{2} \ddot u +\frac{k^2}{6} \dot{\ddot{u}} + \OO(k^3),
\end{equation*}
and using Eq. \eqref{eq:forwardTimeCont} can be written to
\begin{equation}
    \dtp u(t) = \dot u + \OO(k).
\end{equation}
This says that the forward difference operator approximates the continuous first order derivative with an additional error term that depends on $k$.
As the power of $k$ in $\OO$'s argument is $1$, the forward operator is first-order accurate. One can also observe that, as expected, the error gets smaller as the time step $k$ gets smaller and indicates that higher sample rates result in more accurate simulations (through $k=1/\fs$).

One can arrive at a similar result for the backward operator. Applying Eq. \eqref{eq:backwardTimeOperator} to $u(t) $ yields
\begin{equation}
    \dtm u(t) = \frac{u(t) - u(t-k)}{k}.
\end{equation}
One can then approximate $u(t-k)$ by performing a Taylor series expansion of $u$ about $t$ according to
\begin{align}
    u(t-k) &= u(t) + (-k) \dot u + \frac{(-k)^2}{2} \ddot u +\frac{(-k)^3}{6}\dot{\ddot{u}} + \OO(k^4),\label{eq:taylorStartBackward}\\
    \frac{u(t-k) - u(t)}{k} &= -\dot u + \frac{k}{2} \ddot u - \frac{k^2}{6}\dot{\ddot{u}} + \OO(k^3),\nonumber\\
    % \dot u &= \frac{u(t) - u(t-k)}{k} + \OO(k).
    \dtm u(t) &= \dot u + \OO(k).
\end{align}
Notice that the sign of $\OO$ does not matter.

Applying the centred operator in Eq. \eqref{eq:centredTimeOperator} to $u(t)$ yields
\begin{equation}
    \dtd u(t) = \frac{u(t+k) - u(t-k)}{2k},
\end{equation}
indicating that to find the order of accuracy for this operator, both Eqs. \eqref{eq:taylorStartForward} and \eqref{eq:taylorStartBackward} are needed. Subtracting these and substituting their definitions yields
\begin{align}
    u(t+k) - u(t-k) &= 2k\dot u - \frac{2k^3}{6}\dot{\ddot{u}} + 2\OO(k^5),\nonumber\\
    \frac{u(t+k) - u(t-k)}{2k} &= \dot u + \OO(k^2),\nonumber\\
    \dtd u(t) &= \dot u + \OO(k^2),
\end{align}
and shows that the centred difference operator is second-order accurate.

As a first-order derivative indicates the \textit{slope} of a function, the differences in accuracy between the above operators can be visualised as in Figure \ref{fig:taylor}. It can be observed that the derivative approximation -- the slope -- of the centred operator matches much more closely the true derivative of $u$ at $t$.

\begin{figure}[h]
    \includegraphics[width=\textwidth]{figures/fdtd/taylor.eps}
    \caption{\label{fig:taylor} The accuracy of the forward, backward and centred difference operators in \eqref{eq:discFirstTime} visualised. The centred difference operator much more closely approximates the derivative, or the slope, of $u$ at $t$ when compared to the forward and backward difference operators.}
\end{figure}

Higher-order differences, such as the second-order difference in time operator in Eq. \eqref{eq:discSecondTime} can also be applied to $u(t)$ to get
\begin{equation}
    \dtt u(t) =  \frac{u(t+k) - 2u(t) + u(t-k)}{k^2},
\end{equation}
and can be proven to be second-order accurate by adding Eqs. \eqref{eq:taylorStartForward} and \eqref{eq:taylorStartBackward}:
\begin{align}
    u(t+k) + u(t-k) &= 2 u(t) + k^2 \ddot u + \OO(k^4),\nonumber\\
    \frac{u(t+k) -2 u(t) + u(t-k)}{k^2} &= \ddot u + \OO(k^2),\nonumber\\
    \dtt u(t) &= \ddot u + \OO(k^2).
\end{align}

The accuracy of averaging operators can be found in the same way and follow a similar pattern.
\begin{equation}
    \begin{aligned}
    \mtp u(t) = u(t) + \OO(k),\quad &\mtm u(t)= u(t) + \OO(k),\\
    \mtd u(t)= u(t) + \OO(k^2), \quad &\mtt u(t)= u(t) + \OO(k^2).
    \end{aligned}
\end{equation}

\subsection{Identities}
For working with FD schemes, either for implementation or analysis, it can be extremely useful to rewrite the operators presented above to equivalent versions of themselves. These are called \textit{identities} and for future reference, some useful ones are listed below:\footnote{The $\pm$ operators in a single equation are either all `$+$' or all `$-$' and are used for a more compact notation.}
\begin{subequations}
    \begin{align}
        \dtt &= \frac{2}{k}\left(\dtd- \dtm\right),\label{eq:identity1}\\
        \dtd &= \dtp\mtm = \dtm\mtp\label{eq:identity2},\\
        \mu_{t\pm} &= \pm\frac{k}{2}\delta_{t\pm} + 1\label{eq:identity3},\\
        \delta_{t\pm} &= \pm\frac{2}{k}(\mu_{t\pm} - 1),\label{eq:identityLip}\\
        \mtd &= k\dtd + e_{t-}.\label{eq:identity4}
    \end{align}
\end{subequations}
That these equalities hold, can easily be proven by expanding the operators defined in Section \ref{sec:FDoperators}. Naturally, these identities also hold for spatial operators by simply substituting the `$t$' subscripts for `$x$'.

\section{%Intro to ODEs:
The mass-spring system}\label{sec:massSpringSystem}
Though a complete physical modelling field on their own (see Chapter \ref{ch:physMod}), mass-spring systems are also sound-generating systems and lend themselves well to illustrating and explaining FDTD methods in practice. Starting with the continuous-time ODE, the current section follows the discretisation process to a FD scheme using the operators described in Section \ref{sec:FDoperators}. Finally, the scheme is rewritten to an update equation that can be implemented and the output of the system is shown.

\subsection{Continuous time}\label{sec:massSpringCont}
Using dots to indicate a temporal derivative, the ODE of a simple mass-spring system is defined as
\begin{equation}\label{eq:massSpringPDE}
    M\ddot u = -Ku,
\end{equation}
where $u = u(t)$ is the distance from the equilibrium position (in m), $M>0$ is the mass of the mass (in kg) and $K\geq 0$ is the spring constant (in N/m). Equation \eqref{eq:massSpringPDE} can be written as
\begin{equation}\label{eq:massSpringCompact}
    \ddot u = -\omega_0^2u,
\end{equation}
with angular frequency (in rad/s)
\begin{equation}\label{eq:omega0MassSpring}
    \omega_0 = \sqrt{K/M}.
\end{equation}
This way of writing the mass-spring ODE is more compact and can more directly be related to the fundamental frequency $f_0 = \omega_0 / 2 \pi$ (in Hz) of the system.

Apart from the choices of $K$ and $M$, the behaviour of the mass-spring system is determined by its \textit{initial conditions}, being $u(0)$ and $\pt u(0)$, i.e., the displacement and velocity of the mass at $t = 0$. If the initial conditions are non-zero, the path that the displacement of the mass follows over time is sinusoidal (see Figure \ref{fig:massSpring}), which is also why the mass-spring system is often referred to as the \textit{simple harmonic oscillator}. The amplitude of the sinusoid is determined by the initial conditions, whereas the frequency is determined by $M$ and $K$.

\input{introduction/massSpringTikz.tex}

\subsubsection{Intuition}\todo{move section up (stefan's comment)}
The behaviour of the mass-spring system in Eq. \eqref{eq:massSpringPDE} arises from two basic laws of physics: \textit{Newton's second law} and \textit{Hooke's law}.

Starting with Newton's second law -- \textit{force equals mass times acceleration} -- and relating this to the variables used in Eq. \eqref{eq:massSpringPDE} yields an expression for force
\begin{equation}\label{eq:newton2nd}
    F = M\ddot u.
\end{equation}
This equation in isolation can be used to, for example, calculate the force necessary to accelerate a mass of $M$ kg to $\ddot u$ m/s$^2$. Next, the force generated by the spring follows Hooke's law:
\begin{equation}\label{eq:hookesLaw}
    F = -Ku,
\end{equation}
which simply states that the force generated by a spring with stiffness $K$ is negatively proportional to the value of $u$. In other words, the further the spring is extended (from the equilibrium $u=0$), the more force will be generated in the opposite direction. Finally, as the sole force acting on the mass is the one generated by the spring, the two expressions for the force $F$ can be set equal to each other and yield the equation for the mass-spring system in \eqref{eq:massSpringPDE}.

The sinusoidal behaviour of the mass-spring system, or a least the fact that the mass ``gets pulled back'' to the equilibrium, is apparent from the minus-sign in Eq. \eqref{eq:hookesLaw}. The frequency of the sinusoid, depends on the value of $K$ as the ``pull'' happens to a higher degree for a higher spring stiffness.
That the frequency of the system is also dependent on the mass $M$ can be explained by the fact that a lighter object is more easily moved and vice versa, which is apparent from Eq. \eqref{eq:newton2nd}. In other words, the pull of the spring has a greater effect on the acceleration of a lighter object than a heavier one.

%Both these parameters affect the frequency through the definition of $\omega_0$ in Eq. \eqref{eq:omega0MassSpring}.

%From the definition of $\omega_0$ in Eq. \eqref{eq:omega0MassSpring} one can observe that a smaller $M$ will cause a higher frequency explained by the fact that a smaller mass is more easily moved. Similarly, a higher spring constant $K$ will cause a higher frequency, as the force caused by Hooke's law ``pulls'' the mass back to the equilibrium to a higher degree.

Finally, if $u = 0$ there is no spring force present and the acceleration remains unchanged. This is exactly what Newton's first law states: if the net force acting on an object is zero, its velocity will be constant. If the mass is not in motion, this means that it remains stationary. If it is, at the exact moment that $u=0$, the velocity is unchanged.

\subsection{Discrete time}
Following the discretisation process introduced in Section \ref{sec:discUsingFDTD}, one can approximate the PDE in Eq. \eqref{eq:massSpringPDE}. The displacement of the mass is approximated using
\begin{equation}
    u(t) \approx u^n,
\end{equation}
with time $t = nk$, time step $k = 1/\fs$, sample rate $\fs$ and temporal index and $n \in \mathbb{N}^0$. Note that the ``grid function'' does not have a subscript $l$ as $u$ is not distributed in space and is now simply called a \textit{time series}.

Using the operators found in Section
\ref{sec:FDoperators}, Eq. \eqref{eq:massSpringPDE} can be discretised as follows:
\begin{equation}\label{eq:massSpringFDS}
    M \dtt \un = -K\un,
\end{equation}
which is the first appearance of a FD scheme in this work. Expanding the $\delta_{tt}$ operator yields
\begin{equation*}
    \frac{M}{k^2}\left(u^{n+1}-2\un+u^{n-1}\right) = -K\un,\nonumber
\end{equation*}
and solving for $u^{n+1}$ results in the following recursion or \textit{update equation}:
\begin{equation}\label{eq:massSpringUpdate}
    u^{n+1} = \left(2-\frac{Kk^2}{M} \right)\un - u^{n-1},
\end{equation}
which can be implemented in a programming language such as \texttt{MATLAB}.

\subsection{Implementation and output}\label{sec:massSpringImplementation}
A simple \texttt{MATLAB} script implementing the mass-spring system described in this section is shown in Appendix \ref{app:massSpringCode}. The most important part of the algorithm happens in a for-loop recursion, where update equation \eqref{eq:massSpringUpdate} is implemented. At the end of each loop, the system states are updated and prepared for the next iteration. %See Algorithm \ref{alg:massSpringLoop}

To be able to start the simulation of the scheme, the initial conditions given in Section \ref{sec:massSpringCont} must be discretised at $n=0$. As $n$ is only defined for values greater than zero, the forward difference operator is used. A simple way to obtain a sinusoidal motion with an amplitude of $1$, is to set the initial conditions as follows:
%
\begin{equation}
    u^0 = 1 \quad \text{and} \quad \dtp u^0 = 0.
\end{equation}
The latter equality can be expanded and solved for $u^1$ to obtain its definition:
\begin{align*}
    &\frac{1}{k}\left(u^1 - u^0\right) = 0,\\[-1em]
    \xLeftrightarrow{\mystrut\ u^0 = 1\ }\quad &u^1 - 1 = 0,\\[0.1em]
 &u^1 = 1.
\end{align*}
In short, setting $u^0 = u^1 \neq 0$ yields an oscillatory behaviour with an amplitude of $1$. Note that any other non-zero initial condition will also yield oscillatory behaviour, but likely with a different amplitude.

\begin{figure}[b]
    \includegraphics[width=\textwidth]{figures/fdtd/massSpringOutput.eps}
    \caption{The time domain and frequency domain output of a mass-spring system with $f_0 = 440$ Hz. \label{fig:massSpringOutput}}
\end{figure}

The values for $K$ and $M$ are restricted by a stability condition
\begin{equation}
    k < 2\sqrt{\frac{M}{K}},
\end{equation}
which will be elaborated on in Section \ref{sec:stabilityAnalysis}. If this condition is not satisfied, the system will exhibit (exponential) growth and is \textit{unstable}.

The output of the system can be obtained by `recording' the displacement of the mass and listening to this at the given sample rate $\fs$. An example of this can be found in Figure \ref{fig:massSpringOutput} where the frequency of oscillation $f_0 = 440$ Hz.

% \setlstMAT
% \begin{lstlisting}[caption=\texttt{MATLAB} implementation of a simple mass spring system. The code only shows the for-loop. The full code can be found in Appendix \ref{app:massSpringCode}., label=alg:massSpringLoop]
% %%% ... initialisation ... %%%

% %% Simulation loop
% for n = 1:lengthSound

%     % Update equation Eq. %*\eqrefMatlab[eq:massSpringUpdate] *)
%     uNext = (2 - K * k^2 / M) * u - uPrev;

%     out(n) = u;

%     % Update system states
%     uPrev = u;
%     u = uNext;
% end
% \end{lstlisting}

\section{%Intro to PDEs:
The 1D wave equation}\label{sec:1DWave}
Arguably the most important PDE in the field of physical modelling for sound synthesis is the 1D wave equation. It can be used to describe transverse vibration in an ideal string, longitudinal vibration in an ideal bar or the pressure in an acoustic tube (see Chapter \ref{ch:brass}). Although the behaviour of this equation alone does not appear in the real world as such -- as no physical system is ideal -- it is extremely useful as a test case and a basis for more complicated models. %Here, it will be used to introduce various concepts and analysis techniques in the field of FDTD methods.

\subsection{Continuous time\todo{FULL DOC SWEEP: check hyphen in titles (DOING)}}\label{sec:1DwaveContTime}
The 1D wave equation is a PDE that describes the motion of a system distributed in one dimension of space. Consider the state of a 1D system $u=u(x,t)$ \todo{unit?}of length $L$ (in m) defined for time $t\geq 0$ and $x\in \D$ with $\D = [0, L]$. The PDE describing its motion is
\begin{equation}\label{eq:1DwavePDE}
    \ptt u = c^2 \pxx u,
\end{equation}
where $c$ is the wave speed of the system (in m/s). If the PDE is used to model an ideal string, the wave speed can be defined as $c = \sqrt{T / \rho A}$, with tension $T$ (in N), material density $\rho$ (in kg/m$^3$) and cross-sectional area $A$. If instead, it is used to model pressure in an acoustic tube $c$ is the speed of sound in air. Figure \ref{fig:1DWavePropagation} shows the wave propagation of the 1D wave equation excited using a raised cosine.

\def\figWidth{0.32}
\begin{figure}[h]
    \centering
    \subfloat[$t = 0$ ms.\label{fig:1DWaveProp1}]{\includegraphics[width=\figWidth\textwidth]{figures/fdtd/1DWaveProp1.eps}}\hfill
    \subfloat[$t = 1$ ms.\label{fig:1DWaveProp2}]{\includegraphics[width=\figWidth\textwidth]{figures/fdtd/1DWaveProp2.eps}}\hfill
    \subfloat[$t = 2$ ms.\label{fig:1DWaveProp3}]{\includegraphics[width=\figWidth\textwidth]{figures/fdtd/1DWaveProp3.eps}}
    \caption{Wave propagation in the 1D wave equation in Eq. \eqref{eq:1DwavePDE} with $c \approx 127$ m/s.\label{fig:1DWavePropagation}}
\end{figure}

\subsubsection{Intuition}
% Although the 1D wave equation often appears in the literature, an intuition or interpretation of why it works the way it does is hard to find. In the following, $u$ describes the transverse displacement of an ideal string.

% OR

% I would like to use this opportunity to provide some extra explanation as to how and why the 1D wave equation in \eqref{eq:1DwavePDE} works the way it does. This will hopefully provide some basic intuition into the workings of PDEs that will make it easier to work with later on. \SWcomment[somethingsomething]

As with the mass-spring system in Section \ref{sec:massSpringSystem} the working of the PDE in \eqref{eq:1DwavePDE} arises from Newton's second law, even though this connection might be less apparent. %As in the mass-spring case, the acceleration of a system state is calculated, but now this depends on

The 1D wave equation in \eqref{eq:1DwavePDE} states that the acceleration of $u(x,t)$ at location $x$ is determined by the second-order spatial derivative of $u$ at that same location (scaled by a constant $c^2$). In the case that $u$ describes the transverse displacement of an ideal string, this second-order derivative denotes the \textit{curvature} of this string. As $c^2$ is always positive, the sign (or direction) of the acceleration is fully determined by the sign of the curvature. In other words, a `positive' curvature at location $x$ along the ideal string yields a `positive' or upwards acceleration at that same location.

What a `positive' or `negative' curvature implies is more easily seen when we take a simple function describing a parabola, $y(x) = x^2$, and take its second derivative to get $y''(x) = 2$. The answer is a positive number which means that $y$ has a positive curvature.

So, what does this mean for the 1D wave equation? As a positive curvature implies a positive or upwards acceleration as per Eq. \eqref{eq:1DwavePDE}, $u$ with a positive curvature at a location $x$ will start to move upwards and vice versa. Of course, the state of a physical system such as $u$ will rarely have a perfect parabolic shape, but the argument still applies. See Figure \ref{fig:curvature} for a visualisation of the forces acting on $u$ due to curvature.

How the 1D wave equation relates to Newton's second law, becomes apparent by slightly rewriting Eq. \eqref{eq:1DwavePDE}. Recalling the definition of $c$ for an ideal string, one can rewrite the 1D wave equation to
\begin{equation*}
    \rho A \ptt u = T \pxx u,
\end{equation*}
where $\rho A$ describes the \textit{mass per unit length} of the string. As the forces present in the system act on infinitesimally small portions of the string Newton's second law appears by a multiplication of $dx$
\begin{equation*}
    \underbrace{\rho A \ptt u dx}_{ma} = \underbrace{T \pxx u dx}_{F},
\end{equation*}
where $\rho A dx$ is the mass of a (tiny) portion of the string of length $dx$ (in m), $\ptt u$ is the acceleration of that portion and $T \pxx u dx$ describes the force acting on that portion, yielding Newton's second law.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/fdtd/curvature.eps}
    \caption{\label{fig:curvature} The forces acting on the 1D wave equation described by $u(x,t)$ due to curvature. The arrows indicate the direction and magnitude of the force, and simultaneously the acceleration as these are connected through Eq. \eqref{eq:1DwavePDE}.}
\end{figure}\todo{different wording in caption}

\subsubsection{Boundary conditions}
When a system is distributed in space, \textit{boundary conditions} must be determined. Recalling that $x$ is defined over domain $\D = [0, L]$, the boundaries, or end points of the system are located at $x=0$ and $x=L$. Two often-used alternatives for the boundary conditions are
%
\begin{subequations}\label{eq:boundaryCond1DWave}
    \begin{align}
        u(0, t) = u(L, t) &= 0\quad \text{(Dirichlet, fixed)},\label{eq:contDirichlet}\\
        \px u(0, t) = \px u(L, t) &= 0\quad \text{(Neumann, free)}.\label{eq:contNeumann}
    \end{align}
\end{subequations}
%
The Dirichlet boundary condition says that at the end points of the system, the state is 0 at all times. The Neumann condition on the other hand, says that rather the slope of these points needs to be 0, but that the end points are free to move transversely. In the former case, incoming waves invert after reaching the boundary whereas in the latter incoming waves are reflected un-inverted. See Figure \ref{fig:boundaryCondsCont}\todo{add why this is relevant?}.

If both boundaries of the 1D wave equation share the same condition, the fundamental frequency of the simulation can be calculated using
\begin{equation}\label{eq:fundamentalFreq}
    f_0 = \frac{c}{2L}\ .
\end{equation}

\begin{figure}[t]
    \centering
    \subfloat[The Dirichlet boundary condition in Eq. \eqref{eq:contDirichlet} fixes the boundary, which causes the incoming waves to invert.\label{fig:dirichlet}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/dirichletCont.eps}}\hspace{0.06\textwidth}
    \subfloat[The Neumann or free boundary condition in Eq. \eqref{eq:contNeumann} fixes the slope at the boundary, causing the incoming waves to not invert.\label{fig:neumann}]{\includegraphics[width=0.45\textwidth]{figures/fdtd/neumannCont.eps}}
    \caption{The behaviour of the 1D wave equation with (a) Dirichlet or (b) Neumann boundary conditions.\label{fig:boundaryCondsCont}}
\end{figure}

\subsubsection{Scaling}
As this work follows much of Bilbao's \textit{Numerical Sound Synthesis} \cite{theBible}, it might be good to talk about a major discrepancy between the PDEs and FD schemes that appear there and those used here. Non-dimensionalisation, or \textit{scaling}, is extensively used in \cite{theBible} and much of the literature published around that time (fx. \cite{Bilbao2009Modular,Bilbao2009spring}) and can be useful to reduce the number of parameters used to describe a system.

Scaling techniques normalise the domain $x\in[0, L]$ to $x' \in [0, 1]$ with $x' = x/L$ . The 1D wave equation in \eqref{eq:1DwavePDE} can then be rewritten to
\begin{equation}\label{eq:scaled1Dwave}
    \ptt u = \gamma^2\partial_{x'x'}u,
\end{equation}
where scaled wave speed $\gamma = c/L$ has units of frequency. The scaling has removed the necessity for both $c$ and $L$ and simply specifying the scaled wave speed $\gamma$ is enough to parametrise the behaviour of the system. The parameter reduction gets more apparent for more complex systems and could greatly simplify the models used, at least in notation and parameter control.

Although this parameter reduction might be useful for resonators in isolation, when multiple resonators interact with each other (see Part \ref{part:interactions}), it is better to keep the systems dimensional. As a big part of this work includes interaction between multiple resonators, only dimensional systems will appear here.\todo{check whether still correct}

\subsection{Discrete time}\label{sec:1DWaveDisc}
Coming back to the PDE presented in Eq. \eqref{eq:1DwavePDE}, we continue by finding a discrete-time approximation for it. As explained in Section \ref{sec:gridFunctions}, a continuous state variable $u = u(x,t)$ can be discretised using $x=lh$ with grid spacing $h$ (in m) and $t=nk$ with time step $k$ (in s). The grid function $\uln$ approximating $u$ can then be indexed by spatial index $l \in \{0, \hdots, N\}$ with number of intervals between the grid points $N$ and temporal index $n\in \mathbb{N}^0$. Continuing with the approximations of the derivatives in the 1D wave equation, the most straightforward \todo{FULL DOC SWEEP: check straightforward or straight-forward} discretisation of Eq. \eqref{eq:1DwavePDE} is the following FD scheme
\begin{equation}\label{eq:1DwaveFDS}
    \dtt \uln = c^2 \dxx \uln.
\end{equation}
Other schemes exist (see e.g. \cite{theBible}), but are excluded as they have not been used in this work. Expanding the operators using the definitions given in Section \ref{sec:FDoperators} yields
\begin{equation}
    \frac{1}{k^2}\left(u_l^{n+1}-2 \uln + u_l^{n-1}\right) = \frac{c^2}{h^2} \left(u_{l+1}^n - 2 \uln + u_{l-1}^n\right).
\end{equation}
and solving for $u_l^{n+1}$ yields
\begin{equation}\label{eq:1DwaveUpdate}
    u_l^{n+1} = \left(2-2\lambda^2\right) \uln  + \lambda^2\left(u_{l+1}^n + u_{l-1}^n\right) - u_l^{n-1}.
\end{equation}
Here,
\begin{equation}\label{eq:courantNumber}
    \lambda = \frac{ck}{h}
\end{equation}
is called the \textit{Courant number} and plays a big role in stability and quality of the FD scheme. More specifically, $\lambda$ needs to abide the (famous) Courant-Friedrichs-Lewy or \textit{CFL condition} for short \cite{Courant1928}
\begin{equation}\label{eq:CFL}
    \lambda \leq 1,
\end{equation}
which acts as a stability condition for scheme \eqref{eq:1DwaveFDS}. More details on this are given in Section \ref{sec:quality1DWave}.

As $c$, $k$ and $h$ are interdependent due to the CFL condition, it is useful to rewrite Eq. \eqref{eq:CFL} in terms of known variables.
As the time step $k$ is based on the sample rate and thus (usually) fixed, and $c$ is a user-defined wave speed, the CFL condition can be rewritten in terms of the grid spacing $h$:
\begin{equation}\label{eq:1DWaveStabilityCond}
    h \geq ck,
\end{equation}
%
which, in implementation, is used as a stability condition for the scheme. See Section \ref{sec:stabilityAnalysis} for more information on how to derive a stability condition from a FD scheme.

\subsubsection{Stencil}
As was done for several FD operators in Figure \ref{fig:operators}, it can be useful to visualise the \textit{stencil}, or region of operation, of a FD scheme. A stencil of a scheme visualises what grid values are necessary to calculate the state at the next time step $u_l^{n+1}$. Figure \ref{fig:stencil1DWave} shows the stencil for scheme \eqref{eq:1DwaveFDS} and -- in essence -- visualises the various shifts of the grid function in Eq. \eqref{eq:1DwaveUpdate}. One could visualise this stencil to be placed on the left-most points of the grid shown in Figure \ref{fig:gridExp}. The update equation then iterates this stencil over the entire domain and calculates all values of $u_l^{n+1}$ based on known values of $u_l^n$ and $u_l^{n-1}$.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/fdtd/1DWaveStencil.eps}
    \caption{The stencil, or region of operation, for the FD scheme in \eqref{eq:1DwaveFDS}. The time steps of the various grid points are colour-coded by yellow ($n+1$), light blue ($n$) and dark blue ($n-1$).\label{fig:stencil1DWave}}
\end{figure}

\subsubsection{Boundary conditions and virtual grid points}
The end points of the discrete domain are located at $l = 0$ and $l = N$.
Substituting these locations into Eq. \eqref{eq:1DwaveUpdate} shows that grid points outside of the defined domain are needed, namely $u_{-1}^n$ and $u_{N+1}^n$. These can be referred to as \textit{virtual grid points} and can be accounted for %/defined
by discretising the boundary conditions in Eq. \eqref{eq:boundaryCond1DWave}. Discretising these (using the most accurate centred spatial difference operator for the Neumann condition) yields
\begin{subequations}
    \begin{align}
        u_0^n = u_N^n &= 0 \quad\text{(Dirichlet, fixed)}\label{eq:discreteDirichlet},\\
        \delta_{x\cdot} u_0^n = \delta_{x\cdot} u_N^n &= 0 \quad \text{(Neumann, free)}.\label{eq:discreteNeumann}
    \end{align}
\end{subequations}
If Dirichlet boundary conditions are used, the states of the boundary points will always be zero and can therefore be excluded from the calculations. The range of calculation then simply becomes $l\in\{1,\hdots, N-1\}$ and no virtual grid points are needed when performing the update.

If, on the other hand, Neumann conditions are used, the range of calculation remains $l\in\{0,\hdots, N\}$ and definitions for the virtual grid points need to be found.
Expanding the operators in Eq. \eqref{eq:discreteNeumann} and solving for $u_{-1}^n$ and $u_{N+1}^n$ provides the definitions for these virtual grid points based on values inside the defined domain:

\begin{minipage}[c]{0.49\textwidth}
    \begin{align*}
        &\frac{1}{2h} \left(u_1^n - u_{-1}^n\right) = 0,\\
        &u_1^n - u_{-1}^n = 0,\\
        &u_{-1}^n = u_1^n.
    \end{align*}
\end{minipage}
\begin{minipage}[c]{0.49\textwidth}
    \begin{align*}
        &\frac{1}{2h} \left(u_{N+1}^n - u_{N-1}^n\right) = 0,\\
        &u_{N+1}^n - u_{N-1}^n = 0,\\
        &u_{N+1}^n = u_{N-1}^n.
    \end{align*}
\end{minipage}
\\
\\
\noindent At the boundaries, the update equation in Eq. \eqref{eq:1DwaveUpdate} will then have the above definitions for the virtual grid points substituted and will become
\begin{equation}\label{eq:1DWaveLeftBound}
    u_0^{n+1} = \left(2-2\lambda^2\right) u_0^n  + 2\lambda^2 u_1^n - u_0^{n-1},
\end{equation}
and
\begin{equation}\label{eq:1DWaveRightBound}
    u_N^{n+1} = \left(2-2\lambda^2\right) u_N^n  + 2\lambda^2 u_{N-1}^n - u_N^{n-1},
\end{equation}
at the left and right boundary respectively.

\subsection{Implementation}\label{sec:output1DWave}
This section provides information on the excitation and output of the system. See Appendix \ref{app:1DWave} for a \texttt{MATLAB} implementation of the 1D wave equation.

\subsubsection{Excitation}
A simple way to excite the system is to initialise the state using a raised cosine, or Hann window. More information on excitations will be given in Part \ref{part:exciters}, but for completeness, the formula for a discrete raised cosine will be given here.

The discrete raised cosine can be parametrised by its center location $l_0$ and width $w$ (in` grid spacings') \todo{look at this} from which the start index $l_\text{s}$ and end index $l_\text{e}$ can be calculated, according to
\begin{equation}
    l_\text{s} = l_0 - \floor[w/2]\qaq l_\text{e} = l_0 + \floor[w/2],
\end{equation}
where $\floor[\cdot]$ denotes the flooring operation and needs to be used as all the above variables are integers. Furthermore, both $l_\text{s}$ and $l_\text{e}$ must fall into the defined spatial range of calculation. Then, a raised cosine with an amplitude of $1$ can be calculated and used as an initial condition for the system according to
\begin{equation}\label{eq:raisedCos}
    u_l^1 = u_l^0 =
    \begin{cases}
        0.5 - 0.5\cos\left(\frac{2\pi (l - l_\text{s})}{w}\right), & l_\text{s} \leq l \leq l_\text{e},\\
        0, & \text{otherwise}.
    \end{cases}
\end{equation}
As done for the implementation of the mass-spring system in Section \ref{sec:massSpringImplementation}, both $u_l^0$ and $u^1_l$ are initialised with the same state, as to only have an initial displacement, and not an initial velocity.

In \texttt{MATLAB}, an easier way to obtain a raised cosine is to use the \texttt{hann(w)} function which returns a raised cosine (or Hann window) of width \texttt{w} (in grid points).

\subsubsection{Output and modes}
After the system is excited, one can retrieve the output of the system by selecting a grid point $l_\text{out}$ and listening to that at the given sample rate $\fs$. An example using the parameters in Table \ref{tab:1DWaveParams} and Dirichlet boundary conditions is shown in Figure \ref{fig:1DWaveOutput}.

\begin{table}[h]
    \begin{center}
    \begin{tabular}{|l|c|c|}
        \hline
        Name & Symbol (unit) & Value\\ \hline
        \multicolumn{3}{|l|}{\bf User-defined parameters}\\ \hline
        Length & $L$ (m) & $1$\\
        Wave speed & $c$ (m/s) & $1470$\\
        Sample rate & $\fs$ (Hz) & $44100$ \\\hline
        \multicolumn{3}{|l|}{\bf Derived parameters}\\ \hline
        Fundamental frequency & $f_0$ (Hz) & $735$\\
        No. of intervals & $N$ (-) & $30$ \\
        Time step & $k$ (s)& $\approx 2.27\cdot 10^{-5}$ \\
        Grid spacing & $h$ (m)& $\approx 0.033$ \\
        Courant number & $\lambda$ (-)& $1$ \\\hline
        \multicolumn{3}{|l|}{\bf Excitation and output}\\ \hline
        Center location& $l_0$ (-)& $0.2N$\\
        Width& $w$ (-)& 4\\
        Output location & $l_\text{out}$ & 3 \\\hline
    \end{tabular}
    \caption{Parameters used for 1D wave equation example used in this section. The user-defined parameters have been chosen such that $\lambda = 1$. \label{tab:1DWaveParams}}
    \end{center}
\end{table}

As can be seen from Figure \ref{fig:1DWaveOutput}, the output of the 1D wave equation contains many peaks in the frequency spectrum on top of the fundamental frequency. These are called \textit{harmonic partials} or \textit{harmonics} for short and arise from the various modes of vibration present in the system (see Figure \ref{fig:modes}). Although the PDE has not been implemented using modal synthesis (discussed in Chapter \ref{ch:physMod}), the system can still be decomposed into different modes of vibration, each corresponding to a harmonic frequency. These modes are assumed to vibrate independently, and their weighted sum yields the eventual behaviour of the system.\footnote{Modes of the vibrating string were first discovered by Sauveur in 1701 who said that  ``\textit{especially at night}'' he observed ``\textit{other small sounds}'' on top of the fundamental frequency and coined the terms `node' and `harmonic' \cite{Sauveur1701}.}

\begin{figure}[h]
    \includegraphics[width=\textwidth]{figures/fdtd/oneDWaveOutput.eps}
    \caption{The time-domain and frequency domain output of the 1D wave equation with $f_0 = 735$ Hz and $\fs = 44100$ Hz ($N = 30$ and $\lambda = 1$) and Dirichlet boundary conditions. The system is initialised with a raised cosine described in Eq. \eqref{eq:raisedCos} with $l_0 = 0.2N)$ and $w=4$ and the output is retrieved at $l_\text{out} = 3$. \label{fig:1DWaveOutput}}
\end{figure}

\begin{figure}[h]
    \includegraphics[width=\textwidth]{figures/fdtd/modes.eps}
    \caption{The first $10$ modal shapes of the 1D wave equation with Dirichlet boundary conditions defined for $x\in[0, L]$ (only shown for mode $1$). The modes are normalised to have the same amplitude and vibrate at their respective modal frequencies with the extremes indicated by the black and the grey plot. The number of the shape can be determined by the number of antinodes present in the shape. \label{fig:modes}}
\end{figure}

The number of modes present in the continuous PDE of the 1D wave equation is theoretically infinite. The number present in the discrete FD scheme, however, is determined by the number of moving points in the system. If Dirichlet boundary conditions are used, this means that there are $N-1$ modes, and $N+1$ modes for Neumann boundary conditions. If the CFL condition is satisfied with equality, the frequencies of these modes are integer multiples of the fundamental: $f_m = mf_0$ for mode number $m \in \{1, \hdots, N-1\}$ for Dirichlet and $m \in \{0, \hdots, N\}$ when using Neumann boundary conditions. The frequency of the harmonics -- and even the modal shapes -- can be analytically derived using modal analysis as will be explained in Section \ref{sec:modalAnalysis}.

The amplitude of the different modes depends on the excitation location (and type) and the output location.
Figure \ref{fig:1DWaveOutput}, for example, seemingly shows that the system only exhibits $24$ modes, rather than the $29$ ($N-1$) predicted. As the system is excited at $0.2N$, or in other words, $1/5$\th of the length of the system, this means that every $5$\th mode will be attenuated.
To understand how and/or why this happens, one can refer to Figure \ref{fig:modes} and see that every $5$\th modal shape has a node at $1/5$\th of its length. If the system is excited exactly there, this modal shape will not obtain any energy and will thus not resonate. Similarly, if the system is excited exactly in the middle, every 2\textsuperscript{nd} modal frequency will be attenuated as there is a node present in the corresponding modal shape. The output would then only contain odd-numbered modes.

\subsection{Stability and simulation quality}\label{sec:quality1DWave}
As shown in Eq. \eqref{eq:CFL}, the Courant number needs to abide the CFL condition in order for the scheme to be stable. A system is regarded \textit{unstable} if it exhibits (exponential) unbounded growth. If Neumann boundary conditions (free) are used, it is possible that the system drifts off over time. This does not mean that the system is unstable, it is actually entirely physically possible!\footnote{Imagine a `free' guitar string where the ends are not connected to the nut and bridge of a guitar. The string can be taken far away from the guitar without it breaking or exploding.}

Besides stability, the value of $\lambda$ is closely related to the quality of the simulation.
If $\lambda = 1$, Eq. \eqref{eq:1DwaveFDS} is actually an exact solution to Eq. \eqref{eq:1DwavePDE}, which is quite uncommon in the realm of differential equations! See Figure \ref{fig:1DWaveDisp1}. Identically, if Eq. \eqref{eq:1DWaveStabilityCond} is satisfied with equality, the FD scheme is an exact solution to the PDE, and if $h$ deviates from this condition, the quality of the simulation decreases.

If $\lambda < 1$, the quality of the simulation decreases in an effect called \textit{numerical dispersion}. Dispersion is a phenomenon where some frequencies travel faster through a medium than others, which is desired in some models (see fx. Chapter \ref{ch:stiffString}). Numerical dispersion, however, which is due to numerical inaccuracy, never is! Figure \ref{fig:1DWaveDisp09} shows an example when $\lambda = 0.9$, and one can observe that the wave propagation does not match the ideal case as Figure \ref{fig:1DWaveDisp1} shows. Moreover, bandlimiting effects occur, meaning that the highest frequency that the system can generate decreases. See Figure \ref{fig:1DWaveBandwidth}. Higher modes get `squished' together and are not exact multiples of the fundamental anymore. Section \ref{sec:modalAnalysis} elaborates on how to calculate the exact modal frequencies of a FD implementation of the 1D wave equation.

Finally, if $\lambda > 1$ the system becomes unstable. An example is shown in Figure \ref{fig:1DWaveDisp1001}. Unstable behaviour usually comes in the form of high frequencies (around the Nyquist frequency of $\fs / 2$) growing without bounds.

\def\figSpacing{0.01\textwidth}
\def\figWidth{0.32\textwidth}

\begin{figure}[t]
    \centering
    \subfloat[$\lambda = 1$\label{fig:1DWaveDisp1}]{\includegraphics[width=\figWidth]{figures/fdtd/ulnLambda1.eps}}\hspace{\figSpacing}
    \subfloat[$\lambda = 0.9$ \label{fig:1DWaveDisp09}]{\includegraphics[width=\figWidth]{figures/fdtd/ulnLambda09.eps}}\hspace{\figSpacing}
    \subfloat[$\lambda = 1.001$\label{fig:1DWaveDisp1001}]{\includegraphics[width=\figWidth]{figures/fdtd/ulnLambda1001.eps}}
    \caption{Grid function $\uln$ visualised $\sim\!100$ samples after excitation. (a) If $\lambda = 1$, the solution is exact. (b) If $\lambda < 1$ dispersive behaviour shows. (c) If $\lambda > 1$ the CFL condition in Eq. \eqref{eq:CFL} is not satisfied and the system is unstable.\label{fig:1DWaveDispersion}}
\end{figure}

\begin{figure}[t]
    \centering
    \subfloat[$\lambda = 1$\label{fig:1DWaveBand1}]{\includegraphics[width=\figWidth]{figures/fdtd/bandwidthLambda1.eps}}\hspace{\figSpacing}
    \subfloat[$\lambda = 0.9$ \label{fig:1DWaveBand09}]{\includegraphics[width=\figWidth]{figures/fdtd/bandwidthLambda09.eps}}\hspace{\figSpacing}
    \subfloat[$\lambda = 1.001$\label{fig:1DWaveBand1001}]{\includegraphics[width=\figWidth]{figures/fdtd/bandwidthLambda05.eps}}
    \caption{Frequency spectra of the simulation output. The Courant number is set to (a) $\lambda = 1$, (b) $\lambda = 0.9$ and (c) $\lambda = 0.5$. One can observe that for lower values of $
    \lambda$ the bandwidth of the output decreases drastically. \label{fig:1DWaveBandwidth}}
\end{figure}

In what situation would the stability condition then not be satisfied with equality? As mentioned in Section \ref{sec:gridFunctions}, a continuous domain $\D = [0,L]$ for a system of length $L$ needs to be divided into $N$ equal sections of length $h$ in the discretisation process. A logical step to calculate $N$ would be to divide $L$ by $h$ calculated using Eq. \eqref{eq:1DWaveStabilityCond} satisfied with equality to get the highest possible simulation quality. However, this calculation might not result in an integer value, which $N$ should be! To stay as close to the stability condition as possible, the following calculations are performed in order:
\begin{equation}\label{eq:orderOfCalc}
    h := ck, \quad N := \floor[\frac{L}{h}], \quad h := \frac{L}{N}, \quad \lambda := \frac{ck}{h}.
\end{equation}
In other words, Eq. \eqref{eq:1DWaveStabilityCond} is satisfied with equality and used to calculate integer $N$. After this, $h$ is recalculated based on $N$ and used to calculate the Courant number using Eq. \eqref{eq:courantNumber}. This process assures that $N$ is an integer and that the CFL condition is satisfied, though not necessarily with equality.

To understand why $h$ needs to be recalculated, consider the following example. Consider the 1D wave equation defined over domain $\D = [0, L]$ where $L = 1$. Furthermore, we say that the system should produce a fundamental frequency of $f_0 = 750$ Hz which requires a wave speed of $c = 1500$ m/s according to Eq. \eqref{eq:fundamentalFreq}. If we use the commonly-used sample rate of $\fs = 44100$ Hz, and recalling that $k = 1/\fs$, these values can be filled into \eqref{eq:1DWaveStabilityCond} satisfied with equality and yields $h \approx 0.034$. If we divide the length by the grid spacing, we get $L / h = 29.4$, meaning that exactly $29.4$ intervals of size $h$ fit in the domain $\D$. However, the number of intervals needs to be an integer and -- using Eq. \eqref{eq:orderOfCalc} -- we get $N = 29$. If $h$ is not recalculated according to \eqref{eq:orderOfCalc}, the total length will be $29$ times the grid spacing $h$. This results in $L \approx 0.986$ and is slightly less than the original length of $1$. Although the CFL condition will be satisfied with equality, the fundamental frequency will be slightly higher than desired: $f_0 \approx 760.34$ Hz. If $h$ is recalculated based on $N$, then $L$ and $f_0$ will be unchanged, and the system will have the correct fundamental frequency. The Courant number $\lambda \approx 0.986$ is still very close to satisfying the condition in Eq. \eqref{eq:CFL}, and the decrease in quality will be perceptually irrelevant -- or at the very least, less perceptually relevant than the change in $f_0$ if $h$ is not recalculated.

% Together with the total length of the domain $L$, $h$ can be used to determine into how many intervals $N$ the continuous domain can be subdivided in . As this number is an integer,  This number of intervals $N$ and through that the Courant number $\lambda$ can be calculated using the following operations
% \begin{equation}
%     h := ck, \quad N := \floor[\frac{L}{h}], \quad h := \frac{L}{N} \quad \lambda := \frac{ck}{h}.
% \end{equation}
% where $\floor[\cdot]$ denotes the flooring operation and is necessary as $N$ is an integer. The CFL condition in Eq. \eqref{eq:CFL} thus places a limit on the grid spacing $h$ and with that the number of grid points allowed by the simulation.

\subsubsection{Intuition}
It might not be immediately clear why a too low value for $h$ might cause instability. Some intuition is provided in \cite[Fig. 6.9]{theBible}, but here I would like to provide an alternative, hopefully more tangible way to see this.

In a FD implementation of the 1D wave equation, grid points can only affect their neighbours as seen in update equation \eqref{eq:1DwaveUpdate}. Using the values in Table \ref{tab:1DWaveParams} as an example, $N=30$ and if $\lambda = 1$, it takes exactly $30$ samples, or iterations of Eq. \eqref{eq:1DwaveUpdate}, for a wave to travel from one boundary to the other.

If $h$ were to be chosen to be twice as big so that there are only half as many intervals between the grid points as per Eq. \eqref{eq:orderOfCalc} ($N=15$), the grid points could be set to `affect' their neighbours to a lesser degree. This way, the wave still takes the same amount of time to travel between the boundaries and the fundamental frequency stays approximately the same. This is essentially what happens when $\lambda < 1$ (in this case $\lambda = 0.5$) and can be observed from the update equation in Eq. \eqref{eq:1DwaveUpdate}; the effect that the neighbouring grid points have on each other will indeed be less. The output of the system will have approximately the same fundamental frequency as if $\lambda = 1$, but its partials will be detuned due to numerical dispersion as explained in this section.

If, on the other hand, $h$ were to be chosen to be twice as small so that there are twice as many intervals between grid points ($N = 60$), it is impossible for the waves to travel from one boundary to the other in $30$ samples. If they could interact with their second neighbour, this would be possible, but the FD scheme in \eqref{eq:1DwaveFDS} does not allow for this. Indeed, as $\lambda = 2$ in this case, the effect that the grid points have on each other will be disproportionate. In a way, grid points have too much energy that they can not lose to their neighbours, because their effect should have reached their second neighbour over the course of one sample. The way to solve this would be to halve the time step $k$ (or double the sample rate $\fs$), which would allow grid points to interact with their second neighbours over the course of once the old time step (as this is now divided into two time steps). This also shows in the fact that $\lambda = 1$ again (as halving $k$ cancels out halving $h$) and grid points transfer their energy to their neighbours proportionately again.\todo{figure?}

\subsubsection{Possible solution}
One of the main contributions of the PhD project is published in paper \citeP[G] and summarised in Chapter \ref{ch:dynamicGrid}, where a `fractional' number of intervals is introduced. This removes the necessity of the flooring operation in Eq. \eqref{eq:orderOfCalc} and circumvents the recalculation of $h$ to always satisfy the stability condition with equality while retaining the correct fundamental frequency.


\chapter{Feedback gives rise to fixed point equation.}

What happens if we feed the output
of a function \(f\) back to its input?
But firstly, what do we mean by feeding it back?
We mean we wire the input and the output together so that they are the same,
so that the function's input \(x\) is the same as the function's output \(f~x\).
That statement immediately translates to the equation \( x = f~x \),
which is a \emph{fixed point equation}.
Therefore equalizing the output of a function to its input gives rise to a fixed point equation.

But feedback is more general than that.
Feedback is the case where the input also depends on the output.
If the input is \(x\) and the output is \(f~x\),
we can express feedback as the equation \( x = g~(f~x)~x \),
which we can generalize to \( x = g~f~x \)
by letting \(g\) apply \(x\) to \(f\) internally.
We shall call the equation \( x = g~f~x \) the \emph{general feedback equation}.

But it turns out that we can always write \( x = g~f~x \) as \( x = h~x \)
where \( h = g~f \).
We can always write \( f~x = x \) as \( g~x = 0 \) where \( g~x = f~x - x \).
We can also always write \( x = g~f~x \) as \( 0 = g~f~x \).
What about squared fixed point equation \( (f~x)^2 - x^2 = 0 \)?
The equation \( g~x = 0 \) doesn't mean that \(g\) is zero everywhere.
It means that \(g\) is zero at \(x\), and nothing more, unless quantified.

A fixed point equation is an example of functional equations.
Other examples are Cauchy's functional equations.

\section{Feedback is a special case of fixed point.}

The \emph{general fixed point equation} is \(x = f ~ x\).
In this case, we say that \(x\) is a fixed point of \(f\).

\begin{m:thm}[Feedback as special case of fixed point]
The general feedback equation is a special case of the general fixed point equation.
\begin{proof}
Start with the general feedback equation \( f ~ x = g ~ f ~ x \).
Do eta conversion, producing \( f = g ~ f \).
Then, rename \(f\) to \(x\) and rename \(g\) to \(f\),
producing \( x = f ~ x \),
which is the general fixed point equation.
\end{proof}
\end{m:thm}

That theorem relates cybernetics and fixed point theory.
Thus a feedback is a fixed point of a higher-order function
in the sense that if \(f = g f\), then \(f\) is a fixed point of \(g\).

\begin{m:lem}
    If \( f~x = g~x \) for all \(x\), then \(f = g\).
    \begin{proof}
        We can prove this using set theory.
        A function is an injective relation.
        A relation is a set of ordered pairs.
    \end{proof}
\end{m:lem}

Generally, the equation \(f~x = x\) doesn't hold for all \(x\),
because if it does, \(f\) becomes the identity function.
There are two points of view:
you can fix \(f\) and let \(x\) vary, or you can fix \(x\) and let \(f\) vary.

The set of fixed points of \(f\) is \(\sbn{x}{x = f~x}\).

\begin{m:lem}
    If \( x = f~x \) for all \(x\),
    then \(f\) is an identity function.
\begin{proof}
    The step from the second to the third equation
    comes from the previous lemma.
    \begin{align*}
        x &= f~x & \text{from the antecedent of this lemma}
        \\
        \fid~x &= f~x & \text{due to the antecedent of this lemma}
        \\
        \fid &= f
    \end{align*}
\end{proof}
\end{m:lem}

\section{Recurrence relation is an example feedback.}

Consider the function \( f~x = y \).
If we know \(x\), we can compute \(y\).
If we know \(y\), we can limit the possibilities of \(x\).
They depend on each other.
But they aren't feedback.
Why?
What do we mean by `depend'?

Usually we can rewrite a recurrence relation to a fixed point equation.
For example, we can rewrite \( x_k = f~x_{k-1} \)
to another equation \( x = f \circ (d~x) \),
and then to \( x = (f \circ d)~x \),
where \( d \) is the discrete unit delay function,
which is defined as \( d~x~k = x~(k-1) \).
Recurrence relations describe feedback.
Difference equation is a special case of recurrence relations.
Difference equation is a discrete version of differential equation.

\[
    f~x = c \cdot (s - x)
\]

\section{Closed-loop control is feedback.}

A control system has input signal \(x\), control signal \(y\), output parameter \(z\),
control function \(c\),
response function \(r\),
and follows the equations
\( z~t = r~z~y~t \)
and
\( y~t = c~y~x~t \)
where \(t\) represents time.
We use two equations to model the case
that changing the control signal
does not instantaneously change the output parameter.
The control function affects the output parameter indirectly through the control signal.
The control function cannot directly change the output parameter.
The response function models physical limitations.
\begin{align*}
    z &= r~z~y
    \\ y &= c~y~x
\end{align*}
which can also be stated as the vector feedback equation
\begin{align*}
    \bmat{z \\ y} &= \bmat{r~z~y \\ c~y~x}
\end{align*}

That system of equations suggests that a control system is just two feedback loops.
A control system is a system of feedback equations.
However, system of feedback equations is still a special case of the general feedback equation.

\section{Is this a general feedback equation?}

An equation \(f~x = y\) is a feedback equation iff \(f\) appears in \(y\),
that is if \(f\) appears in both sides of the equation.
A feedback equation is just another name for functional equation.
A feedback equation describes a subset of implicit functions.

The \emph{general feedback equation} is \(f ~ x = g ~ f ~ x\).
It can model every feedback system.
We give the name \emph{system function} to \(f\)
and \emph{wiring function} to \(g\).

An \emph{open} system or a \emph{pure feedforward} system
is a system whose \(g\) does not use \(f\).

\section{What are examples of feedbacks?}

The differential equation \( f~x = d~f~x + d~(d~f)~x \)
is a special case of the general feedback equation \( f~x = g~f~x \)
with \( g~f~x = d~f~x + d~(d~f)~x \).
We can also eta-convert \(g\) to \( g = d + d^2 \).

The Fibonacci equation \( f~x = f~(x-1) + f~(x-2) \)
is a special case of the general feedback equation \( f~x = g~f~x \)
with \( g~f = f \circ (s~1) + f \circ (s~2) \)
where \(s~a~b = b - a\) is flipped subtraction.

Indeed all differential equations and difference equations
are special cases of the general feedback equation.

PID (proportional-integral-derivative)
control systems are special cases of the general feedback equation
with \(g~f~x = c_p \cdot x + c_i \cdot i~f~0~x + c_d \cdot d~f~x\).

IIR (infinite impulse response) filters are special cases of the general feedback equation.

Functional equations are special cases of the general feedback equation.

Thermostats are special cases of the general feedback equation.

Organisms, organizations, and planets are
also special cases of the general feedback equation.

\section{What are positive and negative feedback?}

We say that a system \( f~x = g~f~x \) has positive feedback iff \( d~f~x > 0 \).
We say that it has a negative feedback iff \( d~f~x < 0 \).

\section{What is feedback from dynamical systems of view?}

According to \r{A}str\"om and Murray \cite{AstromMurrayFeedback}:
\begin{quote}
A \emph{dynamical system} is a system whose behavior changes over time, often in response
to external stimulation or forcing. The term \emph{feedback} refers to a situation in which
two (or more) dynamical systems are connected together such that each system
influences the other and their dynamics are thus strongly coupled.
\cite[p. 1]{AstromMurrayFeedback}
\end{quote}

\section{What are examples of fixed points?}

Everything is a fixed point of the identity function \(\fb{x}{x}\).
The constant \(c\) is a fixed point of the constant function \(\fb{x}{c}\).
The function \(\exp\) is a fixed point of the real derivative operator \(d\)
because \(d ~ \exp = \exp\).

Consider \(d^4\), the real derivative operator applied four times.
The function \(\exp\) is a fixed point of \(d^4\).
The sine function is another fixed point.
The cosine function is yet another fixed point.

Eigenvalue-eigenvector pairs are related to fixed points.
If \(a \cdot x = c \cdot x\) where \(a\) is a matrix, \(x\) is a vector, and \(c\) is a scalar,
then \(x\) is a fixed point of \(g\) where \(g~x = (a \cdot x) / c\).


%!TEX program = xelatex

\documentclass[12pt,a4paper]{article}
\usepackage{xeCJK}
\usepackage{amsmath}
%\setmainfont{Times New Roman}
\usepackage{setspace}
\usepackage{caption}
\usepackage{graphicx, subfig}
\usepackage{float}
\usepackage{listings}
\usepackage{booktabs}
\usepackage{setspace}%使用间距宏包
\usepackage{mathtools}
\usepackage{amsfonts}
\usepackage{amssymb,amsmath}
\newcommand{\dd}{\mathrm{d}}
\usepackage{enumitem}

\begin{document}
\title{homework10}
	\author{11611118 郭思源}

\begin{spacing}{1.2}
\section{Homework 10}
\textit{Controlling a population}---The fish and game department in a certain state is planning to issue hunting permits to control the deer population(one deer per permit). It is known that if the deer population falls below a certain level m, the deer will become extinct. It is also known that if the deer population rises above the carrying capacity $M$, the population will decrease back to $M$ through disease and malnutrition.
\begin{enumerate}[label={\textbf{\alph*.}}]
    \item Discuss the reasonableness of the following model for the growth rate of the deer population as a function of time:
    \[
        \frac{\dd P}{\dd t} = rP(M-P)(P-m)
    \]
    where $P$ is the population of the deer and $r$ is a positive constant of proportionality. Include a phase line.
    \\\\
    There should be $P >= 0$.  \\\\
    While the $P = m$, the $\frac{dP}{dt} = 0$, the $P$ will stay at value $m$.  \\\\
    While the $P < m$, the $\frac{dP}{dt} <= 0$, the $P$ will tend to equal 0 or stay at value $m$. As we known, if the deer population falls below a certain level $m$, the deer will become extinct. \\\\
    While the $m < P < M$, the $\frac{dP}{dt} > 0$ and as the $P$ increase, the $\frac{dP}{dt}$ will firstly increase and match the maximum and then decrease, which also matches the rule of deer population development. \\\\
    While the $P = M$, the $\frac{dP}{dt} = 0$, the $P$ will stay at value $M$, which also matches the rule of deer population development.  \\\\
    While the $P > M$, the $\frac{dP}{dt} <= 0$, the $P$ will decrease until $P=M$, and then $\frac{dP}{dt} = 0$ and $P$ will not change, which also matches the rule of deer population development.
    \newpage

    \item Explain how this model differs from the logistic model $\dd P/\dd t=rP(M-P)$. Is it better or worse than the logistic model?
    \\\\
    In the logistic model, if $P <= m$, the population will grow, which means that the rule if the deer population falls below a certain level $m$, the deer will become extinct was ignored in this model. But in our new model, we can simulate the extinct behavior of beer population, so our new model is much better than the logistic model, since it could remain people do not to hunt too many deer which will destroy the deer population.\\\\

    \item Show that if $P>M$ for all $t$, then $\lim_{t\to\infty}P(t)=M$.
	\\\\
	If $P > M$, the $\frac{dP}{dt} < 0$, and if $P = M$, the $\frac{dP}{dt} = 0$, we also known that $P$ is continuous, which means that if $P > M$ at all $t$, the $P$ will decrease until $P(t)=M$ and then become stable, so we can get that $\lim_{t\to\infty}P(t)=M$.\\\\

    \item Discuss the solutions to the differential equation. What are the equilibrium points of the model? Explain the dependence of the steady-state value of $P$ on the initial values of $P$. About how many permits should be issued?
    \\\\
    Equilibrium points : $P=0$, $P=m$ and $P=M$\\
    If $P_0<m$, $P_{steady-state}=0$ \\
    If $P_0=m$, $P_{steady-state}=m$ \\
    If $P_0>m$, $P_{steady-state}=M$ \\
    The population of beer should be as small as possible but we cannot let $P<m$, so the best solution is to control to $P$ to $P=m$. The number of permission should be $(P_{current}-m)$.

\end{enumerate}
\end{spacing}
\end{document}


Thesis aimed to cover, explain and experiment with new approaches to facial recognition. The research in Chapter\,\ref{chapter:research} listed and covered mathematical and engineering principles of construction of convolutional neural networks. It stumbled upon some drawbacks of such design and proposed available solutions. It discusses convolutional neural networks as well as the newest capsule network approach. This throughout review of principles behind successful identification is later transformed into a state-of-the-art market scan, providing a quick insight into publicly available solutions as well as available frameworks. Fundamental benefits and advantages of each framework and solution is listed in the Chapter\,\ref{chapter:solutions}.

The very same chapter also provides an overview of available training data sets are listed, with focus on in-the-wild image data. This review includes a short description of each, accompanied by basic metric. Large scale face databases are listed as well.

This knowledge is later leveraged to create own implementation of a capsule network. The Chapter\,\ref{chapter:implementation} covers the actual design and implementation of this solution. A part from this implementation we also demonstrated how such model can be trained and shown the results.

\section{Experiment discussion}
Overall our implementation proved itself to be successful in recognition of 75\,\% individuals when trained with 11 identities. When trained to learn 42 individuals, the achieved performance drops to  This accuracy decreases dramatically with either more identities added or less training data per identity. These effects correlate but the actual cause may be n one factor only. That would require testing against a data set greater in magnitude, while balanced in amount of images per identity.

In the case of a model trained to recognize 11 different identities, we had available 50 or more images per each individual. These images covered different settings, angles and poses. This might have proved to be a great benefit to our model. On the other hand, the fewer amount of individuals to recognise there is much less parameters to be learned in the network and therefore the error margin is smaller. And since our data are small images of $32\times32$ pixels, the capsule network struggles to successfully match a greater amount of identities.

\section{Improvements and suggestions}

A capsule network provides great power at smaller scales, however, when it is facing big problems, it demands great computational powers and resources. Hence quick prototyping and full scale training result in big differences in network configurations and therefore the observed behaviours. To achieve better results with this type of network, over more identities, it would be necessary to provide sufficient amount of input data, great GPU resources. Enhancing detail on the input data form $32\times32$ pixels to more, comes with a great cost as well, since nearly every layer's parameter count is dependent on amount of input pixels. That means the resource demand is raising again. This problem may seem easy to bypass\,--\,forced retraining of the capsule network on a new label. This is topic is even newer than capsule networks itself and so far was not solved sufficiently.

Capsule networks are a still young approach to machine learning. First announcement of this approach is dated to 2017, which didn't yet provided enough time for this technology to mature. It features new and never before tried algorithms and requires more advanced mathematics and thinking than a standard and nowadays classical convolutional neural network. This thesis covered one of the first attempts to publicize and advertise the possibilities and capabilities of a capsule neural network in the field of facial recognition. The success rate of the experiments does not reach the best of the best in the field, however the results can hopefully serve as a base for future research.


\chapter{Definition of terms}

\begin{center}
\begin{longtable}{lp{10cm}}
\caption{Definition of terms.}\\
\label{tab:definition-of-terms}
\vspace{1em}
%\hline
%\textbf{First entry} & \textbf{Second entry}\\
%\hline
\endfirsthead
\multicolumn{2}{c}{\captionlabelfont\captionfont\tablename\  \thetable{}: \rmfamily Definition of terms (continued).} \\
\vspace{1em}
%\hline
%\textbf{First entry} & \textbf{Second entry} \\
%\hline
\endhead
%\hline
\multicolumn{2}{r}{\textit{Continued on next page}} \\
\endfoot
%\hline
\endlastfoot
$f(\cdot{})$&true state operator (nonlinear function)\\
$\hat{f}(\cdot{})$&supposed state operator (nonlinear function)\\
$h(\cdot{})$&true measurement operator (nonlinear function)\\
$\hat{h}(\cdot{})$&supposed measurement operator (nonlinear function)\\
$g(\cdot{})$&nonlinear transformation of the model output\\
$\boldsymbol\Theta$&constrained parameter space\\
$\mathbf{\Omega}$&constrained state space\\
$\mathbb{R}$&real number space\\
$\boldsymbol\theta$&true parameter values pertaining to the state operator\\
$\hat{\boldsymbol\theta}$&estimate of the parameter values pertaining to the state operator\\
$\boldsymbol\phi$&true parameter values pertaining to the measurement operator\\
$\sigma$&standard deviation of the normal distribution\\
$\mu$&mean of the normal distribution\\
$\nu_{t+1}$&random perturbation of the true output at time $t$\\
$\psi_{t+1}$&random perturbation of the true forcing at time $t$\\
$\omega_{t+1}$&random perturbation of the true state at time $t$\\
$u_t$&true forcing at time $t$\\
$\tilde{u}_t$&observed forcing at time $t$\\
$x_t$&true state at time $t$\\
$\tilde{x}_t$&observed state at time $t$\\
$\hat{x}_t$&predicted state at time $t$\\
$y_t$&true output at time $t$\\
$\tilde{y}_t$&observed output at time $t$\\
$\hat{y}_t$&predicted output at time $t$\\
$\boldsymbol\varepsilon$&observation-prediction residual vector\\
$\mathcal{N}$&normal probability distribution\\
$L(\cdot{})$&log likelihood\\
$p(\boldsymbol\theta|\tilde{\mathbf{x}})$&posterior probability of $\boldsymbol{\theta}$ given the observations $\tilde{\mathbf{x}}$\\
$p(\boldsymbol\theta)$&prior probability of $\boldsymbol{\theta}$\\
$p(\tilde{\mathbf{x}}|\boldsymbol\theta)$&likelihood of the observations $\tilde{\mathbf{x}}$ given the parameters $\boldsymbol{\theta}$\\
$p(\tilde{\mathbf{x}})$&probability of the observations $\tilde{\mathbf{x}}$, or \textit{evidence}\\
$s^2$&estimator of variance $\sigma^2$\\
$n_p$&number of parameters in the vector $\hat{\boldsymbol{\theta}}$\\
$n_o$&number of observations in the vector $\tilde{\mathbf{x}}$\\
$n_e$&number of evaluations of $\hat{f}(\cdot{})$ within a parameter optimization context\\
\end{longtable}
\end{center}