\documentclass[11pt,a4paper]{article}

% These are extra packages that you might need for writing the equations:
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{outlines}
\usepackage{mathtools}

\lstset {language=C++,
		 basicstyle=\ttfamily,
         keywordstyle=\color{blue}\ttfamily,
         stringstyle=\color{red}\ttfamily,
         commentstyle=\color{purple}\ttfamily,
         morecomment=[l][\color{magenta}]{\#},
       	 basicstyle=\tiny}

% You need the following package in order to include figures in your report:
\usepackage{graphicx}

% With this package you can set the size of the margins manually:
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}

\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\begin{document}

% Enter the exercise number, your name and date here:
\noindent\parbox{\linewidth}{
 \parbox{.25\linewidth}{ \large ICP, Exercise 12 }\hfill
 \parbox{.5\linewidth}{\begin{center} \large Beat Hubmann \end{center}}\hfill
 \parbox{.2\linewidth}{\begin{flushright} \large Dec 07, 2018 \end{flushright}}
}
\noindent\rule{\linewidth}{2pt}

\section{Introduction}

As a continuation of last week's work on exercise 11, the conjugate gradient method was
implemented to also solve
the discretized Poisson equation in two dimensions for point charges in a grounded box.
This time around, the time to solution was compared to a high performance library Cholesky solver as well
as to the library Conjugate Gradient solver.

\section{Algorithm Description}

Repeating last week's description:\\
For all cases, the two-dimensional Poisson equation(equation~\ref{eqn:1}) on $\Omega$ is
discretized using second-order central finite differences in both the x- and the y-direction (equation~\ref{eqn:2}).
Both axes share a common grid spacing of $\Delta x= \frac{1}{N+1}$ where $N$ is the number of interior points
per axis direction on the grid.
Following the established finite difference method procedure to employ natural ordering, the left-hand side of equation~\ref{eqn:2} then can be written
in form of an $N*N \times N*N$ matrix $\vec{A}$ while the values of $\phi$ on the grid get unrolled into a vector $\vec{b}$ of size $N*N$ on the right-hand side (equation~\ref{eqn:3}).\\
The resulting matrix $A$ is both sparse and block tridiagonal.

\begin{equation}
\Delta \Phi = -\phi \quad \text{on}\ \Omega = (0, 1) \times (0,1)
\label{eqn:1}
\end{equation}

\begin{equation}
4x_{i,j} - x_{i-1, j} - x_{x+1, j} - x_{i, j-1} - x_{i, j+1} = -(\Delta x)^2 \cdot \rho(x_{i,j})
\label{eqn:2}
\end{equation}

\begin{equation}
Ax = b
\label{eqn:3}
\end{equation}

\subsection{Conjugate gradient method}
The conjugate gradient method is the most well known member of the family of Krylov subspace methods~\cite{elman}.
Given the system of equations $\vec{Ax}=\vec{b}$ with $\vec{A}$ a symmetric positive definite (SPD) matrix, the method iteratively constructs a solution $\vec{x}^{(t)} \in \mathcal{K}_t(\vec{A}, \vec{r}^{(0)})$ using a starting solution $\vec{x}^{(0)} = (0,0,\ldots, 0)^T$,
the residual $\vec{r}^{(0)}=\vec{b} - \vec{Ax}^{(0)}$ and the associated Krylov subspace $\mathcal{K}_t(\vec{A}, \vec{r}^{(0)})=\text{span}\{\vec{x}, \vec{Ax}, \vec{A}^2\vec{x}, \ldots, \vec{A}^{t-1}\vec{x}\}$.
In this implementation, we use the simple Richardson iteration with $\vec{x}^{(t)}=\vec{x}^{(t-1)} + \alpha_{t-1}\vec{x}^{(t-1)}$ where $\alpha$ is a scalar factor
calculated as shown in the outline below. $\vec{M}^{-1}\ =\ \frac{\delta_{ij}}{\vec{A}_{ii}}$ is the Jacobi preconditioner matrix. Setting $\vec{M} = \vec{M}^{-1} = \vec{\mathcal{I}}$ instead trivially falls back to
using no preconditioner:
\begin{outline}
\1 $\vec{r}^{(0)} \leftarrow \vec{b} - \vec{Ax}^{(0)}$
\1 $\vec{z}^{(0)} \leftarrow \vec{M}^{-1}\vec{r}^{(0)}$
\1 $\vec{p} \leftarrow \vec{z}^{(0)}$
\1 Iterate until convergence and/or iteration limit:
    \2 $\alpha \leftarrow \frac{(\vec{r}^{(t)})^T \vec{z}^{(t)}}{(\vec{p})^T \vec{Ap}}$
    \2 $\vec{x} \leftarrow \vec{x} + \alpha\vec{p}$
    \2 $\vec{r}^{(t)} \leftarrow \vec{r}^{(t-1)} - \alpha\vec{Ap}$
    \2 check preconditioned residual $\norm{\vec{r}^{(t)}}_2$ for convergence; stop if reached
    \2 $\vec{z}^{(t)} \leftarrow \vec{M}^{-1} \vec{r}^{(t-1)}$
    \2 $\beta \leftarrow \frac{(\vec{z}^{(t)})^T \vec{r}^{(t)}}{(\vec{z}^{(t-1)})^T \vec{r}^{(t-1)}}$
    \2 $\vec{p} \leftarrow \vec{z}^{(t)} = \beta \vec{p} $
    \2 $\vec{r}^{(t-1)} \leftarrow \vec{r}^{(t)}$
    \2 $\vec{z}^{(t-1)} \leftarrow \vec{z}^{(t)}$
\1 return $\vec{x}$
\end{outline}

\section{Results}

The program was implemented as described above and submitted with this report. \\
The conjugate gradient method was iterated until the residual's norm went below the set treshold: $\norm{\vec{r}}_{2} \le 10^{-4}$.
The conjugate gradient method took only $t=82$ iterations and $\sim 1 \text{ms}$ which compares very favourably with the Jacobi relaxation method ($t=3478$ iterations in $\sim 45 \text{ms}$)
and the Gauss-Seidel method ($t=1922$ iterations in $\sim 6400 \text{ms}$ examined in exercise 11. For comparison, Eigen's optimized library Cholesky method
solver obtained the reference solution in $\sim 10 \text{ms}$ while Eigen's own conjugate gradient method set to use a complete matrix took $\sim 3 \text{ms}$.\\
The conjugate gradient method with the set residual treshold reached a very minor deviation from the reference Cholesky solution:\\
$\norm{\vec{x}^*_{\text{Conjugate gradient}} - \vec{x}^*_{\text{Cholesky}}}_{2} \approxeq 0.0002$. \\
The heat map for the conjugate gradient solver is shown in figure~\ref{fig:1}.

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=1.2]{figure_1.png}
\end{center}
\caption{Conjugate gradient solver reference solution for Poisson equation with point charges at $(0.25, 0.75)$, $(0.75, 0.25)$; grid parameter $N=50$.}
\label{fig:1}
\end{figure}

\section{Discussion}
The results are mostly are as expected, the exception being that the Jacobi preconditioner didn't make a difference for the system under consideration.
Then again, neither is the Jacobi preconditioner sophisticated, nor does the given system setup (2D finite differences on a small system with $N=50$) actually
mandate using a preconditioner.

\begin{thebibliography}{99}

\bibitem{elman}
Elman, H., Silvester, D., Wathen, A. \\
\emph{Finite Elements and Fast Iterative Solvers},\\
Oxford University Press,\\
2014.

% \bibitem{metropolis}
% Metropolis, N.,
% Rosenbluth, A.W.,
% Rosenbluth, M.N.,
% Teller, A.H.,
% Teller, E.\\
% \emph{Equations of State Calculations by Fast Computing Machines},\\
% Journal of Chemical Physics. 21 (6): 1087,\\
% 1953.

% \bibitem{herrmann}
% 	Herrmann, H. J.,
% 	Singer, H. M.,
% 	Mueller L.,
% 	Buchmann, M.-A.,\\
% 	\emph{Introduction to Computational Physics - Lecture Notes},\\
% 	ETH Zurich,\\
% 	2017.

% \bibitem{Gottschling}
% Gottschling, Peter\\
% \emph{Discovering Modern C++},\\
% Addison-Wesley,\\
% 2016.

\end{thebibliography}

\end{document}


\section{Air Velocity} \label{sec:velocity}
Did you ever feel the wind blow? Most probably. That's what we will be calculating here. How hard the wind will blow. This is noted as velocity, how fast something moves.

\subsection{Equation of State and the Incompressible Atmosphere}
The equation of state relates one or more variables in a dynamical system (like the atmosphere) to another. The most common equation of state in the atmosphere is the ideal gas equation as
described by \autoref{eq:ideal gas} \cite{idealGas}. The symbols in that equation represent:

\begin{itemize}
    \item $p$: The gas pressure (\si{Pa}).
    \item $V$: The volume of the gas (\si{m^3}).
    \item $n$: The amount of moles in the gas (\si{mol}).
    \item $R$: The Gas constant as defined in \autoref{sec:gas constant} (\si{JK^{-1}mol^{-1}}) \cite{idealGas}.
    \item $T$: The temperature opf the gas ($K$).
\end{itemize}

If we divide everything in \autoref{eq:ideal gas} by $V$ and set it to be unit (in this case, set it to be exactly $1$ \si{m^3}) we can add in the molar mass in both the top and bottom parts of
the division as show in \autoref{eq:gas unit}. We can then replace $\frac{nm}{V}$ by $\rho$ the density of the gas (\si{kgm^{-3}}) and $\frac{R}{m}$ by $R_s$ the specific gas constant (gas
constant that varies per gas in \si{JK^{-1}mol^{-1}}) as shown in \autoref{eq:state gas}. The resulting equation is the equation of state that you get that most atmospheric physicists use when
talking about the atmosphere \cite{simon}.

\begin{subequations}
    \begin{equation}
        \label{eq:ideal gas}
        pV = nRT
    \end{equation}
    \begin{equation}
        \label{eq:gas unit}
        p = \frac{nR}{V}T = \frac{nmR}{Vm}T
    \end{equation}
    \begin{equation}
        \label{eq:state gas}
        p = \rho R_sT
    \end{equation}
\end{subequations}

The pressure is quite important, as air moves from a high pressure point to a low pressure point. So if we know the density and the temperature, then we know the pressure and we can work out
where the air will be moving to (i.e. how the wind will blow). In our current model, we know the atmospheric temperature but we do not know the density. For simplicities sake, we will now assume
that the atmosphere is Incompressible, meaning that we have a constant density. Obviously we know that air can be compressed and hence our atmosphere can be compressed too but that is not
important enough to account for yet, especially considering the current complexity of our model.

The code that corresponds to this is quite simple, the only change that we need to make in \autoref{eq:state gas} is that we need to replace $T$ by $T_a$, the temperature of the atmosphere. As
$T_a$ is a matrix (known to programmers as a double array), $p$ will be a matrix as well. Now we only need to fill in some values. $\rho = 1.2$\cite{densityAir}, $R_s = 287$\cite{specificGasConstantAir}.

\subsection{The Momentum Equations} \label{sec:momentum}
The momentum equations are a set of equations that describe the flow of a fluid on the surface of a rotating body. For our model we will use the f-plane approximation. The equations corresponding
to the f-plane approximation are given in \autoref{eq:x momentum} and \autoref{eq:y momentum} \cite{momentumeqs}. Note that we are ignoring vertical movement, as this does not have a significant
effect on the whole flow. All the symbols in \autoref{eq:x momentum} and \autoref{eq:y momentum} mean:

\begin{itemize}
    \item $u$: The east to west velocity (\si{ms^{-1}}).
    \item $t$: The time (\si{s}).
    \item $f$: The coriolis parameter as in \autoref{eq:coriolis}.
    \item $v$: The north to south velocity (\si{ms^{-1}}).
    \item $\rho$: The density of the atmosphere (\si{kgm^{-3}}).
    \item $p$: The atmospheric pressure (\si{Pa}).
    \item $x$: The local longitude coordinate (\si{m}).
    \item $y$: The local latitude coordinate (\si{m}).
\end{itemize}

If we then define a vector $\bar{u}$ as $(u, v, 0)$, we can rewrite both \autoref{eq:x momentum} as \autoref{eq:x momentum laplace}. Here $\nabla u$ is the gradient of $u$ in both $x$ and $y$
directions. Then if we write out $\nabla u$ we get \autoref{eq:x momentum final}. Similarly, if we want to get $\delta v$ instead of $\delta u$ we rewrite \autoref{eq:y momentum} to get
\autoref{eq:y momentum laplace} and \autoref{eq:y momentum final}.

\begin{subequations}
    \begin{equation}
        \label{eq:x momentum}
        \frac{Du}{Dt} - fv = -\frac{1}{\rho} \frac{\delta p}{\delta x}
    \end{equation}
    \begin{equation}
        \label{eq:y momentum}
        \frac{Dv}{Dt} - fu = -\frac{1}{\rho} \frac{\delta p}{\delta y}
    \end{equation}
    \begin{equation}
        \label{eq:x momentum laplace}
        \frac{\delta u}{\delta t} + \bar{u} \cdot \nabla u - fv = -\frac{1}{\rho}\frac{\delta p}{\delta x}
    \end{equation}
    \begin{equation}
        \label{eq:y momentum laplace}
        \frac{\delta v}{\delta t} + \bar{u} \cdot \nabla v - fu = -\frac{1}{\rho}\frac{\delta p}{\delta y}
    \end{equation}
    \begin{equation}
        \label{eq:x momentum final}
        \frac{\delta u}{\delta t} + u\frac{\delta u}{\delta x} + v\frac{\delta u}{\delta y} - fv = -\frac{1}{\rho}\frac{\delta p}{\delta x}
    \end{equation}
    \begin{equation}
        \label{eq:y momentum final}
        \frac{\delta v}{\delta t} + u\frac{\delta v}{\delta x} + v\frac{\delta v}{\delta y} - fu = -\frac{1}{\rho}\frac{\delta p}{\delta y}
    \end{equation}
\end{subequations}

With the gradient functions defined in \autoref{alg:gradient x} and \autoref{alg:gradient y}, we can move on to the main code for the momentum equations. The main loop is shown in
\autoref{alg:stream3}. Do note that this loop replaces the one in \autoref{alg:stream2v2} as these calculate the same thing, but the new algorithm does it better.

\begin{algorithm}
    \caption{Calculating the flow of the atmosphere (wind)}
    \label{alg:stream3}
    $S_{xu} \leftarrow \texttt{gradient\_x}(u, lat, lon)$ \;
    $S_{yu} \leftarrow \texttt{gradient\_y}(u, lat, lon)$ \;
    $S_{xv} \leftarrow \texttt{gradient\_x}(v, lat, lon)$ \;
    $S_{yv} \leftarrow \texttt{gradient\_y}(v, lat, lon)$ \;
    $S_{px} \leftarrow \texttt{gradient\_x}(p, lat, lon)$ \;
    $S_{py} \leftarrow \texttt{gradient\_x}(p, lat, lon)$ \;
    \For{$lat \leftarrow 1$ \KwTo $nlat - 1$}{
        \For{$lon \leftarrow 0$ \KwTo $nlon$}{
            $u[lat, lon] \leftarrow u[lat, lon] + \delta t \frac{-u[lat, lon]S_{xu} - v[lat, lon]S_{yu} + f[lat]v[lat, lon] - S_{px}}{\rho}$ \;
            $v[lat, lon] \leftarrow v[lat, lon] + \delta t \frac{-u[lat, lon]S_{xv} - v[lat, lon]S_{yv} - f[lat]u[lat, lon] - S_{py}}{\rho}$ \;
        }
    }
\end{algorithm}

\subsection{Improving the Coriolis Parameter}
Another change introduced is in the coriolis parameter. Up until now it has been a constant, however we know that it varies along the latitude. So let's make it vary over the latitude. Recall
\autoref{eq:coriolis}, where $\Theta$ is the latitude. Coriolis ($f$) is currently defined in \autoref{alg:gradient}, so let's replace it with \autoref{alg:coriolis}.

\begin{algorithm}
    \caption{Calculating the coriolis force}
    \label{alg:coriolis}
    \SetAlgoLined
    $\Omega \leftarrow 7.2921 \cdot 10^{-5}$ \;

    \For{$lat \leftarrow -nlat$ \KwTo $nlat$}{
        $f[lat] \leftarrow 2\Omega \sin(lat \frac{\pi}{180})$ \;
    }
\end{algorithm}

\subsection{Adding Friction}
In order to simulate friction, we multiply the speeds $u$ and $v$ by $0.99$. Of course there are equations for friction but that gets complicated very fast, so instead we just assume that we
have a constant friction factor. This multiplication is done directly after \autoref{alg:stream3} in \autoref{alg:stream4v1}.

\subsection{Adding in Layers}
With adding in atmospheric layers we need to add vertical winds, or in other words add the $w$ component of the velocity vectors. We do that by editing \autoref{alg:stream3}. We change it to
\autoref{alg:velocity}. Here we use gravity ($g$) instead of the coriolis force ($f$) and calculate the change in pressure. Therefore we need to store a copy of the pressure before we do any
calculations. This needs to be a copy due to aliasing \footnote{Aliasing is assigning a different name to a variable, while it remains the same variable. Take for instance that we declare a
variable $x$ and set it to be $4$. Then we say $y \leftarrow x$, which you might think is the same as saying they $y \leftarrow 4$ but behind the screen it is pointing to $x$. So if $x$ changes,
then so does $y$.}. Since we use pressure as the vertical coordinate, we must be able to convert that into meters (why we opted for pressure is explained in \autoref{sec:rad layers}) in order to
be able to say something sensible about it. To do that we need the concept of geopotential height.

\subsubsection{Dimensionless Pressure}
Geopotential height is similar to geometric height, except that it also accounts for the variation in gravity over the planet \cite{geopot}. One could say that geopotential height is the
"gravity adjusted" height. That means that it is similar to the height, but not exactly the same. Height is a linear function, whereas the geopotential height is not, though it is very similar
to a linear function if you would plot it. Now to convert easily to and from potential temperature into temperature, we need another function which is known as the Exner function. The Exner
function is a dimensionless \footnote{Being dimensionless means that there is no dimension (unit) attached to the number. This is useful for many applications and is even used in daily life. For
instance when comparing price rises of different products, it is way more useful to talk about percentages (who are unitless) instead of how much you physically pay more (with your favourite
currency as the unit).} pressure. The Exner function is shown in \autoref{eq:exner} \cite{verticalcoords}. The symbols in the equation are:

\begin{itemize}
    \item $c_p$: The specific heat capacity of the atmosphere.
    \item $p$: Pressure (\si{Pa}).
    \item $p_0$: Reference pressure to define the potential temperature (\si{Pa}).
    \item $R$: The gas constant $8.3144621$ (\si{J(mol)^{-1}K}).
    \item $T$: The absolute temperature (\si{K}).
    \item $\theta$: the potential temperature (\si{K}).
\end{itemize}

Since the right hand side contains what we want to convert to and from, we can do some basic rewriting, which tells us what we need to code to convert potential temperature in absolute
temperature and vice versa. This is shown in \autoref{eq:temp exner} and \autoref{eq:potential temp exner} respectively.

\begin{subequations}
    \begin{equation}
        \label{eq:exner}
        \Pi = c_p(\frac{p}{p_0})^{\frac{R}{c_p}} = \frac{T}{\theta}
    \end{equation}
    \begin{equation}
        \label{eq:temp exner}
        T = \Pi\theta
    \end{equation}
    \begin{equation}
        \label{eq:potential temp exner}
        \theta = \frac{T}{\Pi}
    \end{equation}
\end{subequations}

Now onto some code. Let us initialise $\Pi$ before we do any other calculations. This code is already present in the control panel section (\autoref{sec:cp}) as that is where it belongs, so for
further details please have a look at the code there. Now onto the geopotential height.

\subsubsection{Geopotential Height}
As stated before, geopotential height is similar to geometric height, except that it also accounts for the variation in gravity over the planet. One could say that geopotential height is the
"gravity adjusted" height. That means that it is similar to the height, but not exactly the same. Height is a linear function, whereas the geopotential height is not, though it is very similar
to a linear function if you would plot it. Now one could ask why we would discuss dimensionless pressure before geopotential height. The answer is quite simple, in order to define geopotential
height, we need the Exner function to define it. Or rather, we need that function to convert potential temperature into geopotential height. How those three are related is shown in
\autoref{eq:geopot}. Then with a little transformation we can define how the geopotential height will look like, as shown in \autoref{eq:geopot final}. The symbols in both equations are:

\begin{itemize}
    \item $\Pi$: The Exner function.
    \item $\theta$: Potential temperature (\si{K}).
\end{itemize}

\begin{subequations}
    \begin{equation}
        \label{eq:geopot}
        \theta + \frac{\delta\Phi}{\delta\Pi} = 0
    \end{equation}
    \begin{equation}
        \label{eq:geopot final}
        \delta\Phi = -\theta\delta\Pi
    \end{equation}
\end{subequations}

Now to turn this into code we need to be careful about a few things. First we are talking about a change in geopotential height here, so defining one level of geopotential height means that it
is dependent on the level below it. Second this calculatoin needs potential temperature and therefore it should be passed along to the velocity calculations function. With those two things out
of the way, we get the code as shown in \autoref{alg:geopot}. Note that \texttt{Smooth3D} refers to \autoref{alg:smooth}.

\begin{algorithm}
    \caption{Calculating the geopotential height}
    \label{alg:geopot}
    \For{$level \leftarrow 1$ \KwTo $nlevels$}{
        $\Phi[:, :, level] \leftarrow \Phi[:, :, level - 1] - T_{pot}[:, :, level](\Pi[level] - \Pi[level - 1])$ \;
    }
    $\Phi \leftarrow \texttt{Smooth3D}(\Phi, smooth_t)$ \;
\end{algorithm}

\subsubsection{Finally Adding in the Layers}
Now with the geopotential height and dimensionless pressure out of the way, we need to use those two concepts to add layers to the velocity calculations. Before we dive into the code however,
there are slight changes that we need to discuss. The equation shown in \autoref{eq:velocity} is the primitive equation (as discussed in \autoref{sec:primitive}). The momentum equations are
gesostrphic momentum, which are a special form of the primitive equation. Since this whole system must remain in equilibrium, we need to set the right hand side to $0$ as shown in
\autoref{eq:vel eq}. Now let us rewrite \autoref{eq:velocity} into \autoref{eq:velocity int}. We replaze $z$ with pressure as that is our vertical coordinate. $\omega$ is the velocity of the
pressure field, as defined in \autoref{eq:vert vel}. Note that $p_k$ is the pressure for layer $k$ and $p_0$ is the pressure at the planet surface. Now we need to turn the velocity of the
pressure field into the velocity of a packet of air (see it as a box of air being moved), which is done in \autoref{eq:vertical velocity}. Here $\rho$ is the density and $g$ is gravity
(\si{ms^{-2}}).

\begin{subequations}
    \begin{equation}
        \label{eq:velocity}
        \frac{\delta T}{\delta x} + \frac{\delta T}{ \delta y} + \frac{\delta T}{\delta z} = \nabla T
    \end{equation}
    \begin{equation}
        \label{eq:vel eq}
        \nabla T = 0
    \end{equation}
    \begin{equation}
        \label{eq:velocity int}
        \frac{\delta u}{\delta x} + \frac{\delta v}{\delta y} + \frac{\delta\omega}{\delta p} = 0
    \end{equation}
    \begin{equation}
        \label{eq:vert vel}
        \omega_k = -\int^{p_k}_{p_0}\frac{\delta u}{\delta x} + \frac{\delta v}{\delta y} dp
    \end{equation}
    \begin{equation}
        \label{eq:vertical velocity}
        w = \omega \rho g
    \end{equation}
\end{subequations}

But I hear you say, what is the density? Since we have moved to pressure coordinates we can actually calculate the density rather than store it. This is done in \autoref{eq:density}, where each
symbol means:

\begin{itemize}
    \item $\rho$: The density of the atmosphere.
    \item $p$: The pressure of the atmosphere (\si{Pa}).
    \item $c$: Specific heat capacity of the atmosphere (\si{JKg^{-1}K^{-1}}).
    \item $T$: Temperature of the atmosphere (\si{K}).
\end{itemize}

\begin{equation}
    \label{eq:density}
    \rho = \frac{p}{cT}
\end{equation}

Finally, let us convert \autoref{eq:vertical velocity} to code in \autoref{alg:velocity}. Here $T_{trans}$ is a call to the algorithm as described in \autoref{alg:temp to pot}.
\texttt{gradient\_x}, \texttt{gradient\_y} and \texttt{gradient\_z} are calls to \autoref{alg:gradient x}, \autoref{alg:gradient y} and \autoref{alg:gradient z} respectively.

\begin{algorithm}
    \caption{Calculating the flow of the atmosphere (wind)}
    \label{alg:velocity}
    //\texttt{The following variables are function calls to algorithms and their shorthand notations are used in the loops}\\
    $S_{xu} \leftarrow \texttt{gradient\_x}(u, lat, lon, layer)$ \;
    $S_{yu} \leftarrow \texttt{gradient\_y}(u, lat, lon, layer)$ \;
    $S_{xv} \leftarrow \texttt{gradient\_x}(v, lat, lon, layer)$ \;
    $S_{yv} \leftarrow \texttt{gradient\_y}(v, lat, lon, layer)$ \;
    $S_{zu} \leftarrow \texttt{gradient\_z}(u[lat, lon], p_z, layer)$ \;
    $S_{zv} \leftarrow \texttt{gradient\_z}(v[lat, lon], p_z, layer)$ \;
    $S_{px} \leftarrow \texttt{gradient\_x}(geopot, lat, lon, layer)$ \;
    $S_{py} \leftarrow \texttt{gradient\_y}(geopot, lat, lon, layer)$ \;

    //\texttt{The following variables are real variables}\\
    $nlat \leftarrow \Phi.length$ \;
    $nlon \leftarrow \Phi[0].length$ \;
    $u_t \leftarrow $ array like $u$ \;
    $v_t \leftarrow $ array like $v$ \;
    $w_t \leftarrow $ array like $w$ \;
    \For{$lat \leftarrow 1$ \KwTo $nlat - 1$}{
        \For{$lon \leftarrow 0$ \KwTo $nlon$}{
            \For{$layer \leftarrow 0$ \KwTo $nlevels$}{
                $u_t[lat, lon, layer] \leftarrow u[lat, lon, layer] + \delta t \frac{-u[lat, lon, layer]S_{xu} - v[lat, lon, layer]S_{yu} + f[lat]v[lat, lon, layer] - S_{px}}
                    {10^5u[lat, lon, layer]}$ \;
                $v_t[lat, lon, layer] \leftarrow v[lat, lon, layer] + \delta t \frac{-u[lat, lon, layer]S_{xv} - v[lat, lon, layer]S_{yv} - f[lat]u[lat, lon, layer] - S_{py}}
                    {10^5v[lat, lon, layer]}$ \;
            }
        }
    }

    $T_a \leftarrow T_{trans}(T_{pot}, p_z, \texttt{False})$ \;

    \For{$lat \leftarrow 2$ \KwTo $nlat - 2$}{
        \For{$lon \leftarrow 0$ \KwTo $nlon$}{
            \For{$level \leftarrow 1$ \KwTo $nlevels$}{
                $w_t[lat, lon, level] \leftarrow  w_t[lat, lon, level - 1] - \frac{(p_z[level] - p_z[level - 1])p_z[level]g(S_{xu} + S_{yv})}{C_pT_a[lat, lon, layer]}$ \;
            }
        }
    }

    $u \leftarrow u + u_t$ \;
    $v \leftarrow v + v_t$ \;
    $w \leftarrow w + w_t$ \;
    $p_0 \leftarrow copy(p)$ \;
\end{algorithm}


\section{Basics of Reinforcement Learning}
\label{sec:reinforcement_learning}

\begin{frame}{Markov Decision Processes}
	\begin{block}{Reinforcement Learning}
		General class of algorithms that allow an agent to learn how to behave
		in a stochastic and possibly unknown environment by trial-and-error.
	\end{block}

	\begin{block}{Markov Decision Process (MDP)}
		stochastic dynamical system specified by $<\S, \A, \calP, \calR, \gamma>$
		\begin{enumerate}
			\item $(\S, \calS)$ is a measurable state space
			\item $(\A, \calA)$ is a measurable action space
			\item $\calP: \S \times \A \times \calS \to \R$ is a Markov transition kernel
			\item $\calR: \S \times \A \to \R$ is a reward function
			\item $0 < \gamma < 1$ is the discount factor.
		\end{enumerate}
	\end{block}
\end{frame}

\begin{frame}{Interaction Between Agent and Environment}
\begin{figure}[t]
	\centering
	\begin{tikzpicture}[node distance = 6em, auto, thick]
		\node [block] (Agent) {Agent\\$\pi$};
		\node [block, below of=Agent] (Environment) {Environment\\$\calP, \calR$};
		\path [line] (Agent.0) --++ (4em,0em) |- node [near start]{Action $a_t$} (Environment.0);
		\path [line] (Environment.190) --++ (-6em,0em) |- node [near start]{State  $s_{t}$} (Agent.170);
		\path [line] (Environment.170) --++ (-4.25em,0em) |- node [near start, right] {Reward $r_{t+1}$} (Agent.190);
	\end{tikzpicture}
\end{figure}
\end{frame}

\begin{frame}{Policy and Value Function}
	\begin{block}{Policy}
		 A policy is a function $\pi: \S \times \A \to \R$ such that, $\forall s \in \S$, $C \mapsto \pi(s,C)$ is a probability distribution over $(\A, \calA)$
	\end{block}
	\begin{block}{Return}
		\begin{equation*}
			G_t = \sum^{\infty}_{t=0} \gamma^t R_{t+k+1}
		\end{equation*}
	\end{block}
	\begin{block}{Value Function}
		\begin{equation*}
			V_\pi(s) = \E[\pi]{G_t|S_t = s}
		\end{equation*}
	\end{block}
	\begin{block}{Agent's goal}
		Select a policy $\pi^*$ that maximizes his expected return in all possible states. This policy is called \emph{optimal}.
	\end{block}
\end{frame}

\begin{frame}{Policy Gradient Methods}
	\begin{block}{Key idea}
	$\pi_*$ is approximated using a parametrized policy $\pi_{\theta^*}(s,a)$, where
	\begin{equation*}
		\theta^* = \argmax_{\theta \in \Theta} J(\theta) = V_{\pi_\theta}(s_0)
	\end{equation*}
	Using gradient descent, we have the following update scheme
	\begin{equation*}
		\theta_{k+1} = \theta_k + \alpha_k \nabla_\theta J\left(\theta_k\right)
	\end{equation*}
	\end{block}

	\begin{block}{Policy Gradient Theorem}
			Let $\pi_\theta$ be a differentiable policy. The policy gradient for the average reward formulation is given by
			\begin{equation*}
				\nabla_\theta J(\theta) =
				\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
				\pi_\theta(S,A) Q_{\theta}(S, A)}
			\end{equation*}
			$d^\theta$ is the stationary distribution of the Markov chain induced by $\pi_\theta$.
	\end{block}
\end{frame}

\begin{frame}{Monte-Carlo Policy Gradient Method}
	\begin{figure}[h]
		\centering
		\includegraphics[width=1.0\framewidth]{Images/gpomdp}
	\end{figure}
\end{frame}


\chapter{Hybrid Neural Network for State Inference} \label{sec:approach}
In this section, I describe my proposed deep learning approach for the black-box state inference task, in details.

\section{The Model Architecture}
\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{ASE_files/GeneralConvolutionalNet.pdf}
    \caption{The input and output signals of the black-box system are captured as a multivariate time series; they are processed in a deep neural network that consists of 3 sections: convolutional, recurrent, and dense (fully connected) to predict the system's internal state and its changes over time.}
    \label{fig:general_net}
\end{figure*}

The goal of this study is to infer the states of a running software system, over time. Given that my assumption is we don't have access to the source code (or part of it), I only leverage the values of inputs and outputs of the system, over time.
As can be seen in figure~\ref{fig:general_net}, I capture all the inputs and outputs of the system as a time series and then process it in a DNN.
The architecture of my proposed model is a hybrid DNN which is inspired by models proposed in the field of Human Activity Recognition (HAR). This task is quite similar to the subject of this thesis in the sense that they both take in a multivariate time series-data (from sensor readings) and output the state of the system that generated those readings (see section~\ref{sec:related_work_har} for more details on HAR papers).
This DNN is made of three parts in sequence: 1) Convolutional, 2) Recurrent, and 3) Fully connected layers. This architecture addresses the aforementioned traditional methods' challenges; each part serves a different purpose in this process, as follows.

Convolutions, being more generalized than simple sliding windows, can discover patterns and features in the signals, both in temporal and in spatial (how signals affect each other) dimensions \cite{wang2017time}.
The convolutional layers' flexibility allows them to learn some typical preprocessing operations. For example a moving average or a discrete derivative can be learned as simple convolutional filters. They also help the model to be more resilient to varying time delays between noticing a deviation in input signals and the reaction that will appear in the output signals. Applying convolutional layers in sequence has been shown to result in each layer learning more complex features than the previous layers \cite{zeiler2014visualizing}.
The number of layers, filters, and the kernel size are hyper-parameters that should be selected based on the size of data and the complexity of the system being modeled.
Using a sequence of convolutions with a) increasing number of filters and the same kernel size, b) same number of filters and increasing kernel size, and c) decreasing filters with increasing kernel sizes are all different approaches that have been used in the literature by well-known architectures such as VGG and U-net \cite{simonyan2014very, ronneberger2015u}.
I will discuss more details of CNN layers in my approach in Section~\ref{sec:architecture_detail}.

Convolutions are quite powerful in discovering local features. To capture long-term features, recurrent layers which learn sequences of data are leveraged. For example, in my case, they can learn that ``accelerate'' and ``take off'' states only happen in the start of the states sequence, and each ``take off'' state is usually followed by a ``climb'' state. The type of recurrent cell to use (LSTM, GRU, etc.), how many cells to unravel in the layer, and the number of layers are also hyper-parameters that need to be tuned depending on the size and complexity of the system under study.

Finally, one or more dense (fully connected) layers in the end are a common way of reducing the dimensions to match expected output dimensions.
If there are only two states, the last layer can have a sigmoid activation function and be of shape $L$ (the length of the input), otherwise, to match the one-hot encoding of labels, an output of shape $L\times N_s$ with softmax activation along the second axis ($N_s$) is required ($N_s$ being the number of possible states).

In terms of loss function to optimize in the training process, a good choice is a dice overlap loss function, which is used in image semantic segmentation tasks, as well. An important property of this loss function is not getting negatively affected by class imbalances \cite{milletari2016v, sudre2017generalised}.

% To make it more clear, compare it to applying a convolution (or a discrete derivative) to each signal independently. It can only learn temporal features, nothing about the interactions between the signals.
% Using convolutional layers instead of predefined kernels allows the model to learn which subset of possible connections and interactions (which can involve two or more variables) can be used as features. This flexibility has another advantage as well. It helps the model to be more resilient to time lag between noticing a deviation in input signals and the reaction that will appear in the output signals. This time lag can be caused by a number of reasons including: physical limitations, sampling resolution, tuning of the controller, or design choices in the system.
%\subsubsection{Data Collection}
%The object system is treated as a black-box where we can only read its input and output values.
%Capturing the input and output data ($T_k$s) is straightforward. In the first step the system is executed and multiple logs of the input/output values are recorded. After that, state change points ($CP_k$s) data is required to be collected. A typical way to obtain them is with help of a domain expert. They label the logs indicating the time stamps where the system's internal state has changed as well as the new state that it went in. Domain experts are ideal for labeling since they know what the system is and what it is supposed to do, so they can (probably with some inherent human error) detect the system state.
%They can do it based on a variety of information sources such as examining the system's I/O (which we already have captured), logs, or a high level program or set of instructions preloaded into the system.

%We aimed to infer high level states of a black-box system using a machine learning model that observes the system's behavior.
\section{Data Encoding}
% To collect data we run the system $N$ times using $N$ existing test cases and record the input/output values. The recorded data from $k$-th run, $T_k$, forms a discrete multivariate time series data.
The input/output values of the black-box system create a multivariate time-series ($T_k$), which can be defined as a set of $n$ univariate time series ($V_i$) of the same length $l_k$. Each $V_i$ corresponds to the recorded values for one of the inputs or outputs of the system:
\begin{equation} \label{eq:T_k}
    T_k = \{{V_1}_k, {V_2}_k, \ldots, {V_n}_k\}
\end{equation}
\begin{equation} \label{eq:l_k}
    |{V_1}_k|=|{V_2}_k|=\ldots=|{V_n}_k|=l_k
\end{equation}

Note that, as figure~\ref{fig:general_net} shows, we take both inputs and outputs as part of the time-series data to be fed as input into my deep learning models.
This is to make sure that we can model state-based behavior of the system, where the current state depends not only on the inputs, but also on the last state(s) (captured as previous outputs) of the system. As an example, from the case study, if the outputs are not taken into account
% the ``acceleration'' state before ``takeoff'' and the breaking state after landing
a mid-flight ``descend'' state and the ``approach'' state right before landing
are indistinguishable, using the sensor readings (inputs) alone.

Having such a time-series, the only remaining pieces from a training set are the labels. Unlike the input/output values (the features in the dataset) the labels are not usually given. My method to infer the labels is a supervised approach. Thus, we need the domain expert to manually label each individual time stamp with a state name/ID. In practice, what they would do is to identify the approximate time that a state change happens and assign the new state to one of the previous states labels or define a new label for this new state.
Thus I encode the states information over time as a set of tuples in the form of $(t_s, s)$ where $t_s$ denotes the timestamp where the system entered state $s$. We show the set of all possible states with $S$ ($s \in S$) and define $N_s$ as the cardinality of this set.
\begin{equation}\label{eq:change_point}\begin{split}
    CP_k {}&{}= \big\{ (t_{s_1}, s_1), (t_{s_2}, s_2), \ldots, (t_{s_l}, s_l) \big\}\:,\; s_i \in S \\
    N_s  {}&{}= |S|
\end{split}
\end{equation}
So in summary, the dataset consists of $N$ pairs of the I/O values as features and their state information as labels $\big\{(X=T_k,\;y=CP_k)|1\leq~k\leq~N\big\}$.
% Each example in dataset contains one whole test case. The output labels were one-hot encoded to match the model output shape.
%It is worth mentioning that in section~\ref{mp_data_collection} we explain what we did to keep the variety of the collected data (and hence its quality) high.

\subsection{Data Preprocessing}
%\subsubsection{Inputs and Labels} %\label{data_set_properties}
Before being fed into the model $\mathcal{F}$ (as defined below), the inputs and labels need some preprocessing.
\begin{equation}\label{eq:model_F}
    \mathcal{F}(\delta(T), m) \colon \mathbb{R}^{L\times n}\times\mathbb{R}^L\to S^{L}.
\end{equation}
To run more efficiently, TensorFlow expects all the inputs to have the same length. To do that, the shorter $T_k$s should be zero-padded to length $L = max\{l_k\}$. The padding function $\delta$ does that.
Therefore, eventually, the input to the model will be $T_k$s that are rearranged to form a tensor of shape $n \times L$ along with a padding mask (denoted with $m$).
The mask tells the model where the tail starts so the model can ignore all the zeros from there on. % It prevents the added zero values to have a negative effect on the model's performance.
%The inputs are the sampled in/out values of the system given in form of $n$ time series of length $L$.
\begin{equation}\label{eq:model_as_function}
\begin{split}
    \hat{O} {}&{}= \langle \hat{o}_i \in S \rangle^L_{i=1} = \mathcal{F}\left(\left[ {\delta(V_1)}^\intercal \: {\delta(V_2)}^\intercal \; \ldots \; {\delta(V_n)}^\intercal \right],m\right) \\
    % \delta(V_i) {}&{}= \left[V_i \quad \vec{0}_{L-l}\right] \\
    m {}&{}= \delta(\vec{\mathds{1}}_l) \quad\text{i.e.}\quad \langle {m}_j \rangle^l_{j=1} = 1\:, \: \langle {m}_j \rangle^L_{j=l+1} = 0 \\
\end{split}
\end{equation}
Here $l$ denotes the length of the input before padding. It is equal to $l_k$ for the $k$th training data ($T_k$).

As defined in \eqref{eq:change_point}, $CP_k$s are tuples of $(t, s)$ which indicate the system have gone into state $s$ at time $t$. To train the model, $CP_k$ needs to be expanded into a vector of length $L$ denoted by $O$ where each element $o_t$ holds the state at time $t$. To define it formally, the elements can be derived from $CP_k$ using the following formula:
\begin{equation} \label{eq:output}
\begin{split}
O = \langle  \forall t \in \mathbb{N}_L : s_i \:|\:
           (& t_{s_i}, s_i) \in CP_k \: \land  \\
            & t_{s_i} = max\{t_{s_j} \:|\: (t_{s_j}, s_j) \in CP_k \land t_{s_j}\leq t\} \rangle
\end{split}
\end{equation}
For example: Suppose $L=10$ and $CP = \{(0, a),\:(3, b),\:(5, c),\:(8, a)\}$
% which means starting with state $a$ then going to state $b$ at time $t=3$,\ldots and it goes on for 10 steps;
then $O = \langle a\:a\:a\:b\:b\:c\:c\:c\:a\:a \rangle$.
If there are more than two possible states ($N_s > 2$), $O$ needs to be one-hot encoded, at this stage.

\section{The Model Implementation} \label{sec:architecture_detail}
The first few layers of the model are convolutional layers. I used 5 convolutional layers with 64 filters each and a growing kernel size.
The intuition behind this design is that starting with a small kernel guides the training in a way that the first layers learn simpler more local features that fits in their window (kernel size).
Kernel sizes started with 3 since it is a common number in the literature for kernel sizes, then I used multiples of 5 from 5 to 20.
The rationale behind choosing 5 is because the sampling frequency is 5, so each layer with a kernel size of $5n$ processes a whole $n$ seconds worth of simulation data, in each step.
Stopping at kernel size of 20 was a compromise between generalizability and model size. Generally, a larger model has more learning capacity, but it is also more prone to over-fitting. The current models are the smallest I could make the models (to avoid over-fitting), without compromising the performance.

Same compromise was made in the second section of the model (Recurrent layers), the sweet spot for hyper-parameters here was to use two GRU layers with 128 cells each. Their output was fed into a fully connected layer with 128 neurons with a leaky ReLU ($\alpha=0.3$) activation function \cite{maas2013rectifier} and finally to a dense layer with $N_s=25$ units with softmax activation.
I used Adam optimizer \cite{kingma2014adam} that could converge in 60-80 epochs, i.e. validation accuracy plateaued. The full architecture can be seen in figure~\ref{fig:model_arch}.
\begin{landscape}
\begin{figure*}
    \centering
    \includegraphics[width=\columnwidth]{ASE_files/Convolutional_Net.pdf}
    \caption{Model architecture in a nutshell. Tandem convolutional layers with increasing kernel size fed into two sequence-to-sequence recurrent layers with 128 GRU cells each, which is then fed into dense layers to output the predicted system state, as a list of one-hot encoded states. $\hat{O}$ will be the result of applying argmax operation on the last layer's output. $L=18000,\: N_s=25$}
    \label{fig:model_arch}
\end{figure*}
\end{landscape}


\newcommand{\globalassum}{Assumptions \ref{assu:paper_smoothness}---\ref{assu:paper_bounded} }

\section{Detailed assumptions, lemmas, and proofs\label{sec:appendix_proofs}}

\subsection{Tools}

We begin by stating two general propositions that will be useful.
First, we show that a version of Cauchy-Schwartz can be applied to
weighted sums of tensors.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{prop}
\label{propref:tensor_cauchy_schwartz}Tensor array version of H{\"o}lder's
inequality. Let $w$ be an array of scalars and let $a=\left(a_{1},...,a_{N}\right)$
be an array of tensors, were each $a_{n}$ is indexed by $i=1,\ldots,D_{A}$
($i$ may be a multi-index---e.g., if $A$ is a $D\times D$ matrix,
then $i=\left(j,k\right)$, for $j,k\in\left[D\right]$ and $D_{A}=D^{2}$).
Let $p,q\in\left[1,\infty\right]$ be two numbers such that $p^{-1}+q^{-1}=1$.
Then
\begin{align*}
\norm{\frac{1}{N}\sum_{n=1}^{N}w_{n}a_{n}}_{1} & \le\frac{D_{A}^{\frac{1}{p}}}{N}\norm w_{p}\norm a_{q}.
\end{align*}
In particular, with $p=q=2$,
\begin{align*}
\norm{\frac{1}{N}\sum_{n=1}^{N}w_{n}a_{n}}_{1} & \le\sqrt{D_{A}}\frac{\norm w_{2}}{\sqrt{N}}\frac{\norm a_{2}}{\sqrt{N}}.
\end{align*}
\end{prop}
%
\begin{proof}
The conclusion follows from the ordinary H{\"o}lder's inequality
applied term-wise to $n$ and Jensen's inequality applied to the indices
$i$.

\begin{align*}
\norm{\frac{1}{N}\sum_{n=1}^{N}w_{n}a_{n}}_{1} & =\sum_{i=1}^{D_{A}}\left|\frac{1}{N}\sum_{n=1}^{N}w_{n}\left(a_{n}\right)_{i}\right|\\
 & \le\frac{1}{N}\sum_{i=1}^{D_{A}}\left|\left(\sum_{n=1}^{N}\left|w_{n}\right|^{p}\right)^{\frac{1}{p}}\left(\sum_{n=1}^{N}\left|\left(a_{n}\right)_{i}\right|^{q}\right)^{\frac{1}{q}}\right|\text{(H{\"o}lder)}\\
 & =\frac{1}{N}\norm w_{p}\frac{D_{A}}{D_{A}}\sum_{i=1}^{D_{A}}\left(\sum_{n=1}^{N}\left|\left(a_{n}\right)_{i}\right|^{q}\right)^{\frac{1}{q}}\\
 & \le\frac{1}{N}\norm w_{p}D_{A}\left(\frac{1}{D_{A}}\sum_{i=1}^{D_{A}}\sum_{n=1}^{N}\left|\left(a_{n}\right)_{i}\right|^{q}\right)^{\frac{1}{q}}\textrm{ (Jensen applied to }i\textrm{)}\\
 & =\frac{1}{N}\norm w_{p}D_{A}\left(\frac{1}{D_{A}}\sum_{n=1}^{N}\norm{a_{n}}_{q}^{q}\right)^{\frac{1}{q}}\\
 & =\frac{1}{N}\norm w_{p}D_{A}^{1-\frac{1}{q}}\norm a_{q}\\
 & =\frac{D_{A}^{\frac{1}{p}}}{N}\norm w_{p}\norm a_{q}.
\end{align*}
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%
Next, we prove a relationship between the term-wise difference between
matrices and the difference between their operator norms. It is well-known
that the minimum eigenvalue of a non-singular matrix is continuous
in the entries of the matrix. In the next proposition, we quantify
this continuity for the $L_{1}$ norm.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{prop}
\label{propref:operator_norm_continuity}
%
Let $A$ and $B$ be two square matrices of the same size.
Let $\norm{A^{-1}}_{op}\le \constop$ for some finite $\constop$, and  Then
%
\begin{align*}
\norm{A-B}_{1}\le \frac{1}{2} (\constop)^{-1} &
    \quad\Rightarrow\quad\norm{B^{-1}}_{op} \le 2 \constop.
\end{align*}
%
\begin{proof}
%
We will use the results stated in Theorem 4.29 of \citet{schott:2016:matrix} and
the associated discussion in Example 4.14, which establish the following result.
Let $A$ be a square, nonsigular matrix, and let $I$ be the identity matrix of
the same size.  Let $\norm{\cdot}$ denote any matrix norm satisfying $\norm{I} =
1$.  Let $D$ be a matrix of the same size as $A$ satisfying
%
\begin{align}\eqlabel{ab_matrix_condition}
\norm{A^{-1}} \norm{D} \le 1.
\end{align}
%
Then
%
\begin{align}\label{eq:matrix_norm_continuity}
    \norm{A^{-1} - (A + D)^{-1}} \le
    \frac{\norm{A^{-1}}\norm{D}}{1 - \norm{A^{-1}\norm{D}}} \norm{A^{-1}}.
\end{align}
%
We will apply \eqref{matrix_norm_continuity} using the operator norm
$\norm{\cdot}_{op}$, for which $\norm I_{op}=1$ and with $D := B - A$.
Because $\norm{A^{-1}}_{op}\le \constop$, $A$ is invertible.

Assume that $\norm{A-B}_{1}\le \frac{1}{2} (\constop)^{-1}$.  First, note that
%
\begin{align}\label{eq:ab_matrix_condition_fulfilled}
\norm{A^{-1}}_{op} \norm{D}_{op} &=
    \norm{A^{-1}}_{op}\norm{B - A}_{op} \nonumber \\
&\le\norm{A^{-1}}_{op}\norm{B - A}_{1}
    & \textrm{(ordering of matrix norms)}\nonumber \\
 & \le \constop \frac{1}{2} (\constop)^{-1}
    & \textrm{(by assumption)} \nonumber \\
&= \frac{1}{2}  < 1,
\end{align}
%
so \eqref{ab_matrix_condition} is satisfied and we can apply
\eqref{matrix_norm_continuity}. Then
%
\begin{align*}
\norm{B^{-1}}_{op}
 & \le \norm{B^{-1}-A^{-1}}_{op} + \norm{A^{-1}}_{op}
    & \textrm{ (triangle inequality)}\\
 & \le \frac{\norm{A^{-1}}_{op}\norm{B - A}_{op}}
            {1 - \norm{A^{-1}}_{op}\norm{B - A}_{op}}
        \norm{A^{-1}}_{op} + \norm{A^{-1}}_{op}
     & \textrm{(\eqref{matrix_norm_continuity})}\\
 & \le \frac{\frac{1}{2}}{1-\frac{1}{2}}\norm{A^{-1}}_{op} +
    \norm{A^{-1}}_{op}
    &\textrm{(\eqref{ab_matrix_condition_fulfilled})} \\
 & \le 2 \constop.&\textrm{(by assumption)}
\end{align*}
\end{proof}
%
\end{prop}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Next, we define the quantities needed to make use of the integral form of
the Taylor series remainder.\footnote{We are indebted to Pang Wei Koh for
pointing out the need to use the integral form of the remainder for
Taylor series expansions of vector-valued functions.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{prop}
\label{propref:taylor_series_remainder}
%
For any $\theta \in \Omega_{\theta}$ and any $\tilde{w} \in W$,
%
\begin{align*}
G(\theta, \tilde{w}) - G(\thetaone, \tilde{w}) =&
    \left(\int_0^1 H(\thetaone + t (\theta - \thetaone), w) dt\right)
    \left(\theta - \thetaone\right)
\end{align*}
%
\end{prop}
%
\begin{proof}
%
Let $G_d(\theta, \tilde{w})$ denote the $d$-th component of the vector
$G(\theta, \tilde{w})$, and define the function $f_d(t) := G_d(\thetaone + t
(\thetaone - \theta), \tilde{w})$. The proposition follows by taking the
integral remainder form of the zero-th order Taylor series expansion of $f_d(t)$
around $t=0$ \citep[Appendix B.2]{dudley:2018:analysis}, and stacking the result
into a vector.
%
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The Taylor series residual of \proprefref{taylor_series_remainder} will show up
repeatedly, so we will give it a concise name in the following definition.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{defn}
\label{defref:hess_integral}
%
For a fixed weight $w$ and a fixed parameter $\theta$, define the Hessian
integral
%
\begin{align*}
\hint(\theta, w) :=&
    \int_0^1 H(\thetaone + t (\theta - \thetaone), w) dt.
\end{align*}
\end{defn}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Lemmas}

We now prove some useful consequences of our assumptions. The proof
roughly proceeds for all $w\in W_{\delta}$ by the following steps:
%
\begin{enumerate}
\item When $\delta$ is small we can make $\norm{{{\thetaw}}-\thetaone}_{2}$
small. (\lemref{theta_difference_bound} below.)
\item When $\norm{\theta-\thetaone}_{2}$ is small, then the derivatives
$H\left(\theta,w\right)$ are close to their optimal value, $H\left(\thetaone,\onevec\right)$.
(\lemref{bounded_continuous} and \lemref{gh_difference_from_one}
below.)
\item When the derivatives are close to their optimal values, then $H\left(\theta,w\right)$
is uniformly non-singular. (\lemref{continuous_invertibility} below.)
\item When the derivatives are close to their optimal values and $H\left(\theta,w\right)$
is uniformly non-singular we can control the error in $\thetaij-\thetaw$
in terms of $\delta$. (\thmrefref{taylor_error_first} below.)
\end{enumerate}
%
We begin by showing that the difference between $\thetaw$ and $\thetaone$
for $w\in W_{\delta}$ can be made small by making $\delta$ from
\condref{paper_uniform_bound} small.  First, however, we need to prove that
operator norm bounds on $H(\theta, w)$ also apply to the integrated Hessian
$\hint(\theta, w)$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{lem}\label{lem:hess_integral_invertible}
Invertibility of the integrated Hessian.
%
If, for some domain $\Omega$ and some constant $C$, $\sup_{\theta \in \Omega}
\norm{H(\theta, w)^{-1}}_{op} \le C$, then
$\sup_{\theta \in \Omega} \norm{\hint(\theta, w)^{-1}}_{op} \le C$.
%
\end{lem}

\begin{proof}
%
By definition of the operator norm,
%
\begin{align*}
\norm{\hint(\theta, w)^{-1}}_{op}^{-1} =&
    \min_{v \in \mathbb{R}^D: \norm{v}_2 = 1} v^T \hint(\theta, w) v \\
=& \min_{v \in \mathbb{R}^D: \norm{v}_2 = 1}
    \int_0^1 v^T H(\thetaone + t (\theta - \thetaone), w) v dt \\
\ge& \int_0^1 \min_{v \in \mathbb{R}^D: \norm{v}_2 = 1}
    v^T H(\thetaone + t (\theta - \thetaone), w) v dt \\
\ge& \inf_{\theta \in \Omega} \min_{v \in \mathbb{R}^D: \norm{v}_2 = 1}
        v^T H(\theta, w) v \\
\ge& C^{-1}.
\end{align*}
%
The result follows by inverting both sides of the inequality.
%
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
\begin{lem}
\label{lem:theta_difference_bound}Small parameter changes. Under
\globalassum and \condref{paper_uniform_bound},
\begin{align*}
\textrm{for all }w\in W_{\delta},\quad\norm{{{\thetaw}}-\thetaone}_{2} &
    \le\constop\delta.
\end{align*}
\end{lem}
%
\begin{proof}
%
Applying \proprefref{taylor_series_remainder} with $\theta = \thetaw$
and $\tilde{w} = \onevec$ gives
%
%
\begin{align*}
G\left(\thetaw,\onevec\right) &
    =G\left(\thetaone,\onevec\right) +
    \hint\left(\thetaw, \onevec\right)
    \left(\thetaw-\thetaone\right).
\end{align*}
%
By \assuref{paper_hessian} and
% $\sup_{\theta \in \Omega_\theta}
% \norm{H(\theta,\onevec)^{-1}} \le \constop$, so by
\lemref{hess_integral_invertible},
$\sup_{\theta \in \Omega_\theta}
\norm{\hint(\theta,\onevec)^{-1}} \le \constop$.
%
In particular, $\hint(\theta,\onevec)$ is non-singular.
A little manipulation, together with the fact that
$G\left(\thetaw,w\right)=G\left(\thetaone,\onevec\right)=0$ gives
%
\begin{align*}
\thetaw-\thetaone & =
    \hint\left(\thetaw,\onevec\right)^{-1}
    \left(G\left(\thetaw, \onevec\right) - G\left(\thetaw,w\right)\right).
\end{align*}
%
Taking the norm of both sides gives
%
\begin{align*}
\norm{{{\thetaw}}-\thetaone}_{2} & =
    \norm{\hint\left(\thetaw,\onevec\right)^{-1}
        \left(G\left({{\thetaw}},\onevec\right) -
             G\left({{\thetaw}},w\right)\right)}_{2}\\
& \le\norm{\hint\left(\thetaw,\onevec\right)^{-1}}_{op}
    \norm{\left(G\left({{\thetaw}},\onevec\right) -
                G\left({{\thetaw}},w\right)\right)}_{2}\\
& \le\constop\norm{G\left({{\thetaw}},\onevec\right) -
                   G\left({{\thetaw}},w\right)}_{2}
    \textrm{ (Lemma \ref{lem:hess_integral_invertible})}\\
& \le\constop\norm{
    G\left({{\thetaw}},\onevec\right) -
    G\left({{\thetaw}},w\right)}_{1}\textrm{ (relation between norms)}\\
& \le\constop\sup_{\theta\in\Omega_{\theta}}
    \norm{G\left(\theta,\onevec\right)-G\left(\theta,w\right)}_{1}\\
& \le\constop\delta.\textrm{ (Condition \ref{cond:paper_uniform_bound}).}
%
\end{align*}
%
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%
%
Because we will refer to it repeatedly, we give the set of $\theta$
defined in \lemref{theta_difference_bound} a name.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{defn}
For a given $\delta$, define the region around $\thetaone$ given
by \lemref{theta_difference_bound} as

\begin{align*}
\thetadeltaball & :=\left\{ \theta:\norm{\theta-\thetaone}_{2}\le\constop\delta\right\} \bigcap\Omega_{\theta}.
\end{align*}
\end{defn}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
In other words, \lemref{theta_difference_bound} states that \condref{paper_uniform_bound}
implies $\thetaw\in\thetadeltaball$ when $w\in W_{\delta}$.

Next, we show that closeness in $\theta$ will mean closeness in $H\left(\theta,w\right)$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{lem}
\label{lem:bounded_continuous} Boundedness and continuity. Under
\paperallcoreassum and \condref{paper_uniform_bound},
%
\begin{align*}
\textrm{for all }\theta\in\thetaball,\quad\sup_{w\in W}
    \norm{H\left(\theta,w\right) - H\left(\thetaone,w\right)}_{1}
    &\le D\constw\liph\norm{\theta-\thetaone}_{2}.
\end{align*}
\end{lem}
\begin{proof}
%
For $\theta\in\thetaball$,
%
\begin{align*}
\sup_{w\in W}\norm{H\left(\theta,w\right)-H\left(\thetaone,w\right)}_{1} &=
    \sup_{w\in W}\norm{\frac{1}{N}
    \sum_{n=1}^{N}w_{n}\left(h_{n}\left(\theta\right) -h_{n}\left(\thetaone\right)\right)}_{1}\textrm{ (by definition)}\\
 & \le D\sup_{w\in W}\frac{\norm w_{2}}{\sqrt{N}}
    \frac{\norm{h\left(\theta\right) -
                h\left(\thetaone\right)}_{2}}{\sqrt{N}}
        \textrm{ (Proposition \ref{propref:tensor_cauchy_schwartz})}\\
 & \le D\constw\frac{\norm{h\left(\theta\right) -
                           h\left(\thetaone\right)}_{2}}{\sqrt{N}}
       \textrm{ (Assumption \ref{assu:paper_weight_bounded})}\\
 & \le D\constw\liph\norm{\theta-\thetaone}_{2}
    \textrm{ (Assumption \ref{assu:paper_lipschitz} and }
    \theta\in\thetaball\textrm{)}.
\end{align*}
%
\end{proof}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
We now combine \lemref{theta_difference_bound} and \lemref{bounded_continuous}
to show that $H\left(\theta,w\right)$ is close to its value at the
solution $H\left(\thetaone,\onevec\right)$ for sufficiently small
$\delta$ and for all $\theta\in\thetadeltaball$.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{lem}
\label{lem:gh_difference_from_one}Bounds for difference in parameters.
Under \paperallcoreassum and \condref{paper_uniform_bound}, if $\delta\le\thetasize\constop[-1]$,
then
\begin{align*}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}} &
    \norm{H\left(\theta,w\right)-H\left(\thetaone,\onevec\right)}_{1}
    \le\left(1 + D\constw\liph\constop\right)\delta.
\end{align*}
\end{lem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{proof}
By \lemref{theta_difference_bound}, $\delta\le\thetasize\constop[-1]$
implies that $\constop\delta\le\thetasize$ and so $\thetadeltaball\subseteq\thetaball$.
Consequently, we can apply \lemref{bounded_continuous}:
\begin{align*}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\thetaone,w\right)}_{1} & \le\sup_{\theta\in\thetaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\thetaone,w\right)}_{1}\\
 & \le D\constw\liph\norm{\theta-\thetaone}_{2}\textrm{ (Lemma \ref{lem:bounded_continuous})}\\
 & \le D\constw\liph\constop\delta\quad\textrm{ (because }\theta\in\thetadeltaball\textrm{)}.
\end{align*}
%
Next, we can use this to write
\begin{align*}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}} & \norm{H\left(\theta,w\right)-H\left(\thetaone,\onevec\right)}_{1}\\
 & =\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\theta,\onevec\right)+H\left(\theta,\onevec\right)-H\left(\thetaone,\onevec\right)}_{1}\\
 & \le\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\theta,\onevec\right)}_{1}+\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,\onevec\right)-H\left(\thetaone,\onevec\right)}_{1}\\
 & \le\sup_{\theta\in\Omega_{\theta}}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\theta,\onevec\right)}_{1}+\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,\onevec\right)-H\left(\thetaone,\onevec\right)}_{1}\\
 & \le\delta+\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,\onevec\right)-H\left(\thetaone,\onevec\right)}_{1}\textrm{ (Condition \ref{cond:paper_uniform_bound})}\\
 & \le\delta+D\constw\liph\constop\delta.
\end{align*}
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%
The constant that appears multiplying $\delta$ at the end of the proof of
\lemref{gh_difference_from_one} will appear often in what follows, so we give it
the special name $\constij$ in \defrefref{constants_definition}.

Note that \lemref{gh_difference_from_one} places a condition on how small
$\delta$ must be in order for our regularity conditions to apply.
\lemref{theta_difference_bound} will guarantee that $\thetaw\in\thetadeltaball$,
but if we are not able to make $\delta$ arbitrarily small in
\condref{paper_uniform_bound}, then we are not guaranteed to ensure that
$\thetadeltaball\subseteq\thetaball$, will not be able to assume Lipschitz
continuity, and none of our results will apply.

Next, using \lemref{gh_difference_from_one}, we can extend the operator bound on
$\hone^{-1}$ from \assuref{paper_hessian} to $H\left(\theta,w\right)^{-1}$ for
all $w\in W_{\delta}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{lem}
\label{lem:continuous_invertibility}Uniform invertibility of the
Hessian. Under \paperallcoreassum and \condref{paper_uniform_bound}, if $\delta\le\min\left\{ \thetasize\constop[-1],\frac{1}{2}\constij^{-1}\constop[-1]\right\} $,
then
\begin{align*}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)^{-1}}_{op} & \le2\constop.
\end{align*}
\end{lem}
\begin{proof}
By \assuref{paper_hessian}, $\norm{H\left(\thetaone,\onevec\right)^{-1}}_{op}\le\constop$.
So by \proprefref{operator_norm_continuity}, it will suffice to select
$\delta$ so that
\begin{align}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\thetaone,\onevec\right)}_{1} & \le\frac{1}{2}\constop[-1].\label{eq:h_bound}
\end{align}
When we can apply \lemref{gh_difference_from_one}, we have
\begin{align*}
\sup_{\theta\in\thetadeltaball}\sup_{w\in W_{\delta}}\norm{H\left(\theta,w\right)-H\left(\thetaone,\onevec\right)}_{1} & \le\constij\delta.
\end{align*}
So $H\left(\theta,w\right)$ will satisfy \eqref{h_bound} if we can
apply \lemref{gh_difference_from_one} and if
\begin{align*}
\delta\le & \frac{1}{2}\constop[-1]\constij^{-1}.
\end{align*}
To apply \lemref{gh_difference_from_one} we additionally require
that $\delta\le\thetasize\constop[-1]$. By taking $\delta\le\min\left\{ \thetasize\constop[-1],\frac{1}{2}\constop[-1]\constij^{-1}\right\} $,
we satisfy \eqref{h_bound} and the result follows.
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Next, we show that a version of \lemref{gh_difference_from_one} also applies
to the integrated Hessian $\hint(\theta, w)$ when $\theta \in \thetadeltaball$.

%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{lem}
\label{lem:int_h_difference_from_one}
Bounds for difference of the integrated Hessian.
%
Under \paperallcoreassum and \condref{paper_uniform_bound}, if $\delta\le\thetasize\constop[-1]$ and $\theta \in \thetadeltaball$,
%
\begin{align*}
\sup_{w\in W_{\delta}}\norm{
    \hint\left(\theta, w\right) -
    H(\thetaone, \onevec)}_{1}
\le& \left(1 + D\constw\liph\constop\right)\delta.
\end{align*}
%
\end{lem}
%
\begin{proof}
\begin{align*}
\MoveEqLeft
\sup_{w\in W_{\delta}}\norm{
    \hint\left(\theta, w\right) -
    H(\thetaone, \onevec)}_{1} &\\
=& \sup_{w\in W_{\delta}}\norm{
    \int_0^1 \left(
        H(\thetaone + t (\theta - \thetaone), w) dt -
        H(\thetaone, \onevec)\right)}_{1}
        &\textrm{ (Definition \ref{defref:hess_integral})} \\
\le&
    \sup_{w\in W_{\delta}}
    \int_0^1 \norm{
        H(\thetaone + t (\theta - \thetaone), w) -
        H(\thetaone, \onevec)}_{1} dt
        &\textrm{ (Jensen's inequality)} \\
\le&
    \sup_{\theta\in\thetadeltaball} \sup_{w\in W_{\delta}}
    \norm{H(\theta), w) - H(\thetaone, \onevec)}_{1} &\\
\le&
    \left(1 + D\constw\liph\constop\right)\delta
    &\textrm{ (Lemma \ref{lem:gh_difference_from_one})}
\end{align*}
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%

With these results in hand, the upper bound on $\delta$ will at last be
sufficient to control the error terms in our approximation. For compactness, we
give it the upper bound on $\delta$ the name $\deltasize$ in
\defrefref{constants_definition}.

Finally, we state a result that will allow us to define derivatives
of $\thetaw$ with respect to $w$.
\begin{lem}
\label{lem:implicit_function_theorem}Inverse function theorem. Under
\paperallcoreassum and \condref{paper_uniform_bound}, and for $\delta\le\deltasize$,
there exists a continuous, differentiable function of $w$, $\thetahat\left(w\right)$,
such that, for all $w\in W$, G$\left(\thetahat\left(w\right),w\right)=0$.
\end{lem}
\begin{proof}
This follows from \lemref{continuous_invertibility} and the implicit
function theorem.
\end{proof}
By definition, $\thetahat\left(\onevec\right)=\thetaone$.

\subsection{Bounding the errors in a Taylor expansion}

We are now in a position to use \paperallcoreassum and \condref{paper_uniform_bound}
to bound the error terms in a first-order Taylor expansion of $\thetaw$.
We begin by simply calculating the derivative $d\thetahat\left(w\right)/dw$.
\begin{prop}
\label{propref:theta_w_first_derivative}For any $w\in W$ for which
$H\left(\thetaw,w\right)$ is invertible, and for any vector $a\in\mathbb{R}^{N}$,
\begin{align*}
\frac{d\thetaw}{dw^{T}}\at wa & =-H\left(\thetaw,w\right)^{-1}G\left(\thetaw,a\right).
\end{align*}
\end{prop}
\begin{proof}
Because $G\left(\thetaw,w\right)=0$ for all $w\in W$, by direct
calculation,
\begin{align*}
0 & =\frac{d}{dw^{T}}G\left(\thetaw,w\right)\at wa\\
 & =\left(\frac{\partial G}{\partial\theta^{T}}\frac{d\hat{\theta}}{dw^{T}}+\frac{\partial G}{\partial w^{T}}\right)\at{_{w}}a\\
 & =H\left(\thetaw,w\right)\frac{d\hat{\theta}}{dw^{T}}\at{_{w}}a+\left(\frac{\partial}{\partial w^{T}}\frac{1}{N}\sum_{n=1}^{N}w_{n}g_{n}\left(\theta\right)\right)\at{_{w}}a\\
 & =H\left(\thetaw,w\right)\frac{d\hat{\theta}}{dw^{T}}\at{_{w}}a+\frac{1}{N}\sum_{n=1}^{N}g_{n}\left(\thetaw\right)a\\
 & =H\left(\thetaw,w\right)\frac{d\hat{\theta}}{dw^{T}}\at{_{w}}a+G\left(\thetaw,a\right).
\end{align*}
Because $H\left(\thetaw,w\right)$ is invertible by assumption, the
result follows.
\end{proof}
\begin{defn}
\label{defref:theta_infinitesimal_jackknife}Define
\begin{align*}
\thetaij\left(w\right) & :=\thetaone+\frac{d\thetaw}{dw^{T}}\at{\onevec}\left(w-\onevec\right)\\
 & =\thetaone-\hone^{-1}G\left(\thetaone,w\right).\textrm{ (because }G\left(\thetaone,\onevec\right)=0\textrm{)}
\end{align*}
\end{defn}
%
$\thetaij\left(w\right)$ in \defrefref{theta_infinitesimal_jackknife}
is the first term in a Taylor series expansion of $\thetaw$ as a
function of $w$. We want to bound the error, $\thetaij\left(w\right)-\thetaw$.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{thm}
\label{thmref:taylor_error_first}

Under \paperallcoreassum and \condref{paper_uniform_bound},
when $\delta\le\deltasize$,
%
\begin{align*}
\sup_{w\in W_{\delta}}\norm{\thetaij\left(w\right)-\thetahat\left(w\right)}_{2} & \le2\constop[2]\constij\delta^{2}.
\end{align*}
%
\end{thm}
%
\begin{proof}
%
Applying \proprefref{taylor_series_remainder} with $\theta = \thetaw$ and
$\tilde{w} = w$, we have
%
\begin{align*}
0=G\left(\thetaw,w\right) & =
    G\left(\thetaone, w\right)
    +\hint\left(\thetaw, w\right) \left(\thetaw[w] - \thetaone\right).
\end{align*}
%
Because $\delta\in W_{\delta}$, \lemref{theta_difference_bound} implies that
$\thetaw\in\thetadeltaball$ so, \lemref{continuous_invertibility}
and \lemref{hess_integral_invertible} imply that
$\hint\left(\thetaw,w\right)$ is invertible and we can solve for
$\thetaw - \thetaone$.
%
\begin{align*}
\thetaw[w]-\thetaone & =
    -\hint\left(\thetaw, w\right)^{-1}G\left(\thetaone,w\right)\\
 & =\left(-\hint\left(\thetawiggle,w\right)^{-1} +
          H\left(\thetaone,\onevec\right)^{-1} -
          H\left(\thetaone,\onevec\right)^{-1}\right)
            G\left(\thetaone,w\right)\\
 & =\left(H\left(\thetaone,\onevec\right)^{-1} -
          \hint\left(\thetaw,w\right)^{-1}\right) G\left(\thetaone,w\right)+
    \thetaij\left(w\right)-\thetaone.
\end{align*}
%
Eliminating $\thetaone$ and taking the supremum of both sides gives
%
\begin{align*}
\sup_{w\in W_{\delta}} &
    \norm{\thetaij\left(w\right)-{{\thetaw[w]}}}_{2}\\
 & =\sup_{w\in W_{\delta}}\norm{
    \left(H\left(\thetaone,\onevec\right)^{-1} -
          \hint\left(\theta, w\right)^{-1}\right)
            G\left(\thetaone,w\right)}_{2}\\
 & =\sup_{w\in W_{\delta}}\norm{
    \hint\left(\thetaw, w\right)^{-1}\left(
        \hint\left(\thetaw,w\right) -
        H\left(\thetaone,\onevec\right)\right)
            H\left(\thetaone,\onevec\right)^{-1}
                G\left(\thetaone,w\right)}_{2}\\
 & \le2\constop\sup_{w\in W_{\delta}}\norm{
    \left(\hint\left(\thetaw,w\right) -
          H\left(\thetaone,\onevec\right)\right)
            H\left(\thetaone,\onevec\right)^{-1}
                G\left(\thetaone,w\right)}_{2}\textrm{ (Lemma
                    \ref{lem:hess_integral_invertible})}\\
 & \le2\constop\sup_{w\in W_{\delta}}\norm{
    \hint\left(\thetaw,w\right) -
    H\left(\thetaone,\onevec\right)}_{op}
    \norm{H\left(\thetaone,\onevec\right)^{-1}G\left(\thetaone,w\right)}_{2}\\
 & \le2\constop\sup_{w\in W_{\delta}}\norm{
    \hint\left(\thetaw,w\right) -
        H\left(\thetaone,\onevec\right)}_{1}
    \norm{H\left(\thetaone,\onevec\right)^{-1}G\left(\thetaone,w\right)}_{2}
    \textrm{ (ordering of matrix norms)} \\
 & \le 2\constop\constij\delta
    \sup_{w\in W_{\delta}}
        \norm{H\left(\thetaone,\onevec\right)^{-1}
              G\left(\thetaone,w\right)}_{2}
    \textrm{ (Lemma \ref{lem:int_h_difference_from_one})}\\
 & \le2\constop[2]\constij\delta
    \sup_{w\in W_{\delta}}\norm{G\left(\thetaone,w\right)}_{2}
    \textrm{ (Assumption \ref{assu:paper_hessian})}\\
 & =2\constop[2]\constij\delta
    \sup_{w\in W_{\delta}}
        \norm{G\left(\thetaone,w\right) -
              G\left(\thetaone,\onevec\right)}_{2}
    \textrm{ (because }G\left(\thetaone,\onevec\right)=0\textrm{)}\\
 & \le2\constop[2]\constij\delta^{2}
    \textrm{ (Condition \ref{cond:paper_uniform_bound})}.
\end{align*}

\end{proof}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Use cases\label{sec:use_cases}}

First, let us state a simple condition under which \coreassum
hold. It will help to have a lemma for the Lipschitz continuity.
\begin{lem}
Derivative Cauchy Schwartz. Let $a\left(\theta\right)=\left(a_{1}\left(\theta\right),...,a_{N}\left(\theta\right)\right)$
be an array of tensors with multi-index $i\in\left[D_{A}\right]$,
and let $\frac{\partial a\left(\theta\right)}{\partial\theta}=\left(\frac{\partial}{\partial\theta}a_{1}\left(\theta\right),...,\frac{\partial}{\partial\theta}a_{N}\left(\theta\right)\right)$
be an array of tensors of size $D\times D_{A}$. Then
\begin{align*}
\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}}_{2} & \le D_{A}\norm{\frac{\partial a}{\partial\theta}}_{2}.
\end{align*}
\end{lem}
\begin{proof}
By direct calculation,
\begin{align*}
\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}^{2}}_{2}^{2} & =\sum_{r=1}^{D}\left(\frac{\partial}{\partial\theta_{r}}\sum_{n=1}^{N}\sum_{i=1}^{D_{A}}a_{n,i}\left(\theta\right)^{2}\right)^{2}\\
 & =\sum_{r=1}^{D}\left(\sum_{n=1}^{N}\sum_{i=1}^{D_{A}}2a_{n,i}\left(\theta\right)\frac{\partial a_{n,i}\left(\theta\right)}{\partial\theta_{r}}\right)^{2}\\
 & \le\sum_{r=1}^{D}\left(2\sum_{i=1}^{D_{A}}\left(\sum_{n=1}^{N}a_{n,i}\left(\theta\right)^{2}\right)^{\frac{1}{2}}\left(\sum_{n=1}^{N}\left(\frac{\partial a_{n,i}\left(\theta\right)}{\partial\theta_{r}}\right)^{2}\right)^{\frac{1}{2}}\right)^{2}\\
 & \le\sum_{r=1}^{D}\left(2D_{A}^{2}\left(\frac{1}{D_{A}}\sum_{i=1}^{D_{A}}\sum_{n=1}^{N}a_{n,i}\left(\theta\right)^{2}\right)^{\frac{1}{2}}\left(\frac{1}{D_{A}}\sum_{n=1}^{N}\left(\frac{\partial a_{n,i}\left(\theta\right)}{\partial\theta_{r}}\right)^{2}\right)^{\frac{1}{2}}\right)^{2}\\
 & =4D_{A}^{2}\norm a_{2}^{2}\sum_{r=1}^{D}\norm{\frac{\partial a}{\partial\theta_{r}}}_{2}^{2}\\
 & =4D_{A}^{2}\norm a_{2}^{2}\norm{\frac{\partial a}{\partial\theta}}_{2}^{2}.
\end{align*}
By the chain rule,
\begin{align*}
\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}}_{2}^{2} & =\frac{1}{4\norm{a\left(\theta\right)}_{2}^{2}}\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}^{2}}_{2}^{2}\le D_{A}^{2}\norm{\frac{\partial a}{\partial\theta}}_{2}^{2}.
\end{align*}
\end{proof}
%
\begin{lem}
\label{lem:lipschitz_helper}Let $a\left(\theta\right)\in\mathbb{R}^{D\times D}$
be a continuously differentiable random matrix with a $D\times D\times D$
derivative tensor. (Note that the function, not $\theta$, is random.
For example, $\mbe\left[a\left(\theta\right)\right]$ is still a function
of $\theta$.) Suppose that $\mbe\left[\norm{a\left(\theta\right)}_{2}\right]$
is finite for all $\theta\in\Omega_{\theta}$. Then, for all $\theta_{1},\theta_{2}\in\Omega_{\theta}$,
\begin{align*}
\left|\mbe\left[\norm{a\left(\theta_{1}\right)}_{2}\right]-\mbe\left[\norm{a\left(\theta_{2}\right)}_{2}\right]\right| & \le\sqrt{\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial a\left(\theta\right)}{\partial\theta}}_{2}^{2}\right]}\norm{\theta_{1}-\theta_{2}}_{2}.
\end{align*}
\end{lem}
\begin{proof}
For any tensor $a$ with multi-index $i$,
\begin{align*}
\norm{\frac{\partial}{\partial\theta}\norm a_{2}^{2}}_{2}^{2} & =\sum_{r=1}^{D}\left(\frac{\partial}{\partial\theta_{r}}\norm a_{2}^{2}\right)^{2}\\
 & =\sum_{r=1}^{D}\left(\frac{\partial}{\partial\theta_{r}}\sum_{i=1}^{D_{A}}a_{i}^{2}\right)^{2}\\
 & =\sum_{r=1}^{D}\left(2\sum_{i=1}^{D_{A}}a_{i}\frac{\partial a_{i}}{\partial\theta_{r}}\right)^{2}\\
 & \le4\sum_{r=1}^{D}\sum_{i=1}^{D_{A}}a_{i}^{2}\sum_{i=1}^{D_{A}}\left(\frac{\partial a_{i}}{\partial\theta_{r}}\right)^{2}\textrm{ (Cauchy-Schwartz)}\\
 & =4\sum_{i=1}^{D_{A}}a_{i}^{2}\sum_{r=1}^{D}\sum_{i=1}^{D_{A}}\left(\frac{\partial a_{i}}{\partial\theta_{r}}\right)^{2}\\
 & =4\norm a_{2}^{2}\norm{\frac{\partial a}{\partial\theta}}_{2}^{2}.
\end{align*}

Consequently,
\begin{align*}
\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}}_{2}^{2} & =\norm{\frac{1}{2\norm{a\left(\theta\right)}_{2}}\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}^{2}}_{2}^{2}\\
 & =\frac{1}{4\norm{a\left(\theta\right)}_{2}^{2}}\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}^{2}}_{2}^{2}\\
 & \le\frac{4\norm{a\left(\theta\right)}_{2}^{2}}{4\norm{a\left(\theta\right)}_{2}^{2}}\norm{\frac{\partial}{\partial\theta}a\left(\theta\right)}_{2}^{2}\\
 & =\norm{\frac{\partial a\left(\theta\right)}{\partial\theta}}_{2}^{2}.
\end{align*}
So for any $\theta_{1},\theta_{2}\in\Omega_{\theta}$,
\begin{align*}
\left|\mbe\left[\norm{a\left(\theta_{1}\right)}_{2}\right]-\mbe\left[\norm{a\left(\theta_{2}\right)}_{2}\right]\right| & \le\mbe\left[\left|\norm{a\left(\theta_{1}\right)}_{2}-\norm{a\left(\theta_{2}\right)}_{2}\right|\right]\\
 & \le\mbe\left[\left(\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial}{\partial\theta}\norm{a\left(\theta\right)}_{2}}_{2}\right)\right]\norm{\theta_{1}-\theta_{2}}_{2}\textrm{ (}\theta\textrm{ is not random)}\\
 & \le\mbe\left[\left(\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial a\left(\theta\right)}{\partial\theta}}_{2}\right)\right]\norm{\theta_{1}-\theta_{2}}_{2}\\
 & \le\sqrt{\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial a\left(\theta\right)}{\partial\theta}}_{2}^{2}\right]}\norm{\theta_{1}-\theta_{2}}_{2}.
\end{align*}
The result follows. Note that the bound still holds (though vacuously)
if $\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial a\left(\theta\right)}{\partial\theta}}_{2}^{2}\right]$
is infinite.
\end{proof}
\begin{prop}
\label{prop:assumptions_hold}Let $\Omega_{\theta}$ be a compact
set. Let $g_{n}\left(\theta\right)$ be twice continuously differentiable
IID random functions. Define
\begin{align*}
h_{n}\left(\theta\right) & :=\frac{\partial g{}_{n}\left(\theta\right)}{\partial\theta}\\
r_{n}\left(\theta\right) & :=\frac{\partial^{2}g{}_{n}\left(\theta\right)}{\partial\theta\partial\theta},
\end{align*}
where $r_{n}\left(\theta\right)$ is a $D\times D\times D$ tensor.
Assume that

1a) $\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]<\infty$;

1b) $\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{h_{n}\left(\theta\right)}_{2}^{2}\right]<\infty$;

1c) $\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]<\infty;$

2) $\mbe\left[h_{n}\left(\theta\right)\right]$ is non-singular for
all $\theta\in\Omega_{\theta}$;

3) We can exchange expectation and differentiation.

Then $\lim_{N\rightarrow\infty}P\left(\textrm{\coreassum\ hold}\right)=1.$
\end{prop}
%
\begin{proof}
    %
The proof follows from Theorems 9.1 and
9.2 of \citet{keener:2011:theoretical}. We will first show that the expected values of
the needed functions satisfy \coreassum, and then that the sample versions
converge uniformly.

By Jensen's inequality,
\begin{align*}
\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{2}\right] & =\mbe\left[\sqrt{\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{2}^{2}}\right]\le\sqrt{\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]}.
\end{align*}
Also, for the $i^{th}$ component of $g_{n}\left(\theta\right)$
\begin{align*}
\mbe\left[\sup_{\theta\in\Omega_{\theta}}\left|g_{n,i}\left(\theta\right)\right|\right] & \le\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{\infty}\right]\le\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{g_{n}\left(\theta\right)}_{2}\right].
\end{align*}
By Theorem 9.1 of \citet{keener:2011:theoretical}, $\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]$
, $\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}\right]$, and $\mbe\left[g_{n}\left(\theta\right)\right]$
are continuous functions of $\theta$, and because $\Omega_{\theta}$
is compact, they are each bounded. Similar reasoning applies to $h_{n}\left(\theta\right)$
and $r_{n}\left(\theta\right)$. Consequently we can define
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right] & =:Q_{g}^{2}<\infty\\
\sup_{\theta\in\Omega_{\theta}}\mbe\left[\norm{h_{n}\left(\theta\right)}_{2}^{2}\right] & =:Q_{h}^{2}<\infty.
\end{align*}
Below, these constants will be used to satisfy \assuref{paper_smoothness}
and \assuref{paper_bounded} with high probability.

Because $\Omega_{\theta}$ is compact, $\mbe\left[h_{n}\left(\theta\right)\right]$
is continuous, $\mbe\left[h_{n}\left(\theta\right)\right]$ is non-singular,
and the operator norm is a continuous function of $\mbe\left[h_{n}\left(\theta\right)\right]$,
we can also define
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\norm{\mbe\left[h_{n}\left(\theta\right)\right]^{-1}}_{op} & =:Q_{op}<\infty.
\end{align*}
Below, this constant be used to satisfy \assuref{paper_hessian}
with high probability.

Finally, we turn to the Lipschitz condition. \lemref{lipschitz_helper}
implies that
\begin{align*}
\left|\mbe\left[\norm{h_{n}\left(\theta_{1}\right)}_{2}\right]-\mbe\left[\norm{h_{n}\left(\theta_{2}\right)}_{2}\right]\right| & \le\sqrt{\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]}\norm{\theta_{1}-\theta_{2}}_{2}.
\end{align*}
Define
\begin{align*}
\Lambda_{h} & =\sqrt{\mbe\left[\sup_{\theta\in\Omega_{\theta}}\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]},
\end{align*}
so that we have shown that $\mbe\left[\norm{h_{n}\left(\theta\right)}_{2}\right]$
is Lipschitz in $\Omega_{\theta}$ with constant $\Lambda_{h}$, which
is finite by assumption.

We have now shown, essentially, that the expected versions of the
quantities we wish to control satisfy \coreassum with $N=1$.
We now need to show that the sample versions satisfy \coreassum
with high probability, which will follow from the fact that the sample
versions converge uniformly to their expectations by
Theorem 9.2 of \citet{keener:2011:theoretical}.

First, observe that \assuref{paper_smoothness} holds with probability
one by assumption. For the remaining assumption choose an $\epsilon>0$,
and define
\begin{align*}
\constg & :=\sqrt{Q_{g}^{2}+\epsilon}\\
\consth & :=\sqrt{Q_{h}^{2}+\epsilon}\\
\constop & :=2Q_{op}\\
\liph & :=\sqrt{D^{4}\Lambda_{h}^{2}+\epsilon}.
\end{align*}

By \citet{keener:2011:theoretical} Theorem 9.2,
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}-\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]\right| & \xrightarrow[N\rightarrow\infty]{p}0.
\end{align*}
Because
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}\right| & >Q_{g}^{2}+\epsilon\ge\sup_{\theta\in\Omega_{\theta}}\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]+\epsilon\Rightarrow\\
\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}-\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]\right| & >\epsilon,
\end{align*}

we have
\begin{align*}
 & P\left(\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}\right|\ge Q_{g}^{2}+\epsilon\right)\le\\
 & \quad P\left(\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}-\mbe\left[\norm{g_{n}\left(\theta\right)}_{2}^{2}\right]\right|\le\epsilon\right),
\end{align*}
so
\begin{align*}
P\left(\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\sum_{n=1}^{N}\norm{g_{n}\left(\theta\right)}_{2}^{2}\right|\ge\constg^{2}\right) & \xrightarrow[N\rightarrow\infty]{}0.
\end{align*}
An analogous argument holds for $\frac{1}{N}\norm{h_{n}\left(\theta\right)}_{2}^{2}$.
Consequently, $P\left(\textrm{Assumption \ref{assu:paper_bounded} holds}\right)\xrightarrow[N\rightarrow\infty]{}1$.

We now consider \assuref{paper_hessian}. Again, by \citet{keener:2011:theoretical} Theorem 9.2
applied to each element of the matrix $h_{n}\left(\theta\right)$,
using a union bound over each of the $D^{2}$ entries,
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\norm{\frac{1}{N}\sum_{n=1}^{N}h_{n}\left(\theta\right)-\mbe\left[h_{n}\left(\theta\right)\right]}_{1} & \xrightarrow[N\rightarrow\infty]{p}0.
\end{align*}
By the converse of \proprefref{operator_norm_continuity}, because
$\norm{\mbe\left[h_{n}\left(\theta\right)\right]^{-1}}_{op}\le Q_{op}$,
\begin{align*}
\norm{\left(\frac{1}{N}\sum_{n=1}^{N}h_{n}\left(\theta\right)\right)^{-1}}_{op} & >2Q_{op}=\constop\Rightarrow\\
\norm{\frac{1}{N}\sum_{n=1}^{N}h_{n}\left(\theta\right)-\mbe\left[h_{n}\left(\theta\right)\right]}_{1} & >\frac{1}{2}Q_{op}^{-1}.
\end{align*}
Consequently,
\begin{align*}
 & P\left(\norm{\left(\frac{1}{N}\sum_{n=1}^{N}h_{n}\left(\theta\right)\right)^{-1}}_{op}\ge\constop\right)\le\\
 & \quad P\left(\norm{\frac{1}{N}\sum_{n=1}^{N}h_{n}\left(\theta\right)-\mbe\left[h_{n}\left(\theta\right)\right]}_{1}\right)\xrightarrow[N\rightarrow\infty]{p}0,
\end{align*}
and $P\left(\textrm{Assumption \ref{assu:paper_hessian} holds}\right)\xrightarrow[N\rightarrow\infty]{}1.$

Finally, applying \lemref{lipschitz_helper} to $\frac{1}{\sqrt{N}}\norm{h\left(\theta_{2}\right)}_{2}$,
\begin{align*}
\left|\frac{1}{\sqrt{N}}\norm{h\left(\theta_{1}\right)}_{2}-\frac{1}{\sqrt{N}}\norm{h\left(\theta_{2}\right)}_{2}\right| & \le\sup_{\theta\in\Omega_{\theta}}\norm{\frac{\partial}{\partial\theta}\frac{1}{\sqrt{N}}\norm{h\left(\theta\right)}_{2}}_{2}\norm{\theta_{1}-\theta_{2}}_{2}\\
 & \le\frac{D^{2}}{\sqrt{N}}\sup_{\theta\in\Omega_{\theta}}\norm{r\left(\theta\right)}_{2}\norm{\theta_{1}-\theta_{2}}_{2}\\
 & =D^{2}\sqrt{\sup_{\theta\in\Omega_{\theta}}\frac{1}{N}\norm{r\left(\theta\right)}_{2}^{2}}\norm{\theta_{1}-\theta_{2}}_{2}.
\end{align*}
Consequently,
\begin{align*}
\left|\frac{1}{\sqrt{N}}\norm{h\left(\theta_{1}\right)}_{2}-\frac{1}{\sqrt{N}}\norm{h\left(\theta_{2}\right)}_{2}\right| & \ge\liph\norm{\theta_{1}-\theta_{2}}_{2}\Rightarrow\\
D^{2}\sqrt{\sup_{\theta\in\Omega_{\theta}}\frac{1}{N}\norm{r\left(\theta\right)}_{2}^{2}} & \ge\liph\Rightarrow\\
\sup_{\theta\in\Omega_{\theta}}\frac{1}{N}\norm{r\left(\theta\right)}_{2}^{2}-\sup_{\theta\in\Omega_{\theta}}\mbe\left[\norm{r_{n}\left(\theta\right)}_{2}^{2}\right] & \ge\frac{\liph^{2}}{D^{4}}-\sup_{\theta\in\Omega_{\theta}}\mbe\left[\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]\Rightarrow\\
\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\norm{r\left(\theta\right)}_{2}^{2}-\mbe\left[\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]\right| & \ge\frac{\liph^{2}}{D^{4}}-\Lambda_{h}^{2}=\epsilon.
\end{align*}
However, again by \citet{keener:2011:theoretical} Theorem 9.2,
\begin{align*}
\sup_{\theta\in\Omega_{\theta}}\left|\frac{1}{N}\norm{r\left(\theta\right)}_{2}^{2}-\mbe\left[\norm{r_{n}\left(\theta\right)}_{2}^{2}\right]\right| & \xrightarrow[N\rightarrow\infty]{p}0,
\end{align*}
so $P\left(\textrm{Assumption \ref{assu:paper_lipschitz} holds}\right)\xrightarrow[N\rightarrow\infty]{}1$.
\end{proof}


\section{Conclusion}
\label{sec:Conclusion}

In this paper, we propose a two-stages row-by-row parallel VGLCS
algorithm and optimization for both stages.  The proposed two-stage
algorithm is much more efficient than the traditional wavefront method
since it is much regular and much easy to parallelize.

%% it shrinks the length of the
%% critical path of intuitive parallel algorithm.

We use {\em sparse table} to solve the variable gapped longest common
subsequence problem.  The sparse table implementation runs efficiently
in parallel because it improves the thread synchronize and workload
imbalance than the disjoint set.  In particular, we present a
rightmost-pops encoding that is both easy-to-implement and efficient
in a parallel environment.  Our VGLCS algorithm using right-most-pops
encoding sparse table runs in $O(n^2 s / p + n \; \max(\log n, s))$,
where $n$ is the number of data, $p$ is the number of processors, and
$s$ is the block size.

We also present a tree labeling technique that order trees
lexicographically so that we can encode every binary search tree into
a Catalan index.  Then, we apply this technique in building Cartesian
tree building so that the insertion takes amortized $O(1)$ time.  As a
result the VGLCS problem can be solved in $O(n^2 / p + n \log n)$
time with this labeling technique.

Finally, we present a dynamic Catalan index computation algorithm for
sparse table.  We can answer the incremental ranged maximum query run in
amortized $O(1)$ by our sparse table, and it can be applied to the
incremental suffix maximum query as well.  The time complexity of our
VGLCS algorithm is $O(n^2 / p + n \log n)$.  Note that this dynamic
Catalan index computation technique is very general and can be applied
to other circumstances where data is constantly inserted into a binary
tree.

We observed two interesting results from our experiments.  First
blocked sparse table outperforms unblocked sparse tables in a
parallel environment.  When the block size is properly chosen, the
performance of blocked version runs more efficiently than the unblocked
version.  Second, a asymptotically better algorithm may not perform
better in practice.  For example, from our experiments we conclude
that blocked sparse table with rightmost-pops, which has a query time
$O(s)$, actually performs better than a sparse table with $O(1)$
amortized query time.  We believe that an easy and straightforward
implementation is the key of good performance, especially in a
parallel environment.


\chapter{\uppercase{Geometry of Output Quantities and Consistent Solutions} \label{chapter:geometry}}

Here, we describe how the relationship of a geometric quantity called \emph{skewness} in QoI maps impacts the accuracy of consistent solutions to SIPs approximated with random sampling.
While prior work has addressed skewness in the context of solving the SIP, its impact on parameter estimation problems formulated within the Data-Consistent framework has not been previously studied.
We demonstrate that the skewness of a map impacts the \emph{precision} of a parameter estimate.
This is subsequently utilized in Chapter~\ref{chapter:vector-valued} to aggregate data into different components of a QoI map to more precisely estimate parameter values.

In this chapter, we begin with a definition of skewness and overview of a set-based approach for constructing consistent solutions to SIPs in \ref{sec:skewness}.
That review is followed by a series of numerical examples which establish fundamental connections between the skewness of QoI maps and the difficulty of accurately approximating solutions with finite sampling.
Namely, we establish that in addition to the implied invariability of skewness to translations, rotations of maps have no impact on solution accuracy.
Furthermore, we show that the number of samples required to achieve a predefined level of error is directly proportional to the skewness of the QoI map used to solve the inverse problem.

\section{A Brief Literature Review of Skewness}\label{sec:skewness}
\input{set-based/skewness.tex}

We demonstrate that the number of samples required to approximate densities using uniform i.i.d.~sampling is proportional to the skewness of the map used for inversion, though the convergence rate of the algorithm used to solve the SIP is unaffected.
We focus on the accuracy of the consistent solutions to the SIP.
It is illustrative to begin with the original set-based approximations to solutions developed in \cite{BBE11}, \cite{BES12}, and \cite{BET+14} as the dependence of solutions on skewness is more explicit than in the density-based approach.
While the content of this chapter is concerned with estimating distributions and not individual parameter estimates, we remind the reader that the solutions to the SIP are inherently densities.
In Chapter~\ref{chapter:mud}, we used these densities to produce a parameter estimate, and so we are motivated to study the accuracy of these solutions to the SIP.

%%%%%

\section{Set-Based Inversion for Consistent Measures}\label{sec:set-based}
% Intro
\input{set-based/set_derivation.tex}
%% \input{ch02/set_derivation_bayes.tex}

% Numerical Approximation
\input{set-based/set_algorithm.tex}

% Descriptions of Error
\input{set-based/set_error.tex}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% discussion of convergence

\subsection{Convergence}
To study the accuracy of the solutions to SIPs using QoI maps with different skewness values (or number of available model evaluations), we need to select a metric on the space of probability measures.
Rates of convergence depend on this choice.
Since the spaces $\pspace$ we are considering are generally bounded and finite, the Total Variation metric metrizes weak convergence (see Thm. 6 in \cite{GS02}).
The latter property of the metric is of notable importance because the QoI maps we study are indeed (component-wise) functionals on the space of model inputs $\pspace$.
Thus, convergence of a sequence of probability measures under the Total Variation metric implies that the QoIs will also converge component-wise in $\RR$.
In other words, convergence in the Total Variation metric implies the convergence of the sampled QoI map to the exact QoI map since the map is a linear functional of the probability measure.
More formally, if $\PP_{\pspace,\ndiscs,\nsamps,h}$ converges to either $\PP_{\pspace,\ndiscs,\nsamps}$, $\PP_{\pspace,\ndiscs}$, or $P_\pspace$ using the Total Variation metric, this implies that the error converged to zero in the numerically computed $\qoi(\param^{(j)})$.
Thus, convergence in the Total Variation metric implies convergence of the numerical method used to construct the QoI map.
Furthermore, recall that weak convergence $\PP_n \to \PP$ is defined to mean
\[
\int f \PP_n \to \int f \PP \text{ as } n \to \infty
\]
for bounded Lipschitz functions $f:\pspace\to\RR$.
Taking $f = \Chi_A$, this leads to the following implication:
\[
\PP_{\pspace, \ndiscs, \nsamps} \to P_\pspace \implies \PP_{\pspace, \ndiscs, \nsamps} (A) \to P_\pspace (A) \quad \forall{A\in\pborel}.
\]
%provided we rigorously define $\P\PP_{\pspace, \ndiscs, \nsamps}$ to measure sets in $\BB_\pspace$, which we proceed to do in the following section\footnote{\bf{I know this is clumsy, but I'm not exactly sure how to phrase this correctly, because it almost seems like this would be true for all $A\in\pspace$ instead. I suppose the omission of the differential operator above is intentionally vague to gloss over this detail. Can we sharpen this up?}}
We choose Total Variation as the metric used in the numerical results of Section~\ref{sec:ch03-set} because of its common use in the literature as the ``statistical distance'' between densities \cite{GS02, Silverman}, and implications for convergence.

\input{set-based/set_accuracy}
% \input{ch03/sample_accuracy}

% \input{set-based/heatrod.tex}
% \FloatBarrier
%
% \input{ch03/decay.tex}

\section{Conclusions}
In this chapter we reviewed the skewness property of QoI maps and demonstrated that for solutions to the SIP, maps which exhibit lower values of skewness exhibit lower overall finite-sampling approximation error against a reference solution.
We saw that the solutions are invariant to transformations such as rotation (which also holds for translations).
The latter observation may aid in the selection of optimal maps (i.e., one can disregard the direction of the maps' contours and focus solely on the angles among the generalized contours).
In the following chapter, we show how vector-valued QoI maps such as those in Chapter~\ref{chapter:mud} constructed from data relate to the geometry of the resulting data-spaces and subsequently impacts the solutions to the SIP.
We demonstrate that even a basic awareness and consideration of skewness can aid in the construction of maps which are more informative and thus increase the precision of parameter estimates.

\FloatBarrier