\label{s:appendix:interaction}
The BCS theory of superconductivity is often introduced as
a consequence of the effective electron-electron interaction
arising from the underlying electron-phonon interaction.
In this appendix, we review general two-particle interactions,
and derive the form of the electron-electron interaction
when applied to the tight-binding model.
These results are the foundation for deriving the
intrinsic TMD superconducting state.

\section{Electron-electron interaction}

Consider a general spin-independent interaction potential
$v \of{\vc{x}, \vc{x}'}$ which depends on the positions
$\vc{x}$ and $\vc{x}'$ of a pair of particles
\cite{ballentine1998quantum}.
In a multiparticle system of dimension $d$ with no self-interaction,
the interaction $V$ may be written as an additive pair operator
in a given bases,
$C_α^† \Ket{0} = \Ket{α} = \Ket{f_α} ⊗ \Ket{σ_α}$,
of fermionic Fock operators,
\begin{equation}
  \label{eq:interaction:general}
  \begin{aligned}
  V
  & = \frac{1}{2} ∑_{i ≠ j} v \of{\vc{Q}_i, \vc{Q}_j}, \\
  & = \frac{1}{4} \sumInt
    \Braket{α β | v | γ δ} C_α^† C_β^† C_δ C_γ, \\ % chktex 19
  & = \frac{1}{2} \sumInt
    v_{α β, γ δ} C_α^† C_β^† C_δ C_γ. % chktex 19
  \end{aligned}
\end{equation}
Here, $\Ket{α β} = \left( \Ket{α} \Ket{β}
- \Ket{β} \Ket{α} \right) / \sqrt{2}$,
and we introduce the symbol
\begin{equation}
  \begin{aligned}
    v_{α β, γ δ} % chktex 19
    & = \left( \Bra{f_α} \Bra{f_β} \right) v
        \left( \Ket{f_γ} \Ket{f_δ} \right) % chktex 19
        \Braket{σ_α | σ_γ} \Braket{σ_β | σ_δ}, \\ % chktex 19
    & = δ_{σ_α σ_γ} δ_{σ_β σ_δ} % chktex 19
        ∫ \cc{f}_α \of{\vc{x}} \cc{f}_β \of{\vc{x}'} v % chktex 19
        \of{\vc{x}, \vc{x}'} f_γ \of{\vc{x}} f_δ \of{\vc{x}'} % chktex 19
        \dif^d \vc{x} \dif^d \vc{x}',
  \end{aligned}
\end{equation}
so that $\Braket{α β | v | γ δ} = v_{α β, γ δ} - v_{α β, δ γ}$. % chktex 19
For a given interaction, the explicit form of $V$
depends on the choice of basis states.
For illustration, we consider
the position and momentum space descriptions below.

If one chooses the continuous basis of position eigenstates,
$Ψ_σ^† \of{\vc{x}} \Ket{0} = \Ket{\vc{x}} ⊗ \Ket{σ}$,
then \cref{eq:interaction:general} takes the form of an integral over
density-density interactions,
\begin{equation}
  \label{eq:interaction:position-space}
  V
  = \frac{1}{2}
    ∫ v \of{\vc{x}, \vc{x}'}
    \normalorder{ρ \of{\vc{x}} ρ \of{\vc{x}'}}
    \dif^d \vc{x} \dif^d \vc{x}',
\end{equation}
where the colon denotes normal ordering and
\begin{equation}
  ρ \of{\vc{x}} = ∑_σ Ψ_σ^† \of{\vc{x}} Ψ_σ \of{\vc{x}}.
\end{equation}

For interactions $v \of{\vc{r}}$
which depend only on the relative separation,
$\vc{r} = \vc{x}' - \vc{x}$,
if one chooses the countable basis
of box normalized momentum eigenstates in a volume $Ω$
with periodic boundary conditions,
$c_{\vc{p} σ}^† \Ket{0} = \Ket{\vc{p}} ⊗ \Ket{σ}$,
with
$\Braket{\vc{x} | \vc{p}} = Ω^{-1 / 2} e^{i \vc{p} · \vc{x}}$,
then
\begin{equation}
  \label{eq:interaction:momentum-space}
  V
  = \frac{1}{2}
    ∑_{\vc{p}, \vc{p'}, \vc{q}}
    ∑_{σ, σ'}
    \tilde{v}_{\vc{q}}
    c_{\vc{p} + \vc{q} σ}^† c_{\vc{p}' - \vc{q} σ'}^†
    c_{\vc{p}' σ'} c_{\vc{p} σ},
\end{equation}
where
\begin{equation}
  \label{eq:interaction:fourier:coefficients}
  \tilde{v}_{\vc{q}}
  = \frac{1}{Ω} ∫ v \of{\vc{r}} e^{-i \vc{q} · \vc{r}} \dif^d \vc{r}.
\end{equation}

\subsection{Interaction in the tight-binding model}

We now apply the above to the basis introduced in the tight-binding model.
Introduce a Wannier representation,
\begin{equation}
  \Ket{ϕ_{ν \vK}}
  = \frac{1}{\sqrt{N}}
    ∑_{\vR} e^{i \vK ⋅ \vR}
    \Ket{ϕ_{ν \vR}},
\end{equation}
where for convenience we write $\vR$ for $\vRn{n}$.
We adopt the following section-specific notation for the Fock operators,
\begin{subequations}
  \begin{alignat}{2}
    a_{\vK ν σ}^† \Ket{0}
    &  = \Ket{ϕ_{ν \vK σ}}
    && = \Ket{ϕ_{ν \vK}} ⊗ \Ket{σ}, \\
    a_{\vR ν σ}^† \Ket{0}
    &  = \Ket{ϕ_{ν \vR σ}}
    && = \Ket{ϕ_{ν \vR}} ⊗ \Ket{σ},
  \end{alignat}
\end{subequations}
which explicitly separates out the spin index.
Since
\begin{equation}
  \Braket{\vc{x} | ϕ_{ν \vR}}
  = \Braket{\vc{x} - \vR | φ_ν}
  = φ_ν \of{\vc{x} - \vR},
\end{equation}
by keeping only on-center, like-orbital terms,
the interaction may by simplified to the approximate form
\begin{equation}
  V
  ⋍ \frac{1}{2}
    ∑_{\vR, \vR'}
    ∑_{ν, ν'}
    v_{\vR \vR'}^{ν ν'}
    \normalorder{n_{\vR ν} n_{\vR' ν'}},
\end{equation}
with the number operators,
\begin{equation}
  n_{ν \vR}
  = ∑_σ a_{\vR ν σ}^† a_{\vR ν σ},
\end{equation}
and interaction integral,
\begin{equation}
  v_{\vR \vR'}^{ν ν'}
  = ∫ v \of{\vc{x}, \vc{x}'}
    \abs{φ_ν \of{\vc{x} - \vR}}^2
    \abs{φ_{ν'} \of{\vc{x}' - \vR'}}^2
    \dif^d \vc{x} \dif^d \vc{x}'.
\end{equation}
This sum over density-density interactions mimics
\cref{eq:interaction:position-space}.

Further, assuming that the interaction
depends only on the relative separation,
$v_{\vR \vR'}^{ν ν'} = v_{ν ν'} \of{\vR' - \vR}$,
we may introduce the Fourier expansion
\begin{equation}
  \tilde{v}^{ν ν'}_{\vc{q}}
  = ∑_{\vR} v_{ν ν'} \of{\vR} e^{i \vc{q} · \vR},
\end{equation}
and switch back to the Bloch representation,
\begin{multline}
  \label{eq:interaction:tight-binding:momentum}
  V
  = \frac{1}{2} {\left( \frac{Ω}{N} \right)}^2
    ∑_{ν, ν'}
    ∑_{σ, σ'}
    ∑_{\vK', \bar{\vK}}
    ∑_{\vK, \bar{\vK}'}
    ∑_{\vc{q}}
    \tilde{v}^{ν ν'}_{\vc{q}} \\
    δ \left[ \vc{q} - \left(\vK - \bar{\vK} \right) \right] % chktex 19
    δ \left[ \vc{q} - \left(\bar{\vK}' - \vK' \right) \right] % chktex 19
    a_{\bar{\vK} ν σ}^† a_{\bar{\vK}' ν' σ'}^†
    a_{\vK' ν' σ'} a_{\vK ν σ}.
\end{multline}
As a convenience, we have written this
as a sum over a countable set of momentum states,
however any sum over momentum may be converted to an integral
according to the substitution
$∑_{\vK} → \left( N / Ω \right) ∫ \dif^d \vK$.
Using this, we may integrate out the $δ$-functions % chktex 19
and obtain a form that mimics
\cref{eq:interaction:momentum-space},
\begin{equation}
  \label{eq:interaction:tight-binding:final}
  V
  = \frac{1}{2}
    ∑_{\vK, \vK', \vc{q}}
    ∑_{ν, ν'}
    ∑_{σ, σ'}
    \tilde{v}^{ν ν'}_{\vc{q}}
    a_{\vK + \vc{q} ν σ}^† a_{\vc{\vK' - \vc{q}} ν' σ'}^†
    a_{\vK' ν' σ'} a_{\vK ν σ}.
\end{equation}


\label{ch:pert}

\section{Predicting interface states}

The \maestro\ hyperbolic equations come in two forms: advective and
conservative.  The procedure for predicting interface states differs
slightly depending on which form we are dealing with.

\subsection{Advective form}

Most of the time, we are dealing with equations in advective form.
Here, a scalar, $\phi$, obeys:
\begin{equation}
\frac{\partial \phi}{\partial t} = -\Ub \cdot \nabla \phi + f
\end{equation}
where $f$ is the force.  This is the form that the perturbation
equations take, as well as the equation for $X_k$ itself.

A piecewise linear prediction of $\phi$ to the interface
would appear as:
\begin{eqnarray}
\phi_{i+1/2,j}^{n+1/2} &=& \phi_{i,j}^n
    + \left . \frac{\Delta x}{2} \frac{\partial \phi}{\partial x} \right |_{i,j}
    + \left . \frac{\Delta t}{2} \frac{\partial \phi}{\partial t} \right |_{i,j} \\
 &=& \phi_{i,j}^n
    + \left . \frac{\Delta x}{2} \frac{\partial \phi}{\partial x} \right |_{i,j}
    +  \frac{\Delta t}{2} \left ( -u \frac{\partial \phi}{\partial x}
                                         -v \frac{\partial \phi}{\partial y} + f \right ) \\
 &=& \phi_{i,j}^n + \frac{\Delta x}{2} \left ( 1 - \frac{\Delta t}{\Delta x} u \right )
           \frac{\partial \phi}{\partial x}
    \underbrace{- \frac{\Delta t}{2} v \frac{\partial \phi}{\partial y}}_{\text{``transverse~term''}} + \frac{\Delta t}{2} f
\end{eqnarray}
(see the Godunov notes section for more details).  Here, the
``transverse term'' is accounted for in {\tt make\_edge\_scal}.  Any
additional forces should be added to $f$.  For the perturbation form
of equations, we add additional advection-like terms to $f$ by calling
{\tt modify\_scal\_force}.  This will be noted below.

\subsection{Conservative form}

A conservative equation takes the form:
\begin{equation}
\frac{\partial \phi}{\partial t} = -\nabla \cdot ( \phi \Ub) + f
\end{equation}
%
Now a piecewise linear prediction of $\phi$ to the interface is
\begin{eqnarray}
\phi_{i+1/2,j}^{n+1/2} &=& \phi_{i,j}^n
    + \left . \frac{\Delta x}{2} \frac{\partial \phi}{\partial x} \right |_{i,j}
    + \left . \frac{\Delta t}{2} \frac{\partial \phi}{\partial t} \right |_{i,j} \\
 &=& \phi_{i,j}^n
    + \left . \frac{\Delta x}{2} \frac{\partial \phi}{\partial x} \right |_{i,j}
    +  \frac{\Delta t}{2} \left ( -\frac{\partial (\phi u)}{\partial x}
                                  -\frac{\partial (\phi v)}{\partial y} + f \right ) \\
 &=& \phi_{i,j}^n + \frac{\Delta x}{2} \left ( 1 - \frac{\Delta t}{\Delta x} u \right )
           \frac{\partial \phi}{\partial x}
    \underbrace{- \frac{\Delta t}{2} \phi \frac{\partial u}{\partial x} }_{\text{``non-advective~term''}}
                \underbrace{- \frac{\Delta t}{2} \frac{\partial (\phi v)}{\partial y}}_{\text{``transverse~term''}} + \frac{\Delta t}{2} f
\end{eqnarray}
Here the ``transverse term'' is now in conservative form, and an additional
term, the non-advective portion of the
$x$-flux (for the $x$-prediction) appears.  Both of the underbraced terms are
accounted for in {\tt make\_edge\_scal} automatically when we call it
with {\tt is\_conservative = .true.}.

%-----------------------------------------------------------------------------
% density
%-----------------------------------------------------------------------------
\section{Density Evolution}

\label{sec:pred:density}

\subsection{Basic equations}

The full density evolution equation is
\begin{eqnarray}
\frac{\partial\rho}{\partial t} &=& -\nabla\cdot(\rho\Ub) \nonumber \\
&=& -\Ub\cdot\nabla\rho - \rho\nabla\cdot\Ub \, . \label{rho equation}
\end{eqnarray}
The species are evolved according to
\begin{eqnarray}
\frac{\partial(\rho X_k)}{\partial t} &=& -\nabla\cdot(\rho\Ub X_k) + \rho \omegadot_k \nonumber \\
&=& -\Ub\cdot\nabla(\rho X_k) - \rho X_k \nabla\cdot\Ub + \rho \omegadot_k \, . \label{species equation}
\end{eqnarray}
In practice, only the species evolution equation is evolved, and the
total density is set as
\begin{equation}
\rho = \sum_k (\rho X_k)
\end{equation}
To advance $(\rho X_k)$ to the next timestep, we need to predict a
time-centered, interface state.  Algebraically, there are multiple
paths to get this interface state---we can predict $(\rho X)$ to the
edges as a single quantity, or predict $\rho$ and $X$ separately
(either in full or perturbation form).  In the notes below, we use the
subscript `{\tt edge}' to indicate what quantity was {\em predicted} to the
edges.  In \maestro, the different methods of computing $(\rho X)$ on
edges are controlled by the \runparam{species\_pred\_type} parameter.  The
quantities predicted to edges and the
resulting edge state are shown in the table~\ref{table:pred:species}

\begin{table}[h]
\centering
\caption{Summary of species edge state construction}
\label{table:pred:species}
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{l|c|c}
\hline
\hline
{\tt species\_pred\_type} &   {quantities predicted} & {$(\rho X_k)$ edge state} \\[-5pt]
 & {in {\tt make\_edge\_scal}} & \\
\hline
1 / {\tt predict\_rhoprime\_and\_X}  &
  $\rho'_\mathrm{edge}$, $(X_k)_\mathrm{edge}$ &
  $\left(\rho_0^{n+\myhalf,{\rm avg}}
  + \rho'_\mathrm{edge} \right)(X_k)_\mathrm{edge}$ \\
2 / {\tt predict\_rhoX}  &
  $\sum(\rho X_k)_\mathrm{edge}$, $(\rho X_k)_\mathrm{edge}$ &
  $(\rho X_k)_\mathrm{edge}$ \\
3 / {\tt predict\_rho\_and\_X}  &
  $\rho_\mathrm{edge}$, $(X_k)_\mathrm{edge}$ &
  $\rho_\mathrm{edge} (X_k)_\mathrm{edge}$ \\
\hline
\end{tabular}
\end{table}
We note the labels {\tt predict\_rhoprime\_and\_X}, {\tt predict\_rhoX}, and
{\tt predict\_rho\_and\_X} are provided by the {\tt pred\_parameters}
module.

\subsection{Method 1: {\tt species\_pred\_type} = {\tt predict\_rhoprime\_and\_X}}

Here we wish to construct $(\rho_0^{n+\myhalf,{\rm avg}}
+ \rho'_\mathrm{edge})(X_k)_\mathrm{edge}$.

We predict both $\rho'$ and $\rho_0$ to edges separately and later use them to
reconstruct $\rho$ at edges.  The base state density evolution equation is
\begin{equation}
\frac{\partial\rho_0}{\partial t} = -\nabla\cdot(\rho_0 w_0 \eb_r) =
-w_0\frac{\partial\rho_0}{\partial r}
\underbrace{-\rho_0\frac{\partial w_0}{\partial r}}_{``\rho_0 ~ \text{force"}}.
\label{rho0 equation}
\end{equation}
Subtract (\ref{rho0 equation}) from (\ref{rho equation}) and rearrange
terms, noting that $\Ub = \Ubt + w_o\eb_r$, to obtain the
perturbational density equation,
\begin{equation}
\frac{\partial\rho'}{\partial t} = -\Ub\cdot\nabla\rho' \underbrace{- \rho'\nabla\cdot\Ub
- \nabla\cdot(\rho_0\Ubt)}_{\rho' ~ \text{force}} \, .
\label{rhoprime equation}
\end{equation}

We also need $X_k$ at the edges.  Here, we subtract $X_k \times$
Eq.~\ref{rho equation} from Eq.~\ref{species equation} to obtain
\begin{equation}
\frac{\partial X_k}{\partial t} = -\Ub \cdot \nabla X_k + \omegadot_k
\end{equation}
When using Strang-splitting, we ignore the $\omegadot_k$ source terms, and
then the species equation is a pure advection equation with no force.

\subsubsection{Predicting $\rho'$ at edges}
We define $\rho' = \rho^{(1)} - \rho_0^n$.  Then we predict $\rho'$ to
edges using {\tt make\_edge\_scal} in {\tt density\_advance} and the
underbraced term in Eq.~\ref{rhoprime equation} as the forcing.  This
force is computed in {\tt modify\_scal\_force}.  This prediction is
done in advective form.

\subsubsection{Predicting $\rho_0$ at edges}\label{Predicting rho0 at edges}
There are two ways to predict $\rho_0$ at edges.
\begin{enumerate}

\item We call {\tt make\_edge\_state\_1d} using the underbraced term
in (\ref{rho0 equation}) as the forcing.  This gives us
$\rho_0^{n+\myhalf,{\rm pred}}$.  This term is used to advect $\rho_0$
in {\bf Advect Base Density}.  In plane-parallel geometries, we also use
$\rho_0^{n+\myhalf,{\rm pred}}$ to compute $\etarho$, which will be used
to compute $\psi$.

\item We define $\rho_0^{n+\myhalf,{\rm avg}} = (\rho_0^n +
\rho_0^{(2)})/2$.  We compute $\rho_0^{(2)}$ from $\rho_0^n$ using
{\bf Advect Base Density}, which advances equation (\ref{rho0 equation})
through $\Delta t$ in time.  The $(2)$ in the superscript indicates
that we have not called {\bf Correct Base} yet, which computes
$\rho_0^{n+1}$ from $\rho_0^{(2)}$.  We use $\rho_0^{(2)}$ rather than
$\rho_0^{n+1}$ to construct $\rho_0^{n+\myhalf,{\rm avg}}$ since $\rho_0^{n+1}$
is not available yet.  $\rho_0^{n+\myhalf,{\rm avg}}$ is used to construct
$\rho$ at edges from $\rho'$ at edges, and
this $\rho$ at edges is used to compute fluxes for $\rho X_k$.
\end{enumerate}

We note that in essence these choices reflect a hyperbolic (1)
vs.\ elliptic (2) approach.  In \maestro, if we setup a problem with
$\rho = \rho_0$ initially, and enforce a constraint $\nabla \cdot
(\rho_0 \Ub) = 0$ (i.e.\ the anelastic constraint), then analytically,
we should never generate a $\rho'$.  To realize this behavior
numerically, we use $\rho_0^{n+\myhalf,{\rm avg}}$ in the prediction
of $(\rho X_k)$ on the edges to be consistent with the use of the
average of $\rho$ to the interfaces in the projection step at the end
of the algorithm.

\subsubsection{Computing $\rho$ at edges}\label{Computing rho at edges}
For the non-radial edges, we directly add $\rho_0^{n+\myhalf,{\rm avg}}$
to $\rho'$ since $\rho_0^{n+\myhalf,{\rm avg}}$ is a cell-centered
quantity.  For the radial edges, we interpolate to obtain
$\rho_0^{n+\myhalf,{\rm avg}}$ at radial edges before adding it to
$\rho'$.

\subsubsection{Predicting $X_k$ at edges}
\label{sec:pert:predict_X}
Predicting $X_k$ is straightforward.  We convert the cell-centered
$(\rho X_k)$ to $X_k$ by dividing by $\rho$ in each zone and then we
just call {\tt make\_edge\_scal} in {\tt density\_advance} on $X_k$.
The force seen by {\tt make\_edge\_scal} is 0.  The prediction is
done in advective form.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 2: {\tt species\_pred\_type} = {\tt predict\_rhoX}}

Here we wish to construct $(\rho X_k)_\mathrm{edge}$ by predicting
$(\rho X_k)$ to the edges as a single quantity.  We recall
Eq.~\ref{species equation}:
\begin{equation}
\frac{\partial(\rho X_k)}{\partial t} =
  -\nabla \cdot (\rho \Ub X_k) + \rho \omegadot_k \, . \nonumber
\end{equation}
The edge state is created by calling {\tt make\_edge\_scal} in {\tt
  density\_advance} with {\tt is\_conservative = .true.}.
We do not consider the $\rho \omegadot_k$ term in the forcing when
Strang-splitting.

We note that even though it is not needed here, we still compute
$\rho_\mathrm{edge}=\sum(\rho X_k)_\mathrm{edge}$ at the edges since certain
enthalpy formulations need it.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 3: {\tt species\_pred\_type} = {\tt predict\_rho\_and\_X}}

Here we wish to construct $\rho_\mathrm{edge} (X_k)_\mathrm{edge}$
by predicting $\rho$ and $X_k$ to the edges separately.

Predicting $X_k$ to the edges proceeds exactly as described in
\S~\ref{sec:pert:predict_X}.

Predicting the full $\rho$ begins with Eq.~\ref{rho equation}:
\begin{equation}
\frac{\partial\rho}{\partial t}
= -\Ub\cdot\nabla\rho \, \underbrace{- \rho\nabla\cdot\Ub}_{``\rho~\text{force''}} \, . \label{rho equation labeled}
\end{equation}
Using this, $\rho$ is predicted to the edges using {\tt
  make\_edge\_scal} in {\tt density\_advance}, with the underbraced
force computed in {\tt modify\_scal\_force} with {\tt fullform =
  .true.}.  \MarginPar{we need to switch this over to doing the conservative prediction}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Advancing $\rho X_k$}\label{Advancing rhoX_k}
The evolution equation for $\rho X_k$, ignoring the reaction terms
that were already accounted for in {\tt react\_state}, and the
associated discretization is \\

\noindent {\tt species\_pred\_type = predict\_rhoprime\_and\_X}:
\begin{equation}
\frac{\partial\rho X_k}{\partial t} =
-\nabla\cdot\left\{\left[\left({\rho_0}^{n+\myhalf,{\rm avg}}
+ \rho'_\mathrm{edge} \right)(X_k)_\mathrm{edge} \right](\Ubt+w_0\eb_r)\right\}.
\end{equation} \\

\noindent {\tt species\_pred\_type = predict\_rhoX}:
\begin{equation}
\frac{\partial\rho X_k}{\partial t} =
-\nabla\cdot\left\{\left[\left(\rho X_k \right)_\mathrm{edge} \right](\Ubt+w_0\eb_r)\right\}.
\end{equation} \\

\noindent {\tt species\_pred\_type = predict\_rho\_and\_X}:
\begin{equation}
\frac{\partial\rho X_k}{\partial t} =
-\nabla\cdot\left\{\left[\rho_\mathrm{edge} (X_k)_\mathrm{edge} \right](\Ubt+w_0\eb_r)\right\}.
\end{equation}

%-----------------------------------------------------------------------------
% energy
%-----------------------------------------------------------------------------
\section{Energy Evolution}

\label{sec:pred:enthalpy}

\subsection{Basic equations}

\maestro\ solves an enthalpy equation.
The full enthalpy equation is
\begin{eqnarray}
\frac{\partial(\rho h)}{\partial t} &=& -\nabla\cdot(\rho h \Ub) + \frac{Dp_0}{Dt}
+ \nabla\cdot \kth \nabla T + \rho H_{\rm nuc} + \rho H_{\rm ext} \nonumber \\
&=& \underbrace{-\Ub\cdot\nabla(\rho h) - \rho h\nabla\cdot\Ub}_{-\nabla\cdot(\rho h\Ub)}
+ \underbrace{\psi + (\Ubt \cdot \er) \frac{\partial p_0}{\partial r}}_{\frac{Dp_0}{Dt}}
+ \nabla\cdot\kth\nabla T + \rho H_{\rm nuc} + \rho H_{\rm ext}.
\end{eqnarray}
%The full temperature equation is
%\begin{equation}
%\frac{\partial T}{\partial t} = -\Ub\cdot\nabla T
%+ \frac{1}{\rho c_p}\left\{(1-\rho h_p)\left[\psi
%+ (\Ubt \cdot \er )\frac{\partial p_0}{\partial r}\right] - \sum_k\rho\xi_k\dot\omega_k
%+ \rho H_{\rm nuc} + \rho H_{\rm ext}\right\}.
%\end{equation}
Due to Strang-splitting of the reactions, the call to {\tt
  react\_state} has already been made.  Hence, the goal is to compute
an edge state enthalpy starting from $(\rho h)^{(1)}$ using an
enthalpy equation that does not include the $\rho H_{\rm nuc}$ and
$\rho H_{\rm ext}$ terms, where were already accounted for in {\tt
  react\_state}, so our equation becomes
\begin{equation}
\frac{\partial(\rho h)}{\partial t} = -\Ub\cdot\nabla(\rho h) - \rho h\nabla\cdot\Ub
+ \underbrace{\psi + (\Ubt \cdot \er) \frac{\partial p_0}{\partial r} + \nabla\cdot\kth\nabla T}_{``(\rho h) ~ \text{force}"} \label{rhoh equation}
\end{equation}
We define the base state enthalpy evolution equation as
\begin{eqnarray}
\frac{\partial(\rho h)_0}{\partial t} &=& -\nabla\cdot[(\rho h)_0 w_0\eb_r]
+ \frac{D_0p_0}{Dt} \nonumber \\
&=& -w_0\frac{\partial(\rho h)_0}{\partial r}
- \underbrace{(\rho h)_0\frac{\partial w_0}{\partial r}+ \psi}_{``(\rho h)_0 ~ \text{force}"}
\enskip .\label{rhoh0 equation}
\end{eqnarray}

\subsubsection{Perturbational enthalpy formulation}

Subtracting (\ref{rhoh0 equation}) from (\ref{rhoh equation}) and
rearranging terms gives the perturbational enthalpy equation
\begin{eqnarray}
\frac{\partial(\rho h)'}{\partial t} &=& -\nabla\cdot[(\rho h)'\Ub]
- \nabla\cdot[(\rho h)_0\Ubt] + (\Ubt \cdot \er)\frac{\partial p_0}{\partial r}
+ \nabla\cdot\kth\nabla T\nonumber \\
&=& -\Ub\cdot\nabla(\rho h)' \underbrace{- (\rho h)'\nabla\cdot\Ub
- \nabla\cdot[(\rho h)_0\Ubt] + (\Ubt \cdot \er)\frac{\partial p_0}{\partial r}
+ \nabla\cdot\kth\nabla T}_{``(\rho h)' ~ \text{force}"}, \label{rhohprime equation}
\end{eqnarray}

\subsubsection{Temperature formulation}

Alternately, we can consider an temperature evolution equation, derived
from enthalpy, as:
\begin{equation}
\frac{\partial T}{\partial t} = -\Ub\cdot\nabla T
+ \frac{1}{\rho c_p}\left\{(1-\rho h_p)\left[\psi
+ (\Ubt \cdot \er )\frac{\partial p_0}{\partial r}\right]
+ \nabla \cdot \kth \nabla T
- \sum_k\rho\xi_k\omegadot_k
+ \rho H_{\rm nuc} + \rho H_{\rm ext}\right\}.
\end{equation}
Again, we neglect the reaction terms, since that will be handled during
the reaction step, so we can write this as:
\begin{equation}
\frac{\partial T}{\partial t} = -\Ub\cdot\nabla T
\underbrace{
+ \frac{1}{\rho c_p}\left\{(1-\rho h_p)\left[\psi
+ (\Ubt \cdot \er )\frac{\partial p_0}{\partial r}\right]
+ \nabla \cdot \kth \nabla T \right \} }_{``T~\text{force''}} \, .
\label{T equation labeled}
\end{equation}

\subsubsection{Pure enthalpy formulation}

A final alternative is to consider an evolution equation for $h$
alone.  This can be derived by expanding the derivative of $(\rho h)$
in Eq.~\ref{rhoh equation} and subtracting off $h \times$ the
continuity equation (Eq.~\ref{rho equation}):
\begin{equation}
\frac{\partial h}{\partial t} = -\Ub \cdot \nabla h
\underbrace{+ \frac{1}{\rho}
\left \{ \psi + (\Ubt \cdot \er )\frac{\partial p_0}{\partial r}
+ \nabla \cdot \kth \nabla T \right \} }_{``h~\text{force''}} \, .
\label{h equation labeled}
\end{equation}

\subsubsection{Prediction requirements}

To update the enthalpy, we need to compute an interface state for
$(\rho h)$.  As with the species evolution, there are multiple
quantities we can predict to the interfaces to form this state,
controlled by \runparam{enthalpy\_pred\_type}.  A complexity of the
enthalpy evolution is that the formation of this edge state will
depend on \runparam{species\_pred\_type}.

The general procedure for making the $(\rho h)$ edge state is as follows:
\begin{enumerate}
\item predict $(rho h)$, $(\rho h)'$, $h$, or $T$ to the edges (depending on {\tt
  enthalpy\_pred\_type} ) using {\tt make\_edge\_scal} and the forces
  identified in the appropriate evolution equation
  (Eqs.~\ref{rhohprime equation}, \ref{T equation labeled}, or \ref{h
    equation labeled} respectively).

  The appropriate forces are summaried in table~\ref{table:pred:hforce}.

\item if we predicted $T$, convert this predicted
  edge state to an intermediate ``enthalpy'' state (the quotes
  indicate that it may be perturbational or full enthalpy) by calling
  the EOS.

\item construct the final enthalpy edge state in {\tt mkflux}.  The
  precise construction depends on what species and enthalpy quantities
  are input to {\tt mkflux}.

\end{enumerate}

Finally, when \maestro\ is run with \runparam{use\_tfromp} {\tt = T}, the
temperature is derived from the density, basestate pressure ($p_0$),
and $X_k$.  When run with reactions or external heating, {\tt
  react\_state} updates the temperature after the reaction/heating
term is computed.  In {\tt use\_tfromp = T} mode, the temperature will
not see the heat release, since the enthalpy does not feed in.  Only
after the hydro update does the temperature gain the effects of the
heat release due to the adjustment of the density (which in turn sees
it through the velocity field and $S$).  As a result, the {\tt
  enthalpy\_pred\_type}s that predict temperature to the interface
({\tt predict\_T\_then\_rhoprime} and {\tt predict\_T\_then\_h}) will
not work.  \maestro\ will abort if the code is run with this
combination of parameters.

Table~\ref{table:pred:hoverview}
gives a summary
of the {\tt enthalpy\_pred\_type} behavior.

\begin{table*}[h]
\centering
\caption{Forcing term into {\tt make\_edge\_scal}}
\label{table:pred:hforce}
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{l|c}
\hline
\hline
{\tt enthalpy\_pred\_type} &   {advective force} \\
\hline
0 / {\tt predict\_rhoh}  $(\rho h)$ & $ \left [\psi + (\Ubt \cdot \er)
  \frac{\partial p_0}{\partial r} + \nabla \cdot \kth \nabla T \right ]$ \\
1 / {\tt predict\_rhohprime} $((\rho h)^\prime)$ &
 $-(\rho h)^\prime \; \nabla \cdot (\Ubt+w_0 \er) -
 \nabla \cdot (\Ubt (\rho h)_0) + (\Ubt \cdot \er) \frac{\partial p_0}{\partial r} + \nabla \cdot \kth \nabla T$ \\
2 / {\tt predict\_h}  $(h)$ & $\frac{1}{\rho} \left [\psi + (\Ubt \cdot \er)
  \frac{\partial p_0}{\partial r} + \nabla \cdot \kth \nabla T \right ]$ \\
3 / {\tt predict\_T\_then\_rhohprime} $(T)$ & $\frac{1}{\rho c_p} \left \{ (1 - \rho h_p)
   \left [\psi + (\Ubt \cdot \er) \frac{\partial p_0}{\partial r} \right ] + \nabla \cdot \kth \nabla T \right \}$ \\
4 / {\tt predict\_T\_then\_h}  $(T)$ & $\frac{1}{\rho c_p} \left\{ (1 - \rho h_p) \left [\psi + (\Ubt \cdot \er)
\frac{\partial p_0}{\partial r}\right ] +  \nabla \cdot \kth \nabla T \right\}$ \\
\hline
\end{tabular}
\end{table*}

\begin{landscape}
\begin{table}[t]
\caption{Summary of enthalpy edge state construction}
\label{table:pred:hoverview}
\renewcommand{\arraystretch}{1.5}
{\small
\centering
\begin{tabular}{l|l|c|c|c|c}
\hline
\hline
{\tt species\_pred\_type} &
{\tt enthalpy\_pred\_type} &
{cell-centered} &
{intermediate} &
{species quantity} &
{final $(\rho h)$} \\[-5pt]
 &
 &
{quantity predicted} &
{``enthalpy''} &
{available in} &
{edge state} \\[-5pt]
 &
 &
{in {\tt make\_edge\_scal}} &
{edge state} &
{\tt mkflux} &
 \\
\hline
1 / {\tt predict\_rhoprime\_and\_X}  & 0 / {\tt predict\_rhoh} &
  $(\rho h)$ & $(\rho h)_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho'_\mathrm{edge}$ &
  $(\rho h)_\mathrm{edge}$ \\
1 / {\tt predict\_rhoprime\_and\_X}  & 1 / {\tt predict\_rhohprime} &
  $(\rho h)'$ & $(\rho h)'_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho'_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
1 / {\tt predict\_rhoprime\_and\_X}  & 2 / {\tt predict\_h} &
  $h$ & $h_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho'_\mathrm{edge}$ &
  $\left ( \rho_0^{n+\myhalf,{\rm avg}} + \rho'_\mathrm{edge} \right ) h_\mathrm{edge}$ \\
1 / {\tt predict\_rhoprime\_and\_X}  & 3 / {\tt predict\_T\_then\_rhohprime} &
  $T$ & $(\rho h)'_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho'_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
1 / {\tt predict\_rhoprime\_and\_X}  & 4 / {\tt predict\_T\_then\_h} &
  $T$ & $h_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho'_\mathrm{edge}$ &
  $\left ( \rho_0^{n+\myhalf,{\rm avg}} + \rho'_\mathrm{edge} \right ) h_\mathrm{edge}$ \\
\hline
\hline
2 / {\tt predict\_rhoX}  & 0 / {\tt predict\_rhoh} &
  $(\rho h)$ & $(\rho h)_\mathrm{edge}$ &
  $(\rho X)_\mathrm{edge}$, $\sum(\rho X)_\mathrm{edge}$ &
  $(\rho h)_\mathrm{edge}$ \\
2 / {\tt predict\_rhoX}  & 1 / {\tt predict\_rhohprime} &
  $(\rho h)'$ & $(\rho h)'_\mathrm{edge}$ &
  $(\rho X)_\mathrm{edge}$, $\sum(\rho X)_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
2 / {\tt predict\_rhoX}  & 2 / {\tt predict\_h} &
  $h$ & $h_\mathrm{edge}$ &
  $(\rho X)_\mathrm{edge}$, $\sum(\rho X)_\mathrm{edge}$ &
  $\sum(\rho X)_\mathrm{edge} h_\mathrm{edge}$ \\
2 / {\tt predict\_rhoX}  & 3 / {\tt predict\_T\_then\_rhohprime} &
  $T$ & $(\rho h)'_\mathrm{edge}$ &
  $(\rho X)_\mathrm{edge}$, $\sum(\rho X)_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
2 / {\tt predict\_rhoX}  & 4 / {\tt predict\_T\_then\_h} &
  $T$ & $h_\mathrm{edge}$ &
  $(\rho X)_\mathrm{edge}$, $\sum(\rho X)_\mathrm{edge}$ &
  $\sum(\rho X)_\mathrm{edge} h_\mathrm{edge}$ \\
\hline
\hline
3 / {\tt predict\_rho\_and\_X}  & 0 / {\tt predict\_rhoh} &
  $(\rho h)$ & $(\rho h)_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho_\mathrm{edge}$ &
  $(\rho h)_\mathrm{edge}$ \\
3 / {\tt predict\_rho\_and\_X}  & 1 / {\tt predict\_rhohprime} &
  $(\rho h)'$ & $(\rho h)'_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
3 / {\tt predict\_rho\_and\_X}  & 2 / {\tt predict\_h} &
  $h$ & $h_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho_\mathrm{edge}$ &
  $\rho_\mathrm{edge} h_\mathrm{edge}$ \\
3 / {\tt predict\_rho\_and\_X}  & 3 / {\tt predict\_T\_then\_rhohprime} &
  $T$ & $(\rho h)'_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho_\mathrm{edge}$ &
  $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho h)'_\mathrm{edge} \right ]$ \\
3 / {\tt predict\_rho\_and\_X}  & 4 / {\tt predict\_T\_then\_h} &
  $T$ & $h_\mathrm{edge}$ &
  $X_\mathrm{edge}$, $\rho_\mathrm{edge}$ &
  $\rho_\mathrm{edge} h_\mathrm{edge}$ \\
\hline
\end{tabular}
} % end \small
\end{table}
\end{landscape}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 0: {\tt enthalpy\_pred\_type} = {\tt predict\_rhoh}}

Here we wish to construct $(\rho h)_\mathrm{edge}$ by predicting
$(\rho h)$ to the edges directly.  We use {\tt make\_edge\_scal} with
{\tt is\_conservative = .true.} on $(\rho h)$, with the underbraced term
in Eq.~\ref{rhoh equation} as the force (computed in {\tt mkrhohforce}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 1: {\tt enthalpy\_pred\_type} = {\tt predict\_rhohprime}}

Here we wish to construct $\left [ (\rho h)_0^{n+\myhalf,{\rm avg}} + (\rho
  h)'_\mathrm{edge} \right ]$ by predicting $(\rho h)'$ to the edges.

\subsubsection{Predicting $(\rho h)'$ at edges}\label{Predicting rhohprime at edges}

We define $(\rho h)' = (\rho h)^{(1)} - (\rho h)_0^n$.  Then we predict
$(\rho h)'$ to edges using {\tt make\_edge\_scal} in {\tt enthalpy\_advance}
and the underbraced term in (\ref{rhohprime equation}) as the forcing (see
also table~\ref{table:pred:hforce} for the forcing term).
The first two terms in $(\rho h)'$ force are computed in
{\tt modify\_scal\_force}, and the last two terms are accounted for in
{\tt mkrhohforce}.  For spherical problems, we have found that a different
representation of the pressure term in the $(\rho h)'$ force gives better
results, namely:
\begin{equation}
(\Ubt \cdot \er)\frac{\partial p_0}{\partial r} \equiv \Ubt\cdot\nabla p_0 =
\nabla\cdot(\Ubt p_0) - p_0\nabla\cdot\Ubt.
\end{equation}

\subsubsection{Predicting $(\rho h)_0$ at edges}
We use an analogous procedure described in Section \ref{Predicting
rho0 at edges} for computing $\rho_0^{n+\myhalf,\rm{avg}}$ to obtain
$(\rho h)_0^{n+\myhalf,\rm{avg}}$, i.e.,
$(\rho h)_0^{n+\myhalf,{\rm avg}} = [(\rho h)_0^{n} + (\rho h)_0^{n+1}]/2$.

For spherical, however, instead of computing $(\rho h)_0$ on edges
directly, we compute $\rho_0$ and $h_0$ separately at the edges, and
multiply to get $(\rho h)_0$.

\subsubsection{Computing $\rho h$ at edges}
We use an analogous procedure described in Section \ref{Computing rho
  at edges} for computing $\rho$ at edges to compute $\rho h$ at
edges.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 2: {\tt enthalpy\_pred\_type} = {\tt predict\_h}}

Here, the construction of the interface state depends on what species
quantities are present.  In all cases, the enthalpy state is found
by predicting $h$ to the edges.

For {\tt species\_pred\_types}: {\tt predict\_rhoprime\_and\_X}, we wish to construct
$(\rho_0 + \rho'_\mathrm{edge} ) h_\mathrm{edge}$.

For {\tt species\_pred\_types}: {\tt predict\_rho\_and\_X} or
{\tt predict\_rhoX}, we wish to construct $\rho_\mathrm{edge} h_\mathrm{edge}$.

\subsubsection{Predicting $h$ at edges}

We define $h = (\rho h)^{(1)}/\rho^{(1)}$.  Then we predict $h$ to edges
using {\tt make\_edge\_scal} in {\tt enthalpy\_advance} and the
underbraced term in Eq.~\ref{h equation labeled} as the forcing (see
also table~\ref{table:pred:hforce}).  This force is computed by
{\tt mkrhohforce} and then divided by $\rho$.  Note: {\tt mkrhoforce}
knows about the different {\tt enthalpy\_pred\_type}s and computes
the correct force for this type.

\subsubsection{Computing $\rho h$ at edges}

{\tt species\_pred\_types}: {\tt predict\_rhoprime\_and\_X}:\\[1mm]
%
We use the same procedure described in Section \ref{Computing rho at
  edges} for computing $\rho_\mathrm{edge}$ from $\rho_0$ and
$\rho'_\mathrm{edge}$ and then multiply by $h_\mathrm{edge}$.

\ \\
{\tt species\_pred\_types}: {\tt predict\_rhoX}:\\[1mm]
%
We already have $\sum(\rho X_k)_\mathrm{edge}$ and simply multiply by
$h_\mathrm{edge}$.

\ \\
{\tt species\_pred\_types}: {\tt predict\_rho\_and\_X}:\\[1mm]
%
We already have $\rho_\mathrm{edge}$ and simply multiply by
$h_\mathrm{edge}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 3: {\tt enthalpy\_pred\_type} = {\tt predict\_T\_then\_rhohprime}}

Here we wish to construct $\left [ (\rho h)_0 + (\rho
  h)'_\mathrm{edge} \right ]$ by predicting $T$ to the edges and then
converting this to $(\rho h)'_\mathrm{edge}$ via the EOS.

\subsubsection{Predicting $T$ at edges}

We predict $T$ to edges using {\tt make\_edge\_scal} in {\tt
  enthalpy\_advance} and the underbraced term in Eq.~\ref{T equation
  labeled} as the forcing (see also table~\ref{table:pred:hforce}).
This force is computed by {\tt mktempforce}.

\subsubsection{Converting $T_\mathrm{edge}$ to $(\rho h)'_\mathrm{edge}$}

We call the EOS in {\tt makeHfromRhoT\_edge} (called from {\tt
  enthalpy\_advance}) to convert from $T_\mathrm{edge}$ to $(\rho
h)'_\mathrm{edge}$.  For the EOS call, we need $X_\mathrm{edge}$ and
$\rho_\mathrm{edge}$.  This construction depends on {\tt
  species\_pred\_type}, since the species edge states may differ
between the various prediction types (see the ``species quantity''
column in table~\ref{table:pred:hoverview}).  The EOS inputs are
constructed as:

\begin{table*}[h]
\centering
\caption{EOS states in {\tt makeHfromRhoT\_edge}}
\label{table:pred:EOSinputs}
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{l|c|c}
\hline
\hline
{\tt species\_pred\_type} & {$\rho$ edge state} &   {$X_k$ edge state}  \\
\hline
{\tt predict\_rhoprime\_and\_X} &
  $\rho_0^{n+\myhalf,\rm{avg}} + \rho'_\mathrm{edge}$ &
  $(X_k)_\mathrm{edge}$ \\
{\tt predict\_rhoX} &
  $\sum_k (\rho X_k)_\mathrm{edge}$ &
  $(\rho X_k)_\mathrm{edge}/\sum_k (\rho X_k)_\mathrm{edge}$ \\
{\tt predict\_rho\_and\_X} &
  $\rho_\mathrm{edge}$ &
  $(X_k)_\mathrm{edge}$ \\
\hline
\end{tabular}
\end{table*}

After calling the EOS, the output of {\tt makeHfromRhoT\_edge} is
$(\rho h)'_\mathrm{edge}$.

\subsubsection{Computing $\rho h$ at edges}

The computation of the final $(\rho h)$ edge state is done identically
as the {\tt predict\_rhohprime} version.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Method 4: {\tt enthalpy\_pred\_type} = {\tt predict\_T\_then\_h}}

Here, the construction of the interface state depends on what species
quantities are present.  In all cases, the enthalpy state is found by
predicting $T$ to the edges and then converting this to
$h_\mathrm{edge}$ via the EOS.

For {\tt species\_pred\_types}: {\tt predict\_rhoprime\_and\_X}, we wish to
construct $(\rho_0 + \rho'_\mathrm{edge} ) h_\mathrm{edge}$.

For {\tt species\_pred\_types}: {\tt predict\_rhoX}, we wish to
construct $\sum(\rho X_k)_\mathrm{edge} h_\mathrm{edge}$.

For {\tt species\_pred\_types}: {\tt predict\_rho\_and\_X}, we wish to
construct $\rho_\mathrm{edge} h_\mathrm{edge}$.

\subsubsection{Predicting $T$ at edges}

The prediction of $T$ to the edges is done identically as the
{\tt predict\_T\_then\_rhohprime} version.

\subsubsection{Converting $T_\mathrm{edge}$ to $h_\mathrm{edge}$}

This is identical to the {\tt predict\_T\_then\_rhohprime} version,
except that on output, we compute $h_\mathrm{edge}$.

\subsubsection{Computing $\rho h$ at edges}

The computation of the final $(\rho h)$ edge state is done identically
as the {\tt predict\_h} version.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Advancing $\rho h$}
We update the enthalpy analogously to the species update in
Section \ref{Advancing rhoX_k}.  The forcing term does not include reaction
source terms accounted for in {\bf React State}, and is the same
for all {\tt enthalpy\_pred\_type}s.
\begin{equation}
\frac{\partial(\rho h)}{\partial t} =
-\nabla\cdot\left\{\left \langle (\rho h) \right \rangle_\mathrm{edge}
 \left(\Ubt + w_0\eb_r\right)\right\} + (\Ubt \cdot \er)\frac{\partial p_0}{\partial r} + \psi  \enskip .
\end{equation}
where $\left \langle (\rho h) \right \rangle_\mathrm{edge}$ is the
edge state for $(\rho h)$ computed as listed in the final column of
table~\ref{table:pred:hoverview} for the given {\tt enthalpy\_pred\_type}
and {\tt species\_pred\_type}.

%-----------------------------------------------------------------------------
% toy_convect
%-----------------------------------------------------------------------------
\section{Experience from {\tt toy\_convect}}

\label{sec:toyconvect}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Why is {\tt toy\_convect} Interesting?}

The {\tt toy\_convect} problem consists of a carbon-oxygen white dwarf with
an accreted layer of solar composition. There is a steep composition
gradient between the white dwarf and the accreted layer. The convection
that begins as a result of the accretion is extremely sensitive to the
amount of mixing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Initial Observations}

With {\tt use\_tfromp = T} and {\tt cflfac = 0.7} there is a large difference
between \runparam{species\_pred\_type} {\tt = 1} and {\tt species\_pred\_type = 2,3} as
seen in Figure \ref{fig:spec1_vs_23}. {\tt species\_pred\_type = 1} shows
quick heating (peak T vs. t) and there is ok agreement between {\tt tfromh}
and {\tt tfromp}. {\tt species\_pred\_type = 2,3} show cooling (peak T vs. t)
and {\tt tfromh} looks completely unphysical (see Figure
\ref{fig:tfromh_unphysical}). There are also strange filament type features in
the momentum plots shown in Figure \ref{fig:mom_filaments}.

% compare spec 1 with spec 2,3
%-----------------------------------------
\begin{figure}[!h]
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/spec1_vs_23}
\caption{Compare {\tt species\_pred\_type = 1,2,3} with {\tt use\_tfromp =
T, enthalpy\_pred\_type = 1, cflfac = 0.7}}
\label{fig:spec1_vs_23}
\end{figure}

% tfromh is unphysical
%-----------------------------------------
\begin{figure}[!h]
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/tfromh_unphysical}
\caption{{\tt tfromh} is unphysical when using {\tt species\_pred\_type =
2,3, enthalpy\_pred\_type = 1, cflfac = 0.7, use\_tfromp = T}. Shown above
is {\tt species\_pred\_type = 2}}
\label{fig:tfromh_unphysical}
\end{minipage}
\hspace{0.5cm}
% momentum filaments
%-----------------------------------------
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/mom_filaments}
\caption{There are strange filament type features at the bottom of the
domain. {\tt species\_pred\_type = 2, enthalpy\_pred\_type = 1, cflfac = 0.7,
use\_tfromp = T}}
\label{fig:mom_filaments}
\end{minipage}
\end{figure}

Using {\tt use\_tfromp = F} and \runparam{dpdt\_factor} {\tt $>$ 0} results in many runs
crashing very quickly and gives unphyiscal temperature profiles as seen in
Figure \ref{fig:tfrompF_unphys}.

% unphysical temp profiles with use_tfromp = F dpdt > 0
%-----------------------------------------
\begin{figure}[!h]
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/tfrompF_unphys}
\caption{Unphysical temperature profile with {\tt use\_tfromp = F} and
{\tt dpdt\_factor = 0.1}. {\tt dpdt\_factor = 0.2,0.3} lead to the code
crashing.}
\label{fig:tfrompF_unphys}
\end{minipage}
\hspace{0.5cm}
% temp vs time of castro ppm 0,1 and maestro spec = 2
%----------------------------------------------
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/tfrompF_cfl_1vs7}
\caption{Compare {\tt cflfac = 0.1} with {\tt cflfac = 0.7} for
{\tt use\_tfromp = F, dpdt\_factor = 0.0, species\_pred\_type = 2,
enthalpy\_pred\_type = 4}}
\label{fig:tfrompF_cfl_1vs7}
\end{minipage}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Change {\tt cflfac} and {\tt enthalpy\_pred\_type}}

With {\tt species\_pred\_type = 1} and {\tt cflfac = 0.1},
there is much less heating (peak T vs. t) than the {\tt cflfac = 0.7}
(default). There is also a lower overall Mach number (see Figure
\ref{fig:spec1_cfl_1vs7}) with the \runparam{cflfac} {\tt = 0.1} and excellent agreement
between {\tt tfromh} and {\tt tfromp}.

{\tt use\_tfromp = F, dpdt\_factor = 0.0, enthalpy\_pred\_type = 3,4 and
species\_pred\_type = 2,3} shows cooling (as seen in {\tt use\_tfromp = T})
with a comparable rate of cooling (see Figure \ref{fig:compare_tfromp})
to the {\tt use\_tfromp = T} case. The
largest difference between the two runs is that the {\tt use\_tfromp = F}
case shows excellent agreement between {\tt tfromh} and {\tt tfromp} with
{\tt cflfac = 0.7}. The filaments in the momentum plot of Figure
\ref{fig:mom_filaments} are still present.

% Mach number of cfl 0.1 vs cfl 0.7
%-----------------------------------------
\begin{figure}[!h]
\begin{minipage}[!b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/spec1_cfl_1vs7}
\caption{Comparing the Mach number of {\tt cflfac = 0.1} and
{\tt cflfac = 0.7}. {\tt species\_pred\_type = 1, enthalpy\_pred\_type = 1}}
\label{fig:spec1_cfl_1vs7}
\end{minipage}
\hspace{0.5cm}
% comparable cooling rates between tfromp = T and tfromp = F
%-----------------------------------------
\begin{minipage}[!b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/compare_tfromp}
\caption{Illustrate the comparable cooling rates between
{\tt use\_tfromp = T} and {\tt use\_tfromp = F with dpdt\_factor = 0.0}
using {\tt species\_pred\_type = 2, enthalpy\_pred\_type = 3,1}}
\label{fig:compare_tfromp}
\end{minipage}
\end{figure}

For a given {\tt enthalpy\_pred\_type} and {\tt use\_tfromp = F},
{\tt species\_pred\_type = 2} has a lower Mach number (vs. t) compared to
{\tt species\_pred\_type = 3}.

Any {\tt species\_pred\_type} with {\tt use\_tfromp = F, dpdt\_factor = 0.0}
and {\tt enthalpy\_pred\_type = 1} shows significant heating, although
the onset of the heating is delayed in {\tt species\_pred\_type = 2,3} (see
Figure \ref{fig:compare_tF_d0_h1_s123}). Only
{\tt species\_pred\_type = 1} gives good agreement between {\tt tfromh} and
{\tt tfromp}.

Comparing {\tt cflfac = 0.7} and {\tt cflfac = 0.1} with
{\tt use\_tfromp = F, dpdt\_factor = 0.0, species\_pred\_type = 2} and
{\tt enthalpy\_pred\_type = 4} shows good agreement overall (see Figure
\ref{fig:tfrompF_cfl_1vs7}).

% compare tfromp=F%, dpdt=0, h=1, spec=1,2,3
%-----------------------------------------
\begin{figure}[!h]
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/compare_tF_d0_h1_s123}
\caption{Compare various {\tt species\_pred\_type} with {\tt use\_tfromp = F,
dpdt\_factor = 0.0, enthalpy\_pred\_type = 1}}
\label{fig:compare_tF_d0_h1_s123}
\end{minipage}
\hspace{0.5cm}
% compare tfromp=F,dpdt=0,spec=2,h=4, cfl=0.1,0.7
%-----------------------------------------
\begin{minipage}[b]{0.5\linewidth}
\vspace{0pt}
\centering
\includegraphics[height=0.3\textheight]{\pertfigpath/compare_castro}
\caption{Compare the {\tt castro.ppm\_type CASTRO} runs with the
{\tt species\_pred\_type MAESTRO} runs.}
\label{fig:compare_castro}
\end{minipage}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Additional Runs}

\subsubsection{{\tt bds\_type = 1}}

Using \runparam{bds\_type} {\tt = 1, use\_tfromp = F, dpdt\_factor = 0.0,
species\_pred\_type = 2, enthalpy\_pred\_type = 4} and
{\tt cflfac = 0.7} seems to cool off faster, perhaps due to less mixing.
There is also no momentum filaments in the bottom of the domain.

\subsubsection{{\tt evolve\_base\_state = F}}

Using \runparam{evolve\_base\_state} {\tt = F, use\_tfromp = F, dpdt\_factor = 0.0,
species\_pred\_type = 2} and {\tt enthalpy\_pred\_type = 4} seems to agree
well with the normal {\tt evolve\_base\_state = T} run.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{{\tt toy\_convect} in {\tt CASTRO}}

{\tt toy\_convect} was also run using {\tt CASTRO} with
{\tt castro.ppm\_type = 0,1}. These runs show temperatures that cool
off rather than increase (see Figure \ref{fig:compare_castro}) which
suggests using {\tt species\_pred\_type = 2,3} instead of
{\tt species\_pred\_type = 1}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Recommendations}

All of these runs suggest that running under {\tt species\_pred\_type =
2 or 3, enthalpy\_pred\_type = 3 or 4} with either {{\tt use\_tfromp = F} and
{\tt dpdt\_factor = 0.0}} or {\tt use\_tfromp = T} gives the most
consistent results.


\documentclass[authordraft=true,dvipsnames]{acmart}

\usepackage{blindtext}
\usepackage{xpatch}
\usepackage{tikz}
\usepackage{subcaption}
\usepackage{pbox}
\usepackage{url}

\usetikzlibrary{matrix,backgrounds}
\pgfdeclarelayer{myback}
\pgfsetlayers{myback,background,main}

% By default the URLs are put in typewriter type in the body and the
% bibliography of the document when using the \url command.  If you are
% using many long URLs you may want to uncommennt the next line so they
% are typeset a little smaller.
\renewcommand{\UrlFont}{\small\tt}

%% Some hacks to remove all the ACM hardcoded text as I only want to use its style
\makeatletter
\xpatchcmd{\ps@firstpagestyle}{Manuscript submitted to ACM}{}{\typeout{First patch succeeded}}{\typeout{first patch failed}}
\xpatchcmd{\ps@standardpagestyle}{Manuscript submitted to ACM}{}{\typeout{Second patch succeeded}}{\typeout{Second patch failed}}    \@ACM@manuscriptfalse% Also in titlepage
\makeatother
\renewcommand\footnotetextcopyrightpermission[1]{} % removes footnote with conference info
\setcopyright{none}
\pagestyle{plain} % remove running headers
%% End of hack

%% Some shorthands
\newcommand{\pref}[1]{(\ref{#1})}
\newcommand{\bx}[0]{\mathbf{X}}
\newcommand{\bw}[0]{\mathbf{W}}
\newcommand{\by}[0]{\mathbf{Y}}
\newcommand{\bb}[0]{\mathbf{B}}

%% Drawing a cube in tikz
\newcommand{\tikzcube}[3]{% width, height, depth
\foreach \x in {0,...,#1}
{   \draw (\x ,0  ,#3 ) -- (\x ,#2 ,#3 );
    \draw (\x ,#2 ,#3 ) -- (\x ,#2 ,0  );
}
\foreach \x in {0,...,#2}
{   \draw (#1 ,\x ,#3 ) -- (#1 ,\x ,0  );
    \draw (0  ,\x ,#3 ) -- (#1 ,\x ,#3 );
}
\foreach \x in {0,...,#3}
{   \draw (#1 ,0  ,\x ) -- (#1 ,#2 ,\x );
    \draw (0  ,#2 ,\x ) -- (#1 ,#2 ,\x );
}
}

%% Highlight the elements in tikz matrix
\tikzset{mycolor/.style = {dotted,line width=1bp,color=#1}}%
\tikzset{myfillcolor/.style = {draw,fill=#1,opacity=0.3}}%

\NewDocumentCommand{\highlight}{O{blue!40} m m}{%
\draw[mycolor=#1] (#2.north west)rectangle (#3.south east);
}

\NewDocumentCommand{\fhighlight}{O{blue!40} m m}{%
\draw[myfillcolor=#1] (#2.north west)rectangle (#3.south east);
}

\title{Convolutional Layer and Its Matrix Implementation}
\author{Yinghai Lu}

\begin{document}
\begin{abstract}
There are quite some tutorials~\cite{cs231n,andrew14,jefkin16} and textbooks~\cite{goodfellow16} explaining the mathematics of convolutional neural networks. However, in order to implement the CNN efficiently, we usually cast it into matrix operations. Few articles discuss this transformation process. \cite{cs231n} does a good job at the forward propagation pass but the backward propagation part is missing. In addition, many of them discuss simplified model with zero padding and $1$-stride. When I was reading the code of $CAFFE2$, I found myself constantly needing to figure out the all the parameters and all the indexing into the tensors. This notes tries to bridge the gap and provides a reference for matrix implementation of convolutional layer, consider various parameters and flavors.
\end{abstract}
\maketitle

\section{Configuration}
The input to the convolutional layer is a 4D tensor $\bx \in \mathbb{R} ^{N \times C \times H_X \times W_X}$ (assuming NCHM format), which means a batch of $N$ images of size $H \times W$, where each image has $C$ channels. A set of $M$ filters of size $C \times H_K \times W_K$ will be applied to those input and generate the output $Y$. The filters are thus represented by a tensor $\bw \in \mathbb{R}^{M \times C \times H_K \times W_K}$. Optionally, we can have a bias vector $\mathbf{b} \in \mathbb{R}^{M}$, which offsets the result of the convolution. The output tensor $\by$ is of size $N \times M \times H_Y \times W_Y$, where the output image (activation map) dimension $H_Y \times W_Y$ is determined by configurations such as padding and stride and etc.

For now, we assume zero-padding and stride of size $1$ for simplicity. We don't take group filters into account either. Later, we will see how to extend it. We also assume $N=1$ and omit the $n$ subscript.

\section{Forward Propagation}
For a certain filter $m$, we drag it through the image to apply the convolution. Each time, we apply it onto a $H_K \times W_K$ tile of the image with $C$ channels and produce one element (pixel) in the output map:
\begin{equation} \label{forward}
\by_{m, i,j}=\sum_{c=0}^{C-1}{\sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\bw_{m,c,p,q}\bx_{c, i+p,j+q}}}} + \mathbf{b}_m
\end{equation}

The above equation can be naturally converted to maxtrix multiplication. Let's first ignore the bias and focus on the convolution term. Given an input sample $\bx \in \mathbb{R}^{C \times H_X \times W_X}$, we are going to apply one filter $H_Y \times W_Y$ times by going from left to right and then from top to bottom. We introduce an operator $Im2Col()$ which takes  $\by$ as input and creates a 2D matrix $\bb \in \mathbb{R}^{(C \times H_K \times W_K) \times (H_Y \times W_Y)}$, where each column $B_{n,:,j}$ is made by reshaping the input area where the filter is going to be applied to into a $C \times H_K \times W_K$ column. As we are moving the filter, we are going to generate such a column $(H_Y \times W_Y$ times from left to right, top to bottom of the image. Here is an example of how we generate $\mathbf{B}$ from a $2 \times 3 \times 3$ input $\mathbf{X}$ with a $2 \times 2 \times 2$ filter kernel.
\begin{figure}[h]
\begin{subfigure}[b]{0.3\textwidth}
\centering
\begin{tikzpicture}[scale=1]
\tikzcube{3}{3}{2}
\draw[fill=Yellow,opacity=0.3] (0,0,2) -- (0,3,2) -- (3,3,2) -- (3,0,2);
\draw[fill=Yellow,opacity=0.3] (3,0,2) -- (3,0,1) -- (3,3,1) -- (3,3,2);
\draw[fill=Yellow,opacity=0.3] (0,3,2) -- (0,3,1) -- (3,3,1) -- (3,3,2);

\node at (-0.25, -0.25, 0) {$\bx_{0,2,0}$};
\node at (-0.25, 0.75, 0) {$\bx_{0,1,0}$};
\node at (-0.25, 1.75, 0) {\color{Red}{$\bx_{0,0,0}$}};
\node at (0.75, -0.25, 0) {$\bx_{0,2,1}$};
\node at (0.75, 0.75, 0) {\color{Green}{$\bx_{0,1,1}$}};
\node at (0.75, 1.75, 0) {\color{Cyan}{$\bx_{0,0,1}$}};
\node at (1.75, -0.25, 0) {$\bx_{0,2,2}$};
\node at (1.75, 0.75, 0) {$\bx_{0,1,2}$};
\node at (1.75, 1.75, 0) {$\bx_{0,0,2}$};
\end{tikzpicture}
\caption{Input $\bx$ of $2 \times 3 \times 3$}
\end{subfigure}
\hspace{20mm}
\begin{subfigure}[b]{0.3\textwidth}
\centering
\begin{tikzpicture}[baseline=-\the\dimexpr\fontdimen22\textfont2\relax ]
\matrix(m)[matrix of math nodes,left delimiter=(,right delimiter=)]
{
\color{Red}{\bx_{0,0,0}} & \color{Cyan}{\bx_{0,0,1}} & \bx_{0,1,0} & \color{Green}{\bx_{0,1,1}}\\
\color{Cyan}{\bx_{0,0,1}} & \bx_{0,0,2} & \color{Green}{\bx_{0,1,1}} & \bx_{0,1,2}\\
\bx_{0,1,0} & \color{Green}{\bx_{0,1,1}} & \bx_{0,2,0} & \bx_{0,2,1}\\
\color{Green}{\bx_{0,1,1}} & \bx_{0,1,2} & \bx_{0,2,1} & \bx_{0,2,2}\\
\bx_{1,0,0} & \bx_{1,0,1} & \bx_{1,1,0} & \bx_{1,1,1}\\
\bx_{1,0,1} & \bx_{1,0,2} & \bx_{1,1,1} & \bx_{1,1,2}\\
\bx_{1,1,0} & \bx_{1,1,1} & \bx_{1,2,0} & \bx_{1,2,1}\\
\bx_{1,1,1} & \bx_{1,1,2} & \bx_{1,2,1} & \bx_{1,2,2}\\
};
\fhighlight[Yellow]{m-1-1}{m-4-4}
\end{tikzpicture}
\caption{$\bb = Im2Col(\bx)$ of $8 \times 4$ }
\end{subfigure}
\caption{An example of $Im2Col()$} \label{buffer}
\end{figure}

With $\bb$, if we think the weight tensor as a 2D matrix of $M \times (C \times H_K \times W_K)$, where each row is just the filter map $m$ flattened, we can easily get the output of applying $\bw$ to $\bx$ as
\begin{equation} \label{bm}
\by = \bw \bb
\end{equation}
where $\by \in \mathbb{R}^{M \times (H_Y \times W_Y)}$.

Now we amend $\by$ to add bias term. What we want to do is to add $\mathbf{b}_i$ to each element in row $i$ of $\mathbf{Y}$. This can be done by generating a increament matrix and do element-wise add to  $\mathbf{Y}$:
\begin{equation} \label{bias}
 \mathbf{Y'} =  \by + \mathbf{b} \mathbf{u}^T
\end{equation}
where $\mathbf{u} \in \mathbb{R}^{(H_Y \times W_Y) \times 1}$ is a vector filled with $1$s. By doing an outer-product between $\mathbf{b}$ and $\mathbf{u}$, we strectch the incremental matrix we want and add it to $ \by$. Note that most numerical libraries offser GEMM routine which does operation in above equation in one call.

Applying this to each of the input batch, we can get $\by$ for each $n \in \{0,\ldots,N-1\}$.

\section{Backward Propagation}
For backward propagation, we are interested in generating the gradients for inputs ($d\bx$), for weights ($d\bw$) and for the bias terms ($d\mathbf{b}$). In addition to $\mathbf{X}$ and $\mathbf{W}$, we also know the output gradient back-propagated from next layer $d\by \in \mathbb{R}^{N \times M \times H_Y \times W_Y}$. We will focus on one sample of the input batch $N$ and hence omit the index $n$ in the following discussion.

The gradient of weight $\bw_{m,c,p,q}$ with respect to the training error is about how much a perturbation in $\mathbf{W}_{m,c,p,q}$ is going to affect the output. It's worth noticing how much $\mathbf{W}_{m,c,p,q}$ is involved in the computation. Basically, for each filter $m$, if we have zero padding and stride of $1$, $\mathbf{W}_{m,c,p,q}$ will be involved in the computation of each pixel of the output activation map with respect to a specific filter and there are $H_Y \times W_Y$ of them. With that,
applying the chain rule on the gradient and we get:
\begin{equation}
\frac{\partial E}{\partial \bw_{m,c,p,q}} = \sum_{i=0}^{H_Y-1}{\sum_{j=0}^{W_Y-1}{\frac{\partial E}{\partial \by_{m,i,j}}\frac{\partial \mathbf{Y}_{m,i,j}}{\partial  \mathbf{W}_{m,c,p,q}}}}
\end{equation}
Substituting $\by_{m,i,j}$ with \pref{forward}, we have
\begin{equation} \label{dw}
\frac{\partial E}{\partial \bw_{m,c,p,q}} = \sum_{i=0}^{H_Y-1}{\sum_{j=0}^{W_Y-1}{\frac{\partial E}{\partial \by_{m,i,j}}\bx_{c,i+p,j+q}}}
\end{equation}

For a given input of $d\by$ from a batch of $N$, we can treat it as a matrix $d\mathbf{Y}$ of size $M \times (H_Y \times W_Y)$. For each row $m$, each column represents an element $\mathbf{Y}_{m,i,j}$, flatterned into one row. Remember the buffer matrix $\bb=Im2Col(\bx)$ is a matrix of size $(C \times H_K \times W_K) \times (H_Y \times W_Y)$, we have
\begin{equation}
d\bw = d\by \bb^T
\end{equation}
To see why, you can expand the matrix multiplication and verify that each element operation corresponds exactly to \pref{dw}.

Next, let's check the gradient of bias $\mathbf{b}_m$. Similar to $\bw_{m,c,p,q}$, $\mathbf{b}_m$ contributes to the computation of each of the pixels in an activation map of size  $H_Y \times W_Y$. Similarly, applying the chain rule and we have
\begin{equation}
\frac{\partial E}{\partial \mathbf{b}_m} =  \sum_{i=0}^{H_Y-1}{\sum_{j=0}^{W_Y-1}{\frac{\partial E}{\partial \by_{m,i,j}}\frac{\partial \mathbf{Y}_{m,i,j}}{\partial \mathbf{b}_m}}}
\end{equation}
From \pref{forward}, we know that $\frac{\partial \by_{m,i,j}}{\partial \mathbf{b}_m} = 1$. Therefore we have
\begin{equation}
d\mathbf{b} = d\by \mathbf{u}
\end{equation}
where $\mathbf{u}$ is the vector with all ones, as we have already seen in \pref{bias}.

Finally, we still need to compute the gradient of input, $d\bx$. Using chain rule, the gradient of each input $\mathbf{X}_{c,i,j}$ is
\begin{equation}
\frac{\partial E}{\partial \bx_{c,i,j}} = \sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\sum_{m=0}^{M-1}{\frac{\partial E}{\partial \by_{m,i-p,j-q}}\frac{\partial \mathbf{Y}_{m,i-p,j-q}}{\partial \mathbf{X}_{c,i,j}}}}}
\end{equation}
Substituting \pref{forward} into the above euqation and we have
\begin{equation} \label{dx}
\frac{\partial E}{\partial \bx_{c,i,j}} = \sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\sum_{m=0}^{M-1}{\frac{\partial E}{\partial \by_{m,i-p,j-q}} \bw_{m,c,p,q}}}}
\end{equation}
Note that the ranges of the three levels of summation show intuitively how a pixel $\bx_{c,i, j}$ contributes to the computation of output activation map $\by$. The outer two levels show that each pixel will be involved in computing (at most) $H_K \times W_K$ pixels in the activation map of one filtering, as is shown by the highlighted elements in Figure \pref{buffer}. And the innermost summation means that there are $M$ such filters. First, we can tackle the the innermost summation.
Considering the following matrix $\bb^* \in \mathbb{R} ^ {C \times H_K \times W_K} \times (H_Y \times W_Y)$
\begin{equation} \label{rbb}
\bb^* = \bw^T d\by
\end{equation}
where each element in $B^*$ corresponds to a combination of $\sum_{m=0}^{M-1}{ \bw_{m,c,p,q}\by_{m,i,j} }$. Basically, $B^*$ gives the all the information we need to compute $d\bx_{c,i,j}$ and we can rewrite \pref{dx} to
\begin{equation} \label{dx2}
\frac{\partial E}{\partial \bx_{c,i,j}} = \sum_{(a,b) \in A}{\bb^*_{a,b}}
\end{equation}

The problem is how to pick and choose the set $A$ so that the elements we choose satisfies \pref{dx}. Notice the structural similarity between $\bb^*$ and $\mathbf{B}$ in Figure \ref{buffer}. Both of them are of the same size and the elements are arranged in the same way. Since $\mathbf{B} = Im2Col(X)$, we can actually obtain $d\bx$ from $\mathbf{B}^*$ through a dual process of $Im2Col()$, which we call $Col2Im()$. What $Col2Im()$ does is the reverse process of $Im2Col()$, taking each column of the input matrix and patch it back to the output matrix of the same size of the input. Since patches overlap with each other, the values are added up, which corresponds to the summation in \pref{dx2}. The following Figure shows an example of how we reconstruct an element in $d\mathbf{X}$ from $\mathbf{B}^*$, where the input and kernl size are the same as the example in Figure \pref{buffer}. The colored boxes show how each column in $\mathbf{B}^*$ is stretched into a tile of pixels in $d\bx$. Note how the process is exactly the reverse of that in Figure \ref{buffer}.
\begin{figure}[h]
\begin{subfigure}[c]{0.5\textwidth}
\centering
\begin{tikzpicture}[baseline=-\the\dimexpr\fontdimen22\textfont2\relax ]
\matrix(m)[matrix of math nodes,left delimiter=(,right delimiter=)]
{
\color{Red}{\bw_{0,0,0}d\by_{0,0}} & \color{Cyan}{\bw_{0,0,0} d\by_{0,1}} & \bw_{0,0,0}d\by_{1,0} & \color{Green}{\bw_{0,0,0}d\by_{1,1}}\\
\color{Cyan}{\bw_{0,0,1}d\by_{0,0}} & \bw_{0,0,1}d\by_{0,1} & \color{Green}{\bw_{0,0,1}d\by_{1,0}} & \bw_{0,0,1}d\by_{1,1}\\
\bw_{0,1,0}d\by_{0,0} & \color{Green}{\bw_{0,1,0}d\by_{0,1}} & \bw_{0,1,0}d\by_{1,0} & \bw_{0,1,0}d\by_{1,1}\\
\color{Green}{\bw_{0,1,1}d\by_{0,0}} & \bw_{0,1,1}d\by_{0,1} & \bw_{0,1,1}d\by_{1,0} & \bw_{0,1,1}d\by_{1,1}\\
\bw_{1,0,0}d\by_{0,0} & \bw_{1,0,0}d\by_{0,1} & \bw_{1,0,0}d\by_{1,0}& \bw_{1,0,0}d\by_{1,1}\\
\bw_{1,0,1}d\by_{0,0} & \bw_{1,0,1}d\by_{0,1} & \bw_{1,0,1}d\by_{1,0}& \bw_{1,0,1}d\by_{1,1}\\
\bw_{1,1,0}d\by_{0,0} & \bw_{1,1,0}d\by_{0,1} & \bw_{1,1,0}d\by_{1,0}& \bw_{1,1,0}d\by_{1,1}\\
\bw_{1,1,1}d\by_{0,0} & \bw_{1,1,1}d\by_{0,1} & \bw_{1,1,1}d\by_{1,0}& \bw_{1,1,1}d\by_{1,1}\\
};
\begin{pgfonlayer}{myback}
\highlight[Red]{m-1-1}{m-4-1}
\highlight[Blue]{m-1-2}{m-4-2}
\highlight[ForestGreen]{m-1-3}{m-4-3}
\highlight[Orange]{m-1-4}{m-4-4}
\end{pgfonlayer}
\end{tikzpicture}
\caption{$\bb^* = \bw^T d\by$ of $8 \times 4$ }
\end{subfigure} \\
\vspace{5mm}
\begin{subfigure}[c]{0.75\textwidth}
\centering
\begin{tikzpicture}[baseline=-\the\dimexpr\fontdimen22\textfont2\relax ]
\matrix(m)[matrix of math nodes,execute at begin cell = {\pbox[t]{\linewidth}}, left delimiter=(,right delimiter=), nodes={align=left, anchor=center, text width=100pt, minimum height=6ex, inner sep=4pt}]
{
\color{Red}{\bw_{0,0,0}d\by_{0,0}} & \color{Cyan}{\bw_{0,0,1}d\by_{0,0} + \bw_{0,0,0} d\by_{0,1}}  &  \bw_{0,0,1}d\by_{0,1} \\
\bw_{0,1,0}d\by_{0,0} +  \bw_{0,0,0}d\by_{1,0}  & { \color{Green}{$\bw_{0,1,1}d\by_{0,0} + \bw_{0,1,0}d\by_{0,1}$} \\ $+ \bw_{0,0,1}d\by_{1,0} + \bw_{0,0,0}d\by_{1,1}$} &  \bw_{0,1,1}d\by_{0,1} +  \bw_{0,0,1}d\by_{1,1} \\
 \bw_{0,1,0}d\by_{1,0} & \bw_{0,1,1}d\by_{1,0} +  \bw_{0,1,0}d\by_{1,1}  &   \bw_{0,1,1}d\by_{1,1} \\
};
\begin{pgfonlayer}{myback}
\highlight[Red]{m-1-1}{m-2-2}
\highlight[Blue]{m-1-2}{m-2-3}
\highlight[ForestGreen]{m-2-1}{m-3-2}
\highlight[Orange]{m-2-2}{m-3-3}
\end{pgfonlayer}
\end{tikzpicture}
\caption{First channel of $d\bx = Col2Im(\bb^*)$ of $2 \times 3 \times 3$}
\end{subfigure}
\caption{An example of $Col2Im()$ \label{dual}}
\end{figure}

\section{Different Flavors}
Normally, convulational layer will be configured to be more than just zero padding and stride $1$. Hence, we discuss how to take those parameters into consideration.

\subsection{Padding, Stride and Dilation}
The beauty of $Im2Col()$ is that it abstracts away how we grab the pixels from the input to form the receptive field for convolution. Padding, stride and dilation all just change the way we grab the pixels and is taken care of by the $Im2Col()$ function. Padding tells $Im2Col()$ if the pixels are padded one, just put $0$ into $\bb$. Stride tells $Im2Col()$ how to move the convolution window around. And dilation tells $Im2Col()$ to strech the receiptive field and collect pixels that corresponds to non-zero weights. Since $Col2Im()$ is just a reverse process of $Im2Col()$, it will also take padding, stride and dilation into consideration. The core of the computation, i.e. the matrix multiplications, does not change.

Another important influence of padding ($P$), stride ($S$) and dilation ($D$) is that they determine the size of output image, i.e. $H_Y$ and $W_Y$, which has been missing in the discussion. With some geometry intuition, we can see that
\begin{eqnarray}
H_Y &=& (H_X + P_T + P_B - (D_H * (H_K - 1) + 1)) / S_H + 1 \\
W_Y &=& (W_X + P_L + P_R - (D_W * (W_K - 1) + 1)) / S_W + 1
\end{eqnarray}
where $P_{\{T,B,L,R\}}$ are paddings from top, bottom, left and right, and $D_{\{H,W\}}$ and $S_{\{H,W\}}$ are dilation and stride along height and width direction.

\subsection{Group Filters}\footnote{Group filters currently don't work with dilated convolution in $CAFFE2$}
\cite{alexnet} introduces group filters to reduce the number of weight so that the computation can fit into the configeration of GPUs. In addition, it seems that group filters can actually improve the accuracy, and the resulting trained filters look more intuitive when plotted as 2D images. For group filters, we partition a set of $M$ filters groups of $G$, and there will be $M/G$ groups. Each filter in a group, intead of working on all the $C$ channels, works on $C/G$ channels. For all the filters that work on group $g \in \{0,\ldots,M/G-1\}$, we collect them together to form a matrix of size $(M/G) \times ((C/G) \times H_K \times W_K)$.
\begin{equation*}
\bw_g = [\mathbf{w}_{g \times M/G}^l, \mathbf{w}_{g\times M/G+1}^l, \ldots, \mathbf{w}_{(g+1) \times M/G-1}^{l}]^T
\end{equation*}
where $\mathbf{w}_g^l$ means a filter $g$-th group and works on the input channel $[l\times C/G, (l+1)\times C/G-1]$.

In addition, we need to tell the $Im2Col()$ function to grab pixels from $C/G$ instead of $G$ channels to form the buffer matrix $\bb_g$ and its resulting size will be $(C/G \times H_K \times W_K) \times (H_Y \times W_Y)$. Similar to~\pref{bm}, output regarding to this group of filters can be computed as
\begin{equation} \label{bmg}
\by_g = \bw_g \bb_g
\end{equation}
Note that $\by_g \in \mathbb{R} ^ {(M/G) \times H_Y \times W_Y}$. Repeat this for $g \in \{0,\ldots,M/G-1\}$ and combine the results together and you will get $\mathbf{Y}$.

\section{Locally-Connected Layer}
In conventional convolutional layers, the weights of a filters are shared among computation of all the pixels of the output map. In other words, we use this same filter to scan the whole image to extract feature. It makes sense in the case of general images because the objects in interest can appear anywhere in the image. However, for some specific applications such as face recognition, this may not be a good idea, because the input of the face recognition is the frontal face photo where specific features such as eyes and mouth exist in specific region of the image. In this case, we do not want to share the weights of the filters at each location of convolution. We call convolutional layer without weight sharing locally-connected layer.

\subsection{Forward Propagation}
With out weight sharing, \pref{forward} becomes
\begin{equation}
\by_{m, i,j}=\sum_{c=0}^{C-1}{\sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\bw_{m,c,p,q,i,j}\bx_{c, i+p,j+q}}}} + \mathbf{b}_{m,i,j}
\end{equation}
Notice the additional indices $i,j$ in the weights and biases. Since we have different set of weights for each of the $H_Y \times H_W$ convolutional locations. The input weight tensor is now of size $M  \times C \times H_K \times W_K \times H_Y \times W_Y$. Similarly, bias becomes a tenor of size $M \times H_Y \times W_Y$. However, the output $\by$ is still of the same dimension ($N \times M \times H_Y \times W_Y$). Again, we will assume $N=1$ and omit the first dimension in the following discussion.

First, we can still use $Im2Col()$ to take care of the padding, stride and dilation and prepare the buffer matrix $\bb  \in \mathbb{R}^{(C \times H_K \times W_K) \times (H_Y \times W_Y)}$. However, we cannot do direct matrix multiplication with parts of $\bw$ and $\mathbf{B}$ because it will create terms $\bw_{m,c,p,q,i,j}\bx_{c, i+p,j+q}$ where $i \neq j$. Instead, if we arrange $\bw$ into $\bw_m \in \mathbb{R} ^{(C \times H_K \times W_K) \times (H_Y \times W_Y)}$ for $m=0,\ldots,M-1$, we can use Hadamard product to compute the first term of the output, ignoring the bias for now:
\begin{equation} \label{ym2}
\by_m = \mathbf{u}^T (\bw_m \odot \bb)
\end{equation}
where $\odot$ denotes the Hadamard product (element-wise product) and $\mathbf{u}$ is the same all one vector of size $C \times H_K \times W_K$. The result $\by_m$ is a $H_Y \times W_Y$ vector. Repeating this for all $m$ and assemble the result, and we will get $\by$. Some remarks:
\begin{enumerate}
\item When repeating~\pref{ym2} $M$ times, in terms of complexity, it is the same as matrix multiplication in~\pref{bm}. But we are making $2M$ calls ($M$ for Hadamont product and $M$ for AXPY) instead of $1$. This may introduce inefficiency compared to the weigth sharing convolutional layer.
\item cuBLAS and Intel MKL provide support for Hadamond product (DHAD, vdMul) but OpenBLAS doesn't yet~\cite{openblas}. This is not a hard blocker though because it is fairly easy to add this considering OpenBLAS already has elementwise addition.
\item The AXPY operation with $\mathbf{u}$ in~\pref{ym2} is an overkill. Effectively, we just need a reduction operation. BLAS provides ASUM operation for that. If there is support for that in CUDA, we can just use that.
\item We can actually convert the Hadamard product to matrix multiplication by creating a huge block-diagnal matrix. BLAS has built-in support for diagnal matrix but I am not sure whether this will help us in terms of performance or not.
\end{enumerate}

Next, we add the bias terms. Since $Y$ and $\mathbf{b}$ are of the size shape. Adding the bias is as simple as a matrix addition
\begin{equation}
\by' = \by + \mathbf{b}
\end{equation}

\subsection{Backward Propagation}
The input of gradient propagation is the same as normal convulational layer where we have $\bx$, $\bw$ and $d\by$, except for that $\bw$ is of size $M  \times C \times H_K \times W_K\times H_Y \times W_Y$ and $\mathbf{b}$ is of size $M \times H_Y \times W_Y$. For output, $d\bw$ will be the same size as $\bw$ and bias gradient $d\mathbf{b}$ will be the same size of $\mathbf{b}$. $d\bx$ will keep the same size as input $N \times C \times H_X \times H_Y$.

First, we compute the gradient of the weight $\bw_{m,c,p,q,i,j}$. Note that for a given filter $m$, each set of weight for each convolutional operation only contributes to one pixel in the output activation map, instead of $H_Y \times W_Y$ of them. Therefore, we have
\begin{equation}
\frac{\partial E}{\partial \bw_{m,c,p,q,i,j}} = \frac{\partial E}{\partial \by_{m,i,j}} \frac{\partial \by_{m,i,j}}{\partial  \bw_{m,c,p,q,i,j}} =  \frac{\partial E}{\partial \by_{m,i,j}} \bx_{c,i+p,j+q}
\end{equation}
Notice that compared to~\pref{dw}, it is a single multiplication instead of a convulation. Again, we will iterate through filter $m=0,\ldots,M-1$ and suppose we already have $\bb=Im2Col(\bx)$. Considering $d\by$ a $M \times (H_Y \times W_Y)$ matrix, and $\mathbf{B}$ and matrix of size $(C \times H_H \times W_K) \times (H_Y \times W_Y)$,
\begin{equation*}
d\bw_{m,c,p,q,:,:} = \by_{m,:} \odot \bb_{k,:}
\end{equation*}
computes a vector of $H_Y \times W_Y$ weights, where we first iterate on each row of $\by$ for $m=0,\ldots,M-1$ and then on each row of $\bb$ for $k=0,\ldots,C\times H_K \times W_K$. There will be $M \times C \times H_K \times W_K$ Hadamard product of size $H_Y \times W_Y$.

Similarly, the bias gradients can be propagated as
\begin{equation}
\frac{\partial E}{\partial \mathbf{b}_{m,i,j}} = \frac{\partial E} {\partial \by_{m,i,j}} \frac{\partial \by_{m,i,j}} {\partial \mathbf{b}_{m,i,j}} = \frac{\partial E} {\partial \by_{m,i,j}}
\end{equation}
So literally, $d\mathbf{b}=d\by$.

Finally, we compute the input gradient $d\bx$. A pixel in input $\bx_{c,i,j}$ still contributes to the computation of output activation map $\by$ in locally connected layer the same way as that in convolutional layer does. However, the weigths are a bit different. The formula to compute $d\bx_{c,i,j}$ is as follows
\begin{equation}
\frac{\partial E}{\partial \bx_{c,i,j}} = \sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\sum_{m=0}^{M-1}{\frac{\partial E}{\partial \by_{m,i-p,j-q}}\frac{\partial \mathbf{Y}_{m,i-p,j-q}}{\partial \mathbf{X}_{c,i,j}}}}} = \sum_{p=0}^{H_K-1}{\sum_{q=0}^{W_K-1}{\sum_{m=0}^{M-1}{\frac{\partial E}{\partial \by_{m,i-p,j-q}} \bw_{m,c,p,q,i,j}}}}
\end{equation}
The idea is the same as computing $d\bx$ in convulational layer. We want to compute a buffer matrix $\bb^*$ similar to~\pref{rbb} first and get $d\bx = Col2Im(\mathbf{B}^*)$, where each element in $\mathbf{B}^*$ corresponds to $\sum_{m=0}^{M-1}{\bw_{m,c,p,q,i,j}\by_{m,i,j}}$. Observe each element in $\bb^*$ in Figure~\ref{dual} presents multiplication between $\bw$ and $d\by$ along kernel pixel index as the row index increases and along output pixel index as the column index increases. The difference is that for each column, we have different weights. In order to do that, we first initialze $\mathbf{B}^*$ to be all zero.
Next, we iterate on $m=0,\ldots,M-1$ and arrange $\bw$ into $M$ chunks of $\bw_m$ (same as in~\pref{ym2}). Then, we iterate on the rows of $\bw_m$:
\begin{equation*}
\bb^*_{k,:} = \bb^*_{k,:} + \bw_{m_{k,:}} \odot d\by_{m,:}
\end{equation*}
where $k=0,\ldots, C \times H_K \times W_K$. Repeating the process by $M$ times and we will obtain the final result of $\bb^*$. With that, we can obtain $d\bx$ by applying $Col2Im(\bb^*)$.

\subsection{Efficient implementation with Batched GEMM}
From last section, we see that computing the forward and backward propagation usually involves $M$ alls to Hadamard product, which may require multiple kernels calls when implemented on GPU, which is suboptimal. Both Intel MKL and cuBLAS supports an operation called $BatchedGEMM$. $BatchedGEMM$ in a simpled form takes two tensor input $A \in \mathbb{R} ^{N \times P \times Q}$ and $B \in \mathbb{R}^{N \times Q \times R}$ and produce an output tensor $C \in \mathbb{R}^{N \times P \times R}$, where
\begin{equation*}
C_{n, :, :} = A_{n, :, :} B_{n, :, :}
\end{equation*}
It is one single kernel call to the GPU.

To take advantage of that when computing the forward propagation, we now reshape $\bw$ to be of size $H_Y \times W_Y \times M \times C \times H_K \times W_K$. We partition it into $H_Y \times W_Y$ matrices $\bw_k$ of size $M \times (C \times H_K \times W_K)$ and iterate on $k$. Each iteration yields $M$ results of $\by$ on the same convolution location:
\begin{equation*}
\by'_{i,j,:} = \bw_k \bb_{:,k}
\end{equation*}
Basically, we can implement this with one call to $BatchedGEMM(\bw, \bb^T)$, where $\bw$ is considered a 3D tensor of size $(H_Y \times W_Y) \times M \times (C \times H_K \times W_K)$ and $\bb^T$ is considered of size $(H_Y \times W_Y) \times (C \times H_K \times W_K) \times 1$. Notice that $\by'$ is of size $H_Y \times W_Y \times M$, we might need to reshape it into $\by$.

For backward propagation of weight gradients, similarly, we will iterate over output pixels $H_Y \times W_Y$ and use that as our batch size to call $BatchedGEMM$. For each $k=0,\ldots,H_Y \times W_Y-1$, we take a column of $d\by$ and a column of $\bb$ and computes their outer product, which yields a matrix of $M \times (C \times H_K \times W_K)$ weight gradients $d\bw_{i,j,:,:,:,:}$. Implementation is basically one call to $BatchedGEMM(d\by^T, \bb)$, where $d\by^T$ is considered of size $(H_Y \times W_Y) \times M \times 1$.

Finally, for backward propagation of input gradients, we focus on how to compute $\bb^*$ with $BatchedGEMM$. Again, we use slices of input weight $\bw_k$, for $k=0,\ldots,H_Y \times W_Y -1$. In each iteration
\begin{equation*}
\bb^*_{:,k} = \bw_k^T d\by_{:,k}
\end{equation*}
generates one column of $\bb^*$. The whole operation can be wrapped with one call to $BatchedGEMM(\bw_k^T, d\by^T)$. Similarly, resulting $\bb^*$ needs reshaping before applying $Col2Im$ to it.

%\section{Tiled Convulational Layer}

\bibliographystyle{IEEEtran}
\bibliography{cnnref}

\end{document}