 \section{Integration Variable Remapping}

\subsection{Introduction}
This section will layout the jacobian needed for the 10 $\rar$ 10 remapping of
variables for the parton level cross section. The base variables used are shown below.

\begin{myitemize}
\item $p_{3}$: Absolute momentum of the lepton
\item $p_{5}$: Absolute momentum of the first quark
\item $p_{6}$: Absolute momentum of the second quark
\item $p_{tot}^{z}$: Total $p_{z}$ of the system
\item $cos(\theta_{3})$: Cosine($\theta$) of the lepton
\item $\phi_{3}$: $\phi$ of the lepton
\item $cos(\theta_{5})$: Cosine($\theta$) of the first quark
\item $\phi_{5}$: $\phi$ of the first quark
\item $cos(\theta_{6})$: Cosine($\theta$) of the second quark
\item $\phi_{6}$: $\phi$ of the second quark
\end{myitemize}

Other variables that are useful for the integration are
\begin{myitemize}
\item $m_{34}$: Mass of the lepton and neutrino (W mass)
\item $m_{345}$: Mass of the lepton, neutrino, and first quark (top mass)
\item $m_{56}$: Mass of the first and second quark ($b\bar{b}$ mass)
\end{myitemize}

Since some of these variables are sharp peaks (W and top masses), it is much
better to sample from the expected distribution rather than make requirements of
the invariant masses. The W and top masses are expected to follow a Breit-Wigner
distribution shown below.

\begin{equation}
\sigma(M_{34}) = \frac{1}{\pi} \left[ \frac{\gamma}{(M_{34} - M_{W})^{2} +
\gamma^{2}} \right]
\end{equation}

where $M_{34}$ is the mass of the lepton and neutrino. Similarly, the top mass
has the following expected distribution.

\begin{equation}
\sigma(M_{345}) = \frac{1}{\pi} \left[ \frac{\gamma}{(M_{345} - M_{top})^{2} +
\Gamma^{2}} \right]
\end{equation}

where  $M_{345}$ is the mass of the lepton, neutrino, and first quark.

\subsection{Sampling from a Breit-Wigner mass distribution}

Sampling from a Breit-Wigner distribution is done by selecting a random point
between 0 and 1 from the cumulative distribution function of the BW
function. The cumulative distribution function, of cdf, for the Breit-Wigner
distribution is shown below.

\begin{equation}
\label{sample}
\int \sigma(m;m_{0};\Gamma) = F(m;m_{0},\Gamma) = \frac{1}{\pi} \tan^{-1} \left[
\frac{m-m_{0}}{\Gamma} \right] + \frac{1}{2}
\end{equation}

The value of F is taken as a random number between 0 and 1. After selecting a
value of F, the next step is to solve for m. As a function of F, defined as u for
the following, the mass is

\begin{equation}
m = m_{0} + \Gamma \tan \left[ \pi(u - \frac{1}{2}) \right]
\end{equation}

\subsection{Sampling from a Breit-Wigner $S_{cm}$ distribution}

In the previous example, a distribution was sampled using a random number
uniformly distributed from 0 to 1. In, this example, a new random number is used
that is uniformaly samples from 0 to 1, but the maximum and minimum values of
the variable are taken into account in the Jacobian.

The distribution of the variable $S_{cm}$ is the following

\begin{equation}
\label{defines}
s = m_{0}^{2} + m_{0}\Gamma \tan \left[ m_{0} \Gamma r \right]
\end{equation}

where r is defined in terms of the random variable, u, that is uniformaly
distributed between 0 and 1.

\begin{equation}
\label{rtou}
r = (r_{max} - r_{min}) \times u + r_{min}
\end{equation}

where $r_{max}$ and $r_{min}$ are defined in terms of the variable $s_{cm}$.

\begin{eqnarray}
\label{definer}
r = \frac{1}{m_{0}\Gamma} \tan \left[ \frac{s - m_{0}^{2}}{m_{0}\Gamma} \right] \\
r_{min} = \frac{1}{m_{0}\Gamma} \tan \left[ \frac{s_{min} -
m_{0}^{2}}{m_{0}\Gamma} \right] \\
r_{max} = \frac{1}{m_{0}\Gamma} \tan \left[ \frac{s_{max} - m_{0}^{2}}{m_{0}\Gamma} \right]
\end{eqnarray}

\subsection{Jacobian for random sampling of a Breit-Wigner distribution around
the W mass squared, $s_{34}$}

The first case to consider is the Breit-Wigner sampling around the W mass
$S_{34}$ distribution and replace the integration variable, $p_{3}$ or the
lepton momentum.

\begin{equation}
\label{jacobian1}
|J(p_{3}, u)| = \left| \pderiv{p_{3}}{u} \right|
\end{equation}

Because u is redined in terms of the variable r, we can rewrite ~\ref{jacobian1}
in terms of r instead of u.

\begin{equation}
\label{jacobian2}
|J(p_{3}, u)| = \left| \pderiv{p_{3}}{u} \right| = \left| \pderiv{p_{3}}{r}
\times \pderiv{r}{u} \right|
\end{equation}

And since the variable r is sampling the $S_{34}$ distribution it makes sense to
define the Jacobian in terms of this variable instead of $p_{3}$.

\begin{equation}
\label{jacobian3}
|J(p_{3}, u)| = \left| \pderiv{p_{3}}{r} \times \pderiv{r}{u} \right| = \left|
\pderiv{p_{3}}{s_{34}} \times \pderiv{s_{34}}{r} \times \pderiv{r}{u} \right| =
\left | \frac{\pderiv{s_{34}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{34}}{p_{3}}} \right |
\end{equation}

Equation ~\ref{jacobian3} has three components: $\pderiv{s_{34}}{r}$,
$\pderiv{r}{u}$, and $\pderiv{s_{34}}{p_{3}}$. From equation ~\ref{rtou}, the
partial derivative of r with respect to u is

\begin{equation}
\label{drdu}
\pderiv{r}{u} = r_{max} - r_{min} = \Delta r
\end{equation}

Next, the partial of $s_{34}$ with respect to r can be determined from equation
~\ref{defines}.

\begin{equation}
\label{ds34dr}
\pderiv{s_{34}}{r} = (m_{W} \Gamma_{W})^{2} \sec^{2} \left[m_{W} \Gamma_{W} r \right]
\end{equation}

Inserting the value of r(s) as defined in equation ~\ref{definer}, equation
~\ref{ds34dr} can be re-written as

\begin{equation}
\label{ds34dr_2}
\pderiv{s_{34}}{r} = (m_{W} \Gamma_{W})^{2} \sec^{2} \left[m_{W} \Gamma_{W} r \right] =
(m_{W} \Gamma_{W})^{2} \sec^{2} \left[\arctan \left[ \frac{s_{34} - m_{W}^{2}}{m_{W}
\Gamma_{W}}  \right] \right]
\end{equation}

Equation ~\ref{ds34dr_2} is solved by defining a right triangle where

\begin{eqnarray}
\nonumber
\tan(\theta) = \frac{s_{34}-m_{W}^{2}}{m_{W}\Gamma_{W}} \\
\cos(\theta) = \frac{1}{\sqrt{1+\left[ \frac{s_{34}-m_{W}^{2}}{m_{W}\Gamma_{W}}
\right]^{2}}}
\end{eqnarray}

Using these definitions, equation ~\ref{ds34dr_2} is finally defined as

\begin{equation}
\label{ds34dr_3}
\pderiv{s_{34}}{r} = (m_{W} \Gamma_{W})^{2} \sec^{2} \left[\arctan \left[ \frac{s_{34} - m_{W}^{2}}{m_{W}
\Gamma_{W}}  \right] \right] = (m_{W}\Gamma_{W})^{2} + (s_{34} - m_{W}^{2})^{2}
\end{equation}

Finally, we need the partial derivative of $s_{34}$ with repect to
$p_{3}$. First, we define $s_{34}$

\begin{equation}
\label{defines34}
s_{34} = m_{3}^{2} + m_{4}^{2} + 2E_{3}E_{4} - 2p_{3}^{x}p_{4}^{x} - 2p_{3}^{y}p_{4}^{y} - 2p_{3}^{z}p_{4}^{z}
\end{equation}

Since the neutino four-vector is defined in terms of all the other particles in
the event, we need to rewrite equation ~\ref{defines34} to expose all the
dependences on $p_{3}$. For the following, it is assumed that the lepton and
neutrino are massless meaning $E_{3} = p_{3}$.

\begin{eqnarray}
\label{defines34_2}
\nonumber
s_{34} = 2p_{3}\sqrt{(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x})^{2} + (-p_{3}^{y} -p_{5}^{y} -p_{6}^{y})^{2} + (p_{tot}^{z} -p_{3}^{z} -p_{5}^{z} -p_{6}^{z})^{2}} \\
- 2p_{3}^{x}(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x}) -
2p_{3}^{y}(-p_{3}^{y} -p_{5}^{y} -p_{6}^{y}) - 2p_{3}^{z}(p_{tot}^{z} -p_{3}^{z} -p_{5}^{z} -p_{6}^{z})
\end{eqnarray}

After combining like terms, we can evaluate the partial derivative of $s_{34}$
with respect to $p_{3}$ that yields the relatively
simple formula

\begin{equation}
\label{ds34dp3}
\pderiv{s_{34}}{p_{3}} = 2(p_{3} + p_{4})(1 - \hat{p_{3}} \cdot \hat{p_{4}})
\end{equation}

Finally, we can rewrite the Jacobian defined in ~\ref{jacobian3} as

\begin{equation}
\label{jacobian4}
|J(p_{3}, u)| = \left | \frac{\pderiv{s_{34}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{34}}{p_{3}}} \right | = \frac{\Delta R \times \left[
(m_{W}\Gamma_{W})^{2} + (s_{34} - m_{W}^{2})^{2} \right]}{2(p_{3} + p_{4})(1 - \hat{p_{3}} \cdot \hat{p_{4}})}
\end{equation}

In some cases, it is also common to replace the first quark momentum integration
with the Breit-Wigner sampling variable. In that case, we need to evaluate

\begin{equation}
\label{jacobian_5}
|J(p_{5}, u)| = \left | \frac{\pderiv{s_{34}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{34}}{p_{5}}} \right |
\end{equation}

For this substitution, we only need to evaluate the partial derivative of
$s_{34}$ with respect to $p_{5}$. Assuming a massless quark, the result is

\begin{equation}
\label{ds34dp5}
\pderiv{s_{34}}{p_{5}} = 2p_{3}(\hat{p_{3}} \cdot \hat{p_{5}} - \hat{p_{4}} \cdot \hat{p_{5}})
\end{equation}

Combining equation ~\ref{ds34dp5} with ~\ref{jacobian_5} yields

\begin{equation}
\label{jacobian_6}
|J(p_{5}, u)| = \left | \frac{\pderiv{s_{34}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{34}}{p_{5}}} \right | = \frac{\Delta R \times \left[
(m_{W}\Gamma_{W})^{2} + (s_{34} - m_{W}^{2})^{2} \right]}{2p_{3}(\hat{p_{3}} \cdot \hat{p_{5}} - \hat{p_{4}} \cdot \hat{p_{5}})}
\end{equation}

%
%-----------------------------------------------------------------------
%

\subsection{Jacobian for random sampling of a Breit-Wigner distribution around
the top mass squared, $s_{345}$}

The next case to consider is the Breit-Wigner sampling around the top mass squared
$S_{345}$ distribution and replace the integration variable, $p_{3}$ or the
lepton momentum. As before, we need need to calculate the following

\begin{equation}
\label{jacobian_1}
|J(p_{3}, u)| = \left| \pderiv{p_{3}}{r} \times \pderiv{r}{u} \right| = \left|
\pderiv{p_{3}}{s_{345}} \times \pderiv{s_{345}}{r} \times \pderiv{r}{u} \right| =
\left | \frac{\pderiv{s_{345}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{345}}{p_{3}}} \right |
\end{equation}

Equation ~\ref{jacobian_1} has three components: $\pderiv{s_{345}}{r}$,
$\pderiv{r}{u}$, and $\pderiv{s_{345}}{p_{3}}$. We know $\pderiv{r}{u}$ from
equation ~\ref{drdu} where $r_{min}$ and $r_{max}$ are defined by the $s_{345}$
system instead of the $s_{34}$ system. We also know $\pderiv{s_{345}}{r}$ from ~\ref{ds34dr_3}
where we replace $s_{34}$ with $s_{345}$.

\begin{equation}
\label{ds345dr}
\pderiv{s_{345}}{r} = (m_{t}\Gamma_{t})^{2} + (s_{345} - m_{t}^{2})^{2}
\end{equation}

We do need the partial derivative of $s_{345}$ with repect to
$p_{3}$. First, we define $s_{345}$

\begin{eqnarray}
\label{defines345}
\nonumber
s_{345} = m_{3}^{2} + m_{4}^{2} + m_{5}^{2} + 2E_{3}E_{4} + 2E_{3}E_{5} +
2E_{4}E_{5} - \\
2p_{3}^{x}p_{4}^{x} - 2p_{3}^{x}p_{5}^{x} - 2p_{4}^{x}p_{5}^{x} -
2p_{3}^{y}p_{4}^{y} - 2p_{3}^{y}p_{5}^{y} - 2p_{4}^{y}p_{5}^{y} -
2p_{3}^{z}p_{4}^{z} - 2p_{3}^{z}p_{5}^{z} - 2p_{4}^{z}p_{5}^{z}
\end{eqnarray}

As before, the neutino four-vector is defined in terms of all the other particles in
the event so we need to rewrite equation ~\ref{defines345} to expose all the
dependences on $p_{3}$. For the following, it is assumed that the lepton,
neutrino, and quark are massless meaning $E_{3} = p_{3}$.

\begin{eqnarray}
\label{defines345_2}
\nonumber
s_{345} = 2p_{3}\sqrt{(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x})^{2} + (-p_{3}^{y}
-p_{5}^{y} -p_{6}^{y})^{2} + (p_{tot}^{z} -p_{3}^{z} -p_{5}^{z} -p_{6}^{z})^{2}}
+ \\
\nonumber
2p_{3}p_{5} + 2\sqrt{(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x})^{2} + (-p_{3}^{y} -p_{5}^{y}
-p_{6}^{y})^{2} + (p_{tot}^{z} -p_{3}^{z} -p_{5}^{z} -p_{6}^{z})^{2}}p_{5} - \\
\nonumber
2p_{3}^{x}(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x}) - 2p_{3}^{x}p_{5}^{x} -
2(-p_{3}^{x} -p_{5}^{x} -p_{6}^{x})p_{5}^{x} - \\
\nonumber
2p_{3}^{y}(-p_{3}^{y} -p_{5}^{y} -p_{6}^{y}) - 2p_{3}^{y}p_{5}^{y} -
2(-p_{3}^{y} -p_{5}^{y} -p_{6}^{y})p_{5}^{y} - \\
2p_{3}^{z}(-p_{3}^{z} -p_{5}^{z} -p_{6}^{z}) - 2p_{3}^{z}p_{5}^{z} - 2(-p_{3}^{z} -p_{5}^{z} -p_{6}^{z})p_{5}^{z}
\end{eqnarray}

After combining like terms, we can evaluate the partial derivative of $s_{345}$
with respect to $p_{3}$ as

\begin{equation}
\label{ds345dp3}
\pderiv{s_{345}}{p_{3}} = 2(p_{3} + p_{4} + p_{5})(1 - \hat{p_{3}} \cdot \hat{p_{4}})
\end{equation}

Finally, we can rewrite the Jacobian defined in ~\ref{jacobian_1} as

\begin{equation}
\label{jacobian_2}
|J(p_{3}, u)| = \left | \frac{\pderiv{s_{345}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{345}}{p_{3}}} \right | = \frac{\Delta R \times \left[
(m_{t}\Gamma_{t})^{2} + (s_{345} - m_{t}^{2})^{2} \right]}{2(p_{3} + p_{4} + p_{5})(1 - \hat{p_{3}} \cdot \hat{p_{4}})}
\end{equation}

Instead of replacing the lepton momentum integration variable, it is also common
to replace the first quark momentum integration variable, $p_{5}$. In that case,
we need to evaluate the following Jacobian.

\begin{equation}
\label{jacobian_3}
|J(p_{5}, u)| = \left| \pderiv{p_{5}}{r} \times \pderiv{r}{u} \right| = \left|
\pderiv{p_{5}}{s_{345}} \times \pderiv{s_{345}}{r} \times \pderiv{r}{u} \right| =
\left | \frac{\pderiv{s_{345}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{345}}{p_{5}}} \right |
\end{equation}

The only difference is that partial derivative of $s_{345}$ with respect to
$p_{5}$ instead of $p_{3}$. However, since $s_{345}$ is invariant under a change
of $p_{3}$ and $p_{5}$ the partial derivatives must be equal. Thus,

\begin{equation}
\label{jacobian_4}
|J(p_{5}, u)| = \left | \frac{\pderiv{s_{345}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{345}}{p_{5}}} \right | = \frac{\Delta R \times \left[
(m_{t}\Gamma_{t})^{2} + (s_{345} - m_{t}^{2})^{2} \right]}{2(p_{3} + p_{4} + p_{5})(1 - \hat{p_{4}} \cdot \hat{p_{5}})}
\end{equation}

%
%-----------------------------------------------------------------------
%

\subsection{Jacobian for random sampling of two Breit-Wigner distributions around
the top mass squared, $s_{345}$ and W mass squared, $s_{34}$}

The next situation is to sample from a Breit-Wigner around the
top mass squared and the W mass squared, or $s_{345}$ and $s_{34}$. It is common
to replace the lepton momentum and first quark momentum integration variables
with the two new variables. Since we are
replacing two variables, we need to evaluate the following Jacobian

\begin{equation}
|J(p_{3}, p_{5} ; u_{1}, u_{2})| = \left| \begin{array}{cc}
\pderiv{p_{3}}{u_{1}}	& \pderiv{p_{3}}{u_{2}} \\
\pderiv{p_{5}}{u_{1}}	& \pderiv{p_{5}}{u_{2}} \\
\end{array} \right|
\end{equation}

where $u_{1}$ and $u_{2}$ are the sampling variables around the top mass squared
and W mass squared, respectively.

We have already computed the partial derivatives for each of these cases
in the previous two sections, thus the result is

\begin{eqnarray}
\nonumber
|J(p_{3}, p_{5} ; u_{1}, u_{2})| = \left| \begin{array}{cc}
\pderiv{p_{3}}{u_{1}}	& \pderiv{p_{3}}{u_{2}} \\
\pderiv{p_{5}}{u_{1}}	& \pderiv{p_{5}}{u_{2}} \\
\end{array} \right| =
\nonumber
\left| \begin{array}{cc}
\frac{\pderiv{s_{345}}{r} \times \pderiv{r}{u}}{\pderiv{s_{345}}{p_{3}}}	& \frac{\pderiv{s_{345}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{345}}{p_{5}}} \\
\frac{\pderiv{s_{34}}{r} \times \pderiv{r}{u}}{\pderiv{s_{34}}{p_{3}}}	& \frac{\pderiv{s_{34}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{34}}{p_{5}}}
\end{array} \right| = \\
\left| \begin{array}{cc}
\frac{\Delta R_{345} \times \left[
(m_{t}\Gamma_{t})^{2} + (s_{345} - m_{t}^{2})^{2} \right]}{2(p_{3} + p_{4} + p_{5})(1 - \hat{p_{3}} \cdot \hat{p_{4}})}	& \frac{\Delta R_{34} \times \left[
(m_{W}\Gamma_{W})^{2} + (s_{34} - m_{W}^{2})^{2} \right]}{2(p_{3} + p_{4})(1 -
\hat{p_{3}} \cdot \hat{p_{4}})} \\
\frac{\Delta R_{345} \times \left[
(m_{t}\Gamma_{t})^{2} + (s_{345} - m_{t}^{2})^{2} \right]}{2(p_{3} + p_{4} + p_{5})(1 - \hat{p_{4}} \cdot \hat{p_{5}})}	& \frac{\Delta R_{34} \times \left[
(m_{W}\Gamma_{W})^{2} + (s_{34} - m_{0}^{2})^{2} \right]}{2p_{3}(\hat{p_{3}} \cdot \hat{p_{5}} - \hat{p_{4}} \cdot \hat{p_{5}})}
\end{array}
\right|
\end{eqnarray}

\subsection{Sampling from a Polynomial $S_{cm}$ distribution}

*** This is where I am taking a function from Aurelio and I can't seem to derive
it on my own ***

The distribution of the variable $S_{cm}$ according to a polynomial power
distribution is

\begin{equation}
\label{defines_power}
s = m_{0}^{2} + \left[ (1-\alpha)r \right]^{\frac{-1}{\alpha - 1}}
\end{equation}

where r is defined in terms of the random variable, u, that is uniformaly
distributed between 0 and 1.

\begin{equation}
\label{rtou_power}
r = (r_{max} - r_{min}) \times u + r_{min}
\end{equation}

where $r_{max}$ and $r_{min}$ are defined in terms of the variable $s_{cm}$.

\begin{eqnarray}
\label{definer_power}
r = \frac{1}{1-\alpha} \times \left[ s - m_{0}^{2} \right]^{1-\alpha} \\
r_{min} = \frac{1}{1-\alpha} \times \left[ s_{min} - m_{0}^{2}
\right]^{1-\alpha} \\
r_{max} = \frac{1}{1-\alpha} \times \left[ s_{max} - m_{0}^{2}
\right]^{1-\alpha}
\end{eqnarray}

where alpha can not equal 1.

\subsection{Jacobian for random sampling of a polynomial distribution
starting at $m_{pole}$}

The first case to consider is sampling around a falling polynomial distribution
for the mass squared of two quarks in the event, $s_{56}$. We need to define the
Jacobian with respect $p_{5}$ or $p_{6}$. Since $s_{56}$ is invariant under an
interchange of particle 5 and 6, the Jacobian will be the same for each momentum
integration. The following assume $p_{5}$ will be replaced with the variable,
$u$, which is sampled from a polynomial distribution.

\begin{equation}
\label{jacobian_7}
|J(p_{5}, u)| = \left| \pderiv{p_{5}}{u} \right|
\end{equation}

Because u is redined in terms of the variable r, we can rewrite ~\ref{jacobian_7}
in terms of r instead of u.

\begin{equation}
\label{jacobian_8}
|J(p_{5}, u)| = \left| \pderiv{p_{5}}{u} \right| = \left| \pderiv{p_{5}}{r}
\times \pderiv{r}{u} \right|
\end{equation}

And since the variable r is sampling the $S_{56}$ distribution it makes sense to
define the Jacobian in terms of this variable instead of $p_{5}$.

\begin{equation}
\label{jacobian_9}
|J(p_{5}, u)| = \left| \pderiv{p_{5}}{r} \times \pderiv{r}{u} \right| = \left|
\pderiv{p_{5}}{s_{56}} \times \pderiv{s_{56}}{r} \times \pderiv{r}{u} \right| =
\left | \frac{\pderiv{s_{56}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{56}}{p_{5}}} \right |
\end{equation}

Equation ~\ref{jacobian_9} has three components: $\pderiv{s_{56}}{r}$,
$\pderiv{r}{u}$, and $\pderiv{s_{56}}{p_{5}}$. From equation ~\ref{rtou_power}, the
partial derivative of r with respect to u is

\begin{equation}
\label{drdu_power}
\pderiv{r}{u} = r_{max} - r_{min} = \Delta R_{56}
\end{equation}

Next, the partial of $s_{56}$ with respect to r can be determined from equation ~\ref{defines_power}.

\begin{equation}
\label{ds56dr_power}
\pderiv{s_{56}}{r} = \left[ r(1-\alpha) \right]^{\frac{\alpha}{1-\alpha}}
\end{equation}

Inserting the value of r(s) as defined in equation ~\ref{definer_power}, equation ~\ref{ds56dr_power} can be re-written as

\begin{equation}
\label{ds56dr_power2}
\pderiv{s_{56}}{r} = \left[ s_{56} - m_{0}^{2} \right]^{\alpha}
\end{equation}

Finally, we need the partial derivative of $s_{56}$ with repect to $p_{5}$. First, we define $s_{56}$

\begin{equation}
\label{defines56_power}
s_{56} = m_{5}^{2} + m_{6}^{2} + 2E_{5}E_{6} - 2p_{5}^{x}p_{6}^{x} - 2p_{5}^{y}p_{6}^{y} - 2p_{5}^{z}p_{6}^{z}
\end{equation}

we can evaluate the partial derivative of $s_{56}$ with respect to $p_{5}$.

\begin{equation}
\label{ds56dp5_power}
\pderiv{s_{56}}{p_{5}} = 2p_{6}(1 - \hat{p_{5}} \cdot \hat{p_{6}})
\end{equation}

Finally, we can rewrite the Jacobian defined in ~\ref{jacobian_9} as

\begin{equation}
\label{jacobian_10}
|J(p_{5}, u)| = \left | \frac{\pderiv{s_{56}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{56}}{p_{5}}} \right | = \frac{\Delta R_{56} \times \left[ s_{56}-m_{0}^{2} \right]^{\alpha}}{2p_{6}(1 - \hat{p_{5}} \cdot \hat{p_{6}})}
\end{equation}

and

\begin{equation}
\label{jacobian_11}
|J(p_{6}, u)| = \left | \frac{\pderiv{s_{56}}{r} \times
\pderiv{r}{u}}{\pderiv{s_{56}}{p_{6}}} \right | = \frac{\Delta R_{56} \times \left[ s_{56}-m_{0}^{2} \right]^{\alpha}}{2p_{5}(1 - \hat{p_{5}} \cdot \hat{p_{6}})}
\end{equation}


The previous chapter introduced the concepts of \textit{open-loop} and \textit{closed-loop} control laws, and then dove into techniques for designing open-loop control laws for robots based on optimal control and differential flatness. These techniques are useful for determining control inputs that accomplish different objectives, such as ``move from point A to point B in a minimal amount of time while satisfying some constraints''. Additionally, computing open-loop control laws is often computationally less challenging that computing closed-loop control laws.
However in practice open-loop control is not very robust since observations are not leveraged to update the control input. One solution to this robustness problem is to convert the open-loop control law into a closed-loop control law, typically referred to as \textit{trajectory tracking controllers}. Another solution is to not use any open-loop techniques but rather to directly synthesize a closed-loop control law, for example by performing a \textit{Lyapunov stability analysis}.
This chapter will introduce techniques for synthesizing closed-loop controllers in both of these ways.

\notessection{Closed-loop Motion Planning \& Control}
Recall from the previous chapter that open-loop control laws are defined as a function of time for a given initial condition. In contrast, closed-loop control laws are a function of the \textit{current} state, and therefore are reactive.

\begin{definition}[Closed-loop Control]
If the control law is a function of the state and time, i.e.,
\begin{equation}
\bm{u}(t) = \pi(\x(t), t )
\end{equation}
then the control is said to be in closed-loop form.
\end{definition}

Closed-loop controllers (also sometimes referred to as \textit{feedback controllers} or \textit{policies}), are much more robust than open-loop controllers. For example, suppose a controller needs to be designed to make a wheeled robot move from point to point. If the model used for open-loop controller design wasn't perfect, if the initial state was not perfectly known, or if external disturbances affected the system (e.g. wheel slipping), then the robot would not exactly reach its desired destination. Alternatively, a closed-loop control law can continuously correct for these errors since it is always taking in new information.

\subsection{Trajectory Tracking}
One common approach to closed-loop control is to simply extend the open-loop control techniques from the previous chapter to include a feedback component. Such an approach consists of two steps:
\begin{enumerate}
    \item Use open-loop control techniques to design a desired trajectory $\x_d(t)$ and corresponding control $\bm{u}_d(t)$.
    \item Design a closed-loop control law that is designed to make sure the system stays close to the desired trajectory.
\end{enumerate}
These controllers are referred to as \textit{trajectory tracking} controllers, and their control law is defined as
\begin{equation}
\bm{u}(t) =  \bm{u}_d(t)+\pi(\x(t)-\x_d(t),t).
\end{equation}
This type of control law is also said to be a ``feedforward plus feedback'' controller. This is because the term $\bm{u}_d(t)$ is an open-loop ``feedforward'' term that attempts to generally make the system follow the desired trajectory, while the term $\pi(\x(t)-\x_d(t),t)$ is a ``feedback'' term that attempts to correct for any errors.

The previous chapter discussed techniques for solving open-loop control problems to define the desired trajectory, and additionally there are several approaches for designing the feedback component $\pi(\x(t)-\x_d(t),t)$:
\begin{itemize}
  \item \textit{Geometric} approaches generally leverage some sort of insight about the system and are therefore hard to discuss in general settings. They are also typically difficult to derive theoretical guarantees for.
  \item \textit{Linearization} based approaches typically linearize nonlinear dynamics models about points along the desired trajectory. These linearized models are then used to design linear controllers (e.g. linear quadratic regulators). For some nonlinear systems, instead of linearizing about specific points it possible to \textit{feedback linearize} the system. This essentially means that the non-linearities can be exactly ``canceled'' out such that the system can be considered linear. Linear control theory can then be applied to design a feedback control scheme.

  \item \textit{Non-linear control} techniques also exist which do not rely on linearization. These approaches are also heavily system dependent, but one common tool for non-linear control is based on \textit{Lyapunov} theory.

  \item \textit{Optimization-based} feedback control laws can also be designed. These approaches often leverage optimal control theory, some of which was presented in the previous chapter. One common optimization-based approach for closed-loop control is known as \textit{model predictive control} (MPC).
\end{itemize}

\subsubsection{Trajectory Tracking for Differentially Flat Systems}
For differentially flat systems linearization based approaches to designing trajectory tracking controllers are particularly useful\cite{Levine2009}. In fact, every flat system can be linearized via dynamic feedback and a coordinate change to yield a dynamical system of the form
\begin{equation} \label{eq:diff_flat_linearized}
    \z^{(q+1)} = \bm{w},
\end{equation}
where $\z^{(q+1)}$ is the $q+1$-th order derivative of the flat outputs $z$ and $q$ is the degree of the flat output space (i.e. the highest order of derivatives of the flat output that are needed to describe system dynamics), and $\bm{w}$ is a modified ``virtual'' input term.

The set of ODEs \eqref{eq:diff_flat_linearized} are \textit{linear}, which means that techniques from linear control theory can be applied to design a control law for $\bm{w}$. In particular, for trajectory tracking problems suppose a reference flat output trajectory $\z_d(t)$ has been defined which corresponds to the virtual input $\bm{w}_d(t)$. Let the error between the actual flat output and desired flat output be defined as $\bm{e}(t) = \z(t) - \z_d(t)$ and consider a closed-loop control law of the form
\begin{equation}
w_i(t) = w_{i,d}(t) - \sum_{j=0}^q k_{i,j} e_i^{(j)}(t),
\end{equation}
where $(\cdot)_i$ denotes the $i$-th component of the vector, $e^{(j)} = \z^{(j)} - \z_d^{(j)}$ is the $j$-th order derivative of the error, and $k_{i,j}$ are called controller \textit{gains}.
The application of this control law to the system \eqref{eq:diff_flat_linearized} will result in \textit{closed-loop dynamics} of the form
\begin{equation*}
\begin{split}
\z^{(q+1)} &= \bm{w}_d - \sum_{j=0}^q K_j \bm{e}^{(j)}, \\
\end{split}
\end{equation*}
where $K_j$ is a diagonal matrix with $i$-th diagonal element $k_{i,j}$. Since $\z_d^{(q+1)} = \bm{w}_d(t)$ this can be simplified to give the closed-loop error dynamics:
\begin{equation}
\begin{split}
\bm{e}^{(q+1)} + \sum_{j=0}^q K_j \bm{e}^{(j)} = 0. \\
\end{split}
\end{equation}
This set of linear ODEs describes the dynamics of the error, and many classical techniques from linear control theory can be used to choose the gains $k_{i,j}$ that will guarantee this system is \textit{stable}. Having stable error dynamics means that the error will decay to zero, which in this case means the system will track the desired trajectory.

\begin{example}[Extended Unicycle Trajectory Tracking] \label{ex:trajtrack}
\theoremstyle{definition}
Consider the dynamically extended unicycle model
\begin{equation}
\begin{split}
\dot{x}(t) &= v \cos(\theta(t)), \\
\dot{y}(t) &= v \sin(\theta(t)), \\
\dot{v}(t) &= a(t), \\
\dot{\theta}(t) &= \omega(t),
\end{split}
\label{robot_eq_dyn}
\end{equation}
where the two inputs are the acceleration $a(t)$ and the rotation rate $\omega(t)$. This system is differentially flat with flat outputs $x$ and $y$ and order $q=1$. It can therefore be expressed as:
\begin{equation*}
\ddot{\z} = \begin{bmatrix} \ddot{x} \\ \ddot{y} \end{bmatrix} = \underbrace{\begin{bmatrix} \cos(\theta) & -V\sin(\theta) \\ \sin(\theta) & V\cos(\theta) \end{bmatrix}}_{:=J} \begin{bmatrix} a \\ \omega \end{bmatrix} := \begin{bmatrix} w_1 \\ w_2 \end{bmatrix},
\end{equation*}
and a trajectory tracking controller can be defined as
\begin{equation*}
\begin{split}
    w_1 &= \ddot{x}_d - k_{px}(x - x_d)- k_{dx}(\dot{x} - \dot{x}_d),\\
    w_2 &= \ddot{y}_d - k_{py}(y-y_d)- k_{dy}(\dot{y}-\dot{y}_d),\\
\end{split}
\end{equation*}
where $(\cdot)_d$ represents a term associated with the desired trajectory. The control inputs $a(t)$ and $\omega(t)$ can then be computed by solving the linear system
\begin{equation*}
J\begin{bmatrix} a \\ \omega \end{bmatrix} = \begin{bmatrix} w_1 \\ w_2 \end{bmatrix},
\end{equation*}
assuming that $J$ is full rank.
\end{example}

\subsection{Closed-loop Control}
Trajectory tracking is just one example of closed-loop control, which assumes the existence of a desired trajectory for which to track. As previously discussed, one way of computing the desired trajectory is by solving an open-loop optimal control problem. However, in the context of optimal control, modifying an open-loop optimal control with feedback is not always the most desirable option. Instead, it may be preferred to just directly solve a closed-loop optimal control problem to obtain an optimal policy $\bm{u}^* = \pi(\x(t),t)$. Techniques for solving closed-loop optimal control problems typically are based on either the Hamilton-Jacobi-Bellman equation or dynamic programming.

Another common closed-loop control problem is to drive to or stabilize the system about a particular state (often called \textit{regulation}). For systems with linear dynamics models the most controller for regulation problems is called the \textit{linear quadratic regulator}. However, for nonlinear systems, stabilizing closed-loop controllers are commonly designed through \textit{Lyapunov analysis}\cite{SlotineLi1991}.

\subsubsection{Lyapunov-based Control}
A Lyapunov stability analysis is a common tool for analyzing the stability of nonlinear systems. This analysis is based on the definition of a Lyapunov function, which can be thought of as a measure of the ``energy'' of the system. Similar to mechanical systems, if the energy does not increase in time then the system is considered stable\footnote{Note there are more technical definitions of stability, but for simplicity these will not be discussed here}.

The most challenging part of a Lyapunov stability analysis is finding a suitable Lyapunov function, and for many complex systems this may be extremely difficult. However, one of the advantages of the method is that it provides nice theoretical guarantees regarding the stability of the system, and is applicable to any system of interest.

\begin{example}[Pose Stabilization] \label{ex:pose}
\theoremstyle{definition}
\cite{AicardiCasalinoEtAl1995} Consider a robot that is modeled by the unicycle robot model (differential drive robot model) represented graphically in Figure \ref{fig:posecartesian}
\begin{equation} \label{eq:posecartesian}
\begin{split}
\dot{x}(t) &= v(t) \cos\theta(t), \\
\dot{y}(t) &= v(t) \sin\theta(t), \\
\dot{\theta}(t) &= \omega(t),
\end{split}
\end{equation}
where the control inputs are the robot speed $v$ and the rotational rate $\omega$. The objective is to design a closed-loop controller that will drive the robot the origin (i.e. $x=0$, $y=0$, $\theta = 0$).

\begin{marginfigure}
\centering
\includegraphics[width=0.9\textwidth]{tex/figs/ch03_figs/unicycle_cartesian.png}
\caption{Pose stabilization of a unicycle robot in Cartesian coordinates.}
\label{fig:posecartesian}
\end{marginfigure}

To make the controller design easier the dynamics will be alternatively expressed in polar coordinates. This can be accomplished by defining
\begin{equation} \label{eq:polarcoord}
\begin{split}
\rho &= \sqrt{x^2+y^2}, \\
\alpha &= \mathrm{atan2}(y, x) - \theta + \pi, \\
\delta &= \alpha + \theta,
\end{split}
\end{equation}
where $\rho$ is the Euclidean distance to the origin, $\alpha$ is the heading angle with respect to the line from the robot to the origin, and $\delta$ is the angle between the $x$-axis and the line from the robot to the origin. These coordinates are graphically shown in Figure \ref{fig:polarcoord}.
\begin{marginfigure}
\centering
\includegraphics[width=0.95\textwidth]{tex/figs/ch03_figs/unicycle_polar.png}
\caption{Pose stabilization of a unicycle robot using polar coordinates.}
\label{fig:polarcoord}
\end{marginfigure}
Using the newly defined polar coordinates, the dynamics equations \eqref{eq:posecartesian} can be equivalently expressed as
\begin{equation} \label{eq:posepolar}
\begin{split}
\dot{\rho}(t) &= -v(t) \cos\alpha(t), \\
\dot{\alpha}(t) &= \frac{v(t) \sin\alpha(t)}{\rho(t)} - \omega(t), \\
\dot{\delta}(t) &= \frac{v(t) \sin \alpha(t)}{\rho(t)}. \\
\end{split}
\end{equation}

By expressing the dynamics in polar form, a Lyapunov stability analysis can now be easily performed. Consider the following \textit{candidate} Lyapunov function:
\begin{equation}
V(\rho, \alpha, \theta) = \frac{1}{2} \rho^2 + \frac{1}{2} (\alpha^2 + k_3 \delta^2),
\end{equation}
and consider the following closed-loop control law:
\begin{equation} \label{eq:poselaw}
\begin{split}
v &= k_1 \rho \cos \alpha,
\\
\omega &= k_2 \alpha + k_1 \frac{\sin \alpha \cos \alpha}{\alpha}(\alpha + k_3 \delta),
\end{split}
\end{equation}
where $k_1, k_2, k_3>0$.

The candidate Lyapunov function is quadratic and therefore is positive everywhere, $V \geq 0$, and is equal to zero only at the origin with $\rho = 0$, $\alpha = 0$, $\delta=0$. Therefore, if it is possible to show that along \textit{all} closed-loop system trajectories the Lyapunov function is \textit{decreasing} ($\dot{V} < 0$), then it can be guaranteed that the system will converge to the origin! To show that the Lyapunov function decreases along trajectories of the system, begin by taking the derivative of $V$:
\begin{equation*}
\dot{V} = \rho \dot{\rho} + \alpha \dot{\alpha} + k_3 \delta \dot{\delta}.
\end{equation*}
This quantity can now be shown to decrease along \textit{all} closed-loop trajectories by substituting in the dynamics equations \eqref{eq:polarcoord} with the closed-loop control law as defined by \eqref{eq:poselaw}:
\begin{equation*}
\begin{split}
\dot{V} &= \rho \dot{\rho} + \alpha \dot{\alpha} + k_3 \delta \dot{\delta}, \\
&= - \rho v \cos\alpha + \alpha \big(\frac{v \sin\alpha}{\rho} - \omega \big) +  \frac{k_3 \delta v \sin \alpha}{\rho}, \\
&= -k_1 \rho^2 \cos^2 \alpha  - k_2 \alpha^2, \\
\end{split}
\end{equation*}
where in the last line the control laws were substituted in for $v$ and $\omega$ and algebraically simplified. Note that since $k_1$ and $k_2$ have been chosen to be strictly positive, this function must be strictly negative for all values of $\rho$ and $\alpha$! Therefore this Lyapunov stability analysis has theoretically proven that the system under the closed-loop control law \eqref{eq:poselaw} will converge to the origin.
\end{example}

\subsection{Exercises}
Both exercises for this chapter can be found in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW1}.

\subsubsection{Pose Stabilization}
Complete \textit{Problem 2: Pose Stabilization}, where you will implement the Lyapunov-based pose controller for the unicycle robot described in Example \ref{ex:pose}.

\subsubsection{Trajectory Tracking}
Complete \textit{Problem 3: Trajectory Tracking}, where you will implement the differential flatness-based trajectory tracking controller for the extended unicycle robot described in Example \ref{ex:trajtrack}.


\subsection{Type Classes}\label{sec:type-classes}

So far we have only discussed \toolname in the context of
concrete Haskell functions, but many Haskell programs make
use of ad-hoc polymorphism via type-classes.
%
While the implementation of each type-class instance is
different, there is often a common interface that we
expect all instances to satisfy.

\mypara{Class Measures}
For example, consider the classes
%
\begin{code}
  class Indexable f where
    size :: f a -> Int
    at   :: f a -> Int -> a
\end{code}
%
For safe access, we might require that @at@'s second
parameter be bounded by the @size@ of the container.
To this end, we can define a \emph{type-indexed}
measure, using the @class measure@ keyword
%
\begin{code}
  class measure sz :: a -> Nat
\end{code}
%
Now, we can specify the safe-access preconditions
independant of particular instances of @Indexable@
as:
%
\begin{code}
  class Indexable f where
    size :: xs:f a -> {v:Nat | v = (sz xs)}
    at   :: xs:f a -> {v:Nat | v < (sz xs)} -> a
\end{code}

\mypara{Instance Measures}
For each concrete type for which a class is instantiated, we require
a corresponding definition for the measure. For example, we may want
to define lists to be an instance of @Indexable@. To do so, we would
define the @sz@ instance for lists as:
%
\begin{code}
  instance measure sz :: [a] -> Nat
  sz []     = 0
  sz (x:xs) = 1 + (sz xs)
\end{code}
%
%% Similarly, for a binary tree, we might define:
%% %
%% \begin{code}
%%   instance measure sz :: Tree a -> Nat
%%   size Leaf         = 0
%%   size (Node x l r) = 1 + (sz l) + (sz r)
%% \end{code}
%
Class measures work just like regular measures.
The above definitions are used to refine the types
of the data constructors, \eg
%
\begin{code}
  []  :: {v:[a] | (sz v) = 0}
  (:) :: a -> xs:[a]
      -> {v:[a] | (sz v) = 1 + (sz xs)}
\end{code}

Once we have defined the measure, we can define the actual
instance as:
%
\begin{code}
  instance Indexable [] where
    size []        = 0
    size (x:xs)    = 1 + size xs

    (x:xs) `at` 0  = x
    (x:xs) `at` i  = index xs (i-1)
\end{code}
%
Now, \toolname will use the definition of the @sz@ measure for lists
to check that @size@ and @at@ satisfy the refined class specifications,
and hence, that the above instance creates a valid instance dictionary.

\mypara{Client Verification}
On the client side of a type-class we can use the refined
types of class methods just as we would any regular function.
For example, consider a @sum@ function that works for any
@Indexable@.
%
\begin{code}
  sum :: (Indexable f) => f Int -> Int
  sum xs = go 0
  where
    go i
      | i < size xs = (xs `at` i) + go (i+1)
      | otherwise   = 0
\end{code}
%
\toolname proves that each call to @at@ is safe, by using the refined
class specifications of @Indexable@.
Specifically, each call to @at@ is guarded by a check @i < size xs@,
which, combined with the fact that @i@ is monotonically increasing
from 0, allows \toolname to prove that @xs `at` i@ will always be safe.

%%%%%%%%   %%
%%%%%%%%   %% We now have the task of recognizing each instance declaration
%%%%%%%%   %% and matching it up with the correct class specification, a
%%%%%%%%   %% non-trivial task in GHC's Core representation.
%%%%%%%%   %%
%%%%%%%%   %% Luckily for us, Haskell type-classes are implemented by passing around
%%%%%%%%   %% dictionaries of functions~\ES{CITE}. A typical instance for @Sized@
%%%%%%%%   %
%%%%%%%%   %% then \toolname will check that the generated class dictionary's @size@ field
%%%%%%%%   %% satisfies the above refinement type, \ie returns an
%%%%%%%%   %% will look something like the following in Core
%%%%%%%%   %
%%%%%%%%   \begin{code}
%%%%%%%%     $csize    = \xs -> case xs of
%%%%%%%%       []      -> 0
%%%%%%%%       (x:xs') -> 1 + $csize xs'
%%%%%%%%     $fSized[] = D:Sized $csize
%%%%%%%%   \end{code}
%%%%%%%%   %
%%%%%%%%   where @D:Sized@ is the dictionary data constructor. Instead of
%%%%%%%%   assigning a type directly to @$csize@ %$
%%%%%%%%   we assign a refined type to the data constructor @D:Sized@, specifically
%%%%%%%%   %
%%%%%%%%   % When \toolname sees a refined type-class definition it does two
%%%%%%%%   % things:
%%%%%%%%   % \begin{enumerate}
%%%%%%%%   % \item it \emph{assumes} the types of the class methods
%%%%%%%%   % \item it \emph{strengthens} the type of the dictionary data constructor to
%%%%%%%%   %   enforce the method types
%%%%%%%%   % \end{enumerate}
%%%%%%%%   % For our @Sized@ class this results in the type
%%%%%%%%   %
%%%%%%%%   \begin{code}
%%%%%%%%     D:Sized :: (xs:f a -> {v:Nat | v = size xs})
%%%%%%%%             -> Sized f
%%%%%%%%   \end{code}
%%%%%%%%   %
%%%%%%%%   from which \toolname creates the subtyping query
%%%%%%%%   %
%%%%%%%%   \begin{code}
%%%%%%%%     $csize <: (xs:f a -> {v:Nat | v = size xs})
%%%%%%%%   \end{code}
%%%%%%%%   %$
%%%%%%%%   which is valid, thus the instance declaration is safe.
%%%%%%%%   %
%%%%%%%%   Similarly for @Indexable@ we might find the instance
%%%%%%%%   %
%%%%%%%%   \begin{code}
%%%%%%%%     instance Indexable [] where
%%%%%%%%       index []     i = error "impossible"
%%%%%%%%       index (x:xs) 0 = x
%%%%%%%%       index (x:xs) i = index xs (i-1)
%%%%%%%%   \end{code}
%%%%%%%%   %
%%%%%%%%   translated into
%%%%%%%%   %
%%%%%%%%   \ES{FIXME: why does this next code block break latex???}
%%%%%%%%   % \begin{code}
%%%%%%%%   %   $cindex = \xs i -> case xs of
%%%%%%%%   %     []      -> error "impossible"
%%%%%%%%   %     (x:xs') -> case i of
%%%%%%%%   %       0 -> x
%%%%%%%%   %       _ -> $cindex xs' (i-1)
%%%%%%%%   %   $fIndexable[] = D:Indexable $fSized[] $cindex
%%%%%%%%   % \end{code}
%%%%%%%%   %$
%%%%%%%%   which will result in a similar subtyping query as above. The actual class
%%%%%%%%   methods merely select the correct function from the dictionary, \eg
%%%%%%%%   %
%%%%%%%%   \begin{code}
%%%%%%%%     index (D:Indexable sz idx) = idx
%%%%%%%%   \end{code}
%%%%%%%%   %
%%%%%%%%   so verification is trivial, as \toolname gets to assume the type of
%%%%%%%%   @D:Indexable@ when pattern-matching against it.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Test Problems}
\label{sec:53testProblems}

It is impossible to assess the capability of optimization methods
for every possible optimization problem.
The most widespread approach in literature
is the selection of a subset of specific problems
with different characteristics \term{(test problems)} and
the application of the methods to only these problems,
in the hope that the methods perform similarly in
actual application settings.

\paragraph{Trivial test functions}

When testing methods that involve sparse grid interpolation,
one has to consider that the function to be interpolated
does not satisfy a specific \term{trivial property.}
A test function $\objfun\colon \clint{\*0, \*1} \to \real$ is trivial if
$\objfun$ is a sum of tensor products of which at
least one factor is a linear polynomial, i.e., if
$\objfun$ is of the form
{%
  \setlength{\abovedisplayskip}{9pt}%
  \setlength{\belowdisplayskip}{9pt}%
  \begin{equation}
    \objfun(\*x) \equiv \sum_{q=1}^m \prod_{t=1}^d \objfun_{q,t}(x_t),\quad
    m \in \natz,\;\;
    \objfun_{q,t}\colon \clint{0, 1} \to \real,\;\;
    \faex{q = 1, \dotsc, m}{t \in \{1, \dotsc, d\}}{
      \objfun_{q,t} \in \polyspace{1},
    }
  \end{equation}%
}%
where $\polyspace{1}$ is the space of univariate polynomials
up to linear degree.
This is already fulfilled if the summands of $\objfun(\*x)$
do not depend on all coordinates $x_t$ of $\*x$.
One can show that for hat functions on sparse grids,
the hierarchical surpluses $\surplus{\*l,\*i}$ for trivial functions
vanish if $\*l \ge \*1$.
This means that trivial functions can be well-approximated by hat functions
on sparse grids just with boundary points, without placing any points
in the interior.
As this would distort our results,
we avoid trivial test functions in the following,
which include popular functions such as the
Branin01, Rosenbrock, and Schwefel26 functions.

\paragraph{Selection of test problems}

In the following, we select six unconstrained test problems
and two constrained test problems, which are listed in
\cref{tbl:optimizationProblem} and plotted in
\cref{fig:unconstrainedOptimizationProblem,fig:constrainedOptimizationProblem}.
The definitions of the problems are given in \cref{chap:a20testProblems}.
For the unconstrained case and the standard hierarchical
B-spline basis, a more exhaustive list of test functions has been
studied previously \cite{Valentin14Hierarchische}.
Gavana \cite{Gavana13Global} and Runarsson/Yao \cite{Runarsson00Stochastic}
provide a good overview of unconstrained and constrained test problems,
respectively.

\begin{table}
  \setnumberoftableheaderrows{1}%
  \begin{tabular}{%
    >{\kern\tabcolsep}=l<{\kern5mm}+l<{\kern5mm}*{5}{+c}%
    <{\kern5mm}+l<{\kern\tabcolsep}%
  }
    \toprulec
    \headerrow
    Name&           Abbr.& $d$& $m_{\ineqconfun}$& C&    CD&   MM&   Reference\\
    \midrulec
    Branin02&       Bra02& 2&   0&                 \yes& \yes& \yes& \cite{Munteanu98Global}\\
    GoldsteinPrice& GoP&   2&   0&                 \yes& \yes& \yes& \cite{Goldstein71Descent}\\
    Schwefel06&     Sch06& 2&   0&                 \yes& \no&  \no&  \cite{Schwefel77Numerische}\\
    Ackley&         Ack&   $d$& 0&                 \yes& \yes& \yes& \cite{Ackley87Connectionist}\\
    Alpine02&       Alp02& $d$& 0&                 \yes& \yes& \yes& \cite{Clerc99Swarm}\\
    Schwefel22&     Sch22& $d$& 0&                 \yes& \no&  \no&  \cite{Schwefel77Numerische}\\
    \midrulec
    G08&            G08&   2&   2&                 \yes& \yes& \yes& \cite{Schoenauer93Constrained}\\
    G04Squared&     G04Sq& 5&   6&                 \yes& \yes& \no&  \cite{Colville68Comparative}\\
    \bottomrulec
  \end{tabular}
  \caption[Selection of test problems in optimization]{%
    Unconstrained \emph{(top)} and constrained \emph{(bottom)} test problems.
    The columns state the full and abbreviated names,
    the dimensionality $d$ of the objective function $\objfun$,
    the number $m_{\ineqconfun}$ of constraints,
    whether $\objfun$ is continuous in the domain
    $\clint{\*0, \*1}$ (C),
    whether $\objfun$ is continuously differentiable in the domain
    $\clint{\*0, \*1}$ (CD),
    whether $\objfun$ is multi-modal (MM, i.e.,
    whether there are multiple local minima), and
    a reference to the original literature that defines the problem.%
  }%
  \label{tbl:optimizationProblem}%
\end{table}

\begin{figure}
  \subcaptionbox{%
    Bra02%
  }[71mm]{%
    \includegraphics{optimizationProblem_1}%
  }%
  \hfill%
  \subcaptionbox{%
    GoP%
  }[76mm]{%
    \includegraphics{optimizationProblem_2}%
  }\\[2.5mm]%
  \subcaptionbox{%
    Sch06%
  }[71mm]{%
    \includegraphics{optimizationProblem_3}%
  }%
  \hfill%
  \subcaptionbox{%
    Ack for $d = 2$%
  }[76mm]{%
    \includegraphics{optimizationProblem_4}%
  }\\[2.5mm]%
  \subcaptionbox{%
    Alp02 for $d = 2$%
  }[71mm]{%
    \includegraphics{optimizationProblem_5}%
  }%
  \hfill%
  \subcaptionbox{%
    Sch22 for $d = 2$%
  }[76mm]{%
    \includegraphics{optimizationProblem_6}%
  }%
  \caption[%
    Unconstrained test problems%
  ]{%
    Bivariate test functions $\objfunscaled$ in unconstrained optimization.
    The \textcolor{C1}{red dot} indicates the location of the
    global minimum.%
  }%
  \label{fig:unconstrainedOptimizationProblem}%
\end{figure}

\begin{figure}
  \subcaptionbox{%
    G08%
  }[72mm]{%
    \includegraphics{optimizationProblem_7}%
  }%
  \hfill%
  \subcaptionbox{%
    G04Sq (bivariate projection over $\xscaledentry{3}$ and $\xscaledentry{5}$
    onto $\xscaledentry{t} = \xoptscaledentry{t}$ for $t = 1, 2, 4$)%
  }[72mm]{%
    \includegraphics{optimizationProblem_8}%
  }%
  \caption[%
    Constrained test problems%
  ]{%
    Test problems in constrained optimization.
    The \textcolor{C0}{blue areas} denote the inequality constraints and
    the \textcolor{C1}{red dot} indicates the location of the
    global minimum.%
  }%
  \label{fig:constrainedOptimizationProblem}%
\end{figure}

For each test problem, we state unscaled versions of objective functions
$\objfunscaled\colon \clint{\*a, \*b} \to \real$,
$\xscaled \mapsto \objfunscaled(\xscaled)$
(and the unscaled constraint function $\ineqconfunscaled$, if present).
The actual objective function $\objfun\colon \clint{\*0, \*1} \to \real$
can be obtained by $\objfun(\*x) \ceq \objfunscaled(\xscaled)$
with the affine parameter transformation
$x_t = \tfrac{\xscaledentry{t} - a_t}{b_t - a_t}$, $t = 1, \dotsc, d$
(similarly for the constraint function).

\vspace*{1em}

The parameter domains of some test problems have been slightly translated
compared to the literature
to avoid that the minima are located exactly at or close to
the center of the domain.
In these cases, sparse grids would be in advantage as
they tend to place more points near the center of the domain
(especially for high dimensionalities).


\documentclass[notitlepage]{problem-solving}

\author{Matt McCarthy}
\title{Trigonometric Sum and Difference Formulae in $\CC$}
\date{June 2016}

\addbibresource{complex-trig-sum.bib}
\nocite{*}

\begin{document}

\maketitle

\begin{problem*}
	Show that for any $z,w\in\CC$,
	\[
		\sin(z+w)=\cos z\sin w+\sin z\cos w \text{ and } \cos(z+w)=\cos z\cos w - \sin z\sin w.
	\]
\end{problem*}

\section{Background}

In the world of $\CC$, differentiability is a stronger condition than in $\RR$.
To be precise, once a function is differentiable in $\CC$, it is infinitely differentiable.
This happens because differentiability has more stringent requirements in $\CC$.
In order for a $\CC$-valued function to be differentiable on a domain, it must be \textit{analytic} on that domain.
\begin{definition}
	Let $f:\CC\rightarrow\CC$, where $f(x+iy)=u(x,y)+iv(x,y)$, be a function such that $\partial u/\partial x$, $\partial u/\partial y$, $\partial v/\partial x$, and $\partial v/\partial y$ exist and are continuous on some disk $D\subseteq\CC$ with a nonzero radius.
	If $f$ satisfies the \textit{Cauchy-Riemann equations}
	\[
		\frac{\partial u}{\partial x} = \frac{\partial v}{\partial y} \text{ and } \frac{\partial u}{\partial y} = -\frac{\partial v}{\partial x}
	\]
	then $f$ is said to be \textit{analytic on $D$}.
	Furthermore, the largest subset of $\CC$ on which $f$ is analytic is called $f$'s \textit{domain of analyticity}.
	If the domain of analyticity is $\CC$, then $f$ is said to be \textit{entire}.
\end{definition}
Since any analytic function is infinitely differentiable, its Taylor expansion exists as well.
Furthermore, the Taylor series converges on any disk in the domain of analyticity.
\begin{thm}
	Let $f$ be analytic on a disk $D(z_0,r)$, then $f$'s Taylor series about $z_0$ converges for all $z\in D$.
\end{thm}

\section{Solution}

\begin{problem*}
	Show that for any $z,w\in\CC$,
	\[
		\sin(z+w)=\cos z\sin w+\sin z\cos w \text{ and } \cos(z+w)=\cos z\cos w - \sin z\sin w.
	\]
\end{problem*}

To prove these statements, we will find $\sin z$ and $\cos z$ in terms of $e^z$.
From there, we will show the sum formulae.
However, before we can get to either of those steps, we need to show that $e^z$ is entire.

\subsection{Analyticity of $e^{z}$}

\begin{proposition}
	$e^z$ is entire.
\end{proposition}
\begin{proof}
	Let $z=x+iy$.
	We want to find $\RR$-valued functions $u,v$ such that $e^{x+iy}=u(x,y)+iv(x,y)$.
	\begin{align*}
		e^z &= e^{x+iy}\\
		&= e^x e^{iy}\\
		&= e^x \cos y + ie^x\sin y\\
		&= u(x,y) +iv(x,y)
	\end{align*}
	Taking partial derivatives yields the following.
	\[
		\begin{array}{rlrl}
			\partial u/\partial x =& e^x \cos y  \hspace{1cm}&  \partial v/\partial y =& e^x \cos y\\
			\partial u/\partial y =& -e^x \sin y \hspace{1cm}& -\partial v/\partial x =& -e^x \sin y
		\end{array}
	\]
	Therefore $e^z$ satisfies the Cauchy-Riemann equations.
	Furthermore, since the equations hold for all $z\in\CC$, $e^z$ is entire.
\end{proof}

\subsection{Complex Trigonometric Functions in Terms of the Complex Exponential}

Our next step is to use the Taylor expansion of $e^z$ to get the Taylor expansion of $\cos z$ and $\sin z$.

\begin{proposition}
	Let $z\in\CC$.
	Then
	\[
		\cos z = \frac{ e^{iz} + e^{-iz} }{2} \text{ and } \sin z = \frac{ e^{iz} - e^{-iz} }{2i}.
	\]
\end{proposition}
\begin{proof}
	Consider $\paren{e^{iz}+e^{-iz}}/2$.
	Since $e^z$ is entire, its Taylor series converges everywhere in $\CC$.
	\begin{align*}
		\frac{1}{2}\paren{e^{iz}+e^{-iz}}
		&= \frac{1}{2}\paren{\sum_{n=0}^\infty \frac{1}{n!} i^n z^n + \frac{(-1)^n}{n!} i^n z^n}\\
		&= \frac{1}{2}\paren{\sum_{n=0}^\infty \frac{1+(-1)^n}{n!} i^n z^n}\\
		&= \frac{1}{2}\paren{\sum_{n=0}^\infty \frac{1+(-1)^{2n}}{(2n)!}i^{2n}z^{2n} + \frac{1+(-1)^{2n+1}}{(2n+1)!}i^{2n+1}z^{2n+1}}\\
		&= \frac{1}{2}\paren{\sum_{n=0}^\infty \frac{2}{(2n)!}(i^2)^nz^{2n}}\\
		&= \sum_{n=0}^\infty \frac{(-1)^n}{(2n!)} z^{2n}\\
		&= \cos z
	\end{align*}
	Consider $\paren{e^{iz}-e^{-iz}}/(2i)$.
	\begin{align*}
		\frac{1}{2i}\paren{e^{iz}-e^{-iz}}
		&= \frac{1}{2i}\paren{\sum_{n=0}^\infty \frac{1}{n!} i^n z^n - \frac{(-1)^n}{n!} i^n z^n}\\
		&= \frac{1}{2i}\paren{\sum_{n=0}^\infty \frac{1-(-1)^n}{n!} i^n z^n}\\
		&= \frac{1}{2i}\paren{\sum_{n=0}^\infty \frac{1-(-1)^{2n}}{(2n)!}i^{2n}z^{2n} + \frac{1-(-1)^{2n+1}}{(2n+1)!}i^{2n+1}z^{2n+1}}\\
		&= \frac{1}{2i}\paren{\sum_{n=0}^\infty \frac{2i}{(2n+1)!}(i^2)^nz^{2n+1}}\\
		&= \sum_{n=0}^\infty \frac{(-1)^n}{(2n+1)!} z^{2n+1}\\
		&= \sin z
	\end{align*}
\end{proof}

\subsection{Trigonometric Sum Identities}

We will now use our new identities to show the sum formulae.
Afterwards, we will get the difference formulae as corollaries.

\begin{thm}
	Let $z,w\in\CC$.
	Then $\sin(z+w)=\cos z\sin w+\sin z\cos w$.
\end{thm}
\begin{proof}
	Consider $\cos z\sin w+\sin z\cos w$.
	\begin{align*}
		\cos z\sin w+\sin z\cos w &=
		\paren{\frac{e^{iz}+e^{-iz}}{2}}\paren{\frac{e^{iw}-e^{-iw}}{2i}}+\paren{\frac{e^{iz}-e^{-iz}}{2i}}\paren{\frac{e^{iw}+e^{-iw}}{2}}\\
		&= \frac{e^{i(z+w)} +e^{i(w-z)} - e^{i(z-w)} - e^{-i(z+w)}}{4i} + \frac{e^{i(z+w)} -e^{i(w-z)} + e^{i(z-w)} - e^{-i(z+w)}}{4i}\\
		&= \frac{2e^{i(z+w)}-2e^{-i(z+w)}}{4i}\\
		&= \frac{e^{i(z+w)}-e^{-i(z+w)}}{2i}\\
		&= \sin(z+w)
	\end{align*}
\end{proof}

\begin{thm}
	Let $z,w\in\CC$.
	Then $\cos(z+w)=\cos z\cos w - \sin z\sin w$.
\end{thm}
\begin{proof}
	Consider $\cos z\cos w - \sin z\sin w$.
	\begin{align*}
		\cos z\cos w - \sin z\sin w &=
		\paren{\frac{e^{iz}+e^{-iz}}{2}}\paren{\frac{e^{iw}+e^{-iw}}{2}} - \paren{\frac{e^{iz}-e^{-iz}}{2i}}\paren{\frac{e^{iw}-e^{-iw}}{2i}}\\
		&= \frac{e^{i(z+w)} + e^{i(w-z)} + e^{i(z-w)} + e^{-i(z+w)}}{4} + \frac{e^{i(z+w)} -e^{i(w-z)} - e^{i(z-w)} + e^{-i(z+w)}}{4}\\
		&= \frac{e^{i(z+w)}+e^{-i(z+w)}}{2}\\
		&= \cos(z+w)
	\end{align*}
\end{proof}

\begin{corollary}
	Let $z,w\in\CC$.
	Then
	\[
		\sin(z-w)=\sin z\cos w - \cos z\sin w \text{ and }\cos(z-w)=\cos z\cos w + \sin z\sin w.
	\]
\end{corollary}
\begin{proof}
	Since $\sin w$ is an odd function, $\sin(-w)=-\sin w$.
	Since $\cos w$ is an even function, $\cos (-w)=\cos w$.
	These facts combined with the sum formulae for $\cos(z+(-w))$ and $\sin(z+(-w))$ yield the difference formulae.
\end{proof}

\printbibliography

\end{document}


% Preamble.
\documentclass[12pt]{article}
\usepackage[margin=1.25in]{geometry}
\usepackage[fleqn]{amsmath}
\usepackage{textcomp}
\usepackage{gensymb}
\usepackage{amsfonts}
\usepackage{enumitem}
%\usepackage{tikz}  % Include for figures.
%\usepackage{subfiles}  % Include for subfiles.

%% Title macros.
\newcommand{\HOMEWORKNUM}{14}
\newcommand{\NAME}{D. Choi}
\newcommand{\DATE}{2020-06-15}

\title{\vspace{-2\baselineskip}MATH 225 - Homework \#\HOMEWORKNUM}
\author{\NAME}
\date{\DATE}

%% Formatting options.
%\pagenumbering{gobble}  % Include for single-page document.

% Macros.
%% Norm.
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

% Document.
\begin{document}
\maketitle

\section*{1.}
\textit{For a generic orthographic projection $A$ in $\mathbb{R}^3$, show that
$\text{det}(A) = 0$.} \\[\baselineskip]
Let $\vec{n}$ be the normal vector so that the orthographic projection $A$ of
a vector $\vec{v}$ is equivalent to the orthographic projection onto the plane
\begin{equation*}
	\vec{n} \cdot \vec{v}
	=
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	\cdot
	\begin{pmatrix} x \\ y \\ z \end{pmatrix}
	=
	0
	.
\end{equation*}
Such a projection is also equivalent to the vector rejection of $\vec{v}$ from
$\vec{n}$. Thus,
\begin{equation*}
	A\vec{v}
	=
	\vec{v} - \text{proj}_{\vec{n}}\vec{v}
	=
	\begin{pmatrix} x \\ y \\ z \end{pmatrix}
	-
	\frac{
		\begin{pmatrix} x \\ y \\ z \end{pmatrix}
		\cdot
		\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	}{
		\norm{\begin{pmatrix} a \\ b \\ c \end{pmatrix}}^2
	}
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	=
	\begin{pmatrix} x \\ y \\ z \end{pmatrix}
	-
	\frac{ax + by + cz}{a^2 + b^2 + c^2}
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	.
\end{equation*}
The columns of $A$ can be found by computing the vector rejections of the
standard basis unit vectors from $\vec{n}$:
\begin{gather*}
	A \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}
	=
	\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}
	-
	\frac{a}{a^2 + b^2 + c^2}
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	=
	\frac{1}{a^2 + b^2 + c^2}
	\begin{pmatrix}
		b^2 + c^2 \\
		-ab \\
		-ac
	\end{pmatrix}
	\\
	A \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}
	=
	\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}
	-
	\frac{b}{a^2 + b^2 + c^2}
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	=
	\frac{1}{a^2 + b^2 + c^2}
	\begin{pmatrix}
		-ab \\
		a^2 + c^2 \\
		-bc
	\end{pmatrix}
	\\
	A \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
	=
	\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
	-
	\frac{c}{a^2 + b^2 + c^2}
	\begin{pmatrix} a \\ b \\ c \end{pmatrix}
	=
	\frac{1}{a^2 + b^2 + c^2}
	\begin{pmatrix}
		-ac \\
		-bc \\
		a^2 + b^2
	\end{pmatrix}
	.
\end{gather*}
Thus,
\begin{equation*}
	A
	=
	\frac{1}{a^2 + b^2 + c^2}
	\begin{pmatrix}
		b^2 + c^2 & -ab & -ac \\
		-ab & a^2 + c^2 & -bc \\
		-ac & -bc & a^2 + b^2
	\end{pmatrix}
	.
\end{equation*}
For the sake of convenience, let
\begin{equation*}
	B
	=
	\begin{pmatrix}
		b^2 + c^2 & -ab & -ac \\
		-ab & a^2 + c^2 & -bc \\
		-ac & -bc & a^2 + b^2
	\end{pmatrix}
\end{equation*}
so that
\begin{equation*}
	A = \frac{B}{a^2 + b^2 + c^2},
\end{equation*}
which allows computing the determinant of $A$ as
\begin{equation*}
	\text{det}(A)
	=
	\left( \frac{1}{a^2 + b^2 + c^2} \right)^3
	\text{det}(B)
	=
	s \cdot \text{det}(B)
	.
\end{equation*}
The row reduction of $B$ results in
\begin{equation*}
	\text{rref}(B)
	=
	\begin{pmatrix}
		1 & 0 & -\frac{a}{c} \\
		0 & 1 & -\frac{b}{c} \\
		0 & 0 & 0
	\end{pmatrix}
	.
\end{equation*}
Because $\text{rref}(B)$ contains a row of zeros, the determinant of $B$ must
be $0$. \\
Thus,
\begin{equation*}
	\text{det}(A)
	=
	s \cdot \text{det}(B)
	=
	s \cdot 0
	=
	\boxed{0}
	.
\end{equation*}

\end{document}


\section{Jachimowsky Combustion Model}

The calculation of a chemical source term is explained in detail in Anderson. \cite{gen:anderson2}.
A brief summary of that exaplanation follows here.
Each species conservation equation has a source term for each species $k$ given by:

\begin{displaymath}
\dot{W}_k = {\cal M}_k \sum_{l=1}^{N_e} \left( \nu''_{l,k} - \nu'_{l,k} \right)
\left[ k_{f,l} \prod_{m=1}^{N_l} \left[ \chi_m \right]^{\nu'_{l,m}}
- k_{b,l} \prod_{m=1}^{N_l} \left[ \chi_m \right]^{\nu''_{l,m}} \right]
\end{displaymath}

$N_e$ is the number of elementary reactions involving species $k$.
$N_l$ is the number of species involved in the elementary reaction.
${\cal M}_k$ is the molecular weight of species $k$.
$\chi_m$ is the mole fraction of species $m$.
$\nu'_{l,k}$ is the stoichiometric coefficient for the reactants of reaction $l$.
$\nu''_{l,k}$ is the stoichiometric coefficient for the products of reaction $l$.
$k_b$ is the backward reaction rate constant and
$k_f$ is the forward reaction rate constant, given by the
modified Arrhenius equation:
\begin{displaymath}
k_f = AT^n \exp(-E/{\cal R}T)
\end{displaymath}
where the coefficients $A$, $n$ and $E$ are germane to a particular combustion
model. In this study, the Jachimowsky hydrogen-air combustion model is used
\cite{chem:jachimowsky} and the coefficients are given in table \ref{model}
below.

The backward and the forward reaction rate constants are related by the
equilibrium constant, $K_c$
\begin{displaymath}
\frac{k_f}{k_b} = K_c
\end{displaymath}
The equilibrium constant, $K_c$ is given by:
\begin{displaymath}
K_c = ({\cal R} T)^{-\Delta \nu}
       \exp \left( \frac{-\Delta G^0}{{\cal R} T} \right)
\end{displaymath}

where
\begin{displaymath}
\Delta \nu = \sum_{i=1}^{N_l} (\nu''_i - \nu'_i)
\end{displaymath}
The Gibbs free energy for species $i$
is defined as $G_i = H_i - T S_i$, where $H_i$ is the
enthalpy, $T$ is the temperature and $S_i$ is the entropy.
The difference in Gibbs free energy $\Delta G^0$ of products and reactants is
\begin{displaymath}
\Delta G^0 = \sum_{i=1}^{N_l} (\nu''_i - \nu'_i) \left( h^0_i - T s^0_i \right)
\end{displaymath}
where $h^0_i$ is the molar enthalpy (including the heat of formation at a
reference temperature of 298 K)
of the species $i$, and $s^0_i$ is its the molar entropy,
each calculated at a reference pressure (one atmosphere).\\
\\

\begin{table}
\begin{threeparttable}
\caption{Jachimowsky Hydrogen-Air Combustion Model with Nitrogen Inert}
\begin{tabular}{|cc|c|c|c|} \hline
\multicolumn{2}{|c|}{Reaction} & A & n & E  \\ \hline \hline
(1) & H$_{2}$ + O$_{2} \rightarrow$ OH + OH & 1.70 $\times$ 10$^{13}$ & 0 & 48 000 \\
(2) & H + O$_{2} \rightarrow$ OH + O & 2.60 $\times$ 10$^{14}$ & 0 & 16 800 \\
(3) & O + H$_{2} \rightarrow$ OH + H & 1.80 $\times$ 10$^{10}$ & 1.00 & 8 900 \\
(4) & OH + H$_{2} \rightarrow$ H$_{2}$O + H & 2.20 $\times$ 10$^{13}$ & 0 & 5 150 \\
(5) & OH + OH $\rightarrow$ H$_{2}$O + O & 6.30 $\times$ 10$^{12}$ & 0 & 1 090 \\
(6) & H + OH + M $\rightarrow$ H$_{2}$O + M & 2.20 $\times$ 10$^{22}$ & -2.00 & 0 \\
(7) & H + H + M $\rightarrow$ H$_{2}$ + M & 6.40 $\times$ 10$^{17}$ & -1.00 & 0 \\
(8) & H + O + M $\rightarrow$ OH + M & 6.00 $\times$ 10$^{16}$ & -0.60 & 0 \\
(9) & H + O$_{2}$ + M $\rightarrow$ HO$_{2}$ + M & 2.10 $\times$ 10$^{15}$ & 0 & -1 000 \\
(10) & HO$_{2}$ + H $\rightarrow$ H$_{2}$ + O$_{2}$ & 1.30 $\times$ 10$^{13}$ & 0 & 0 \\
(11) & HO$_{2}$ + H $\rightarrow$ OH + OH & 1.40 $\times$ 10$^{14}$ & 0 & 1 080 \\
(12) & HO$_{2}$ + H $\rightarrow$ H$_{2}$O + O & 1.00 $\times$ 10$^{13}$ & 0 & 1 080 \\
(13) & HO$_{2}$ + O $\rightarrow$ O$_{2}$ + OH & 1.50 $\times$ 10$^{13}$ & 0 &  950 \\
(14) & HO$_{2}$ + OH $\rightarrow$ H$_{2}$O + O$_{2}$ & 8.00 $\times$ 10$^{12}$ & 0 & 0 \\
(15) & HO$_{2}$ + HO$_{2} \rightarrow$ H$_{2}$O$_{2}$ + O$_{2}$ & 2.00 $\times$ 10$^{12}$ & 0 & 0 \\
(16) & H + H$_{2}$O$_{2} \rightarrow$ H$_{2}$ + HO$_{2}$ & 1.40 $\times$ 10$^{12}$ & 0 & 3 600 \\
(17) & O + H$_{2}$O$_{2} \rightarrow$ OH + HO$_{2}$ & 1.40 $\times$ 10$^{13}$ & 0 & 6 400 \\
(18) & OH + H$_{2}$O$_{2} \rightarrow$ H$_{2}$O + HO$_{2}$ & 6.10 $\times$ 10$^{12}$ & 0 & 1 430 \\
(19) & M + H$_{2}$O$_{2} \rightarrow$ OH + OH + M & 1.20 $\times$ 10$^{17}$ & 0 & 45 500 \\
(20) & O + O + M $\rightarrow$ O$_{2}$ + M & 6.00 $\times$ 10$^{17}$ & 0 & -1 800 \\
\hline
\end{tabular}
\label{model}
\end{threeparttable}
\end{table}

The units for $A$ are in $[\frac{cm^{3b}}{(mole-s)^b}]$ where $b=1$ for two body reactions and
$b=2$ for three body reactions. $E$ is in $[\frac{cal}{mole}]$.\\
\\

The symbol $M$ denotes a third-body collision partner, a species acting as a catalyst only.
The concentration of $M$ is simply determined from the equation :\\

\begin{displaymath}
X_M = \sum_{k=1}^{n_s} \eta_k X_k
\end{displaymath}
\clearpage

where $\eta_k$ is the third-body efficiency. $\eta_k$ is unity
for most species and reactions except those listed in the table
\ref{thirds} above.

\vspace*{-3cm}
\begin{table}
\begin{threeparttable}
\caption{Third Body Efficiencies for Jachimowsky Model}
\begin{tabular}{|cc|cccc|} \hline
\multicolumn{2}{|c|}{Reaction} & \multicolumn{4}{|c|}{third body efficiency} \\ \hline \hline
(6) & H + OH + M $\rightarrow$ H$_{2}$O + M & H$_{2}$ & 1.0 & H$_{2}$O & 6.0 \\
(7) & H + H + M $\rightarrow$ H$_{2}$ + M & H$_{2}$ & 2.0 & H$_{2}$O & 6.0 \\
(8) & H + O + M $\rightarrow$ OH + M & H$_{2}$ & 1.0 & H$_{2}$O & 5.0 \\
(9) & H + O$_{2}$ + M $\rightarrow$ HO$_{2}$ + M & H$_{2}$ & 2.0 & H$_{2}$O & 16.0 \\
(19) & M + H$_{2}$O$_{2} \rightarrow$ OH + OH + M & H$_{2}$ & 1.0 & H$_{2}$O & 15.0 \\
\hline
\end{tabular}
\label{thirds}
\end{threeparttable}
\end{table}


\section{Equivalence of DFAs and NFAs}

NFAs are easier to build than DFAs because one does not have to worry,
for any state, of having out-going edges carrying a unique label. The
surprising fact is that NFAs and DFAs actually have the same
expressiveness, that is, all that can be defined by means of a NFA can
also be defined with a DFA (the converse is trivial since a DFA is
already a NFA). More precisely, there is a procedure, called \emph{the
  subset construction}, which converts any NFA to a DFA.

Consider that, in a NFA, from a state~\(q\) with several out-going
edges with the same label~\(a\), the transition function~\(\delta_N\)
leads, in general, to several states. The idea of the \emph{subset
  construction} is to create a new automaton where these edges are
merged. So we create a state~\(p\) which corresponds to the set of
states \(\delta_N (q,a)\) in the NFA. Accordingly, we create a
state~\(r\) which corresponds to the set~\(\{q\}\) in the NFA. We
create an edge labelled~\(a\) between~\(r\) and~\(p\). The important
point is that \emph{this edge is unique}. This is the first step to
create a DFA from a NFA.

Graphically, instead of the non-determinism of
\fig~\vref{fig:nfa_edges},
\begin{figure}
\centering
\subfloat[Non-determinism\label{fig:nfa_edges}]{
\includegraphics[bb=62 660 222 730]{nfa_edges}
}
\qquad
\subfloat[Determinism\label{fig:nfa_subset}]{
\includegraphics[bb=60 712 144 730]{nfa_subset}
}
\caption{From non-determinism to determinism}
\end{figure}
where we have \(\delta_N (q, a) = \{p_0, p_1, \dots, p_n\}\), we get
the determinism of \fig~\vref{fig:nfa_subset}.

Let us present the complete algorithm for the subset construction.

Let us start from a NFA \(\mathcal{N} = (Q_N, \Sigma, \delta_N, q_0,
F_N)\). The goal is to construct a DFA \(\mathcal{D} = (Q_D, \Sigma,
\delta_D, \{q_0\}, F_D)\) such that \(L(\mathcal{D}) =
L(\mathcal{N})\). Notice that the input alphabet of the two automata
are the same and the initial state of~\(\mathcal{D}\) if the set
containing only the initial state of~\(\mathcal{N}\).

The other components of~\(\mathcal{D}\) are constructed as
follows. First, \(Q_D\)~is the set of subsets of~\(Q_N\); that is,
\(Q_D\)~is the \emph{power set} of~\(Q_N\). Thus, if~\(Q_D\) has~\(n\)
states, \(Q_D\)~has~\(2^n\) states. Fortunately, often not all these
states are accessible from the initial state of~\(Q_D\), so these
inaccessible states can be discarded.

\label{state_explosion}
Why is~\(2^n\) the number of subsets of a finite set of
cardinal~\(n\)?

Let us order the~\(n\) elements and represent each subset by an
\(n\)-bit string where bit \(i\) corresponds to the \(i\)th element:
it is~\(1\) if the \(i\)th element is present in the subset and~\(0\)
if not. This way, we counted all the subsets and not more (a bit
cannot always be \(0\) since all elements are used to form subsets and
cannot always be \(1\) if there is more than one element). There
are~\(2\) possibilities, \(0\)~or~\(1\), for the first bit;
\(2\)~possibilities for the second bit etc. Since the choices are
independent, we multiply all of them: \(\underbrace{2 \times 2 \times
  \dots \times 2}_{n \; \text{times}} = 2^n\), yielding the number of
subsets of an \(n\)-element set.

Resuming the definition of DFA~\(\mathcal{D}\), the remaining
components are defined as follows.
\begin{itemize}

  \item \(F_D\)~is the set of subsets~\(S\) of~\(Q_N\) such that \(S
    \cap F_N \neq \varnothing\). That is, \(F_D\)~is all sets of
    \(N\)'s states that include at least one final state
    of~\(\mathcal{N}\).

  \item for each set \(S \subseteq Q_N\) and for each input \(a \in
    \Sigma\),
  \begin{equation*}
    \delta_D(S, a) = \bigcup_{q \in S}{\delta_N (q, a)}.
  \end{equation*}
  In other words, to compute~\(\delta_D (S, a)\), we look at all the
  states~\(q\) in~\(S\), see what states of~\(\mathcal{N}\) are
  reached from~\(q\) on input~\(a\) and take the union of all those
  states to make the next state of~\(\mathcal{D}\).

\end{itemize}
Let us reconsider the NFA given by its transition table in
\fig~\vref{fig:initial_nfa_table}
\begin{figure}
\centering
\(\begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{NFA} \; \mathcal{N}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & q_0 & \{q_0, q_1\} & \{q_0\}\\
                & q_1 & \varnothing  & \{q_2\}\\
    \#          & q_2 & \varnothing  & \varnothing
\end{array}\)
\caption{Initial NFA table\label{fig:initial_nfa_table}}
\end{figure}
and let us create an equivalent DFA. Firstly, we form all the subsets
of the sets of the NFA and put them in the first column in
\fig~\vref{fig:first_stage}.
\begin{figure}
\centering
\subfloat[First stage\label{fig:first_stage}]{
\(\begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    & \varnothing & &\\
    & \{q_0\}           & &\\
    & \{q_1\}           & &\\
    & \{q_2\}           & &\\
    & \{q_0, q_1\}      & &\\
    & \{q_0, q_2\}      & &\\
    & \{q_1, q_2\}      & &\\
    & \{q_0, q_1, q_2\} & &
\end{array}\)
}
\qquad
\subfloat[Second stage\label{fig:second_stage}]{
\(\begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    & \varnothing & &\\
    \rightarrow & \{q_0\}  & &\\
       & \{q_1\}           & &\\
    \# & \{q_2\}           & &\\
       & \{q_0, q_1\}      & &\\
    \# & \{q_0, q_2\}      & &\\
    \# & \{q_1, q_2\}      & &\\
    \# & \{q_0, q_1, q_2\} & &
\end{array}\)
}
\caption{First two stages of the subset construction}
\end{figure}

Secondly, we annotate in this first column the states
with~\(\rightarrow\) if and only if they contain the initial state of
the NFA, here~\(q_0\), and we add a~\(\#\) if and only if the states
contain at least a final state of the NFA, here~\(q_2\). See
\fig~\vref{fig:second_stage}.

Thirdly, we form the subsets as follows:
\begin{equation*}
\small
\begin{array}{@{}r@{}l@{\,}||@{\,}c@{\,}|@{\,}c@{}}
\multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
\hhline{==::==}
            & \varnothing & \varnothing & \varnothing\\
\rightarrow & \{q_0\} & \delta_N(q_0, 0) & \delta_N(q_0, 1)\\
            & \{q_1\} & \delta_N(q_1, 0) & \delta_N(q_1, 1)\\
         \# & \{q_2\} & \delta_N(q_2, 0) & \delta_N(q_2, 1)\\
            & \{q_0, q_1\} & \delta_N(q_0,0) \cup \delta_N(q_1,0)
                           & \delta_N(q_0,1) \cup \delta_N(q_1,1)\\
         \# & \{q_0, q_2\} & \delta_N(q_0,0) \cup \delta_N(q_2,0)
                           & \delta_N(q_0,1) \cup \delta_N(q_2,1)\\
         \# & \{q_1, q_2\} & \delta_N(q_1,0) \cup \delta_N(q_2,0)
                           & \delta_N(q_1,1) \cup \delta_N(q_2,1)\\
         \# & \{q_0, q_1, q_2\}
            & \delta_N(q_0,0) \cup \delta_N(q_1,0) \cup \delta_N(q_2,0)
            & \delta_N(q_0,1) \cup \delta_N(q_1,1) \cup \delta_N(q_2,1)
\end{array}
\end{equation*}
Finally, we compute those subsets and obtain the table in
\fig~\ref{fig:first_dfa_table}.
\begin{figure}[t]
\centering
\(\begin{array}{r@{}l||c|c}
\multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
\hhline{==::==}
            & \varnothing & \varnothing & \varnothing\\
\rightarrow & \{q_0\} & \{q_0, q_1\} & \{q_0\}\\
            & \{q_1\} & \varnothing & \{q_2\}\\
         \# & \{q_2\} & \varnothing & \varnothing\\
            & \{q_0, q_1\} & \{q_0,q_1\} & \{q_0,q_2\}\\
         \# & \{q_0, q_2\} & \{q_0,q_1\} & \{q_0\}\\
         \# & \{q_1, q_2\} & \varnothing & \{q_2\}\\
         \# & \{q_0, q_1, q_2\} & \{q_0, q_1\} & \{q_0,q_2\}
\end{array}\)
\caption{First DFA obtained\label{fig:first_dfa_table}}
\end{figure}
The transition diagram of the DFA~\(\mathcal{D}\) is showed in
\fig~\vref{fig:dfa_from_nfa}
\begin{figure}[b]
\centering
\includegraphics[bb=42 628 291 758]{dfa_from_nfa}
\caption{Transition diagram of \fig~\vref{fig:first_dfa_table}
\label{fig:dfa_from_nfa}}
\end{figure}
where states with out-going edges which have no end are final
states. If we look carefully at the transition diagram, we see that
the DFA is actually made of two disconnected sub\hyp{}automata. In
particular, since we have only one initial state, this means that one
part is not accessible, therefore its states are never used to
recognise or reject an input word, and we can remove this part, as
shown in \fig~\ref{fig:dfa_from_nfa_simple1}.
\begin{figure}
\centering
\subfloat[Simplification of \fig~\ref{fig:dfa_from_nfa}
\label{fig:dfa_from_nfa_simple1}]{
\includegraphics[bb=43 628 162 758]{dfa_from_nfa_simple1}
}
\qquad
\subfloat[State renamings of \fig~\ref{fig:dfa_from_nfa_simple1}
\label{fig:dfa_from_nfa_simple}]{
\includegraphics[bb=48 625 150 755]{dfa_from_nfa_simple}
}
\caption{Simplifications}
\end{figure}
It is important to understand that the states of the DFA are subsets
of the NFA states. This is due to the construction and, when finished,
it is possible to hide this by renaming the states. For example, we
can rename the states of the previous DFA in the following manner:
\(\{q_0\}\) into~\(A\), \(\{q_0, q_1\}\) in~\(B\) and \(\{q_0, q_2\}\)
in~\(C\). So the transition table changes:
\begin{equation*}
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & \{q_0\}     & \{q_0, q_1\} & \{q_0\}\\
                & \{q_0,q_1\} & \{q_0,q_1\}  & \{q_0,q_2\}\\
             \# & \{q_0,q_2\} & \{q_0,q_1\}  & \{q_0\}
  \end{array}
\qquad
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & A & B & A\\
                & B & B & C\\
             \# & C & B & A
  \end{array}
\end{equation*}
So, finally, the DFA is simply as in \fig~\vref{fig:dfa_from_nfa_simple}.

\subsection*{Optimisation}

Even if in the worst case the resulting DFA has an exponential number
of states of the corresponding NFA, it is in practice often possible
to avoid the construction of inaccessible states.
\begin{itemize*}

  \item The singleton containing the initial state (in our example,
  \(\{q_0\}\))~is accessible;

  \item let us assume we have a set~\(S\) of accessible states; then
    for each input symbol~\(a\), we compute \(\delta_D(S,a)\): this
    new set is also accessible;

  \item let us repeat the last step, starting with~\(\{q_0\}\), until
    no new (accessible) sets are found.

\end{itemize*}
Let us reconsider the NFA given by the transition table
\begin{equation*}
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{NFA} \; \mathcal{N}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & q_0 & \{q_0, q_1\} & \{q_0\}\\
                & q_1 & \varnothing  & \{q_2\}\\
    \#          & q_2 & \varnothing  & \varnothing
  \end{array}
\end{equation*}
Initially, the sole subset of accessible states is \(\{q_0\}\):
\begin{equation*}
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & \{q_0\} & \delta_N(q_0,0) & \delta_N(q_0,1)
  \end{array}
\quad\text{that is}\quad
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & \{q_0\} & \{q_0,q_1\} & \{q_0\}
  \end{array}
\end{equation*}
Therefore \(\{q_0,q_1\}\) and~\(\{q_0\}\) are accessible sets,
but~\(\{q_0\}\) is not a new set, so we only add to the table entries
\(\{q_0, q_1\}\) and compute the transitions from it:
\begin{equation*}
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & \{q_0\}      & \{q_0, q_1\} & \{q_0\}\\
                & \{q_0, q_1\} & \{q_0, q_1\} & \{q_0, q_2\}
  \end{array}
\end{equation*}
This step uncovered a new set of accessible states, \(\{q_0, q_2\}\),
which we add to the table and repeat the procedure, and mark it as
final state since \(q_2 \in \{q_0, q_2\}\):
\begin{equation*}
  \begin{array}{r@{}l||c|c}
    \multicolumn{2}{c||}{\text{DFA} \; \mathcal{D}} & 0 & 1\\
    \hhline{==::==}
    \rightarrow & \{q_0\}      & \{q_0, q_1\} & \{q_0\}\\
                & \{q_0, q_1\} & \{q_0, q_1\} & \{q_0, q_2\}\\
             \# & \{q_0, q_2\} & \{q_0, q_1\} & \{q_0\}
  \end{array}
\end{equation*}
We are done since there are no more new accessible sets.

\subsection*{Tries}

Lexical analysis tries to recognise a prefix of the input character
stream (in other words, the first lexeme of the given
program). Consider the C~keywords \term{const} and \term{continue}:
\begin{center}
\includegraphics[bb=48 675 353 730]{nfa_kwd}
\end{center}
This example shows that a NFA is much more comfortable than a DFA for
specifying tokens for lexical analysis: we design separately the
automata for each token and then merge their initial states into one,
leading to one, possibly large NFA. It is possible to apply the subset
construction to this NFA.

After forming the corresponding NFA as in the previous example, it is
actually easy to construct an equivalent DFA by sharing their
prefixes, hence obtaining a tree-like automaton called \emph{trie}
(pronounced as the word `try'):
\begin{center}
\includegraphics[bb=48 675 353 730]{trie_kwd}
\label{trie_kwd}
\end{center}
Note that this construction only works for a list of constant words,
like keywords.

This technique can easily be generalised for searching constant
strings (like keywords) in a text, that is, not only as a prefix of a
text, but \emph{at any position}. It suffices to add a loop on the
initial state for each possible input symbol. If we note~\(\Sigma\)
the language alphabet, we get
\begin{center}
\includegraphics[bb=48 675 355 738]{trie_kwd_search}
\end{center}
It is possible to apply the subset construction to this NFA or to use
it directly for searching the two keywords at any position in a
text. In case of direct use, the difference between this NFA and the
trie one page~\pageref{trie_kwd} is that there is no need here to
`restart' by hand the recognition process once a keyword has been
recognised: we just continue. This works because of the loop on the
initial state, which always allows a new start. (The reader may try
for instance the input \texttt{constantcontinue}.)

The subset construction can lead, in the worst case, to a number of
states which is the total number of state subsets of the NFA. In other
words, if the NFA has~\(n\) states, the equivalent DFA by subset
construction can have~\(2^n\) states (see
page~\pageref{state_explosion} for the count of all the subsets of a
finite set). For instance, consider the following NFA, which
recognises all binary strings which have~\(1\) at the \(n\)th
position from the end:
\begin{center}
\includegraphics[bb=48 711 305 758]{subset_bad_case}
\end{center}
The language recognised by this NFA is \(\Sigma^{*} 1 \Sigma^{n-1}\),
where \(\Sigma=\{0,1\}\), that is: all words of length greater or
equal to~\(n\) are accepted as long as the \(n\)th bit from the
\emph{end} is~\(1\). Therefore, in any equivalent DFA, all the
prefixes of length~\(n\) should not lead to a stuck state, because the
automaton must wait until the \emph{end} of the word to accept or
reject it. If the states reached by these prefixes are all different,
then there are at least \(2^n\)~states in the DFA. Equivalently (by
contraposition), if there are less than \(2^n\)~states, then some
states can be reached by several strings of length~\(n\):
\begin{center}
\includegraphics[bb=48 643 235 730]{dfa_bad_case}
\end{center}
where words \(x1w\) and \(x'0w\) have length~\(n\). Let us define the
DFA as follows: \(\mathcal{D} = (Q_D, \Sigma, \delta_D, q_D, F_D)\),
where \(q_D=\{q_0\}\). The extended transition function is noted
\(\hat{\delta}_D\) as usual. The situation of the previous picture can
be formally expressed as
\begin{gather}
\hat{\delta}_D (q_D, x1) = \hat{\delta}_D (q_D, x'0) = q, \label{ext:1}\\
\lvert{x1w}\lvert = \lvert{x'0w}\lvert = n,\notag
\end{gather}
where \(\lvert{u}\lvert\) is the length of~\(u\). Let~\(y\) be a any
string of~0 and~1 such that \(\lvert{wy}\lvert = n - 1\). Then
\(\hat{\delta}_D(q_D, x1wy) \in F_D\) since there is a~1 at the
\(n\)th position from the end:
\begin{center}
\includegraphics[bb=48 643 287 730]{dfa_bad_case_complete}
\end{center}
Also, \(\hat{\delta}_D(q_D,x'0wy) \not\in F_D\) because there is a 0
at the \(n\)th position from the end. On the other hand,
equation~\eqref{ext:1} implies
\begin{equation*}
\hat{\delta}_D (q_D, x1wy) = \hat{\delta}_D (q_D, x'0wy) = p.
\end{equation*}
So we stumble upon a contradiction because a state (\(p\)) must be
either final or not final, it cannot be both. As a consequence, we
must reject our initial assumption: there are at least \(2^n\)~states
in the equivalent DFA. This is a very bad case, even if it is not the
worst case (\(2^{n+1}\) states).


\section{On-policy Control with Approximation}
We consider attempts to solve the control problem using parametrised function approximation to estimate action-values. We consider only the on-policy case for now.

\subsection{Episodic Semi-gradient Control}
Extension of the semi-gradient update rules to action-values is straightforward
\begin{equation}
    \vec{w}_{t+1} = \vec{w}_t + \alpha \left[ U_t - \hat{q}(S_t, A_t, \vec{w}_t) \right] \grad_{\vec{w}_t} \hat{q}(S_t, A_t, \vec{w}_t)
\end{equation}
where $U_t$ is the update target at time $t$. For example, one-step Sarsa has the update target is
\[
    U_t = R_{t+1} + \gamma \hat{q}(S_{t+1}, A_{t+1}, \vec{w}_t).
\]
We call this method \emph{episodic semi-gradient one-step Sarsa}. For a constant policy, this method converges in the same with as TD(0), with a similar kind of error bound.\\

In order to form control methods, we must couple the prediction ideas developed in the previous chapter with methods for policy improvement. Policy improvement methods for continuous actions or actions from large discrete spaces are an active area of research, with no clear resolution. For actions drawn from smaller discrete sets, we can use the same idea as we have before, which is to compute action values and then take an $\varepsilon$-greedy action selection. Episodic semi-gradient sarsa can be used to estimate the optimal action-values as in the box below.\\

\includegraphics[width=\textwidth]{\NotesImages/episodic_semi_gradient_sarsa.png}\\

\subsection{Semi-gradient $n$-step Sarsa}
We can use an $n$-step version of the episodic Sarsa that we defined above by incorporating the bootstrapped $n$-step return
\begin{equation}
    G_{t:t+n} \doteq \sum_{i=1}^{n-1} \gamma^i R_{i+1} + \gamma^n \hat{q}(S_{t+n}, A_{t+n}, W_{t+n-1})
\end{equation}
where $G_{t:t+n} = G_{t}$ if $t + n \geq T$, as usual. This update target is used in the pseudocode in the box below. As we have seen before, performance is generally best with amn intermediate value of $n$. \\

\includegraphics[width=\textwidth]{\NotesImages/n_step_episodic_semi_gradient_sarsa.png}\\

\subsection{Average Reward: A New Problem Setting for Continuing Tasks}
We introduce a third classical setting for formulating the goal in Markov decision problems (MDPs) (to go along with episodic and continuing). This new setting is called the \emph{average reward setting}. This setting applies to continuing problems with no start or end state, but also no discounting. (Later we will see that the lack of a start state introduces a symmetry that makes discounting with function approximation pointless.)\\

In the average reward setting, the ordering of policies is (most often) defined with respect to the \emph{average reward}  while following the policy

\begin{align}
    r(\pi) &\doteq \lim_{h\to\infty} \frac1h \sum_{t=1}^{h} \E{}[R_{t} \vert{} S_0, A_{0:t-1} \sim \pi]\\
             &= \lim_{t\to\infty} \E{}[R_{t} \vert{} S_0, A_{0:t-1} \sim \pi] \\
             &= \sum_s \mu_\pi(s) \sum_a \pi(a \vert{} s) \sum_{s', r} p(s', r \vert{} s, a) r.
\end{align}

We will consider policies that attain the maximal value of $r(\pi)$ to be optimal (though there are apparently some subtly distinctions here that are not gone into).\\

The distribution $\mu_\pi(s)$ is the steady-state distribution defined by
\begin{equation}
    \mu_\pi(s) \doteq \lim_{t\to \infty} \P{} (S_t = s \vert{} A_{0:t-1} \sim \pi)
\end{equation}
which we assume to exist for any $\pi$ and to be independent of the starting state $S_0$. This assumption is known as \emph{ergodicity}, and it means that the long run expectation of being in a state depends only on the policy and MDP transition probabilities -- not on the start state. The steady-state distribution has the property that it is invariant under actions taken by $\pi$, in the sense that the following holds
\[
    \sum_s \mu_\pi(s) \sum_a \pi(a \vert{} s) p(s' \vert{} s, a) = \mu_\pi(s').
\]\\

In the average-reward setting we define returns in terms of the difference between the reward and the expected reward for the policy
\begin{equation}
    G_t \doteq \sum_{i \geq t} \left(R_{i+1} - r(\pi)\right)
\end{equation}
we call this quantity the \emph{differential return} and the corresponding value functions (defined in the same way, just with this return instead) \emph{differential value functions}. These new value functions also have Bellman equations:

\begin{align}
    v_\pi(s) &= \sum_a \pi(a \vert{} s) \sum_{s', r} p(s', r \vert{} s, a) \left[ r - r(\pi) + v_\pi(s')\right] \\
    q_\pi(s, a) &= \sum_{s', r} p(s', r \vert{} s, a) \left[ r - r(\pi) + \sum_{a'} \pi(a' \vert{} s') q_\pi(s', a')\right] \\
    v_*(s) &= \max_a \sum_{s', r} p(s', r \vert{} s, a) \left[ r - r(\pi) + v_*(s')\right] \\
    q_\pi(s, a) &= \sum_{s', r} p(s', r \vert{} s, a) \left[ r - r(\pi) + \max_{a'} q_*(s', a')\right].
\end{align}
We also have differential forms of the TD errors, where $\bar{R}_t$ is the estimate of $r(\pi)$ at $t$,
\begin{align}
    \delta_t &\doteq R_{t+1} - \bar{R}_{t+1} + \hat{v}(S_{t+1}, \vec{w}_t) - \hat{v}(S_t, \vec{w}_t)\\
    \delta_t &\doteq R_{t+1} - \bar{R}_{t+1} + \hat{q}(S_{t+1}, A_{t+1}, \vec{w}_t) - \hat{q}(S_t, A_t, \vec{w}_t).
\end{align}\\

Many of the previous algorithms and theoretical results carry over to this new setting without change. For instance, the update for the semi-gradient Sarsa is defined in the same way just with the new TD error, corresponding pseudocode given in the box below.\\

\includegraphics[width=\textwidth]{\NotesImages/differential_semi_gradient_sarsa.png}\\

\subsection{Deprecating the Discounted Setting}
Suppose we want to optimise the discounted value function $v_\pi^\gamma(s)$ over the on-policy distribution, we would choose an objective $J(\pi)$ with
\begin{align}
    J(\pi) &\doteq \sum_s \mu_\pi(s) v_\pi^\gamma(s) \\
           &= \sum_s \mu_\pi(s) \sum_a \pi(a \vert{} s) \sum_{s', r} p(s', r \vert{} s, a) \left[ r + \gamma v_\pi^\gamma(s')\right] \\
           &= r(\pi) + \sum_s \mu_\pi(s) \sum_a \pi(a \vert{} s) \sum_{s', r} p(s', r \vert{}  s, a) \gamma v_\pi^\gamma(s') \\
           &= r(\pi) + \gamma \sum_{s'} v_\pi^\gamma(s') \sum_{s} \mu_\pi(s) \sum_a \pi(a \vert{} s)p(s', \vert{}  s, a) \\
           &= r(\pi) + \gamma \sum_{s'} v_\pi^\gamma(s') \mu_\pi(s') \\
           &= r(\pi) + \gamma J(\pi) \\
           &\vdotswithin{=} \\
           &= \frac{1}{1 - \gamma} r(\pi)
\end{align}
so we may as well have optimised for the \emph{undiscounted} average reward.\\

The root cause (note: why \emph{root} cause?) of the difficulties with the discounted control setting is that when we introduce function approximation we lose the policy improvement theorem. This is because when we change the discounted value of one state, we are not guaranteed to have improved the policy in any useful sense (e.g. generalisation could ruin the policy elsewhere). This is an area of open research.

\subsection{Differential Semi-gradient $n$-step Sarsa}
We generalise $n$-step bootstrapping by introducing an $n$-step version of the TD error  in this new setting. In order to do that, we first introduce the differential $n$-step return using function approximation
\begin{equation}
    G_{t:t+n} \doteq \sum_{i=t}^{n-1} \left( R_{i +1} - \bar{R}_{i+1} \right) + \hat{q}(S_{t+n}, A_{t+n}, \vec{w}_{t+n-1})
\end{equation}
with $G_{t:t+n} = G_t$ if $t+n \geq T$ as usual and where $\bar{R}_i$ are the estimates of $\bar{R}$. The $n$-step TD error is then defined as before just with the new $n$-step return
\[
    \delta_t \doteq G_{t:t+n} - \hat{q}(S_t, A_t, \vec{w}_t).
\]
Pseudocode for the use of this return in the Sarsa framework is given in the box below. Note that $\bar{R}$ is updated using the TD error rather than the latest reward (see Exercise 10.9).\\

\includegraphics[width=\textwidth]{\NotesImages/differential_semi_gradient_n_step_sarsa.png}\\