\section{Introduction and motivation}

The models that we have studied until now are \textbf{deterministic} : given an input a specific output is provided. Now, we want to change the point of view and study \textbf{statistical models}, where we can deal with \emph{probabilities} of each output.

We will now introduce Hidden Markv Models (HMMs) from a intuitive point of view.\\

HMMs are probabilistic models that assume that the underlying system is a Markov process \(X\) with unobservable (or \emph{hidden}) states. We assume that there is another \textbf{observable} process \(Y\) whose outcomes we assume are influenced by the process \(X\). Our \textbf{goal} is to determine the process \(X\) by observing \(Y\). In \textbf{real world]} signals, we have observed outputs which we will want to model, so we can apply this paradigmn to our problem.

\begin{ndef}
A \textbf{latent variable} is a variable that is not directly observed but is rather infered from observed variables. If a model aims to explain observed variables in terms of latent variables is a \textbf{latent variable models}.
\end{ndef}

HMMs are a example of latent variable models.

\begin{example}
Let us set in the speech recognition problem. In this topic we have the following features:

\begin{itemize}
  \item The \emph{latent states} are the different symbols existing in the language.
  \item The observations are the features extracted from the voice (parameterization). Example: Linear Prediction coefficients of a short term fragment of speech (LPCC).
\end{itemize}
In this case, the goal would be to obtain the latent language symbols using the speech segment.

\begin{note}
  The parameterization is also called feature extraction. In the speech recognition setting, it is usual to transform each audio window in a \(\mathbb R^n\) vector, using the same \(n \in \mathbb N\) for each window. The coefficients of an LPC filter or a MFCC filter are examples of extracted features.
\end{note}
\end{example}

\subsection{HMMs for Speech recognition}

One particular way of modeling speech is by using HMMs. In this case, we assume that the speech sequence is generated by a Markov model as the one shown in Figure \ref{fig:hmm:speech},

\begin{figure}[H]
\centering
\includegraphics[scale=0.3]{Figures/ExampleSpeechHmm}
\caption{HMM Speech model.}
\label{fig:hmm:speech}
\end{figure}

This finite state machine changes state every time unit and each time \(t\) that we jump from state \(i\) to state \(j\), a speech vector \(\mathbf{o}_t\) is generated from the p.d.f. \(b_j(\mathbf{o}_t)\). Furthermore, the transition from state \(i\) to state \(j\) is also probabilistic and it is modeled by the discrete probability \(a_{ij}\).

\begin{ndef}
A HMM is \textbf{ergodic} if \(a_{ij} > 0\) for all \(i,j\).
\end{ndef}

\begin{example}
An example of \textbf{left to right} HMM for speech modeling is called \emph{Bakis}. The example of HMM represented in Figure \ref{fig:hmm:speech} is a left-to-right HMM. In these models, we can not go backwards (typical in temporal information processing) and a phone is typically represented using between \(3\) and \(5\) states.
\end{example}

\subsection{Gaussian Mixture Models as HMMs}

\textbf{Gaussian Mixture Models (GMMs)} tries to fit a multivariate normal vector to a set of data. For each state of the HMM, we can define a GMM by its mean vector, its covariance vector and the weights of each mixture. That is, for a state \(p\),
\[
\lambda_p = \{\mu_{ip},\Sigma_{ip},\omega_{ip}\},
\]
where \(\omega_{ip}\) are the weights.\\

If we have a dataset and we assume that it is gaussian, it is easy to learn its meand and variance. However, learning the whole mixture is not so simple, since we \textbf{we do not know} to which mixture does each point belong to. \\

With this in mind, our problem is \textbf{the training}. To train, we need:

\begin{enumerate}
  \item Associate observations to HMM states (\textbf{state alignment}).
  \item A \emph{score} function that evaluates the fit of a model.
\end{enumerate}

After fitting the model, we will be able to generate observations from the model or recognise a non-labeled speech sentence.

\begin{example}
This example is again related to the speech recognition topic. Consider that we want to \textbf{recognize isolated digits} that have been spoken out ("one", "two",...).
\begin{enumerate}
\item To \textbf{train} we would need labeled observations, and we would use them to train a GMM-HMM for each digit, using all digit observations
\item To \textbf{recognize} a digit, having a sequence of observations that are not labeled, we would compare this sequence with the trained HMMs and choose the one that fits the best the pronounced digit.
\end{enumerate}
\end{example}

\section{HMMs: Formal definition}

Let us make use of the previous intuitions in order to give a formal definition of HMMs. The elements of an HMM are:

\begin{itemize}
\item We will consider that we have \(N\) \textbf{states}, \(\{S_1,\dots,S_N\}\). At time \(t\), we will be at state \(q_t\), which is a priori unknown. If we know that at time \(t\) the state is at state \(S_i\), we will write \(q_t = S_i\).

\item We will have \(M\) possible \textbf{observable symbols}, \(\{v_1,\dots,v_M\}\). We denote the observation at time \(t\) as \(O_t\), which is also a priori unknown. If we know that at time \(t\) the observation is \(v_j\), we write \(O_t = v_j\)

\item A \textbf{transition matrix}, \(A = \{a_{ij}\}\), which has the probability of transitioning from state \(i\) to state \(j\), that is:
\[
a_{ij} = P[q_{t+1} = S_j \ | \ q_t = S_i], \quad 1 \leq i,j \leq N.
\]

\item We have a \textbf{likelihood} \(B = \{b_j(k)\}\) of a observation \(v_j\) happening in each state \(S_i\), that is:
\[
b_j(k) = P\left[ v_k \ \text{at} \ t \ | \ q_t = S_j\right], \quad 1 \leq j \leq N, 1 \leq k \leq M.
\]

\item An \textbf{a priori} distribution of each state \(\pi = \{\pi_i\}\), where
\[
\pi_i = P \left[q_1 = S_i\right], \quad  1 \leq i \leq N.
\]
\end{itemize}

\begin{ndef}
A \textbf{Hidden Markov Model} is defined by its parameters \(\lambda = (A,B,\pi)\).
\end{ndef}

HMMs can be seen as \textbf{generative models}. Consider the following \textbf{algorithm}:

\begin{enumerate}
\item Choose an initial state \(q_1 = S_i\), following the a priori state distribution \(\pi\).
\item Increment the time \(t\), making a jump.
\item Choose an observation \(O_t = v_k\) using the likelihood distribution \(b_i\) of the observations at the state \(S_i\).
\item Jump to a new state \(q_{t+1}\) using the probability of jumping from state \(i\) to state \(j\), \(a_{ij}\).
\item Increment the time \(t = t+1\) and return to step 3 if more more observations are wanted.
\end{enumerate}

We want to solve a few problems using HMMs. We will study them one by one:

\subsection{Scoring problem}

The problem can be stated as:
\begin{quote}
Given a sequence of observations \(O = O_1,\dots,O_T\) and a HMM model \(\lambda = (A,B,\pi)\), how do we compute the likelihood of observing that sequence \(P(O| \lambda)\)
\end{quote}

\subsubsection{Analitical solution}

Considering the given HMM \(\lambda = (A,B,\pi)\), we want to compute \(P(O|\lambda)\) where \(O = O_1\dots O_T\). We consider that \textbf{a sequence of states has occurred}. Let the sequence of states be
\[
Q = q_1 \dots q_T,
\]
which is unknown. Assuming \textbf{independent observations}, we have that
\[
P(O|Q,\lambda) = \prod_{t=1}^T P(O_t  \  |  \  q_t, \lambda) = \prod_{t=1}^T b_{q_t}(O_t).
\]
Also, using the conditional probability we have that:
\[
P(O  \  |  \  Q,\lambda) = \frac{P(O,Q|\lambda)}{P(Q \ | \ \lambda)} \implies P(O,Q | \lambda) = P(O \ | \ Q,\lambda)P(Q \ | \ \lambda).
\]
Now, since we have a Markov model, the probability of jumping to a state only depends on the actual state, so
\[
P(Q \ | \ \lambda) = \pi_{q_1} a_{q_1 q_2}\dots a_{q_{T-1}q_T}.
\]
Lastly, since the probability \(P(O|\lambda)\) can be obtaining summing the last probability in all possible state sequences, we have that
\begin{align*}
P(O|\lambda) & = \sum_{\text{all } Q} P(O|Q,\lambda) P(Q|\lambda)\\
& = \sum_{q_1,\dots,q_T} \pi_{q_1}b_{q_1}(O_1) \  a_{q_1 q_2}b_{q_2}(O_2)\dots a_{q_{T-1}q_T}b_{q_T}(Q_T)
\end{align*}

This is the analitical solution of the problem. However, this solution has a clear \textbf{problem}: it is very computationally expensive so it is not escalable.

\subsubsection{Forward/Backward Algorithm}

We will define an algorithm that is more computationally efficient to find the solution of the Scoring problem.

Consider the probability of observing the sequence \(O = O_1\dots O_t\) until instant \(t\) and be at state \(S_i\) in instant \(t\). That is called the \textbf{forward variable} and it can be written as:
\[
\alpha_t(i) = P(O_1\dots O_t, q_t = S_i| \lambda).
\]

This forward variable can be computed using the following \textbf{algorithm}:

\begin{enumerate}
\item Initialization:
\[
\alpha_1(i) = \pi_i b_i(O_1), \quad 1 \leq i \leq N.
\]
\item Induction:
\[
\alpha_{t+1}(j) = \left[\sum_{i = 1}^N \alpha_t(i)a_{ij}\right]b_j(O_{t+1}).
\]
\item Termination:
\[
P(O|\lambda) = \sum_{i = 1}^N \alpha_T(i).
\]
\end{enumerate}

As we can see, we are computing the forward variable \(\alpha_{t+1}\) using the previous forward variables \(\alpha_t\) and the probabilities of observation and transition.\\

Remember that the state \(S_j\) can be reached at time \(t+1\) from \(N\) different states \(S_i\), \(1 \leq i \leq N\) at time \(t\). Now, since \(\alpha_t(i)\) is the probability of the join of the observations \(O_1\dots O_t\) and the state at time \(t\) is \(S_i\),
\textbf{the product} \(\alpha_t(i)a_{ij}\) is the probability of the join event that \(O_1\dots O_t\) are observed, the state \(S_j\) is reached at time \(t+1\) is reached from the state \(S_i\) at time \(t\). \\

When summing this product over all possible states \(S_i\), \(1\leq i \leq N\), results in the probability of \(S_j\) at time \(t+1\) with all the previous partial observation. This way we obtain \(S_j\).\\

Now that we know \(S_j\), we obtain \(\alpha_{t+1}(j)\) by multiplying the summed quantity by the probability \(b_j(O_{t+1})\), and this is done \textbf{for all states j}, \(1 \leq j \leq N\) for a given \(t\).\\

With this algorithm, we can only compute \textbf{the probability of an observation given the model}. We can alternatively use the \textbf{backward variable}, that represent the probability of observing the partial sequence \(O_{t+1}\dots O_T\) from instant \(t+1\) and being in the state \(S_i\) at instant \(t\)
\[
\beta_t(i) = P(O_{t+1}\dots O_T|q_t = S_i , \lambda).
\]

We can compute \(\beta_T(i)\) inductively as follows:
\begin{enumerate}
\item Initialization:
\[
\beta_T(i) = 1, \quad 1\leq i \leq N.
\]
This initialization is arbitrary.

\item Induction:
\[
\beta_t(i) =  \sum_{j=1}^N a_{ij}b_j(O_{t+1})\beta_{t+1}(j),
\]
with \(t = T-1,T_2, \dots, 1\), \(1 \leq i \leq N\).
\end{enumerate}

\subsection{State recognition problem}

The problem that we will now treat is the following one:

\begin{quote}
Given a sequence of observations \(O = O_1\dots O_T\) and a HMM model \(\lambda\), how do we choose a state sequence \(Q = q_1\dots q_T\) which is optimal in some meaninful sense? (that is, best explains the observations).
\end{quote}

As we have mentioned, we have to way of measuring how optimal is a state sequence. We have a few possibilities, let us present them.

\begin{itemize}
\item Choosing the states \(q_t\) which are \textbf{individually most likely}. This criterion maximizes the \textbf{expected number} of correct individual states. To obtain this, we define \(\gamma_t(i)\) as the probability of being in state \(S_i\) at time \(t\), given the observation sequence \(O\) and the model \(\lambda\). That is:
\[
\gamma_t(i) = P(q_t = S_i \ | \ O,\lambda).
\]
\end{itemize}


\newcommand*{\yts}{y_t^\ast}
\newcommand*{\ets}{\varepsilon_t^\ast}

\section{Model}

The Stochastic Volatility model was introduced in the seminal work of~\citet{Taylor1982}.
By choosing SV, one aims at capturing time varying and seasonal volatility using an AR(1) process.
The model used in this thesis is the Stochastic Volatility with leverage, which, additional to the AR(1) process, also models the leverage effect by letting the stock return and the increment of the log variance have a constant correlation.

\subsection{Formulation}

The SV model with leverage is, as formulated in~\citet{Omori2007},
\begin{equation}
\begin{alignedat}{2}\label{form:orig_model}
y_t & = \varepsilon_t\exp\left(h_t/2\right), && \quad t=1,\dots,T, \\
h_{t+1} & = \mu+\phi(h_t-\mu)+\eta_t, && \quad t=1,\dots,T-1, \\
\begin{pmatrix}
\varepsilon_t \\
\eta_t
\end{pmatrix}
\bigg\vert\left(\rho,\sigma\right) & \sim\text{ i.i.d. }\mathcal{N}_2\left(\bm{0},\bm{\Sigma}\right), && \quad t=1,\dots,T-1, \\
\varepsilon_T &\sim\mathcal{N}(0,1), \\
\bm{\Sigma} & =
\begin{pmatrix}
1 & \rho\sigma \\
\rho\sigma & \sigma^2
\end{pmatrix},
\end{alignedat}
\end{equation}
where $T$ is the number of time points, the only observed variables are the demeaned log returns, $y_t$.
The return at time $t$ is thus conditionally normally distributed, given $h_t$.
The log variance, $\bm{h}$, is a latent vector, and it constitutes an AR(1) process with mean $\mu$, persistence $\phi$ and variance $\sigma^2$.
Leverage is the fourth parameter, $\rho$, which is the correlation between $\varepsilon_t$ and $\eta_t$, i.e.\ the increment of the stock price and the increment of the log variance.

The first equation in~\eqref{form:orig_model} is not linear in $h_t$, which makes the model difficult to estimate. For the ease of notation, let
\begin{align*}
\yts &=\log(y^2_t), \\
d_t &=I(y_t\ge0)-I(y_t<0), \qquad\text{$y_t$'s sign,} \\
\ets &=\log(\varepsilon^2_t),
\end{align*}
thus knowing $y_t$ is equivalent to knowing the pair $(\yts, d_t)$\footnote{Except for the case $\{y_t=0\}$, which is a null set in the model, and it causes identifiability issues for $h_t$. In practice, $\yts =\log(y^2_t+\epsilon)$ is used with some small $\epsilon$ for robustness.}. By storing $d_t$ and applying $x\mapsto\log(x^2)$ to the first equation of~\eqref{form:orig_model} we get the linearised form,
\begin{align}
\begin{split}\label{form:lin_model}
\yts & = h_t+\ets, \\
h_{t+1} & = \mu+\phi(h_t-\mu)+\eta_t,
\end{split}
\end{align}
where the error term of the first equation has a $\log(\chi_1^2)$ distribution. The observed variable is $\yts$ and $h_t$ is the latent state.

\subsubsection*{Other forms}

The SV model with leverage was formulated differently in~\citet{Jacquier2004}, where $\varepsilon_t$ and $\eta_{t-1}$ are correlated.
A comparison provided in~\citet{yu2005leverage} revealed that model~\eqref{form:orig_model} is more attractive as it is an Euler approximation to the log-normal Ornstein--Uhlenbeck process.
Hence, the method that fits~\eqref{form:orig_model} also fits the corresponding continuous time process with discretely sampled data.
Moreover, in the alternative specification, $y_t$ is not a martingale difference sequence, and $\rho$ has two roles: leverage and the skewness of $y_t\mid y_1,\dots,y_{t-1}$, which makes it more difficult to interpret its value.
Finally, an empirical comparison also showed the model by~\citet{Jacquier2004} to be inferior to~\eqref{form:orig_model}.
For these reasons, we use~\eqref{form:orig_model} in the paper at hand.

\subsection{Estimation without leverage}

SV models are an attractive alternative to GARCH type models, the main difference\footnote{For a more in-depth comparison see~\citet{Harvey1994}.} being that while the volatility of GARCH at $t+1$ is conditionally deterministic, given the information known at $t$, it is random in SV.
On the one hand, this lets SV fit the data better in some cases~\citep{Kim1998,kastner2016dealing,Chan2016}, on the other hand, it makes its estimation more difficult. In the following parts, the fitting methods for SV without leverage considered in the literature are briefly summarised.

\subsubsection{Maximum likelihood estimation}

Let $\bm{y}=(y_1,\dots,y_n)$.
In order to obtain an ML estimate for $(\phi,\sigma^2,\rho,\mu)$, we need to evaluate the likelihood function $\ell(\phi,\sigma^2,\rho,\mu\mid\bm{y})$, for which we need to integrate over the space of vector $\bm{h}$.
This is unfortunately difficult due to the non-linear dependence between $y_t$ and $h_t$, or, in the linear form, due to the non-Gaussian error term $\ets$.

The issue of non-normality was resolved in~\citet{Harvey1994} using a Gaussian approximation to $\ets$, i.e.\ by matching the first two moments of the $\log(\chi_1^2)$ distribution.
In the resulting approximate model, a quasi-maximum likelihood estimate can be obtained by maximising a so-called quasi-likelihood function.
That function is the result of the application of the Kalman filter that integrates over the vector $\bm{h}$.
This estimator is consistent and asymptotically normally distributed, but it has bad performance in small samples because the $\ets$ is poorly approximated by the normal distribution~\citep{Kim1998}.

\subsubsection{Bayesian approach}

The lack of a closed form likelihood function also means that there are no closed form posteriors for the model.
This suggests the usage of Markov chain Monte Carlo algorithms (MCMC), which, with the help of Markov chains and Bayes' theorem, make it possible to draw samples from the posterior distribution of the latent variables and the parameters.
With enough such samples we get a picture of these distributions.
For an introduction, see, e.g.,~\citet{Geyer2011}.

In~\citet{Kim1998}, two different Bayesian ideas were compared for SV without leverage based on how the latent variables are sampled.
A single move (one-at-a-time volatility update) sampler was introduced first. It draws $h_t$ from $h_t\mid\bm{h}_{-t},\bm{y},\phi,\sigma^2,\mu$ one by one, where $\bm{h}_{-t}$ is $\bm{h}$ excluding $h_t$.
Due to the high intercorrelation in $\bm{h}$, slow convergence and poor mixing characterise this approach even though the algorithm used by~\citet{Kim1998} performs better than the other ones in the literature~\citep{shephard1993fitting,jacquier2002bayesian,shephard1994comment,shephard1997likelihood,geweke1994bayesian}.

To avoid the issues with high intercorrelation in $\bm{h}$, a multi-move sampler was used that draws $\bm{h}$ from $\bm{h}\mid\bm{y},\phi,\sigma^2,\mu$ at once.
By approximating the marginal distribution of $\ets$ with a $K=7$ component mixture of normals,~\citet{Kim1998} managed to reduce the task to the known framework of conditionally Gaussian state spaces\footnote{For an introduction see~\citet{Kitagawa1996}.}.
Since the marginal of $\ets$ does not include any model-dependent values, this mixture of normals can be specified before fitting the model.
The approximation errors to the original SV model can be corrected for by a reweighting scheme.
However, this correction is known to have minor influence due to the good choice of the mixture approximation~\citep{Kim1998}.

\subsection{Approximate model}

Both the single move and the multi-move samplers were generalised to SV with leverage in~\citet{Omori2007}, and a particle filtering\footnote{For an introduction see~\citet{Johannes2009}.} method was also derived.
In this work, we favor MCMC methods over sequential Monte Carlo (particle filtering) due to the availability of computers with a multitude of strong processors.
Finally, due to its better sampling efficiency, the multi-move sampler was chosen over the single move sampler for fitting the model.

\subsubsection{Bivariate normal approximation}

\begin{table}[t!]
	\centering
	\begin{tabular}{cccccc}
		$j$                       & $p_j$    & $m_j$      & $v_j^2$ & $a_j$    & $b_j$    \\ \hline
		\multicolumn{1}{l|}{1}  & 0.00609 & 1.92677   & 0.11265                & 1.01418 & 0.50710 \\
		\multicolumn{1}{l|}{2}  & 0.04775 & 1.34744   & 0.17788                & 1.02248 & 0.51124 \\
		\multicolumn{1}{l|}{3}  & 0.13057 & 0.73504   & 0.26768                & 1.03403 & 0.51701 \\
		\multicolumn{1}{l|}{4}  & 0.20674 & 0.02266   & 0.40611                & 1.05207 & 0.52604 \\
		\multicolumn{1}{l|}{5}  & 0.22715 & -0.85173  & 0.62699                & 1.08153 & 0.54076 \\
		\multicolumn{1}{l|}{6}  & 0.18842 & -1.97278  & 0.98583                & 1.13114 & 0.56557 \\
		\multicolumn{1}{l|}{7}  & 0.12047 & -3.46788  & 1.57469                & 1.21754 & 0.60877 \\
		\multicolumn{1}{l|}{8}  & 0.05591 & -5.55246  & 2.54498                & 1.37454 & 0.68728 \\
		\multicolumn{1}{l|}{9}  & 0.01575 & -8.68384  & 4.16591                & 1.68327 & 0.84163 \\
		\multicolumn{1}{l|}{10} & 0.00115 & -14.65000 & 7.33342                & 2.50097 & 1.25049
	\end{tabular}
	\caption{Constants of the bivariate approximation~\citep{Omori2007}.}
	\label{tab:constants}
\end{table}

Due to the correlation between $\varepsilon_t$ and $\eta_t$, approximating $\ets$ affects $\eta_t$ as well.
Thus,~\citet{Omori2007} used a mixture of bivariate normals as an approximation to the conditional distribution of the pair $(\ets, \eta_t)$.
Let $(\xi_t,\nu_t\mid d_t,\rho,\sigma)$ denote the approximate random variable to $(\ets,\eta_t\mid d_t,\rho,\sigma)$, and let $\pi(X)$ denote the density of the random variable $X$ and $\pi(X=x)$ denote its value at $x$.

In order to derive the approximation, the bivariate conditional density is decomposed first, and then the parts are separately approximated,
\begin{align}
\pi(\ets,\eta_t\mid d_t,\rho,\sigma) &= \pi(\ets\mid d_t,\rho,\sigma) \pi(\eta_t\mid\ets,d_t,\rho,\sigma)\nonumber \\
&= \pi(\ets) \pi(\eta_t\mid\ets,d_t,\rho,\sigma)\label{eq:decomp},
\end{align}
because the marginal of $\ets$ is independent of $d_t$, $\rho$, and $\sigma$.
\citet{Omori2007} now used an improved normal mixture approximation to the marginal $\pi(\ets)$ with $K=10$,
\begin{equation}\label{eq:ets}
\pi(\xi_t)\triangleq\sum_{j=1}^{10}p_j\pi\left(\mathcal{N}\left(m_j,v_j^2\right)\right),
\end{equation}
where $\pi\left(\mathcal{N}\left(m_j,v_j^2\right)\right)$ denotes the normal density with mean $m_j$ and variance $v_j^2$.
The constants $m_j$, $p_j$, and $v_j$ were found by matching the first four moments of $\ets$ and $\exp(\ets)$ with the moments of $\xi_t$ and $\exp(\xi_t)$, and then applying a non-linear optimiser to minimise the distance of the densities~\citep{Kim1998}.
The constants are specified in Table~\ref{tab:constants}.

The conditional distribution of $\eta_t$ is
\begin{equation*}
\eta_t\mid\ets,d_t,\rho,\sigma\sim\mathcal{N}\left(d_t\rho\sigma\exp(\ets/2),\sigma^2\left(1-\rho^2\right)\right),
\end{equation*}
thus, we could use
\begin{equation}\label{eq:eta}
\nu_t\mid\xi_t,d_t,\rho,\sigma\sim\mathcal{N}\left(d_t\rho\sigma\exp(\xi_t/2),\sigma^2\left(1-\rho^2\right)\right),
\end{equation}
but the term $\exp(\xi_t/2)$ introduces difficulties.
These are mitigated by a linear approximation
\begin{equation}\label{eq:etslinear}
\exp(\xi_t/2)\approx\exp(m_j/2)(a_j+b_j(\xi_t-m_j))
\end{equation}
when the $j$th mixture component is used for $\xi_t$, i.e.\ $\mathcal{N}(m_j,v_j^2)$.
The constants $a_j$ and $b_j$ are the results of the mean square norm minimisation,
\begin{equation*}
\E{\left[\exp(\xi_t/2)-\exp(m_j/2)(a+b(\xi_t-m_j))\right]^2}
\end{equation*}
w.r.t.\ $a$ and $b$, separately for each $j$, and they are listed in Table~\ref{tab:constants}.

By combining~\eqref{eq:decomp},~\eqref{eq:ets},~\eqref{eq:eta}, and~\eqref{eq:etslinear}, the final approximation is
\begin{align*}
\pi(\ets,\eta_t\mid d_t,\rho,\sigma) &\approx \pi(\xi_t,\nu_t\mid d_t,\rho,\sigma) \\
&= \pi(\xi_t)\pi(\nu_t\mid\xi_t,d_t,\rho,\sigma) \\
&= \sum_{j=1}^{10}p_j\pi\left(\mathcal{N}\left(m_j,v_j^2\right)\right)\times \\
&\times\pi\left(\mathcal{N}\left(d_t\rho\sigma\exp(m_j/2)(a_j+b_j(\xi_t-m_j)),\sigma^2\left(1-\rho^2\right)\right)\right).
\end{align*}

\subsubsection[State space form]{Conditional Gaussian state space form}

Due to the normal mixture approximation, a new variable $\bm s=(s_1,\dots,s_T)$ is included in the model, the vector of mixture components. There is one component $s_t\in\{1,\dots,K\}$ for each time point $t$. Given $s_t=j$, linear model with normal errors is obtained,
\begin{equation}
\begin{alignedat}{2}\label{form:appr_model}
\yts & = h_t+\xi_t, && \quad t=1,\dots,T, \\
h_{t+1} & = \mu+\phi(h_t-\mu)+\nu_t, && \quad t=1,\dots,T-1, \\
\begin{pmatrix}
\xi_t \\
\nu_t
\end{pmatrix} &\overset{d}{=}
\begin{pmatrix}
m_j \\
a_j\gamma_t^j
\end{pmatrix} +
\begin{pmatrix}
v_j && 0 \\
b_jv_j\gamma_t^j && \sigma\sqrt{1-\rho^2}
\end{pmatrix}
\begin{pmatrix}
z_t^1 \\
z_t^2
\end{pmatrix}, && \quad t=1,\dots,T-1, \\
\xi_T &\overset{d}{=} m_j+v_jz_T^1, \\
\begin{pmatrix}
z_t^1 \\
z_t^2
\end{pmatrix}
&\sim\text{ i.i.d. }\mathcal{N}_2\left(\bm{0},\bm{I_2}\right), && \quad t=1,\dots,T,
\end{alignedat}
\end{equation}
where $\gamma_t^j\triangleq d_t\rho\sigma\exp(m_j/2)$ and ``$\overset{d}{=}$'' means equivalence in distribution. Note that $(\xi_t,\nu_t)$ depends on $d_t,\rho,\sigma$, and $s_t$.

In order to reduce the estimation of model~\eqref{form:appr_model} to the estimation of a well-known framework, the first two equations of~\eqref{form:appr_model} are reformulated equivalently as
\begin{equation}
\begin{alignedat}{2}\label{form:gauss_model}
\begin{pmatrix}
\yts \\
h_{t+1} \\
\tilde{\mu}_{t+1}
\end{pmatrix} &=
\begin{pmatrix}
h_t \\
\tilde{\mu}_t+\phi(h_t-\tilde{\mu}_t) \\
\tilde \mu_t
\end{pmatrix} +
\begin{pmatrix}
\xi_t \\
\nu_t \\
0
\end{pmatrix}, \quad t=1,\dots,T-1, \\
y_T^\ast &= h_T+\xi_T.
\end{alignedat}
\end{equation}
The error $(\xi_t,\nu_t,0)$ is a (degenerate) normal white-noise series. Hence, by assuming a Gaussian prior for $\left(h_1,\tilde\mu_1\right)$, model~\eqref{form:gauss_model} becomes a linear Gaussian state space (GSS) with hidden state $(h_t,\tilde{\mu}_t)$.
We copy the priors of the initial latent state used by~\citet{Omori2007}, for arbitrary constants $\mu_0$ and $\sigma_0$,
\begin{equation*}
\begin{pmatrix}
h_1 \\
\tilde\mu_1
\end{pmatrix} \sim
\mathcal{N}\left(
\begin{pmatrix}
\mu_0 \\
\mu_0
\end{pmatrix},
\begin{pmatrix}
\sigma^2/(1-\phi^2)+\sigma_0^2 & \sigma_0^2 \\
\sigma_0^2 & \sigma_0^2
\end{pmatrix}
\right).
\end{equation*}
Note that $\tilde{\mu}_t$ is constant in~\eqref{form:gauss_model} through $t$, and $\mu\equiv\tilde{\mu}_t$ is used to obtain $\mu$ from~\eqref{form:gauss_model}.
This ``trick'', the inclusion of $\mu$ in the hidden state, was the key in~\citet{Omori2007} for showing that~\eqref{form:gauss_model} is a special case of the model specified by~\citet{de1995simulation}.
The algorithm used by~\citet{Omori2007} and by the paper at hand is also heavily based on the algorithm defined by~\citet{de1995simulation}.

\subsubsection{Correcting for misspecification}\label{sec:reweight}

Let $\theta$ denote $(\sigma,\rho,\phi)$.
By using an approximate distribution for the true $\pi(\ets,\eta_t\mid d_t,\rho,\sigma)$, the model is misspecified, and the draws of $\bm{h}$, $\theta$, and $\mu$ are from an approximate posterior density as well.
This can be corrected, and draws can be produced from the true posterior $\pi(\bm{h},\theta,\mu\mid\bm{y})$.
First, the weights $w_k$ need to be calculated for all the draws $k=1,\dots,N$, and then the original sample needs to be resampled using $w_k$ as probabilities.
After obtaining the error terms
\begin{align*}
\xi_t^k &= y_t^\ast-h_t^k, \\
\nu_t^k &= (h_{t+1}^k-\mu^k)-\phi^k(h_t^k-\mu^k),
\end{align*}
the non-normalised weights are computed,
\begin{equation*}
w_k^\ast=\prod_{t=1}^{T}\frac{\pi\left(\ets=\xi_t^k,\eta_t=\nu_t^k\mid d_t,\mu=\mu^k,\theta=\theta^k\right)}{\pi\left(\xi_t=\xi_t^k,\nu_t=\nu_t^k\mid d_t,\mu=\mu^k,\theta=\theta^k\right)},
\end{equation*}
Finally, we normalise the weights
\begin{equation*}
w_k=\frac{w_k^\ast}{\sum_{l=1}^{T}w_l^\ast}
\end{equation*}
to get the probabilities that we use for resampling the existing draws.
\citet{Omori2007} found that the weights have quite small variance, which makes the effect of this correction procedure modest.
That is in line with the demonstrated high precision of the approximate distribution~\citep{Omori2007}.

\subsection{Estimation with leverage}\label{sec:estimlev}

\comment{
\subsubsection[MCMC algorithms]{Markov Chain Monte Carlo algorithms}

The term Monte Carlo is used for the simulation of random processes, while the term Markov chain denotes sequences of random variables $X_1,X_2,\dots$ for which, for each $n\in\mathbb{N}$, the conditional distribution of $X_{n+1}$ given $X_1,\dots,X_n$ only depends on $X_n$ (it is ``memoryless'').
This way, the value of $X_n$ can be thought of as the state of the chain at time $n$, and then $f_{X_{n+1}\mid X_n}(v\mid u)$ is the probability or density of transition from state $u$ to state $v$ at time $n$.
Under sufficient conditions, the states of a Markov chain converge in distribution to $\pi$, called the equilibrium distribution, from every initial state having positive density~\citep{grinstead2012introduction}.

In practice, we cannot prove the conditions for the existence of $\pi$.
Instead we only have the output of the algorithm, a sequence of realisations, from which we can try to imply that we have a converged chain using, e.g., visualisations and autocorrelation functions.
\citet{Geyer2011} mentions the issues that pop up here and argues about some solution ideas.

	These transition probabilities are called stationary if they don't depend on $n$.

	For the majority of the Markov chain Monte Carlo (MCMC) algorithms stationary transition probability distributions are needed.
	These are easier to handle, e.g. the joint distribution of such a Markov chain is characterised by the initial distribution of $X_1$ and on the general transition distribution from $X_n$ to $X_{n+1}$.

	The sequence $X_1,X_2,\dots$ itself is called stationary if for each $k\in\mathbb{N}$ and $n\in\mathbb{N}$ the joint distribution of the tuple $(X_n,\dots,X_{n+k})$ is independent of $n$.}

\subsubsection{Steps overview}

After initialising all the latent variables and parameters, there are three main steps when doing a Bayesian estimation of a conditional linear GSS.
\begin{enumerate}[start=0]
	\item Initialise $\bm h,\bm s,\theta,\mu$,
	\item Draw $\bm{s}\mid\bm{y}^\ast,\bm{d},\bm{h},\mu,\theta$\label{enum:draw-s},
	\item Draw $\theta,\bm h,\mu\mid\bm y^\ast,\bm d,\bm s$\label{enum:draw-other},
	\begin{enumerate}
		\item Draw $\theta\mid\bm{y}^\ast,\bm{d},\bm s$\label{enum:draw-theta},
		\item Draw $\bm{h},\mu\mid\bm{y}^\ast,\bm{d},\theta,\bm s$\label{enum:draw-latent},
	\end{enumerate}
	\item Goto Step~\ref{enum:draw-s}.
\end{enumerate}
Steps~\ref{enum:draw-s} and~\ref{enum:draw-other} are interchangeable in the algorithm, but steps~\ref{enum:draw-theta} and~\ref{enum:draw-latent} are not because the former's by-products are needed for the latter.
The steps are detailed below based on~\citet{Omori2007}.

If the Markov chain specified by the algorithm above has an equilibrium distribution, then it is an approximation to the posterior $\bm h,\bm s,\theta,\mu\mid\bm y$, and it is the true one after the reweighting scheme (Section~\ref{sec:reweight}).
So by repeating the steps sufficiently many times, after ``reaching convergence'', draws from the posterior are obtained.
With such a sample any point estimate, credible intervals, or transformations of the distribution can be estimated.

\subsubsection{Drawing the mixture states}

A closed form can be derived for $\pi(s_t=j\mid\bm{y},\bm{h},\mu,\sigma,\rho,\phi)$, $j=1,\dots,K$.
Let
\begin{align*}
\hat\xi_t &\triangleq y_t^\ast-h_t, \\
\hat{\nu}_t &\triangleq h_{t+1}-\mu-\phi(h_t-\mu), \\
\gamma_t^j &\triangleq d_t\rho\sigma\exp(m_j/2).
\end{align*}
Then, based on~\citet{Omori2007} with $K=10$, for $t=1,\dots,T-1$ and $j=1,\dots,10$,
\begin{equation}
\begin{aligned}[1]
\pi\left(s_t=j\mid\bm{y},\bm{h},\mu,\theta\right) &\propto P\left(s_t=j\right)\pi\left(\xi_t=\hat\xi_t\mid s_t=j\right)\times \\
&\times\pi\left(\nu_t=\hat\nu_t\mid\xi_t=\hat\xi_t,s_t=j,h_{t+1},h_t,\mu,\theta\right), \\
\pi\left(s_T=j\mid\bm{y},\bm{h},\mu,\theta\right)
&\propto P\left(s_T=j\right)\pi\left(\xi_T=\hat\xi_T\mid s_T=j\right)\times 1,
\end{aligned}
\end{equation}
where
\begin{align*}
P\left(s_t=j\right) &= p_j, \\
\pi\left(\xi_t=\hat\xi_t\mid s_t=j\right) &\propto \frac{1}{v_j}\exp\left(-\frac{\left(\hat\xi_t-m_j\right)^2}{2v_j^2}\right),
\end{align*}
and
\begin{multline*}
\pi\left(\nu_t=\hat\nu_t\mid\xi_t=\hat\xi_t,s_t=j,h_{t+1},h_t,\mu,\theta\right) \propto \\
\propto \exp\left(-\frac{\left(\hat{\nu}_t-\left(\gamma_t^j\left[a_j+b_j\left(\hat\xi_t-m_j\right)\right]\right)\right)^2}{2\sigma^2\left(1-\rho^2\right)}\right).
\end{multline*}
This way, values proportional to the posterior probabilities are obtained, so they need to be normalised.
We can use the inverse sampling method to draw $\bm s\mid\bm{y},\bm{h},\mu,\sigma,\rho,\phi$~\citep{grinstead2012introduction}.

\subsubsection{Drawing $\rho,\sigma,\phi$}

The remaining two steps are more involved and based on~\citet{de1995simulation}.
They are detailed and derived for SV with leverage in~\citet{Nakajima2009}, where SV with leverage is extended to jumps and t distributed residuals.

In this step, the Metropolis-Hastings (MH) algorithm is used to obtain a draw from $\theta\mid\bm{y}^\ast,\bm{d},\bm s$.
MH is useful when the posterior density can be evaluated up to a constant factor.
That can be done using Bayes' theorem, by factoring the posterior into the likelihood, the prior, and a constant factor: $\pi(\theta\mid\bm{y}^\ast,\bm{d},\bm s)\propto\pi(\bm{y}^\ast\mid\theta,\bm{d},\bm s)\pi(\theta)$.
The prior is usually chosen to be from a well-known family, and, based on~\cite{Nakajima2009}, the likelihood can be computed using the Kalman filter.

A proposal density $g(\theta\mid\bm{y}^\ast,\bm{d},\bm s)$ is needed for MH as well, whose support is a superset of the true posterior's support.
It is also important that the evaluation of and sampling from the proposal density are efficient.

Given everything above, an MH step in the $n$th loop first produces a candidate $\theta_\ast$ from $g(\theta\mid\bm{y}^\ast,\bm{d},\bm s)$.
Then, the new sample $\theta_{n+1}$ is chosen randomly from among the current value $\theta_n$ and the candidate $\theta_\ast$.
The so-called acceptance ratio drives the decision,
\begin{equation*}
\text{AR}=\frac{\pi(\bm{y}^\ast\mid\theta_\ast,\bm{d},\bm s)\pi(\theta_\ast)}{\pi(\bm{y}^\ast\mid\theta_n,\bm{d},\bm s)\pi(\theta_n)}\frac{g(\theta_n\mid\bm{y}^\ast,\bm{d},\bm s)}{g(\theta_\ast\mid\bm{y}^\ast,\bm{d},\bm s)}
\end{equation*}
Finally, $\theta_\ast$ is accepted with probability $\min\left\{\text{AR},1\right\}$, and $\theta_{n+1}=\theta_\ast$.
Otherwise $\theta_{n+1}=\theta_n$ remains the state.

\citet{Nakajima2009} chose the proposal to be $\mathcal{N}(c, C)$ truncated to the region $R=\left\{(\sigma,\rho,\phi)\mid\sigma>0,-1<\rho<1,-1<\phi<1\right\}$.
The mean vector $c$ and the covariance matrix $C$ are calculated in each loop such that the proposal's log density is the second order Taylor expansion of the true log density around the true log density's mode.\footnote{
	Even though the true posterior density can only be evaluated up to a (positive) constant factor, the Taylor approximation can be still calculated.
	The reason is that the mode is invariant under positive scaling, and differentiation is only applied to the log density, i.e.\ constant multipliers vanish.}
More precisely,
\begin{align*}
c &= \hat{\theta}+Cv, \\
C^{-1} &= -\frac{\partial^2\log\pi(\theta\mid\bm{y}^\ast,\bm{d},\bm s)}{\partial\theta\partial\theta^\prime},
\end{align*}
where
\begin{align*}
v &= \frac{\partial\log\pi(\theta\mid\bm{y}^\ast,\bm{d},\bm s)}{\partial\theta}, \\
\hat{\theta} &= \text{arg}\max_\theta\pi(\theta\mid\bm{y}^\ast,\bm{d},\bm s).
\end{align*}

\subsubsection{Drawing $\mu$ and the log variance}

Both $\mu$ and $\bm h$ are sampled using by-products of the Kalman filter used when calculating $\pi(\bm{y}^\ast\mid\theta_{n+1},\bm{d},\bm s)$ for the AR.
The new value of $\mu$ comes by simply drawing from $\mathcal{N}(q_{T+1},Q_{T+1})$, where $q_{T+1}$ and $Q_{T+1}$ are readily available at this point.
In order to get the latent vector $\bm h$, we first sample $\bm{\nu}\mid\bm{y}^\ast,\bm{d},\theta,\bm s$ using a Gaussian simulation smoother~\citep{fruhwirth1994data,carter1994gibbs} and then reconstruct $\bm h$ from the formulas in~\citet{de1995simulation}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\comment{
	\begin{align}
	\pi(s_t=j\mid\bm{y},\bm{h},\mu,\theta)
	&= \pi(s_t=j\mid\bm{y^*},\bm{d},\bm{h},\mu,\theta),\nonumber \\
	&\overset{(1)}{=} \pi(s_t=j\mid y^*_t,d_t,h_{t+1},h_t,\mu,\theta),\nonumber \\
	&\overset{(2)}{=} \pi(s_t=j\mid \xi_t,d_t,\nu_t,\mu,\theta),\nonumber \\
	&\propto P(s_t=j)\cdot\pi(\xi_t,\nu_t\mid s_t=j,d_t,\mu,\theta),\nonumber \\
	&\overset{(3)}{=} P(s_t=j)\cdot\pi(\xi_t=y^*_t-h_t\mid s_t=j)\cdot\nonumber \\
	&\qquad\cdot\pi(\nu_t=h_{t+1}-\mu-\phi(h_t-\mu)\mid \xi_t,s_t=j,d_t,\mu,\theta),\nonumber \\
	&\propto p_j\frac{1}{v_j}\exp\left(\frac{(y^*_t-h_t-m_j)^2}{2v_j^2}\right)\cdot \\
	&\qquad\exp\left(\dots\right)
	\end{align}
}


\section{An Example Session}
\label{exampleSession}

To provide a first impression of  the capabilities of the CEC-System, we
describe the development of a quicksort algorithm on lists of
natural numbers.

\subsection{The Specification of Quicksort}

Specifications can be written and completed in a {\em modular}
way. CEC can combine completed specifications without
repeating previous work. The hierarchy of modules for the
specification of quicksort on lists of natural numbers is given
in the following diagram:

\begin{picture}(450,100)
\put(50,0){\makebox(400,100){}}
\put(205,85){\makebox(0,0){{\tt qsortnat}}}
\put(200,80){\vector(-2,-1){45}}
\put(200,80){\vector(2,-1){45}}

\put(150,50){\makebox(0,0){{\tt nat}}}

\put(257,50){\makebox(0,0){{\tt qsort}}}
\put(255,45){\vector(-3,-2){30}}
\put(255,45){\vector(3,-2){30}}

\put(215,20){\makebox(0,0){{\tt totalOrder}}}

\put(300,20){\makebox(0,0){{\tt lists}}}
\end{picture}

\noindent
The module \cec{qsort} --- saved in a file named \cec{qsort.eqn} ---
describes the quicksort algorithm:
\begin{spec}
module qsort using lists + totalOrder.

op sort  : (list -> list).
op split : (elem * list -> pair).
cons (',') : (list * list -> pair).

sort([]) = [].
split(x, l) = (l1 , l2) => sort([x|l]) = append(sort(l1), [x|sort(l2)]).

split(x, []) = ([] , []).
(y =< x) = true and split(x, l) = (l1,l2) => split(x, [y|l]) = ([y|l1],l2).
(y =< x) = false and split(x, l) = (l1,l2) => split(x, [y|l]) = (l1,[y|l2]).
\end{spec}

Complex specifications can be constructed from modules by the
elementary operations {\em combination}, {\em enrichment} and {\em renaming}
(\refArrow chapter \ref{OperationsOnSpecifications}).
The combine operator (\cec{+}) forms the union of two specifications and
the rename operator (\cec{<-}) allows to renaming sorts and operators.
Any specification in CEC is the enrichment of a possibly empty base
specification (``{\tt using} $<$base$>$'') by new vocabulary and axioms.
The module \cec{qsort} is based on \cec{lists}
and \cec{totalOrder}. Here \cec{lists} is the imported module
\begin{spec}
module lists.

cons []   : list .
cons '.'  : (elem * list -> list).
op append : (list * list -> list).

append([], l) = l.
append([e | l1], l2) = [e | append(l1, l2)].
\end{spec}

We use the constructor \cec{[]} to denote the empty list and
the constructor `\cec{.}' for adding an element to a list
(This is exactly the way how lists are constructed in Prolog and we can
use the usual Prolog notation for lists in which \cec{[e|l]} is
a synonym for \cec{(e '.' l)}).
Additionally we have defined an
operator \cec{append} for the concatenation of two lists and described
its behaviour through two equations.

\cec{totalOrder} plays the r$\hat{o}$le of a formal parameter
of \cec{qsort}:
\begin{spec}
module totalOrder.

op (=<) : (elem * elem -> bool) .

(x =< x) = true .
(x =< y) = false => (y =< x) = true .
(x =< y) = true and (y =< z) = true => (x =< z) = true .
(x =< y) = true and (y =< x) = true => x = y .
\end{spec}

It describes a usable approximation of the usual first-order axioms
for total orders in a Horn clause setting.
An actual parameter candidate for \cec{totalOrder} is
the following specification of natural numbers:
\begin{spec}
module nat using totalOrder(elem <- nat).

cons 0 : nat.
cons(s, 100, fy) :  (nat -> nat).

(0 =< n ) = true.
(s n =< 0) = false .
(s n =< s m) = (n =< m).
\end{spec}

Computing with the above specification one has to write the numbers
$n$ as \cec{s}$^n$\cec{0} which is a tedious work. In CEC it is possible to provide different external
representations for the terms of a specification by specifying
the translations from the external to the internal
and from the internal to the external representation
(\refArrow chapter \ref{ParseAndPretty}).

The instantiation of the formal parameter of \cec{qsort} with
nat yields \cec{qsortnat}:
\begin{spec}
module qsortnat using qsort(elem <- nat) + nat.
\end{spec}

The modules \cec{qsort} and \cec{nat} are combined after
renaming the sort \cec{elem} of \cec{qsort} into \cec{nat}.
The completion of \cec{qsortnat} will check the consistency of the axioms
for \cec{=<} in the actual parameter \cec{nat} and in the formal
parameter \cec{totalOrder}. The latter proves the correctness
of the actual parameter in cases where the actual parameter is
constructor-complete. \cec{nat} is constructor-complete.

\subsection{Order Specifications}

In CEC the following termination orderings are available:
\begin{itemize}
\item
Two  {\em precedence orderings} \kw{kns} and \kw{neqkns}, according to
Kapur et. al \cite{KNS85} (\refArrow chapter \ref{kns}).
%They are based on the recursive comparison of paths occurring
%in the terms to be compared. This order is induced by a partial
%order on operators called the {\it precedence}. The precedence ordering
%\kw{neqkns} forbids that two different operators have the same precedence.
%All constructors are given a precedence less than any nonconstructor operator.
\item
The method of {\it polynomial interpretations}
where \[t_1 > t_2 : \iff I(t_1) > I(t_2). \]
if $I(t)$ is the polynomial or tuple of polynomials associated with
it.
The concrete version of this technique as it is used in CEC is due to
\cite{CL86} (\refArrow chapter \ref{poly}).
\kw{poly}\nt{N} stands for polynomial interpretations with
tuplelength $N$.
\end{itemize}
The precedence declarations or polynomial interpretations
for the operators of a specification are given interactively
during the completion process or in a {\em order specification}
associated with the specification
(\refArrow chapter \ref{OrderSpecification}).
%The order specification determines the termination ordering to be used for
%\nt{specificationName}, the order names for its direct imports
%and gives precedence declarations for the operators of the
%specification (if the termination ordering is \kw{kns} or \kw{neqkns}) or
%polynomial interpretations (if the termination ordering is \kw{poly}\nt{N}).

For example, the order specification for the module \cec{lists}
may have the following form:

\begin{spec}
order poly3 for lists.

setInterpretation(['[]'        : [2, 2, 2],
                   '.'(x,y)    : [3 * x + y + 1, x + y, x + y],
                   append(x,y) : [x + y, x + y, 2 * x + y]]).
\end{spec}

In this case, polynomial interpretations with tuple length 3
have been chosen. If this information is stored in the file \cec{lists.pol.ord},
\cec{pol} is called the {\em order name} of this order specification.

\subsection{Reading and Displaying}

We want to demonstrate how CEC handles this specification.
First, we read in the specification of \cec{lists} using
the \comRef{in}-command (\refArrow chapter \ref{InCommand})
with two arguments, the name of the module and the order name
(\refArrow chapter \ref{orderBase}):

\begin{screen}
| ?- in(lists,pol).
[collecting garbage...]
[evaluating base of lists.eqn with lists.pol.ord...]
 [thawing standard.poly3.q2.0 into user...]
 [storing to standard.poly3]
[reading body of lists.eqn and lists.pol.ord...]
[analyzing axioms...]
[collecting garbage...]

Time used: 3.634 sec.
\end{screen}

\noindent
We can use the \comRef{sig}-command to display the signature:

\begin{screen}
| ?- sig.

Signature :

cons true       : bool.
cons false      : bool.
cons []	: list.
cons .  : (elem * list -> list).
op append       : (list * list -> list).
\end{screen}

The boolean constants \cec{true} and \cec{false} are imported
from the \cec{standard}-module, which is automatically imported into
every specification. The fact that \cec{true} and \cec{false}
are constructors implies that any equation, explicitly given in
a specification or derived through completion, equating these
constants is forbidden.
As expected, we also see our two constructors \cec{[]} and
`\cec{.}' and the operator \cec{append}.

The \comRef{show}-command can be used to display the
set of equations and rules (\refArrow chapter \ref{Displaying}):

\begin{screen}
| ?- show.

Current equations

  1    append([],l) = l
  2    append([e|l1],l2) = [e|append(l1,l2)]

Current rules

Current nonoperational equations

All axioms reduced.
All superpositions computed.
The set of equations is not empty.
\end{screen}

\noindent
The information about the polynomial interpretations can be made
visible using the \comRef{interpretation}-command. If \cec{kns}
or \cec{neqkns} is used, this can be achieved by the
\comRef{operators}-command (\refArrow chapter \ref{InspecTermination}).

Every consulted specification is saved in a special {\em specification
variable}. The user can restore an older state of his system
using the \comRef{load}-command (\refArrow chapter \ref{LoadCommand})
or can update the content of the variable using the \comRef{store}-command
(\refArrow chapter \ref{StoreCommand}).
Whenever a specification is to be imported, the CEC system uses the content
of the corresponding specification variable, if this specification variable
exists. So it increases one's speed, if the completion process of a hierarchical
specification follows the tree structure from its leaves to its root,
storing every participating specification after successful completion.
We assume that attention is paid to this advice for our example.
Information about the current specification variables
can be retrieved using the \comRef{specifications}-command.

It is also possible to save a system externally using the
\comRef{freeze}-command (\refArrow chapter \ref{FreezeCommand}).
To restore a externally saved system use the \comRef{thaw}-command
(\refArrow chapter \ref{ThawCommand}).

\subsection{Completion}
We now apply the completion procedure (\refArrow chapter
\ref{Completion}) to our example.

\subsubsection{Completing {\tt lists}}

The completion process --- started with the \comRef{c}-command ---
is able to orient the two equations of \cec{lists}
without requesting any additional information:

\begin{screen}
| ?- c.

new rule   1    append([],l) = l .

new rule   2    append([e|l1],l2) = [e|append(l1,l2)] .

[4 superpositions yet to be considered.]

0 superpositions have been computed.
Time used: 0.883011 sec.
\end{screen}

\noindent
The system is complete, and has the following rules, displayed
by the \comRef{show}-command:

\begin{screen}
| ?- show.

Current equations

Current rules

  1    append([],l) = l
  2    append([e|l1],l2) = [e|append(l1,l2)]

Current nonoperational equations

All axioms reduced.
All superpositions computed.
No more equations, the system is complete.
\end{screen}

\subsubsection{Completing {\tt totalOrder}}

An important feature is that CEC does not require to transform all
equations into rewrite rules. This can be demonstrated during the
completion of \cec{totalOrder}.

After orienting the first equation into a rewrite rule,
the system discovers that it is unable to orient the second
equation:

\begin{screen}
Checking reductivity constraints for rule
        x=<y = false => y=<x = true:
The current ordering fails to prove
[y=<x]  >  [x=<y,false].
At this point you may take any of the the following actions:
a. for assume to be proved
c. for checking quasi-reductivity of the equation
p. for postpone
n. for considering the equation as nonoperational
   Please answer with a. or c. or p. or n. (Type A. to abort) > \user{n.}
\end{screen}

\noindent
We want to consider this equation as nonoperational
(\refArrow chapter \ref{Completion}), so we answer with \user{n.}.

Nonoperational equations become useless for the equational theory and, hence,
in fact nonoperational, if they are superposed on at least one of their
conditions by all rewrite rules. This yields new conditional equations.
What concerns the equational theory, the new equations together have
the same power as the original equation.
However, they may have better operational properties than the
equation they have been generated from.
Equations that cannot be oriented into a reductive rule must
either be eliminated eventually or considered nonoperational.

Checking the convergence of conditional equations requires to compare different
applications of equations and rewrite rules. The comparison of equation application is performed
by comparing the literals of the equation instance. The {\em status} of an equation
determines the order in which the literals of the equation should be
inspected. The status $ms$ means that the literals are compared as a multiset.
Instead of $ms$ the user can choose an arbitrary sequence of the literals
by entering a permutation of [0 .. n] where n is the number of conditions
(\refArrow \cite{Gan88a}).

We want to use $ms$ here:

\begin{screen}
In which order should the literals of the equation be
inspected when comparing proofs that use this equation?
Please enter ms (for multiset ordering) or a permutation of [0 .. 1]
(0 stands for the consequent, i>0 for the ith condition). > \user{ms.}
\end{screen}

\noindent
The other two nonreductive equations are handled in the same way and we get the
following complete system:

\begin{screen}
| ?- \user{show.}

Current equations

Current rules

  1    x=<x = true

Current nonoperational equations

  1    x=<y = false => y=<x = true
  2    x=<y = true and y=<x = true => x = y
  3    x=<y = true and y=<z = true => x=<z = true

All axioms reduced.
All superpositions computed.
No more equations, the system is complete.
\end{screen}

In this case, superposing the only rule \cec{x =< x = true} on the first
condition of the nonoperational equations does not generate any nontrivial
(nonconvergent) consequences.

\subsubsection{Completing {\tt qsort}}

The next step is the completion of the \cec{qsort} specification.
The completion procedure is able to orient the first and third
equation of our specification, but fails to orient the second one.

\begin{screen}
| ?- c.

new rule   9    sort([]) = [] .

new rule  10    split(x,[]) = [],[] .
The equation
        split(x,l) = l1,l2 => sort([x|l]) = append(sort(l1),[x|sort(l2)])
is not reductive.
\end{screen}

\noindent
This is due to the presence of the {\em extra variables} \cec{l1} and \cec{l2}
in the condition of the equation.

Equations with extra variables in the condition or in the right side of
the consequence are usually not admitted as rewrite rules.
Fortunately some of these equations belong to the class of what we call
{\em quasi-reductive rules} (\refArrow chapter \ref{Completion}).
CEC is able to prove the remaining equations 2, 4
and 5 of the module \cec{qsort} to be quasi-reductive.
Quasi-reductive rules are a generalization
of reductive conditional rewrite rules and the associated rewrite
process is similarly efficient.
It specifies e.g. for the second equation the replacement
of \cec{sort([x|l])} by the term \cec{append(sort(l1), [x|sort(l2)])}
if the normalform of \cec{split(x, l)} matches \cec{(l1,l2)}.
After this match the extra variables \cec{l1} and \cec{l2} are instantiated to
terms which include only variables of the left hand side.
In quasi-reductive rules, conditions are oriented, too. To solve an instance
of a condition equation means to rewrite its left side into an instance of the
right side.

\begin{screen}
Do you want a check for quasi-reductivity?
c. for check
n. for considering the equation as nonoperational
p. for postpone
   Please answer with c. or n. or p. (Type A. to abort) >\user{c.}
\end{screen}

\noindent
To orient an equation into a quasi-reductive rule, we must first indicate the
desired orientation of the equations in the condition and the conclusion.
In this example, the equation in the condition
and the conclusion should be oriented from left to right (literal annotation
``\user{l}'').

\begin{screen}
Enter annotations of literals in
	split(x,l) = l1,l2 => sort([x|l]) = append(sort(l1),[x|sort(l2)])
	: \user{[l,l].}
\end{screen}

\noindent
Now CEC attemps to prove the quasi-reductivity of the equation according to the
definition in chapter \ref{Completion}. This involves giving  polynomial
interpretations to some auxiliary operators.
Here we must give an appropriate interpretation for \cec{$h5_0}:

\begin{screen}
Checking reductivity constraint:
Consider the terms
        $h5_0((l1,l2),x)
and
        append(sort(l1),[x|sort(l2)])

There is no interpretation of operator '$h5_0' with arity 2
The default interpretation is :
   [ 2 * x * y ,
     2 * x * y ,
     2 * x * y ]
Do you want to change it ? (if so type 'y') \user{y}
Do you want to change it component for component ? (if so type 'y') \user{n}
Type in the new interpretation tuple
[ $h5_0 ] (x,y) = \user{[2*x+3*y+1,2*x+2*y,2*x+2*y].}
Resulting interpretation for Operator '$h5_0' with arity 2 :
   [ 2 * x + 3 * y + 1 ,
     2 * x + 2 * y ,
     2 * x + 2 * y ]
Do you accept it ? (if not, type 'n') \user{y}
\end{screen}

\noindent
Now the proof of the quasi-reductivity of the equation is completed:

\begin{screen}
new rule  11    split(x,l) = l1,l2 => sort([x|l]) =
                                      append(sort(l1),[x|sort(l2)]) .
\end{screen}

\noindent
In the same way the equations

\cec{y=<x = true and split(x,l) = (l1,l2) => split(x,[y|l]) = ([y|l1],l2)}

\noindent
and

\cec{y=<x = false and split(x,l) = (l1,l2) => split(x,[y|l]) = (l1,[y|l2])}

\noindent
can be oriented into quasi-reductive rules.

For the quicksort specification three nontrivial superposition instances are
computed. For any nontrivial equation with at least one condition that is
generated during completion, CEC will ask the user what to do with it. In the
example we decide to declare these consequences as ``nonoperational''.

\begin{screen}
instance   6    l1,l2 = l4,l3 and split(x1,l5) = l1,l2 =>
                                     append(sort(l1),[x1|sort(l2)]) =
                                     append(sort(l4),[x1|sort(l3)])
of        6    split(x,l) = l1,l2 => sort([x|l]) =
                                     append(sort(l1),[x|sort(l2)])
by superposing
          6    split(x,l) = l1,l2 => sort([x|l]) =
                                     append(sort(l1),[x|sort(l2)])
on the left side.

Consider the equation
        l1,l2 = l4,l3 and split(x1,l5) = l1,l2 =>
                                     append(sort(l1),[x1|sort(l2)]) =
                                     append(sort(l4),[x1|sort(l3)]).
The following actions may be taken:
o. for attempting to orient into a (quasi-)reductive rule
p. for postpone
n. for considering equation as nonoperational
   Please answer with o. or p. or n. (Type A. to abort) >\user{n.}
\end{screen}

As mentioned before it is sufficient to superpose all rewrite rules
on just one condition of a nonoperational conditional equation. So
CEC asks the user on which condition superposition should be
applied.
We will choose the first equation of the condition, since we know it
will generate no nontrivial superpositions:

\begin{screen}
Which of the condition equations in
        l1,l2 = l4,l3 and split(x1,l5) = l1,l2
should be selected for superposition?
Please enter index from 1 to 2. > \user{1.}

The equation l1,l2 = l4,l3 and split(x1,l5) = l1,l2 =>
                                          append(sort(l1),[x1|sort(l2)]) =
                                          append(sort(l4),[x1|sort(l3)])
will be considered as nonoperational.
\end{screen}

\subsubsection{Completing {\tt nat}}

The completion of the \cec{nat} specification is straightforward using
the following order specification:

\begin{spec}
order poly1 for nat.

setInterpretation([0 : 2,
                   s(x) : 8 * x]).
\end{spec}

\subsubsection{Combining {\tt qsort} and {\tt nat}}

We now want to combine the \cec{nat} specification with the
\cec{qsort} specification.

\begin{screen}
| ?- store.
yes
| ?- load(qsort,poly3).
\end{screen}

\noindent
The \comRef{store}-command saves the completed {\tt nat} specification
into its {\em specification variable}.
The \comRef{load}-command loads the completed {\tt qsort} specification
from its specification variable.

\begin{screen}
| ?- renameSpec(elem <- nat).
yes
| ?- sig.

Signature :

cons [] : list.
cons .  : (nat * list -> list).
op append       : (list * list -> list).
cons true       : bool.
cons false      : bool.
op =<   : (nat * nat -> bool).
op sort	: (list -> list).
op split        : (nat * list -> pair).
cons ,    : (list * list -> pair).
yes
| ?- store(qsortnatSpec).
yes
\end{screen}

\noindent
Now we combine the two specifications and make the result
to the current specification:

\begin{screen}
| ?- combineSpecs(qsortnatSpec,'nat.poly1_qsort',user).
yes
\end{screen}

\noindent
Because \cec{totalOrder} is a formal parameter for \cec{qsort},
completion now checks the consistency of the axioms for
\cec{=<} in the actual parameter \cec{nat} and in the formal
parameter \cec{totalOrder}.
The completion process will not do unnecessary work again.
For the above example the completion process
will only compute overlaps between axioms
of the renamed module \cec{qsort} and axioms of the module \cec{nat}
but it will not recompute overlaps between axioms of one of these
modules.

If two constructor terms are shown to be equal then the specification
is inconsistent. In our case the system is completed without any user
interaction, no inconsistency showing up.

\subsection{Computing in Completed Specifications}

Computation in specifications is realized by term reduction with the
rules of the completed specification. The result of such computations are unique
normal forms (\refArrow chapter \ref{NormCommand}):

\begin{screen}
| ?- norm(sort([5,3,6,1])).
The normalform of sort([5,3,6,1]) is [1,3,5,6] .
\end{screen}

\noindent
If confluence can be achieved and if all rules are reductive equational theorems
become decidable, e.g. it is decidable if two terms are equivalent with respect to the
equations in the specification (\refArrow chapter \ref{ProveCommand}):

\begin{screen}
| ?- prove(sort([5,3]) = [3,5]).
Normal forms are: [3,5] and [3,5]
yes
\end{screen}

\noindent
Conditional narrowing (\refArrow chapter \ref{NarrowCommand}) can then be used to solve
equations.

\begin{screen}
| ?- solve(sort([1,x]) = [x,1],U).

Time used: 7.43298 sec.

U = {x-nat/0} ;

Time used: 7.91595 sec.

U = {x-nat/1} ;

no
\end{screen}

\noindent
Here we proved that the only substitutions for \cec{x} such that
\cec{sort([1,x])} is equal to \cec{[x,1]} are \cec{0} and \cec{1}.

\subsection{Saving the CEC System}

The whole state of the CEC system can be saved using the
\comRef{saveCEC}-command.

\begin{screen}
| ?- saveCEC('qsortCEC').
[ Prolog state saved into /home/helga/cec/cec/qsortCEC ]
\end{screen}

Using \cec{qsortCEC} instead of CEC offers the possibility to have
the complete hierarchy of our \cec{qsort} example present, without
wasting time to reconsult the necessary frozen states of all the
specification used for this example.


\section{Policy Learning Details}
\label{sec:supp-lqr}

Given a TVLG dynamics model and quadratic cost approximation, we can approximate our Q and value functions to second order with the following dynamic programming updates, which proceed from the last time step $t = T$ to the first step $t = 1$:
\begin{align*}
    Q_{\mathbf{s},t}=c_{\mathbf{s},t}&+\dynmat_{\mathbf{s},t}^\top{V}_{\mathbf{s},t+1}\,,~~Q_{\mathbf{ss},t}=c_{\mathbf{ss},t}+\dynmat_{\mathbf{s},t}^\top{V}_{\mathbf{ss},t+1}\dynmat_{\mathbf{s},t}\,,\\
    Q_{\mathbf{a},t}=c_{\mathbf{a},t}&+\dynmat_{\mathbf{a},t}^\top{V}_{\mathbf{s},t+1}\,,~~Q_{\mathbf{aa},t}=c_{\mathbf{aa},t}+\dynmat_{\mathbf{a},t}^\top{V}_{\mathbf{ss},t+1}\dynmat_{\mathbf{a},t}\,,\\
    &Q_{\mathbf{sa},t}=c_{\mathbf{sa},t}+\dynmat_{\mathbf{s},t}^\top{V}_{\mathbf{ss},t+1}\dynmat_{\mathbf{a},t}\,,\\
    &V_{\mathbf{s},t}=Q_{\mathbf{s},t}-Q_{\mathbf{sa},t}Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{a},t}\,,\\
    &V_{\mathbf{ss},t}=Q_{\mathbf{ss},t}-Q_{\mathbf{sa},t}Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{as},t}\,.
\end{align*}
It can be shown (e.g., by \citet{synthesis}) that the action $\mathbf{a}_t$ that minimizes the second-order approximation of the Q-function at every time step $t$ is given by
\[
\mathbf{a}_t=-Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{as},t}\mathbf{s}_t-Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{a},t}\,.
\]
This action is a linear function of the state $\mathbf{s}_t$, thus we can construct an optimal linear policy by setting $\mathbf{K}_t=-Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{as},t}$ and $\mathbf{k}_t=-Q_{\mathbf{aa},t}^{-1}Q_{\mathbf{a},t}$. We can also show that the maximum-entropy policy that minimizes the approximate Q-function is given by
\[
\pi^\star(\mathbf{a}_t|\mathbf{s}_t)=\N(\mathbf{K}_t\mathbf{s}_t+\mathbf{k}_t,Q_{\mathbf{aa},t}).
\]
Furthermore, as in \citet{mfcgps}, we can impose a constraint on the total KL-divergence between the old and new trajectory distributions induced by the policies through an augmented cost function $\tilde{\cost}(\state_t,\action_t)=\frac{1}{\lambda}\costmodel(\state_t,\action_t)-\log\bar{\policy}(\action_t|\state_t)$, where solving for $\lambda$ via dual gradient descent can yield an exact solution to a KL-constrained LQR problem.

\section{Parameterizing the Cost Model}
\label{sec:supp-cost}

The simplest choice that we consider for parameterizing the cost model is as a full quadratic function of the state and action, i.e.,     $\costmodel(\state_t,\action_t)=\frac{1}{2}\state_t^\top\costmat\state_t+\costvec^\top\state_t+\alpha\|\action_t\|_2^2+b$ where we assume that the action-dependent part of the cost -- i.e., $\alpha$ -- is known, and we impose no restrictions on the learned parameters $\costmat$ and $\costvec$. This is our default option due to its simplicity and the added benefit that fitting this model locally can be done in closed form through least-squares quadratic regression on the observed states. However, another option we consider is to choose $\costmodel(\state_t,\action_t)=\frac{1}{2}\state_t^\top\mathbf{L}\mathbf{L}^\top\state_t+\costvec^\top\state_t+\alpha\|\action_t\|_2^2+b$. $\mathbf{L}$ is a lower-triangular matrix with non-negative diagonal entries, and thus by constructing our cost matrix as $\costmat=\mathbf{L}\mathbf{L}^\top$ we guarantee that the learned cost matrix is positive semidefinite, which can improve the behavior of the policy update.

In general, in this work, we consider quadratic parameterizations of the cost model since we wish to build a LQS model. However, in general it may be possible to use non-quadratic but twice-differentiable cost models, such as a neural network model, and compute local quadratic cost models using a second-order Taylor approximation as in \citet{mfcgps}. We also do not assume access to a goal observation, though if provided with such information we can use construct a quadratic cost function that penalizes distance to this goal in the learned latent space, as in \citet{spatial-ae} and \citet{e2c}.

\section{The SVAE Algorithm}
\label{sec:supp-svae}

\citet{Johnson2016} builds off of \citet{Hoffman2013} and \citet{Winn2005}, who show that, for conjugate exponential models, the variational model parameters can be updated using natural gradients of the form
\begin{equation}
    \tilde{\nabla}_{\omega}\L[\q]=\omega^0+B\mathbb{E}_\q\left[t_{\dynmat,\dyncovar}(\dynmat,\dyncovar)\right]-\omega\,,
\end{equation}
Where $\omega$ denotes the MNIW parameters of the variational factors on $\dynmat,\dyncovar$, $B$ is the number of minibatches in the dataset, $\omega^0$ is the parameter for the prior distribution $p(\dynmat,\dyncovar)$, and $t_{\dynmat,\dyncovar}(\dynmat,\dyncovar)$ is the sufficient statistic function for $p(\dynmat,\dyncovar)$. Thus, we can use this equation to compute the natural gradient update for $\omega$, whereas for $\gamma$, $\phi$, and the parameters of the cost model, we use stochastic gradient updates on Monte Carlo estimates of the ELBO, specifically using the Adam optimizer \citep{Kingma2014}. This leads to two simultaneous optimizations, and their learning rates are treated as separate hyperparameters. We have found $10^{-4}$ and $10^{-3}$ to be good default settings for the natural gradient step size and stochastic gradient step size, respectively.

\section{Fitting the Local Dynamics Model}
\label{sec:supp-fit}

In the pretraining phase described in \autoref{sec:solar-modeling}, we are learning the following sets of parameters from observed trajectories:
\vspace{-.5em}
\begin{enumerate}
    \itemsep0em
    \item The parameters of the variational posterior over global dynamics $q_\textrm{global}(\dynmat, \dyncovar)$;
    \item The weights of the encoder and decoder networks $\decoder_\gamma(\state)$ and $\encoder_\phi(\observation)$;
    \item The parameters of the cost function $\costmodel(\state, \action)$.
\end{enumerate}
\vspace{-.5em}
In the RL phase described in \autoref{sec:inference}, after learning the representation and global models, we fit local, linear-Gaussian dynamics models to additional trajectories. The conjugacy of the Bayesian LQS model enables a computationally efficient expectation-maximization procedure to learn the local dynamics. We assume the same graphical model as in \autoref{eq:gm-start} to \autoref{eq:gm-end} except we modify \autoref{eq:dyn} and \autoref{eq:dyn2} to be
\begin{align*}
    \dynmat_t, \dyncovar_t &\sim p(\dynmat_t, \dyncovar_t) \triangleq q_{\textrm{global}}(\dynmat, \dyncovar)\,,\\
    \state_{t + 1} | \state_t, \action_t, \dynmat_t, \dyncovar_t &\sim \N(\dynmat_t\colvec{\state_t\\\action_t}, \dyncovar_t)\,.
\end{align*}
The model assumes that the TVLG dynamics are independent samples from our global dynamics, followed by a deep Bayesian LDS to generate trajectories. This is similar to the globally trained model, with the exception that we explicitly assume time-varying dynamics.

Now suppose we have collected a set of trajectories of the form $\trajectory$ and aim to fit a local dynamics model. We use variational inference to approximate the posterior distributions by setting up the variational factors
\vspace{-.5em}
\begin{enumerate}
    \itemsep0em
    \item $q(\state_{1:\horizon} | \dynmat_{1:\horizon}, \dyncovar_{1:\horizon}; \observation_{1:\horizon}, \action_{1:\horizon})$, which approximates the posterior distribution $p(\state_{1:\horizon} | \observation_{1:\horizon}, \action_{1:\horizon}, \dynmat_{1:\horizon}, \dyncovar_{1:\horizon})$;
    \item $q(\dynmat_t, \dyncovar_t)$, which approximates the posterior distribution $p(\dynmat_t, \dyncovar_t | \state_{1:\horizon}, \action_{1:\horizon})$
\end{enumerate}
\vspace{-.5em}
The ELBO under these variational factors is:
\begin{align*}
    \L[\q] &= \mathbb{E}_\q \big[\sum_t^\horizon \log p(\observation_t | \state_t)\\ &- \mathrm{KL}\left(q(\state_{1:\horizon})\|p(\state_{1:\horizon} | \action_{1:\horizon}, \dynmat_{1:\horizon}, \dyncovar_{1:\horizon})\right)\\
    &- \sum_t^{\horizon - 1} \mathrm{KL}\left(q(\dynmat_t, \dyncovar_t)\|p(\dynmat_t, \dyncovar_t)\right)
    \big]
\end{align*}

We use variational EM to alternatively optimize $q(\state_{1:\horizon} | \dynmat_{1:\horizon}, \dyncovar_{1:\horizon}; \observation_{1:\horizon}, \action_{1:\horizon})$ and $q(\dynmat_t, \dyncovar_t)$. Using evidence potentials $\psi(\state_t;\observation_t,\phi)$ output by the recognition network $\encoder_\phi(\observation_t)$, both of these optimizations can be done in closed form. Specifically, the optimal $q(\state_{1:\horizon} | \dynmat_{1:\horizon}, \dyncovar_{1:\horizon}; \observation_{1:\horizon}, \action_{1:\horizon})$ is computed via Kalman smoothing using evidence potentials from the recognition network, and the optimal $q(\dynmat_t, \dyncovar_t)$ can be computed via Bayesian linear regression using expected sufficient statistics from $q(\state_{1:\horizon} | \dynmat_{1:\horizon}, \dyncovar_{1:\horizon}; \observation_{1:\horizon}, \action_{1:\horizon})$.

\section{Experiment Setup}
\label{sec:supp-set}

{\bf 2D navigation.} Our recognition model architecture for the 2D navigation domain consists of two convolution layers with \mbox{2-by-2} filters and 32 channels each, with no pooling layers and ReLU non-linearities, followed by another convolution with \mbox{2-by-2} filters and 2 channels. The output of the last convolution layer is fed into a fully-connected layer which then outputs a Gaussian distribution with diagonal covariance. Our observation model consists of FC hidden layers with 256 ReLU activations, and the last layer outputs a categorical distribution over pixels. We initially collect 100 episodes which we use to train our model, and for every subsequent RL iteration we collect 10 episodes. The cost function we use is the sum of the $L^2$-norm squared of the distance to the target and the commanded action, with weights of 1 and 0.001, respectively.

As discussed in \autoref{sec:solar-experiments}, we modify the 2D navigation task from \citet{e2c} and \citet{rce} to randomize the location of the target every episode, and we set this location uniformly at random between $-2.8$ and $2.8$ for both the x and y coordinates, as coordinates outside of $[-3,3]$ are not visible in the image. We similarly randomize the initial position of the agent. In this setup, we use two \mbox{32-by-32} images as the observation, one with the location of the agent and the other with the location of the target, and in the fixed-target version of the task we only use one \mbox{32-by-32} image.

{\bf Nonholonomic car.} The nonholonomic car domain consists of \mbox{64-by-64} image observations. Our recognition model is a convolutional neural network with four convolutional layers with \mbox{4-by-4} filters with 4 channels each, and the first two convolution layers are followed by a ReLU non-linearity. The output of the last convolutional layer is fed into three FC ReLU layers of width 2048, 512, and 128, respectively. Our final layer outputs a Gaussian distribution with dimension 8. Our observation model consists of four FC ReLU layers of width 256, 512, 1024, and 2048, respectively, followed by a Bernoulli distribution layer that models the image. For this domain, we collect 100 episodes initially to train our model, and then for RL we collect 100 episodes per iteration. The cost function we use is the sum of the $L^2$-norm squared of the distance from the center of the car to the target and the commanded action, with weights of 1 and 0.001, respectively.

{\bf Reacher.} The reacher domain consists of \mbox{64-by-64-by-3} image observations. Our recognition model consists of three convolutional layers with \mbox{7-by-7}, \mbox{5-by-5}, and \mbox{3-by-3} filters with 64, 32 and 8 channels respectively. The first convolutional layer is followed by a ReLU non-linearity. The output of the last convolutional layer is fed into an FC ReLU layer of width 256, which outputs a Gaussian distribution with dimension 10. Our observation model consists of one FC ReLU layers of width 512, followed by three deconvolutional layers with the reverse order of filters and channels as the recognition model. This is followed by a Bernoulli distribution layer that models each image. We collect 200 episodes initially to train our model, and then for RL we collect 100 episodes per iteration. The cost function we use is the sum of the $L^2$-norm of the distance from the fingertip to the target and the $L^2$-norm squared of the commanded action, which is the negative of the reward function as defined in Gym.

{\bf Sawyer Lego block stacking.} The image-based Sawyer block-stacking domain consists of \mbox{64-by-64-by-3} image observations. The policy outputs velocities on the end effector in order to control the robot. Our recognition model is a convolutional neural network with the following architecture: a \mbox{5-by-5} filter convolutional layer with 16 channels followed by two convolutional layers using \mbox{5-by-5} filters with 32 channels each. The convolutional layers are followed by ReLU activations leading to a 12 dimensional Gaussian distribution layer. Our observation model consists of a FC ReLU layer of width 128 feeding into three deconvolutional layers, the first with \mbox{5-by-5} filters with 16 channels and the last two of \mbox{6-by-6} filters with 8 channels each. These are followed by a final Bernoulli distribution layer.

For this domain, we collect 400 episodes initially to train our model and 10 per iteration thereafter. Note that this pretraining data is collected only once across solving all of the tasks that we test on. The cost function is the cubed root of the $L^2$-norm of the displacement vector between the end-effector and the target in 3D-space.

{\bf Sawyer pushing.} The image-based Sawyer pushing domain also operates on \mbox{64-by-64-by-3} image observations. Our recognition and observation models are the same as those used in the block-stacking domain. The dynamics model is learned by a network with two FC ReLU layers of width 128 followed by a 12 dimensional Gaussian distribution layer. The cost model is learned jointly with the representation and dynamics by optimizing the ELBO, which with regards to the cost corresponds to logistic regression on the observed sparse reward using a sampled latent state as the input. We collect 200 episodes to train our model and 20 per iteration for RL.

During the RL phase, the human supervisor uses keyboard input to provide the sparse reward signal to the learning algorithm, indicating whether or not the mug was successfully pushed onto the coaster. In practice, for simplicity, we label the last five images of the trajectory as either $0$ or $1$ depending on whether or not the keyboard was pressed at any time during the trajectory, as for this task a successful push is typically reflected in the end state. In order to overcome the exploration problem and provide a diverse dataset for pretraining the cost model, we manually collect $180$ ``goal images'' where the mug is on the coaster and the robot arm is in various locations.

\section{Implementation of Comparisons}
\label{sec:comp}

{\bf PPO.} We use the open source implementation of PPO (named ``PPO2'') from the OpenAI Baselines project: \mbox{\footnotesize{\url{https://github.com/openai/baselines}}}. We write OpenAI gym wrappers for our simulated environments in order to test PPO on our simulated tasks.

{\bf LQR-FLM.} We implement LQR-FLM based on the open-source implementation from the Guided Policy Search project: \mbox{\footnotesize{\url{https://github.com/cbfinn/gps}}}. The only modification to the LQR-FLM algorithm that we make is to handle unknown cost functions by fitting a quadratic cost model to data from the current policy.

{\bf DVF.} We train a video prediction model using the open source Stochastic Adversarial Video Prediction project: \mbox{\footnotesize{\url{https://github.com/alexlee-gk/video_prediction}}}. To define the task, we specify the location of a pixel whose movement to a specified goal location indicates success. The cost function is then the predicted probability of successfully moving the selected pixel to the goal. We then use MPC, specifically the cross-entropy method (CEM) for offline planning: we sample sequences of actions from a Gaussian, predict the corresponding sequence of images using the video prediction model, evaluate the cost of the imagined trajectory with the cost model, and refit the parameters of the Gaussian to the best predicted action sequences. This iterative process eventually outputs an action sequence to perform in the real world in order to try and solve the task.

{\bf RCE.} We use model learning code directly from the authors of RCE \citep{rce}, though this code is not publicly available and to our knowledge there are no open source implementations of RCE or E2C \citep{e2c} that are able to reproduce the results from the respective papers. In addition to LQR-based control, we also experiment with MPC with neural network dynamics and cost models in the learned latent representation. In our experiments, we report the best results using either of these control methods.

{\bf VAE ablation.} In the VAE ablation, we replace our representation and global models with a standard VAE \citep{Kingma2014, Rezende2014}, which imposes a unit Gaussian prior on the latent representation. Because we cannot infer local dynamics as described in \autoref{sec:inference}, we instead use a GMM dynamics prior that is trained on all data as described by \citet{gps}. After fitting a local quadratic cost model, we again have a local LQS model that we can use in conjunction with an LQR-FLM policy update.

{\bf MPC baseline.} MPC involves planning $H$ time steps ahead using a dynamics and cost model, executing an action based on this plan, and then re-planning after receiving the next observation \citep{mpc}. Recently, MPC has proven to be a successful control method when combined with neural network dynamics models, where many trajectories are sampled using the model and then the first action corresponding to the best imagined trajectory is executed \citep{nn-dyn,pets}. Similar to LQR-FLM, we can extend MPC to handle image-based domains by learning dynamics and cost models within a learned latent representation. As MPC does not require an LQS model, we can instead utilize neural network dynamics and cost models which are more expressive.

\section{Additional Experiments}
\label{sec:supp-exp}

\subsection{RCE on Fixed-Target 2D Navigation}

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{img/solar/2dnav-rce.png}
    \caption{On 2D navigation with the goal fixed to the bottom right, RCE is able to successfully learning a policy for navigating to the goal.}
    \label{fig:pm-e2c}
\end{figure}

As mentioned in \autoref{sec:solar-experiments}, RCE was unable to make progress for the 2D navigation task, though we were able to get more successful results by fixing the position of the goal to the bottom right as is done in the image-based 2D navigation task considered in E2C \citep{e2c} and RCE \citep{rce}. \autoref{fig:pm-e2c} details this experiment, which we ran for three random seeds and report the mean and standard deviation of the average final distance to the goal as a function of the number of training episodes. This indicates that RCE can indeed solve some tasks from image observations, though we were unable to use RCE succesfully on any of the tasks we consider.

\subsection{Full Learning Progress of PPO}

In \autoref{fig:log-plots} we include the plots for the simulated tasks comparing \metabbr\ and PPO. Note that the x-axis is on a log scale, i.e., though our method is sometimes worse in final policy performance, we use one to three orders of magnitude fewer samples. This demonstrates our method's sample efficiency compared to model-free methods, while being able to solve complex image-based domains that are difficult for model-based methods.

\begin{figure}
    \centering
    \begin{subfigure}{0.32\linewidth}
        \includegraphics[width=0.9\linewidth]{img/solar/2dnav-log.png}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.31\linewidth}
        \includegraphics[width=0.9\linewidth]{img/solar/car-log.png}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.33\linewidth}
        \includegraphics[width=0.9\linewidth]{img/solar/reacher-log.png}
        \caption{}
    \end{subfigure}
    \caption[Comparison of \metabbr\ to PPO]{(a)~Comparison of our method to PPO on the 2D navigation task presented in the paper. Our method uses roughly three orders of magnitude fewer samples to solve the task compared to PPO. (b)~On the car from images task, our method achieves slightly worse performance than PPO though with about 25 times fewer samples. (c)~Comparison of our method to PPO for the reacher task. Our method achieves worse final performance but uses about 40 times fewer samples than these methods.}
    \label{fig:log-plots}
\end{figure}


\chapter{Lecture 3 May 07th 2018}
  \label{chapter:lecture_3_may_07th_2018}

\section{Groups} % (fold)
\label{sec:groups}

\subsection{Groups} % (fold)
\label{sub:groups}

\begin{defn}[Groups]\label{defn:groups}
\index{Groups}
  Let $G$ be a set and $*$ an operation on $G \times G$. We say that $G = (G, *)$ is a \hlnoteb{group} if it satisfies\sidenote{If you wonder why the uniqueness is not specified for \hlnoteb{Identity} and \hlnoteb{Inverse}, see \cref{propo:uniqueness_of_group_identity_and_group_element_inverse}.}
  \begin{enumerate}
    \item \hlnoteb{Closure}: $\forall a, b \in G \quad a * b \in G$
    \item \hlnoteb{Associativity}: $\forall a, b, c \in G \quad a * (b * c) = (a * b) * c$
    \item \hlnoteb{Identity}: $\exists e \in G \enspace \forall a \in G \quad a * e = a = e * a$
    \item \hlnoteb{Inverse}: $\forall a \in G \enspace \exists b \in G \quad a * b = e = b * a$
  \end{enumerate}
\end{defn}

\begin{defn}[Abelian Group]\label{defn:abelian_group}
\index{Abelian Group}
  A group $G$ is said to be abelian if $\forall a, b \in G$, we have $a * b = b * a$.
\end{defn}

\begin{propo}[Group Identity and Group Element Inverse]\label{propo:uniqueness_of_group_identity_and_group_element_inverse}
  Let $G$ be a group and $a \in G$.
  \begin{enumerate}
    \item The identity of $G$ is unique.
    \item The inverse of $a$ is unique.
  \end{enumerate}
\end{propo}

\begin{proof}
  \begin{enumerate}
    \item If $e_1, e_2 \in G$ are both identities of $G$, then we have
      \begin{equation*}
        e_1 \overset{(1)}{=} e_1 * e_2 \overset{(2)}{=} e_2
      \end{equation*}
      where $(1)$ is because $e_2$ is an identity and $(2)$ is because $e_1$ is an identity.

    \item Let $a \in G$. If $b_1, b_2 \in G$ are both the inverses of $a$, then we have
      \begin{equation*}
        b_1 = b_1 * e = b_1 * (a * b_2) \overset{(1)}{=} e * b_2 = b_2
      \end{equation*}
      where $(1)$ is by associativity.
  \end{enumerate}
\end{proof}

\begin{eg}
  The sets $(\mathbb{Z}, +), \, (\mathbb{Q}, +), \, (\mathbb{R}, +)$, and $(\mathbb{C}, +)$ are all abelian, where the additive identity is $0$, and the additive inverse of an element $r$ is $(-r)$.
\end{eg}

\begin{note}
  $(\mathbb{N}, +)$ is not a group for neither does it have an identity nor an inverse for any of its elements.
\end{note}

\begin{eg}
  The sets $(\mathbb{Q}, \cdot), \, (\mathbb{R}, \cdot)$ and $(\mathbb{C}, \cdot)$ are \hlwarn{not} groups, since $0$ has no multiplicative inverse in $\mathbb{Q}, \mathbb{R}$ or $\mathbb{C}$.
\end{eg}

We may define that for a set $S$, let $S^* \subseteq S$ contain all the elements of $S$ that has a multiplicative inverse. For example, $\mathbb{Q}^* = \mathbb{Q} \setminus \{0\}$. Then, $(\mathbb{Q}, \cdot), (\mathbb{R}, \cdot)$ and $(\mathbb{C}, \cdot)$ are groups and are in fact abelian, where the multiplicative identity is $1$ and the multiplicative of an element $r$ is $\frac{1}{r}$.

\begin{eg}
  The set $\big( M_n(\mathbb{R}), + \big)$ is an abelian group, where the additive identity is the zero matrix, $0 \in M_n(\mathbb{R})$, and the additive inverse of an element $M = [a_{ij}] \in M_n(\mathbb{R})$ is $-M = [-a_{ij}] \in M_n(\mathbb{R})$.
\end{eg}

\newthought{Consider} the set $M_n(\mathbb{R})$ under the matrix mutiplication operation that we have introduced in \nameref{chapter:lecture_1_may_02nd_2018}. We found that the identity matrix is
\begin{equation*}
  I = \begin{bmatrix}
    1 & 0 & \hdots & 0 \\
    0 & 1 & \hdots & 0 \\
    \vdots & \vdots & & \vdots \\
    0 & 0 & \hdots & 1
  \end{bmatrix} \in M_n(\mathbb{R}).
\end{equation*}
But since not all elements of $M_n(\mathbb{R})$ have a multiplicative inverse\sidenote{The multiplicative inverse of a matrix does not exist if its determinant is $0$.}, $(M_n(\mathbb{R}), \cdot)$ is not a group.

\newthought{We can try} to do something similar as to what we did before: by excluding the elements that do not have an inverse. In this case, we exclude elements whose determinant is $0$. We define the following set

\begin{defn}[General Linear Group]\index{General Linear Group}
\label{defn:general_linear_group}
  The \hlnoteb{general linear group of degree $n$ over $\mathbb{R}$} is defined as
  \begin{equation*}
    GL_n(\mathbb{R}) := \{ M \in M_n(\mathbb{R}) \, : \, \det M \neq 0 \}
  \end{equation*}
\end{defn}

Note that $\because \det I = 1 \neq 0$, we have that $I \in GL_n(\mathbb{R})$. \\
Also, $\forall A, B \in GL_n(\mathbb{R} )$, we have that $\because \det A \neq 0 \, \land \, \det B \neq 0$,
\begin{equation*}
  \det AB = \det A \det B \neq 0,
\end{equation*}
and therefore $AB \in GL_n(\mathbb{R} )$. Finally, $\forall M \in GL_n(\mathbb{R})$, $\exists M^{-1} \in GL_n(\mathbb{R})$ such that
\begin{equation*}
  MM^{-1} = I = M^{-1} M
\end{equation*}
since $\det M \neq 0$. $\therefore (GL_n(\mathbb{R}), \cdot)$ is a group.

\newthought{Since} we have introduced permutations in \nameref{chapter:lecture_2_may_04th_2018}, we shall formalize the purpose of its introduction below.

\begin{eg}
  Consider $S_n$, the set of all permutations on $\{1, 2, ..., n\}$. By \cref{propo:properties_of_Sn}, we know that $S_n$ is a group. We call $S_n$ the \hldefn{symmetry group} \hlnoteb{of degree $n$}. For $n \geq 3$, the group $S_n$ is not abelian\sidenote{Let us make this an exercise.
  \begin{ex}
    For $n \geq 3$, prove that the group $S_n$ is not abelian.
  \end{ex}}.
\end{eg}

\newthought{Now that} we have a fairly good idea of the basic concept of a group, we will now proceed to look into handling multiple groups. One such operation is known as the \hldefn{direct product}.

\begin{eg}
  \label{eg:direct_product}
  Let $G$ and $H$ be groups. Their direct product is the set $G \times H$ with the component-wise operation defined by
  \begin{equation*}
    (g_1, h_1) * (g_2, h_2) = (g_1 *_G g_2, h_1 *_H h_2)
  \end{equation*}
  where $g_1, g_2 \in G$, $h_1, h_2 \in H$, $*_G$ is the operation on $G$, and $*_H$ is the operation on $H$.

  The \hlnoteb{closure} and \hlnoteb{associativity} property follow immediately from the definition of the operation. The identity is $(1_G, \, 1_H)$ where $1_G$ is the identity of $G$ and $1_H$ is the identity of $H$. The inverse of an element $(g_1, \, h_1) \in G \times H$ is $(g_1^{-1}, \, h_1^{-1})$.
\end{eg}

By induction, we can show that if $G_1, G_2, ..., G_n$ are groups, then so is $G_1 \times G_2 \times \hdots \times G_n$.

To facilitate our writing, use shall use the following notations:

\begin{notation}
  Given a group $G$ and $g_1, g_2 \in G$, we often denote its identity by $1$, and write $g_1 * g_2 = g_1 g_2$. Also, we denote the unique inverse of an element $g \in G$ as $g^{-1}$.

  We will write $g^0 = 1$. Also, for $n \in \mathbb{N}$, we define
  \begin{equation*}
     g^n = \underbrace{g * g * \hdots * g}_{n \text{ times}}
  \end{equation*}
  and
  \begin{equation*}
    g^{-n} = (g^{-1})^n
  \end{equation*}
\end{notation}

With the above notations,

\begin{propo}\label{propo:group_notations}
  Let $G$ be a group and $g, h \in G$. We have \marginnote{
    \begin{ex}
      Prove \cref{propo:group_notations} as an exercise.
    \end{ex}
  }
  \begin{enumerate}
    \item $(g^{-1})^{-1} = g$
    \item $(gh)^{-1} = h^{-1} g^{-1}$
    \item $g^n g^m = g^{n + m}$ for all $n, m \in \mathbb{Z}$
    \item $(g^n)^m = g^{nm}$ for all $n, m \in \mathbb{Z}$
  \end{enumerate}
\end{propo}

\begin{warning}
  In general, it is not true that if $g, h \in G$, then $(gh)^n = g^n h^n$. For example,
  \begin{equation*}
    (gh)^2 = ghgh \quad \text{but} \quad g^2 h^2 = gghh.
  \end{equation*}
  The two are only equal if and only if $G$ is abelian.
\end{warning}

% subsection groups (end)

% section groups (end)

% chapter lecture_3_may_07th_2018 (end)


\chapter{Discrete Fourier Transform (DFT) Project}
\glsresetall
\label{chapter:DFT_project}

\section{Introduction}
The goal of this project is design architectures that implement the Discrete Fourier Transform (DFT).  The DFT is a common operation in signal processing which generates a frequency domain representation of the discrete input signal. We start with the direct implementation of the DFT which is a matrix-vector multiplication. The input signal is a vector of samples and the matrix is composed of a set of basis functions corresponding to discrete cosine and sine waveforms of different frequencies. The multiplication of the input signal with these basis functions describes how well the input signal correlates with those waveforms, which is the value of the Fourier series at that frequency.

\section{Materials}
You can find the following files in \texttt{project2-dft.zip}. These are divided into four folders: \texttt{dft\_8\_precomputed}, \texttt{dft\_32\_precomputed}, \texttt{dft\_256\_precomputed}, and \texttt{dft\_1024\_precomputed}. Each of these folders has its own testbench, \texttt{out.gold.dat} file, and \texttt{coefficients.h} file.

Each folder contains following files:
\begin{itemize}
\item \texttt{dft.cpp}: The baseline implementation for the dft function.
\item \texttt{dft.h}: Header file holding important constants and other variables.
\item \texttt{dft\_test.cpp}: Testbench
\item \texttt{coefficientsX.h}: A file containing the values of corresponding to one sine/cosine period sampled based upon the DFT points. For example, an 8 point DFT has 8 samples across both the sine and cosine function evenly spaced across one period. This is equivalent to dividing one rotation in the complex plane equally by the number of points in the DFT.
\item \texttt{out.gold.dat}: ``Golden'' output. The testbench (dft\_test.cpp) generates a sample input and calls the function \texttt{dft} in \texttt{dft.cpp} with that sample input. This output of the function is compared to the expected output. This will indicate PASS or FAIL. If it fails, then the code in \texttt{dft} is incorrect. There are four different versions of depending on the DFT size and way in which the DFT coefficients were generated.
\item \texttt{script.tcl} and \texttt{directives.tcl}: Used to create the project.
\end{itemize}

\section{Project Goal}
You should modify the code to create a number of different architectures that perform tradeoffs between performance and area.  For \texttt{dft\_256\_precomputed} and \texttt{dft\_1024\_precomputed} designs, you need to use precomputed values from \texttt{coefficients256.h} and \texttt{coefficients1024.h}.

For 256-point and 1024-point DFTs, you will create a report describing how you generated these different architectures (code restructuring, pragmas utilized, etc.). For each architecture you should provide its results including the resource utilization (BRAMs, DSP48, LUT, FF), and performance in terms of throughput (number of FFT operations/second), latency, clock cycles, clock frequency (which is fixed to 10 ns). You can do most of the design space exploration on the 256 point DFT. You should pick your ``best'' 256 architecture and synthesize that as a 1024 DFT.

The 8 and 32 point folders are provided for your convenience. If you would like, you can do some of your initial design space optimization on these smaller architectures. But it is not necessary to use these at all.

The key in this project is to understand the tradeoffs between loop optimizations (unrolling and pipelining) and data partitioning. Therefore you should focus on these optimizations.

\section{Optimization Hints and Guidelines}
\begin{itemize}
\item You should use a clock period of 10 ns.
\item The output of your architecture must closely match the golden output. Be sure to generate a working function before performing any optimizations. If the result does not match exactly, but is close, please explain why in the report.
\item You should use float for all data types. You do not need to perform bitwidth optimization of this project.
\item The current design is set up to do the DFT in-place, i.e., you put the output results back into the same arrays that give you the input results. You can change this if you think it will give you better results.
\item There are many different ways to generate the DFT coefficients. These are constants when the DFT size is fixed. We have given you the coefficients for both 256 (in \texttt{coefficients256.h}) and 1024 (in \texttt{coefficients1024.h}). They each have two constant arrays, \texttt{sin\_table} and \texttt{cos\_table}. You can use these coefficient arrays directly as memories in your architectures. You are also free to create your own arrays using a different structure (e.g., 2D array, reordering of elements in the given 1D arrays, etc.). Or you could dynamically generate the coefficients.
\item There is significant amount of parallelism that can be exploited by (partially) unrolling the for loops. Pipelining these (partially) unrolled for loops should lead to higher throughputs.
\item There are more efficient methods for performing the DFT that exploit the symmetries of the Fourier constants, e.g., the fast Fourier transform (FFT). Do not use these symmetries. In other words, treat this like a matrix-vector multiply with unknown matrix values. Do not worry, we will implement FFT architectures in a following project that will fully take advantage of these symmetries.
\item You do not need to report your optimizations for your 8 point and 32 point DFT; these folders are provided for your convenience. Since these will very likely synthesize much faster than larger point DFT functions, it may be useful to use these to debug your code or in your initial design space exploration.
\item Your report must explicitly state how you calculated the throughput results. Note that this is often not simply a function of the latency and the clock period, and involves using the initiation interval.
\end{itemize}