\section{Dependent type theory}
\label{ch:dtt}

\index{dependent type theory|(}
Dependent type theory is a system of inference rules that can be combined to make \emph{derivations}. In these derivations, the goal is often to construct a term of a certain type. Such a term can be a function if the type of the constructed term is a function type; a proof of a property if the type of the constructed term is a proposition; an identification if the type of the constructed term is an identity type, and so on. In some respect, a type is just a collection of mathematical objects and constructing terms of a type is the everyday mathematical task or challenge. The system of inference rules that we call type theory offers a principled way of engaging in mathematical activity.

\subsection{Judgments and contexts in type theory}\label{sec:judgments}
\index{judgment|(}
\index{context|(}
An \define{inference rule}\index{inference rule|see {rule}} is an expression of the form
\begin{prooftree}
  \AxiomC{$\mathcal{H}_1$\quad $\mathcal{H}_2$ \quad \dots \quad $\mathcal{H}_n$}
  \UnaryInfC{$\mathcal{C}$}
\end{prooftree}
containing above the horizontal line\index{horizontal line|see {inference rule}} a finite list $\mathcal{H}_1$, $\mathcal{H}_2$, \dots, $\mathcal{H}_n$ of \emph{judgments} for the hypotheses\index{inference rule!hypotheses}, and below the horizontal line a single judgment $\mathcal{C}$ for the conclusion\index{inference rule!conclusion}. A very simple example that we will encounter in \cref{ch:pi} when we introduce function types\index{function type}, is the inference rule
\begin{prooftree}
  \AxiomC{$\Gamma\vdash a:A$}
  \AxiomC{$\Gamma\vdash f:A\to B$}
  \BinaryInfC{$\Gamma\vdash f(a):B$}
\end{prooftree}
This rule asserts that in any context $\Gamma$ we may use a term $a:A$ and a function $f:A\to B$ to obtain a term $f(a):B$. Each of the expressions
\begin{align*}
  \Gamma & \vdash a :A \\
  \Gamma & \vdash f : A \to B \\
  \Gamma & \vdash f(a):B
\end{align*}
are examples of judgments. There are four kinds of judgments in type theory:
\begin{enumerate}
\item \emph{$A$ is a (well-formed) \define{type} in context $\Gamma$.}
  \index{well-formed type}\index{type}
  The symbolic expression for this judgment is\index{Gamma turnstile A type@{$\Gamma\vdash A~\type$}}\index{judgment!Gamma turnstile A type@{$\Gamma\vdash A~\type$}}
  \begin{equation*}
    \Gamma\vdash A~\type
  \end{equation*}
\item \emph{$A$ and $B$ are \define{judgmentally equal types} in context $\Gamma$.}
  \index{judgmental equality!of types} The symbolic expression for this judgment is\index{Gamma turnstile A is B type@{$\Gamma\vdash A\jdeq B~\type$}}\index{judgment!Gamma turnstile A is B type@{$\Gamma\vdash A\jdeq B~\type$}}
  \begin{equation*}
    \Gamma\vdash A \jdeq B~\type
  \end{equation*}
\item \emph{$a$ is a (well-formed) \define{term}\index{well-formed term}\index{term} of type $A$ in context $\Gamma$.} The symbolic expression for this judgment is\index{Gamma turnstile a in A@{$\Gamma\vdash a:A$}}\index{judgment!Gamma turnstile a in A@{$\Gamma\vdash a:A$}}
  \begin{equation*}
    \Gamma \vdash a:A
  \end{equation*}
\item \emph{$a$ and $b$ are \define{judgmentally equal terms} of type $A$ in context $\Gamma$.}\index{judgmental equality!of terms} The symbolic expression for this judgment is\index{Gamma turnstile a is b in A@{$\Gamma\vdash a\jdeq b:A$}}\index{judgment!Gamma turnstile a is b in A@{$\Gamma\vdash a\jdeq b:A$}}
  \begin{equation*}
    \Gamma\vdash a\jdeq b:A
  \end{equation*}
\end{enumerate}
Thus we see that any judgment is of the form $\Gamma\vdash\mathcal{J}$, consisting of a context $\Gamma$ and an expression $\mathcal{J}$ asserting that $A$ is a type, that $A$ and $B$ are equal types, that $a$ is a term of type $A$, or that $a$ and $b$ are equal terms of type $A$. The role of a context is to declare what hypothetical terms\index{hypothetical term} are assumed, along with their types. More formally, a \define{context} is an expression of the form
\begin{equation}\label{eq:context}
x_1:A_1,~x_2:A_2(x_1),~\ldots,~x_n:A_n(x_1,\ldots,x_{n-1})
\end{equation}
satisfying the condition that for each $1\leq k\leq n$ we can derive, using the inference rules of type theory, that
\begin{equation}\label{eq:context-condition}
  x_1:A_1,~x_2:A_2(x_1),~\ldots,~x_{k-1}:A_{k-1}(x_1,\ldots,x_{k-2})\vdash A_k(x_1,\ldots,x_{k-1})~\type.
\end{equation}
In other words, to check that an expression of the form \cref{eq:context} is a context, one starts on the left and works their way to the right verifying that each hypothetical term $x_k$ is assigned a well-formed type. Hypothetical terms are commonly called \define{variables}\index{variable}, and we say that a context as in \cref{eq:context} \define{declares the variables}\index{variable declaration} $x_1,\ldots,x_n$. We may use variable names other than $x_1,\ldots,x_n$, as long as no variable is declared more than once.

The condition in \cref{eq:context-condition} that each of the hypothetical terms is assigned a well-formed type, is checked recursively. Note that the context of length $0$ satisfies the requirement in \cref{eq:context-condition} vacuously. This context is called the \define{empty context}\index{context!empty context}\index{empty context}. An expression of the form $x_1:A_1$ is a context if and only if $A_1$ is a well-formed type in the empty context. Such types are called \define{closed types}\index{closed type}\index{type!closed type}. We will soon encounter the type $\N$ of natural numbers\index{natural numbers}, which is an example of a closed type\index{natural numbers!is a closed type}. There is also the notion of \define{closed term}\index{closed term}\index{term!closed term}, which is simply a term in the empty context. The next case is that an expression of the form $x_1:A_1,~x_2:A_2(x_1)$ is a context if and only if $A_1$ is a well-formed type in the empty context, and $A_2(x_1)$ is a well-formed type, given a hypothetical term $x_1:A_1$. This process repeats itself for longer contexts.

It is a feature of \emph{dependent} type theory that all judgments are context-dependent, and indeed that even the types of the variables may depend on any previously declared variables. For example, when we introduce the \emph{identity type}\index{identity type} in \cref{chap:identity}, we make full use of the machinery of type dependency, as is clear from how they are introduced:
\begin{prooftree}
  \AxiomC{$\Gamma\vdash A~\type$}
  \UnaryInfC{$\Gamma,x:A,y:A\vdash x=y~\type$}
\end{prooftree}
This rule asserts that given a type $A$ in context $\Gamma$, we may form a type $x=y$ in context $\Gamma,x:A,y:A$. Note that in order to know that the expression $\Gamma,x:A,y:A$ is indeed a well-formed context, we need to know that $A$ is a well-formed type in context $\Gamma,x:A$. This is an instance of \emph{weakening}\index{weakening}, which we will describe shortly.

In the situation where we have
\begin{equation*}
  \Gamma,x:A\vdash B(x)~\type,
\end{equation*}
we say that $B$ is a \define{family}\index{family}\index{type family} of types over $A$ in context $\Gamma$. Alternatively, we say that $B(x)$ is a type \define{indexed}\index{indexed type}\index{type!indexed} by $x:A$, in context $\Gamma$. Similarly, in the situation where we have
\begin{equation*}
  \Gamma,x:A\vdash b(x):B(x),
\end{equation*}
we say that $b$ is a \define{section}\index{section of a family} of the family $B$ over $A$ in context $\Gamma$. Alternatively, we say that $b(x)$ is a term of type $B(x)$, \define{indexed}\index{indexed term}\index{term!indexed} by $x:A$ in context $\Gamma$. Note that in the above situations $A$, $B$, and $b$ also depend on the variables declared in the context $\Gamma$, even though we have not explicitly mentioned them. It is common practice to not mention every variable in the context $\Gamma$ in such situations.

\index{judgment|)}
\index{context|)}

\subsection{Inference rules}\label{sec:rules}

In this section we present the basic inference rules of dependent type theory. Those rules are valid to be used in any type theoretic derivation. There are only four sets of inference rules:
\begin{enumerate}
\item Rules for judgmental equality
\item Rules for substitution
\item Rules for weakening
\item The ``variable rule''
\end{enumerate}

\subsubsection*{Judgmental equality}

\index{rules!for type dependency!rules for judgmental equality|(}
In this set of inference rules we ensure that judgmental equality (both on types and on terms) are equivalence relations, and we make sure that in any context $\Gamma$, we can change the type of any variable to a judgmentally equal type.

\begin{samepage}
The rules postulating that judgmental equality on types and on terms is an equivalence relation are as follows\index{judgmental equality!is an equivalence relation}:
\begin{center}
%\begin{small}
\begin{minipage}{.2\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash A~\textrm{type}$}
\UnaryInfC{$\Gamma\vdash A\jdeq A~\textrm{type}$}
\end{prooftree}
\end{minipage}
\begin{minipage}{.25\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash A\jdeq A'~\textrm{type}$}
\UnaryInfC{$\Gamma\vdash A'\jdeq A~\textrm{type}$}
\end{prooftree}
\end{minipage}
\begin{minipage}{.5\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash A\jdeq A'~\textrm{type}$}
\AxiomC{$\Gamma\vdash A'\jdeq A''~\textrm{type}$}
\BinaryInfC{$\Gamma\vdash A\jdeq A''~\textrm{type}$}
\end{prooftree}
\end{minipage}
\\*
\bigskip
\begin{minipage}{.2\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash a:A$}
\UnaryInfC{$\Gamma\vdash a\jdeq a : A$}
\end{prooftree}
\end{minipage}
\begin{minipage}{.25\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash a\jdeq a':A$}
\UnaryInfC{$\Gamma\vdash a'\jdeq a: A$}
\end{prooftree}
\end{minipage}
\begin{minipage}{.5\textwidth}
\begin{prooftree}
\AxiomC{$\Gamma\vdash a\jdeq a' : A$}
\AxiomC{$\Gamma\vdash a'\jdeq a'': A$}
\BinaryInfC{$\Gamma\vdash a\jdeq a'': A$}
\end{prooftree}
\end{minipage}
%\end{small}
\end{center}
\end{samepage}

\bigskip
Apart from the rules postulating that judgmental equality is an equivalence relation, there are also \define{variable conversion rules}\index{judgmental equality!conversion rules}\index{variable conversion rules}\index{conversion rule!variable}\index{rules!for type dependency!variable conversion}.
Informally, these are rules stating that if $A$ and $A'$ are judgmentally equal types in context $\Gamma$, then any valid judgment in context $\Gamma,x:A$ is also a valid judgment in context $\Gamma,x:A'$. In other words: we can convert the type of a variable to a judgmentally equal type.

The first variable conversion rule states that
\begin{prooftree}
\AxiomC{$\Gamma\vdash A\jdeq A'~\textrm{type}$}
\AxiomC{$\Gamma,x:A,\Delta\vdash B(x)~\type$}
\BinaryInfC{$\Gamma,x:A',\Delta\vdash B(x)~\type$}
\end{prooftree}
In this conversion rule, the context of the form $\Gamma,x:A,\Delta$ is just any extension of the context $\Gamma,x:A$, i.e., a context of the form
\begin{equation*}
  x_1:A_1,\ldots,x_{n-1}:A_{n-1},x:A,x_{n+1}:A_{n+1},\ldots,x_{n+m}:A_{n+m}.
\end{equation*}

Similarly, there are variable conversion rules for judgmental equality of types, for terms, and for judgmental equality of terms. To avoid having to state essentially the same rule four times, we state all four variable conversion rules at once using a \emph{generic judgment} $\mathcal{J}$, which can be any of the four kinds of judgments.
\begin{prooftree}
\AxiomC{$\Gamma\vdash A\jdeq A'~\textrm{type}$}
\AxiomC{$\Gamma,x:A,\Delta\vdash \mathcal{J}$}
\BinaryInfC{$\Gamma,x:A',\Delta\vdash \mathcal{J}$}
\end{prooftree}
An analogous \emph{term conversion rule}, stated in \cref{ex:term_conversion}, converting the type of a term to a judgmentally equal type, is derivable using the rules for substitution and weakening, and the variable rule.
\index{rules!for type dependency!rules for judgmental equality|)}

\subsubsection*{Substitution}
\index{substitution|(}\index{rules!for type dependency!rules for substitution|(}
If we are given a term $a:A$ in context $\Gamma$, then for any type $B$ in context $\Gamma,x:A,\Delta$ we can simultaneously substitute $a$ for all occurences of the variable $x$ in $\Delta$ and in $B$, to obtain a type $B[a/x]$ in context $\Gamma,\Delta[a/x]$. You are already familiar with simultaneous substitution, e.g., substituting $0$ for $x$ in the polynomial
\begin{equation*}
  1+x+x^2+x^3
\end{equation*}
resuts in the number $1+0+0^2+0^3$, which can be computed to the value $1$.

Type theoretic substitution is similar. In a bit more detail, suppose we have well-formed type
\begin{equation*}
  x_1:A_1,\ldots,x_{n-1}:A_{n-1},x_n:A_n,x_{n+1}:A_{n+1},\ldots,x_{n+m}:A_{n+m}\vdash B~\textrm{type}
\end{equation*}
and a term $a:A_n$ in context $x_1:A_1,\ldots,x_{n-1}:A_{n-1}$. Then we can form the type
\begin{equation*}
  x_1:A_1,\ldots,x_{n-1}:A_{n-1},x_{n+1}:A_{n+1}[a/x_n],\ldots,x_{n+m}:A_{n+m}[a/x_n]\vdash B[a/x_n]~\textrm{type}
\end{equation*}
by substituting $a$ for all occurences of $x_n$. Note that the variables $x_{n+1},\ldots,x_{n+m}$ are assigned new types after performing the substitution of $a$ for $x_n$.

This operation of substituting $a$ for $x$ is understood to be defined recursively over the length of $\Delta$. When $B$ is a family of types over $A$ and $a:A$, we also say that $B[a/x]$ is the \define{fiber}\index{family!fiber of}\index{fiber!of a family} of $B$ at $a$. We will usually write $B(a)$ for $B[a/x]$. Similarly we obtain for any term $b:B$ in context $\Gamma,x:A,\Delta$ a term $b[a/x]:B[a/x]$. The term $b[a/x]$ is called the \define{value} of $b$ at $a$. When we substitute in a judgmental equality, either of types or terms, we simply subtitute on both sides of the equation.

We can now postulate the \define{substitution rule} as follows:
\begin{prooftree}
\AxiomC{$\Gamma\vdash a:A$}
\AxiomC{$\Gamma,x:A,\Delta\vdash \mathcal{J}$}
\RightLabel{$S$}
\BinaryInfC{$\Gamma,\Delta[a/x]\vdash \mathcal{J}[a/x]$}
\end{prooftree}
In other words, the substitution rule asserts that substitution preserves well-formedness and judgmental equality of types and terms. Furthermore, we postulate that substitution by judgmentally equal terms results in judgmentally equal types
\begin{prooftree}
\AxiomC{$\Gamma\vdash a\jdeq a':A$}
\AxiomC{$\Gamma,x:A,\Delta\vdash B~\type$}
\BinaryInfC{$\Gamma,\Delta[a/x]\vdash B[a/x]\jdeq B[a'/x]~\type$}
\end{prooftree}
and it also results in judgmentally equal terms
\begin{prooftree}
\AxiomC{$\Gamma\vdash a\jdeq a':A$}
\AxiomC{$\Gamma,x:A,\Delta\vdash b:B$}
\BinaryInfC{$\Gamma,\Delta[a/x]\vdash b[a/x]\jdeq b[a'/x]:B[a/x]$}
\end{prooftree}
To see that these rules make sense, we observe that both $B[a/x]$ and $B[a'/x]$ are types in context $\Delta[a/x]$, provided that $a\jdeq a'$. This is immediate by recursion on the length of $\Delta$.
\index{substitution|)}\index{rules!for type dependency!rules for substitution|)}

\subsubsection*{Weakening}
\index{weakening|(}\index{rules!for type dependency!rules for weakening|(}
If we are given a type $A$ in context $\Gamma$, then any judgment made in a longer context $\Gamma,\Delta$ can also be made in the context $\Gamma,x:A,\Delta$, for a fresh variable $x$. The \define{weakening rule}\index{weakening} asserts that weakening by a type $A$ in context preserves well-formedness and judgmental equality of types and terms.
\begin{prooftree}
\AxiomC{$\Gamma\vdash A~\textrm{type}$}
\AxiomC{$\Gamma,\Delta\vdash \mathcal{J}$}
\RightLabel{$W$}
\BinaryInfC{$\Gamma,x:A,\Delta \vdash \mathcal{J}$}
\end{prooftree}
This process of expanding the context by a fresh variable of type $A$ is called \define{weakening} (by $A$).

In the simplest situation where weakening applies, we have two types $A$ and $B$ in context $\Gamma$. Then we can weaken $B$ by $A$ as follows
\begin{prooftree}
  \AxiomC{$\Gamma\vdash A~\textrm{type}$}
  \AxiomC{$\Gamma\vdash B~\textrm{type}$}
  \RightLabel{$W$}
  \BinaryInfC{$\Gamma,x:A\vdash B~\type$}
\end{prooftree}
in order to form the type $B$ in context $\Gamma,x:A$. The type $B$ in context $\Gamma,x:A$ is called the \define{constant family}\index{family!constant family}\index{constant family} $B$, or the \define{trivial family}\index{family!trivial family}\index{trivial family} $B$.
\index{weakening|)}\index{rules!for type dependency!rules for weakening|)}

\subsubsection*{The variable rule}
If we are given a type $A$ in context $\Gamma$, then we can weaken $A$ by itself to obtain that $A$ is a type in context $\Gamma,x:A$. The \define{variable rule}\index{variable rule}\index{rules!for type dependency!variable rule} now asserts that any hypothetical term $x:A$ in context $\Gamma$ is a well-formed term of type $A$ in context $\Gamma,x:A$.
\begin{prooftree}
\AxiomC{$\Gamma\vdash A~\textrm{type}$}
\RightLabel{$\delta$}
\UnaryInfC{$\Gamma,x:A\vdash x:A$}
\end{prooftree}
One of the reasons for including the variable rule is that it provides an \emph{identity function}\index{identity function} on the type $A$ in context $\Gamma$.

\subsection{Derivations}\label{sec:derivations}

\index{derivation|(}
A derivation in type theory is a finite tree in which each node is a valid rule of inference. At the root of the tree we find the conclusion, and in the leaves of the tree we find the hypotheses. We give two examples of derivations: a derivation showing that any variable can be changed to a fresh one, and a derivation showing that any two variables that do not mutually depend on one another can be swapped in order.

Given a derivation with hypotheses $\mathcal{H}_1,\ldots,\mathcal{H}_n$ and conclusion $\mathcal{C}$, we can form a new inference rule
\begin{prooftree}
  \AxiomC{$\mathcal{H}_1$}
  \AxiomC{$\cdots$}
  \AxiomC{$\mathcal{H}_n$}
  \TrinaryInfC{$\mathcal{C}$}
\end{prooftree}
Such a rule is called \define{derivable}, because we have a derivation for it. In order to keep proof trees reasonably short and manageable, we use the convention that any derived rules can be used in future derivations.

\subsubsection*{Changing variables}

\index{change of variables}
Variables can always be changed to fresh variables. We show that this is the case by showing that the inference rule\index{rules!for type dependency!change of variables}
\begin{prooftree}
  \AxiomC{$\Gamma,x:A,\Delta\vdash \mathcal{J}$}
  \RightLabel{$x'/x$}
  \UnaryInfC{$\Gamma,x':A,\Delta[x'/x]\vdash \mathcal{J}[x'/x]$}
\end{prooftree}
is derivable, where $x'$ is a variable that does not occur in the context $\Gamma,x:A,\Delta$.

Indeed, we have the following derivation using substitution, weakening, and the variable rule:
\begin{prooftree}
  \AxiomC{$\Gamma\vdash A~\type$}
  \RightLabel{$\delta$}
  \UnaryInfC{$\Gamma,x':A\vdash x':A$}
  \AxiomC{$\Gamma\vdash A~\type$}
  \AxiomC{$\Gamma,x:A,\Delta\vdash \mathcal{J}$}
  \RightLabel{$W$}
  \BinaryInfC{$\Gamma,x':A,x:A,\Delta\vdash \mathcal{J}$}
  \RightLabel{$S$}
  \BinaryInfC{$\Gamma,x':A,\Delta[x'/x]\vdash \mathcal{J}[x'/x]$}
\end{prooftree}
In this derivation it is the application of the weakening rule where we have to check that $x'$ does not occur in the context $\Gamma,x:A,\Delta$.

\subsubsection*{Interchanging variables}

The \define{interchange rule}\index{rules!for type dependency!interchange}\index{interchange rule} states that if we have two types $A$ and $B$ in context $\Gamma$, and we make a judgment in context $\Gamma,x:A,y:B,\Delta$, then we can make that same judgment in context $\Gamma,y:B,x:A,\Delta$ where the order of $x:A$ and $y:B$ is swapped. More formally, the interchange rule is the following inference rule
\begin{prooftree}
\AxiomC{$\Gamma\vdash B~\textrm{type}$}
\AxiomC{$\Gamma,x:A,y:B,\Delta\vdash \mathcal{J}$}
\BinaryInfC{$\Gamma,y:B,x:A,\Delta\vdash \mathcal{J}$}
\end{prooftree}
Just as the rule for changing variables, we claim that the interchange rule is a derivable rule.

The idea of the derivation for the interchange rule is as follows: If we have a judgment
\begin{equation*}
  \Gamma,x:A,y:B,\Delta\vdash\mathcal{J},
\end{equation*}
then we can change the variable $y$ to a fresh variable $y'$ and weaken the judgment to obtain the judgment
\begin{equation*}
  \Gamma,y:B,x:A,y':B,\Delta[y'/y]\vdash\mathcal{J}[y'/y].
\end{equation*}
Now we can substitute $y$ for $y'$ to obtain the desired judgment $\Gamma,y:B,x:A,\Delta\vdash\mathcal{J}$. The formal derivation is as follows:
%\begin{small}
\begin{prooftree}
\AxiomC{$\Gamma\vdash B~\textrm{type}$}
\RightLabel{$\delta$}
\UnaryInfC{$\Gamma,y:B\vdash y:B$}
\RightLabel{$W$}
\UnaryInfC{$\Gamma,y:B,x:A\vdash y:B$}
\AxiomC{$\Gamma,x:A,y:B,\Delta\vdash \mathcal{J}$}
\RightLabel{$y'/y$}
\UnaryInfC{$\Gamma,x:A,y':B,\Delta[y'/y]\vdash \mathcal{J}[y'/y]$}
\RightLabel{$W$}
\UnaryInfC{$\Gamma,y:B,x:A,y':B,\Delta[y'/y]\vdash \mathcal{J}[y'/y]$}
\RightLabel{$S$}
\BinaryInfC{$\Gamma,y:B,x:A,\Delta\vdash \mathcal{J}$}
\end{prooftree}
%\end{small}
\index{derivation|)}

\begin{exercises}
\exercise \label{ex:term_conversion}Give a derivation for the following \define{term conversion rule}\index{term conversion rule}\index{rules!for type dependency!term conversion}\index{term conversion rule}\index{conversion rule!term}:
\begin{prooftree}
\AxiomC{$\Gamma\vdash A\jdeq A'~\textrm{type}$}
\AxiomC{$\Gamma\vdash a:A$}
\BinaryInfC{$\Gamma\vdash a:A'$}
\end{prooftree}
\exercise Consider a type $A$ in context $\Gamma$. In this exercise we establish a correspondence between types in context $\Gamma,x:A$, and uniform choices of types $B_a$, where $a$ ranges over terms of $A$ in a uniform way. A similar connection is made for terms.
  \begin{subexenum}
  \item We define a \define{uniform family}\index{uniform family} over $A$ to consist of a type
    \begin{equation*}
      \Delta,\Gamma\vdash B_a~\type
    \end{equation*}
    for every context $\Delta$, and every term $\Delta,\Gamma\vdash a:A$, subject to the condition that one can derive
    \begin{prooftree}
      \AxiomC{$\Delta\vdash d:D$}
      \AxiomC{$\Delta,y:D,\Gamma\vdash a:A$}
      \BinaryInfC{$\Delta,\Gamma\vdash B_a[d/y]\jdeq B_{a[d/y]}~\type$}
    \end{prooftree}
    Define a bijection between the set of types in context $\Gamma,x:A$ modulo judgmental equality, and the set of uniform families over $A$ modulo judgmental equality.
  \item Consider a type $\Gamma,x:A\vdash B$. We define a \define{uniform term}\index{uniform term} of $B$ over $A$ to consist of a type
    \begin{equation*}
      \Delta,\Gamma\vdash b_a:B[a/x]~\type
    \end{equation*}
    for every context $\Delta$, and every term $\Delta,\Gamma\vdash a:A$, subject to the condition that one can derive
    \begin{prooftree}
      \AxiomC{$\Delta\vdash d:D$}
      \AxiomC{$\Delta,y:D,\Gamma\vdash a:A$}
      \BinaryInfC{$\Delta,\Gamma\vdash b_a[d/y]\jdeq b_{a[d/y]}:B[a/x][d/y]$}
    \end{prooftree}
    Define a bijection between the set of terms of $B$ in context $\Gamma,x:A$ modulo judgmental equality, and the set of uniform terms of $B$ over $A$ modulo judgmental equality.
  \end{subexenum}
\end{exercises}
\index{dependent type theory|)}


\section{Ground State to an Excited Bound State}
We mentioned in Chapter 5 that ideally the total cross section would have both
amplitudes corresponding to ground-continuum transitions and bound-bound
transitions. The first step that is required is to calculate a photoabsorption
matrix element from the ground state to one of the excited bound
states of hydrogen.

This chapter provides semi analytic results for photoabsorption matrix elements
from the ground state to the first three excited states of hydrogen.
We will consider the the relativistic absorption operator $\mathcal{A} = \Alpha_x \Exp{k}{r}
= \Alpha_x e^{i\eta(\mb{k},\theta,\phi)r}$ for a photon polarised in the $x$
direction and we define $\eta$ as:
\begin{equation}
    \eta(\mb{k},\theta,\phi) = k_x \sin\phi \cos\phi +
                               k_y \sin\phi \sin\theta +
                               k_z \cos\phi
\end{equation}
In the electric dipole approximation $\mb{k} = 0$ and hence $\eta = 0$.
The use of $\eta$ allows us to separate the angular and radial integrals
therefore writing the photoabsorption matrix element from the ground
state $\ket{\psi_0}$ to an excited bound state $\ket{\psi_i}$ as:
\begin{equation}
    \bra{\psi_i} X \ket{\psi_0}
    =
    \int (A_{i1}A_{01} + A_{ii}A_{0i}) R^g_{0i} \; d\Omega
    +
    \int (A_{i3}A_{03} + A_{i4}A_{04}) R^h_{0i} \; d\Omega
\end{equation}
where $A_{ij}(\theta,\phi)$ are the angular components of the excited
state Dirac spinors as defined in Chapter 2.
$R^g_{0i}$ and $R^h_{0i}$ are the large and small component radial
integrals which are defined in the sections below for each excited state.
Some of the radial integrals are analytic while others have to be solved
numerically. The final integration over all angles $ d\Omega $ has to
be performed numerically.

All the radial integrals are written in terms of the constants defined in
Chapter 2 and in terms of the integrals $\mathcal{J}$ and $\mathcal{K}$
defined below.
\begin{equation}
    \mathcal{J}[a,n] = \int_0^\infty e^{-ar} r^n \; dr
\end{equation}
\begin{equation}
    \mathcal{K}[a,n,A,B,C,D] = \int_0^\infty
                \left( \frac{A-Br}{C-Dr} \right) e^{-ar} r^{n} \; dr
\end{equation}

\section{Ground State to First Excited State}
\begin{equation}
R^g_{01} = G_0 G'_1 \mathcal{J}[\Half(\sigma_1 + \sigma_2) - i\eta,2\gamma_1] -
           G_0 G''_1 \mathcal{J}[\Half(\sigma_1 + \sigma_2) - i\eta,2\gamma_1 + 1]
\end{equation}

\begin{equation}
\begin{split}
R^h_{01} = H_0 G_0 H_1
            G'_1 \mathcal{K}[\Half(\sigma_1 + \sigma_2) - i\eta, 2\gamma_1,H'_1,H''_1,H'''_1,H''''_1]
            \\ -
   H_0 G_0 H_1          G''_1 \mathcal{K}[\Half(\sigma_1 + \sigma_2) - i\eta, 2\gamma_1,  1H'_1,H''_1,H'''_1,H''''_1]
\end{split}
\end{equation}

\section{Ground State to Second Excited State}
\begin{equation}
R^g_{02} = G_0 G'_2 \mathcal{J}[\Half(\sigma_1 + \sigma_2) - i\eta,2\gamma_1] -
           G_0 G''_2 \mathcal{J}[\Half(\sigma_1 + \sigma_2) - i\eta,2\gamma_1 + 1]
\end{equation}

\begin{equation}
\begin{split}
R^h_{02} = H_0 G_0 H_2
            G'_1 \mathcal{K}[\Half(\sigma_1 + \sigma_2) - i\eta, 2\gamma_1,H'_2,H''_2,H'''_2,H''''_2]
            \\ -
   H_0 G_0 H_2          G''_1 \mathcal{K}[\Half(\sigma_1 + \sigma_2) - i\eta, 2\gamma_1, 1H'_2,H''_2,H'''_2,H''''_2]
\end{split}
\end{equation}

\section{Ground State to Third Excited State}
\begin{equation}
R^g_{03} = G_0 G_3 \mathcal{J}[\Half(\sigma_1 + \sigma_3) - i\eta, \gamma_1 + \gamma_2]
\end{equation}

\begin{equation}
R^h_{03} = H_0 G_0 H_3 G_3 \mathcal{J}[\Half(\sigma_1 + \sigma_3) - i\eta, \gamma_1 + \gamma_2]
\end{equation}


%!TEX program = xelatex

\documentclass[12pt,a4paper]{article}
\usepackage{xeCJK}
\usepackage{amsmath}
\setmainfont{Times New Roman}
\usepackage{setspace}
\usepackage{caption}
\usepackage{graphicx, subfig}
\usepackage{float}
\usepackage{listings}
\usepackage{booktabs}
\usepackage{setspace}%使用间距宏包
\usepackage{mathtools}
\usepackage{amsfonts}

\begin{document}
\title{homework4}
	\author{11611118 郭思源}

\section{Problem 1}
\textbf{Question: State and verify the Lagrange Interpolation Theorem.}

\begin{spacing}{1.5}%%行间距变为double-space
\subsection{What is Lagrange Interpolation Theorem?}
\noindent In numerical analysis, \textbf{Lagrangian interpolation} is a polynomial interpolation method named after French 18th century mathematician Joseph Lagrange. Many practical problems use functions to represent some kind of internal connection or law, and many functions can only be understood through experiments and observations. If you observe a physical quantity in practice and get the corresponding observations in several different places, Lagrangian interpolation can find a polynomial that takes the observed value at each observation point. Such a polynomial is called a \textbf{Lagrangian (interpolation) polynomial}. Mathematically,  Lagrangian interpolation gives a polynomial function that happens to traverse several known points on a two-dimensional plane. The Lagrange interpolation method was first discovered by the British mathematician Edward Waring in 1779, and soon after (1783) was discovered again by Leonhard Euler. In 1795, Lagrange published this interpolation method in his book ``Basic Tutorials for Normal Schools'', and his name has since been associated with this method.\\

\noindent
For a given ${n+1}$ points ${(x_0,y_0), (x_1,y_1),\ldots,(x_n,y_n)}$, there is only one Lagrangian polynomial L corresponding to their number of times not exceeding n. If a polynomial with a higher number of counts is counted, there is an infinite number because all polynomials that differ from L by
${\lambda(x_0,y_0)(x_1,y_1)\ldots(x_n,y_n)}$ satisfy the condition.

\newpage
\subsection{Definition}
\noindent For a polynomial function, the given ${n+1}$ value points are known:\\
${(x_0,y_0), (x_1,y_1),\ldots,(x_n,y_n)}$\\
Where ${x_{j}}$ corresponds to the position of the argument, \\
and ${y_{j}}$ corresponds to the value of the function at this position.\\

\noindent Assuming that any two different ${x_{j}}$ are different from each other, the \textbf{Lagrangian interpolation polynomial} obtained by applying the Lagrangian interpolation formula is:

\begin{equation}
	L(x) \coloneqq \sum_{j=0}^k y_j \ell_j(x)
\end{equation}\\

\noindent Each of $\ell _{j}(x)$ is a \textbf{Lagrangian basic polynomial} (or \textbf{interpolation basis function}) whose expression is:

\begin{equation}
\begin{aligned}
	\ell_j(x) \coloneqq &\prod_{i = 0,i = j}^k
	\frac{x-x_i}{x_j-x_i} \\
	= &\frac{x-x_0}{x_j-x_0} \ldots
	\frac{x-x_{j-1}}{x_j-x_{j-1}}
	\frac{x-x_{j+1}}{x_j-x_{j+1}} \ldots
	\frac{x-x_k}{x_j-x_k}
\end{aligned}
\end{equation}\\

\noindent The \textbf{Lagrangian basic polynomial} $\ell _{j}(x)$ is characterized by a value of \textbf{1} on ${x_{j}}$ and a value of
\textbf{0} on other points ${x_{j}}$, ${i\neq j}$.

\newpage
\subsection{Proof of existence}
For a given $k+1$ point: $(x_{0}, y_{0}), \ldots, (x_{k}, y_{k})$, the idea of Lagrange interpolation is to find one at a point $x_{j}$ takes a value of \textbf{1}, while at other points the polynomial $\ell _{j}(x)$ whose value is \textbf{0}. Thus, the polynomial $y_{j}\ell _{j}(x)$ takes the value $y_{j}$ at point $x_{j}$ and \textbf{0} at other points. \\

\noindent And the polynomial
\begin{equation*}
	L(x) \coloneqq \sum_{j=0}^k y_j \ell_j(x)
\end{equation*}

\noindent can satisfy :
\begin{equation*}
	L(x_{j})=\sum _{{i=0}}^{{k}}y_{i}\ell _{i}(x_{j})=0+0+\cdots +y_{ j}+\cdots +0=y_{j}
\end{equation*}

\noindent Polynomials with a value of \textbf{0} at other points are easy to find, for example:\\
$(x-x_{0})\cdots (x-x_{{j-1}})(x-x_{{j+1}})\cdots (x-x_{{k}})$\\

\noindent It takes the value $(x_{j}-x_{0})\cdots (x_{j}-x_{{j-1}})(x_{j}-x_{{j+ 1}})\cdots (x_{j}-x_{{k}})$ at point $x_{j}$.\\

\noindent Since $x_{i}$ has been assumed to be different from each other, the above value is not equal to zero. Thus, by dividing the polynomial by this value, you get a polynomial that satisfies "\textbf{1} at $x_{j}$ and \textbf{0} at other points":

\begin{equation*}
\begin{aligned}
	\ell_j(x) \coloneqq &\prod_{i = 0,i = j}^k
	\frac{x-x_i}{x_j-x_i} \\
	= &\frac{x-x_0}{x_j-x_0} \ldots
	\frac{x-x_{j-1}}{x_j-x_{j-1}}
	\frac{x-x_{j+1}}{x_j-x_{j+1}} \ldots
	\frac{x-x_k}{x_j-x_k}
\end{aligned}
\end{equation*}
\newline
\noindent This is the Lagrangian basic polynomial.

\newpage
\subsection{Proof of uniqueness}

There is only one Lagrangian polynomial with a number not exceeding $k$, because the Lagrangian polynomial does not exceed k for any two times:
$P_{1}$ and $P_{2}$, their difference $P_{1}-P_{ 2}$ is \textbf{0} at all $k+1$ points, so it must be a polynomial $(x-x_{0})(x-x_{{1}})\cdots (x-x_{{k}})$ multiples. \\\\
Therefore, if the difference $P_{1}-P_{2}$ is not equal to \textbf{0}, the number of times must not be less than $k+1$. However, $P_{1}-P_{2}$ is the difference between two polynomials whose number does not exceed $k$, and the number of times does not exceed k. So $P_{1}-P_{2}=0$, that is to say $P_{1}=P_{2}$. This proves the uniqueness.

\newpage
\subsection{Geometric property}

Lagrangian basic polynomial $\ell_{0}, \ell_{1}, \cdots, \ell_{n}$ used in Lagrangian interpolation (by a group $x_{0} < x_{1}< \cdots <x_{n}$ determined) can be thought of as a linear space consisting of polynomials of not more than n a set of bases of ${\mathbb {K}}_{n}[X]$.\\

\noindent First, if there is a set of coefficients: $\lambda _{0}, \lambda _{1}, \cdots, \lambda _{n}$\\
let $P=\lambda _{0}\ell _{0}+\lambda _{1}\ell _{1}+\cdots +\lambda _{n}\ell _{n}=\textbf{0}$\\

\noindent Then, on the one hand, the polynomial $P$ is he Lagrangian interpolation polynomial satisfies: \\
$P(x_{0})=\lambda _{0}, P(x_{1})=\lambda _{1}, \cdots, P(x_{n})=\lambda _{n}$ \\

\noindent On the other hand, $P$ is a zero polynomial, so the value is always \textbf{0}. \\
So:\\
$\lambda _{0}=\lambda _{1}=\cdots =\lambda _{n}=\textbf{0}$\\

\noindent This proves that $\ell _{0}, \ell _{1}, \cdots, \ell _{n}$ are linearly independent. At the same time it contains a total of $n+1$ polynomials, which is exactly equal to the dimension of ${\mathbb {K}}_{n}[X]$. So $\ell _{0}, \ell _{1}, \cdots, \ell _{n}$ form a set of bases for ${\mathbb {K}}_{n}[X]$.\\

\noindent The advantage of the Lagrangian basic polynomial as a base is that all polynomials are homogeneous (both nth polynomials).

\newpage
\section{Problem 2}
\textbf{Question: Give some data}
\[
	\begin{array}{r|r}
	t(days) & 7 \quad 14 \quad\ \ 21 \quad\ \ 28 \quad\ \ 35\quad \ \ 42 \\
	\hline
	P(number\;of\;observed\;flies) & 8\quad41\quad133\quad250\quad280\quad297 \\
	\end{array}
\]

\noindent(a) Use the one-term model to fit the data via the least-squares method.\\
(b) Use fifth-order polynomial to interpolate the data.\\
(c) Use low-order polynomial to minimize the $L_1$ norm, $L_2$ norm and $L_\infty$ norm.

Try second, third order polynomials, respectively.

\subsection{(a)}
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.3]{figure/HW4_2_a.png}
\end{figure}

\begin{center}
$y = (2279*x)/245 - 896/15$
\end{center}

\newpage
\subsection{(b)}
\begin{center}
The Lagrangian interpolation is used to interpolate the data:
\end{center}
\begin{figure}[htbp]
\centering
\includegraphics[scale=.3]{figure/HW4_2_b.png}
\end{figure}
\[ \begin{split}
y = (11*x^5)/84035 &- (145*x^4)/9604 + (1283*x^3)/2058 \\
					&- (2181*x^2)/196 + (9712*x)/105 - 274
\end{split} \]

\newpage
\subsection{(c)}
\begin{figure}[htbp]
\centering
\includegraphics[scale=.3]{figure/HW4_2_c.png}
\end{figure}

Second order polynomials:
\begin{center}
$y = (3714*x)/245 - (41*x^2)/343 - 572/5$
\end{center}

Third order polynomials:
\begin{center}
$y = -(58*x^3)/3087 + (1298*x^2)/1029 - (6185*x)/441 + 48$
\end{center}

\newpage
\subsection{Matlab code}
\begin{lstlisting}[language=matlab]
% HW4_2_a.m
x = [7,14,21,28,35,42];
y = [8,41,133,250,280,297];
p = polyfit(x,y,1);
poly2sym(p)
x1 = linspace(0,49);
y1 = polyval(p,x1);
figure
plot(x,y,'o')
hold on
plot(x1,y1)
hold off

% Hw4_2_b.m
x = [7 14 21 28 35 42];
y = [8 41 133 250 280 297];
x1 = linspace(0,49);
y1 = la(x,y,x1);
plot(x,y,'o')
hold on
plot(x1,y1)
hold off

% la.m
function yy=la(x1,y1,xx)
syms x
n=length(x1);
    for i=1:n
    t=x1;t(i)=[];
    L(i)=prod((x-t)./(x1(i)-t));% L向量用来存放插值基函数
    end
u=sum(L.*y1);
p=simplify(u) % p是简化后的Lagrange插值函数（字符串）
yy=subs(p,x,xx);    % p是以x为自变量的函数，并求xx处的函数值
end

% Hw4_2_c.m
x = [7 14 21 28 35 42];
y = [8 41 133 250 280 297];
p2 = polyfit(x,y,2);
p3 = polyfit(x,y,3);
poly2sym(p2)
poly2sym(p3)
x1 = linspace(0,49);
y2 = polyval(p2,x1);
y3 = polyval(p3,x1);
plot(x,y,'o')
hold on
plot(x1,y2,'-')
plot(x1,y3,'--')
\end{lstlisting}

\end{spacing}

\end{document}


%!TEX root = ../Thesis.tex
\chapter{Arnoldi's Method for Monte Carlo Particle Transport \label{ch:ArnoldiMethod}}
As described in the introduction, the power method has been the method of choice for calculating eigenvalues and eigenvectors for Monte Carlo criticality applications.  The power method is suited well for this type of calculation because of its simple, matrix-free implementation.

The power method, however, has a few drawbacks.  The rate of convergence to the fundamental eigenvalue---determined by the dominance ratio of the first higher-order eigenvalue to the fundamental eigenvalue---can be too small.  It is also limited to calculating one eigenvalue at a time.  Other Krylov subspace methods exist that have some of the same positive benefits of the power method, but also address the negative aspects that make the power method slow and limited.  Chief among these is Arnoldi's method.

Arnoldi's method \cite{Arnoldi:1951The-P-0} is just one such method from the class of Krylov subspace methods.  While its implementation is not as straightforward as with the power method, an explicit form of the transport-fission operator remains unnecessary; we only need to know how to apply the operator to a fission source.  In this sense, Arnoldi's method is as equally suited to Monte Carlo criticality applications as the power method, but it has never been studied in this application.  In this dissertation, the first Monte Carlo application of Arnoldi's method for particle transport is demonstrated.

Arnoldi's method has advantages over the power method that may prove to make the difficulty in its implementation a small matter.  One benefit Arnoldi's method brings to Monte Carlo particle transport is the ability to calculate multiple eigenvalues and eigenvectors with minimal extra computational expense over what is required to calculate the fundamental mode.  In addition, the fission source converges faster in Arnoldi's method than in the power method, reducing the number of inactive iterations required.  In this chapter the basic Arnoldi method and the Monte Carlo implementation for particle transport is introduced.   The ability of Monte Carlo Arnoldi's method to calculate multiple eigenpairs of the fission-transport operator is also demonstrated.  Arnoldi's superior performance in converging the eigenvalue and the fission source is shown in \Fref{ch:AdvancedArnoldi}.

\section{Arnoldi's Method \label{sec:ArnoldiMethod}}
Arnoldi's method generates a Krylov subspace similar to \Fref{eq:KrylovSubspaceBasisVectors} except at each iteration the basis vectors $\left\{v_i\right\}_{i=1}^m$ are orthogonalized against all the previously calculated Arnoldi vectors and normalized.  The basis vectors that form the Krylov subspace are called Arnoldi vectors \citep[see][pp. 435-438]{Watkins:2002Funda-0}.

An Arnoldi process begins with a normalized vector
\begin{equation}
    v_1 = \frac{v}{\left\|v\right\|_2},
\end{equation}
where
\begin{equation}
    \left\|v\right\|_2 = \left(\sum_{i=1}^n\left|v_i\right|^2\right)^{1/2}
\end{equation}
is the Euclidean norm.  We then apply the linear operator to $v_1$ and orthogonalize the result against $v_1$, yielding
\begin{equation}
    \tilde{v}_2 = \A v_1 - \langle\A v_1, v_1\rangle v_1 = \A v_1 - h_{1,1} v_1.
\end{equation}
where
\begin{equation}
    h_{jm} = \langle \A v_m, v_j\rangle
    \label{eq:ArnoldiInnerProduct}
\end{equation}
is the inner product between the vectors $v_m$ and $v_j$ \[\int v_m(x)v_j(x) \dd x.\]  Then we normalize $\tilde{v}_2$
\begin{equation}
    v_2 = \frac{\tilde{v}_2}{\left\|\tilde{v}_2\right\|_2} = \frac{\tilde{v}_2}{h_{2,1}}.
\end{equation}
This process continues iteratively; at the $m$-th iteration we have
\begin{subequations}\begin{gather}
    \tilde{v}_{m+1} = \A v_m - \sum_{j=1}^m h_{jm} v_j \qquad \mathrm{(Orthogonalization)} \label{eq:ArnoldiOrthogonalization} \\[2ex]
    v_{m+1} = \frac{\tilde{v}_{m+1}}{h_{m,m+1}} \qquad \mathrm{(Normalization)}. \label{eq:ArnoldiNormalization}
\end{gather}\end{subequations}

The process of orthogonalization and normalization involves the calculation of the values $h_{jm}$.  These values are the elements of an upper-Hessenberg matrix, $H_{m+1,m}$.  (An upper Hessenberg matrix is upper triangular except that the first subdiagonal is non-zero.)  We can write the $m$-th Arnoldi iteration in matrix form as
\begin{equation}
    \A V_m = V_{m+1}H_{m+1,m},
    \label{eq:ArnoldiMatrixFormulation}
\end{equation}
where the columns of $V_m$ are the Arnoldi vectors and the elements of $H_{m+1,m}$ are the results of the inner products of Arnoldi vectors as described in \Fref{eq:ArnoldiInnerProduct}.  If we separate the last column of $V_{m+1}$ and the last row of $H_{m+1,m}$ we obtain from \Fref{eq:ArnoldiMatrixFormulation}
\begin{equation}
    \A V_m = V_mH_m + v_{m+1}h_{m+1,m}e_m^T
    \label{eq:ArnoldiFactorization}
\end{equation}
where $e_m$ is the $m$-th standard basis vector, $v_{m+1}$ is the Arnoldi vector calculated during the $m$-th Arnoldi iteration and $h_{m+1,m}$ is the normalization factor for the new Arnoldi vector.  Thus we see that at the $m$-th Arnoldi iteration we add a row and a column to $V_mH_m$.

Equation \eqref{eq:ArnoldiFactorization} is called the \emph{Arnoldi factorization} and is an important equation in further analysis of Arnoldi's method.

\subsection{Finding Ritz Pairs from Arnoldi Factorization \label{sec:RitzPairs}}
The Arnoldi process generates an upper-Hessenberg matrix $H_m$ which is the projection of \A{} onto the Krylov subspace defined by the Arnoldi vectors.  Since $H_m$ is generated after a small number of iterations its size is small and we can find its eigenvalues and eigenvectors with relative ease.  The eigenpairs of $H_m$, \mbox{$\left(\mu_i, x_i\right)$} can be used to find Ritz pairs---approximate eigenpairs---of \A.  To see this multiply the Arnoldi factorization, \Fref{eq:ArnoldiFactorization} on the right by an eigenvector of $H_m$, $x_i$
\begin{equation}
    \begin{split}
        \A V_mx_i &= V_m\left( H_mx_i \right) + v_{m+1}h_{m+1,m}e_m^Tx_i \\
        \A V_mx_i &= V_m\left( \mu_ix_i \right) + v_{m+1}h_{m+1,m}e_m^Tx_i \\
        \A y_i &= \mu_iy_i + v_{m+1}h_{m+1,m}e_m^Tx_i,
    \end{split}
    \label{eq:ArnoldiFactorizationRitzPair}
\end{equation}
where $y_i = V_mx_i$. \mbox{$\left(\mu_i, y_i\right)$} is a Ritz pair of \A{} or an approximation to an eigenpair of \A.  The Ritz vector $y_i$ is just the product of the Arnoldi vectors and an eigenvector of $H_m$.

The residual is defined as
\begin{equation}
    r_m \equiv \A y - \mu y.
\end{equation}
We can see from \Fref{eq:ArnoldiFactorizationRitzPair} that the magnitude of the residual is just the magnitude of the last element of the eigenvector $x_i$ times $h_{m+1,m}$
\begin{equation}
    \left|r_m\right| = \left\|\A y - \mu y\right\| = \left|h_{m+1,m}\right|\left|e_mx_i\right|.
    \label{eq:Residual}
\end{equation}
It is easy to see that if the residual is zero, then the Ritz pair is an eigenpair of \A.  The residual is therefore a good indication---but not a guarantee---of the quality of the eigenpair approximation; in fact a small residual guarantees that \mbox{$\left(\mu, x\right)$} is an exact eigenpair of a matrix close to \A{} \citep[see][]{Watkins:2002Funda-0}.

\subsection{Explicitly Restarted Arnoldi \label{sec:ERAM}}
\begin{comment}
The rate at which Arnoldi's method converges to good approximations of eigenpairs depends upon the starting vector.  Take for example starting with the eigenvector associated with the desired eigenvalue, i.e. largest eigenvalue in magnitude.  If we are so fortunate to know this in advance, the Krylov subspace calculated by Arnoldi's method will become invariant after just one iteration and the eigenpair will have been found exactly.  If we desire multiple, say $j$, eigenpairs and the starting vector is a linear combination of the $j$ eigenvectors associated with the desired eigenvalues then Arnoldi's method will be complete (the Krylov subspace will become invariant) in $j$ steps with all the desired eigenpairs known.

Unfortunately we don't know \emph{a priori} the desired eigenvector(s) and eigenvalue(s); if we did, we wouldn't need to perform the calculation.  We may not know beforehand a good guess to the solution, but after a few iterations of Arnoldi's method we have a better estimate of the desired solution than with what we started.  We could start Arnoldi's method over using the estimate of our desired eigenvectors (Arnoldi's Ritz vectors) as the new initial guess.
\end{comment}
As Arnoldi's method proceeds, each iteration adds one Arnoldi vector, and the size of Krylov subspace expands.  The increase in the number of Arnoldi vectors and the size of the Krylov subspace is problematic.  First, the memory requirements increase and second, it is computationally more expensive  because there are more vectors that the newest Arnoldi vector must be orthogonalized against.

Arnoldi's method begins with an estimate of the fundamental eigenvector  After a few Arnoldi iterations we have a better estimate of the fundamental eigenvector than what we started with.  We can therefore restart Arnoldi's method using the better estimate of the eigenvector as the starting vector for the Arnoldi process.  The idea of starting Arnoldi's method using the results of several previous iterations of Arnoldi's method is known as Restarted Arnoldi's method (RAM) and was first proposed by \citet{Saad:1980Varia-0}.

Arnoldi's method can be restarted repeatedly, each time starting with an improved starting vector that is a linear combination of the estimates of the desired eigenvectors of $\A$.  Each sequence of several iterations and a calculation of the Ritz pairs of \A{} is called a \emph{restart}.  Restarting Arnoldi's method saves computational expense by reducing the number of Arnoldi vectors that we must orthogonalize against and reduces memory requirements as fewer Arnoldi vectors must be stored.  Each iteration adds one basis vector to the Krylov subspace so the size of the Krylov subspace is the same as the number Arnoldi vectors.  The size, $m$, of the Krylov subspace is also the size of the upper-Hessenberg matrix \mbox{$H_m \in \mathbb{R}^{m \times m}$} and is therefore the number of eigenpairs that can be estimated in one restart.  Although not necessary, the number of iterations in each restart is generally the same.

The estimated eigenvectors of $\A$ form a basis that we can use in a linear expansion to represent a vector,
\begin{equation}
    \hat{v} = c_1y_1 + c_2y_2 + \cdots + c_ny_n,
\end{equation}
where the $y_i$'s are the estimated eigenvectors of the linear operator $\A$ and the $c_i$'s are some expansion coefficients.  If the $j$ eigenvalues largest in magnitude are desired, then we would like to have our initial vector be
\begin{equation}
    \hat{v} = c_1y_1 + \cdots + c_jy_j + 0\,y_{j+1} + \cdots + 0\,y_n;
\end{equation}
i.e., we want to suppress any information from the undesired portion of the spectrum of $\A$.

The initial vector will most likely contain significant components of all the eigenvectors.  We can reduce the components from eigenvectors from the undesired region of the spectrum by forcing \mbox{$c_{j+1} = \cdots = c_n = 0$}.  The initial vector then becomes
\begin{equation}
    \hat{v} = c_1y_1 + \cdots + c_jy_j.
    \label{eq:ExplicitRestartVector}
\end{equation}
The value of the coefficients \mbox{$c_1, c_2, \ldots, c_j$} are not important and are conveniently chosen to be 1.  When Arnoldi's method is restarted using $\hat{v}$ from \Fref{eq:ExplicitRestartVector} as the initial vector for a new Arnoldi restart, it is an explicit restart and will be referred to as \emph{explicitly} restarted Arnoldi's method (ERAM).  It is implemented by only summing the vectors \mbox{$x_1 + \cdots + x_j$} and ignoring the other eigenvectors.

\section{Monte Carlo Implementation of Explicitly Restarted Arnoldi's Method \label{sec:MCERAM}}
Now that the basic restarted Arnoldi's method has been described, we can proceed to show how it can be implemented in a Monte Carlo particle transport application.  First we note that eigenvalue estimates are made at the end of every restart.  Eigenvalue estimates could have been made at every iteration, but that takes additional computational expense.  A choice was made to calculate and store the eigenvalue estimates at the end of every restart to reduce this cost.  The mean and standard deviation of these stored eigenvalue estimates can be calculated.  In this sense, an Arnoldi restart is similar to a power method iteration.

In conjunction with the eigenvalues, the eigenvectors of the transport-fission operator are estimated at the end of each restart.  The eigenvectors ($y_i$) are normalized such that
\begin{equation}
    \int y_i(x)^2 \dd x = 1
    \label{eq:NormalizedEigenvectors}
\end{equation}
and stored.  The mean of the eigenvectors can also be calculated, but we must be careful that we sum appropriate values of the eigenvectors.  For example, suppose the fundamental eigenvector estimate from one restart has the opposite sign as the fundamental eigenvector estimate from another restart.  (Eigenvectors are unique only up to a multiplicative constant so both estimates are valid.)  Prior to adding a new eigenvector estimate when calculating the mean, the dot product of the new eigenvector estimate with the previous eigenvector estimate is calculated.  If the dot product is negative, the new eigenvector is multiplied by $-1$.  This preserves the normalization given in \Fref{eq:NormalizedEigenvectors}.

The application of the linear operator in \Fref{eq:ArnoldiOrthogonalization} was described in \Fref{sec:ParticleTransport}.  In brief, neutrons are sampled from the fission source $v_k$, transported and the position of the neutrons at the time they initiate fission is stored, creating a new fission source, $\tilde{v}_{k+1}$.

\subsection{Negative Sources \label{sec:NegativeSource}}
When using the power method to calculate the fundamental eigenmode for criticality calculations, we are guaranteed that the solution is everywhere positive and the vectors that are sampled from at each iteration are also everywhere positive.  With Arnoldi's method we are not so fortunate.  The process of orthogonalization guarantees that some of the Arnoldi vectors will be partially negative.  This presents two challenges to Monte Carlo particle transport: first, a negative source means a negative flux which is not physical and second, to sample from a source it must be assumed to be everywhere positive.

To sample from a distribution, it must be everywhere positive and integrate to 1.  If $p(x)$ is a probability distribution function, the quantity $p(x)\dd x$ is the probability of choosing a point in $\dd x$ about $x$.  For a fission source $v(x)$ which may have negative regions it is first normalized such that
\begin{subequations}\begin{align}
    \int \left|v(x)\right| \dd x &= q \\
    p(x) = \frac{\left|v(x)\right|}{q}.
    \label{eq:FissionSourceNormalization}
\end{align}\end{subequations}
With this normalization, the quantity $p(x) \dd x$ is the probability of choosing a point in $\dd x$ about $x$.  A neutron position $x_s$ is sampled from $p(x)$ and is given an initial weight of
\begin{equation}
    \omega = \frac{v(x_s)}{\left|v(x_s)\right|},
    \label{eq:InitialWeight}
\end{equation}
or, alternatively
\begin{equation}
    \omega = \begin{cases}
        1, & v(x_s) > 0 \\
        -1, & v(x_s) < 1.
    \end{cases}
    \label{eq:OtherInitialWeight}
\end{equation}
Neutrons sampled where \mbox{$v(x_s) < 0$} reduce the tally where they score; neutrons sampled where \mbox{$v(x_s) > 0$} contribute positively to the tally where they score.  A neutron is never sampled where \mbox{$v(x_s) = 0$} because there the probability of choosing this point is exactly zero.

Once a particle has been sampled, Monte Carlo transport proceeds as usual, with the neutron scoring $\omega\left(\nu\Sigma_f/\Sigma_T\right)$ in the proper bin at each collision.  If non-analog Monte Carlo is being done, particle weight is reduced at each collision and Russian Roulette is played if the absolute value of the weight becomes small.  Giving a neutron a signed weight does not interfere with any variance reduction or tallying techniques.

At alternative approach to applying \A{} to $v(x)$ this way is to separate $v(x)$ into its positive and negative parts, \vp{} and \vm{} respectively, and apply the transport-fission operator to each part independently
\begin{equation}
    \A v(x) = \A\vp - \A\left|\vm\right|.
\end{equation}
The first approach does this directly by assigning the weights as described.  The first approach is also superior because it samples the positive and negative parts proportionately to the magnitude of the positive or negative part; the second approach would use the same number of particles to apply the linear operator for both the positive and negative part, even if one part is significantly larger than the other.

Once the sampling of a fission source, transporting particles, and creation of a new fission source is completed the new fission source is normalized per source particle and multiplied by integral of the absolute value of the previous source
\begin{equation}
    v_{j+1}(x) = \A v_j(x) \frac{1}{N} \int |v_j(x)| \dd x,
    \label{eq:SourceScaling}
\end{equation}
where $v_j(x)$ is the source we sample from and $\A v(x)$ is the new source created after sampling from $v_j(x)$ and transporting.  This scaling is similar to the scaling performed in the power method shown in \Fref{eq:eVectorIterative} and \Fref{eq:eValueIterative}.  After the vector has been properly scaled, Arnoldi's method continues with orthogonalization and normalization of the newest Arnoldi vector.

\subsection{Spatial Discretization \label{sec:SpatialDiscretization}}
The orthogonalization and normalization of the basis vectors in Arnoldi's method requires taking the inner product of two vectors
\begin{equation}
    \langle v_j,v_k\rangle = \int v_j(x)v_k(x) \dd x.
    \label{eq:InnerProduct}
\end{equation}
In Monte Carlo Arnoldi's method we must take the inner product of two fission sources.  To do this, the fission source is represented as a linear combination of piecewise constant functions
\begin{equation}
    \vP(x) = \sum_{b=1}^B a_b \Pi_b(x),
    \label{eq:source_function}
\end{equation}
where $B$ is the number of spatial bins and
\begin{equation}
    \Pi_b(x) = \begin{cases}
        \left(\frac{1}{\Delta x_b}\right)^{1/2}, & x_b \leq x < x_{b+1} \\
        0, & \mathrm{otherwise},
    \end{cases}
    \label{eq:PiFunction}
\end{equation}
where $\Delta x_b = \left(x_{b+1}-x_b\right)$ is the width of bin $b$.  The term $a_b\sqrt{\Delta x_b}$ is the number of fission neutrons generated in the spatial bin $b$, \mbox{$x \in \left[x_b, x_{b+1}\right)$}.  Note that, to sample from $\vP(x)$, we first sample a bin and then sample uniformly within the bin to determine the position of the particle.

Representing the fission source with a piecewise-constant-in-space approximation, the elements of the Arnoldi vectors are just the expansion coefficients
\begin{equation}
    \vP = \left[a_1, a_2, \ldots, a_B\right]^T,
\end{equation}
and the inner product between two fission sources is defined as
\begin{equation}
    \langle \vP^{(j)},\vP^{(k)}\rangle = \sum_{b=1}^B a_b^{(j)}a_b^{(k)},
    \label{eq:InnerProductFissionSources}
\end{equation}
where $a_b^{(j)}$ and $a_b^{(k)}$ are the expansion coefficients from the fission sources $\vP^{(j)}$ and $\vP^{(k)}$ respectively.

Applying $\A$ to a vector of coefficients $\{a_b\}_{b=1}^B$ simply requires sampling the piecewise constant source function $\vP(x)$ in \Fref{eq:source_function}, transporting these neutrons until they cause another fission, and tallying the resulting fission neutrons over the bins \citep[see][]{Conlin:2008Arnol-0}.  This generates a truncation error.

With this representation of the fission source, the sampling techniques described in \Fref{sec:NegativeSource} can be used and the inner product between two fission sources have been defined.  To estimate the mean eigenvector, we can  simply calculate the mean value of the coefficient in each bin.

\section{Numerical Results \label{sec:ArnoldiResults}}
Everything necessary for a Monte Carlo application of explicitly restarted Arnoldi's method has been described.  To demonstrate how Monte Carlo Arnoldi's method can calculate multiple eigenvalues and eigenvectors of the transport-fission operatorm a few simulations of a homogeneous, semi-infinite slab of multiplying material are shown.  The simulations shown here were chosen to match results published by \cite{Garis:1991One-s-0,Modak:1995Evalu-0} and \cite{Dahl:1979Eigen-0}.  The cross sections are: \mbox{$\nu\Sigma_f = 1.0$}, \mbox{$\Sigma_a = 0.2$}, \mbox{$\Sigma_s = 0.8$} with \mbox{$\Sigma_t = 1.0$}, thus the mean free path for this geometry is $1/\Sigma_t = 1.0$.  I will show the results of slabs with width 0.2, 2.0 or 20 mfp.

For each slab width I have run two simulations; one simulation using Arnoldi's method and the other simulation using the power method for comparison.  In every simulation $10^5$ particles are tracked in an iteration.  The power method has 250 inactive and 1000 active iterations while Arnoldi's method has 25 inactive and 100 active restarts with 10 iterations in each restart.  Therefore the Krylov subspace size is 10.  The number of inactive and active iterations in each method is the same.  The total number of particles tracked in each simulation is also the same.  The slab is discretized into 50 spatial bins for the 0.2 mfp problem and 75 spatial bins for the 2.0 and 20 mfp problems.

\begin{comment}
\begin{table}[h] \centering
    \begin{tabular}{ccccccc}
        \toprule
        Width & Spatial & Particles &\multirow{2}{*}{Method} &  \multirow{2}{*}{Iterations} & Inactive & Active \\
        (mfp) & Bins & per Iteration &&  & Cycles & Cycles \\
        \midrule
        \multirow{2}{*}{0.2} & \multirow{2}{*}{50} & \multirow{2}{*}{$1 \times 10^5$} & Power & -- & 250 & 1000 \\
         & & & Arnoldi & 10 & 25 & 100 \\
        \midrule
        \multirow{2}{*}{2.0} & \multirow{2}{*}{75} & \multirow{2}{*}{$1 \times 10^5$} & Power & -- & 250 & 1000 \\
         & & & Arnoldi & 10 & 25 & 100 \\
        \midrule
        \multirow{2}{*}{20} & \multirow{2}{*}{75} & \multirow{2}{*}{$1 \times 10^5$} & Power & -- & 250 & 1000 \\
         & & & Arnoldi & 10 & 25 & 100 \\
        \bottomrule
    \end{tabular}
    \caption{Problem Parameters.  Power method cycles are power iterations while Arnoldi method cycles are explicit restarts.}
    \label{tab:BasicParameters}
\end{table}
\end{comment}

The results of these simulations are shown in \Fref{tab:BasicResults}, along with the dominance ratio (DR) for each of the three slab widths.  The fundamental eigenvalue estimates are shown for both the power method and Arnoldi's as well as the first and second harmonic eigenvalue estimates from Arnoldi's method.  The published results from \cite{Garis:1991One-s-0,Modak:1995Evalu-0} and \cite{Dahl:1979Eigen-0} are given as the reference.  Almost all the eigenvalue estimates are within one standard deviation of the reference solution.  The only exceptions are the second harmonic estimates for the 2.0 and 20 mfp thick problems and they are both within two standard deviations of the reference solution.

\begin{table}[h]\centering
    \begin{tabular}{cccccc}
        \toprule
        Width & \multirow{2}{*}{Method} & \multirow{2}{*}{Eigenvalue} & Standard & \multirow{2}{*}{Reference} & \multirow{2}{*}{Error} \\
        (mfp) & & & Deviation & \\
        \midrule
        \multirow{2}{*}{0.2} & Power    & 0.329979 & 6.3\e{-5} & 0.330000 & 2.1\e{-5} \\
        \cmidrule{2-6}
        & \multirow{3}{*}{Arnoldi}      &  0.33008 & 1.8\e{-4} &  0.33000 & 8.3\e{-5} \\
        \multirow{2}{*}{DR = 0.23997} & &  0.07911 & 1.5\e{-4} &  0.07919 & 7.6\e{-5} \\
        &                               &  0.04493 & 1.6\e{-4} &  0.04499 & 5.8\e{-5} \\
        \midrule
        \multirow{2}{*}{2.0} & Power    &  2.09593 & 2.7\e{-4} &  2.09599 & 6.0\e{-5} \\
        \cmidrule{2-6}
        & \multirow{3}{*}{Arnoldi}      &  2.09652 & 6.9\e{-4} &  2.09599 & 5.3\e{-4} \\
        \multirow{2}{*}{DR = 0.4015} &  &  0.84183 & 5.8\e{-4} &  0.84150 & 3.3\e{-4} \\
        &                               &  0.48279 & 4.5\e{-4} &  0.48230 & 4.9\e{-4} \\
        \midrule
        \multirow{2}{*}{20} & Power     &  4.82734 & 6.3\e{-4} &  4.82780 & 4.6\e{-4} \\
        \cmidrule{2-6}
        & \multirow{3}{*}{Arnoldi}      &   4.8290 & 1.5\e{-3} &   4.8278 & 1.2\e{-3} \\
        \multirow{2}{*}{DR = 0.9079}&   &   4.3827 & 1.4\e{-3} &   4.3831 & 4.2\e{-4} \\
        &                               &   3.8152 & 1.4\e{-3} &   3.8174 & 2.2\e{-3} \\
         \bottomrule
     \end{tabular}
     \caption{Eigenvalue estimates from power method and Arnoldi's method for slab geometries of width 0.2, 2.0 and 20 mfp.  Reference values taken from \cite{Garis:1991One-s-0}, and \cite{Dahl:1979Eigen-0}. }
     \label{tab:BasicResults}
\end{table}

The figure of merit (FOM) and computational time for each simulation is given in \Fref{tab:BasicFOM}.  The figure of merit is a measure of the efficiency of a Monte Carlo calculation.  The variance of a Monte Carlo eigenvalue calculation goes as $\sqrt{1/N}$, where $N$ is the number of eigenvalue estimates.  The computational expense should be directly proportional to the number of eigenvalue estimates.  The figure of merit is therefore defined to be
\begin{equation}
    \mathrm{FOM} \equiv \frac{1}{\sigma^2 T}
    \label{eq:FOM}
\end{equation}
where $\sigma^2$ is the variance and $T$ is the time required to perform the Monte Carlo calculation.

\begin{table}[h] \centering
    \begin{tabular}{cccccc}
        \toprule
        Width & \multirow{2}{*}{Method} & Fundamental & Standard & \multirow{2}{*}{FOM} & Time \\
        (mfp) & & Eigenvalue & Deviation & & (sec)\\
        \midrule
        \multirow{2}{*}{0.2}    & Power   & 0.329979 & 6.3\e{-5} & 1.7\e{6} &  149.0 \\
                                & Arnoldi &  0.33008 & 1.8\e{-4} & 3.3\e{5} &   95.3 \\
        \cmidrule{2-6}
        \multirow{2}{*}{2.0}    & Power   &  2.09593 & 2.7\e{-4} & 5.5\e{4} &  258.1 \\
                                & Arnoldi &  2.09652 & 6.9\e{-4} & 9.8\e{3} &  212.5 \\
        \cmidrule{2-6}
        \multirow{2}{*}{20}     & Power   &  4.82734 & 6.3\e{-4} & 5.4\e{3} &  463.0 \\
                                & Arnoldi &   4.8290 & 1.5\e{-3} & 1.1\e{3} &  378.5 \\
        \bottomrule
    \end{tabular}
    \caption{Eigenvalue estimates, figure of merit for fundamental eigenvalue from the power method and Arnoldi's method for slab geometries of width 0.2, 2.0, and 20 mfp.}
    \label{tab:BasicFOM}
\end{table}

We see that the figure of merit is always larger for the power method than for Arnoldi's method, but that the Arnoldi simulations run faster.  The figure of merit for the power method is larger than the figure of merit for Arnoldi's method by a factor of 5 while the computational time for the power method is 1.6 times larger than Arnoldi's method for the 0.2 mfp simulation and 1.2 times larger than Arnoldi's method for the 2.0 and 20 mfp simulations.

The power method has many (10 times for these simulations) more eigenvalue estimates because it calculates an estimate after every iteration while Arnoldi's method calculates an estimate after each restart consisting of many iterations.  Thus for the same number of particles tracked (computational expense, $T$) the power method has many more eigenvalue estimates and therefore its variance is smaller, and the FOM is larger.

We can calculate the spread of the eigenvalue estimates from each method.  The spread is the root mean sqared difference of the eigenvalue estimates, $x_i$, from $\overline{x}$, the mean eigenvalue estimate; \[\mathscr{s} \equiv \sqrt{\frac{1}{N}\sum_{i=1}^N \left(x_i - \overline{x}\right)^2}.\]  This should not be confused with the \emph{population} standard deviation \[\sigma \equiv \sqrt{\frac{1}{N-1}\left(\frac{1}{N}\sum_{i=1}^N \left(x_i - \overline{x}\right)^2\right)}\] which is called simply the standard deviation in this dissertation.  The spread of the fundamental eigenvalue estimates from the active iterations/restarts for the three slab widths is shown in \Fref{tab:BasicSpread}.  We can see that even though the standard deviation of the mean is larger in Arnoldi's method than in the power method, the spread of the eigenvalue estimates is smaller in Arnold's method than in the power method.  From this we can conclude that the estimate of the standard deviation (and thus the figure of merit) is dominated by the number of eigenvalue estimates.

The figure of merit isn't a complete comparison between these two methods.  It only measures the efficiency of estimating one eigenvalue, but in Arnoldi's method we have estimates of the first three eigenvalues at no additional cost.  Furthermore, because we are electing to compare multiple eigenvalues, we have fewer eigenvalue estimates to average together in Arnoldi's method.
\begin{table}[h]
    \centering
    \begin{tabular}{cccc}
        \toprule
        & 0.2 mfp & 2.0 mfp & 20 mfp \\
        \midrule
        Power    & 0.0020 & 0.0084 & 0.0201 \\
        Arnoldi  & 0.0018 & 0.0069 & 0.0153 \\
        \bottomrule
    \end{tabular}
    \caption{Spread of active fundamental eigenvalue estimates from the power method and Arnoldi's method for slab geometries of width 0.2, 2.0, and 20 mfp.}
    \label{tab:BasicSpread}
\end{table}

Graphical results for the 20 mfp thick slab are shown in figures \ref{fig:BasicValuesW20}--\ref{fig:BasicFundamentalW20}.  In \Fref{fig:BasicValuesW20} we see the eigenvalue convergence for the fundamental and first two harmonic eigenvalue estimates.  Both inactive and active iterations are shown; the active iterations are the running average of the active eigenvalue estimates.  We can see that the spread of the estimates of the fundamental eigenvalue from Arnoldi's method is smaller than the estimates from the power method.  Black lines are drawn indicating the reference values published in \cite{Garis:1991One-s-0} and \cite{Dahl:1979Eigen-0}.  It appears that all three eigenvalue estimates agree with the reference solution.  However we know from \Fref{tab:BasicResults} that the estimate of the second harmonic is just outside of one standard deviation.

\Fref{fig:BasicFundamentalW20} shows the fundamental eigenvector estimate from the power method and Arnoldi's method as well as a reference solution from an S$_\mathrm{N}$ code.  We see that both the power method and Arnoldi's method accurately estimate the fundamental eigenmode.  \Fref{fig:BasicVectorsW20} shows the fundamental eigenvector and the first two harmonics all calculated by Arnoldi's method.  The higher-order eigenmodes of a semi-infinite slab are similar to the higher modes of the cosine function as expected \citep[see][pg. 173]{Duderstadt:1976Nucle-0}.

The results of the 0.2 and 2.0 mfp simulations are shown  in Figures \ref{fig:BasicValuesW02}--\ref{fig:BasicVectorsW2}.  The results are similar to what we have seen for the 20 mfp problem.

\begin{sidewaysfigure}[hp]\centering
    \input{Arnoldi/Data/BasicValues-w20}
    \caption{Eigenvalue estimates for the power method and Arnoldi's method for the 20 mfp thick slab.  The heavy lines indicate the reference eigenvalues.}
    \label{fig:BasicValuesW20}
\end{sidewaysfigure}

\begin{sidewaysfigure}[hp]\centering
    \input{Arnoldi/Data/BasicFundamental-w20}
    \caption{Fundamental eigenvector estimates from the power method and Arnoldi's method for the 20 mfp thick slab.  The heavy line shows the S$_\mathrm{N}$ solution.}
    \label{fig:BasicFundamentalW20}
\end{sidewaysfigure}

\begin{sidewaysfigure}[hp]\centering
    \input{Arnoldi/Data/BasicVectors-w20}
    \caption{Fundamental and first and second harmonic eigenvector estimates from Arnoldi's method for the 20 mfp thick slab.}
    \label{fig:BasicVectorsW20}
\end{sidewaysfigure}

\begin{figure}\centering
    \subfloat[Eigenvalue estimates]{\label{fig:BasicValuesW02}\input{Arnoldi/Data/BasicValues-w0.2}}

    \subfloat[Eigenvector Estimates]{\label{fig:BasicVectorsW02}\input{Arnoldi/Data/BasicVectors-w0.2}}
    \caption{Eigenvalue and eigenvector estimates from power method and Arnoldi's method for the 0.2 mfp thick slab.  Heavy lines show reference solution from \cite{Garis:1991One-s-0} and \cite{Dahl:1979Eigen-0}.}
\end{figure}

\begin{comment}
\begin{sidewaysfigure}[hp]\centering
%    \includegraphics[width=\textwidth, keepaspectratio]{Arnoldi/Data/BasicFundamental-w02}
    \input{Arnoldi/Data/BasicFundamental-w0.2}
    \caption{Fundamental eigenvector estimates from the power method and Arnoldi's method for the 0.2 mfp wide slab.  The heavy line shows the S$_\mathrm{N}$ solution.}
    \label{fig:BasicFundamentalW02}
\end{sidewaysfigure}
\end{comment}

\begin{figure}\centering
    \subfloat[Eigenvalue estimates]{\label{fig:BasicValuesW2}\input{Arnoldi/Data/BasicValues-w2}}

    \subfloat[Eigenvector estimates]{\label{fig:BasicVectorsW2}\input{Arnoldi/Data/BasicVectors-w2}}
    \caption{Eigenvalue and eigenvector estimates from power method and Arnoldi's method for the 2.0 mfp thick slab.  Heavy lines show reference solution from \cite{Garis:1991One-s-0} and \cite{Dahl:1979Eigen-0}.}

\end{figure}

\begin{comment}
\begin{sidewaysfigure}[hp]\centering
%    \includegraphics[width=\textwidth, keepaspectratio]{Arnoldi/Data/BasicFundamental-w2}
    \input{Arnoldi/Data/BasicFundamental-w2}
    \caption{Fundamental eigenvector estimates from the power method and Arnoldi's method for the 2.0 mfp wide slab.  The heavy line shows the S$_\mathrm{N}$ solution.}
    \label{fig:BasicFundamentalW2}
\end{sidewaysfigure}
\end{comment}

\clearpage
\subsection{Discretization Error \label{sec:DiscretizationBias}}

One of the benefits of Monte Carlo particle transport is the ability to use exact geometry without discretization.  This is true for the transport of particles in the power method, but the fission source must be discretized for tallying.  Arnoldi's method, on the other hand, must have a discretized source  in order to orthogonalize the Arnoldi vectors, as described in \Fref{sec:SpatialDiscretization}.

The discretization of the fission source can lead to an error in the estimated eigenvalue if an insufficient number of spatial bins are used to represent the fission source.  To illustrate this effect a series of simulations is shown using the same slab of multiplying material with varying number of spatial bins.  The slab here is exactly the same as for the 20 mfp problem shown earlier (\mbox{$\nu\Sigma_f = 1.0$}, \mbox{$\Sigma_a = 0.2$}, \mbox{$\Sigma_s = 0.8$} with \mbox{$\Sigma_t = 1.0$}).  This time $10^5$ histories are tracked per iteration with 50 inactive restarts and 500 active restarts.  The increase in the number of histories and restarts is to reduce the statistical uncertainty to ensure the error can be seen outside the noise.  This simulation was performed eleven times varying the number of spatial discretization bins from 10 to 150.

The results of these simulations are shown in \Fref{tab:Discretization} for the fundamental eigenvalue.  We can see that the uncertainty in the eigenvalue estimate (standard deviation) is relatively independent of the number of spatial bins.  The error in the eigenvalue estimate is the absolute value of the difference between the eigenvalue estimate and the reference solution.  We see that the error in the eigenvalue estimate is larger than the statistical uncertainty for bin widths \mbox{$\leq 0.5$} mfp thick.  For bin widths greater than 0.5 mfp thick the error is less than the statistical uncertainty.

The data from \Fref{tab:Discretization} is shown graphically in \Fref{fig:BasicBias}.  The error in the eigenvalue estimate for the first two harmonics are also shown in \Fref{fig:BasicBias}.  The error in the eigenvalue estimate is denoted marked as $\mathcal{B}$ for each of the eigenvalue estimates.  Best fit lines are drawn through the points on the graph.  We see that there is a very good linear fit to these data points.

\begin{table}[h] \centering
    \begin{tabular}{cccccc}
        \toprule
        \# of Bins & Bin Width (mfp) & Eigenvalue & Uncertainty & Error & FOM\\
        \midrule
         10 & 2.00 & 4.8003 & 6.6\e{-4} & 2.7\e{-2} & 831.2 \\
         25 & 0.80 & 4.8224 & 6.8\e{-4} & 5.3\e{-3} & 773.2 \\
         40 & 0.50 & 4.8251 & 6.3\e{-4} & 2.6\e{-3} & 872.0 \\
         50 & 0.40 & 4.8273 & 6.5\e{-4} & 4.2\e{-4} & 829.0 \\
         60 & 0.33 & 4.8258 & 6.9\e{-4} & 2.0\e{-3} & 704.3 \\
         75 & 0.27 & 4.8275 & 6.7\e{-4} & 2.4\e{-4} & 753.0 \\
         90 & 0.22 & 4.8277 & 6.7\e{-4} & 4.2\e{-5} & 746.1 \\
        105 & 0.19 & 4.8277 & 6.9\e{-4} & 5.2\e{-5} & 698.3 \\
        120 & 0.17 & 4.8282 & 6.5\e{-4} & 4.1\e{-4} & 767.8 \\
        135 & 0.15 & 4.8274 & 7.0\e{-4} & 3.1\e{-4} & 656.9 \\
        150 & 0.13 & 4.8285 & 6.4\e{-4} & 7.5\e{-4} & 792.0 \\
        \bottomrule
    \end{tabular}
    \caption{Eigenvalue estimates of the fundamental eigenvalue from Arnoldi's method and the error in the estimate.  The error is the difference between the estimate and the reference value ($\lambda_0 = 4.8278$, see \cite{Garis:1991One-s-0} and \cite{Dahl:1979Eigen-0}).}
    \label{tab:Discretization}
\end{table}

\begin{sidewaysfigure}[h] \centering
    \input{Arnoldi/Data/BiasHistogram}
    \caption{Discretization error for Arnoldi's method.  The error is the difference between the eigenvalue estimate from Arnoldi's method and the reference value given in \Fref{tab:BasicResults}.}
    \label{fig:BasicBias}
\end{sidewaysfigure}

\section{Variance}
One of the problems with Monte Carlo particle transport is the underestimation of the variance of the mean eigenvalue estimate.  This topic has received considerable attention lately \cite{Brown:2009A-Rev-0}.  In this section I will investigate how this issue manifests itself in Arnoldi's method.

The process of using the fission source calculated in a previous iteration as the source of neutrons for the current iteration causes the uncertainty in the eigenvalue (or some other tally) to be too small.  The mean $\overline{X}$ and standard deviation $\sigma_{\overline{X}}$ for some tally $X$ are calculated as
\begin{subequations}
    \begin{gather}
        \overline{X} = \frac{1}{N}\sum_{n=1}^N X_n, \label{eq:Mean} \\
        \sigma_{\overline{X}} = \left[\frac{1}{N-1}\left(\frac{1}{N}\sum_{n=1}^NX_n^2\right) - \overline{X}^2\right]^{\sfrac{1}{2}}, \label{eq:StD}
    \end{gather}
    \label{eq:MeanStD}
\end{subequations}
where $X_n$ is one estimate of the tally and $N$ is the number of estimates.  Equations \eqref{eq:MeanStD} assume that each estimate is independent of all the others.  Because of the procedure of using previous sources to generate the next source, the sources are correlated.  \citet{Kiedrowski:1009An-In-0} explain it best, ``If the concentration of fission neutrons at a location within a cycle [iteration] is statistically high, the concentration of fission neutrons in the next cycle is likely to be higher than average as well.  The same applies if the concentration is statistically low.  This implies a positive correlation between the fission source distributions.''

Using \Fref{eq:StD} to estimate the standard deviation with correlated estimates causes the calculated standard deviation to be too small \cite[see][]{Brown:2009A-Rev-0}.  A standard deviation that is too small would give immoderate confidence in the eigenvalue.

\subsection{Numerical Results}
While there is no immediate way to reduce or eliminate the correlation between iterations, we can calculate the true standard deviation by running many independent, identical simulations and compute the mean and standard deviation of all of these.  This can then be compared to the standard deviation of an individual run.

For this calculation a 50 mfp thick, homogeneous slab is used with cross sections: \mbox{$\nu\Sigma_f = 1.0$}, \mbox{$\Sigma_a = 0.2$}, and \mbox{$\Sigma_s = 0.8$}; \mbox{$\Sigma_t = 1.0$}.  The fundamental eigenvalue for this geometry is 0.997520.  Both the power method and Arnoldi's method are run so as to compare the results.  In Arnoldi's method, 100 inactive and 100 active restarts are used with 25 iterations per restart.  For the power method 2500 inactive and 2500 active iterations are used.  Both methods track 500,000 particles per iteration.  Each method has 100 independent simulations.

The results of this study are shown in \Fref{tab:TrueVariance}.  I show the mean of the eigenvalue estimate from the 100 simulations, the mean of the standard deviations from the simulations and the true standard deviation.  The true standard deviation is the standard deviation of eigenvalue estimates from all the simulations, while the mean reported standard deviation is the mean of the reported standard deviations from the simulations.
\begin{table}[h]\centering
    \begin{tabular}{ccccc}
        \toprule
        \multirow{2}{*}{Method} & Mean & Mean Reported & True Standard & Percent\\
        & Eigenvalue & Standard Deviation & Deviation & Difference \\
        \midrule
        Power   &  0.99752 & 2.4\e{-5} & 2.7\e{-5} & -11.1 \\
        Arnoldi &  0.9974  & 1.1\e{-4} & 9.7\e{-5} &  13.4 \\
        \bottomrule
    \end{tabular}
    \caption{Mean eigenvalue estimate of fundamental eigenvalue from Arnoldi's method and the power method from 100 independent simulations.  The mean reported standard deviation is the mean of the standard deviation from the 100 independent simulations.  The true standard deviation is the standard deviation of the eigenvalue estimates from the 100 independent simulations.  The difference is (Reported-True)/True.}
    \label{tab:TrueVariance}
\end{table}

We see from these results that both Arnoldi's method and the power method report a standard deviation that is different from the true standard deviation by about 10\%.  The problem is that the power method underpredicts the standard deviation.  For Arnoldi's method we can have confidence that---at least for this problem---the reported standard deviation is larger than true standard deviation.

\section{Summary}
In this chapter the basic explicitly restarted Arnoldi's method for Monte Carlo particle transport has been described.  It has been shown that Monte Carlo Arnoldi's method estimates the fundamental eigenvalue as well as first two higher-order eigenmodes within statistical uncertainty of published results; the eigenvectors are similar to cosine functions as they are expected to be.

Arnoldi's method suffers from two problems; the discretization of the fission source causes an error in the estimate of the eigenvalue, and a smaller figure of merit than the power method.  These issues are addressed in \Fref{ch:SpatialDiscretization} and \Fref{ch:RelaxedArnoldi} respectively.


\chapter{Optimization Problems}
\label{sec:optProGenOpt}
\section{Classification of Optimization Problems}
\label{sec:claOptPro}
We will now classify some optimization problems that can be solved with GenOpt's
optimization algorithms. The classification will be used
in Section~\ref{sec:AlgSel} to recommend suitable optimization algorithms.\\

We distinguish between problems whose
design parameters are continuous
variables\footnote{Continuous variables can take on any value on the real line,
possibly between lower and upper bounds.}, discrete variables\footnote{Discrete
variables can take on only integer values.},
or both.
In addition, we distinguish between
problems with and without inequality constraints on the dependent variables.

% ----------------------------------------------------------------
\subsection{Problems with Continuous Variables}
To denote box-constraints on independent continuous variables,
we will use the notation
\begin{equation}
  \mathbf X \triangleq \bigl\{ x \in \Re^n \ | \
  l^i \le x^i \le u^i, \ i \in \{1, \ldots, n \} \bigr\},
\label{eq:setXPc}
\end{equation}
where $ - \infty \le l^i < u^i \le \infty$ for $i \in \{1, \ldots, n \}$.\\

We will consider optimization problems of the form
\begin{eqnarray}
\lefteqn{ \mathbf P_{c}   } \qquad && \min_{x \in \mathbf X} f(x),
\label{sub:Proc}
\end{eqnarray}
where $f \colon \Re^n \to \Re$ is a once continuously differentiable cost function.

\begin{subequations}
Now, we add inequality constraints on the dependent variables to~\eqref{sub:Proc} and obtain
\begin{eqnarray}
\lefteqn{ \mathbf P_{cg}   } \qquad && \min_{x \in \mathbf X} f(x), \\
&& g(x) \le 0,
\end{eqnarray}
where everything is as in \eqref{sub:Proc} and, in addition,
$g \colon \Re^n \to \Re^m$ is a once continuously differentiable constraint function
(for some $m \in \Na$).
We will assume that there exists an $x^* \in \mathbf X$ that satisfies $g(x^*) < 0$.\\
\label{sub:Procg}
\end{subequations}

% ----------------------------------------------------------------
\subsection{Problems with Discrete Variables}
Next, we will discuss the situation where all design parameters
can only take on user-specified discrete values.\\

Let $\mathbf X_d \subset \Z^{n_d}$ denote the constraint set
with a finite, non-zero number of integers for each variable.\\

We will consider integer programming problems of the form
\begin{eqnarray}
\lefteqn{ \mathbf P_{d}   } \qquad && \min_{x \in \mathbf X_d} f(x).
\label{sub:Prod}
\end{eqnarray}

% ----------------------------------------------------------------
\subsection{Problems with Continuous and Discrete Variables}
Next, we will allow for continuous and discrete independent variables.\\

\begin{subequations}
We will use the notation
\begin{eqnarray}
\mathbf X & \triangleq & \mathbf X_c \times \mathbf X_d, \\
\mathbf X_c & \triangleq & \bigl\{ x \in \Re^{n_c} \ | \
l^i \le x^i \le u^i, \ i \in \{1, \ldots, n_c \} \bigr\},
\label{eq:feaSetXc}
\end{eqnarray}
where the bounds on the continuous independent variables satisfy
$ - \infty \le l^i < u^i \le \infty$ for $i \in \{1, \ldots, n_c \}$,
and the constraint set $\mathbf X_d \subset \Z^{n_d}$ for the discrete variables
is a user-specified set with a finite, non-zero number of integers for each variable.\\
\label{sub:setXd}
\end{subequations}

We will consider mixed-integer programming problems of the form
\begin{subequations}
\begin{eqnarray}
\lefteqn{ \mathbf P_{cd}   } \qquad && \min_{x \in \mathbf X} f(x), \\
\end{eqnarray}
\label{sub:Procd}
\end{subequations}
where $x \triangleq (x_c, x_d) \in \Re^{n_c} \times \Z^{n_d}$,
$f \colon \Re^{n_c} \times \Z^{n_d} \to \Re$ and $\mathbf X$ is as in~\eqref{sub:setXd}.

\begin{subequations}
Now, we add inequality constraints on the dependent variables to~\eqref{sub:Procd} and obtain
\begin{eqnarray}
\lefteqn{ \mathbf P_{cdg}   } \qquad && \min_{x \in \mathbf X} f(x), \\
&& g(x) \le 0,
\end{eqnarray}
\label{sub:Procdg}
\end{subequations}
where everything is as in \eqref{sub:Procd} and in addition
$g \colon \Re^{n_c} \times \Re^{n_d} \to \Re^m$ (for some $m \in \Na$).
We will assume that there exists an $x^* \in \mathbf X$ that satisfies
$g(x^*) < 0$.\\

% ----------------------------------------------------------------
\subsection[Problems that use a Building Simulation Program]{Problems whose Cost Function is Evaluated by a Building Simulation Program}
\label{sec:proAppCosFun}
Next, we will discuss problem $\mathbf P_{c}$ defined in~\eqref{sub:Proc} for
the situation where the cost function $f \colon \Re^n \to \Re$
cannot be evaluated,
but can be approximated numerically by approximating cost functions
$f^* \colon \Re_+^p \times \Re^n \to \Re$,
where the first argument is the precision parameter of the numerical solvers.
This is typically the case when the cost is computed
by a thermal building simulation program,
such as
EnergyPlus~\cite{Crawley2001:1},
TRNSYS~\cite{KleinDuffieBeckman1976}, or
DOE-2~\cite{WinkelmannBirsdall1993}.
In such programs,
computing the cost involves
solving a system of partial and
ordinary differential equations that are coupled to algebraic equations.
In general, one cannot obtain an exact solution, but
one can obtain an approximate numerical solution.
Hence,
the cost function $f(x)$ can only be approximated by an approximating cost function
$f^*(\epsilon,x)$,
where $\epsilon \in \Re_+^q$ is a vector that contains precision parameters of
the numerical solvers.
Consequently, the optimization algorithm can only be applied to $f^*(\epsilon,x)$
and not to $f(x)$.\\

In such thermal building simulation programs
it is common that the termination criteria of the solvers that are used
to solve the partial differential equations, ordinary differential equations, and
algebraic equations depend on the independent variable $x$.
Therefore, a perturbation of $x$ can cause
a change in the sequence of solver iterations,
which causes the approximating cost functions $f^*(\epsilon,x)$
to be discontinuous in $x$.
Furthermore, if variable step size integration methods are used,
then the integration mesh can change from one simulation to the next.
Therefore, part of the change in function values between different points is
caused by a change of the number of solver iterations, and
by a change of the integration mesh.
Consequently, $f^*(\epsilon,\cdot)$ is discontinuous, and
a descent direction for $f^*(\epsilon,\cdot)$ may not be a descent direction
for $f(\cdot)$.
Therefore, optimization algorithms
can terminate at points that are non-optimal.\\

The best one can do in trying to solve optimization problems where the cost and constraint functions are evaluated by a thermal building simulation program
that does not allow controlling the approximation error
is to find points that are close to a local minimizer of $f(\cdot)$.
Numerical experiments show that by using tight enough precision and
starting the optimization algorithm with coarse initial values,
one often comes close to a minimizer of $f(\cdot)$.
Furthermore, by selecting different initial iterates for the optimization,
or by using different optimization algorithms, one can increase the chance of
finding a point that is close to a minimizer of $f(\cdot)$.
However, even if the optimization terminates at a point
that is non-optimal for $f(\cdot)$,
one may have obtained a better system performance compared to not doing any optimization.\\

See~\cite{WetterPolak2003:1,WetterWright2003:1} for a further discussion
of optimization problems in which the cost function value is computed
by a building simulation program.

% ----------------------------------------------------------------
\section{Algorithm Selection}
\label{sec:AlgSel}
In this section, we will discuss which of GenOpt's algorithms can be selected
for the optimization problems that we introduced in Section~\ref{sec:claOptPro}.\\

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{c}$ with $n>1$}
To solve $\mathbf P_{c}$ with $n>1$,
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg}) or
the GPS implementation of the Hooke-Jeeves algorithm
(Section~\ref{sec:GPSHooJeeImp}, page~\pageref{sec:GPSHooJeeImp}) can be used,
possibly with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}).
If $f(\cdot)$ is once continuously differentiable
and has bounded level sets (or if the constraint set $\mathbf X$
defined in~\eqref{eq:setXPc} is compact)
then these algorithms construct for problem~\eqref{sub:Proc}
a sequence of iterates with stationary accumulation points
(see Theorem~\ref{the:feaPoiConv}).

Alternatively, the Discrete Armijo Gradient algorithm
(Section~\ref{sec:DAG}, page~\pageref{sec:DAG}) can be used.
Every accumulation point of the
Discrete Armijo Gradient algorithm is a feasible stationary point.\\

If $f(\cdot)$ is not continuously differentiable, or if
$f(\cdot)$ must be approximated by an approximating cost function $f^*(\epsilon,\cdot)$
where the approximation error cannot be controlled,
as described in Section~\ref{sec:proAppCosFun}, then $\mathbf P_c$ can only be solved
heuristically.
We recommend using
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg}),
the GPS implementation of the Hooke-Jeeves algorithm
(Section~\ref{sec:GPSHooJeeImp}, page~\pageref{sec:GPSHooJeeImp}),
possibly with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}),
or a Particle Swarm Optimization algorithm
(Section~\ref{sec:PSOAlg}, page~\pageref{sec:PSOAlg}).\\

We do not recommend using the Nelder-Mead Simplex algorithm
(Section~\ref{sec:simAlgNelMea}, page~\pageref{sec:simAlgNelMea}) or
the Discrete Armijo Gradient algorithm
(Section~\ref{sec:DAG}, page~\pageref{sec:DAG}).\\

The following approach reduces the risk of failing at a point which is
non-optimal and far from a minimizer of $f(\cdot)$:
\begin{enumerate}
\item
Selecting large values for the parameter \texttt{Step} in
the optimization command file (see page~\pageref{ite:parStep}).
\item
Selecting different initial iterates.
\item
Using the hybrid algorithm of Section~\ref{sec:GPSPSOAlg},
the GPS implementation of the Hooke-Jeeves algorithm,
possibly with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}),
and/or
a Particle Swarm Optimization algorithm
and select the best of the solutions.
\item
Doing a parametric study around the solution that has been obtained by
any of the above optimization algorithms.
The parametric study can be done using the algorithms \texttt{Parametric}
(Section~\ref{sec:algParametric}, page~\pageref{sec:algParametric})
and/or
\texttt{EquMesh}
(Section~\ref{sec:algParRunGen}, page~\pageref{sec:algParRunGen}).
If the parametric study yields a further reduction in cost,
then the optimization failed at a non-optimal point.
In this situation, one may want to try another optimization algorithm.
\end{enumerate}

If $f(\cdot)$ is continuously differentiable but
must be approximated by approximating cost functions $f^*(\epsilon,\cdot)$
where the approximation error can be controlled
as described in Section~\ref{sec:proAppCosFun},
then $\mathbf P_c$ can be solved using
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg}) or
the GPS implementation of the Hooke-Jeeves algorithm
(Section~\ref{sec:GPSHooJeeImp}, page~\pageref{sec:GPSHooJeeImp}),
both with the error control scheme described in
the Model GPS Algorithm~\ref{al:GPSImp}
(page~\pageref{al:GPSImp}).
The GPS implementation of the Hooke-Jeeves algorithm can be used
with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}).
The error control scheme can be implemented using the value of GenOpt's variable
\texttt{stepNumber} (page~\pageref{sec:ImpWeiFac})
and GenOpt's pre-processing capabilities
(Section~\ref{par:posPro}, page~\pageref{par:posPro}).
A more detailed description of how to use the error control scheme can
be found in~\cite{PolakWetter2003:1,WetterPolak2003:1}.

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{cg}$ with $n>1$}
To solve $\mathbf P_{cg}$,
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg})
or
the GPS implementation of the Hooke-Jeeves algorithm
(Section~\ref{sec:GPSHooJeeImp}, page~\pageref{sec:GPSHooJeeImp})
can be used,
possibly with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}).
Constraints $g(\cdot) \le 0$ can be implemented
using barrier and penalty functions
(Section~\ref{cha:conGen}, page~\pageref{cha:conGen}).\\

If $f(\cdot)$ or $g(\cdot)$ are not continuously differentiable,
we recommend using
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg})
or
the GPS implementation of the Hooke-Jeeves algorithm
(Section~\ref{sec:GPSHooJeeImp}, page~\pageref{sec:GPSHooJeeImp}),
possibly with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}),
and implement the constraints $g(\cdot) \le 0$
using barrier and penalty functions
(Section~\ref{cha:conGen}, page~\pageref{cha:conGen}).
To reduce the risk of terminating far from a minimum point of $f(\cdot)$,
we recommend the same measures as for solving $\mathbf P_c$.\\

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{c}$ with $n=1$}
To solve $\mathbf P_c$ with $n=1$,
any of the interval division algorithms can be used
(Section~\ref{sec:IntDivAlg}, page~\pageref{sec:IntDivAlg}).
Since only a few function evaluations are required for parametric studies
in one dimension, the algorithm \texttt{Parametric} can also be used for this problem
(Section~\ref{sec:algParametric}, page~\pageref{sec:algParametric}).
We recommend doing a parametric study if $f(\cdot)$ is expected to have
several local minima.

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{cg}$ with $n=1$}
To solve $\mathbf P_{cg}$ with $n=1$,
the same applies as for $\mathbf P_{c}$ with $n=1$.
Constraints $g(\cdot) \le 0$ can be implemented by setting
the penalty weighting factor $\mu$ in~\eqref{eq:penFun} to a large value.
This may still cause small constraint violations,
but it is easy to check whether the violation is acceptable.

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{d}$}
To solve $\mathbf P_{d}$,
a Particle Swarm Optimization algorithm can be used
(Section~\ref{sec:PSOAlg}, page~\pageref{sec:PSOAlg}).

% ----------------------------------------------------------------
\subsection{Problem $\mathbf P_{cd}$ and $\mathbf P_{cdg}$}
To solve $\mathbf P_{cd}$, or $\mathbf P_{cdg}$,
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg})
or
a Particle Swarm Optimization algorithm can be used
(Section~\ref{sec:PSOAlg}, page~\pageref{sec:PSOAlg}).

% ----------------------------------------------------------------
\subsection{Functions with Several Local Minima}
If the problem has several local minima, we recommend using
the GPS implementation of the Hooke-Jeeves algorithm
with multiple starting points (Section~\ref{sec:GPSMulSta}, page~\pageref{sec:GPSMulSta}),
the hybrid algorithm (Section~\ref{sec:GPSPSOAlg}, page~\pageref{sec:GPSPSOAlg}), or
a Particle Swarm Optimization algorithm
(Section~\ref{sec:PSOAlg}, page~\pageref{sec:PSOAlg}).


\subsection{Binary mixture LMN flow}

In general, the dimensional system of governing equations reads under the LMN approximation as
\begin{align}
&\frac{\partial {\rho}}{\partial t}+\frac{\partial }{\partial x_i} \bigg({\rho} {u}_i\bigg) = 0, \label{mass}\\
&\frac{\partial {\rho}{ u}_j }{\partial t}+ \frac{\partial }{\partial x_i} \bigg({\rho} {u}_j {u}_i \bigg) =-\frac{\partial{ P}}{\partial x_j} + \frac{\partial{\tau}_{ij}}{\partial x_i}+{\rho} g_j, \label{mom}\\
&  \frac{\partial{\rho}{ Y}_1}{\partial t}+ \frac{\partial }{\partial x_i} \bigg({\rho} {u}_i {Y}_1\bigg)=\frac{\partial }{\partial x_i}\bigg( {\rho} \  D \frac{\partial {Y }_1}{\partial x_i}\bigg),\label{spec}\\
&{\rho}={M}_{mix}\frac{p}{RT}, \label{state}
\end{align}
where ${\rho}$ is the mixture's density and ${u}_i$ the mass average component of the velocity vector ${\textbf{u}}=({u}_1,{u}_2,{u}_3)$ and $g_j=(0,0,-g)$ the gravity vector. ${Y}_1$ and ${Y}_2$  are respectively the mass fractions of species 1 ans species 2 satisfying ${Y}_1+{Y}_2=1$.

$D$ corresponds to the mixture diffusion coefficient of both species,  $R= 8.314$ J.K$^{-1}$.mol$^{-1}$ the specific gas constant and $M_{mix}=\displaystyle\left(\sum_{i=1}^2  \frac{{Y}_i}{M_i}\right)^{-1}$ the mixing molar mass where  $M_1=M_{in}$ and $M_2 =M_{am}$, the injection and ambient molar mass respectively.

 ${\tau}_{ij}= 2\mu{e}_{ij}$ is the viscous stress tensor for Newtonian fluids with
$${e}_{ij}= \frac{1}{2}\left(\frac{\partial  u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i}\right) -\frac{1}{3}\delta_{ij}\frac{\partial u_k}{\partial x_k},$$
and $\delta_{ij}$ the Kronecker symbol.  $\mu$ denotes the mixture's dynamic viscosity calculated as a function of the mass fractions and fluids physical properties using the Wilke's formulation as follows
\begin{equation}\label{wilke}
        \mu = \frac{Y_1\mu_1}{Y_1\phi_{11}+Y_2\phi_{12}}+\frac{Y_2\mu_2}{Y_1\phi_{21}+Y_2\phi_{22}},
\end{equation}
        where $\phi_{ij}$ is a set of dimensionless constants calculated as
\begin{equation}
        \phi_{ij}= \displaystyle\frac{\displaystyle\frac{\textrm{M}_i}{\textrm{M}_j}\left[ 1+\left(\displaystyle\frac{\mu_i}{\mu_j} \right)^{1/2} \left( \displaystyle\frac{\textrm{M}_j}{\textrm{M}_i} \right)^{1/4}\right]^2}{ \left[ 8\left(1+\displaystyle\frac{\textrm{M}_i}{\textrm{M}_j} \right) \right]^{1/2}}\quad : \quad i,j=\{1,2\}.
\end{equation}

In this study, we consider identical species (inj = amb) with
\begin{enumerate}
  \item $D = 7.72\times 10^{-5}$ m$^{2}$.s$^{-1}$,
  \item $p = 100000$ Pa,
  \item $T= 284.15$ K,
  \item $M_1=M_2=0.02897$ kg.mol$^{-1}$,
  \item $\mu_1=\mu_2=1.792\times 10^{-5}$ kg.m$^{-1}$.s$^{-1}$.
\end{enumerate}

\subsection{Non-compressible thermo-hydraulic flow}

The dimensional system of governing equations of a non-compressible thermo-hydraulic flow (without a source term) is expressed as

\begin{equation}\label{ns1}
\frac{\partial { u}_j }{\partial t}+ {u}_j\frac{\partial {u}_i }{\partial x_j}=-\frac{\partial{ P^{\ast}}}{\partial x_j} + \frac{\partial}{\partial x_i} \bigg( \nu \frac{\partial  u_i}{\partial x_j}\bigg),
\end{equation}
\begin{equation}\label{ns2}
\frac{\partial T}{\partial t}+  {u}_i\frac{\partial T}{\partial x_i}=\frac{\partial }{\partial x_i}\bigg( {\rho} \frac{\lambda}{\rho c_p} \frac{\partial T}{\partial x_i}\bigg),
\end{equation}
where $\nu=\mu/\rho$ is the kinematic viscosity, $P^{\ast}$ the reduced pressure expressed as a function of the pressure $P$,
the density $\rho$ and the gravity vector $\textbf{g}$ as
$$P^{\ast}=P/\rho + \textbf{g}.$$

The physical properties are considered in this case in accordance to the LMN problem, and described as follows
\begin{enumerate}
  \item $\lambda = 9.466369670132119\times 10^{-5}$ m$^{2}$.s$^{-1}$,
  \item $c_p=1$ J.kg$^{-1}$.K$^{-1}$
  \item $\rho=1.226213687840948$ kg.m$^{-3}$,
  \item $\mu=1.792\times 10^{-5}$ kg.m$^{-1}$.s$^{-1}$.
\end{enumerate}

\subsection{Post processing integral quantities}

To validate the implementation of the binary mixture LMN flow problem, comparisons are provided by comparing two integral quantities
\begin{enumerate}
  \item $I_1 =\displaystyle\int_V \rho Y_1 dV$,
  \item $I_2 =\displaystyle\int_V \rho c_p T dV$,
\end{enumerate}
which denote respectively the total mass of the injected species and the total energy.


\section*{Exercises}

\begin{ex}
  In this section, we observed that $\rho V^2AB$ has
  the units of force. Describe a systematic way to obtain such
  combinations of the variables that will yield something that has
  the units of force.
\end{ex}