%!TEX root = forallx-ubc.tex
\chapter{Soundness and Completeness for QL Trees}
\label{ch.QLsoundcomplete}

In Chapter \ref{ch.SLsoundcomplete} we proved that our SL tree method works the way it is supposed to: any tree that closed was guaranteed to have a root that was unsatisfiable in SL, and any completed tree that remained open was guaranteed to describe an SL model that satisfies the root. These two theorems together are called \emph{soundness} and \emph{completeness}.

\factoidbox{
\define{soundness}: If a tree closes, that guarantees that its root is unsatisfiable. In other words: $$\metaSetX{}\vdash{}\bot\Rightarrow\metaSetX{}\models{}\bot$$
\define{completeness}: If a root is unsatisfiable, that guarantees that the tree will close. In other words: $$\metaSetX{}\models\bot{}\Rightarrow\metaSetX{}\vdash{}\bot$$
}

These definitions were first given on pages \pageref{definesound} and \pageref{definecomplete}. They are equally applicable to our extended tree system for QL. The system introduced in Chapter \ref{ch.QLTrees} is also sound and complete. The rules are designed in a way so as to guarantee that if the tree closes, there is no model for the root (sound), and to guarantee that an open branch describes a model for the root (complete). The proofs for these two theorems are structurally very similar to the proofs given in Chapter \ref{ch.SLsoundcomplete}.

\section{Soundness}

If our tree method is sound, then there is no possible set of QL sentences \metaSetX{} that are satisfiable, where a tree with root \metaSetX{} closes. Not every possible tree method is sound. For example, suppose we dropped the requirement, for a negated universal, that one take an instance with a new name.

In other words, suppose we replaced this rule (introduced on page \pageref{negunrule})

\begin{center}
\begin{prooftree}
{not line numbering}
[\enot\forall\script{x}\metaA{}, checked={\script{a}}
	[\enot\metaA{}\substitute{\script{x}}{\script{a}}, just=where \script{a} is \emph{new}
	]
]
\end{prooftree}
\end{center}

with this alternate one:

\begin{center}
\begin{prooftree}
{not line numbering}
[\enot\forall\script{x}\metaA{}, checked={\script{a}}
	[\enot\metaA{}\substitute{\script{x}}{\script{a}}, just=for \emph{any} \script{a}
	]
]
\end{prooftree}
\end{center}

If we had this rule, there would be counterexamples to soundness --- trees with satisfiable roots, which nevertheless closed. Here is an example:

\begin{prooftree}
{}
	[Fa
	[\enot \forall x Fx, grouped, checked
		[\enot Fa, just=2 alt\. \enot $\forall$, close={1, 3}]
	]
	]
\end{prooftree}

This tree is a counterexample to the soundness of the alternate tree system just described. To prove that our system is sound is to prove that our actual rules do not allow for this kind of tree. To prove this, we'll begin by assuming that \metaSetX{} is satisfiable, and demonstrate from that assumption that at least one branch of any tree that develops according to our rules will remain open. As in the case of the parallel discussion in Chapter \ref{ch.SLsoundcomplete}, our proof will be a recursive one. We will demonstrate that, if the tree \emph{starts} with a set of satisfiable sentences, then, for each legal way the tree may develop, at least one branch will continue to have a set of satisfiable sentences. This is effectively to show that such a branch will never close, since branches only close when they contain some sentences \metaA{} and \enot\metaA{}, which are of course never jointly satisfiable.

Suppose, then, we have some satisfiable set of sentences \metaSetX{} in the root of a tree. If the root is satisfiable, then there is some model that satisfies it. Call this interpretation $\mathcal{I}$. We prove that, if our tree follows the rules given in Chapter \ref{ch.QLTrees}, then $\mathcal{I}$ doesn't just satisfy the root --- it satisfies every sentence in any completed branch. We will prove this recursively.

\subsection{Root}

We assume that the tree begins with a satisfiable root. Given this assumption, $\mathcal{I}$ is just our name for one of the interpretations we are assuming must exist. So $\mathcal{I}$ trivially satisfies everything in the root.

Now we must prove, for each possible way of developing the tree, that if the sentences in the branch we \emph{begin} with are satisfiable, then the sentences we have after applying the rule are satisfiable too. There are thirteen possible ways a tree can develop, corresponding to the thirteen kinds of non-atomic sentences in QL, each of which has a particular processing rule. The thirteen kinds of sentences are:

\begin{itemize}
\item double negation
\item conjunction
\item negated conjunction
\item disjunction
\item negated disjunction
\item conditional
\item negated conditional
\item biconditional
\item negated biconditional
\item existential
\item negated existential
\item universal
\item negated universal
\end{itemize}

Fortunately, we've already proven what we need to prove for the first nine items on the list. Our QL tree method uses all the same rules as the SL method did for the sentential connectives; we proved in \S\ref{sec.sl.soundnessproof.begin}, for each of those rules, that it has the key property: if some interpretation $\mathcal{I}$ satisfies what comes above the rule, then the development below it is also satisfiable. (Indeed, we proved there that the very same interpretation, $\mathcal{I}$, satisfied it.)

So to extend our soundness proof to our QL system, we need only prove the same thing for the four rules for quantifiers. This is the project of the next four subsections.

\subsection{Existentials}

Suppose some satisfiable set of QL sentences \metaSetX{} is developed according to the existential rule:

\begin{center}
\begin{prooftree}
{not line numbering}
[\exists\script{x}\metaA{}, checked={\script{a}}
	[\metaA{}\substitute{\script{x}}{\script{a}}, just=where \script{a} is \emph{new}
	]
]
\end{prooftree}
\end{center}

We assume that $\mathcal{I}$ models \metaSetX{}, which includes some existential $\exists\script{x}\metaA{}$. We want to prove that there is a model for the expanded branch which comprises both \metaSetX{} and $\metaA{}\substitute{\script{x}}{\script{a}}$, i.e.,  $\metaSetX{} \cup \metaA{}\substitute{\script{x}}{\script{a}}$ (the set containing \metaSetX{} and the new development sentence too).  Unlike in the parallel proof for the sentential rules, we cannot be sure that $\mathcal{I}$ itself satisfies our new development, because our new development introduces a new name; we cannot assume that $\mathcal{I}$ included any assignment for the new name \script{a}. Nor, if it did, are we sure that the object \script{a} denotes satisfies $\metaA{}$. But we \emph{can} be assured that $\mathcal{I}$ can be expanded into a new, similar interpretation, $\mathcal{I}\mbox{*}$, which does include \script{a}. Moreover, since we know that $\mathcal{I}$ satisfied the existential $\exists\script{x}\metaA{}$, we know that there was some object in $\mathcal{I}$'s domain that satisfied \metaA{}. So it will be possible to construct our new interpretation $\mathcal{I}\mbox{*}$ so that it includes the new name, and assigns it to that object. This will ensure that $\mathcal{I}\mbox{*}(\metaA{}\substitute{\script{x}}{\script{a}})=1$. And since $\mathcal{I}\mbox{*}$ is just like $\mathcal{I}$ with respect to everything other than \script{a} --- and since we are assured that \script{a} was not in \metaSetX{} (the rule requires that it be new) --- $\mathcal{I}\mbox{*}$ will satisfy \metaSetX{} in just the same way that $\mathcal{I}$ did.

This will be clearer with an example. Suppose $\exists x Fx$ is part of a satisfiable set of sentences. So assume that $\mathcal{I}$ interprets it. The tree resolution for rule for existentials requires that one take an instance corresponding to a new name. Suppose this is done thus:

\begin{center}
\begin{prooftree}
{not line numbering}
[\exists x Fx, checked={a}
	[Fa]
]
\end{prooftree}
\end{center}

We cannot simply say that since some model $\mathcal{I}$ satisfies the existential $\exists x Fx$, it thereby must also satisfy $Fa$; plenty of interpretations satisfy the former while falsifying the latter. But what we \emph{can} say, since $a$ is a new name, with no previous commitments specific to it earlier in the tree, is that we can construct a model $\mathcal{I}\mbox{*}$ that satisfies every sentence above in the tree (in this case, just the one existential), and that also satisfies the new development (in this case, $Fa$). We do so by assigning the name $a$ to refer to some object that is $F$ in $\mathcal{I}$. We're sure there is such an object there because it is stipulated that $\mathcal{I}$ satisfies the existential.

In general, assuming that $\mathcal{I}$ satisfies every sentence in a branch before the existential rule is performed, there is guaranteed to be a new interpretation, $\mathcal{I}\mbox{*}$, which is an extension of $\mathcal{I}$ that includes a new name attached to an object in the UD satisfying the existential, which satisfies everything in the branch up to the point after the existential rule is performed. That is to say, like the nine sentential rules considered in Chapter \ref{ch.SLsoundcomplete}, the existential rule can never take you from a satisfiable branch to an unsatisfiable one.

\subsection{Universals}

Suppose a branch of a tree uses the rule for universals:

\begin{center}
\begin{prooftree}
{not line numbering}
[\forall\script{x}\metaA{}, subs={\script{a}}
	[\metaA{}\substitute{\script{x}}{\script{a}}, just=for \emph{any} \script{a}
	]
]
\end{prooftree}
\end{center}

Assume that the set of QL sentences \metaSetX{} above this development is satisfiable. Then some model $\mathcal{I}$ makes it true. The universal rule allows an instance to be developed using any name, so, as before, we cannot guarantee that $\mathcal{I}$ makes the development true, because $\mathcal{I}$ may or may not interpret the name \script{a}; but as before, we can be assured if it doesn't, we can extend the interpretation to include it. So consider a new model $\mathcal{I}\mbox{*}$, which includes the name \script{a}. If \script{a} wasn't interpreted by $\mathcal{I}$, then it can be assigned to any element of the UD. Since the rest of $\mathcal{I}$ is unchanged, and since $\mathcal{I}(\forall \script{x}\metaA{})=1$, we know that our new extended interpretation will satisfy $\metaA{}\substitute{\script{x}}{\script{a}}$ too.

That is to say, once again, if we assume that $\mathcal{I}$ satisfies every sentence in a branch before the universal rule is performed, there is guaranteed to be a model --- either the very same one, $\mathcal{I}$, or a modification of it, $\mathcal{I}\mbox{*}$, which assigns a new name to any object in the UD, which satisfies everything in the branch up to the point after the universal rule is performed. In other words, the universal rule can never take you from a satisfiable branch to an unsatisfiable one.

Note that the fact that we can perform this rule multiple times does not interfere with the soundness proof. We have proven that \emph{each} time you perform it, you are guaranteed not to switch from a satisfiable branch to an unsatisfiable one. So no matter how many times you take an instance of a universal, you won't be able to go from a satisfiable set of sentences to an unsatisfiable one.

\subsection{Negated Existential}

The reasoning behind soundness for the negated existential rule is exactly parallel to that for the universal rule. We begin by assuming that some negated universal $\enot\exists\script{x}\metaA{}$, is satisfied by $\mathcal{I}$. Here is the rule for negated existentials:

\begin{center}
\begin{prooftree}
{not line numbering}
[\enot\exists\script{x}\metaA{}, subs={\script{a}}
	[\enot\metaA{}\substitute{\script{x}}{\script{a}}, just=for \emph{any} \script{a}
	]
]
\end{prooftree}
\end{center}

We want to prove that the result of this rule is also satisfied, either by $\mathcal{I}$ itself (if \script{a} was interpreted in $\mathcal{I}$), or by an extension of it $\mathcal{I}\mbox{*}$, that preserves the satisfaction of everything above (if \script{a} was a new name). Since $\mathcal{I}$ satisfies $\enot\exists\script{x}\metaA{}$, it makes every substitution instance for \script{x} of \metaA{} false. If \script{a} was interpreted by $\mathcal{I}$ already, then $\mathcal{I}(\metaA{}\substitute{\script{x}}{\script{a}})=0$. If it wasn't, the new model $\mathcal{I}\mbox{*}$ will assign the new name to some object in the UD of the original model; since no object in that model satisfied \metaA{}, \mbox{$\mathcal{I}\mbox{*}(\metaA{}\substitute{\script{x}}{\script{a}})=0$}. Either way, our interpretation falsifies \mbox{\metaA{}\substitute{\script{x}}{\script{a}}}, and so satisfies that sentence's negation, which is the continuation of the branch.

So this rule too can never take us from a satisfiable set of QL sentences to an unsatisfiable one.

\subsection{Negated Universal}

Negated universals are similar to existentials. Assume that a negated universal is part of a set of sentences satisfied by $\mathcal{I}$, and that this rule is then applied:

\begin{center}
\begin{prooftree}
{not line numbering}
[\enot\forall\script{x}\metaA{}, checked={\script{a}}
	[\enot\metaA{}\substitute{\script{x}}{\script{a}}, just=where \script{a} is \emph{new}
	]
]
\end{prooftree}
\end{center}

Construct a new interpretation $\mathcal{I}\mbox{*}$, which differs from $\mathcal{I}$ only in that it includes an interpretation of the new name \script{a}, and assigns that name to some object that falsifies \metaA{}. We know there is at least one such object because we are assuming that $\mathcal{I}$ satisfies the negated universal. Then our new interpretation $\mathcal{I}\mbox{*}$ satisfies the new development of the branch. It also satisfies everything above the branch, just like $\mathcal{I}$ did, because nothing above the branch included the name \script{a}.

That last bit of reasoning relied centrally on the requirement that we're taking a new name. We saw in the introduction to this chapter that if we do not include that requirement, soundness would be violated.

\subsection{Summarizing the soundness proof}

We have now shown, for our four quantifier rules, that each of them has the following property: you can never start with a branch that is satisfiable, and use that rule to extend that branch into one that is unsatisfiable. Since we've also shown that the nine sentential rules also have this property, we've effectively shown that there is no possible way to start with a satisfiable set of sentences and develop the branch into one that is not satisfiable. This in turn means that if the branch starts with a satisfiable set of sentences, the branch will never close. But that's just what soundness says: if the root is satisfiable, the tree is guaranteed to remain open. Soundness is proven.

\section{Completeness}

Completeness says that if a branch of a completed tree remains open, then the root is satisfiable. We prove this by assuming that we have an open completed branch, and use it to construct an interpretation that satisfies every sentence in that branch, which includes the root. The proof for completeness of our QL tree system is structurally just like the one given in Chapter \ref{ch.SLsoundcomplete}.

Given a completed open branch, we construct a model $\mathcal{I}$, based on that branch, as follows: for any predicate $\script{F}$\!, if some QL atom $\script{F}\script{a}_{1}, \ldots, \script{a}_{n}$ is in the branch --- i.e., if $P$ or $Fa$ or $Rab$ is in the branch --- then $\mathcal{I}$ makes that atom true, by putting \ntuple{$\script{a}_{1}$, \ldots, $\script{a}_{n}$} in the extension of $\script{F}$\!. And if $\enot \script{F}\script{a}_{1}, \ldots, \script{a}_{n}$ is in the branch, $\mathcal{I}$ excludes \ntuple{$\script{a}_{1}$, \ldots, $\script{a}_{n}$} from the extension of $\script{F}$\!, thus making the negation of the atom true. This is of course just the way that we construct interpretations from open branches of completed trees.

Now we will prove, for every sentence in QL, that if it is in the open branch, it is made true by $\mathcal{I}$. The QL atoms trivially meet this criterion --- $\mathcal{I}$ was designed precisely to satisfy them. We will prove by induction that every possible QL sentence also meets this criterion. In \S\ref{sec.completenessproof} we showed, for each propositional connective, if you construct a more complex SL sentence out of simpler SL sentences that have this criterion, the more complex one will too. That proof carries on unchanged here. So it remains only to show that the same is true of our four quantifier rules.

\subsection{Existential}

Consider an existential --- a QL sentence of the form $\exists \script{x} \metaA{}$. We need to prove that if it is in the open, completed branch, $\mathcal{I}$ satisfies it. Since the branch is complete, we know that the existential rule has been performed to resolve this sentence. So the branch includes a substitution instance of \metaA{} that used a new name. For our present purposes, it doesn't actually matter whether the name was new --- the fact that there is some instance of \metaA{} in the branch already is enough to prove what we need to prove. Since there is an instance of \metaA{} in the branch, if it is satisfied by $\mathcal{I}$, the existential $\exists \script{x} \metaA{}$ must be satisfied by $\mathcal{I}$ too.

So, just as we showed for the nine sentential rules, the existential rule has this important property: in a completed tree, any interpretation that satisfies the simpler sentences below the existential development, must also satisfy the existential above it.

\subsection{Universal}

Suppose a universal sentence $\forall \script{x} \metaA{}$ appears in a completed open branch. Since the branch is complete, that means that, for every name \script{a} in the branch, \mbox{\metaA{}\substitute{\script{x}}{\script{a}}} is also in the branch. We therefore assume that $\mathcal{I}$ satisfies each \metaA{}\substitute{\script{x}}{\script{a}}; so $\mathcal{I}$ must also satisfy $\forall \script{x} \metaA{}$. Because the UD for $\mathcal{I}$ includes only those names that occur in the branch, every instance of \metaA{} is included, so the universal is true.

Once again, any interpretation that satisfies everything below the universal development, must also satisfy the universal above it.

\subsection{Negated Existential}

Negated existentials work just like universals. If $\enot \exists \script{x} \metaA{}$ is in a completed open branch, then for every name \script{a} in $\mathcal{I}$,
\mbox{\enot\metaA{}\substitute{\script{x}}{\script{a}}} is below it in the branch. And if $\mathcal{I}$ satisfies each of these negations, it will also satisfy the negated existential.

\subsection{Negated Universal}

Negated universals work just like existentials. If $\enot \forall \script{x} \metaA{}$ is in the branch, then some instance of the negation \enot \metaA{} is in the branch below. If $\mathcal{I}$ satisfies some instance of \enot \metaA{}, then, given the definition of truth for negation and universals in QL, it will also satisfy $\enot \forall \script{x} \metaA{}$.

\subsection{Summarizing the completeness proof}

The sentence shapes just considered, combined with the nine shapes considered in \S\ref{sec.completenessproof}, correspond to all the possible QL sentences. So we have proven that, for any possible QL sentence \metaA{}, if an interpretation satisfies the simpler sentences below it in the branch, that interpretation also satisfies \metaA{} itself. Since we also have a recipe for constructing an interpretation $\mathcal{I}$ that is guaranteed to satisfy the atoms, we can prove by induction that it can satisfy everything in the branch, including the root. A completed open branch guarantees a satisfiable root. Completeness is proven.

\practiceproblems

\solutions
\problempart
\label{pr.QLalttrees-sound}
Following are possible modifications to our QL tree system. For each, imagine a system that is like the system laid out in this chapter, except for the indicated change. Would the modified tree system be sound? If so, explain how the proof given in this chapter would extend to a system with this rule; if not, give a tree that is a counterexample to the soundness of the modified system.
\begin{earg}
\item Change the rule for existentials to this rule:
	\factoidbox{
	\begin{center}
	\begin{prooftree}
	{not line numbering}
	[\exists\script{x}\metaA{}, checked={\script{a}}
		[\metaA{}\substitute{\script{x}}{\script{a}}, just=for \emph{any} \script{a}
		]
	]
	\end{prooftree}
	\end{center}
	}

\item Change the rule for existentials to this rule:
	\factoidbox{
	\begin{center}
	\begin{prooftree}
	{not line numbering}
	[\exists\script{x}\metaA{}, checked=d
		[\metaA{}\substitute{\script{x}}{d}, just=(whether or not $d$ is new)
		]
	]
	\end{prooftree}
	\end{center}
	}

\item Change the rule for existentials to this rule:
	\factoidbox{
	\begin{center}
	\begin{prooftree}
	{not line numbering}
	[\exists\script{x}\metaA{}, checked
		[\metaA{}\substitute{\script{x}}{\script{a}}, just={for 3 different names, old or new}
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{b}}, grouped
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{c}}, grouped
		]
		]
		]
		]
		]
	]
	\end{prooftree}
	\end{center}
	}

\item Change the rule for universals to this rule:
	\factoidbox{
	\begin{center}
	\begin{prooftree}
	{not line numbering}
	[\forall\script{x}\metaA{}, checked
		[\metaA{}\substitute{\script{x}}{\script{a}}, just={for 3 different names, old or new}
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{b}}, grouped
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{c}}, grouped
		]
		]
		]
		]
		]
	]
	\end{prooftree}
	\end{center}
	}

\item Change the rule for existentials to this rule:
	\factoidbox{
	\begin{center}
	\begin{prooftree}
	{not line numbering}
	[\exists\script{x}\metaA{}, checked
		[\metaA{}\substitute{\script{x}}{\script{a}}, just={for 3 new names}
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{b}}, grouped
		[ , grouped
		[\metaA{}\substitute{\script{x}}{\script{c}}, grouped
		]
		]
		]
		]
		]
	]
	\end{prooftree}
	\end{center}
	}

\item Change the rule for universals to this rule:
	\factoidbox{
            	\begin{center}
            \begin{prooftree}
            {not line numbering}
            [\forall\script{x}\metaA{}, checked={\script{a}}
            	[\metaA{}\substitute{\script{x}}{\script{a}}, just=where \script{a} is \emph{new}
            	]
            ]
            \end{prooftree}
            \end{center}
	}

\item Change the rule for conjunction to this rule:
	\factoidbox{
            	\begin{center}
            \begin{prooftree}
            {not line numbering}
            	[\metaA{} \eand \metaB{}, checked
            		[\exists \script{x} \metaA{}, just=where \script{x} does not occur in \metaA{}
			[\metaB{}, grouped
            		]
            		]
		]
            \end{prooftree}
            \end{center}
	}

\item Change this requirement (given on page \pageref{branchcompletion.defined})...
	\factoidbox{A branch is \define{complete} if and only if either (i) it is closed, or (ii) every resolvable sentence in every branch has been resolved, and for every general sentence and every name \script{a} in the branch, the \script{a} instance of that general sentence has been taken.}
	...to this one:
	\factoidbox{A branch is \define{complete} if and only if either (i) it is closed, or (ii) every resolvable sentence in every branch has been resolved, and for every general sentence, \emph{at least one instance of} that general sentence has been taken.}

\item Change the branch completion requirement to:
	\factoidbox{\ldots and for every general sentence and every name \script{a} \emph{that is above that general sentence in the branch}, the \script{a} instance of that general sentence has been taken.}

\item Change the branch completion requirement to:
	\factoidbox{\ldots and for every general sentence and every name \script{a} in the branch, the \script{a} instance of that general sentence has been taken, \emph{and at least one additional new instance of that general sentence has also been taken}.}

	\end{earg}

\solutions
\problempart
\label{pr.QLalttrees-complete}
For each of the rule modifications given in Part \ref{pr.QLalttrees-sound}, would the modified tree system be complete? If so, explain how the proof given in this chapter would extend to a system with this rule; if not, give a tree that is a counterexample to the completeness of the modified system.


\newcommand{\defeq}{\vcentcolon=}
\newcommand{\tn}[1]{\textnormal{#1}}

\section{Elementary Definitions and Properties}
\label{sec:c*-algebras-elementary-definitions-properties}

A $C^*$-algebra is a Banach algebra with additional structure. We recall the definition of a Banach algebra below.

\begin{definition}
A \emph{Banach algebra} is a Banach space $A$ with an associative, bilinear multiplication operation $A \times A \rightarrow A$, $(a,b) \mapsto ab$ which is \emph{submultiplicative} with respect to the norm:
\begin{equation}\label{eq:submultiplicativity}
\norm{ab} \leq \norm{a} \norm{b}, \qquad \forall \, a, b \in A.
\end{equation}
We say $A$ is \emph{unital} if there exists a \emph{unit} $1 \in A$ satisfying $\norm{1} = 1$ and
\begin{equation}\label{eq:algebra_unit}
1a = a1 = a, \qquad \forall \, a \in A.
\end{equation}
In a unital Banach algebra we can speak of inverses of elements, but not every element of a Banach algebra is invertible. By an inverse we mean a two-sided inverse unless otherwise specified. We write $A^\times$ for the set of invertible elements in $A$.
\end{definition}

We state without proof some obvious facts about Banach algebras.

\begin{proposition}
Let $A$ be a Banach algebra.
\begin{enumerate}
	\item For all $a \in A$, we have $0a = a0 = 0$.
\end{enumerate}
If $A$ is unital, then the following hold.
\begin{enumerate}\setcounter{enumi}{1}
	\item The unit is unique.
	\item Inverses are unique.
	\item The additive identity is not invertible.
	\item The multiplicative identity is its own inverse.
	\item If $a, b \in A^\times$, then $ab \in A^\times$ and $(ab)^{-1} = b^{-1}a^{-1}$.
	\item If $a \in A$ has a left inverse and a right inverse, then these inverses are equal, so $a \in A^\times$.
\end{enumerate}
\end{proposition}

\begin{proposition}
The multiplication operation on a Banach algebra is continuous.
\end{proposition}

\begin{proof}
Take sequences $(a_n)$ and $(b_n)$ in $A$ such that $a_n \rightarrow a$ and $b_n \rightarrow b$. Using the triangle inequality and \eqref{eq:submultiplicativity},
\begin{equation}
\begin{aligned}
\norm{ab - a_n b_n} &\leq \norm{ab - ab_n} + \norm{ab_n - a_nb_n}\\
&\leq \norm{a} \norm{b - b_n} + \norm{a - a_n} \norm{b_n}
\end{aligned}
\end{equation}
This manifestly approaches zero, so we conclude that $a_nb_n \rightarrow ab$.
\end{proof}

\begin{proposition}
If $A$ is a Banach algebra and $a \in A^\times$, then
\begin{equation}
\norm{a}^{-1} \leq \norm{a^{-1}}.
\end{equation}
\end{proposition}

\begin{proof}
By \eqref{eq:submultiplicativity}, we have
\begin{equation}
1 = \norm{1} =\norm{a^{-1}a} \leq \norm{a^{-1}}\norm{a}.
\end{equation}
The result follows by dividing by $\norm{a}$.
\end{proof}

\begin{proposition}
Inversion $A^\times \rightarrow A^\times$, $a \mapsto a^{-1}$ is continuous.
\end{proposition}

\begin{proof}
Take a sequence $(a_n)$ in $A^\times$ such that $a_n \rightarrow a \in A^\times$. We compute
\begin{equation}
\begin{aligned}
\norm{a^{-1} - a_n^{-1}} &= \norm{a^{-1} (a_n - a)a^{-1}_n}\\
&\leq \norm{a^{-1}} \norm{a_n - a}\norm{a_n^{-1}}\\
&\leq \norm{a^{-1}}\norm{a_n - a}\norm{a^{-1}} + \norm{a^{-1}}\norm{a_n - a}\norm{a^{-1} - a_n^{-1}}.
\end{aligned}
\end{equation}
Moving the rightmost term to the other side yields
\begin{equation}
\qty(1 - \norm{a^{-1}}\norm{a_n - a})\norm{a^{-1} - a_n^{-1}} \leq  \norm{a_n - a} \norm{a^{-1}}^2
\end{equation}
For large enough $n$, the term in parentheses is nonzero, and we may divide by it, yielding
\begin{equation}
\norm{a^{-1} - a_n^{-1}} \leq \frac{\norm{a_n - a}\norm{a^{-1}}^2}{1 - \norm{a^{-1}}\norm{a_n - a}}.
\end{equation}
The left hand side manifestly approaches zero, so we conclude $a_n^{-1} \rightarrow a^{-1}$.
\end{proof}

\begin{definition}
A \emph{$\boldsymbol{C^*}$-algebra} is a Banach algebra $A$ with an antilinear \emph{star operation} $A \rightarrow A$, $a \mapsto a^*$ satisfying
\begin{enumerate}
	\item[(i)] \emph{involutivity:} $a^{**} = a$ for all $a \in A$,
	\item[(ii)] \emph{contravariance:} $(ab)^* = b^*a^*$ for all $a, b \in A$,
	\item[(iii)] the \emph{$\boldsymbol{C^*}$-property:} $\norm{a^*a} = \norm{a}^2$ for all $a \in A$.
\end{enumerate}
An element $a \in A$ is \emph{self-adjoint} if $a^* = a$.

A subset $B \subset A$ is a \emph{$\boldsymbol{C^*}$-subalgebra} of $A$ if $B$ is a $C^*$-algebra under the restrictions to $B$ of all operations defined on $A$. Equivalently, $B$ must be a topologically closed subset which is closed under all the operations on $A$. Topological closedness is equivalent to completeness. We say $B$ is a \emph{unital} $C^*$-subalgebra of $A$ if $B$ is a unital $C^*$-algebra and the unit in $B$ is the same as the unit in $A$.\footnote{The unitization of a unital $C^*$-algebra (discussed in a later section) provides an instance where we have a unital $C^*$-algebra and a $C^*$-subalgebra which has a different unit.}

If $S$ is a subset of a $C^*$-algebra $A$, then the intersection of all $C^*$-subalgebras of $A$ containing $S$ is a $C^*$-subalgebra, called the \emph{$\boldsymbol{C^*}$-subalgebra generated by $\boldsymbol{S}$}.

If $A$ and $B$ are $C^*$-algebras, a \emph{$\boldsymbol{*}$-homomorphism} from $A$ to $B$ is a map $\pi:A \rightarrow B$ respecting the algebraic operations on $A$ and $B$. More precisely, $\pi$ is a linear map satisfying
\begin{equation}
\begin{aligned}
\pi(ab) &= \pi(a)\pi(b)\\
\pi(a^*) &= \pi(a)^*
\end{aligned}
\end{equation}
for all $a, b \in A$. Notice that we do not require $\pi$ to be continuous; we will show later that this is automatically so. If $A$ and $B$ are unital, we say $\pi$ is a \emph{unital $\boldsymbol{*}$-homomorphism} if $\pi$ is a $*$-homomorphism and $\pi(1) = 1$. The terms $*$-isomorphism and $*$-automorphism will be used in the natural sense.
\end{definition}

\begin{proposition}
If $A$ and $B$ are $C^*$-algebras and $\pi:A \rightarrow B$ is a $*$-isomorphism, then $\pi^{-1}:B \rightarrow A$ is a $*$-isomorphism. If $\pi$ is unital, then so is $\pi^{-1}$.
\end{proposition}

\begin{proof}
We know from linear algebra that $\pi^{-1}$ is a linear map. Given $a, b \in B$, let $a', b' \in A$ such that $\pi(a') = a$ and $\pi(b') = b$. Then
\begin{equation}
\begin{aligned}
\pi^{-1}(ab) &= \pi^{-1}(\pi(a')\pi(b')) = \pi^{-1}(\pi(a'b')) = a'b' = \pi^{-1}(a)\pi^{-1}(b)\\
\pi^{-1}(a^*) &= \pi^{-1}(\pi(a')^*) = \pi^{-1}(\pi(a'^*)) = a'^* = \pi^{-1}(a)^*.
\end{aligned}
\end{equation}
If $\pi$ is unital, then $\pi(1) = 1$, so $\pi^{-1}(1) = 1$ as well.
\end{proof}

We will study $*$-homomorphisms more in a later section, for now focusing on properties of elements of $C^*$-algebras.

\begin{definition}
If $A$ is a $C^*$-algebra, a subset $B \subset A$ is a \emph{Banach subalgebra} of $A$ if it is a Banach algebra under the restrictions of all operations defined on $A$. Equivalently, $B$ must be a topologically closed subset which is closed under all the operations on $A$. Topological closedness is equivalent to completeness. We say $B$ is a \emph{unital} Banach subalgebra of $A$ if $B$ is a unital $C^*$-algebra and the unit in $B$ is the same as the unit in $A$.  I pray I never have to consider a case where $B$ is a Banach subalgebra of $A$ and has a different unit from $A$.

If $S$ is a subset of a Banach algebra or $C^*$-algebra $A$, then the intersection of all subalgebras of $A$ containing $S$ is a (Banach or $C^*$) subalgebra, called the \emph{subalgebra generated by $\boldsymbol{S}$}.

A (unital) $C^*$-subalgebra is a (unital) Banach subalgebra $B\subset A$ which is closed under the star operation.
\end{definition}

\begin{definition}
Let $A$ and $B$ be $C^*$-algebras. A \emph{$\boldsymbol{*}$-homomorphism} is a linear map $\pi:A \rightarrow B$ such that
\begin{equation}
\begin{aligned}
\pi(ab) &= \pi(a)\pi(b)\\
\pi(a^*) &= \pi(a)^*
\end{aligned}
\end{equation}
for all $a, b \in A$. In other words, $\pi$ respects all algebraic operations on $A$ and $B$. Note that we do not require $\pi$ to be continuous. We will use the terms $*$-isomorphism and $*$-automorphism in the natural way.
\end{definition}

We will study $*$-homomorphisms more in a later section. For now, let us establish a few more basic properties of $C^*$-algebras.

\begin{proposition}
If $A$ is a $C^*$-algebra, then $0$ is self-adjoint. If $A$ is unital, then $1$ is self-adjoint as well.
\end{proposition}

\begin{proof}
For all $a \in A$, we have
\begin{equation}
0^* + a = 0^* + a^{**} = (0 + a^*)^* = a^{**} = a.
\end{equation}
Hence, $0^* = 0$ by uniqueness of the additive identity. Furthermore,
\begin{equation}
1^*a = 1^*a^{**} = (a^*1)^* = a^{**} = a,
\end{equation}
from which it follows that $1$ is self-adjoint by uniqueness of the multiplicative identity.
\end{proof}

\begin{proposition}
Every $a \in A$ has a unique expression in the form $a = a_1 + ia_2$, where $a_1$ and $a_2$ are self-adjoint.
\end{proposition}

\begin{proof}
This is evident upon setting $a_1 = (a + a^*)/2$ and $a_2 = (a - a^*)/2i$. If $a = a_1' + ia_2'$ for some self-adjoint $a_1', a_2'$, then $a_1 - a_1' = i(a_2' - a_2)$, which can be self-adjoint only if it is zero.
\end{proof}

\begin{proposition}
Let $A$ be a nontrivial $C^*$-algebra. If there exists $1 \in A$ satisfying
\begin{equation}
1a = a1 = a, \qquad \forall \, a \in A,
\end{equation}
then $\norm{1} = 1$, i.e.\ $A$ is unital.
\end{proposition}

\begin{proof}
Setting $a = 1$ in the $C^*$-property yields
\begin{equation}
\norm{1} = \norm{1^*1} = \norm{1}^2.
\end{equation}
Hence, $\norm{1} = 0$ or $\norm{1} = 1$. If $\norm{1} = 0$, then $1 = 0$, so that $A = \qty{0}$. Since $A$ is nontrivial by hypothesis, this cannot be the case, so $\norm{1} = 1$.
\end{proof}

\begin{proposition}\label{prop:inverse_star_commute}
Let $A$ be a unital $C^*$-algebra and let $a \in A^\times$. Then $a^* \in A^\times$ and
\begin{equation}
(a^*)^{-1} = (a^{-1})^*.
\end{equation}
\end{proposition}

\begin{proof}
We compute
\begin{equation}
a^*(a^{-1})^* = (a^{-1} a)^* = 1 = (aa^{-1})^* = (a^{-1})^* a^*,
\end{equation}
which proves the result.
\end{proof}

\begin{proposition}\label{prop:star_is_isometry}
If $A$ is a $C^*$-algebra and $a \in A$, then
\begin{equation}
\norm{a^*} = \norm{a}.
\end{equation}
\end{proposition}

\begin{proof}
The conclusion is trivial if $a = 0$, so suppose $a \neq 0$. The $C^*$-property and submultiplicativity yield
\begin{equation}
\norm{a}^2 = \norm{a^*a} \leq \norm{a} \norm{a^*}.
\end{equation}
Dividing by $\norm{a}$ yields $\norm{a} \leq \norm{a^*}$. Applying this result to $a^*$ yields $\norm{a^*} \leq \norm{a^{**}} = \norm{a}$.
\end{proof}

\begin{corollary}
The star operation $A \rightarrow A$, $a \mapsto a^*$ is continuous.
\end{corollary}

\begin{proof}
This is immediate from Proposition \ref{prop:star_is_isometry}.
\end{proof}

We conclude this section with several examples.

\begin{example}
The complex numbers $\C$ give a fairly trivial unital $C^*$-algebra.
\end{example}

\begin{example}
The bounded linear operators $\blinOps(\hilbertH)$ on a Hilbert space $\hilbertH$ are the prototypical example of a $C^*$-algebra. The star operation is given by the adjoint. Note that this is, of course, a unital $C^*$-algebra.
\end{example}

\begin{example}
In a similar vein to the previous example, the set $M_{n}(\C)$ of $n \times n$ matrices with complex entries is a $C^*$-algebra, where the star operation is given by Hermitian conjugation.
\end{example}

\begin{example}
Let $X$ be a compact Hausdorff space and let $C(X)$ be the space of continuous functions $X \rightarrow \C$. This is a unital $C^*$-algebra with the norm given by the supremum norm and the star operation given by complex conjugation. We note that
\begin{equation}
\norm{fg} = \sup_{x \in X} \abs{fg} \leq \sup_{x \in X} \abs{f} \cdot \sup_{x \in X} \abs{g} = \norm{f} \norm{g}
\end{equation}
and
\begin{equation}
\norm{f^*f} = \sup_{x \in X} \abs{f}^2  = \qty(\sup_{x \in X} \abs{f})^2 = \norm{f}^2,
\end{equation}
so that this satisfies the nontrivial properties of a $C^*$-algebra.
\end{example}

\begin{example}
Let $X$ be a locally compact Hausdorff space and let $C_0(X)$ be the space of continuous functions $f: X \rightarrow \C$ which \emph{vanish at infinity}, meaning for every $\varepsilon > 0$ there exists a compact $K \subset X$ such that $\abs{f(x)} < \varepsilon$ for $x \notin K$. This space is a $C^*$-algebra with the supremum norm and the star operation given by complex conjugation. If $X$ is not compact, then this is a non-unital $C^*$-algebra.
\end{example}

\subsection*{Finite Direct Sums}

Let $A_1,\ldots, A_n$ be a finite collection of Banach algebras. We define the \emph{direct sum}
\begin{equation}
\bigoplus_{i=1}^n A_i \defeq \qty{(a_1,\ldots, a_n): a_i \in A_i}
\end{equation}
with addition and multiplication defined componentwise. If $A_1,\ldots, A_n$ are $C^*$-algebras, define the star operation on $A$ componentwise as well. Finally, set
\begin{equation}
\norm{(a_1,\ldots, a_n)} = \max\qty(\norm{a_1},\ldots, \norm{a_n}).
\end{equation}

We want to show that the direct sum so defined is a Banach algebra. It is easy to check that the norm above is indeed a norm using the properties of the max and the definition of the norms on $A_i$. Submultiplicativity also follows easily from submultiplicativity of the norms on the $A_i$. If $(a_{1,k},\ldots, a_{n,k})_{k \in \N}$ is a Cauchy sequence in the direct sum, then each sequence $(a_{i,k})_{k \in \N}$ is Cauchy, hence convergent, in $A_i$ for $i = 1, \ldots, n$. If $a_{i,k} \rightarrow a_i$ for each $i$, then $(a_{1,k},\ldots, a_{n,k}) \rightarrow (a_1,\ldots, a_n)$ by definition of the norm on the direct sum. Thus, $\bigoplus_{i=1}^n A_i$ is complete, and is therefore a Banach algebra. If each $A_i$ is a $C^*$-algebra, it is again easy to check that $\bigoplus_{i=1}^n A_i$ is a $C^*$-algebra using the properties of the $C^*$-algebras $A_i$.

We may also define the direct sum of a sequence of Banach algebras or $C^*$-algebras $\qty{A_n}_{n \in \N}$. We define
\begin{equation}
\bigoplus_{n=1}^\infty A_n \defeq \qty{(a_n)_{n \in \N}: a_n \in A_n \text{ and } \lim_{n \rightarrow \infty} \norm{a_n} = 0}
\end{equation}
Again, the algebraic operations are defined componentwise and the norm is defined by
\begin{equation}
\norm{(a_n)_{n \in \N}} = \max_{n \in \N} \norm{a_n}.
\end{equation}
The definition of $\bigoplus_{n = 1}^\infty A_n$ ensures that the max exists.

One easily checks that this satisfies all algebraic properties of a Banach or $C^*$-algebra, but completeness is more subtle. If $\mathbf{a}_k = (a_{n,k})_{n \in \N} \in \bigoplus_{n=1}^\infty A_n$ and $(\mathbf{a}_k)_{k \in \N}$ is a Cauchy sequence in the direct sum, then it follows in the same way as before that the sequence $(a_{n,k})_{k \in \N}$ is Cauchy in $A_n$, hence convergent with limit $a_{n,k} \rightarrow a_n \in A_n$. We must show that $\lim_{n \rightarrow \infty} \norm{a_n} = 0$. For any $n, k, K \in \N$, we have
\begin{equation}
\norm{a_n} \leq \norm{a_n - a_{n,k}} + \norm{a_{n,k} - a_{n,K}} + \norm{a_{n,K}} \leq \norm{a_n - a_{n,k}} + \norm{\mathbf{a}_k - \mathbf{a}_K} + \norm{a_{n,K}}
\end{equation}
Fix $\varepsilon > 0$. Since $(\mathbf{a}_k)_{k \in \N}$ is Cauchy, we may choose $K \in \N$ such that $k, \ell \geq K$ implies $\norm{\mathbf{a}_k - \mathbf{a}_\ell} < \varepsilon/3$. We may choose $N \in \N$ such that $n \geq N$ implies $\norm{a_{n,K}} < \varepsilon/3$. Finally, for any $n \geq N$, we may choose $k \geq K$ such that $\norm{a_n - a_{n,k}} < \varepsilon/3$. Thus, for $n \geq N$, we have $\norm{a_n} < \varepsilon$, so $\lim_{n \rightarrow \infty} \norm{a_n} = 0$.

Finally, we show that $\mathbf{a}_k \rightarrow \mathbf{a} \defeq (a_n)_{n \in \N}$. Fix $\varepsilon > 0$. Choose $N \in \N$ such that $\norm{a_n} < \varepsilon/2$ if $n \geq N$. Choose $K \in \N$ such that $k,\ell \geq K$ and $n < N$ implies
\begin{equation}
\norm{\mathbf{a}_k - \mathbf{a}_\ell} < \frac{\varepsilon}{2} \quad\text{and}\quad \norm{a_n - a_{n,k}} < \varepsilon.
\end{equation}
Then for $k \geq K$, we have
\begin{equation}
\norm{\mathbf{a} - \mathbf{a}_k} < \max\qty(\varepsilon, \max_{n \geq N} \norm{a_n - a_{n,k}}).
\end{equation}
But for $n \geq N$ we may choose $\ell$ large enough such that
\begin{equation}
\norm{a_n - a_{n,k}} \leq \norm{a_n - a_{n,\ell}} + \norm{\mathbf{a}_\ell - \mathbf{a}_k} < \varepsilon.
\end{equation}
Thus, $\norm{\mathbf{a} - \mathbf{a}_k} < \varepsilon$, so $\mathbf{a}_k \rightarrow \mathbf{a}$. This proves that the direct sum is complete.

%I believe one can form arbitrary direct sums by completing the vector space direct sum with respect to the max norm.

Both of these constructions come equipped with natural algebra homomorphisms $\iota_i: A_i \rightarrow \bigoplus A_n$ satisfying the following universal property. If $B$ is another Banach or $C^*$-algebra with algebra homomorphisms $f_i:A_i \rightarrow B$, then there exists a unique algebra homomorphism $f:\bigoplus A_n \rightarrow  B$ such that the diagram
\begin{equation}
\begin{tikzcd}
 \bigoplus A_i \arrow[r, "f"] & B\\
 A_i \arrow[ur, "f_i"'] \arrow[u, "\iota_i"] &
\end{tikzcd}
\end{equation}
commutes. We define
\begin{equation}
f(a) = \sum f_i(\pi_i(a))
\end{equation}


\section{Week 1}
\subsection{1/5/2021}
\begin{itemize}
    \item Vector review!
    \item Two dimensional vector: $\langle v_1, v_2 \rangle \begin{bmatrix}
        w_1 \\
        w_2 \\
    \end{bmatrix}$
    \item The zero vector = $\langle 0, 0 \rangle$ or $\textbf{0}$
    \item Linear combinations: linear combinations of $v$ and $w$ refer to any sum of multiples of the vectors. $$c\textbf{v} + d\textbf{w}$$ is a linear combination of $v$ and $w$
    \item In 3-space, a 3 dimensional vector: $\textbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ v_3 \end{bmatrix}, \textbf{v} = (v_1, v_2, v_3)$
    \item Dot products: $\Vec{v} \cdot \Vec{w} = v_1w_1+v_2w_2...+v_nw_n$
    \item Dot products are commutative.
    \item A dot product of zero means the vectors orthogonal.
    \item Contrived example: $v$ is a vector of weights and $w$ is a vector of distances. $v \cdot w = 0$ means the  system is balanced.
    \item Contrived example 2: Expanding the previous problem to 3 dimensions, we can add more weights, $v= (4,2,100), w=(-1,2,0), v\cdot w = 0$
    \item Economics example: Five products with prices $p_1, p_2, ..., p_5$ with quantity $q_1, q_2, ... q_5$. A dot product of zero would mean it breaks even.
    \item Technically, $\textbf{0}$ is orthogonal to all vectors.
    \item $\begin{bmatrix} 4 & 1 \\ 0 & 2 \\ 4 & 3\end{bmatrix} \begin{bmatrix} 3 \\ -2 \end{bmatrix} = \begin{bmatrix} 10 \\ -4 \\ 6\end{bmatrix}$
    \item Linear combinations can also be seen as dot products.
    \item The length or norm of a vector $||v|| = \sqrt{v_1^2 + v_2^2 + ... + v_n^2} = \sqrt{\Vec{v} \cdot \Vec{v}}$
    \item Unit vector trick: if $\Vec{v} \cdot \Vec{v} = 1$ yes!
    \item $\Vec{u} = \frac{\Vec{v}}{||\vec{v}||}$
    \item If we know the angle $\theta$ a vector makes with the x-axis, the unit vector is $\langle cos\theta, sin\theta \rangle$
    \item Given unit vectors with form $\langle cos\beta, sin\beta \rangle$ then the angle in between can be denoted by $\vec{u_1}\cdot \vec{u_2}=cos(\beta - \alpha)$
    \item Schwarz Inequality: $|v\cdot w| \le ||v||||w||$
\end{itemize}

\subsection{1/6/2021}
\begin{itemize}
    \item $$n\cdot v=ax+by+cz=d$$
    \item Planes are also linear combinations. Ex. $x-y-3z=0$ is a combination of vectors.
    \item $x=y+3z$, set $y=z=0$, so $\langle 1, 1, 0\rangle$ is a vector. Then, set $y=0, z=1$, so we get $\langle 3, 0 ,1 \rangle$. Our combination is $y\begin{bmatrix}1\\1\\0\end{bmatrix}+z\begin{bmatrix}3\\0\\1\end{bmatrix}$
    \item In case of a constant, ex. $x-y-3z=2$.
    \item Start as if $d=0$, in this case giving us the same $y\begin{bmatrix}1\\1\\0\end{bmatrix}+z\begin{bmatrix}3\\0\\1\end{bmatrix}$, then simply add $\begin{bmatrix}2\\0\\0\end{bmatrix}$ for $y\begin{bmatrix}1\\1\\0\end{bmatrix}+z\begin{bmatrix}3\\0\\1\end{bmatrix}+\begin{bmatrix}2\\0\\0\end{bmatrix}$
    \item Matrix notation. Ex. $a_{4 7}$, row 4, col 7
    \item Identity matrix, denoted by $I$, is a square matrix with 1's along the main diagonal (where $i=j$)
    \item $AI = A = IA$ if their multiplication dimensions are valid.
    \item $$\begin{bmatrix}2&-1\\3&5\end{bmatrix}\begin{bmatrix}1&0&-1\\3&1&3\end{bmatrix}=\begin{bmatrix}-1&-1&-5\\ 18&5&12\end{bmatrix}$$
    \item An item $A_{ij}$ = row $i \cdot$ column $j$
    \item $\begin{bmatrix}0&1\\1&0\end{bmatrix}\begin{bmatrix}4\\5\end{bmatrix}=\begin{bmatrix}5\\4\end{bmatrix}$
    \item Linear equations:
    $$x+2y+3z=6$$
    $$2x+5y+2z=4$$
    $$6x-3y+z=2$$
    \item Coefficient matrix $A=\begin{bmatrix}1&2&3\\2&5&2\\6&-3&-1\end{bmatrix}$
    \item $$\begin{bmatrix}1&2&3\\2&5&2\\6&-3&-1\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}6\\4\\2\end{bmatrix}$$
    \item $A\vec{x}=\vec{b}$
    \item Different options
    \begin{itemize}
        \item No solutions
        \item All planes are parallel
        \item 2 are parallel + 3 intercept
        \item Infinite solutions (share plane or share line)
        \item One solution.
    \end{itemize}
\end{itemize}

\subsection{1/7/2021}
\begin{itemize}
    \item In Algebra I, we solved 2x2 examples such as $$x-2y=1$$$$3x+2y=11$$
    \item We would solve by adding together to get $4x=12$, $x=3$
    \item We want our solution to form a triangle.
    \item Elimination fails when there are no solutions.
    \item Elimination also fails when there are infinitely many solutions. $$x-2y=1$$$$3x-6y=3$$
    \item Take the equation $$\begin{bmatrix}2&4&-2\\4&9&-3\\-2&-3&7\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}2\\8\\10\end{bmatrix}$$
    \item We first solve to get $$\begin{bmatrix}2&4&-2\\1&3&11\\0&1&5\end{bmatrix}$$
    \item Then we get the triangular system $$\begin{bmatrix}2&4&-2\\0&3&11\\0&0&4\end{bmatrix}$$
    \item We have to multiply matrices. Each element is a dot product of a row of the first matrix with a column of the second ex. $b_{23}$ would come from taking the dot product of row 2 of the first matrix and column 3 of the second.
    \item Associative $(AB)C=A(BC)$
    \item \textbf{NOT COMMUTATIVE} $AB\neq BA$
    \item $A\textbf{x}=\textbf{b}$
    \item $A\longrightarrow$ Coefficient matrix
    \item $x\longrightarrow$ Column vector of unknowns
    \item $b\longrightarrow$ Column of scalars
    \item What if we had $b=\begin{bmatrix}2\\8\\10\end{bmatrix}$ and wanted $b=\begin{bmatrix}2\\8\\10\end{bmatrix}$
    \item To form our matrices, we will start with the identity matrix.
    \item In the example above, we would have $$\begin{bmatrix}1&0&0\\-2&1&0\\0&0&1\end{bmatrix}\begin{bmatrix}a\\b\\c\end{bmatrix}=\begin{bmatrix}a\\-2a+b\\c\end{bmatrix}$$
    \item What would we use to add 5 times the second row to the third? $E_{32}$ do $i+j$ to the position to get $\begin{bmatrix}1&0&0\\0&1&0\\0&5&1\end{bmatrix}$
    \item To subtract a multiple k of row $j$ from row $i$ is the identify matrix with a $-k$ in the $i, j$ position.
    \item $$\begin{bmatrix}1&0&0\\0&1&0\\-4&0&1\end{bmatrix}\begin{bmatrix}1\\3\\9\end{bmatrix}=\begin{bmatrix}1\\3\\5\end{bmatrix}$$
    \item Row exchange matrix (exchanges i and j) exchange them through the identity matrix.
    \item Ex. $i=1$ $j=3$, then we get matrix $\begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}$
    \item Augmented Matrix includes the two sides of the equation.
    \item Our example 4 images above would be $\begin{bmatrix}1&0&0&a\\-2&1&0&b\\0&0&0&c\end{bmatrix}$

\end{itemize}


% !TEX root = ../master/master.tex

%% Dexter Barrows, 2016
%% dbarrows.github.io

\section{Fitting Setup}

	Now that we have established which methods we wish to evaluate the efficacy of for epidemic forecasting, it is prudent to see how they perform when fitting parameters for a known epidemic model. We have already seen how they perform when fitting parameters for a model with a deterministic evolution process and observation noise, but a more realistic model will have both process and observation noise.

	To form such a model, we will take a deterministic SIR ODE model specified in Equation [\ref{sirode}] and add process noise by allowing $\beta$ to follow a geometric random walk given by

	\begin{equation}\label{betaautoreg}
		\beta_{t+1} = \exp \left( \log(\beta_{t}) + \eta (\log(\bar{\beta}) - \log(\beta_{t})) + \epsilon_{t} \right).
	\end{equation}

	We will take $\epsilon_{t}$ to be normally distributed with standard deviation $\rho^2$ such that $\epsilon_{t} \sim \mathcal{N}(0,\rho^2)$. The geometric attraction term constrains the random walk, the force of which is $\eta \in [0,1]$. If we take $\eta = 0$ then the walk will be unconstrained; if we let $\eta = 1$ then all values of $\beta_t$ will be independent from the previous value (and consequently all other values in the sequence).

	When $\eta \in (0,1)$, we have an autoregressive process of order 1 on the logarithmic scale of the form

	\begin{equation}
		X_{t+1} = c + \rho X_t + \epsilon_t ,
	\end{equation}

	where $\epsilon_t$ is normally distributed noise with mean 0 and standard deviation $\sigma_E$. This process has a theoretical expected mean of $\mu = c / (1 - \rho)$ and variance $\sigma = \sigma_E^2 / (1 - \rho^2)$. If we choose $\eta = 0.5$, the resulting log-normal distribution has a mean of $6.80 \times 10^{-4}$ and standard deviation of $4.46 \times 10^{-4}$.

	Figure [\ref{betaplot}] shows the result of simulating the process in Equation [\ref{betaautoreg}] with $\eta = 0.5$.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/betaplot.pdf}
        \caption{Simulated geometric autoregressive process shown in Equation [\ref{betaautoreg}]. \label{betaplot}}
    \end{figure}

    Figure [\ref{betadensity}] shows the density plot corresponding to the values in Figure [\ref{betaplot}].

    \begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/betadensity.pdf}
        \caption{Density plot of values shown if Figure[\ref{betaplot}]. \label{betadensity}}
    \end{figure}

    We see a density plot similar in shape to the desired density, and the geometric random walk displays dependence on previous values. Further the mean of this distribution was calculated to be $6.92 \times 10^{-4}$ and standard the deviation to be $3.99 \times 10^{-4}$, which are very close to the theoretical values.

    If we take the full stochastic SIR system and evolve it using an Euler stepping scheme with a step size of $h = 1/7$, for 1 step per day, we obtain the plot in Figure [\ref{sirmean}].

    \begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/sirmean.pdf}
        \caption{Stochastic SIR model simulated using an explicit Euler stepping scheme. The solid line is a single random trajectory, the dots show the data points obtained by adding in observation error defined as $\epsilon_{\small\text{obvs}} = \mathcal{N}(0,10)$, and the grey ribbon is centre 95th quantile from 100 random trajectories. \label{sirmean}}
    \end{figure}

\section{Calibrating Samples}

	In order to compare HMC and IF2 we need to set up a fair and theoretically justified way to select the number of samples to draw for the HMC iterations and the number of particles to use for IF2. As we wish to compare, among other things, approximate computational cost using runtimes, we need to determine how many sample draws for each method are required to obtain a certain accuracy. Sample draws are typically not comparable in terms of quality when considering multiple methods. For example, vanilla MCMC draws are computationally cheap compared to those from HMC, but HMC produces draws that more efficiently cover the sampling space. Thus we cannot just set the number of HMC draws equal to the number of particles used in IF2 -- we must calibrate both quantities based on a desired target error. We assume that we are working with a problem that has an unknown real solution, so we use the Monte Carlo Standard Error (MCSE) \cite{Harding2014}.

	Suppose we are using a Monte-Carlo based method to obtain a mean estimate $\hat{\mu}_{n}$ for a quantity $\mu$, where $n$ is the number of samples. Then the Law of Large Numbers says that $\hat{\mu}_{n} \rightarrow \mu$ as $n \rightarrow \infty$. Further, the Central Limit Theorem says that the error $\hat{\mu}_{n} - \mu$ should shrink with number of samples such that $\sqrt{n} (\hat{\mu}_{n} - \mu) \rightarrow \mathcal{N}(0,\sigma^2)$ as $n \rightarrow \infty$, where $\sigma^2$ is the variance of the samples drawn.

	We of course do not know $\mu$, but the above allows us to obtain an estimate $\hat{\sigma}_n$ for $\sigma$ given a number of samples $n$ as

	\begin{equation}
		\hat{\sigma}_n = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (X_i - \hat{\mu}) },
	\end{equation}

	which is known as the Monte Carlo Standard Error.

	We can modify this formula to account for multiple, potentially correlated, variables by replacing the single variance measure sum by

	\begin{equation}
		\Theta^* V (\Theta^*)^T
	\end{equation}

	where $\Theta^*$ is a row vector containing the reciprocals of the means of the parameters of interest, and $V$ is the variance-covariance matrix with respect to the same parameters. This in effect scales the variances with respect to their magnitudes and accounts for covariation between parameters in one fell swoop. We also divide by the number of parameters, yielding

	\begin{equation}
		\hat{\sigma}_n = \sqrt{\frac{1}{n} \frac{1}{P} \Theta^* V (\Theta^*)^T }
	\end{equation}

	where $P$ is the number of particles.

	The goal here is to then pick the number of HMC samples and IF2 particles to yield similar MCSE values. To do this we picked a combination of parameters for RStan that yielded decent results when applied to the stochastic SIR model specified above, calculated the resulting mean MCSE across several model fits, and isolated the expected number of IF2 particles needed to obtain the same value. This was used as a starting value to ``titrate'' the IF2 iterations to the same point.

	The resulting values were 1000 HMC warm-up iterations with 1000 samples drawn post-warm-up, and 2500 IF2 particles sent through 50 passes, each method giving an approximate MCSE of 0.0065.

\section{IF2 Fitting}

	Now we will use an implementation of the IF2 algorithm to attempt to fit the stochastic SIR model to the previous data. The goal here is just parameter inference, but since IF2 works by applying a series of particle filters we essentially get the average system state estimates for a very small additional computational cost. Hence, we will will also look at that estimated behaviour in addition the parameter estimates.

	The code used here is a mix of R and C++ implemented using Rcpp. The fitting was undertaken using $2500$ particles with 50 IF2 passes and a cooling schedule given by a reduction in particle spread determined by $0.975^{p}$, where p is the pass number starting with 0. This geometric cooling scheme is standard for use with IF2 \cite{King2015a}\cite{King2016}, with the cooling rate chosen to neatly scale the perturbation factor from 1 to 0.02 (almost 0) over 50 passes.

	The MLE parameter estimates, taken to be the mean of the particle swarm values after the final pass, are shown in the table in Figure [\ref{fiterror}], along with the true values and the relative error.

	\begin{figure}
		\centering
		{\small
		\begin{tabular}{cccccc}
			& & \multicolumn{2}{c}{IF2} & \multicolumn{2}{c}{HMC} \\
			\cmidrule[1.0pt](r){3-4} \cmidrule[1.0pt](r){5-6}
			Name & True	& Fit & Error & Fit & Error \\
			\cmidrule[1.0pt](r){1-1} \cmidrule[1.0pt](r){2-2} \cmidrule[1.0pt](r){3-3} \cmidrule[1.0pt](r){4-4} \cmidrule[1.0pt](r){5-5} \cmidrule[1.0pt](r){6-6}
			$\mathcal{R}_0$ 		& 3.0 				 & 3.27 					& $9.08 \times 10^{-2}$ 	& $3.12$ 				& $1.05 \times 10^{-1}$		\\
			$\gamma$ 				& $10^{-1}$  		 & $1.04 \times 10^{-1}$ 	& $3.61 \times 10^{-2}$ 	& $9.99 \times 10^{-2}$ & $-7.56 \times 10^{-4}$	\\
			Initial Infected 	& 5  				 & 7.90 					& $5.80 \times 10^{-1}$ 	& $6.64$ 				& $3.28 \times 10^{-1}$ 	\\
			$\sigma$ 			& 10   				 & 8.84 					& $-1.15 \times 10^{-1}$ 	& $8.5$ 				& $-1.50 \times 10^{-1}$	\\
			$\eta$ 				& $5 \times 10^{-1}$ & $5.87 \times 10^{-1}$ 	& $1.73 \times 10^{-1}$ 	& $4.57 \times 10^{-1}$ & $-8.27 \times 10^{-2}$ 	\\
			$\varepsilon_{err}$ & $5 \times 10^{-1}$ & $1.63 \times 10^{-1}$ 	& $-6.73 \times 10^{-1}$ 	& $1.60 \times10^{-1}$  & $-6.80 \times 10^{-1}$
		\end{tabular}}
	    \caption{Fitting errors. \label{fiterror}}
    \end{figure}

	From last IF2 particle filtering iteration, the mean state values from the particle swarm at each time step are shown with the true underlying state and data in the plot in Figure [\ref{if2state}].

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/if2state.pdf}
        \caption{True system trajectory (solid line), observed data (dots), and IF2 estimated real state (dashed line). \label{if2state}}
    \end{figure}

\section{IF2 Convergence}

	Since IF2 is an iterative algorithm where each pass through he data is expected to push the parameter estimates towards the MLE, we can see the evolution of these estimates as a function of the pass number. We expect near-convergence in the parameter estimates as IF2 nears the maximum number of passes specified. Unconvincing convergence plots may signal suboptimal algorithm parameters. If sensible algorithm parameters have been chosen, we should see the convergence plots display ``flattening'' over time.

	Figure [\ref{if2convergence}] shows evolution of the mean estimates for the six most critical parameters.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/if2convergence.pdf}
        \caption{The horizontal axis shows the IF2 pass number. The solid black lines show the evolution of the ML estimates, the solid grey lines show the true value, and the dashed grey lines show the mean parameter estimates from the particle swarm after the final pass. \label{if2convergence}}
    \end{figure}

    Figure [\ref{if2sdconvergence}] shows the evolution of the standard deviations of the parameter estimates from the particle swarm as a function of the pass number. We should expect to see asymptotic convergence to zero if the filter is converging.

    \begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/if2sdconvergence.pdf}
        \caption{The horizontal axis shows the IF2 pass number and the solid black lines show the evolution of the standard deviations of the particle swarm values. \label{if2sdconvergence}}
    \end{figure}

\section{IF2 Densities}

	Of diagnostic importance are the densities of the parameter estimates given by the final parameter swarm. If the swarm has collapsed, these densities will be extremely narrow, almost resembling a vertical line. A ``healthy'' swarm should display relatively smooth kernels of reasonable breadth.

	Figure [\ref{sc1if2kernels}] shows the parameter sample distributions from the final parameter swarm.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/if2kernels.pdf}
        \caption{The solid grey lines show the true parameter values and the dashed grey lines show the density medians. \label{sc1if2kernels}}
    \end{figure}

    The IF2 parameters chosen were in part chosen so as to not artificially narrow these densities; a more aggressive cooling schedule and/or an increased number of passes would have resulted in much narrower densities, and indeed have the potential to collapse them to point estimates. This is undesirable as it may indicate instability -- the particles may have become ``trapped'' in a region of the sampling space.

\section{HMC Fitting}

	We can use the Hamiltonian Monte Carlo algorithm implemented in the RStan package to fit the stochastic SIR model as above. This was done with a single HMC chain of 2000 iterations with 1000 of those being warm-up iterations.

	The MLE parameter estimates, taken to be the means of the samples in the chain, were shown in the table in Figure [\ref{fiterror}] along with the true values and relative error.

\section{HMC Densities}

	Figure [\ref{sc1hmckernels}] shows the parameter estimation densities from the Stan HMC fitting.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/hmckernels.pdf}
        \caption{As before, the solid grey lines show the true parameter values and the dashed grey lines show the density medians. \label{sc1hmckernels}}
    \end{figure}

    The densities shown here represent a ``true'' MLE density estimate in that they represent HMC's attempt to directly sample from the parameter space according to the likelihood surface, unlike IF2 which is in theory only trying to get a ML point estimate. Hence, these densities are potentially more robust than those produced by the IF2 implementation.

\section{HMC and Bootstrapping}

	Unlike in some models, our RStan epidemic model does not keep track of state estimates directly, but does keep track of the autoregressive process latent variable draws, which allow us to reconstruct states. This was done to ease implementation as RStan places some restrictions on how interactions between parameters and states can be specified.

	Figure [\ref{hmcboot}] shows the results of 100 bootstrap trajectories generated from the RStan HMC samples.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/hmcboot.pdf}
        \caption{Result from 100 HMC bootstrap trajectories. The solid line shows the true states, the dots show the data, the dotted line shows the average system behaviour, the dashed line shows the bootstrap mean, and the grey ribbon shows the centre 95th quantile of the bootstrap trajectories. \label{hmcboot}}
    \end{figure}

\section{Multi-trajectory Parameter Estimation}

	Here we fit the stochastic SIR model to 200 random independent trajectories using each method and examine the density of the point estimates produced.

	Figure [\ref{combinedmulti}] shows the results of the mean parameter fits from IF2 and HMC for 200 independent data sets generates using the previously described model.

    \begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/combined-multi.pdf}
        \caption{IF2 point estimate densities are shown in black and HMC point estimate densities are shown in grey. The vertical lines show the true parameter values. \label{combinedmulti}}
    \end{figure}

    The densities by and large display similar coverage, with the IF2 densities for $r$ and $\varepsilon_{proc}$ showing slightly wider coverage than the HMC densities for the same parameters.

    Figure [\ref{sc1timeplot}] summarizes the running times for each algorithm.

	\begin{figure}
        \centering
        \captionsetup{width=.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/timeplot.pdf}
        \caption{Fitting times for IF2 and HMC, in seconds. The centre box in each plot shows the centre 50th percent, with the bold centre line showing the median. \label{sc1timeplot}}
    \end{figure}

    The average running times were approximately 45.5 seconds and 257.4 seconds for IF2 and HMC respectively, representing a 5.7x speedup for IF2 over HMC. While IF2 may be able to fit the model to data faster than HMC, we are obtaining less information; this will become important in the next section. Further, the results in Figure [\ref{sc1timeplot}] show that while the running time for IF2 is relatively fixed, the times for HMC are anything but, showing a wide spread of potential times.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS637: Database-Backed Websites
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 4}

Write a simple PHP program that calculates future value of a potential investment based on key parameters investment amount, yearly interest rate and number of years.

\subsection*{Solution}

Desired PHP program has been developed on \href{http://ghorbanzade.com/cs637?p=2&s=1}{Ghorbanzade.com/637} under the name of \textit{Future Value Calculator} and is readily available for evaluation.

Following is the code snippet (slightly different than the one on the website) that calculates future value of an investment, upon submission of its related parameters.

\lstset{language=PHP}
\lstset{frame=tb}
\lstset{numbers=left}
\begin{lstlisting}
if (is_numeric($_POST['invest'])) {
	if (is_numeric($_POST['interest'])) {
		if (is_numeric($_POST['year'])) {
			$future_value = $_POST['invest'];
			for ($i = 1; $i <= $_POST['year']; $i++) {
				$future_value += $future_value * $_POST['interest'] * 0.01;
			}
			$future_value = number_format($future_value,2);
			echo "Future value is $$future_value";
		}
		else {
			echo "Year not valid.";
		}
	}
	else {
		echo "Interest rate not valid.";
	}
}
else {
	echo "Investment amount not valid.";
}
\end{lstlisting}

\section*{Question 5}
Write another simple non-web PHP program that counts and reports how many \textit{a}'s, \textit{e}'s, \textit{i}'s, \textit{o}'s and \textit{u}'s show up in a string set up at the start as \textit{this is a test, only a test}. Use an associative array with keys \textit{a}, \textit{e}, \textit{i}, \textit{o}, and \textit{u}. Use \texttt{strlen} function. You can test if a key exists in an associative array with \texttt{array\_key\_exists} function.

\subsection*{Solution}
Desired PHP program has been developed on \href{http://ghorbanzade.com/cs637?p=2&s=2}{Ghorbanzade.com/637} under the name of \textit{Vowel Counter} and is readily available for evaluation.

Following is a code snippet that analyzes a given string, upon its submission, and gives the number of occurrences of each specified vowel.

\begin{lstlisting}
	// Get the string submitted by user
	$string = $_POST['string'];
	// Initialize array of vowels
	$letter_desired = array("a","e","i","o","u");
	$list = array_fill_keys($letter_desired, 0);
	// Find occurance of each vowel
	for ($i=0; $i < strlen($string); $i++) {
		if (array_key_exists($string[$i], $list)) {
			$list[$string[$i]] += 1;
		}
	}
	// Sort array based on value in descending order
	arsort($list);
	// Prepare message
	$result = "There are ";
	$i = 0;
	foreach ($list as $char => $number) {
		$result .= ($number == 0) ? "no" : $number;
		$result .= " '<i>$char</i>'";
		$result .= ($number != 1) ? "s": "";
		$i++;
		$result .= ($i == count($list)-1) ? " and " : ", ";
	}
	$result .= "in specified string.";
	// Output message
	echo $result;
\end{lstlisting}


\documentclass[10pt, twocolumn]{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage[margin=0.25in]{geometry}
\usepackage{pdfpages}

\newcommand*{\Perm}[2]{{}^{#1}\!P_{#2}}%
\newcommand*{\Comb}[2]{{}^{#1}C_{#2}}%
\newcommand\given[1][]{\:#1\vert\:}

\begin{document}

\section{STAT 400, Aryn Harmon}

\subsection{Midterm 1 Material}

\textbf{Probability} is a real-valued function:
1. $\mathbb{P}(S) = 1$; 2. $\mathbb{P}(A) \geq 0$; 3. If $A_1, A_2$ are mutually exclusive events, $\mathbb{P}(A_1 \cup A_2) = \mathbb{P}(A_1) + \mathbb{P}(A_2)$ and so on.\\
\textbf{Inclusion-Exclusion:} $\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A \cap B)$.\\
\textbf{Conditional Probability:} $\mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}$ given $\mathbb{P}(B) > 0$. $\mathbb{P}(A|B) \neq \mathbb{P}(A)$ unless $A$ and $B$ are independent. Also see Bayes's Theorem. Probability of a string of unions given $B$ is equal to the sum of the individual conditional probabilities.\\
\textbf{Multiplication Rule:} Probability of two events both occurring: $\mathbb{P}(A \cap B) = \mathbb{P}(A) \cdot \mathbb{P}(B|A)$ or $\mathbb{P}(A \cap B) = \mathbb{P}(B) \cdot \mathbb{P}(A|B)$ (one is easier than the other).\\
\textbf{Bayes's Rule:} $\mathbb{P}(A|B) = \frac{\mathbb{P}(A) \cdot \mathbb{P}(B|A)}{\mathbb{P}(B)}$. $\mathbb{P}(A)$ is the \textit{prior probability} of $A$. $\mathbb{P}(A|B)$ is the \textit{posterior probability} of $A$ given that $B$ occurred. Use to invert probabilities.\\
\textbf{Bayes's Rule 2:} $\mathbb{P}(A|B) = \frac{\mathbb{P}(A) \cdot \mathbb{P}(B|A)}{\mathbb{P}(A) \cdot \mathbb{P}(B|A) + \mathbb{P}(A') \cdot \mathbb{P}(B|A')}$.\\
\textbf{Bayes's Rule Full:} Given some partition of $S$: $A_1 \cup \dots \cup A_k = S$. $\mathbb{P}(A_i|B) = \frac{\mathbb{P}(A_i) \cdot \mathbb{P}(B|A_i)}{\sum_{m=1}^{k} \mathbb{P}(A_m) \cdot \mathbb{P}(B|A_m)}$.\\
\textbf{Independence:} $\mathbb{P}(A|B) = \mathbb{P}(A)$, $\mathbb{P}(B|A) = \mathbb{P}(B)$, and $\mathbb{P}(A \cap B) = \mathbb{P}(A) \cdot \mathbb{P}(B)$.\\
\textbf{Pairwise Independence:} All of the following must hold: $\mathbb{P}(A \cap B) = \mathbb{P}(A) \cdot \mathbb{P}(B)$, $\mathbb{P}(A \cap C) = \mathbb{P}(A) \cdot \mathbb{P}(C)$, and $\mathbb{P}(B \cap C) = \mathbb{P}(B) \cdot \mathbb{P}(C)$.\\
\textbf{Mutual Independence:} $A$, $B$, and $C$ must be pairwise independent and in addition: $\mathbb{P}(A \cap B \cap C) = \mathbb{P}(A) \cdot \mathbb{P}(B) \cdot \mathbb{P}(C)$.
\textbf{Number of ways to take $r$ objects from $n$ candidates:}

\begin{tabular}{lll}
        & Ordered Sample & Unordered Sample      \\
W. R.   & $n^r$          & ${n+r-1 \choose r}$ \\
W/o. R. & $\Perm{n}{r}$  & ${n \choose r}$
\end{tabular}

$\Perm{n}{k}=\frac{n!}{(n-k)!}$ - permutation\\
$\binom nk=\Comb{n}{k}=\frac{n!}{k!(n-k)!}$ - combination

\textbf{Binomial Theorem:} $(a+b)^n = \sum_{r=0}^n {n \choose r} a^r b^{n-r}$. ${n \choose n-r} = {n \choose r}$, ${n \choose 0} = {n \choose n} = 1$, ${n \choose 1} = {n \choose n-1} = n$.\\
\textit{Some magic formulae:} 1. $\sum_{r=0}^n {n \choose r} = 2^n$. 2. $\sum_{r=0}^n (-1)^r {n \choose r} = 0$. 3. $\sum_{r=0}^n {n \choose r} p^r (1-p)^{n-r} = 1$.\\
\textbf{Pascal's Equation:} ${n \choose r} = {n-1 \choose r} + {n-1 \choose r-1}$.\\
\textbf{Hypergeometric Distribution:} $f(x) = \mathbb{P}(X=x) = \frac{{N_1 \choose x} \cdot {N_2 \choose n-x}}{{N \choose n}}$, where $x \leq n$, $x \leq N_1$, $n-x \leq N_2$ and $N = N_1 + N_2$. $\mathbb{E}(X) = n\frac{N_1}{N}$ and $Var(X) = n \cdot \frac{N_1}{N} \cdot \frac{N_2}{N} \cdot \frac{N-n}{N-1}$. Example: Urn model: $N_1$ red balls, $N_2$ blue balls, draw $n$ balls from $N_1 + N_2$ balls, then look at the probability that there are $x$ red balls in the selected $n$ balls.\\
\textbf{Mean:} $\mu = \mathbb{E}(X) = \sum_{x \in S} x f(x)$.\\
\textbf{Variance:} $\sigma^2 = Var(X) = \mathbb{E}(X - \mu)^2 = \mathbb{E}(X^2) - \mu^2 = \sum_{i=1}^{k} x_i^2 f(x_i) - \mu^2$. \textbf{Standard Deviation:} $\sigma$.\\
\textbf{r-th Moment:} $\mathbb{E}(|X|^r) = \sum_{x \in S} |x|^r f(x) < \infty$; is the moment about the origin.\\
\textbf{r-th Moment about $b$:} $\mathbb{E}((X-b)^r) = \sum_{x \in S} (x-b)^r f(x)$; is the moment about $b$. Facts: $\mu$ is the first moment of $X$ about the origin. $\sigma^2$ is the second moment of $X$ about $\mu$.\\
\textit{Example of Variance Properties:} $Var(aX+b) = a^2 \cdot Var(X)$.
\textbf{Bernoulli Distribution:} A random experiment is called a set of Bernoulli trials if each trial has only two outcomes, has a constant $p$, and each trial is independent. $f(x) = p$ if $x=1$, $1-p$ if $x=0$, with $0 \leq p \leq 1$. $\mathbb{E}(X) = p$ and $Var(X) = p(1-p)$.\\
\textbf{Binomial Distribution:} Let $X$ be the number of successes in $n$ independent Bernoulli trials with $p$. Then $X \sim b(n,p)$. $f(x) = \mathbb{P}(X=x) = {n \choose x} p^x (1-p)^{n-x}$. $\mathbb{E}(X) = np$ and $Var(X) = np(1-p)$.\\
A note: binomial and hypergeometric are similar, but binomial has replacement and one category, and hypergeometric has two categories and no replacement.\\
\textbf{Cumulative Distribution Function:} $F(x) = \mathbb{P}(X \leq x), x \in (-\infty, \infty)$. For discrete random variables: $f(x) = \mathbb{P}(X=x) = \mathbb{P}(X \leq x) - \mathbb{P}(X \leq x - 1) = F(x) - F(x-1)$.\\
\textbf{Geometric Distribution:} $f(x) = p(1-p)^{x-1}, x = 1,2,3,\dots$. $X$ represents the draw in which the first success is drawn. $f(x)$ is the probability of getting a success in the $x$-th draw.\\
\textbf{Negative Binomial Distribution:} $f(x) = {x-1 \choose r-1} p^r (1-p)^{x-r}$ and $x = r,r+1,r+2,\dots$. $X$ is the number of trials it takes to get $r$ successes. $f(x)$ is the probability that the $r$-th success occurs on the $x$-th trial.

\subsection{Midterm 2 Material}
\textbf{Moment Generating Function:} $M(t) = \mathbb{E}(e^{tX}) = \sum_{x \in S}{e^{tX}f(x)}$ if $f(x)$ is the p.d.f. of some distribution and $t \in V_h(0)$ is finite. \textit{Theorem:} $\mathbb{E}(X^r) = M^{(r)}(0)$, so $\mu = \mathbb{E}(X) = M'(0)$ and $\sigma^2 = \mathbb{E}(X^2) - [\mathbb{E}(X)]^2 = M''(0) - [M'(0)]^2$. To calculate the m.g.f. and p.d.f. of some random variable given only its moments, use the Taylor series expansion centered at zero. $M(t) = M(0) + M'(0)\left(\frac{t}{1!}\right) + M''(0)\left(\frac{t^2}{2!}\right) + \dots = 1 + \mathbb{E}(X)\left(\frac{t}{1!}\right) + \mathbb{E}(X^2)\left(\frac{t^2}{2!}\right) + \dots$.\\
\textbf{Poisson distribution:} \textit{Definition:} Poisson process counts the number of events occurring in a fixed time/space given a rate $\lambda$. Let $X_t$ be the number events which occur in $t$ unit time intervals. $\mathbb{P}(X_t = x) = \frac{(\lambda t)^x}{x!} e^{-\lambda t}$. $\mu = \lambda, \sigma^2 = \lambda$.\\
\textbf{Poisson Approximation} of Binomial Distribution: If $X \sim b(n,p)$ and $n$ is large while $p$ is small, then $X$ can be approximated as $\hat{X} \sim poi(\lambda) \, s.t. \, \lambda = np$.\\
\textbf{Mean:} $\mu = \int_{-\infty}^{\infty} x f(x) dx$.\\
\textbf{Variance:} $\sigma^2 = \int_{-\infty}^{\infty} (x - \mu)^2 f(x) dx$.\\
\textbf{M.G.F.:} $M(t) = \mathbb{E}(e^{tX}) = \int_{-\infty}^{\infty} e^{tx} f(x) dx, t \in (-h,h)$.\\
\textbf{Percentile:} $p = \int_{-\infty}^{\pi_p} f(x) dx = F(\pi_p)$.\\
\textbf{Uniform Distribution:} $f(x) = \frac{1}{b-a}, a \leq x \leq b$. If $X \sim U(a,b)$, then $\mathbb{E}(X) = \frac{a+b}{2}$, $\sigma^2 = \frac{(b-a)^2}{12}$, and $M(t) = \mathbb{E}(e^{tX}) = \frac{e^{tb} - e^{ta}}{t(b-a)}, t \neq 0$ and $M(0) = 1$.\\
\textbf{Exponential Distribution:} This describes the waiting time between events in a Poisson process with rate $\lambda$. Let $\theta = \frac{1}{\lambda}$. Then if $X \sim Exp(\theta)$, $f(x) = \frac{1}{\theta}e^{-\frac{x}{\theta}}, x \geq 0$ and 0 otherwise. $\mu = \theta$, $\sigma^2 = \theta^2$, and $M(t) = (1 - \theta t)^{-1}, t < \frac{1}{\theta}$.\\
\textbf{Memoryless Property} of the Exponential Distribution: What happened in the past does not matter now. Only the present can determine the future. $\mathbb{P}(X > a+b \given[\big] X > a) = \mathbb{P}(X > b), \forall a, b \geq 0$.\\
\textbf{Gamma Function:} $\Gamma(x) = \int_{0}^{\infty} y^{x-1} e^{-y} dy, \, x > 0$.\\
\textbf{Gamma Distribution:} $f(t)$ represents the waiting time unitl the $\alpha$-th occurrence of some Poisson process with rate $\lambda$. $f(t) = \frac{\lambda^\alpha t^{\alpha-1}}{(\alpha-1)!} e^{-\lambda t}, \, t \geq 0$. Then let $\theta = \lambda^{-1}$, then $f(t) = \frac{1}{\Gamma(\alpha)} \theta^{\alpha} t^{\alpha - 1} e^{\frac{-t}{\theta}}, \, t \geq 0, \, \alpha \in \mathbb{R}$. $\mu = \alpha \theta$, $\sigma^2 = \alpha \theta^2$, and $M(t) = (1 - \theta t)^{-\alpha}$. If $\alpha = 1$, $\mathrm{gamma}(1,\theta) = \mathrm{Exp}(\theta)$.\\
\textbf{$\chi^2$ Distribution:} If $X \sim \mathrm{gamma}(\alpha,\theta)$, and $\theta = 2$ and $\alpha = \frac{r}{2}$, where $r$ is a positive integer, then $X$ is a $\chi^2$ distribution with degree of freedom $r$. $X \sim \chi^2(r)$. $\mu = r$, $\sigma^2 = 2r$, and $M(t) = (1-2t)^{\frac{-r}{2}}, \, t < \frac{1}{2}$. $e^2 = \chi^2(2)$.\\
\textbf{Normal Distribution:} Bell curve! $f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}}, \, x \in \mathbb{R}$. $X \sim N(\mu, \sigma^2)$. $X \sim N(0, 1)$ is the standard normal distribution. $\mu = \mu$, $\sigma^2 = \sigma^2$, and $M(t) = e^{\mu t + \frac{\sigma^2 t^2}{2}}$. Standardization: $X \sim N(\mu, \sigma^2)$, then $Z = \frac{X-\mu}{\sigma} \sim N(0, 1)$.\\
\textbf{Normal Square Distribution:} Let $X \sim N(\mu, \sigma^2)$ and $Z = \frac{X-\mu}{\sigma}$. Then the random variable $V = Z^2 \sim \chi^2(1)$.\\
\textbf{Cauchy Distribution:} (Why do we even need this distribution?) $f(x) = \frac{1}{\pi (1 + x^2)}, \, x \in \mathbb{R}$. Symmetric about zero, so median is zero, but $\mu$ is undefined because the tail of the p.d.f. is too heavy (i.e. each integral of the distribution does not converge). $c.d.f. = F(x) = \frac{1}{\pi} \arctan(x) + \frac{1}{2}, \, x \in \mathbb{R}$.\\
\textbf{Joint Probability Density Function:} $\mathbb{P}((X, Y) \in A) = \int\int f(x,y) dx dy$.\\
\textbf{Marginal Probability Density Function:} $f_x(x) = \int_{-\infty}^{\infty} f(x,y) dy$ and the other way around for $y$.\\
\textbf{Mathematical Expectation:} $\mathbb{E}[g(X,Y)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x,y) f(x,y) dx dy$.\\
\textbf{Independent Random Variables:} Two events $A$ and $B$ are independent iff $f(x,y) = f_x(x) \cdot f_y(y) \, \forall x,y$. This works for both the p.d.f. and the c.d.f.\\
\textbf{Trinomial Distribution:} This is an extension of the binomial distribution into two dimensions. $f(x_1,x_2) = \mathbb{P}(X_1 = x_1, X_2 = x_2) = \frac{n!}{x_1! x_2! (n - x_1 - x_2)!} p_1^{x_1} p_2^{x_2} (1 - p_1 - p_2)^{n - x_1 - x_2}$ for $x_1, x_2 \geq 0$, $x_1 + x_2 \leq n$.\\
\textbf{Covariance:} $Cov(X,Y) = \mathbb{E}[(X - \mu_x)(Y - \mu_y)], \mu_x = \mathbb{E}(X), \mu_y = \mathbb{E}(Y)$. Useful identity: $Cov(X,Y) = \mathbb{E}(XY) - \mathbb{X}\mathbb{E}(Y)$.\\
\textbf{Properties of Covariance:} Zero-angle: $Cov(X,X) = Var(X)$. Symmetry: $Cov(X,Y) = Cov(Y,X)$. $Cov(aX + b, Y) = a \cdot Cov(X,Y)$. $Cov(X+Y, W) = Cov(X,W) + Cov(Y,W)$. $Cov(aX+bY, cX+dY) = (ac) \cdot Var(X)+(bd) \cdot Var(Y)+(ad+bc) \cdot Cov(X, Y)$. $Var(aX + bY) = a^2 \cdot Var(X) + 2ab \cdot Cov(X, Y) + b^2 \cdot Var(Y)$.\\
\textbf{Correlation Coefficient:} $\rho_{XY} = \frac{\sigma_{XY}}{\sigma_X \cdot \sigma_Y} = \frac{Cov(X,Y)}{\sqrt{Var(X)} \cdot \sqrt{Var(Y)}}$. Uncorrelated: $\rho_{XY} = 0 \Leftrightarrow \sigma_{XY} = 0$. If $X$ and $Y$ are independent, then $\rho_{XY} = 0$, but the implication \textbf{does not} go the other way.\\
\textbf{I.I.D.:} If every marginal probability density function is equal and independent, then all of the random variables are independent and identically distributed (i.i.d.).\\
\textbf{Combinations:} If $Y$ is a linear combination of independent random variables with constant multiples, then the expectation of $Y$ is simply the sum of the expectations of random variables. The variance of $Y$ is the sum of the variances of the random variables with their constant multiples squared. The moment of a linear combination of random variables is the product of their individual moments.\\
\textbf{Summary of Combinations:} Suppose $X$ and $Y$ are independent. Then,
\begin{itemize}
        \item $bern(p) + bern(p) = b(2,p)$
        \item $b(n_1,p) + b(n_2,p) = b(n_1 + n_2, p)$
        \item $geo(p) + geo(p) = negbin(2,p)$
        \item $negbin(r_1,p) + negbin(r_2,p) = negbin(r_1 + r_2, p)$
        \item $poi(\lambda_1) + poi(\lambda_2) = poi(\lambda_1 + \lambda_2)$
        \item $exp(\theta) + exp(\theta) = gamma(2,\theta)$
        \item $gamma(\alpha_1,\theta) + gamma(\alpha_2,\theta) = gamma(\alpha_1 + \alpha_2, \theta)$
        \item $N(\mu_1,\sigma_1^2) + N(\mu_2,\sigma_2^2) = N(\mu_1 + \mu_2,\sigma_1^2 + \sigma_2^2)$
\end{itemize}

\subsection{Final Material}
Everything taught in the class is on the final.\\
\textbf{Chebyshev's Inequality:} If $k \geq 1$: $\mathbb{P}(|X - \mu| \geq k \sigma) \leq \frac{\sigma^2}{\epsilon^2}$. Let $\epsilon = k \sigma$: $\mathbb{P}(|X - \mu| \geq \epsilon) \leq \frac{1}{k^2}$.\\
\textbf{Central Limit Theorem (CLT):} $\frac{\sqrt{n}(\overline{X} - \mu)}{\sigma} \rightarrow Z, \;\mathrm{as}\; n \rightarrow \infty$.\\
\textbf{Normal Approximation to Binomial:} If $np \geq 5$ and $n(1-p) \geq 5$, then\\
\begin{tabular}{lll}
        & $b(n,p)$ & $N(\mu,\sigma^2)$     \\
Mean    & $np$          & $\mu$ \\
Variance& $np(1-p)$  & $\sigma^2$
\end{tabular}\\
\textbf{Normal Approximation to Poisson:}\\
\begin{tabular}{lll}
        & Poisson$(\lambda)$ & $N(\mu,\sigma^2)$     \\
Mean    & $\lambda$          & $\mu$ \\
Variance& $\lambda$  & $\sigma^2$
\end{tabular}\\
\textbf{Jensen's Inequality:} If $g : \mathbb{R} \rightarrow \mathbb{R}$ is a convex function and $X$ is a random variable, then $\mathbb{E}[g(X)] \geq g(\mathbb{E}(X))$.\\
\textbf{Maximum Likelihood Estimator:} Maximize the likelihood function subject to the parameter space.
\begin{enumerate}
	\item Construct likelihood function, $F(x; \theta) = \Pi_{i=1}^{n}(f(x_i))$.
	\item If applicable, take the log of this to make things easier to differentiate and to change the product into a sum. $l(x; \theta)$.
	\item Differentiate to get $\frac{dl(x;\theta)}{d\theta}$.
	\item Assume convexity (ha) and find optimal $\theta$ by solving $\frac{dl(x;\theta)}{d\theta} = 0$.
\end{enumerate}
\textbf{Sample Mean:} $\frac{1}{n}\sum_{i=1}^{n}X_i \equiv \overline{X}$.\\
\textbf{Sample Variance:} For $z$ distribution: $\frac{1}{n}\sum_{i=1}^{n}(X_i - \overline{X})^2$, and for $t$ distribution: $\frac{1}{n-1}\sum_{i=1}^{n}(X_i - \overline{X})^2$.\\
\textbf{Method of Moments Estimator:} Construct a system of equations using moments and sample data. For example, equate theoretical mean with sample mean, and theoretical variance with sample variance, then solve. \texttt{<><><maybe put an example here<><>><}\\
\textbf{Helpful Distribution Values:}\\
\begin{tabular}{ll}
$\alpha$ & $z_{1-\alpha}$ \\
0.2      & 0.840 \\
0.1      & 1.280 \\
0.05     & 1.645 \\
0.025    & 1.960 \\
\end{tabular}\\
\textbf{Confidence Intervals for Means:}\\
\tiny
\begin{tabular}{rcc}
                & Known Variance & Unknown Variance     \\
Two Side       & $\left[ \overline{X} - z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}}, \overline{X} + z_{\frac{\alpha}{2}} \frac{\sigma}{\sqrt{n}} \right]$ & $\left[ \overline{X} - t_{\frac{\alpha}{2}}(n-1) \frac{S}{\sqrt{n}}, \overline{X} + t_{\frac{\alpha}{2}}(n-1) \frac{S}{\sqrt{n}} \right]$ \\
Lower & $\left[ \overline{X} - z_{\alpha} \frac{\sigma}{\sqrt{n}}, \infty \right)$ & you can \\
Upper & $\left( -\infty, \overline{X} + z_{\alpha} \frac{\sigma}{\sqrt{n}} \right]$ & figure it out \\
\end{tabular}\\
\normalsize
$S^2 = \frac{1}{n-1}\sum_{i=1}^{n}(X_i - \overline{X})^2$ be the sample variance.
When the population distribution is normal, use the ($n-1$) version unless you need to use MLE for which the ($n$) version is needed. For other distributions, use the ($n$) version. If no distribution is mentioned, then assume it to be normal and use the ($n-1$) version.\\
\textbf{Confidence Interval for Variance:} $\left[ \frac{(n-1)S^2}{b}, \frac{(n-1)S^2}{a} \right], a=\chi_{1-\frac{\alpha}{2}}^2(n-1), b=\chi_{\frac{\alpha}{2}}^2(n-1)$.\\
\textbf{Confidence Interval for Standard Deviation:} Use $a$ and $b$ from above, and just square root the expressions for the variance C.I.\\
\textbf{Confidence Interval for Proportions (and Wilcox Interval for large $n$):} Let $\hat{p} = \frac{Y}{n}$. Then $\hat{p} \pm z_{\frac{\alpha}{2}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$. Lower bound: $\left[ \hat{p} - z_{\alpha} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, 1 \right]$. Upper bound: $\left[ 0, \hat{p} + z_{\alpha} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} \right]$.\\

\includepdf[pages={1-},scale=1.0]{gdoc.pdf}

\end{document}


%!TEX root = ../thesis.tex
% ******************************* Thesis Appendix A ****************************
\chapter{Experiment Specifications}

\section*{Bayesian Settings}

\subsection{Image Recognition}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}[c]{c | c}
     \hline
     variable & value \\ [0.5ex]
     \hline
     learning rate &  0.001\\

     epochs & 100 \\

     batch size & 256 \\

     sample size & 10-25 \\

     loss & cross-entropy \\

     $(\alpha \mu^2)_{init}$ of approximate posterior $q_{\theta}(w|\mathcal{D})$ & -10 \\

     optimizer & Adam \cite{kingma2014adam} \\

     $\lambda$ in $\ell$-2 normalisation & 0.0005 \\

     $\beta_i$ & $\frac{2^{M-i}}{2^M-1}$ \cite{blundell2015weight} \\ [1ex]
     \hline
    \end{tabular}
    \renewcommand{\arraystretch}{2}
\end{table}

Sample size can vary from 10 to 25 as this range provided the best results. However, it can be played around with. For most of our experiments, it is either 10 or 25 unless specified otherwise.

\subsection{Image Super Resolution}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}[c]{c | c}
     \hline
     variable & value \\ [0.5ex]
     \hline
     learning rate &  0.01\\

     epochs & 200 \\

     batch size & 64 \\

     upscale factor & 3 \\

     loss & Mean Squared Error \\

     seed & 123 \\

     $(\alpha \mu^2)_{init}$ of approximate posterior $q_{\theta}(w|\mathcal{D})$ & -10 \\

     optimizer & Adam \cite{kingma2014adam} \\

     $\lambda$ in $\ell$-2 normalisation & 0.0005 \\

     $\beta_i$ & $\frac{2^{M-i}}{2^M-1}$ \cite{blundell2015weight} \\ [1ex]
     \hline
    \end{tabular}
    \renewcommand{\arraystretch}{2}
\end{table}

\subsection{Generative Adversarial Network}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}[c]{c | c}
     \hline
     variable & value \\ [0.5ex]
     \hline
     learning rate &  0.001\\

     epochs & 100 \\

     batch size & 64 \\

     image size & 64 \\

     latent vector (nz) & 100 \\

     number of generator factor (ndf) & 64 \\

     number of discriminator factor (ndf) & 64 \\

     upscale factor & 3 \\

     loss & Mean Squared Error \\

     number of channels (nc) & 3 \\

     $(\alpha \mu^2)_{init}$ of approximate posterior $q_{\theta}(w|\mathcal{D})$ & -10 \\

     optimizer & Adam \cite{kingma2014adam} \\

     $\lambda$ in $\ell$-2 normalisation & 0.0005 \\

     $\beta_i$ & $\frac{2^{M-i}}{2^M-1}$ \cite{blundell2015weight} \\ [1ex]
     \hline
    \end{tabular}
    \renewcommand{\arraystretch}{2}
\end{table}

\section*{Non Bayesian Settings}

\subsection{Image Recognition}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}[c]{c | c}
     \hline
     variable & value \\ [0.5ex]
     \hline
     learning rate &  0.001\\

     epochs & 100 \\

     batch size & 256 \\

     loss & cross-entropy \\

     initializer & Xavier \cite{glorot2010understanding} or Normal \\

     optimizer & Adam \cite{kingma2014adam} \\ [1ex]
     \hline
    \end{tabular}
    \renewcommand{\arraystretch}{2}
\end{table}

The weights were initialized with Xavier initialization \cite{glorot2010understanding} at first, but to make it consistent with the Bayesian networks where initialization was Normal initialization (mean = 0 and variance = 1), the initializer was changed to Normal initialization.

\pagebreak

\section*{Architectures}

\subsection{LeNet-5}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}{c c c c c c}
     \hline
     layer type & width & stride & padding & input shape & nonlinearity \\ [0.5ex]
     \hline
     convolution ($5\times5$) & 6 & 1 & 0 & $M\times1\times32\times32$ & Softplus \\

     Mmax-pooling ($2\times2$) & \empty & 2 & 0 & $M\times6\times28\times28$ & \empty \\

     convolution ($5\times5$) & 16 & 1 & 0 & $M\times1\times14\times14$ & Softplus \\

     max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times16\times10\times10$ & \empty \\

     fully-connected & 120 & \empty & \empty & $M\times400$ & Softplus \\

     fully-connected & 84 & \empty & \empty & $M\times120$ & Softplus \\

     fully-connected & 10 & \empty & \empty & $M\times84$ & \empty \\ [1ex]
     \hline
    \end{tabular}
    \renewcommand{\arraystretch}{1}
    \label{tab:LeNet}
    \caption{LeNet architecture with original configurations as defined in the paper. \cite{lecun1998gradient}}
\label{tab:AlexNet}

\end{table}

\subsection{AlexNet}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}{c c c c c c}
 \hline
 layer type & width & stride & padding & input shape & nonlinearity \\ [0.5ex]
 \hline
 convolution ($11\times11$) & 64 & 4 & 5 & $M\times3\times32\times32$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times64\times32\times32$ & \empty \\

 convolution ($5\times5$) & 192 & 1 & 2 & $M\times64\times15\times15$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times192\times15\times15$ & \empty \\

 convolution ($3\times3$) & 384 & 1 & 1 & $M\times192\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 256 & 1 & 1 & $M\times384\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 128 & 1 & 1 & $M\times256\times7\times7$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times128\times7\times7$ & \empty \\

 fully-connected & 128 & \empty & \empty & $M\times128$ & \empty \\ [1ex]
 \hline
\end{tabular}
\renewcommand{\arraystretch}{1}
\caption{AlexNet architecture with original configurations as defined in the paper. \cite{krizhevsky2012imagenet}}
\label{tab:AlexNet}
\end{table}

\subsection{AlexNetHalf}

\begin{table}[H]
    \centering
    \renewcommand{\arraystretch}{2}
    \begin{tabular}{c c c c c c}
 \hline
 layer type & width & stride & padding & input shape & nonlinearity \\ [0.5ex]
 \hline
 convolution ($11\times11$) & 32 & 4 & 5 & $M\times3\times32\times32$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times32\times32\times32$ & \empty \\

 convolution ($5\times5$) & 96 & 1 & 2 & $M\times32\times15\times15$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times96\times15\times15$ & \empty \\

 convolution ($3\times3$) & 192 & 1 & 1 & $M\times96\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 128 & 1 & 1 & $M\times192\times7\times7$ & Softplus \\

 convolution ($3\times3$) & 64 & 1 & 1 & $M\times128\times7\times7$ & Softplus \\

 max-pooling ($2\times2$) & \empty & 2 & 0 & $M\times64\times7\times7$ & \empty \\

 fully-connected & 64 & \empty & \empty & $M\times64$ & \empty \\ [1ex]
 \hline
\end{tabular}
\renewcommand{\arraystretch}{1.5}
\label{tab:AlexNetHalfArchitecture}
\caption{AlexNetHalf with number of filters halved compared to the original architecture.}
\end{table}