\section{Extensions of the Base Logic}

In this section we discuss some additional constructions that we define within and on top of the base logic.
These are not ``extensions'' in the sense that they change the proof power of the logic, they just form useful derived principles.

\subsection{Derived Rules about Base Connectives}
We collect here some important and frequently used derived proof rules.

\begin{mathparpagebreakable}
  \infer{}
  {\prop \Ra \propB \proves \prop \wand \propB}

  \infer{}
  {\prop * \Exists\var.\propB \provesIff \Exists\var. \prop * \propB}

  \infer{}
  {\prop * \All\var.\propB \proves \All\var. \prop * \propB}
\end{mathparpagebreakable}
Verifying that existential quantifiers commute with separating conjunction requires an intermediate step using a magic wand: From $P * \exists x, Q \vdash \Exists x. P * Q$ we can deduce $\Exists x. Q \vdash P \wand \Exists x. P * Q$ and then proceed via $\exists$-elimination.

\subsection{Derived Rules about Modalities}

Iris comes with 4 built-in modalities ($\always$, $\plainly$, $\upd$ and $\later$) and, as we will see, plenty of derived modalities.
However, almost all of them fall into one of two categories (except for $\later$, as we will see): they are either \emph{always-style} modalities (``something holds in all/many (future) worlds'') or \emph{eventually-style} modalities (``something holds in a possible (future) world'').

\emph{Eventually-style modalities} are characterized by being easy to ``add''/introduce, but hard to ``remove''/eliminate.
Consider, for example, the basic update modality $\upd$:
we have $\prop \proves \upd\prop$ (\ruleref{upd-intro}), but the inverse direction does not hold.
Instead, from \ruleref{upd-mono} and \ruleref{upd-trans}, we can derive the following elimination principle:
\begin{mathpar}
  \infer[upd-E]
  {\prop \proves \upd\propB}
  {\upd\prop \proves \upd\propB}
\end{mathpar}
In other words, we can remove an $\upd$ in front of an assumption \emph{if} the goal is itself wrapped in $\upd$.
Another way to view this rule is to think of it as a \emph{bind rule}.
Indeed, together with \ruleref{upd-intro}, this rule shows that $\upd$ forms a monad.

\emph{Always-style modalities}, on the other hand, are easy to ``remove''/eliminate, but hard to ``add''/introduce.
The most widely used example of that in Iris is the persistence modality $\always$:
we have $\always\prop \proves \prop$ (\ruleref{pers-elim}), but the inverse direction does not hold.
Instead, from \ruleref{pers-mono} and $\always{\prop} \proves \always\always\prop$, we can derive the following introduction principle:
\begin{mathpar}
  \infer[$\always$-I]
  {\always\prop \proves \propB}
  {\always\prop \proves \always\propB}
\end{mathpar}
In other words, we can remove an $\always$ from the goal \emph{if} all our assumptions are wrapped in $\always$.
This matches the algebraic structure of a comonad.

In particular, both eventually-style and always-style modalities are \emph{idempotent}: we have $\upd\upd\prop \provesIff \upd\prop$ and $\always\always\prop \provesIff \always\prop$.

Beyond this, all modalities come with plenty of rules that show how they commute around other connectives and modalities.
And, of course, they come with a few ``defining rules'' that give the modalities their individual meaning, \ie for the update modality, that would be \ruleref{upd-update}.

In the following, we briefly discuss each of the modalities.

\paragraph{Update modality.}
As already mentioned, the update modality is an eventually-style modality:
\begin{mathpar}
  \inferhref{upd-E}{upd-elim}
  {\prop \proves \upd\propB}
  {\upd\prop \proves \upd\propB}

  \inferH{upd-idemp}
  {}{\upd\upd\prop \provesIff \upd\prop}
\end{mathpar}
Beyond this (and the obvious variant of \ruleref{upd-frame} that exploits commutativity of separating conjunction), there are no outstandingly interesting derived rules.

\paragraph{Persistence modality.}
As already mentioned, the persistence modality is an always-style modality:
\begin{mathpar}
  \inferhref{$\always$-I}{pers-intro}
  {\always\prop \proves \propB}
  {\always\prop \proves \always\propB}

  \inferhref{$\always$-idemp}{pers-idemp}
  {}{\always\always\prop \provesIff \always\prop}
\end{mathpar}
Some further interesting derived rules include:
\begin{mathparpagebreakable}
  \infer{}
  {\always(\prop\land\propB) \provesIff \always\prop \land \always\propB}

  \infer{}
  {\always(\prop\lor\propB) \provesIff \always\prop \lor \always\propB}

  \infer{}
  {\always\TRUE \provesIff \TRUE}

  \infer{}
  {\always\FALSE \provesIff \FALSE}
\\
  \infer{}
  {\always(\prop*\propB) \provesIff \always\prop * \always\propB}

  \infer{}
  {\always\prop*\propB \provesIff \always\prop \land \propB}

  \infer{}
  {\always(\prop \wand \propB) \provesIff \always(\prop \Ra \propB)}
\\
  \infer{}
  {\always(\prop \Ra \propB) \proves \always\prop \Ra \always\propB}

  \infer{}
  {\always(\prop \wand \propB) \proves \always\prop \wand \always\propB}

\end{mathparpagebreakable}
In particular, the persistence modality commutes around conjunction, disjunction, separating conjunction as well as universal and existential quantification.
Commuting around conjunction can be derived from the primitive rule that says it commutes around universal quantification (as conjunction is equivalent to a universal quantification of a Boolean), and similar for disjunction.
$\TRUE \provesIff \always\TRUE$ (which is basically persistence ``commuting around'' the nullary operator $\TRUE$) can be derived via $\always$ commuting with universal quantification ranging over the empty type.
A similar rule holds for $\FALSE$.

Moreover, if (at least) one conjunct is below the persistence modality, then conjunction and separating conjunction coincide.

\paragraph{Plainness modality.}
The plainness modality is very similar to the persistence modality (in fact, we have $\plainly\prop \proves \always\prop$, but the inverse does not hold).
It is always-style:
\begin{mathpar}
  \infer[$\plainly$-I]
  {\plainly\prop \proves \propB}
  {\plainly\prop \proves \plainly\propB}

  \infer{}{\plainly\plainly\prop \provesIff \plainly\prop}
\end{mathpar}
It also commutes around separating conjunction, conjunction, disjunction, universal and existential quantification (and $\TRUE$ and $\FALSE$).

The key difference to the persistence modality $\always$ is that $\plainly$ provides a \emph{propositional extensionality} principle:
\[ \plainly ( ( P \Ra Q) \land (Q \Ra P ) ) \proves P =_{\Prop} Q \]
In contrast, $\always$ permits using some forms of ghost state ($\ownM\melt \proves \always{\ownM{\mcore\melt}}$).

Having both of these principles for the same modality would lead to a contradiction:
imagine we have an RA with elements $\melt$, $\meltB$ such that $\mcore\melt$ is incompatible with $\meltB$ (\ie $\neg\mvalFull(\mcore\melt \mtimes \meltB)$).
Then we can prove:
\[
\ownM{\mcore\melt} \proves
\always\ownM{\mcore\melt} \proves
\always ( ( \FALSE \Ra \ownM\meltB ) \land ( \ownM\meltB \Ra \FALSE ) )
\]
The first implication is trivial, the second implication follows because $\always\ownM{\mcore\melt} \land \ownM\meltB \proves \ownM{\mcore\melt} * \ownM\meltB \proves \mval(\mcore\melt \mtimes \meltB)$.

But now, if we had propositional extensionality for $\always$ the way we do for $\plainly$, we could deduce $\FALSE =_{\Prop} \ownM\meltB$, and that is clearly wrong.
This issue arises because $\always$, as we have seen, still lets us use some resources from the context, while propositional equality has to hold completely disregarding current resources.

\paragraph{Later modality.}
The later modality is the ``odd one out'' in the sense that it is neither eventually-style nor always-style, because it is not idempotent:%
\footnote{This means $\later$ is neither a monad nor a comonad---it does form an applicative functor, though.}
with $\later$, the number of times the modality is applied matters, and we can get rid of \emph{exactly one} layer of $\later$ in the assumptions only by doing the same in the conclusion (\ruleref{later-mono}).

Some derived rules:
\begin{mathparpagebreakable}
  \inferhref{L{\"o}b}{Loeb}
  {}
  {(\later\prop\Ra\prop) \proves \prop}

  \infer{}
  {\later(\prop \Ra \propB) \proves \later\prop \Ra \later\propB}

  \infer{}
  {\later(\prop \wand \propB) \proves \later\prop \wand \later\propB}
\\
  \infer{}
  {\later(\prop\land\propB) \provesIff \later\prop \land \later\propB}

  \infer{}
  {\later(\prop\lor\propB) \provesIff \later\prop \lor \later\propB}

  \infer{\text{$\type$ is inhabited}}
  {\later(\Exists x:\type. \prop) \provesIff \Exists x:\type. \later\prop}

  \infer{}
  {\later\TRUE \provesIff \TRUE}

  \infer{}
  {\later(\prop*\propB) \provesIff \later\prop * \later\propB}

  \infer{}
  {\later\always\prop \provesIff \always\later\prop}

  \infer{}
  {\later\plainly\prop \provesIff \plainly\later\prop}
\end{mathparpagebreakable}
Noteworthy here is the fact that Löb induction (\ruleref{Loeb}) can be derived from $\later$-introduction and the fact that we can take fixed-points of functions where the recursive occurrences are below $\later$~\cite{Loeb}.%
\footnote{Also see \url{https://en.wikipedia.org/wiki/L\%C3\%B6b\%27s_theorem}.}
Also, $\later$ commutes over separating conjunction, conjunction, disjunction, universal quantification and \emph{non-empty} existential quantification, as well as both the persistence and the plainness modality.

\subsection{Persistent Propositions}
We call a proposition $\prop$ \emph{persistent} if $\prop \proves \always\prop$.
These are propositions that ``do not own anything'', so we can (and will) treat them like ``normal'' intuitionistic propositions.

Of course, $\always\prop$ is persistent for any $\prop$.
Furthermore, by the proof rules given in \Sref{sec:proof-rules}, $\TRUE$, $\FALSE$, $t = t'$ as well as $\ownGhost\gname{\mcore\melt}$ and $\mval(\melt)$ are persistent.
Persistence is preserved by conjunction, disjunction, separating conjunction as well as universal and existential quantification and $\later$.

\subsection{Timeless Propositions and Except-0}

One of the troubles of working in a step-indexed logic is the ``later'' modality $\later$.
It turns out that we can somewhat mitigate this trouble by working below the following \emph{except-0} modality:
\[ \diamond \prop \eqdef \later\FALSE \lor \prop \]
Except-0 satisfies the usual laws of a ``monadic'' modality (similar to, \eg the update modalities):
\begin{mathpar}
  \inferH{ex0-mono}
  {\prop \proves \propB}
  {\diamond\prop \proves \diamond\propB}

  \axiomH{ex0-intro}
  {\prop \proves \diamond\prop}

  \axiomH{ex0-idem}
  {\diamond\diamond\prop \proves \diamond\prop}

\begin{array}[c]{rMcMl}
  \diamond{(\prop * \propB)} &\provesIff& \diamond\prop * \diamond\propB \\
  \diamond{(\prop \land \propB)} &\provesIff& \diamond\prop \land \diamond\propB \\
  \diamond{(\prop \lor \propB)} &\provesIff& \diamond\prop \lor \diamond\propB
\end{array}

\begin{array}[c]{rMcMl}
  \diamond{\All x. \prop} &\provesIff& \All x. \diamond{\prop}   \\
  \diamond{\Exists x. \prop} &\provesIff& \Exists x. \diamond{\prop} \\
  \diamond\always{\prop} &\provesIff& \always\diamond{\prop} \\
  \diamond\later\prop &\proves& \later{\prop}
\end{array}
\end{mathpar}
In particular, from \ruleref{ex0-mono} and \ruleref{ex0-idem} we can derive a ``bind''-like elimination rule:
\begin{mathpar}
  \inferH{ex0-elim}
  {\prop \proves \diamond\propB}
  {\diamond\prop \proves \diamond\propB}
\end{mathpar}

This modality is useful because there is a class of propositions which we call \emph{timeless} propositions, for which we have
\[ \timeless{\prop} \eqdef \later\prop \proves \diamond\prop  \]
In other words, when working below the except-0 modality, we can \emph{strip
  away} the later from timeless propositions (using \ruleref{ex0-elim}):
\begin{mathpar}
  \inferH{ex0-timeless-strip}{\timeless{\prop} \and \prop \proves \diamond\propB}
  {\later\prop \proves \diamond\propB}
\end{mathpar}

 In fact, it turns out that we can strip away later from timeless propositions even when working under the later modality:
\begin{mathpar}
  \inferH{later-timeless-strip}{\timeless{\prop} \and \prop \proves \later \propB}
  {\later\prop \proves \later\propB}
\end{mathpar}
This follows from $\later \prop \proves \later\FALSE \lor \prop$, and then by straightforward disjunction elimination.

The following rules identify the class of timeless propositions:
\begin{mathparpagebreakable}
\infer
{\vctx \proves \timeless{\prop} \and \vctx \proves \timeless{\propB}}
{\vctx \proves \timeless{\prop \land \propB}}

\infer
{\vctx \proves \timeless{\prop} \and \vctx \proves \timeless{\propB}}
{\vctx \proves \timeless{\prop \lor \propB}}

\infer
{\vctx \proves \timeless{\prop} \and \vctx \proves \timeless{\propB}}
{\vctx \proves \timeless{\prop * \propB}}

\infer
{\vctx \proves \timeless{\prop}}
{\vctx \proves \timeless{\always\prop}}

\infer
{\vctx \proves \timeless{\propB}}
{\vctx \proves \timeless{\prop \Ra \propB}}

\infer
{\vctx \proves \timeless{\propB}}
{\vctx \proves \timeless{\prop \wand \propB}}

\infer
{\vctx,\var:\type \proves \timeless{\prop}}
{\vctx \proves \timeless{\All\var:\type.\prop}}

\infer
{\vctx,\var:\type \proves \timeless{\prop}}
{\vctx \proves \timeless{\Exists\var:\type.\prop}}

\axiom{\timeless{\TRUE}}

\axiom{\timeless{\FALSE}}

\infer
{\text{$\term$ or $\term'$ is a discrete OFE element}}
{\timeless{\term =_\type \term'}}

\infer
{\text{$\melt$ is a discrete OFE element}}
{\timeless{\ownM\melt}}

\infer
{\text{$\melt$ is an element of a discrete camera}}
{\timeless{\mval(\melt)}}
\end{mathparpagebreakable}

\subsection{Dynamic Composeable Higher-Order Resources}
\label{sec:composeable-resources}

The base logic described in \Sref{sec:base-logic} works over an arbitrary camera $\monoid$ defining the structure of the resources.
It turns out that we can generalize this further and permit picking cameras ``$\iFunc(\Prop)$'' that depend on the structure of propositions themselves.
Of course, $\Prop$ is just the syntactic type of propositions; for this to make sense we have to look at the semantics.

Furthermore, there is a composability problem with the given logic: if we have one proof performed with camera $\monoid_1$, and another proof carried out with a \emph{different} camera $\monoid_2$, then the two proofs are actually carried out in two \emph{entirely separate logics} and hence cannot be combined.

Finally, in many cases just having a single ``instance'' of a camera available for reasoning is not enough.
For example, when reasoning about a dynamically allocated data structure, every time a new instance of that data structure is created, we will want a fresh resource governing the state of this particular instance.
While it would be possible to handle this problem whenever it comes up, it turns out to be useful to provide a general solution.

The purpose of this section is to describe how we solve these issues.

\paragraph{Picking the resources.}
The key ingredient that we will employ on top of the base logic is to give some more fixed structure to the resources.
To instantiate the logic with dynamic higher-order ghost state, the user picks a family of locally contractive bifunctors $(\iFunc_i : \COFEs^\op \times \COFEs \to \CMRAs)_{i \in \mathcal{I}}$.
(This is in contrast to the base logic, where the user picks a single, fixed camera that has a unit.)

From this, we construct the bifunctor defining the overall resources as follows:
\begin{align*}
  \GName \eqdef{}& \nat \\
  \textdom{ResF}(\ofe^\op, \ofe) \eqdef{}& \prod_{i \in \mathcal I} \GName \fpfn \iFunc_i(\ofe^\op, \ofe)
\end{align*}
We will motivate both the use of a product and the finite partial function below.
$\textdom{ResF}(\ofe^\op, \ofe)$ is a camera by lifting the individual cameras pointwise, and it has a unit (using the empty finite partial function).
Furthermore, since the $\iFunc_i$ are locally contractive, so is $\textdom{ResF}$.

Now we can write down the recursive domain equation:
\[ \iPreProp \cong \UPred(\textdom{ResF}(\iPreProp, \iPreProp)) \]
Here, $\iPreProp$ is a COFE defined as the fixed-point of a locally contractive bifunctor, which exists and is unique up to isomorphism by \thmref{thm:america_rutten}, so we obtain some object $\iPreProp$ such that:
\begin{align*}
  \Res &\eqdef \textdom{ResF}(\iPreProp, \iPreProp) \\
  \iProp &\eqdef \UPred(\Res) \\
	\wIso &: \iProp \nfn \iPreProp \\
	\wIso^{-1} &: \iPreProp \nfn \iProp \\
  \wIso(\wIso^{-1}(x)) &\eqdef x \\
  \wIso^{-1}(\wIso(x)) &\eqdef x
\end{align*}
Now we can instantiate the base logic described in \Sref{sec:base-logic} with $\Res$ as the chosen camera:
\[ \Sem{\Prop} \eqdef \UPred(\Res) \]
We obtain that $\Sem{\Prop} = \iProp$.
Effectively, we just defined a way to instantiate the base logic with $\Res$ as the camera of resources, while providing a way for $\Res$ to depend on $\iPreProp$, which is isomorphic to $\Sem\Prop$.

We thus obtain all the rules of \Sref{sec:base-logic}, and furthermore, we can use the maps $\wIso$ and $\wIso^{-1}$ \emph{in the logic} to convert between logical propositions $\Sem\Prop$ and the domain $\iPreProp$ which is used in the construction of $\Res$ -- so from elements of $\iPreProp$, we can construct elements of $\Sem{\textlog M}$, which are the elements that can be owned in our logic.

\paragraph{Proof composability.}
To make our proofs composeable, we \emph{generalize} our proofs over the family of functors.
This is possible because we made $\Res$ a \emph{product} of all the cameras picked by the user, and because we can actually work with that product ``pointwise''.
So instead of picking a \emph{concrete} family, proofs will assume to be given an \emph{arbitrary} family of functors, plus a proof that this family \emph{contains the functors they need}.
Composing two proofs is then merely a matter of conjoining the assumptions they make about the functors.
Since the logic is entirely parametric in the choice of functors, there is no trouble reasoning without full knowledge of the family of functors.

Only when the top-level proof is completed we will ``close'' the proof by picking a concrete family that contains exactly those functors the proof needs.

\paragraph{Dynamic resources.}
Finally, the use of finite partial functions lets us have as many instances of any camera as we could wish for:
Because there can only ever be finitely many instances already allocated, it is always possible to create a fresh instance with any desired (valid) starting state.
This is best demonstrated by giving some proof rules.

So let us first define the notion of ghost ownership that we use in this logic.
Assuming that the family of functors contains the functor $\Sigma_i$ at index $i$, and furthermore assuming that $\monoid_i = \Sigma_i(\iPreProp, \iPreProp)$, given some $\melt \in \monoid_i$ we define:
\[ \ownGhost\gname{\melt:\monoid_i} \eqdef \ownM{(\ldots, \emptyset, i:\mapsingleton \gname \melt, \emptyset, \ldots)} \]
This is ownership of the pair (element of the product over all the functors) that has the empty finite partial function in all components \emph{except for} the component corresponding to index $i$, where we own the element $\melt$ at index $\gname$ in the finite partial function.

We can show the following properties for this form of ownership:
\begin{mathparpagebreakable}
  \inferH{res-alloc}{\text{$G$ infinite} \and \melt \in \mval_{M_i}}
  {  \TRUE \proves \upd \Exists\gname\in G. \ownGhost\gname{\melt : M_i}
  }
  \and
  \inferH{res-update}
    {\melt \mupd_{M_i} B}
    {\ownGhost\gname{\melt : M_i} \proves \upd \Exists \meltB\in B. \ownGhost\gname{\meltB : M_i}}

  \inferH{res-empty}
  {\text{$\munit$ is a unit of $M_i$}}
  {\TRUE \proves \upd \ownGhost\gname\munit}

  \axiomH{res-op}
    {\ownGhost\gname{\melt : M_i} * \ownGhost\gname{\meltB : M_i} \provesIff \ownGhost\gname{\melt\mtimes\meltB : M_i}}

  \axiomH{res-valid}
    {\ownGhost\gname{\melt : M_i} \Ra \mval_{M_i}(\melt)}

  \inferH{res-timeless}
    {\text{$\melt$ is a discrete OFE element}}
    {\timeless{\ownGhost\gname{\melt : M_i}}}
\end{mathparpagebreakable}

Below, we will always work within (an instance of) the logic as described here.
Whenever a camera is used in a proof, we implicitly assume it to be available in the global family of functors.
We will typically leave the $M_i$ implicit when asserting ghost ownership, as the type of $\melt$ will be clear from the context.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "iris"
%%% End:


% !TEX root = cs1textbook.tex

\chapter{Introduction to Creating Classes}
\label{chapter:classes}

\minitoc

%\section{The \texttt{Point} Class}

We will begin our geometry case study by creating a data type to represent the smallest unit of geometry -- the point -- and build up in complexity from there.  \href{https://en.wiktionary.org/wiki/point}{Wiktionary defines a geometrical point} as ``A zero-dimensional mathematical object representing a location in one or more dimensions; something considered to have position but no magnitude or direction''.  We will define our \mintinline{java}{Point} class to represent an object on a two-dimensional Cartesian plane, since this is a good analogy for a computer screen.  A detail that will become important later is that several computer graphics paradigms, including the one we'll be working under, fix the origin of the screen's Cartesian plane at the top left of the screen, so that $x$ values increase from right to left and $y$ values increase from the top of the screen to the bottom; this results in all $x$ and $y$ values being non-negative.  A monitor with 1024$\times$768 resolution, therefore, would find the point $(1023,767)$ at the bottom right of the screen, as in Figure \ref{fig:monitor}.

\begin{figure}[ht]
    \color{nccblue}
    \begin{center}
        \begin{tikzpicture}
            \draw [thick,nccblue,fill=nccorange] (0,0) -- (5,0) -- (5,4) -- (0,4) -- (0,0) {};
            \draw [thin,nccblue] (1,0) -- (0.5,-0.5) -- (4.5,-0.5) -- (4,0);

            \draw (5,0) node [circle,fill=nccblue] {};
            \draw (0,4) node [circle,fill=nccblue] {};

            \draw (-0.5,3.65) node {$(0,0)$};
            \draw (6,0.3) node {$(1023,767)$};
        \end{tikzpicture}
    \end{center}
    \caption{Cartesian Coordinates of a Computer Monitor with 1024$\times$768 Resolution}
    \label{fig:monitor}
\end{figure}

\section{Class Header}

Every class definition begins with a class \textit{header}.  The header defines the class' visibility and its name.  This is how a simple class header looks:

\begin{javaformat}{Class Header}
\begin{minted}{text}
<visibility-modifier> class <identifier>
\end{minted}
\end{javaformat}

The visibility modifier for classes will be \mintinline{java}{public} throughout this textbook.

In Section \ref{sec:using-memory} we learned some rules that Java imposes on us regarding what we can choose as an identifier:
\bi
\item An identifier \textbf{must} start with a letter.
\item An identifier \textbf{must} only contain letters, digits, and the underscore character (\lstinline{'_'}).
\item An identifier \textbf{must not} be the same as a \textit{reserved word} in Java (like \lstinline{class} or \lstinline{static}).
\ei

Section \ref{sec:using-memory} also lists some guidelines regarding naming a variable.  The guidelines for naming a class are a bit different, however:
\bi
\item The name of a class should \textit{start with a capital letter}.
\item Classes define \textit{things}, so the name should be a \textit{noun}.
\ei

Finally, each class should have a JavaDoc comment above the header that explains what this class provides to the programmer.

For the \texttt{Point} class, the header will look like the first line of this code:

\begin{minted}{java}
/**
 * Provides a two-dimensional point on a Cartesian plane.
 * <p>
 * All x and y values are non-negative.
 */
public class Point {  // This line is the header
    /* The body of the class will go here */
}
\end{minted}

Notice that I have put the opening curly brace of the body at the end of the header.  Some programmers choose instead to place an opening curly brace on the next line.

\section{Instance Variables}

After the class header, the rest of the class' definition -- the \textit{body} of the class -- appear inside curly braces.  The first thing that most programmers put at the top of the class' body is a list of the class' \textit{properties}.  This list consists of variable declarations that describe the data an object of this class will contain.

One instance of the \texttt{Point} class should contain enough information to complete an ordered pair in $(x,y)$ format.  We'll assume some knowledge about computer monitors -- specifically, that there are no partial pixels -- and specify that both the $x$ value and the $y$ value must be integers.

\begin{minted}{java}
public class Point {
    int x;  // x component of Cartesian ordered pair
    int y;  // y component of Cartesian ordered pair
}
\end{minted}

Variables like these, declared in class scope, are called \textbf{instance variables}\index{Variable!Instance variable}, because every instance of the \texttt{Point} class that we create will store values in these variables.

In a different class, we will create a \texttt{Point} object and place some data in these instance variables:

\begin{minted}{java}
public class TestPoint {
    public static void main( String args[] ) {
        Point p1 = new Point();  // p1 represents a point on the screen
        p1.x = 10;   // p1's x coordinate is now 10
        p1.y = 15;   // p1's y coordinate is now 15

        System.out.println( "The point is at (" + p1.x + "x" + p1.y + ")." );
    }
}
\end{minted}

So far, it seems like we've solved our problem.  The output from this program will be what you expect: \texttt{The point is at (10x15).}  There exists, however, a new problem, illustrated here:

\begin{minted}{java}
public class TestPoint {
    public static void main( String args[] ) {
        Point p1 = new Point();  // p1 should represent a point on the screen
        p1.x = -35;   // But can p1's x coordinate be negative?
        p1.y = -17;   // Or its y coordinate?
    }
}
\end{minted}

What happens here might surprise you.  We human readers of the English language know that a point on the screen isn't supposed to have negative values, but the Java compiler doesn't understand English.  Java \textit{will not} interpret this as an error, because to Java it isn't an error.  It's a \textit{semantic error}, because we know it's wrong, but it doesn't violate the rules of the Java language, so it will run and be wrong.

\begin{defn}{Semantic Error}
    A \textbf{semantic error}\index{Error!Semantic error} is an error in logic or meaning.  Typically, we notice semantic errors in code that has compiled and is running, because only a program that's running can generate incorrect output.
\end{defn}

\section{Visibility}

Thus, we learn the most important responsibility of Java programmers: We must protect the data in our classes, because relying on others to do the right thing will not work.

\begin{tip}{Data Protection}
    When designing a Java class, you must ensure that the data in its instance variables will always be valid.  If you don't work to ensure this, then there will come a point where the data will become invalid.
\end{tip}

The path to ensuring data validity starts with changing the \textit{visibility} of the instance variables.  We will mark \texttt{x} and \texttt{y} as \mintinline{java}{private}, like this:

\begin{minted}{java}
public class Point {
    private int x;  // x component of Cartesian ordered pair
    private int y;  // y component of Cartesian ordered pair
}
\end{minted}

The value of a \mintinline{java}{private} variable can not be retrieved or changed by any code outside the class.  This protects our data from being made negative by the \texttt{TestPoint} class:

\begin{minted}{java}
public class TestPoint {
    public static void main( String args[] ) {
        Point p1 = new Point();  // p1 should represent a point on the screen
        p1.x = -35;   // This is now a syntax error!
        p1.y = -17;   // And so is this!  Success!
    }
}
\end{minted}

But we have now done too good a job of protecting our data, because we can't set the variables with good data.  We can't even print their values!

\begin{minted}{java}
public class TestPoint {
    public static void main( String args[] ) {
        Point p1 = new Point();  // p1 represents a point on the screen
        p1.x = 10;   // This seemingly-valid request is now a syntax error
        p1.y = 15;   // So is this

        // And we can't even do this any more:
        System.out.println( "The point is at (" + p1.x + "x" + p1.y + ")." );
    }
}
\end{minted}

The instance variables still need to be \mintinline{java}{private}; the way to fix this problem is to give limited \mintinline{java}{public} access to viewing and changing the instance variables' values.  We will explore how to grant this access in the next chapter.


\subsection{Blockchain}

\subsubsection{Definition}

This is how a block is defined in this work:

\agda{Blockchain}{block}

\emph{nextTXTree} assures that the second transaction tree is from the first transaction tree.
\emph{firstTreesInBlock} guarantees that the last transaction in the first transaction tree
is the first in the block.
\emph{coinBaseTree} assures that the last transaction in the second transaction tree is
a coinbase transaction.

Blockchain is a chain of valid blocks.
Every new block must be a continuation of the previous one.
Here is the definition of the blockchain:

\agda{Blockchain}{blockchain}

In the first case, blockchain just has one block, called \emph{fstBlock}.
In the second case, the blockchain is an addition of a valid block from a previous blockchain.

\subsubsection{Creation}

In this section, there will be explanation of how blockchain is created from blocks
and how blocks are created from transaction trees.
To create a blockchain, it is first needed to create the last block.
From the last block, it is possible to create all the chain.

\plabel{blockblockchain}
\agda{Blockchain}{blockblockchain}

In this \hyperref[blockblockchain]{function}, if the first transaction tree of the block
is a genesis tree,
it will return a blockchain of just one block.
If it is a regular tree, it tries to find the first transaction tree of this block.
Using a recursive definition of block to blockchain,
it is possible to generate all the rest of this blockchain from this block.

It is not always possible to generate a block from the transaction tree.
It is because the last transaction of a transaction tree must be a coinbase transaction.
Here, the function that returns a decidable if it is possible to generate a block from
the transaction tree.

\agda{Blockchain}{treeblock}

The definition of the raw block gets just the coinbase transaction tree as an explicit type.
The other transaction tree can be founded opening the record.

\agda{Blockchain}{rawblock}

The code of the definition of what is a coinbase tree:

\agda{Blockchain}{coinbasetree}

The definition of a coinbase tree is the one that the last transaction is a coinbase.

The code verifies if the last transaction tree is a coinbase tree:

\agda{Blockchain}{iscoinbase}

If it is, it returns that it is possible to create a block from that with the block definition.
If it is not, it returns that it is impossible to create a block from this transaction tree.

But to create a block from this coinbase transaction tree, it is necessary to find the first tree
of the block.

\agda{Blockchain}{fsttree}

The definition of \emph{fstTree} is that it has a tree that is before this tree in the type.
And this tree before is the first in the block.

\agda{Blockchain}{firsttreeinblock}

The decidable version of this \emph{Set}:

\agda{Blockchain}{isfirsttreeinblock}

In this case, it pattern match trees that are genesis tree or if the last transaction was a coinbase
transaction.

\agda{Blockchain}{firsttree}

To find the first tree in the block, there are two cases.
The first case is that if the tree is a genesis tree, so the result is itself.
The second case is if it a regular tree, so it still has to divide it in many cases.
If this tree is already the first tree in the block, it will return itself.
If this tree is not, it has to verify if the block number of the tree is the same as this tree.
If the block number is equal, it can recursively find the first tree.
If it is not, it has to provide proof that this tree must be
the first and the blocks numbers are different.

To define what it means of one tree is next to another:

\agda{Blockchain}{nexttree}

Both trees (\emph{txTree1} and \emph{txTree2}) can be equal or different.
If both trees are the same, they are next to each other.
If there is a proof that both trees are next to each other and
if there is one tree that was generated from the last one,
so the first tree is next to the last one.


\section{Hierarchization on Dimensionally Adaptive Sparse Grids}
\label{sec:43dimAdaptive}

\minitoc{77mm}{7}

\noindent
Dimensionally adaptive sparse grids,
which are sums of different hierarchical subspaces
as described in \cref{sec:232dimensionallyAdaptiveSG},
have the advantage over general spatially adaptive sparse grids
that algorithms can be formulated and applied more easily.
In this section, we describe two methods:
first, the well-known combination technique, which was
already mentioned in \cref{sec:232dimensionallyAdaptiveSG},
and second, a new algorithm based on residual interpolation.

\subsection{The Combination Technique and Its Combinatorial Proof}
\label{sec:431combiTechniqueProof}

The combination technique was one of the first methods that
were developed by Griebel et al. in \cite{Griebel92Combination}
(for two and three dimensions)
after the term ``sparse grids'' was coined in 1991 \cite{Zenger91Sparse}.
However, the combination technique predates the development of sparse grids
by decades, as it was already mentioned by Smolyak in 1963
\multicite{Smolyak63Quadrature,Hegland07Combination}.
Delvos developed and proved the standard combination formula in the
framework of Boolean interpolation operators in 1982
\multicite{Delvos82Dvariate,Delvos89Boolean}.

\paragraph{Formal description and outline of a combinatorial proof}

In the following, we give a formal description of the
sparse grid combination technique, and we outline a new combinatorial proof
of its correctness.
While we discuss a high-level explanation of the proofs in this section,
the proofs themselves can be found in \cref{sec:a131proofCombiTechnique},
since most of them are rather technical.
For simplicity,
we formulate the combination technique and its proof for regular
sparse grids (see \cref{sec:231regularSG}).
However, the main ideas of the chain of proofs are also applicable
to dimensionally adaptive sparse grids
(see \cref{sec:232dimensionallyAdaptiveSG}).

\begin{theorem}[sparse grid combination technique]
  \label{thm:combiTechnique}
  Let $\liset \ceq \{(\*l, \*i) \mid
  \normone{\*l} \le n,\; \*i \in \hiset{\*l}\}$
  correspond to the regular sparse grid
  $\regsgset{n}{d}$ and let $(\fcnval{\*l,\*i})_{(\*l,\*i) \in \liset}$
  be given function values on $\regsgset{n}{d}$.
  If we define
  \begin{itemize}
    \item
    the combined sparse grid interpolant $\regsgintp[\ct]{n}{d}$ via
    \eqref{eq:combiTechnique}, i.e.,
    \begin{equation}
      \regsgintp[\ct]{n}{d}
      = \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \sum_{\normone{\*l'} = n-q}
      \fgintp{\*l'},
    \end{equation}
    where $\fgintp{\*l'} \in \ns{\*l'}$ is the full grid interpolant
    of $\objfun$ with level $\*l'$, and

    \item
    the hierarchical sparse grid interpolant $\regsgintp{n}{d}$
    via \eqref{eq:hierarchizationProblem} and
    \eqref{eq:hierarchizationInterpolant}
  \end{itemize}
  and if we assume that the hierarchical splitting equation
  \eqref{eq:hierSplittingMV} holds,
  then the combined and the hierarchical sparse grid interpolants coincide:
  \begin{equation}
    \regsgintp[\ct]{n}{d}
    = \regsgintp{n}{d}.
  \end{equation}
\end{theorem}

\begin{proof}[Proof (sketch)]
  Let $\gp{\*l,\*i} \in \regsgset{n}{d}$ be an arbitrary
  point of the regular sparse grid.
  First, we split the inner sum of $\regsgintp[\ct]{n}{d}(\gp{\*l,\*i})$
  into levels $\*l'$ whose full grid sets $\fgset{\*l'}$
  contain $\gp{\*l,\*i}$ and levels whose full grid sets
  do not contain $\gp{\*l,\*i}$:
  \begin{equation}
    \regsgintp[\ct]{n}{d}(\gp{\*l,\*i})
    = \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \cdot \paren*{
      \sum_{\substack{\normone{\*l'} = n - q\\\fgset{\*l'} \ni \gp{\*l,\*i}}}
      \fgintp{\*l'}(\gp{\*l,\*i}) +
      \sum_{\substack{\normone{\*l'} = n - q\\\fgset{\*l'} \notni \gp{\*l,\*i}}}
      \fgintp{\*l'}(\gp{\*l,\*i})
    }.
  \end{equation}
  The summands $\fgintp{\*l'}(\gp{\*l,\*i})$ of the first inner sum
  each equal $\fcnval{\*l,\*i}$ due to the full grid interpolation
  property \eqref{eq:interpFullGridMV}.
  Therefore, the first inner sum is equal to the product of
  $\fcnval{\*l,\*i}$ with the number of summands:
  \begin{equation}
    \label{eq:combiTechniqueSplitSum}
    \begin{split}
      \regsgintp[\ct]{n}{d}(\gp{\*l,\*i})
      &= \fcnval{\*l,\*i} \cdot \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \cdot
      \setsize{
        \{\*l' \mid \normone{\*l'} = n - q,\; \fgset{\*l'} \ni \gp{\*l,\*i}\}
      }\\
      &\qquad{} {} + \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \cdot
      \sum_{\substack{\normone{\*l'} = n - q\\\fgset{\*l'} \notni \gp{\*l,\*i}}}
      \fgintp{\*l'}(\gp{\*l,\*i}).
    \end{split}
  \end{equation}
  After this sketch of proof,
  we will prove that the first of the two summands in
  \cref{eq:combiTechniqueSplitSum}
  equals one (see \cref{prop:combiTechniqueOne})
  and that the second of the two summands
  equals zero (see \cref{prop:combiTechniqueZero}).
  Consequently, we infer
  \begin{equation}
    \regsgintp[\ct]{n}{d}(\gp{\*l,\*i})
    = \fcnval{\*l,\*i},
  \end{equation}
  i.e., $\regsgintp[\ct]{n}{d}$ interpolates $\objfun$ at $\regsgset{n}{d}$.
  Note that $\regsgintp[\ct]{n}{d}$ is contained in $\regsgspace{n}{d}$,
  if the hierarchical splitting equation \eqref{eq:hierSplittingMV} holds,
  as
  \begin{equation}
    \fgintp{\*l'} \in
    \ns{\*l'}
    = \bigoplus_{\*l''=\*0}^{\*l'} \hs{\*l''}
    \subset \regsgspace{n}{d},\quad
    \normone{\*l'} \le n,
  \end{equation}
  due to $\normone{\*l''} \le \normone{\*l'} \le n$
  for $\*l'' \le \*l'$, i.e.,
  $\hs{\*l''} \subset \regsgspace{n}{d}$ for
  $\*l'' = \*0, \dotsc, \*l'$.%
  \footnote{%
    This argumentation can be straightforwardly adapted
    for general dimensionally adaptive sparse grids
    with downward closed level sets as mentioned in
    \cref{sec:232dimensionallyAdaptiveSG}.%
  }
  As both $\regsgintp[\ct]{n}{d}$ and $\regsgintp{n}{d}$
  are contained in $\regsgspace{n}{d}$ and
  interpolate $\objfun$ on $\regsgset{n}{d}$, they coincide
  due to the uniqueness of sparse grid interpolation
  (linear independence of the hierarchical basis).
\end{proof}

\paragraph{Inclusion-exclusion principle}

It remains to prove that the first sum in \eqref{eq:combiTechniqueSplitSum}
is indeed one and that the second sum vanishes.
The first statement is a direct consequence of the
\term{inclusion-exclusion principle} \cite{Hegland07Combination}.
In its simplest form, the idea of the principle is that the cardinality
of the union of two finite subsets $A$ and $B$ of some set is given by
\begin{equation}
  \setsize{A \cup B}
  = \setsize{A} + \setsize{B} - \setsize{A \cap B},
\end{equation}
i.e., we first count (include)
the elements in $A$ and then in $B$,
but as we have counted the elements of $A \cap B$ twice,
we have to subtract (exclude) its cardinality again.

The setting is similar for the combination technique.
If we add all grids in \cref{fig:combinationTechnique}
on the green diagonal, then every point whose index is not odd
will be counted multiple times.
By subtracting the number of occurrences of the points on the
red diagonal,
the result of the ``weighted counting'' is exactly one for every point.
The following proposition, whose proof is of purely combinatorial nature,
generalizes this argument to higher dimensions:

\begin{restatable}[inclusion-exclusion principle]{%
  proposition%
}{%
  propCombiTechniqueOne%
}
  \label{prop:combiTechniqueOne}
  For every $\gp{\*l,\*i} \in \regsgset{n}{d}$, we have
  \begin{equation}
    \label{eq:combiTechniqueOne}
    \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \cdot
    \setsize{
      \{\*l' \mid \normone{\*l'} = n - q,\; \fgset{\*l'} \ni \gp{\*l,\*i}\}
    }
    = 1.
  \end{equation}
\end{restatable}

\begin{proof}
  See \cref{sec:a131proofCombiTechnique}.
\end{proof}

\paragraph{Canceling out function values}

The second statement about the vanishing second sum in
\eqref{eq:combiTechniqueSplitSum} is much harder to prove.
It says that at every grid point $\gp{\*l,\*i}$,
the contributions $\fgintp{\*l'}$ of levels $\*l'$
that do not contain that point cancel out,
which may seem quite surprising.
The key observation is as follows:
The values of $\fgintp{\*l'}, \fgintp{\*l''}$ for two levels
$\*l', \*l''$ are the same at $\gp{\*l,\*i}$,
if all non-equal entries $l'_t, l''_t$ of the levels are
each greater or equal to $l_t$.

For a higher-level explanation,
note that the statement $l'_t \ge l_t$ is equivalent to
$\fgset{l'_t} \ni \gp{l_t,i_t}$.
Both $\fgintp{\*l'}, \fgintp{\*l''}$ interpolate at
$\gp{\*l,\*i}$ when projected onto the $t$-th dimension,
so their contribution to $\fgintp{\*l'}(\gp{\*l,\*i})$ and
$\fgintp{\*l''}(\gp{\*l,\*i})$ must be the same.
Although there may be dimensions $t$ for which
$\fgset{l'_t} \notni \gp{l_t,i_t}$,
these dimensions do not matter if $l'_t = l''_t$,
as the univariate restrictions of $\fgintp{\*l'}, \fgintp{\*l''}$
interpolate the same data, and they are evaluated at the same point
$\gp{l_t,i_t}$.

One can formalize these considerations by defining an
equivalence relation on the set of levels such that the values of
$\fgintp{\*l'}$ at $\gp{\*l,\*i}$ are constant
on the equivalence classes.

\begin{definition}[%
  equivalence relation for the proof of the combination technique%
]
  \label{def:combiTechniqueEquivalenceRelation}
  Let $\gp{\*l,\*i} \in \regsgset{n}{d}$ be fixed and
  \begin{equation}
    \label{eq:combiTechniqueSpecialLevelSet}
    L
    \ceq \{\*l' \mid \ex{q=0,\dotsc,d-1}{
      \normone{\*l'} = n - q,\; \fgset{\*l'} \notni \gp{\*l,\*i}
    }\}
  \end{equation}
  be the set of levels that do not contain $\gp{\*l,\*i}$.
  We define a relation $\eq$ on $L$ as follows:
  For $\*l', \*l'' \in L$, we set $\*l' \eq \*l''$ if and only if
  \begin{equation}
    \falarge{t \notin T_{\*l',\*l''}}{\min\{l'_t, l''_t\} \ge l_t},\quad
    T_{\*l',\*l''}
    \ceq \{t \mid l'_t = l''_t < l_t\}.
  \end{equation}
\end{definition}

\begin{restatable}[identical values in equivalence classes]{%
  shortlemma%
}{%
  lemmaCombiTechniqueIdenticalValues%
}
  \label{lemma:combiTechniqueIdenticalValues}
  Let $\*l', \*l'' \in L$ with $\*l' \eq \*l''$.
  Then, $\fgintp{\*l'}(\gp{\*l,\*i})
  = \fgintp{\*l''}(\gp{\*l,\*i})$.
\end{restatable}

\begin{proof}
  See \cref{sec:a131proofCombiTechnique}.
\end{proof}

By exploiting the tensor product structure of the basis functions,
the proof shows an even stronger version, which is shown
in \cref{fig:combiTechniqueProof}:
The components $\fgintp{\*l'}$ and $\fgintp{\*l''}$ are equal
on the $m$-dimensional affine subspace through $\gp{\*l,\*i}$
parallel to the $m$ coordinates in $T_{\*l',\*l''}$
(where $m \ceq \setsize{T_{\*l',\*l''}}$).
The lemma allows us to group summands in the second sum of
\eqref{eq:combiTechniqueSplitSum} by function values.
Hence, it remains to count the number of levels in
each equivalence class of $\eq$.
Therefore, we need a characterization of the equivalence classes:

\begin{SCfigure}
  \includegraphics{combiTechniqueProof_1}%
  \caption[%
    Canceling out function values in the proof of the combination technique%
  ]{%
    Nodal subspaces $\ns{\*l}$ contributing to the combination
    technique solution for the two-dimensional regular sparse grid
    $\regsgspace{n}{d}$ of level $n = 3$ \emph{(bottom right).}
    After picking a point $\gp{\*l,\*i} \in \regsgset{n}{d}$
    (\emph{cross,} here $\*l = (2, 1)$, $\*i = (1, 1)$),
    the set $L$ of levels whose grids do not contain $\gp{\*l,\*i}$
    \emph{(colored subspaces)}
    decompose into three disjoint equivalence classes
    \emph{(colors)} given by the relation $\eq$.
    In every equivalence class $L_0 \in \eqclasses{L}{\eq}$,
    the interpolants $\fgintp{\*l'}$ ($\*l' \in L_0$)
    equal on an affine subspace
    \emph{(dark lines),} which contains $\gp{\*l,\*i}$.
    Due to the combination coefficients,
    the contribution to the combined solution
    vanishes per equivalence class.%
  }%
  \label{fig:combiTechniqueProof}%
\end{SCfigure}

\begin{restatable}[characterization of equivalence classes]{%
  lemma%
}{%
  lemmaCombiTechniqueCharacterization%
}
  \label{lemma:combiTechniqueCharacterization}
  Let $L_0 \in \eqclasses{L}{\eq}$ be an equivalence class of $\eq$.
  If we define
  \begin{equation}
    T_{L_0}
    \ceq \{t \mid \exfa{l^\ast_t < l_t}{\*l' \in L_0}{l'_t = l^\ast_t}\}
  \end{equation}
  as the set of dimensions $t$ in which all levels in $L_0$
  have the same entry $l^\ast_t < l_t$, then
  \begin{equation}
    L_0
    = \{\*l' \in L \mid
    \fa{t \in T_{L_0}}{l'_t = l^\ast_t},\;
    \fa{t \notin T_{L_0}}{l'_t \ge l_t}\}.
  \end{equation}
\end{restatable}

\begin{proof}
  See \cref{sec:a131proofCombiTechnique}.
\end{proof}

The lemma states that every equivalence class $L_0$ is exactly the
set of the levels whose entries are equal
and smaller than $l_t$ in some dimensions
(which are contained in $T_{L_0}$)
and whose entries are greater or equal than $l_t$ in all other dimensions.
While this statement may seem intuitively correct,
the proof is rather technical.
Finally, we are now able to show that the second sum in
\eqref{eq:combiTechniqueSplitSum} vanishes:

\begin{restatable}[function value cancellation]{%
  proposition%
}{%
  propCombiTechniqueZero%
}
  \label{prop:combiTechniqueZero}
  For every $\gp{\*l,\*i} \in \regsgset{n}{d}$, we have
  \begin{equation}
    \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \cdot
    \sum_{\substack{\normone{\*l'} = n - q\\\fgset{\*l'} \notni \gp{\*l,\*i}}}
    \fgintp{\*l'}(\gp{\*l,\*i})
    = 0.
  \end{equation}
\end{restatable}

\begin{proof}
  See \cref{sec:a131proofCombiTechnique}.
\end{proof}

The proof essentially first counts the number of possible levels in
an equivalence class and then applies known combinatorial identities
to prove that the sum must vanish.
This proves \thmref{thm:combiTechnique}.

\subsection{Hierarchization with the Combination Technique}
\label{sec:432hierarchizationCombiTechnique}

It is straightforward to hierarchize function values
$\fcnval{\*l,\*i}$ on dimensionally adaptive sparse grids
with the combination technique.
The resulting hierarchization algorithm is given as \cref{alg:combiTechnique}.
In \cref{line:algCombiTechnique1},
the hierarchical surpluses corresponding to the full grid
interpolant $\fgintp{\*l'} \in \ns{\*l'}$ have to be computed
(see \eqref{eq:interpFullGridMV}).
As shown in \cref{sec:42fullGrids}, we can easily calculate these
surpluses with the unidirectional principle in
\cref{alg:unidirectionalPrinciple}.
The surpluses are then combined with the same combination formula
as in \thmref{thm:combiTechnique}.
Note that it is imperative to employ the hierarchical basis functions
$\basis{\*l,\*i}$ with $\*l = \*0, \dotsc, \*l'$ and $\*i \in I_{\*l}$
and not the nodal basis,
i.e., $\basis{\*l',\*i'}$ with $\*i' = \*0, \dotsc, \*2^{\*l'}$.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{$\vlinout = \texttt{combinationTechnique}$}{%
      $\vlinin$, $n$, $d$%
    }
      \For{$q = 0, \dotsc, d - 1$}
        \For{$\*l' \in \natz^d$ with $\normone{\*l'} = n - q$}
          \State{%
            Let $(\surplus[(\*l')]{\*l,\*i})_{
              \*l = \*0, \dotsc, \*l'\!,\, \*i \in \hiset{\*l}
            }$ be such that
            $\sum_{\*l=\*0}^{\*l'} \sum_{\*i \in \hiset{\*l}}
            \surplus[(\*l')]{\*l,\*i} \basis{\*l,\*i} \equiv
            \fgintp{\*l'}$%
          }
          \label{line:algCombiTechnique1}
          \State{%
            $\surplus[(\*l')]{\*l,\*i} \gets 0$
            for all $(\*l,\*i) \in \liset$
            with $\lnot(\*l \le \*l')$%
          }
          \Comment{extend surpluses}%
          \label{line:algCombiTechnique2}
        \EndFor{}
      \EndFor{}
      \State{%
        $\linout{\*l,\*i}
        = \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q}
        \sum_{\normone{\*l'} = n-q} \surplus[(\*l')]{\*l,\*i}$
        for all $(\*l, \*i) \in \liset$%
      }
      \Comment{combine surpluses}%
    \EndFunction{}
  \end{algorithmic}
  \caption[%
    Hierarchization with the combination technique%
  ]{%
    Application of the hierarchization operator $\linop = \intpmatinv$
    with the combination technique.
    For simplicity,
    the algorithm is described for regular sparse grids,
    but it can be generalized to arbitrary dimensionally adaptive sparse grids.
    Inputs are
    the vector $\vlinin = (\linin{\*l,\*i})_{(\*l,\*i) \in \liset}$
    of input data (function values $\fcnval{\*l,\*i}$ at the grid points),
    the level $n$, and the dimensionality $d$ of the regular sparse grid,
    where $\liset$ is the set of all feasible level-index pairs $(\*l,\*i)$,
    i.e., $\normone{\*l} \le n$, $\*i \in \hiset{\*l}$.
    The output is the vector
    $\vlinout = (\linout{\*l,\*i})_{(\*l,\*i) \in \liset}$
    of output data (hierarchical surpluses $\surplus{\*l,\*i}$).%
  }%
  \label{alg:combiTechnique}%
\end{algorithm}

\paragraph{Correctness}

Of course, the proof of the correctness of \cref{alg:combiTechnique}
relies on the correctness of the combination technique
(see \cref{thm:combiTechnique}).
If determining the combination coefficients correctly
\cite{Nobile16Adaptive}, the algorithm can even be applied to
all dimensionally adaptive sparse grids.
The proof of the following proposition can be generalized accordingly.

\begin{proposition}[correctness of combination technique]
  \label{prop:correctnessAlgCombiTechnique}
  \Cref{alg:combiTechnique}
  is correct for hierarchization on regular sparse grids.
\end{proposition}

\begin{proof}
  According to \cref{line:algCombiTechnique1} of \cref{alg:combiTechnique},
  the full grid interpolants $\fgintp{\*l'}$ can be written as
  \begin{equation}
    \fgintp{\*l'}
    = \sum_{\normone{\*l} \le n} \sum_{\*i \in \hiset{\*l}}
    \surplus[(\*l')]{\*l,\*i} \basis{\*l,\*i}
  \end{equation}
  where the surpluses have been extended
  from $\*l = \*0, \dotsc, \*l'$ to all $\*l$ with $\normone{\*l} \le n$
  by zero in \cref{line:algCombiTechnique2}.
  \Cref{thm:combiTechnique} now allows to write the hierarchical
  interpolant $\regsgintp{n}{d}$ in terms of the full grid components:
  \begin{subequations}
    \begin{align}
      \regsgintp{n}{d}
      = \regsgintp[\ct]{n}{d}
      &= \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \sum_{\normone{\*l'} = n-q}
      \fgintp{\*l'}\\
      &= \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \sum_{\normone{\*l'} = n-q}
      \sum_{\normone{\*l} \le n} \sum_{\*i \in \hiset{\*l}}
      \surplus[(\*l')]{\*l,\*i} \basis{\*l,\*i}.\\
      \label{eq:propCorrectnessAlgCombiTechnique1}
      &= \sum_{\normone{\*l} \le n} \sum_{\*i \in \hiset{\*l}}
      \underbrace{
        \paren*{
          \sum_{q=0}^{d-1} (-1)^q \binom{d-1}{q} \sum_{\normone{\*l'} = n-q}
          \surplus[(\*l')]{\*l,\*i}
        }
      }_{= \linout{\*l,\*i}}
      \basis{\*l,\*i},
    \end{align}
  \end{subequations}
  where $\linout{\*l,\*i}$ is the $(\*l,\*i)$-th entry of the output vector
  of \cref{alg:combiTechnique}.
  Note that the hierarchical interpolant $\regsgintp{n}{d}$
  can be written as
  $\regsgintp{n}{d} = \sum_{\normone{\*l} \le n} \sum_{\*i \in \hiset{\*l}}
  \surplus{\*l,\*i} \basis{\*l,\*i}$
  (see \eqref{eq:regularSGInterpolant}),
  where the surpluses $\surplus{\*l,\*i}$ are unique due to the
  linear independence of the hierarchical basis.
  As \eqref{eq:propCorrectnessAlgCombiTechnique1}
  equals $\regsgintp{n}{d}$ and has the same form,
  the coefficients $\linout{\*l,\*i}$
  must coincide with the surpluses $\surplus{\*l,\*i}$.
\end{proof}

\subsection{Hierarchization with Residual Interpolation}
\label{sec:433residualInterpolation}

Another method to hierarchize function values on
dimensionally adaptive sparse grids is the
\term{method of residual interpolation.}
The advantage over the combination technique is that
it only needs to operate on so-called \term{active nodal spaces.}
In contrast, the combination technique needs to perform computations
on additional non-active nodal subspaces
(for the regular sparse grid case:
summands with $q \ge 1$ in \eqref{eq:combiTechnique}).

\paragraph{Active nodal spaces}

\Cref{alg:residualInterpolation} describes the procedure of
the method of residual interpolation,
given the function values $\vlinin$ corresponding to the grid points
and the levels $L$ contained in the sparse grid
(see \eqref{eq:dimensionallyAdaptiveSG}).
The list $\*l^{(1)}, \dotsc, \*l^{(m)}$ of active nodal spaces
in \cref{line:algResidualInterpolation1} is determined by the condition
\begin{equation}
  \label{eq:activeNodalSpaces}
  \bigcup_{j=1}^m \{\*l \in \natz^d \mid \*l \le \*l^{(j)}\} = L,\quad
  \falarge{j_1 \not= j_2}{\lnot(\*l^{(j_1)} \le \*l^{(j_2)})}.
\end{equation}
This means that the corresponding sparse grid $\sgset$
is the (non-disjoint) union of the full grid sets $\fgset{\*l^{(j)}}$
($j = 1, \dotsc, m$)
and no full grid set is contained in another, i.e.,
no full grid set can be omitted without
removing points from the union $\sgset$.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{$\vlinout = \texttt{residualInterpolation}$}{%
      $\vlinin$, $\levelset$%
    }
      \State{%
        $r^{(0)}(\gp{\*l,\*i}) \gets \fcnval{\*l,\*i}$
        for all $(\*l,\*i) \in \liset$%
      }
      \State{%
        Compute list $\*l^{(1)}, \dotsc, \*l^{(m)}$
        of active nodal spaces from $L$ (see \eqref{eq:activeNodalSpaces})%
      }
      \label{line:algResidualInterpolation1}
      \State{%
        Sort $\*l^{(1)}, \dotsc, \*l^{(m)}$ by decreasing level sum%
      }
      \For{$j = 1, \dotsc, m$}
        \State{%
          Let $r_{\*l^{(j)}}^{(j-1)} \in \ns{\*l^{(j)}}$ be the
          interpolant of $r^{(j-1)}$ on $\fgset{\*l^{(j)}}$%
        }
        \label{line:algResidualInterpolation3}
        \State{%
          Let $(\surplus[(j)]{\*l,\*i})_{(\*l,\*i) \in \liset}$ be such that
          $\sum_{\*l=\*0}^{\*l^{(j)}} \sum_{\*i \in \hiset{\*l}}
          \surplus[(j)]{\*l,\*i} \basis{\*l,\*i}
          \equiv r_{\*l^{(j)}}^{(j-1)}$%
        }
        \Comment{interpolation}%
        \label{line:algResidualInterpolation2}
        \State{%
          $r^{(j)}(\gp{\*l,\*i}) \gets
          r^{(j-1)}(\gp{\*l,\*i}) - r_{\*l^{(j)}}^{(j-1)}(\gp{\*l,\*i})$
          for all $(\*l,\*i) \in \liset$%
        }
        \Comment{new residuals}%
        \label{line:algResidualInterpolation4}
      \EndFor{}
      \State{%
        $\vlinout \gets \sum_{j=1}^{m} \vsurplus^{(j)}$
        (where $\surplus[(j)]{\*l,\*i} = 0$,
        $(\*l,\*i) \in \liset$,
        if $\lnot(\*l \le \*l^{(j)})$)%
      }
      \Comment{combine surpluses}%
    \EndFunction{}
  \end{algorithmic}
  \caption[%
    Hierarchization with residual interpolation%
  ]{%
    Application of the hierarchization operator $\linop = \intpmatinv$
    with residual interpolation
    for dimensionally adaptive sparse grids.
    Inputs are
    the vector $\vlinin = (\linin{\*l,\*i})_{(\*l,\*i) \in \liset}$
    of input data (function values $\fcnval{\*l,\*i}$ at the grid points) and
    the set $\levelset$ of levels that are part of
    the sparse grid (see \eqref{eq:dimensionallyAdaptiveSG}),
    where $\liset$ is the set of all feasible level-index pairs $(\*l,\*i)$,
    i.e., $\*l \in \levelset$, $\*i \in \hiset{\*l}$.
    The output is the vector
    $\vlinout = (\linout{\*l,\*i})_{(\*l,\*i) \in \liset}$
    of output data (hierarchical surpluses $\surplus{\*l,\*i}$).%
  }%
  \label{alg:residualInterpolation}%
\end{algorithm}

\paragraph{Correctness}

The principle of \Cref{alg:residualInterpolation} is maintaining
a vector $(r^{(j)}(\gp{\*l,\*i}))_{(\*l,\*i) \in \liset}$ of residuals
and interpolating the residual data subsequently on the active nodal spaces.
Again, note that it is necessary to compute the coefficients
$\surplus[(j)]{\*l,\*i}$ in the hierarchical basis, despite interpolating
on the full grid $\fgset{\*l^{(j)}}$.
In \cref{chap:a10proofs}, we prove that the algorithm satisfies
the following invariant, which can be used to show its correctness:

\begin{restatable}[invariant of residual interpolation]{%
  proposition%
}{%
  propInvariantResidualInterpolation%
}
  \label{prop:invariantResidualInterpolation}
  For $j = 1, \dotsc, m$, it holds
  \begin{subequations}
    \label{eq:propInvariantResidualInterpolationStatements}
    \begin{alignat}{4}
      \label{eq:propInvariantResidualInterpolation1}
      r_{\*l^{(j)}}^{(j-1)}(\gp{\*l,\*i})
      &= 0,\quad
      &&\*l \le \*l^{(j')},\;\;
      &&\*i \in \hiset{\*l},\quad
      &j'
      &= 1, \dotsc, j - 1,\\
      \label{eq:propInvariantResidualInterpolation2}
      r^{(j)}(\gp{\*l,\*i})
      &= 0,\quad
      &&\*l \le \*l^{(j')},\;\;
      &&\*i \in \hiset{\*l},\quad
      &j'
      &= 1, \dotsc, j,\\
      \label{eq:propInvariantResidualInterpolation3}
      r^{(j)}(\gp{\*l,\*i})
      &= \fcnval{\*l,\*i} - f^{\sparse,(j)}(\gp{\*l,\*i}),\quad
      &&\*l \in L,\;\;
      &&\*i \in \hiset{\*l},&&
    \end{alignat}
  \end{subequations}%
  \setlength{\abovedisplayskip}{0pt}%
  \begin{equation}
    \label{eq:propInvariantResidualInterpolation4}
    \text{where}\quad
    f^{\sparse,(j)}
    \ceq \sum_{\*l' \in \levelset} \sum_{\*i' \in \hiset{\*l'}}
    \paren*{\sum_{j'=1}^{j} \surplus[(j')]{\*l',\*i'}} \basis{\*l',\*i'}.
  \end{equation}
\end{restatable}

\begin{proof}
  See \cref{sec:a132proofResidualInterpolation}.
\end{proof}

\begin{corollary}[correctness of residual interpolation]
  \label{cor:algResidualInterpolationCorrectness}
  \Cref{alg:residualInterpolation} is correct for hierarchization
  on dimensionally adaptive sparse grids.
\end{corollary}

\begin{proof}
  Let $\*l \in \levelset$ and $\*i \in \hiset{\*l}$.
  By construction of the active nodal spaces,
  there exists some $j' \in \{1, \dotsc, m\}$ such that $\*l \le \*l^{(j')}$.
  By \cref{prop:invariantResidualInterpolation}, we obtain
  for $j = m$
  \begin{subequations}
    \label{eq:proofCorAlgResidualInterpolationCorrectness1}
    \begin{align}
      \sum_{\*l' \in L} \sum_{\*i' \in \hiset{\*l'}}
      \smash{
        \underbrace{
          \paren*{\sum_{j''=1}^{m} \surplus[(j'')]{\*l',\*i'}}
        }_{= \linout{\*l',\*i'}}
      }
      \basis{\*l',\*i'}(\gp{\*l,\*i})
      &\quad\;
      \mathclap{\overset{\eqref{eq:propInvariantResidualInterpolation4}}{=}}
      \quad\;
      f^{\sparse,(m)}(\gp{\*l,\*i})
      \overset{\eqref{eq:propInvariantResidualInterpolation3}}{=}
      \fcnval{\*l,\*i} - r^{(m)}(\gp{\*l,\*i})\\
      &\quad\;
      \mathclap{\overset{\eqref{eq:propInvariantResidualInterpolation2}}{=}}
      \quad\;
      \fcnval{\*l,\*i}.
    \end{align}
  \end{subequations}
  As the hierarchical interpolant $\sgintp$
  (see \eqref{eq:hierarchizationInterpolant})
  has the same form
  $\sum_{\*l' \in \levelset} \sum_{\*i' \in \hiset{\*l'}}
  \surplus{\*l',\*i'} \basis{\*l',\*i'}$ as the \lhs of
  \eqref{eq:proofCorAlgResidualInterpolationCorrectness1}
  with unique surpluses $\surplus{\*l',\*i'}$ such that the function values
  are interpolated (see \eqref{eq:hierarchizationProblem}),
  the coefficients $\linout{\*l',\*i'}$
  (output of \cref{alg:residualInterpolation})
  coincide with the surpluses $\surplus{\*l',\*i'}$.
\end{proof}

\vspace{1em}

\Cref{prop:invariantResidualInterpolation} shows that
$r^{(j)}(\gp{\*l,\*i})$ is the residual of the
interpolant $f^{\sparse,(j)}$ of iteration~$j$
to the objective function $\objfun$ at the grid points $\gp{\*l,\*i}$
(\cref{eq:propInvariantResidualInterpolation3}).
After interpolating $r^{(j-1)}$ on the grid $\fgset{\*l^{(j)}}$
to obtain the function $r_{\*l^{(j)}}^{(j-1)}$
and subtracting the resulting values from the old residual values,
the new residual values $r^{(j)}(\gp{\*l,\*i})$ vanish
not only on the grid $\{(\*l^{(j)}, \*i) \mid \*i \in \hiset{\*l^{(j)}}\}$,
but also on all previous grids
$\{(\*l^{(j')}, \*i) \mid \*i \in \hiset{\*l^{(j')}}\}$, $j' \le j$
(\cref{eq:propInvariantResidualInterpolation2}).
The proof of \cref{prop:invariantResidualInterpolation}
shows this by exploiting the auxiliary statement of
\cref{eq:propInvariantResidualInterpolation1}
and the tensor product structure of the hierarchical basis.

An example for the application of \cref{alg:residualInterpolation}
on a two-dimensional sparse grid can be seen in
\cref{fig:residualInterpolation}.
Note that
$\surplus[(j)]{\*l,\*i} \not= 0$ can only be true if $\*l \le \*l^{(j)}$.
Therefore, if $(\*l, \*i)$ is not contained in one of the
grids that are processed in one of the remaining iterations $j+1, \dotsc, m$,
then $\linout[(j)]{\*l,\*i}$ is already equal to the correct surplus
$\surplus{\*l,\*i}$,
where $\linout[(j)]{\*l,\*i} \ceq \sum_{j'=1}^j \surplus[(j')]{\*l,\*i}$
denotes the intermediate result obtained after $j$ iterations.

\begin{SCfigure}
  \includegraphics{residualInterpolation_1}%
  \caption[%
    Hierarchization with residual interpolation%
  ]{%
    Hierarchization of function value data on the
    two-dimensional regular sparse grid
    $\regsgspace{n}{d}$ of level $n = 3$ \emph{(top left)}
    using the method of residual interpolation.
    In this figure, we use
    $\linout[(j)]{\*l,\*i} \ceq \sum_{j'=1}^j \surplus[(j')]{\*l,\*i}$
    as an abbreviation.
    The order of the nodal spaces (here: bottom left to top right)
    does not matter.
    The data $\linout[(j)]{\*l,\*i}$
    corresponding to \textcolor{C0}{blue grid points}
    will not be modified in the remaining iterations
    and, therefore, already equals the correct surpluses $\surplus{\*l,\*i}$.
    The data corresponding to \textcolor{C1}{red grid points}
    will be modified as the grid points appear in one of the remaining
    nodal grids.%
  }%
  \label{fig:residualInterpolation}%
\end{SCfigure}


\section{Analysis}

\subsection{Algorithm selection}

To select algorithm I use the simulated dataset. This dataset contains the latent response variable (the Sharpe ratio of each stock). We evaluate algorithm by the lowest \textit{mean squared error (mse)} of Sharpe ratios.

\begin{equation}\label{eq:meansquarederror}
    mse = \sum_{t=1}^{T} \sum_{i=1}^{k}\lp \hat{sr}_{i}^{(t)} - sr_{i}^{(t)} \rp^2
\end{equation}

The algorithm with the lowest $mse$ is the algorithm selected to be tested on real data. Before testing each algorithm we need to tune, and in the case of the $LSTM$ train the algorithm. Training the algorithm consists of tuning the parameters that are endogenous to the algorithm, and tuning consists of finding the correct values for the hyper parameters.

The LSTM is trained by splitting the dataset into two separate parts (a training set and validation set). The LSTM is trained for 4 epochs, which is the number of times the LSTM goes through the entire training data.

The Naive Rolling Sharpe is tuned on 5000 observations, where a grid search over $h= \{50, 100, 150, 200, 250, 300\}$ is used. The $h$ which corresponds to lowest $mse$ is used for the final evaluation between the algorithms. We find that $h=200$ is the best hyper parameter value for the Naive Rolling Sharpe.

The Rolling Sharpe needs the three hyper parameters: $ h_{\text{long term}}$, $h_{\text{short term}}, \tau$. We create a grid of $3^3=27$ different values as shown in equation \ref{eq:rs_grid}. The algorithm is tuned over this grid, and we find the best hyper parameters to be $h_{\text{short term}} = 30, h_{\text{long term}} = 150, \tau = 30 $.

\begin{equation}\label{eq:rs_grid}
    grid = \underset{\tau}{\{0.3, 0.5, 0.8\}} \times \underset{h_{\text{long term}}}{\{50, 100, 150\}} \times \underset{h_{\text{short term}}}{\{15, 30, 45\}}
\end{equation}

Comparing the three trained and tuned algorithms on the same $5000$ observations yields: $MSE_{\text{LSTM}}= 0.0156$, $MSE_{\text{RS}_{naive}}= 0.0181$, $MSE_{\text{RS}}= 0.0131$. We conclude that the Rolling Sharpe algorithm performs the best. In the appendix figure \ref{fig:rollingsharptest} shows the underlying true Sharpe ratio for a given stock and the corresponding prediction made by the rolling Sharpe algorithm.

\subsection{Performance on real data}

Before applying the Rolling Sharpe algorithm on the real data, a couple of benchmark is established. The \textit{perfect stock pick} benchmark, is made by assuming that we as trader had perfect foresight over the next period, and was able to pick the stock in each period which yields the highest return. The second benchmark is the \textit{Apple} stock (with ticker \textbf{AAPL}). The Apple stock is chosen, since this is the stock with the highest individual Sharpe ratio of all the stocks throughout the period. The last benchmarks are two different tangency portfolios. One based on the first 1000 observations, and a tangency portfolio based on the entire period.

Equation \ref{eq:tangencyport} displays the formula for the tangency portfolio, where $\w$ is the weights of the portfolio, and: $\mu_{adj} = \mu - \bar{r}\cdot \mathbf{1}$. Which is the risk adjusted return. It should be noted that $\w$ is a vector of length $k$ and $\sum_{w \in \w} w = 1$.

\begin{equation}\label{eq:tangencyport}
    \w = \frac{\Omega^{-1} \cdot \mu_{adj}}{\mathbf{1}^{T} \cdot \Omega^{-1} \cdot \mu_{adj}}
\end{equation}

Looking to table \ref{tab:performance} we find that the rolling Sharpe algorithm performs very well. Comparing to \textbf{Apple} we find that the sharp ratio is 4 times higher, and the expected returns is over twice as high, and a somewhat lower standard deviation. Comparing to the tangency portfolio compared over the entire period we find the Sharpe ratio to be twice as high for the rolling Sharpe algorithm. The standard deviation of the tangency portfolio is lower than that of the rolling Sharpe algorithm. However that expected return from the rolling Sharpe algorithm is almost 4 times higher. Lastly comparing to tangency portfolio of only the first 1000 observations, which more realistically would represent how an investor would generate portfolio weights, i.e. the investor cannot use the future innovations of the data generating process to construct portfolio weights. Here again we find that the rolling Sharpe algorithm has 4 times as high Sharpe ratio.

\begin{table}[ht]
\centering
\caption{Summary statistics of performance of different portfolios}
\input{tables/analysis_portfolio_performance.tex}
\label{tab:performance}
\end{table}

Figure \ref{fig:loginvest} displays (the log transformed) portfolio given one had invested following the different benchmarks, using the real data. We see again that the rolling Sharpe ratio is by far the best way to invest throughout the period. Interestingly the poorest performing portfolio is the tangency portfolio calculated on the first 1000 observations.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.45]{figures/log_investment_experiment.png}
\caption{Counter factual portfolio performance (in logs)}
\label{fig:loginvest}
\end{figure}

Figure \ref{fig:logmontecarlo} uses Monte Carlo simulation to compare the different algorithms. By assuming that returns are normal distributed, we simulate portfolios following each of the four strategies: \textit{Rolling Sharpe, Apple Portfolio, Tangency portfolio (1000 obs), Tangency Portfolio (full)}. Since each portfolio have given both a mean and a standard deviation, we can use these distribution moments to randomly draw a return from given portfolio strategy. For each strategy we simulate 10000 portfolios. We run each portfolio for 7013 time steps.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.45]{figures/boxplot_monte_carlo.png}
\caption{Monte Carlo simulation - performance (in logs)}
\label{fig:logmontecarlo}
\end{figure}

Here we find again the rolling Sharpe algorithm performs best. The apple and tangency portfolio for the entire period seems to be doing approximately equal, comparing the mean, however, the variance is considerably lower for the tangency portfolio. Note the logarithmic \textit{y}-scale.


\section{Gram-Schmidt and Orthogonal Complement}
Here we only interested in inner product spaces over $F=\mathbb R$ or $\mathbb C$.
\begin{lemma}[Cauchy-Schwartz Inequality]
    $|\langle u,v\rangle|\le\|u\|\|v\|$.
\end{lemma}
In particular, equality hold iff $u,v$ are linearly dependent.
\begin{proof}
    For $t\in F$, expanding $\langle tu-v,tu-v\rangle \ge 0$ gives
    $$0\le|t|^2\|u\|^2-2\operatorname{Re}(t\langle u,v\rangle)+\|v\|^2$$
    Picking $t=\overline{\langle u,v\rangle}/\|u\|^2$ ends the proof.
\end{proof}
\begin{corollary}[Triangle Inequality]
    $\|u+v\|\le \|u\|+\|v\|$.
\end{corollary}
Consequently $\|\cdot\|$ is a indeed a norm.
\begin{proof}
    Square both sides and use Cauchy-Schwartz.
\end{proof}
\begin{definition}
    Fix an inner product $\langle\cdot,\cdot\rangle$.
    A set $\{e_1,\ldots,e_k\}$ of vectors in $V$ is orthogonal if $\langle e_i,e_j\rangle=0$ for $i\neq j$ and orthonormal if in addition they all have norm $1$, that is $\langle e_i,e_j\rangle=\delta_{ij}$.
\end{definition}
Note that both notion depends on our choice of inner product.
\begin{lemma}
    A set of orthogonal vectors $e_1,\ldots,e_k$ has to be linearly independent.
    In fact, if $v=\sum_i\lambda_ie_i$ then $\lambda_i=\langle v,e_i\rangle/\|e_i\|$.
\end{lemma}
\begin{proof}
    Immediate from bilinearity.
\end{proof}
\begin{lemma}[Parseval's Identity]
    If $V$ is a finite dimensional inner product space and $e_1,\ldots,e_n$ is an orthonormal basis, then
    $$\langle u,v\rangle=\sum_{i=1}^n\langle u,e_i\rangle\overline{\langle v,e_i\rangle}$$
\end{lemma}
\begin{proof}
    Obvious from the preceding lemma.
\end{proof}
In particular, in an orthogonal basis, $\|v\|^2=\sum_i|\langle v,e_i\rangle|^2$.
Does an orthogonal basis always exist?
\begin{theorem}[Gram-Schmidt Orthogonalisation]
    If we have an inner product space $V$ and a sequence of linearly independent vectors $(v_i)_{i\in I}\in V$ where $I=\{1,2,\ldots\}$ (which may or may not terminate), then there exists a sequence $(e_i)_{i\in I}$ of orthonormal vectors such that $\langle v_1,\ldots,v_k\rangle=\langle e_1,\ldots,e_k\rangle$ for any $k\in I$.
\end{theorem}
\begin{proof}
    We shall define $(e_i)$ inductively on $k$.
    For $k=1$, just take $e_1=v_1/\|v_1\|$.
    Say we have found $e_1,\ldots,e_k$, then define
    $$e_{k+1}'=v_{k+1}-\sum_{i=1}^k\langle v_{k+1},e_i\rangle e_i,e_{k+1}=\frac{1}{|e_{k+1}'|}e_{k+1}'$$
    This is well-defined as $(v_i)$ is linearly independent (so $e_{k+1}'\neq 0$) and it is easy to verify that $\langle v_1,\ldots,v_{k+1}\rangle=\langle e_1,\ldots,e_{k+1}\rangle$.
    This completes the proof.
\end{proof}
So not only does there exists such a set of orthonormal vectors, we also get an algorithm to compute it.
\begin{corollary}
    Let $V$ be a finite dimensioanl inner product space, then any orthonormal set of vectors can be extend to an orthonormal basis of $V$.
\end{corollary}
\begin{proof}
    Extend it to a basis, then apply the Gram-Schmidt algorithm (which fixes the original set).
\end{proof}
\begin{note}
    A matrix $A\in M_{m,n}(F)$ has orthogonal columns if $A^\top\bar{A}=I$.
\end{note}
\begin{definition}
    $A\in M_n(\mathbb R)$ is orthogonal if $A^\top A=I$.
    $A\in M_n(\mathbb C)$ is unitary if $A^\top\bar{A}=I$.
\end{definition}
\begin{proposition}
    Any nonsingular $A\in M_n(\mathbb R)$ (resp. $M_n(\mathbb C)$) can be written as $A=RT$ where $T$ is upper-triangular and $R$ is orthogonal (resp. unitary).
\end{proposition}
\begin{proof}
    Do Gram-Schmidt on columns of $A$.
\end{proof}
\begin{definition}
    Let $V$ be an inner product space and $V_1,V_2\le V$.
    We say $V$ is the orthogonal sum of $V_1,V_2$ (written as $V=V_1\oplus^\perp V_2$) if $V=V_1\oplus V_2$ and $\forall v_1,\in V_1,v_2\in V_2$, we have $\langle v_1,v_2\rangle=0$.
\end{definition}
\begin{definition}
    Let $V$ be an inner product space and $W\le V$.
    We define $W^\perp=\{v\in V:\forall w\in W,\langle v,w\rangle=0\}$.
\end{definition}
\begin{lemma}
    $W\oplus^\perp W^\perp=V$ if $V$ is finite dimensional.
\end{lemma}
\begin{proof}
    Clearly $W^\perp\le V$ and by definition the sum $W+W^\perp$ is direct and orthogonal.
    So it suffices to show that $V=W+W^\perp$, which is obvious since we can obtain a basis of $W^\perp$ of the right size by extending an orthonormal basis on $W$ orthonormally to $V$.
\end{proof}


\section{Project Overview}

One of the focus areas for DIANA~\cite{DIANA-proposal-2014} is to ``establish infrastructure for a higher-level of collaborative analysis, building on the successful patterns used for the Higgs boson discovery''.
A large component of this focus is statistical software.
\code{RooFit}~\cite{Verkerke:2003ir} is one of the primary tools used now, but it is facing scalability challenges.
To address these issues, this fellowship project investigated the ability of software libraries for numerical computations using data flow graphs and automatic differentiation (e.g., TensorFlow~\cite{tensorflow2015-whitepaper}, PyTorch~\cite{paszke2017automatic}, and MXNet~\cite{DBLP:journals/corr/ChenLLLWWXXZZ15}) to improve the performance of statistical fits through parallelism and GPU assisted speed up.\\

Under the mentorship of Gilles Louppe and Vince Croft, Matthew became familiar with and investigated the behavior and benefits of different computational graph frameworks.
Specifically, declarative frameworks (i.e., TensorFlow) and imperative frameworks (i.e., PyTorch and MXNet).
Declarative frameworks require the computational graph to be fully declared in advance of computation, allowing for the graph to be compiled in a sublanguage virtual machine and then run in separate sessions.
This offers the ability to efficiently reuse both memory and graphs.
However, the declarative nature requires that for implementation of a new idea into code a graph must always be implemented first, potentially slowing the exploratory analysis stage.
Alternatively, imperative frameworks eagerly run computational graphs within the defining language in which the graph is written (i.e., the programming language itself is the execution engine).
Imperative frameworks allow more easily for interactive and exploratory computing, though at the loss of reusable graphs.~\cite{Chintala:2302087}\\

After investigation, it was decided that the project scope should not be limited prematurely to a single framework, given that both declarative and imperative frameworks have clear use cases in high energy physics --- where the physicists writing the code tend to simultaneously embody the roles of software researcher, software engineer, and end user.
Instead, comparisons of the performance of TensorFlow, PyTorch, and MXNet would be made in different scenarios.\\

Partnering with Lukas Heinrich, who then assumed responsibility of further mentoring and work with Matthew, pyhf~\cite{lukas_heinrich_2018_1172961} was developed.
pyhf is a Python based implementation of the \texttt{HistFactory}~\cite{Cranmer:2012sba} specification that allows different computational graph frameworks to be used as computational backends.
The pyhf project easily allows for testing of the performance of the frameworks in different scenarios, given its easily understandable Pythonic API and ability to switch between computational graph backends with a single function call.
In addition to supporting TensorFlow, PyTorch, and MXNet as backends, a NumPy based backend was also implemented.
Auto differentiable best fit optimizers were also implemented for the respective backends.
The optimizers currently implement Newton's method, requiring the computationally expensive (in high dimensional space) inverse of the Hessian matrix.
However, the use of the graph frameworks greatly mitigates this cost as the underlying libraries have been written for high performance in exactly these situations.


\subsection{Physically Meaningful Model Parameterization}
To accelerate the process of finding a near-optimal set of parameters, it is important to limit the search space of the parameterization algorithm. This is especially important when the number of parameters is large. We have found that it is beneficial for the parameters to have physical meaning in order to specify realistic bounds.

The elastic behaviour is parameterized by Young's modulus, $E$, and Poisson's ratio, $\nu$. Bounds on these quantities are well known. The yield function and flow potential function are parameterized in terms of the friction angle, dilation angle, and the stress ratio $K$. Again, bounds on these quantities is relatively well known.

The hardening function (\ref{eqn:param2-1}) is given in terms of two empirical coefficients $\alpha$ and $\beta$ and the initial compressive yield stress $\sigma_c^{iy}$.  While it is possible to set bounds on $\sigma_c^{iy}$, it is less straightforward to set bounds for $\alpha$ and $\beta$ as they do not have obvious physical meaning. The hardening function for the Barcelona model is shown in Figure \ref{fig:barcelona}. The coefficients $\alpha$ and $\beta$ can be rewritten in terms of the peak compressive yield strength, $\sigma_{c}^{p}$, the plastic strain at the peak compressive yield strength, $\epsilon_c^{p}$, and the initial compressive yield stress $\sigma_c^{iy}$:


\documentclass[t,usenames,dvipsnames]{beamer}
\usetheme{Copenhagen}
\setbeamertemplate{headline}{} % remove toc from headers
\beamertemplatenavigationsymbolsempty

\usepackage{amsmath, tcolorbox, bm, tikz, pgfplots}
\pgfplotsset{compat = 1.16}

\everymath{\displaystyle}

\title{Simplifying Radical Expressions}
\author{}
\date{}

\AtBeginSection[]
{
  \begin{frame}
    \frametitle{Objectives}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
    \maketitle
\end{frame}

\section{Write radicals as rational exponents and vice versa.}

\begin{frame}{Radicand}
\begin{tcolorbox}[colframe=green!20!black, colback = green!30!white,title=\textbf{Radicand}]
For $\sqrt[n]{a}$, the \textbf{radicand} is $\bm{a}$.
\end{tcolorbox}
\end{frame}

\begin{frame}{Radicals}
What is $\left(\sqrt[3]{27}\right)^2$? \newline\\

Well, if we evaluate $\sqrt[3]{27}$ first, we get 3. Then when we square that, we get the final answer of 9.
\end{frame}

\begin{frame}{Radicals}
To put things visually, suppose the block below represents 27.

\begin{center}
    \begin{tikzpicture}[scale=0.5]
    \draw (0,0) rectangle (15,1);
    \end{tikzpicture}
\end{center}

Since we are taking the cube root, we can divide the large block up into 3 equal blocks where we multiply by 3 to get each new number. (\emph{Note}: We can divide it up into 2 equal blocks for square root, 3 for cube root, 4 for fourth root, etc.):
\begin{center}
    \begin{tikzpicture}[scale=0.5]
    \draw (0,0) rectangle (15,1);
    \node at (0,0) [anchor=north] {1};
    \node at (15,0) [anchor=north] {27};
    \draw (5,0) -- (5,1);
    \node at (5,0) [anchor=north] {3};
    \draw (10,0) -- (10,1);
    \node at (10,0) [anchor=north] {9};
    \end{tikzpicture}
\end{center}
\end{frame}

\begin{frame}{Radicals}
Now let's look what happens as we shade in the block from the left, up to our answer of 9:
\begin{center}
    \begin{tikzpicture}[scale=0.5]
    \draw (0,0) [fill=yellow, color=yellow] rectangle (5,1);
    \draw (5,0) [fill=yellow, color=yellow] rectangle (10,1);
    \draw (0,0) rectangle (15,1);
    \node at (0,0) [anchor=north] {1};
    \node at (15,0) [anchor=north] {27};
    \draw (5,0) -- (5,1);
    \node at (5,0) [anchor=north] {3};
    \draw (10,0) -- (10,1);
    \node at (10,0) [anchor=north] {9};
    \end{tikzpicture}
\end{center}

Notice that out of the three equal areas, two of them are shaded. \newline\\

This is a visual approach to the idea that $\left(\sqrt[3]{27}\right)^2 = 27^{2/3}$   \newline\\

In general,
\[
\sqrt[\text{root}]{x^{\text{power}}} = x^{\text{power/root}}
\]
\end{frame}

\begin{frame}{Example 1}
Write each of the following using rational exponents.	\newline\\
(a) \quad $\sqrt{6}$		\pause	\vspace{6pt}
\[ 6^{1/2} \]	\pause	\vspace{6pt}
(b) \quad $\sqrt[3]{8}$	\pause	\vspace{6pt}
\[ 8^{1/3} \] \pause	\vspace{6pt}
(c) \quad $\sqrt[4]{x^3}$	\pause	\vspace{6pt}
\[ x^{3/4} \]
\end{frame}

\begin{frame}{Example 2}
Write each of the following in radical form.	\newline\\
(a)	\quad	$5^{1/2}$	\pause \vspace{6pt}
\[ \sqrt{5} \]	\pause \vspace{6pt}
(b)	\quad	$(-9)^{5/3}$	\pause \vspace{6pt}
\[	\sqrt[3]{(-9)^5} \] \pause \vspace{6pt}
(c)	\quad	$x^{-1/3}$	\pause \vspace{6pt}
\[ \sqrt[3]{x^{-1}} \text{\quad or \quad} \sqrt[3]{\frac{1}{x}} \]
\end{frame}

\section{Simplify square root and higher root expressions.}

\begin{frame}{Even Roots and Exponents}
When dealing with \textbf{even} roots and exponents, keep in mind that the root and exponent don't ``cancel each other out."
\newline\\

For instance, $\sqrt{5^2}=\sqrt{25}=5$, but $\sqrt{(-5)^2}=\sqrt{25}=5$.
\end{frame}

\begin{frame}{Even Roots and Exponents}
The graphs of $y = \sqrt{x^2}$ and $y=|x|$ are shown. Notice they are identical.
\newline\\

\begin{tabular}{cc}
    {\color{blue}$y = \sqrt{x^2}$}    &   {\color{red}$y = |x|$}   \\[11pt]
    \begin{tikzpicture}[scale=0.65]
    \begin{axis}
    [
        xmin = -5,
        xmax = 5,
        ymin = -5,
        ymax = 5,
        axis lines = middle,
        grid,
        minor tick num = 1
    ]
    \addplot [<->, color = blue, line width = 1.5, domain=-4.5:4.5, samples = 300] {abs(x)};
    \end{axis}
    \end{tikzpicture}
    &
    \begin{tikzpicture}[scale=0.65]
    \begin{axis}
    [
        xmin = -5,
        xmax = 5,
        ymin = -5,
        ymax = 5,
        axis lines = middle,
        grid,
        minor tick num = 1
    ]
    \addplot [<->, color = red, line width = 1.5, domain=-4.5:4.5, samples = 300] {abs(x)};
    \end{axis}
    \end{tikzpicture}
\end{tabular}
\vspace{8pt}

Thus, for any real number $x$, $\sqrt{x^2} = |x|$.
\end{frame}

\begin{frame}{Odd Roots and Exponents}
Odd roots, such as $\sqrt[3]{\quad}$ do not follow the same rule as even roots.
\newline\\

So for any real number $x$, $\sqrt[3]{x^3} = x$.
\newline\\

And, in general, for any real number $x$,	\newline\\
\begin{enumerate}
	\item If $n$ is even, $\sqrt[n]{x^n} = |x|$.	\newline\\
	\item If $n$ is odd, $\sqrt[n]{x^n} = x$.
\end{enumerate}
\end{frame}

\begin{frame}{Example 3}
Simplify each of the following. Exact answers only. Use absolute value bars when necessary.		\newline\\
(a) \quad $\sqrt{72x^2}$
\begin{align*}
\onslide<2->{\sqrt{72x^2} &= \sqrt{72} \cdot \sqrt{x^2}} \\[8pt]
\onslide<3->{&= 6\sqrt{2} \cdot |x|} \\[8pt]
\onslide<4->{&= 6|x| \sqrt{2}}
\end{align*}
\end{frame}

\begin{frame}{Example 3}
(b) \quad $\sqrt{175x^3}$
\begin{align*}
\onslide<2->{\sqrt{175x^3} &= \sqrt{175} \cdot \sqrt{x^3}} \\[8pt]
\onslide<3->{&= 5\sqrt{7} \cdot |x|\sqrt{x}} \\[8pt]
\onslide<4->{&= 5|x|\sqrt{7x}}
\end{align*}
\end{frame}

\begin{frame}{Example 3}
(c) \quad $\sqrt{18x^4}$
\begin{align*}
\onslide<2->{\sqrt{18x^4} &= \sqrt{18} \cdot \sqrt{x^4}} \\[8pt]
\onslide<3->{&= 3\sqrt{2} \cdot x^2} \\[8pt]
\onslide<4->{&= 3x^2\cdot \sqrt{2}}
\end{align*}
\end{frame}

\begin{frame}{Example 3}
(d) \quad $\sqrt{65x^5}$
\begin{align*}
\onslide<2->{\sqrt{65x^5} &= \sqrt{65} \cdot \sqrt{x^5}} \\[8pt]
\onslide<3->{&= \sqrt{65} \cdot x^2\sqrt{x}} \\[8pt]
\onslide<4->{&= x^2\cdot \sqrt{65x}}
\end{align*}
\end{frame}

\begin{frame}{Example 3}
(e) \quad $\sqrt[3]{27x^7}$
\begin{align*}
\onslide<2->{\sqrt[3]{27x^7} &= \sqrt[3]{27} \cdot \sqrt[3]{x^7}} \\[8pt]
\onslide<3->{&= 3 \cdot x^2\sqrt[3]{x}} \\[8pt]
\onslide<4->{&= 3x^2\cdot \sqrt[3]{x}}
\end{align*}
\end{frame}

\begin{frame}{Example 3}
(f) \quad $\sqrt[3]{128x^6}$
\begin{align*}
\onslide<2->{\sqrt[3]{128x^6} &= \sqrt[3]{128} \cdot \sqrt[3]{x^6}} \\[8pt]
\onslide<3->{&= 4\cdot\sqrt[3]{2} \cdot x^2} \\[8pt]
\onslide<4->{&= 4x^2\cdot \sqrt[3]{2}}
\end{align*}
\end{frame}

\section{Perform operations with radical expressions.}

\begin{frame}{Performing Operations with Radical Expressions}
Once we have simplified the radicals, we can add and subtract radical expressions with the same radicands and roots. This is essentially combining like terms.
\end{frame}

\begin{frame}{Example 4}
Simplify each of the following. Exact answers only.	\newline\\
(a)	\quad	$\sqrt{98} + \sqrt{8}$
\begin{align*}
\onslide<2->{\sqrt{98} + \sqrt{8} &= 7\sqrt{2} + 2\sqrt{2}} \\[8pt]
\onslide<3->{&= 9\sqrt{2}}
\end{align*}
\end{frame}

\begin{frame}{Example 4}
(b)	\quad	$\sqrt{108x} - \sqrt{300x}$
\begin{align*}
\onslide<2->{\sqrt{108x} - \sqrt{300x} &= 6\sqrt{3x} - 10\sqrt{3x}} \\[8pt]
\onslide<3->{&= -4\sqrt{3x}}
\end{align*}
\end{frame}

\begin{frame}{Example 4}
(c)	\quad	$4\sqrt{6} + 3\sqrt{54} - 5\sqrt{45}$
\begin{align*}
\onslide<2->{4\sqrt{6} + 3\sqrt{54} - 5\sqrt{45} &= 4\sqrt{6} + 3(3\sqrt{6}) - 5(3\sqrt{5})} \\[8pt]
\onslide<3->{&= 4\sqrt{6} + 9\sqrt{6} - 15\sqrt{5}} \\[8pt]
\onslide<4->{&= 13\sqrt{6} - 15\sqrt{5}}
\end{align*}
\end{frame}

\begin{frame}{Multiplying and Dividing Radical Expressions}
We can also multiply and divide radical expressions. It may be helpful to convert them to rational exponent form first, and then use your laws of exponents.
\end{frame}

\begin{frame}{Example 5}
Simplify each of the following. Leave no radical expressions in a denominator.	\newline\\
(a)	\quad $\sqrt{x^5} \cdot \sqrt{x^2}$
\begin{align*}
\onslide<2->{\sqrt{x^5} \cdot \sqrt{x^2} &= x^{5/2} \cdot x^{2/2}} \\[8pt]
\onslide<3->{&= x^{7/2}} \\[8pt]
\onslide<4->{&= \sqrt{x^7}} \\[8pt]
\onslide<5->{&= |x^3|\sqrt{x}}
\end{align*}
\end{frame}

\begin{frame}{Example 5}
(b)	\quad $\sqrt{a^3} \cdot \sqrt[3]{a^2}$
\begin{align*}
\onslide<2->{\sqrt{a^3} \cdot \sqrt[3]{a^2} &= a^{3/2} \cdot a^{2/3}} \\[8pt]
\onslide<3->{&= a^{13/6}} \\[8pt]
\onslide<4->{&= \sqrt[6]{a^{13}}} \\[8pt]
\onslide<5->{&= a^2 \cdot \sqrt[6]{a}}
\end{align*}
\end{frame}

\begin{frame}{Example 5}
(c)	\quad $\sqrt[3]{y^6} \cdot \sqrt{y^3}$
\begin{align*}
\onslide<2->{\sqrt[3]{y^6} \cdot \sqrt{y^3} &= y^{6/3} \cdot y^{3/2}} \\[8pt]
\onslide<3->{&= y^{7/2}} \\[8pt]
\onslide<4->{&= \sqrt{y^7}} \\[8pt]
\onslide<5->{&= |y^3| \cdot \sqrt{y}}
\end{align*}
\end{frame}

\begin{frame}{Example 5}
(d)	\quad $\dfrac{\sqrt{x^5}}{\sqrt{x}}$
\begin{align*}
\onslide<2->{\frac{\sqrt{x^5}}{\sqrt{x}} &= \frac{x^{5/2}}{x^{1/2}}} \\[12pt]
\onslide<3->{&= x^{4/2}} \\[12pt]
\onslide<4->{&= x^2}
\end{align*}
\end{frame}

\begin{frame}{Example 5}
(e)	\quad $\dfrac{7}{\sqrt{2}}$
\begin{align*}
\onslide<2->{\frac{7}{\sqrt{2}}&=\frac{7}{2^{1/2}}} \\[12pt]
\onslide<3->{&= \frac{7}{2^{1/2}} \left(\frac{2^{1/2}}{2^{1/2}} \right)} \\[12pt]
\onslide<4->{&= \frac{7\cdot 2^{1/2}}{2}} \\[12pt]
\onslide<5->{&= \frac{7\sqrt{2}}{2}}
\end{align*}
\end{frame}

\begin{frame}{Example 5}
(f) \quad $\dfrac{\sqrt[3]{y^6}}{\sqrt{y^3}}$
\begin{align*}
\onslide<2->{\frac{\sqrt[3]{y^6}}{\sqrt{y^3}} &= \frac{y^{6/3}}{y^{3/2}}}	\\[12pt]
\onslide<3->{&= y^{1/2}} \\[12pt]
\onslide<4->{&= \sqrt{y}}
\end{align*}
\end{frame}

\end{document}


\documentclass[10pt, handout]{beamer}
\setbeamertemplate{navigation symbols}{}
\usefonttheme{serif}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{cite}
\usepackage{color}
\usepackage{setspace}
\usepackage{hyperref}

\newcommand{\xx}{{\bf{x}}}

\begin{document}
\title{Machine Learning I Lecture VI:\\ Linear models for Classification}
\author{Jakob H Macke\\ Max Planck Institute for Biological Cybernetics\\ Bernstein Center for Computational Neuroscience}
\date{XY.XY.2012}

\frame{\titlepage}

%\frame{\frametitle{Today: Back to basics of probability theory}}

\frame{\frametitle{Plan for today}\tableofcontents}

\section{Binary classification}
\frame{\frametitle{Binary Classification: Assign each data point to one of two classes.}

%\multicolumn
\begin{columns}
\begin{column}{4cm}
\includegraphics[width=1.1\textwidth]{LinClassification.pdf}
\end{column}
\begin{column}{7.5cm}
Examples:
\begin{itemize}
 \item Is there a face in this image?
\item Will this neuron spike in response to this stimulus?
\item Based on this brain-scan, does this patient have a given disease or not?
\item  Will this customer buy this product or not?
\item Is this person likely to be a democrat/republican?
\end{itemize}
%\item
 \pause Notation: we have data $D=\{(x_1, t_1),\ldots, (x_N, t_N)\}$, with $t_n=1$ if $x_n$ belongs to class $1$ and $t_n=-1$ if $x_n$ belongs to class $-1$.
\end{column}
\end{columns}
}

\frame{\frametitle{We focus on linear decision rules, also known as `linear discriminant functions'.}

%\multicolumn

\includegraphics[width=.5\textwidth]{LinClassification.pdf}
\pause
\includegraphics[width=.5\textwidth]{NonLinClassification.pdf}

\pause

Of course, linear algorithms can be used together with \alert{nonlinear feature spaces} or \alert{nonlinear basis functions} in order to solve nonlinear classification problems!
}

\frame{\frametitle{Linear discriminants separate the space by a hyperplane, and the parameters define its normal vector.}

%\multicolumn

%\includegraphics[width=.4\textwidth]{LinearSeparators.pdf}
\begin{itemize}
\item Decision function: $y(\xx)=\omega^\top \xx + \omega_o$
\item \pause Classification: \begin{align}
\mbox{if~}y(\xx)>0 & \mbox{~say $\xx$ belongs to class 1}\\
\mbox{if~}y(\xx)<0 & \mbox{~say $\xx$ belongs to class -1}\\
 \end{align}
\item The decision-surface has equation $y(\xx)=0$, and is a hyperplane of dimensionality $D-1$.
\item \pause $\omega$ is the normal vector to the plane, and points into the positive class.
\item \pause $\omega_o$ determines the location of the decision-surface
\item \pause $|y(\xx)|$ is proproptional to the perpendicular distance to the decision-surface (with factor $1$ if $|| \omega ||=1$).
\end{itemize}

}

\frame{\frametitle{Multiple algorithms and methods exist for finding a good $\omega$.}
\begin{itemize}
\item Mis-classification rate $C(\omega)= \frac{1}{N} \sum_n \delta\left[y(\xx_n) =t_n\right]$ (i.e. average number of errors) difficult to optimize over $\omega$, and might have multiple solutions.
\item \pause Many algorithms can be derived by replacing $C$ by another cost-function which can be optimized.
\item \pause Linear classification algorithms include Least-square classification, Fisher's linear Discriminant, Logistic regression, Support Vector Machines and Rosenblatts' perceptron.
\end{itemize}
}

\section{Least Square Classification}
\frame{\frametitle{You already know one algorithm for linear classification: least square classification.}
\begin{itemize}
\item We have to fit the function $y(\xx)= \omega^\top \xx+ \omega_o $ to data.
\item \pause Simply do a linear regression from $\xx$ to $t$ by minimizing the sum-of-squared errors $\sum_n (y(\xx_n)-t_n)^2$.
\item \pause $\omega_{reg}=  \left(\sum_n x_n x_n^\top  \right)^{-1} \sum_n x_n t_n$
\item \pause Q: In what situations might this be a bad idea?
\end{itemize}
\pause
\includegraphics[width=.4\textwidth]{Figure44a.pdf}
\pause
\includegraphics[width=.4\textwidth]{Figure44b.pdf}
\\
\tiny Bishop PRML Figure 4.4
}

\section{Fisher's linear discriminant}
\frame[shrink=5]{\frametitle{'Fisher's linear discriminant' is a classical and simple algorithm for linear classification}
\begin{columns}
\begin{column}{3.5cm}

\includegraphics[width=1.2\textwidth]{Figure46a.pdf}

\includegraphics[width=1.2\textwidth]{Figure46b.pdf}

~\\
\tiny Bishop PRML Figure 4.6
\end{column}
\begin{column}{7.5cm}
\begin{itemize}
\item $\mathbf{m_+}= \frac{1}{N_+}\sum_{n \in C_+} x_n$ \hspace{.5cm} $\mathbf{m_{-}}= \frac{1}{N_-}\sum_{n \in C_{-}} x_n$
\item \pause Maximize projection-distance of class means
[projected mean/variance: on board]
$\omega_{simple} \propto \mathbf{m}_+-\mathbf{m}_-$
\item \pause Maximizing distance between means ignores that the projected variances might also be big.
\item \pause Fix:  Maximize the ratio of between-class variance to within-class variance ('signal to noise'). Fisher criterion
\begin{align}
J_\omega = 2\frac{(m_+-m_-)^2}{s_+^2+s_-^2}
\end{align}
[Details and solution: on board]
%\item
\pause
$\omega_{lda}= \Sigma_w^{-1} (\mathbf{m}_+-\mathbf{m}_-)$
\end{itemize}
\end{column}
\end{columns}

}
%http://users.informatik.uni-halle.de/~hinnebur/Lehre/BN_seminar_web/bn_05_ag.pdf

\frame{\frametitle{Aside: The multivariate Gaussian}
\begin{itemize}
\item Probability density function of $D$ dimensional Gaussian with mean $\mu$ an covariance $\Sigma$: \begin{align}p(x| \mu, \Sigma)&= (2\pi)^{-D/2}|\Sigma|^{-1/2} \exp \left(-\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu) \right) \\
 \end{align}
\item \pause Maximum likelihood estimation of parameters: \begin{align}
\hat\mu= &\frac{1}{N}\sum_n x_n\mbox{~~(empirical mean)}\\
\hat \Sigma= &\frac{1}{N} \sum_n x_n x_n^\top- \hat\mu \hat \mu^\top\mbox{~~(empirical covariance)}
 \end{align}
\end{itemize}
\begin{center}
\pause
\includegraphics[width=.25\textwidth]{Figure28a.pdf}
\includegraphics[width=.25\textwidth]{Figure28b.pdf}
\includegraphics[width=.25\textwidth]{Figure28c.pdf}
\end{center}
~\\
\tiny Bishop PRML Figure 2.8
}

\frame[shrink=5]{\frametitle{A (super brief) primer on covariance matrices. (more details/intuition in second half of course?)}
%\begin{columns}
%\begin{column}{4.5cm}

%\end{column}
\begin{center}
\includegraphics[width=.45\textwidth]{Figure27.pdf}
\\
\tiny Bishop PRML Figure 2.7
%\end{column}
\end{center}
%\begin{column}{8cm}
\begin{itemize}
\item \pause Covariance matrices are symmetric.
\item \pause Diagonal entries: variances along coordinate-axes
\item \pause Eigenvectors: principal axes of ellipsoid
\item Eigenvalues:  variances along eigen-vectors
\item Eigenvector with maximal/minimal eigen-value: Direction of maximal/minimal variance
\item \pause Covariance matrices are `positive definite', i.e. all their eigenvalues are non-negative.
\item \pause Most of this can be derived from $a^\top \mbox{Cov}(X) a= \mbox{Var}(a^\top X)$
\end{itemize}
%\end{column}
%\end{columns}

}

\section{A generative model: Class-conditional Gaussians}
\frame{\frametitle{A tale of two Gaussian: We can use a probablistic model of the data for classification}
\begin{center}
\includegraphics[width=.35\textwidth]{Figure410a.pdf}
\includegraphics[width=.35\textwidth]{Figure410b.pdf}
\end{center}

\begin{itemize}
\item Suppose that each of the two classes is modelled by a Gaussian: $x | x \in C_+ \sim \mathcal{N}\left(\mu_+, \Sigma_+\right)$, $x | x \in C_- \sim \mathcal{N}\left(\mu_-, \Sigma_-\right)$,
\item ~[On board] Calculation of posterior class probabilities and decision criterion
\item \pause If we assume $\Sigma_+=\Sigma_-$, we get $\omega_{gauss} \propto \Sigma_+^{-1} (\mathbf{m}_+-\mathbf{m}_-)$
\item Note: We take the $t_n$ as given and built a model of $x_n | t_n$, contrast with linear regression, where we took $x_n$ as given and modelled $t_n |x_n$.
\end{itemize}
\tiny Bishop PRML Figure 4.10
}

\frame{\frametitle{This approach directly generalizes to classification with unequal covariances and multi-class classification.}
\begin{center}
\includegraphics[width=.35\textwidth]{Figure411a.pdf}
\includegraphics[width=.35\textwidth]{Figure411b.pdf}
\end{center}
\begin{itemize}
\item Quadratic discriminant analysis: $\Sigma_+ \neq \Sigma_o$, decisison boundary is of form $y(\xx) = \xx^\top A \xx+\omega^\top \xx +\omega_o$
\item Multi-class: Assign each data-point to class with highest posterior probability (or calculate best assignment from cost-function).
\end{itemize}
\tiny Bishop PRML Figure 4.10
}

\frame{\frametitle{A simple nonlinear classifier can be constructed from kernel density estimates of the probability densities.}

\begin{columns}
\begin{column}{4cm}

\includegraphics[width=\textwidth]{NonLinClassification.pdf}
\end{column}
\begin{column}{7cm}
\begin{itemize}
\item Idea: Once we have an estimate of the class-conditional densities $P(x| t=\pm 1)$, we can construct a rule from $d(x)=P(x| t=+ 1)-P(x| t=- 1)$.
\item \pause Use \alert{kernel density estimation} to estimate  $P(x| t=\pm 1)$, i.e. place a `Gaussian bump' on each data-point:
\begin{align}
P(x| t=1) =\frac{1}{Z}\sum_{n_+} \exp\left( \frac{x-x_n}{\sigma}\right)^2
\end{align}
\end{itemize}
\end{column}
\end{columns}
\pause
This leads to a classifier of the form
\begin{align}
d(x)= \sum_{n} \alpha  t_n \exp\left( \frac{x-x_n}{\sigma}\right)^2
\end{align}
\pause
\alert{Support vector machine with radial basis functions} has decision rule \begin{align}
d(x)=\sum_{n}  \alpha_n \exp\left( \frac{x-x_n}{\sigma}\right)^2
\end{align}
}

\frame{\frametitle{Summary: One for the price of three.}
\begin{itemize}
\item Today, you learned about three different algorithms for binary classification with linear decision rules.
\item \pause One was based on a hack, the second one on a plausible (but ad-hoc) criterion, and the third one an a probababilistic model of the data.
\item \pause All three algorithms are equivalent.
\item \pause We showed that the Fisher discriminant and the probabilistic model based on two Gaussians have the same  decision criterion. In fact, it can be shown that linear regression has the same weights (Bishop 4.1.5)
%\item \pause The moral: Great motivations are great, but the actual algorithm matters, and it is important to check connections with other algorithms.
\item The third motivation had immediate extensions to nonlinear algorithms and multi-class classification, and posterior probabilities.
\item \pause Next week, we will learn an algorithm which actually is different, and usually better than the ones discussed today.
\end{itemize}
}

\end{document}