%%% -*- mode: LaTeX; TeX-master: "main.tex"; -*-

\ifx\master\undefined
\documentclass[a4paper]{easychair}
\usepackage{submission}
\begin{document}
{\let\master\relax\input{frontmatter}}
\fi
%%%% PLEASE DO NOT EDIT ABOVE THIS LINE

\section{\tlaplus and its Proof Language}
\label{sec:proof-language}

\subsection{TLA}
\label{sec:proof-language.tla}

The \tlaplus\ language is based on the Temporal Logic of Actions
(TLA)~\cite{lamport:newtla}, a linear-time temporal logic. The rigid
variables of TLA are called \emph{constants} and the flexible
variables are called simply \emph{variables}.  TLA assumes an
underlying ordinary (non-modal) logic for constructing expressions.
Operators of that logic are called \emph{constant} operators.  A
\emph{state function} is an expression built from constant operators
and TLA constants and variables.  The elementary (non-temporal)
formulas of TLA are \textit{actions}, which are formulas built with
constant operators, constants, variables, and expressions of the form
$f'$, where $f$ is a state function.  (TLA also has an \ENABLED
operator that is used in reasoning about liveness, but we ignore it
for brevity.)  An action is interpreted as a predicate on pairs of
states that describes a set of possible state transitions, where state
functions refer to the starting state and primed state functions refer
to the ending state.  Because priming distributes over constant
operators and because $c'$ is equal to $c$ for any constant $c$, an
action can be reduced to a formula built from constant operators,
constants, variables, and primed variables.

TLA is practical for describing systems because all the complexity of
a specification is in the action formulas.  Temporal operators are
essentially used only to assert liveness properties, including
fairness of system actions.  Most of the work in a TLA proof is in
proving action formulas; temporal reasoning occurs only in proving
liveness properties and is limited to propositional temporal logic and
to applying a handful of proof rules whose main premises are action
formulas.  Because temporal reasoning is such a small part of TLA
proofs, we have deferred its implementation.  The \PM\ now handles
only action formulas.  We have enough experience mechanizing TLA's
temporal reasoning~\cite{engberg:mechanical} to be fairly confident
that it will not be hard to extend the \PM to support it.

A formula built from constant operators, constants, variables, and
primed variables is valid iff it is a valid formula of the underlying
logic when constants, variables, and primed variables are treated as
distinct variables of the logic---that is, if $v$ and $v'$ are
considered to be two distinct variables of the underlying logic, for
any TLA variable $v$.  Since any action formula is reducible to such a
formula, action reasoning is immediately reducible to reasoning in the
underlying logic.  We therefore ignore variables and priming here and
consider only constant formulas.

\subsection{\tlaplus}

The \tlaplus\ language adds the following to the TLA logic:
\begin{icom}
\item An underlying logic that is essentially ZFC set theory plus
  classical untyped first-order logic with Hilbert's
  $\varepsilon$~\cite{leisenring:mathematical-logic}.
% LL: I deleted
%
%    which is written \CHOOSE\ in \tlaplus.
%
% to save a line and avoid unnecessarily introducing a name.
%
The major difference between this underlying
  logic and traditional ZFC is that functions are defined axiomatically
  rather than being represented as sets of ordered pairs.

\item A mechanism for defining operators, where a user-defined
operator is essentially a macro that is expanded syntactically.
(\tlaplus\ permits recursive function definitions, but they are
translated to ordinary definitions using Hilbert's $\varepsilon$.)

%% LL: changed ``A module language'' to ``Modules''
\item Modules, where one module can import definitions
  and theorems from other modules.  A module is parameterized by its
  declared variables and constants, and it may be instantiated in another
  module by substituting expressions for its parameters. The
  combination of substitution and the \ENABLED\ operator introduces
  some complications, but space limitations prevent us from discussing
  them, so we largely ignore modules in this paper.
\end{icom}
\tlaplus\ has been extensively documented~\cite{lamport03tla}.  Since
we are concerned only with reasoning about its underlying logic, which
is a very familiar one, we do not bother to describe \tlaplus\ in any
detail.  All of its nonstandard notation that appears in our examples is
explained.

\subsection{The Proof Language}
\label{sec:proof-language.lang}

The major new feature of \tlatwo\ is its proof language.  (For reasons
having nothing to do with proofs, \tlatwo\ also introduces recursive
operator definitions, which we ignore here for brevity.)  We describe
the basic proof language, omitting a few constructs
%%
% LL: I don't think we omitted any ``uninteresting'' constructs,
%     so I deleted:
%
%
%    for simplicity a few constructs that are either uninteresting or
%
%
that concern aspects such as module instantiation that we are not
discussing.  \tlatwo\ also adds constructs for naming subexpressions
of a definition or theorem, which is important in practice for writing
proofs but is orthogonal to the concerns of this paper.

The goal of the language is to make proofs easy to read and write for
someone with no knowledge of how the proofs are being checked.  This
leads to a mostly declarative language, built around the uses and
proofs of assertions rather than around the application of
proof-search tactics.  It is therefore more akin to
Isabelle/Isar~\cite{isar} than to more operational interactive
languages such as Coq's Vernacular~\cite{coq}.
Nevertheless, the proof language does include a few operational
constructs that can eliminate the repetition of common idioms, albeit
with some loss of perspicuity.

% at the cost of some loss of perspicuity.

At any point in a \tlaplus\ proof, there is a current obligation that
is to be proved.  The obligation contains a \emph{context} of
\emph{known} facts, definitions, and declarations, and a
\emph{goal}.
%%
%% SM: slight rewording of the following
%%
% which is a proposition that the obligation claims to be entailed by
% the context.
The obligation claims that the goal is logically entailed by the context.
Some of the facts and definitions in the
context are marked as \emph{usable} for reasoning, while the remaining
facts and definitions are \textit{hidden}.
%
%% LL: We'd better say more about this below, so it's redundant
%% here:
%
% and may not be used in the
% proof unless explicitly made usable with a \USE declaration (see
% below).

\ednote{SM}{I don't think we are quite consistent about what is an
  assertion. This presentation suggests that an assertion is not just
  an \ASSUME{} \PROOF{}, but also ``contains'' the context in which
  that \ASSUME{} \PROOF{} appears. I think that the term ``assertion''
  is used differently towards the end of this section, although the
  description here matches what we say in Section 3.}

Proofs are structured hierarchically.
%
%% LL: This is deferred until the first use of PROOF.
% and optionally begin with the token \PROOF.
%
% At the lowest levels of the hierarchy are the
% \textit{leaf proofs}.
The leaf (lowest-level) proof \OBVIOUS\ asserts that the
current goal follows easily from the usable facts and definitions.
The leaf proof
\begin{gather*}
  \BY\ e_{1},\ldots, e_{m} \ \DEFS\ o_{1},\ldots, o_{n}
\end{gather*}
asserts that the current goal follows easily from the usable facts and
definitions together with (i)~the facts $e_{i}$ that must themselves
follow easily from the context and (ii)~the known definitions of
$o_{j}$.  Whether a goal follows easily from definitions and facts
depends on who is trying to prove it.  The \PM\ generates proof
obligations for each leaf proof, so in practice ``follows easily''
means that a back-end prover can prove them.

A non-leaf proof is a sequence of \textit{steps}, each consisting of a
begin-step token and a proof construct.  For some constructs
(including a simple assertion of a proposition) the step takes a
subproof, which may be omitted.  The final step in the sequence simply
asserts the current goal, which is represented by the token \QED.
%
A begin-step token is either a \emph{level token} of the form \s{n} or
a \emph{label} of the form \s{n}"l", where $n$ is a level number that
is the same for all steps of this non-leaf proof, and "l" is an
arbitrary name.  The hierarchical structure is deduced from the level
numbers of the begin-step tokens, a higher level number beginning a
subproof.

%% LL: I changed
%   A step that makes declarations or definitions or that changes the form
%   of the current goal does not require a proof,
%   but one that makes an assertion is followed by its proof
%
% because (a) SUFFICES changes the goal and requires a proof and
% (b) PICK makes a definition and requires a proof
% (c) WITNESS e \in S makes an assertion but does not take a proof.
%  Also assertions whose proof is omitted are not followed by their proof--
%  at least not with the reader's current understanding of proof.
%
Some steps make declarations or definitions or change the current
goal and do not require a proof.  Other steps make assertions that
become the current goals for their proofs.  An omitted proof (or one
consisting of the token \OMITTED) is considered to be a leaf proof
that instructs the assertion to be accepted as true.  Of course, the
proof is then incomplete and cannot be certified.  Following a step
that makes an assertion (and the step's proof), until the end of the
current proof (after the \QED\ step), the contexts contain that
assertion in their sets of known facts.  The assertion is marked
usable iff the begin-step token is a level token; otherwise it can be
referred to by its label in a \BY\ proof.

The hierarchical structure of proofs not only aids in reading finished
proof but is also quite useful in incrementally writing proofs.  The
steps of a non-leaf proof are first written with all proofs but that
of the \QED\ step omitted.  After checking the proof of the \QED
step, the proofs omitted for other steps in this or earlier levels
are written in any order.  When writing the proof, one may discover
facts that are needed in the proofs of multiple steps.
%
% LL: I modified the following text because we say in conclusion
%     that adding lemmas is bad.
%
Such a fact is then added to the proof as an early step, or added at a
higher level.  It can also be removed from the proof of the theorem
and proved separately as a lemma.  However, the hierarchical proof
language encourages facts relevant only for a particular proof to be
kept within the proof, making the proof's structure easier to see and
simplifying maintenance of the proof.  For correctness proofs of
systems, the first few levels of the hierarchy are generally
determined by the structure of the formula to be proved---for example,
the proof that a formula implies a conjunction usually consists of steps
asserting that it implies each conjunct.

As an example, we incrementally construct a hierarchical proof of
Cantor's theorem, which states that there is no surjective function
from a set to its powerset. It is written in \tlaplus as:
%
\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] :
        \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A"
  \end{tabbing}
\end{quote}
%
%% SM: cosmetic change of the following that heeds Leslie's advice below
%% (I think it used to be more or less like that before, why did it
%% change?)
%%
%where "f[x]" is the application of the function "f" to "x",
where function application is written using square brackets,
"\SUBSET\ S" is the powerset of "S", and "[S -> T]" is the set of
functions from $S$ to $T$.
% \llnote{One should not write two mathematical operators like
% $x$ and $\SUBSET$ separated only by punctuation, since that is hard to
% read.  The sentence above needs to be rewritten.}

The statement of the theorem is the current goal for
its top-level proof. A goal of the form $\forall v:e$ is
proved by introducing a generic constant and proving the formula
obtained by
substituting it for the bound identifier. We express this as follows,
using the \ASSUME/\PROVE construct of \tlatwo:
% \llnote{\ASSUME/\PROVE should be introduced earlier, where we mention
% assertions---probably back where we introduce "||-".}

\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] :
                \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \\
%%    \PROOF \\
    \LSP \= \s11.\ \= \ASSUME \= "\NEW\ S", \\
         \>        \>         \> "\NEW\ f \in [S -> \SUBSET\ S]"\\
         \>        \> \PROVE "\exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \\
         \> \s12.  \> \QED \BY \s11
  \end{tabbing}
\end{quote}
%
%
Step \s11 asserts that for any constants "S" and "f" with "f \in [S
-> \SUBSET\ S]", the proposition to the right of the \PROVE is true.
More precisely, the current context for the proof of \s11 (which we
have yet to write) contains the declarations of $S$ and $f$ and the
usable fact "f \in [S -> \SUBSET\ S]", and the \PROVE\ assertion is
its goal.  The \QED step states that the original goal (the theorem)
follows from the assertion in step~\s11.

We tell the \PM to check this (incomplete) proof, which it does by
having the back-end provers verify the proof obligation for the \QED
step.  The verification succeeds, and we now continue by writing the
proof of \s11.  (Had the verification failed because \s11 did not
imply the current goal, we would have caught the error before
attempting to prove \s11, which we expect to be harder to do.)

\llnote{Check minor edit below.}

We optimistically start with the proof \OBVIOUS, but it is too hard
for the back-end to prove, and the \PM reports a timeout.  Often this
means that a necessary fact or definition in the context is hidden and
we merely have to make it usable with a \USE step or a \BY proof.  In
this case we have no such hidden assumptions, so we must refine the
goal into simpler goals with a non-leaf proof.  We let this proof have
level 2 (we can use any level greater than 1).  Since the goal itself
is existentially quantified, we must supply a witness.  In this case,
the witness is the classic diagonal set, which we call~"T".
%
\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] : \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
    \PROOF \kill
    \LSP \= \s11.\ \= \ASSUME \= "\NEW\ S", \\
         \>        \>         \> "\NEW\ f \in [S -> \SUBSET\ S]" \\
         \>        \> \PROVE "\exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \\
%         \>        \> \PROOF \\
         \>   \hspace{1em}     \= \s21.\ \= \DEFINE "T \DEF \{z \in S : z \notin f[z]\}" \\
         \>        \> \s22.  \> "\forall x \in S : f[x] \neq T" \\
         \>        \> \s23.  \> \QED \BY \s22
  \end{tabbing}
\end{quote}
Because definitions made within a proof are usable by default, the
definition of $T$ is usable in the proofs of \s22\ and \s23.  Once
again, the proof of the \QED\ step is automatically verified, so all
that remains is to prove \s22.  (The \DEFINE\ step requires no proof.)

% \llnote{I added to the paragraph above an explanation of why the
%   definition of $T$ is usable.  Stephan asks if it wouldn't it be more
%   consistent to give the definition a level token.  In a comment that
%   got deleted along the way, I proposed changing the language to make
%   labeled definitions unusable and allow the user to write $\BY\
%   \DEFS\ \s21$ as well as $\BY\ \DEFS\ T$.  However, we had already
%   decided that the user is most likely to want local definitions to be
%   \USE{}d by default, and many users might prefer to give step numbers
%   to all their steps.  So, we decided to defer a decision on this
%   point.  In the conclusion, I added this as the kind of fine tuning
%   we are likely to make to the language.}
%
% \ednote{SM}{Fine with me. There must be a way to hide definitions
%   because they can get in the way. Your suggestion to make labeled
%   definitions unusable looks clean and consistent to me, but in its
%   absence we can at least write HIDE DEF.}
%
% \ednote{KC}{In my opinion, the presence or absence of a label should
%   not be the deciding factor on whether the contents of the step, be
%   they definitions or facts, are usable or hidden. Instead, the label
%   should determine whether the contents of the step are namable by the
%   step token or not. Since definitions are not namable, it does not
%   matter whether we number "\DEFINE" steps or not. (Subexpressions of
%   the body of the definition are reachable via the defined operator
%   name.)}

%
% \llnote{This example suggests that we should modify the current language
% definition so that definitions in a labeled step are not automatically
% usable.  There's no reason why they need to be, and it would make it
% clear that the \QED\ step's proof doesn't depend on the definition.
% We could allow BY DEF \s21\ to include all the definitions made
% in the step.}

The system
%% LL: changed ``will accept'' to ``accepts'' because it does so right now.
accepts \OBVIOUS\ as the proof of \s22 because the only
difficulty in the proof of \s11 is finding the
witness. However, suppose we want to add another level of proof for the benefit
of a human reader.  The universal quantification is proved as above,
by introducing a fresh constant:

\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] : \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
    \PROOF \kill
    \LSP \= \s11.\ \= \ASSUME \= "\NEW\ S", \kill
         \>        \>         \> "\NEW\ f \in [S -> \SUBSET\ S]" \kill
         \>        \> \PROVE "\exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
         \>        \> \PROOF \kill
         \>   \hspace{1em} \= \s21.\ \= \DEFINE "T == \{z \in S : z \notin f[z]\}" \kill
         \>        \> \s22.  \> "\forall x \in S : f[x] \neq T" \\
         \>        \> \hspace{1em} \= \s31.\ \= \ASSUME "\NEW\ x \in S" \PROVE "f[x] \neq T" \\
         \>        \>        \> \s32.\ \> \QED \BY \s31
  \end{tabbing}
\end{quote}
%
% Naturally, the system verifies the proof of the \QED\ step.  (Remember
% that it could verify \s22\ by itself.)
%     the QED step has nothing to do with \s22  -- KC
\ednote{KC}{Should be able to trim the text for this last level below}
Naturally, the \QED step is verified.  Although the system accepts
\OBVIOUS\ as the proof of \s31 (remember that it could verify \s22\ by
itself), we can provide more detail with yet another level
of proof.  We write this proof the way it would seem natural to a
person---by breaking it into two cases:
%
\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] : \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
    \PROOF \kill
    \LSP \= \s11.\ \= \ASSUME \= "\NEW\ S", \kill
         \>        \>         \> "\NEW\ f \in [S -> \SUBSET\ S]" \kill
         \>        \> \PROVE "\exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
         \>        \> \PROOF \kill
         \> \hspace{1em} \= \s21.\ \= \DEFINE "T == \{z \in S : z \notin f[z]\}" \kill
         \>        \> \s22.  \> "\forall x \in S : f[x] \neq T" \kill
         \>        \>  \hspace{1em} \= \s31.\ \= \ASSUME "\NEW\ x \in S" \PROVE "f[x] \neq T" \\
         \>        \>        \> \hspace{1em} \= \s41.\ \= \CASE "x \in T" \\
         \>        \>        \>        \> \s42.\ \> \CASE "x \notin T" \\
         \>        \>        \>        \> \s43.\ \> \QED \BY \s41, \s42
  \end{tabbing}
\end{quote}
The (omitted) proof of the \CASE\ statement \s41\ has as its goal
"f[x]\neq T" and has the additional usable fact $x\in T$ in its context.

\llnote{Minor edit below.}

We continue refining the proof in this way, stopping
with an \OBVIOUS or \BY proof when a goal is
obvious enough for the back-end prover or for a human reader,
depending on who the proof is being written for.
A \BY\ statement can guide the prover or the human reader
by listing helpful obvious consequences of known facts.  For example,
the proof of \s41\ might be "\BY\ x \notin f[x]".
%
The complete proof appears in Appendix~\ref{apx:cantor}.

This example illustrates how the proof language supports a
hierarchical, non-linear, and incremental development of proofs.  The
proof writer can work on the most problematic unproved steps first,
leaving the easier ones for later.  Finding that a step cannot be
proved (for example, because it is invalid) may require changing other
steps, making proofs of those other steps wasted effort.  We intend to
provide an interface to the \PM\ that will make it easy for the user
to indicate which proofs should be checked and will avoid
unnecessarily rechecking proofs.

The example also shows how already-proved facts are generally not made
usable, but are invoked explicitly in \BY\ proofs.  Global definitions
are also hidden by default and the user must explicitly make them
usable.  This makes proofs easier to read by telling the reader what
facts and definitions are being used to prove each step.  It also
helps constrain the search space for an automated back-end prover,
leading to more efficient verification.  Facts and definitions can be
switched between usable and hidden by \USE\ and \HIDE\ steps, which
have the same syntax as \BY. As noted above, omitting the label from a
step's starting token (for example, writing \s4 instead of \s42) makes
the fact it asserts usable.  This might be done for compactness at
the lowest levels of a proof.

% The Isar proof language~\cite{isar} is similarly explicit about
% usable assumptions, but the facility is local to a proof and does
% not persist into sub-proofs, causing repetition in sibling
% sub-proofs. In the \tlatwo proof language, the visibility of an
% assumption persists until hidden, so one \USE declaration at a
% suitable level is enough. (The assumption can be selectively hidden
% at lower levels if needed for performance reasons.)

% \llnote{The preceding stuff about Isar needs to be moved to the conclusion.}

The example also indicates how the current proof obligation at every
step of the proof is clear, having been written explicitly in a parent
assertion.  This clear structure comes at the cost of introducing many
levels of proof, which can be inconvenient.  One way of avoiding these
extra levels is by using an assertion of the form "\SUFFICES\ A",
which asserts that proving $A$ proves the current goal, and makes $A$
the new current goal in subsequent steps.  In our example proof, one
level in the proof of step \s22\ can be eliminated by writing the
proof as:
\begin{quote} \small
  \begin{tabbing}
    \THEOREM\ "\forall S : \forall f \in [S -> \SUBSET\ S] : \exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
    \PROOF \kill
    \LSP \= \s11.\ \= \ASSUME \= "\NEW\ S", \kill
         \>        \>         \> "\NEW\ f \in [S -> \SUBSET\ S]" \kill
         \>        \> \PROVE "\exists A \in \SUBSET\ S : \forall x \in S : f[x] \neq A" \kill
         \>        \> \PROOF \kill
         \> \hspace{1em} \= \s21.\ \= \DEFINE "T == \{z \in S : z \notin f[z]\}" \kill
         \>        \> \s22.  \> "\forall x \in S : f[x] \neq T" \\
         \>        \>  \hspace{1em} \= \s31.\ \= \SUFFICES \ASSUME "\NEW\ x \in S" \PROVE "f[x] \neq T" \\
         \>        \>        \> \hspace{1em} \PROOF \OBVIOUS \\
         \>        \>        \> \s32.\ \> \CASE "x \in T" \\
         \>        \>        \> \s33.\ \> \CASE "x \notin T" \\
         \>        \>        \> \s34.\ \> \QED \BY \s32, \s33
  \end{tabbing}
\end{quote}
where the proofs of the \CASE\ steps are the same as before.  The
\SUFFICES\ statement changes the current goal of the level-3 proof to
$f[x]\neq T$ after adding a declaration of "x" and the usable fact "x
\in S" to the context. This way of proving a universally quantified
formula is sufficiently common that \tlatwo\ provides a \TAKE\
construct that allows the \SUFFICES\ assertion \s31 and its \OBVIOUS
proof to be written \mbox{\,$\TAKE\ x \in S$\,}.

There is a similar construct, "\WITNESS\ f \in S" for proving an
existentially quantified goal $\exists x\in S: e$, which changes the
goal to "e[x := f]".
% the := notation for capture-avoiding substitution is universal
% enough, especially in the community reading this paper
For implicational goals "e => f", the construct $\HAVE\ e$ changes the
goal to $f$.  No other constructs in the \tlatwo proof language change
the form of the current goal. We advise that these constructs be used
only at the lowest levels of the proof, since the new goal they create
must be derived %% SM: commented out "algebraically"
instead of being available textually in
a parent assertion.  (As a check and an aid to the reader, one can at
any point insert a redundant \SUFFICES\ step that simply asserts the
current goal.)

The final \tlatwo\ proof construct is $\PICK\ x : e$, which introduces
a new symbol $x$ that satisfies $e$.  The goal of the proof of this
\PICK step is $\exists x : e$, and it changes the context of
subsequent steps by adding a declaration of "x" and the fact "e".
%
A more formal summary of the language appears in
Appendix~\ref{apx}.

The semantics of a \tlatwo\ proof is independent of any back-end
prover. Different provers will have different notions of what
``follows easily'', so an \OBVIOUS\ proof may be verified by one
prover and not another.  In practice, many provers such as Isabelle
must be directed to use decision procedures or special tactics to
prove some assertions.  For this purpose, special standard modules
will contain dummy theorems for giving directives to the
\PM.  Using such a theorem (with a \USE\ step or \BY\ proof) will
% LL: What generated proof obligation?  And why filter something
%     rather than not putting it into proof obligations in the
%     first place?
cause the \PM\ not to use it as a fact,
but instead to generate special directives for back-end
provers.  It could even cause the \PM\ to use a different back-end
prover.  (The dummy theorem will assert a true fact that suggests the
purpose of the directive.)
For instance, using the theorem \emph{Arithmetic}
%% LL: Engineers have never heard of Peano Arithmetic, and we don't
%%     have so much space that we can squander it on such an example.
%%   \textit{PeanoArithmetic} might assert the existence of a set of
%%   natural numbers satisfying the axioms of Peano Arithmetic; a "\BY\
%%   \mathit{PeanoArithmetic}" proof
%
might be interpreted as an instruction to use a decision procedure for
integers.

\llnote{I replaced ``natural numbers'' by ``integers'' above, which seems
to me what arithmetic reasoning is more likely to mean.}

We hope that almost all uses of this feature will leave the \tlatwo
proof independent of the back-end prover(s).  The proof will not have
to be changed if the \PM\ is reconfigured to replace one decision
procedure with a different one.

%%%% PLEASE DO NOT EDIT BELOW THIS LINE
\ifx\master\undefined
{\let\master\relax\input{rearmatter}}
\end{document}
\fi


% Part: first-order-logic
% Chapter: sequent-calculus
% Section: quantifier-rules

\documentclass[../../../include/open-logic-section]{subfiles}

\begin{document}

\olfileid{fol}{seq}{qrl}

\olsection{Quantifier Rules}

\subsection{Rules for $\lforall$}

\begin{defish}
\Axiom$ !A(t), \Gamma \fCenter \Delta$
\RightLabel{\LeftR{\lforall}}
\UnaryInf$ \lforall[x][!A(x)],\Gamma \fCenter \Delta$
\DisplayProof
\hfill
\Axiom$ \Gamma \fCenter \Delta, !A(a) $
\RightLabel{\RightR{\lforall}}
\UnaryInf$ \Gamma \fCenter \Delta, \lforall[x][!A(x)]$
\DisplayProof
\end{defish}

In \LeftR{\lforall}, $t$ is a closed term (i.e., one without
variables). In \RightR{\lforall}, $a$~is !!a{constant} which must not
occur anywhere in the lower sequent of the \RightR{\lforall} rule. We
call $a$ the \emph{eigenvariable} of the \RightR{\forall}
inference.\footnote{We use the term ``eigenvariable'' even though $a$
in the above rule is !!a{constant}. This has historical reasons.}

\subsection{Rules for $\lexists$}

\begin{defish}
\Axiom$ !A(a), \Gamma \fCenter \Delta $
\RightLabel{\LeftR{\lexists}}
\UnaryInf$ \lexists[x][!A(x)], \Gamma \fCenter \Delta$
\DisplayProof
\hfill
\Axiom$ \Gamma \fCenter \Delta, !A(t) $
\RightLabel{\RightR{\lexists}}
\UnaryInf$ \Gamma \fCenter \Delta, \lexists[x][!A(x)]$
\DisplayProof
\end{defish}

Again, $t$~is a closed term, and $a$~is !!a{constant} which does not
occur in the lower sequent of the \LeftR{\lexists} rule. We call $a$
the \emph{eigenvariable} of the \LeftR{\lexists} inference.

The condition that an eigenvariable not occur in the lower sequent of
the \RightR{\lforall} or \LeftR{\lexists} inference is called the
\emph{eigenvariable condition}.

\begin{explain}
Recall the convention that when $!A$ is !!a{formula} with the
!!{variable}~$x$ free, we indicate this by writing~$!A(x)$. In the
same context, $!A(t)$ then is short for~$\Subst{!A}{t}{x}$. So we
could also write the \RightR{\lexists} rule as:
\begin{prooftree}
  \Axiom$\Gamma \fCenter \Delta, \Subst{!A}{t}{x}$
  \RightLabel{\RightR{\lexists}}
  \UnaryInf$\Gamma \fCenter \Delta, \lexists[x][!A]$
\end{prooftree}
Note that $t$ may already occur in~$!A$, e.g., $!A$~might
be~$\Atom{\Obj P}{t,x}$. Thus, inferring $\Gamma \Sequent \Delta,
\lexists[x][\Atom{\Obj P}{t,x}]$ from~$\Gamma \Sequent \Delta,
\Atom{\Obj P}{t,t}$ is a correct application
of~\RightR{\lexists}---you may ``replace'' one or more, and not
necessarily all, occurrences of~$t$ in the premise by the bound
!!{variable}~$x$. However, the eigenvariable conditions in
\RightR{\lforall} and~\LeftR{\lexists} require that the
!!{constant}~$a$ does not occur in~$!A$. So, you cannot correctly
infer $\Gamma \Sequent \Delta, \lforall[x][\Atom{\Obj P}{a,x}]$ from
$\Gamma \Sequent \Delta, \Atom{\Obj P}{a,a}$ using~$\RightR{\lforall}$.
\end{explain}

\begin{explain}
In \RightR{\lexists} and \LeftR{\lforall} there are no restrictions on
the term~$t$. On the other hand, in the \LeftR{\lexists} and
\RightR{\lforall} rules, the eigenvariable condition requires that the
!!{constant}~$a$ does not occur anywhere outside of~$!A(a)$ in the
upper sequent. It is necessary to ensure that the system is sound,
i.e., only !!{derive}s sequents that are valid. Without this
condition, the following would be allowed:
\begin{prooftree}
  \Axiom$!A(a) \fCenter !A(a)$
  \RightLabel{*\LeftR{\lexists}}
  \UnaryInf$\lexists[x][!A(x)] \fCenter !A(a)$
  \RightLabel{\RightR{\lforall}}
  \UnaryInf$\lexists[x][!A(x)] \fCenter \lforall[x][!A(x)]$
  \DisplayProof\bottomAlignProof
  \qquad
  \Axiom$!A(a) \fCenter !A(a)$
  \RightLabel{*\RightR{\lforall}}
  \UnaryInf$!A(a) \fCenter \lforall[x][!A(x)]$
  \RightLabel{\LeftR{\lexists}}
  \UnaryInf$\lexists[x][!A(x)] \fCenter \lforall[x][!A(x)]$
\end{prooftree}
However, $\lexists[x][!A(x)] \Sequent \lforall[x][!A(x)]$ is not valid.
\end{explain}

\end{document}


\section{DP on Graphs}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Minimum vertex cover on trees}
  \begin{exampleblock}{Minimum vertex cover on trees \pno{2.2.18}}
    \begin{itemize}
	  \item Undirected tree $T = (V, E)$; \textcolor{red}{No designated root!}
      \item Compute (the size of) a minimum vertex cover of $T$
    \end{itemize}
  \end{exampleblock}

  \fignocaption{width = 0.30\textwidth}{figs/vertex-cover.png}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Minimum vertex cover on trees}
  \centerline{Rooted $T$ at any node $r$.}
  \pause
  \vspace{0.30cm}

  \begin{description}
	\item[Subproblem:] $I(u)$: the size of an MVC of subtree $T_{u}$ rooted at $u$
	\item[Goal:] $I(r)$
	  \pause
	\item[Make choice:] Is $u$ in $\text{MVC}[u]$?
	\item[Recurrence:]
	  \begin{align*}
		I(u) = \min \{\text{\# children of } u &+ \sum_{v: \text{ grandchildren of } u} I(v), \\
			1 &+ \sum_{v: \text{ children of } u} I(v)\}
	  \end{align*}
	  \pause
	\item[Init:] $I(u) = 0$, if $u$ is a leave
  \end{description}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Minimum vertex cover on trees}
  \begin{columns}
	\column{0.35\textwidth}
	  DFS on $T$ from root $r$:

	  \vspace{0.50cm}
	  \begin{algorithmic}
		\State when $u$ is ``finished'':
		\If{$u$ is a leave}
		  \State $I(u) \gets 0$
		\Else
		  \State $I(u) \gets \dots$
		\EndIf
	  \end{algorithmic}
	  \pause
	\column{0.60\textwidth}
	Greedy algorithm (\textcolor{red}{Rough Proof!}):

	  \vspace{0.50cm}
	  \begin{theorem}
		There is an MVC which contains no leaves.
	  \end{theorem}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{DP on DAG}
  \begin{exampleblock}{Longest path in DAG (Problem 7.17)}
	\begin{itemize}
	  \item Direction: $\downarrow$ OR $\rightarrow$
	  \item Score: $>=< 0$
	\end{itemize}
  \end{exampleblock}

  \pause
  \begin{enumerate}
	\item digraph $G$
	\item node weight $\to$ edge weight
	\item adding an extra sink $s$
	\item $G \to G^{T}$
  \end{enumerate}

  \pause
  \centerline{Compute a longest path from $s$ in DAG}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{DP on DAG}
  \begin{description}
	\item[Subproblem:] $\text{dist}[v]$: longest distance from $s$ to $v$
	\item[Goal:] $\text{dist}[v], \forall v \in V$
	  \pause
	\item[Make choice:] What is the previous node before $v$ on the longest path?
	\item[Recurrence:]
	  \[
		\text{dist}[v] = \max_{u \to v} \left(\text{dist}[u] + w(u \to v)\right)
	  \]
	  \pause
	\item[Init:] $\text{dist}[s] = 0$
  \end{description}

  \pause
  \vspace{0.60cm}
  \centerline{Compute $\text{dist}[v]$ in topo. order}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Bitonic tour}
  \begin{exampleblock}{Bitonic tour (Problem 7.18)}
	\begin{itemize}
	  \item Points: $P[1 \dots n], \; p_i = (x_i, y_i)$
	  \item $x_1 < x_2 < \dots < x_n$
	  \item Bitonic tour: $p_1 \leadsto^{x_i < x_{i+1}} p_n \leadsto^{x_i > x_{i+1}} p_1$
	  \item Compute a shorest bitonic tour.
	\end{itemize}
  \end{exampleblock}

  \fignocaption{width = 0.30\textwidth}{figs/bitonic-tour-wiki.png}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Bitonic tour}
  \centerline{$P_{i,j} (i \le j)$: bitonic path $p_i \leadsto^{x_i > x_{i+1}} p_1 \leadsto^{x_i < x_{i+1}} p_j$ includes all $p_1, p_2, \dots, p_j$}

  \vspace{0.20cm}
  \begin{description}
	\item[Subproblem:] $d[i,j]$: the length of a shortest bitonic path $P_{i,j}$
	\item[Goal:] $d[n,n] = d[n-1, n] + l(p_{n-1}p_{n})$
	  \pause
	\item[Make choice:] Is $p_{j-1}$ on the increasing path or the decreasing path?
	\item[Recurrence:]
	  \begin{align*}
		d[i,j] &=  d[i,j-1] + l(p_{j-1}p_j) \quad \forall i < j-1 \\
		d[i,j] &= \min_{1 \le k < j-1} \set{d[k, j-1] + l(p_kp_j)} \quad \forall i = j-1
	  \end{align*}
	  \pause
	\item[Init:] $d[1,2] = l(p_1p_2)$
	\item[Time:]
	  \[
		O(n^2) = O(n \log n) + O(n^2) \cdot O(1) + O(n) \cdot O(n)
	  \]
  \end{description}
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\chapter{Applications I:  Pure Binary Arithmetic}\label{arithchapter}

This chapter presents relations for arithmetic over the non-negative
integers: addition, subtraction, multiplication, division,
exponentiation, and logarithm.  Importantly, these relations are
refutationally complete---if an individual arithmetic relation is
called with arguments that do not satisfy the relation, the relation
will fail in finite time rather than diverge. The conjunction of two
or more arithmetic relations may not fail finitely, however. This is
because the conjunction of arithmetic relations can express
Diophantine equations; were such conjunctions guaranteed to terminate,
we would be able to solve Hilbert's 10$^{th}$ problem, which is
undecidable~\cite{hilbertstenth}.  We also do not guarantee
termination if the goal's arguments share variables, since sharing can
express the conjunction of sharing-free relations.

\citet{conf/flops/KiselyovBFS08} gives proofs of refutational
completeness for these relations.  \citet{trs} and
\citet{conf/flops/KiselyovBFS08} give additional examples and
exposition of these arithmetic relations\footnote{The definition of
  \scheme|logo| in the first printing of \citet{trs} contains an
  error, which has been corrected in the second printing and in
  section~\ref{arithexplog}.}.

This chapter is organized as follows.  Section~\ref{arithrep}
describes our representation of numbers.  In section~\ref{arithnaive}
we present a naive implementation of addition and show its
limitations.  Section~\ref{arithrevisited} presents a more
sophisticated implementation of addition, inspired by the half-adders
and full-adders of digital hardware.  Sections~\ref{arithmult} and
\ref{arithdivision} present the multiplication and division relations,
respectively.  Finally in section~\ref{arithexplog} we define
relations for logarithm and exponentiation.

\section{Representation of Numbers}\label{arithrep}

Before we can write our arithmetic relations, we must decide how we
will represent numbers.  For simplicity, we restrict the domain of our
arithmetic relations to non-negative integers\footnote{We could extend
  our treatment to negative integers by adding a sign tag to each
  number.}.  We might be tempted to use Scheme's built-in numbers for
our arithmetic relations.  Unfortunately, unification cannot decompose
Scheme numbers.  Instead, we need an inductively defined
representation of numbers that can be constructed and deconstructed
using unification.  We will therefore represent numbers as lists.

The simplest approach would be to use a unary
representation\footnote{Even when using unary numbers, defining
  refutationally complete arithmetic relations is non-trivial, as
  demonstrated by~\citet{conf/flops/KiselyovBFS08}.}; however, for
efficiency we will represent numbers as lists of binary digits.  Our
lists of binary digits are \emph{little-endian}: the \scheme|car| of
the list contains the least-significant-bit, which is convenient when
performing arithmetic.  We can define the \scheme|build-num| helper
function, which constructs binary little-endian lists from Scheme
numbers.

\schemedisplayspace
\begin{schemedisplay}
(define build-num
  (lambda (n)
    (cond
      ((zero? n) '())
      ((and (not (zero? n)) (even? n))
       (cons 0 (build-num (quotient n 2))))
      ((odd? n)
       (cons 1 (build-num (quotient (- n 1) 2)))))))
\end{schemedisplay}

\noindent For example \mbox{\scheme|(build-num 6)|} returns
\mbox{\scheme|`(0 1 1)|}, while \mbox{\scheme|(build-num 19)|} returns
\mbox{\scheme|`(1 1 0 0 1)|}.

To ensure there is a unique representation of every number, we suppress
trailing $0$'s. Thus \mbox{\scheme|`(0 1)|} is the unique
representation of the number two; both \mbox{\scheme|`(0 1 0)|} and
\mbox{\scheme|`(0 1 0 0)|} are illegal.  Similarly, \scheme|'()| is
the unique representation of zero; \mbox{\scheme|`(0)|} is illegal.
Lists representing numbers may be partially instantiated:
\mbox{\scheme|`(1 . ,x)|} represents any odd integer, while
\mbox{\scheme|`(0 . ,y)|} represents any \emph{positive} even number.
We must ensure that our relations never instantiate variables
representing numbers to illegal values---in these examples, \scheme|x|
can be instantiated to any legal number, while \scheme|y| can be
instantiated to any number \emph{other} than zero to avoid creating
the illegal value \mbox{\scheme|`(0)|}.

We can now define the simplest useful arithmetic relations,
\scheme|poso| and \scheme|>1o|.  The \scheme|poso| relation is
satisfied if its argument represents a positive
integer.

\schemedisplayspace
\begin{schemedisplay}
(define poso
  (lambda-e (n)
    (`((,a . ,d)))))
\end{schemedisplay}

\noindent The \scheme|>1o| relation is satisfied if its argument
represents an integer greater than one.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(define >1o
  (lambda-e (n)
    (`((,a ,b . ,d)))))
\end{schemedisplay}

\noindent We will use \scheme|poso| and \scheme|>1o| in more
sophisticated arithmetic relations, starting with addition.

\section{Naive Addition}\label{arithnaive}

Now that we have decided on a representation for numbers, we can
define the addition relation, \scheme|pluso|.

\schemedisplayspace
\begin{schemedisplay}
(define pluso
  (lambda-e (n m s)
    (`(,x () ,x))
    (`(() ,y ,y))
    (`((0 . ,x) (,b . ,y) (,b . ,res))
     (pluso x y res))
    (`((,b . ,x) (0 . ,y) (,b . ,res))
     (pluso x y res))
    (`((1 . ,x) (1 . ,y) (0 . ,res))
     (exist (res-1)
       (pluso x y res-1)
       (pluso '(1) res-1 res)))))
\end{schemedisplay}

\noindent The first two clauses handle when \scheme|n| or \scheme|m|
is zero.  The next two clauses handle when both \scheme|n| and
\scheme|m| are positive integers, at least one of which is even.  The
final clause handles when \scheme|n| and \scheme|m| are both positive
odd integers.

At first glance, our definition of \scheme|pluso| seems to work fine.

\wspace

\noindent\scheme|(run1 (q) (pluso '(1 1) '(0 1 1) q))| $\Rightarrow$ \scheme|`((1 0 0 1))|

\wspace

\noindent As expected, adding three and six yields nine.  However,
replacing \scheme|run1| with \scheme|run*| results in the answer
\mbox{\scheme|((1 0 0 1) (1 0 0 1))|}.  The duplicate value is due to
the overlapping of clauses in \scheme|pluso|---for example, both of
the first two clauses succeed when \scheme|n|, \scheme|m|, and
\scheme|s| are all zero.  Even worse, \mbox{\scheme|(run* (q) (pluso '(0 1) q '(1 0 1)))|}
 returns \mbox{\scheme|`((1 1) (1 1) (1 1 0) (1 1 0))|}. The
last two values are not even legal representations of a number, since
the most-significant bit is zero.

We can fix these problems by making the clauses of \scheme|pluso|
non-overlapping, and by adding calls to \scheme|poso| to ensure the
most-significant bit of a positive number is never instantiated to
zero.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(define pluso
  (lambda-e (n m k)
    (`(,x () ,x))
    (`(() (,x . ,y) (,x . ,y)))
    (`((0 . ,x) (0 . ,y) (0 . ,res)) (poso x) (poso y)
     (pluso x y res))
    (`((0 . ,x) (1 . ,y) (1 . ,res)) (poso x)
     (pluso x y res))
    (`((1 . ,x) (0 . ,y) (1 . ,res)) (poso y)
     (pluso x y res))
    (`((1 . ,x) (1 . ,y) (0 . ,res))
     (exist (res-1)
       (pluso x y res-1)
       (pluso '(1) res-1 res)))))
\end{schemedisplay}

\enlargethispage{1em}

\noindent We separated the third clause of the original \scheme|pluso|
into two clauses, so we can use \scheme|poso| to avoid illegal
instantiations of numbers.

The improved definition of \scheme|pluso| no longer produces duplicate
or illegal values.

\wspace

\noindent\scheme|(run* (q) (pluso '(1 1) '(0 1 1) q))| $\Rightarrow$ \scheme|`((1 0 0 1))|

\noindent\scheme|(run* (q) (pluso '(0 1) q '(1 0 1)))| $\Rightarrow$ \scheme|`((1 1))|

\wspace

It may appear that our new \scheme|pluso| is refutationally complete,
since attempting to add eight to some number \scheme|q| to produce six
fails finitely:

\wspace

\noindent\scheme|(run* (q) (pluso '(0 0 0 1) q '(0 1 1)))| $\Rightarrow$ \scheme|'()|

\wspace

\noindent Unfortunately, this example is misleading---\scheme|pluso|
is not refutationally complete.  The expression
\mbox{\scheme|(run1 (q) (pluso q '(1 0 1) '(0 0 0 1)))|}
returns \mbox{\scheme|'((1 1))|} as expected, but replacing
\scheme|run1| with \scheme|run2| results in divergence.  Similarly,

\schemedisplayspace
\begin{schemedisplay}
(run6 (q)
  (exist (x y)
    (pluso x y '(1 0 1))
    (== `(,x ,y) q)))
\end{schemedisplay}

\noindent returns

\schemedisplayspace
\begin{schemeresponse}
(((1 0 1) ())
 (() (1 0 1))
 ((0 0 1) (1))
 ((1) (0 0 1))
 ((0 1) (1 1))
 ((1 1) (0 1)))
\end{schemeresponse}

\noindent but \scheme|run7| diverges. If we were to swap the recursive
calls in last clause of \scheme|pluso|, the previous expressions would
converge when using \scheme|run*|; unfortunately, many previously
convergent expressions would then diverge\footnote{These examples
  demonstrate why an efficient implementation (or simulation) of
  commutative conjunction would be useful.}.  If we want
\scheme|pluso| to be refutationally complete, we must reconsider our
approach.

\section{Arithmetic Revisited}\label{arithrevisited}

In this section we develop a refutationally complete definition of
\scheme|pluso|, inspired by the half-adders and full-adders of digital
logic\footnote{See \citet{hennessy-computer} for a description of
  hardware adders.}.

We first define \scheme|half-addero|, which, when given the binary
digits \scheme|x|, \scheme|y|, \scheme|r|, and \scheme|c|, satisfies
the equation $x + y = r + 2 \cdot c$.

\schemedisplayspace
\begin{schemedisplay}
(define half-addero
  (lambda (x y r c)
    (exist ()
      (bit-xoro x y r)
      (bit-ando x y c))))
\end{schemedisplay}

\scheme|half-addero| is defined using bit-wise relations for logical
{\tt and} and {\tt exclusive-or}.

\schemedisplayspace
\begin{schemedisplay}
(define bit-ando
  (lambda-e (x y r)
    (`(0 0 0))
    (`(1 0 0))
    (`(0 1 0))
    (`(1 1 1))))
\end{schemedisplay}

\begin{schemedisplay}
(define bit-xoro
  (lambda-e (x y r)
    (`(0 0 0))
    (`(0 1 1))
    (`(1 0 1))
    (`(1 1 0))))
\end{schemedisplay}

Now that we have defined \scheme|half-addero|, we can define
\scheme|full-addero|.  \scheme|full-addero| is similar to
\scheme|half-addero|, but takes a carry-in bit \scheme|b|; given bits
\scheme|b|, \scheme|x|, \scheme|y|, \scheme|r|, and \scheme|c|,
\scheme|full-addero| satisfies $b + x + y = r + 2 \cdot c$.

\schemedisplayspace
\begin{schemedisplay}
(define full-addero
  (lambda (b x y r c)
    (exist (w xy wz)
      (half-addero x y w xy)
      (half-addero w b r wz)
      (bit-xoro xy wz c))))
\end{schemedisplay}

\scheme|half-addero| and \scheme|full-addero| add individual bits.  We
now define \scheme|addero| in terms of \scheme|full-addero|;
\scheme|addero| adds a carry-in bit \scheme|d| to arbitrarily large
numbers \scheme|n| and \scheme|m| to produce a number \scheme|r|.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(define addero
  (lambda (d n m r)
    (match-e `(,d ,n ,m)
      (`(0 __ ()) (== n r))
      (`(0 () __) (== m r) (poso m))
      (`(1 __ ())
       (addero 0 n '(1) r))
      (`(1 () __)
       (poso m)
       (addero 0 '(1) m r))
      (`(__ (1) (1))
       (exist (a c)
         (== `(,a ,c) r)
         (full-addero d 1 1 a c)))
      (`(__ (1) __)
       (gen-addero d n m r))
      (`(__ __ (1))
       (>1o n) (>1o r)
       (addero d '(1) n r))
      (`(__ __ __)
       (>1o n)
       (gen-addero d n m r)))))
\end{schemedisplay}

The last clause of \scheme|addero| calls \scheme|gen-addero|; given
the bit \scheme|d| and numbers \scheme|n|, \scheme|m|, and \scheme|r|,
\scheme|gen-addero| satisfies $d + n + m = r$, provided that
\scheme|m| and \scheme|r| are greater than one and \scheme|n| is
positive.

\schemedisplayspace
\begin{schemedisplay}
(define gen-addero
  (lambda (d n m r)
    (match-e `(,n ,m ,r)
      (`((,a . ,x) (,b . ,y) (,c . ,z))
       (exist (e)
         (poso y) (poso z)
         (full-addero d a b c e)
         (addero e x y z))))))
\end{schemedisplay}

We are finally ready to redefine \scheme|pluso|.

\schemedisplayspace
\begin{schemedisplay}
(define pluso (lambda (n m k) (addero 0 n m k)))
\end{schemedisplay}

\noindent As proved by~\citet{conf/flops/KiselyovBFS08}, this
definition of \scheme|pluso| is refutationally complete.  Using the
new \scheme|pluso| all the addition examples from the previous section
terminate, even when using \scheme|run*|.  We can also generate
triples of numbers, where the sum of the first two numbers equals the
third.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(run9 (q)
  (exist (x y r)
    (pluso x y r)
    (== `(,x ,y ,r) q))) $\Rightarrow$
\end{schemedisplay}
\nspace
\begin{schemeresponse}
((_.0 () _.0)
 (() (_.0 . _.1) (_.0 . _.1))
 ((1) (1) (0 1))
 ((1) (0 _.0 . _.1) (1 _.0 . _.1))
 ((1) (1 1) (0 0 1))
 ((0 _.0 . _.1) (1) (1 _.0 . _.1))
 ((1) (1 0 _.0 . _.1) (0 1 _.0 . _.1))
 ((0 1) (0 1) (0 0 1))
 ((1) (1 1 1) (0 0 0 1)))
\end{schemeresponse}
% \begin{schemeresponse}
% ((_.0 () _.0)
%  (() (_.0 . _.1) (_.0 . _.1))
%  ((1) (1) (0 1))
%  ((1) (0 _.0 . _.1) (1 _.0 . _.1))
%  ((1) (1 1) (0 0 1))
%  ((0 _.0 . _.1) (1) (1 _.0 . _.1))
%  ((1) (1 0 _.0 . _.1) (0 1 _.0 . _.1))
%  ((0 1) (0 1) (0 0 1))
%  ((1) (1 1 1) (0 0 0 1))
%  ((1 1) (1) (0 0 1))
%  ((1) (1 1 0 _.0 . _.1) (0 0 1 _.0 . _.1))
%  ((1 1) (0 1) (1 0 1))
%  ((1) (1 1 1 1) (0 0 0 0 1))
%  ((1 0 _.0 . _.1) (1) (0 1 _.0 . _.1))
%  ((1) (1 1 1 0 _.0 . _.1) (0 0 0 1 _.0 . _.1)))
% \end{schemeresponse}

We can take advantage of the flexibility of the relational approach by
defining subtraction in terms of addition.

\schemedisplayspace
\begin{schemedisplay}
(define minuso (lambda (n m k) (pluso m k n)))
\end{schemedisplay}

\noindent \scheme|minuso| works as expected:

\wspace

\noindent\scheme|(run* (q) (minuso '(0 0 0 1) '(1 0 1) q))| $\Rightarrow$ \scheme|`((1 1))|

\wspace

\noindent eight minus five is indeed three. \scheme|minuso| is also
refutationally complete:

\wspace

\noindent\scheme|(run* (q) (minuso '(0 1 1) q '(0 0 0 1)))| $\Rightarrow$ \scheme|`()|

\wspace

\noindent there is no non-negative integer \scheme|q| that, when
subtracted from six, produces eight.

\section{Multiplication}\label{arithmult}
\enlargethispage{2em}

Next we define the multiplication relation \scheme|mulo|, which
satisfies $n \cdot m = p$.

\schemedisplayspace
\begin{schemedisplay}
(define mulo
  (lambda (n m p)
    (match-e `(,n ,m)
      (`(() __) (== () p))
      (`(__ ()) (== () p) (poso n))
      (`((1) __) (== m p) (poso m))
      (`(__ (1)) (== n p) (>1o n))
      (`((0 . ,x) __)
       (exist (z)
         (== `(0 . ,z) p)
         (poso x) (poso z) (>1o m)
         (mulo x m z)))
      (`((1 . ,x) (0 . ,y))
       (poso x) (poso y)
       (mulo m n p))
      (`((1 . ,x) (1 . ,y))
       (poso x) (poso y)
       (odd-mulo x n m p)))))
\end{schemedisplay}

\noindent \scheme|mulo| is defined in terms of the helper relation
\scheme|odd-mulo|.

\schemedisplayspace
\begin{schemedisplay}
(define odd-mulo
  (lambda (x n m p)
    (exist (q)
      (bound-mulo q p n m)
      (mulo x m q)
      (pluso `(0 . ,q) m p))))
\end{schemedisplay}

\noindent For detailed descriptions of \scheme|mulo| and
\scheme|odd-mulo|, see~\cite{trs} and \cite{conf/flops/KiselyovBFS08}.
From a refutational-completeness perspective, the definition of
\scheme|bound-mulo| is most interesting.

\scheme|bound-mulo| ensures that the product of \scheme|n| and
\scheme|m| is no larger than \scheme|p| by enforcing that the
length\footnote{More correctly, the length of the list representing
  the number.}  of \scheme|n| plus the length of \scheme|m| is an
upper bound for the length of \scheme|p|.  In the process of enforcing
this bound, \scheme|bound-mulo| length-instantiates \scheme|q|---that
is, \scheme|q| becomes a list of fixed length containing
uninstantiated variables representing binary digits.  The length of
\scheme|q|, written $\norm{q}$, satisfies $\norm{q} <
\min(\norm{p},\penalty\binoppenalty \norm{n} + \norm{m} + 1)$.

\schemedisplayspace
\begin{schemedisplay}
(define bound-mulo
  (lambda (q p n m)
    (match-e `(,q ,p)
      (`(() (__ . __)))
      (`((__ . ,x) (__ . ,y))
       (exist (a z)
         (conde
           ((== '() n)
            (== `(,a . ,z) m)
            (bound-mulo x y z '()))
           ((== `(,a . ,z) n)
            (bound-mulo x y z m))))))))
\end{schemedisplay}

\scheme|mulo| works as expected:

\wspace

\noindent\scheme|(run* (p) (mulo '(1 0 1) '(1 1) p))| $\Rightarrow$ \scheme|`(1 1 1 1)|

\wspace

\noindent multiplying five by three yields fifteen. Thanks to the
bounds on term sizes enforced by \scheme|bound-mulo|, \scheme|mulo| is
refutationally complete:

\wspace

\noindent\scheme|(run* (q) (mulo '(0 1) q '(1 1)))| $\Rightarrow$ \scheme|`()|

\wspace

\noindent there exists no non-negative integer \scheme|q| that, when
multiplied by two, yields three.

As we expect of all our relations, \scheme|mulo| is flexible---it can
even be used to factor numbers.  For example, this \scheme|run*|
expression returns all the factors of twelve.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(run* (q)
  (exist (m)
    (mulo q m '(0 0 1 1)))) $\Rightarrow$
\end{schemedisplay}
\nspace
\begin{schemeresponse}
((1) (0 0 1 1) (0 1) (0 0 1) (1 1) (0 1 1))
\end{schemeresponse}

% From FLOPS paper.
%
% We guarantee termination only for stand-alone base arithmetic goals but
% not their conjunctions (see \S\ref{s:solution-set}). This
% non-compositionality is expected, since conjunctions of arithmetic
% goals can express Diophantine equations; were such conjunctions
% guaranteed to terminate, we would be able to solve Hilbert's 10th
% problem, which is undecidable~\cite{hilbertstenth}.  We also do not
% guarantee termination if the goal's arguments share variables.  Such a
% goal can be expressed by conjoining a sharing-free base goal and
% equalities.

% [TODO discuss Presberger Arithmetic versus Peano Arithmetic; Hilbert's
% 10th problem; Diophantine equations; conjunction of arithmetic
% relations, perhaps by sharing variables.

% Should be able to lift wording for this.]

\section{Division}\label{arithdivision}

Next we define a relation that performs division with remainder.  We
will need additional bounds on term sizes to define division (and
logarithm in section~\ref{arithexplog}).

The relation \scheme|=lo| ensures that the lists representing the numbers
\scheme|n| and \scheme|m| are the same length.  As before, we must
take care to avoid instantiating either number to an illegal value
like \scheme|`(0)|.

\schemedisplayspace
\begin{schemedisplay}
(define =lo
  (lambda-e (n m)
    (`(() ()))
    (`((1) (1)))
    (`((,a . ,x) (,b . ,y)) (poso x) (poso y)
     (=lo x y))))
\end{schemedisplay}

\scheme|<lo| ensures that the length of the list representing \scheme|n| is
less than that of \scheme|m|.

\schemedisplayspace
\begin{schemedisplay}
(define <lo
  (lambda-e (n m)
    (`(() __) (poso m))
    (`((1) __) (>1o m))
    (`((,a . ,x) (,b . ,y)) (poso x) (poso y)
     (<lo x y))))
\end{schemedisplay}

We can now define \scheme|<=lo| by combining \scheme|=lo| and \scheme|<lo|.

\schemedisplayspace
\begin{schemedisplay}
(define <=lo
  (lambda (n m)
    (conde
      ((=lo n m))
      ((<lo n m)))))
\end{schemedisplay}

Using \scheme|<lo| and \scheme|=lo| we can define \scheme|<o|, which
ensures that the value of \scheme|n| is less than that of \scheme|m|.

\schemedisplayspace
\begin{schemedisplay}
(define <o
  (lambda (n m)
    (conde
      ((<lo n m))
      ((=lo n m)
       (exist (x)
         (poso x)
         (pluso n x m))))))
\end{schemedisplay}

Combining \scheme|<o| and \scheme|==| leads to the definition of
\scheme|<=o|.

\schemedisplayspace
\begin{schemedisplay}
(define <=o
  (lambda (n m)
    (conde
      ((== n m))
      ((<o n m)))))
\end{schemedisplay}

With the bounds relations in place, we can define division with
remainder.  The \scheme|divo| relation takes numbers \scheme|n|,
\scheme|m|, \scheme|q|, and \scheme|r|, and satisfies $n = m \cdot q +
r$, with $0 \leq r < m$; this is equivalent to the equation $\frac{n}{m} = q$
with remainder $r$, with $0 \leq r < m$.
A simple definition of \scheme|divo| is

\schemedisplayspace
\begin{schemedisplay}
(define divo
  (lambda (n m q r)
    (exist (mq)
      (<o r m)
      (<=lo mq n)
      (mulo m q mq)
      (pluso mq r n))))
\end{schemedisplay}

\noindent Unfortunately,
\mbox{\scheme|(run* (m) (exist (r) (divo '(1 0 1) m '(1 1 1) r)))|}
diverges. Because we want refutational completeness,
we instead use the more sophisticated definition

\schemedisplayspace
\begin{schemedisplay}
(define divo
  (lambda (n m q r)
    (match-e q
      (`() (== r n) (<o n m))
      (`(1) (=lo n m) (pluso r m n) (<o r m))
      (__ (<lo m n) (<o r m) (poso q)
       (exist (nh nl qh ql qlm qlmr rr rh)
         (splito n r nl nh)
         (splito q r ql qh)
         (conde
           ((== '() nh)
            (== '() qh)
            (minuso nl r qlm)
            (mulo ql m qlm))
           ((poso nh)
            (mulo ql m qlm)
            (pluso qlm r qlmr)
            (minuso qlmr nl rr)
            (splito rr r '() rh)
            (divo nh m qh rh))))))))
\end{schemedisplay}

\noindent The refutational completeness of \scheme|divo| is largely due to the
use of \scheme|<o|, \scheme|<lo|, and \scheme|=lo| to establish bounds
on term sizes. \scheme|divo| is described in detail in~\citet{trs}.

\scheme|divo| relies on the relation \scheme|splito| to `split' a
binary numeral at a given length: \mbox{\scheme|(splito n r l h)|}
holds if $n = 2^{s+1} \cdot l + h$ where $s = \norm{r}$ and $h < 2^{s+1}$.
\scheme|splito| can construct \scheme|n| by combining the lower-order
bits\footnote{The lowest bit of a positive number \mbox{\scheme|n|} is
  the car of \mbox{\scheme|n|}.} of \scheme|l| with
the higher-order bits of \scheme|h|, inserting \emph{padding} bits as
specified by the length of \scheme|r|---\scheme|splito| is essentially
a specialized version of \scheme|appendo|.  \scheme|splito| ensures
that illegal values like \scheme|'(0)| are not constructed by removing
the rightmost zeros after splitting the number \scheme|n| into its
lower-order bits and its higher-order bits.

\schemedisplayspace
\begin{schemedisplay}
(define splito
  (lambda-e (n r l h)
    (`(() __ () ()))
    (`((0 ,b . ,n^) () () (,b . ,n^)))
    (`((1 . ,n^) () (1) ,n^))
    (`((0 ,b . ,n^) (,a . ,r^) () __)
     (splito `(,b . ,n^) r^ '() h))
    (`((1 . ,n^) (,a . ,r^) (1) __)
     (splito n^ r^ '() h))
    (`((,b . ,n^) (,a . ,r^) (,b . ,l^) __)
     (poso l^)
     (splito n^ r^ l^ h))))
\end{schemedisplay}

% Second, the definition of the \scheme|splito| helper
% relation is so complicated because it must avoid instantiating numbers
% to illegal values.

% \scheme|splito|'s definition is so complicated
% because \scheme|splito| must not allow the list \scheme|'(0)| to
% represent a number.  For example, \mbox{\scheme|(splito '(0 0 1) '()
%   '() '(0 1))|} should succeed, but \mbox{\scheme|(splito '(0 0 1) '()
%   '(0) '(0 1))|} should fail.

% \scheme|l| contains the lowest bits of \scheme|n| (if any), while \scheme|h|
% contains \scheme|n|'s highest bits.

% The \scheme|splito| relation is similar to the ternary
% \scheme|appendo| relation: appending \scheme|l|

% $\norm{r} - \norm{l} + 1$ padding bits

%  and \scheme|h| yields \scheme|n|, where \scheme|l|,
% \scheme|h|, and \scheme|n| represent non-negative integers using our
% little-endian list scheme.  \scheme|l| contains the lowest bits of
% \scheme|n|, while \scheme|h| contains \scheme|n|'s highest bits.

% The call \mbox{\scheme|(splito n '() l h)|} \emph{moves} the lowest
% bit\footnote{The lowest bit of a positive number \mbox{\scheme|n|} is
%   the \mbox{\scheme|car|} of \mbox{\scheme|n|}.} of \scheme|n|, if
% any, into \scheme|l|, and moves the remaining bits of \scheme|n| into
% \scheme|h|; \mbox{\scheme|(splito n '(1) l h)|} moves the two lowest
% bits of \scheme|n| into \scheme|l| and moves the remaining bits of
% \scheme|n| into \scheme|h|; and \mbox{\scheme|(splito n '(1 1 1 1) l h)|},
% \mbox{\scheme|(splito n '(0 1 1 1) l h)|}, or
% \mbox{\scheme|(splito n '(0 0 0 1) l h)|} move the five lowest bits of
% \scheme|n| into \scheme|l| and move the remaining bits into
% \scheme|h|; and so on.

% Since \scheme|splito| is a relation, it can construct \scheme|n| by
% combining the lower-order bits of \scheme|l| with the higher-order
% bits of \scheme|h|, inserting \emph{padding} bits as specified by the
% length of \scheme|r|.

% \scheme|splito|'s definition is so complicated
% because \scheme|splito| must not allow the list \scheme|'(0)| to
% represent a number.  For example,
% \mbox{\scheme|(splito '(0 0 1) '() '() '(0 1))|} should succeed, but
% \mbox{\scheme|(splito '(0 0 1) '() '(0) '(0 1))|} should fail.
% \scheme|splito| ensures that illegal values like \scheme|'(0)| are not
% constructed by removing the rightmost zeros after splitting the number
% \scheme|n| into its lower-order bits and its higher-order bits.

\section{Logarithm and Exponentiation}\label{arithexplog}

We end this chapter by defining relations for logarithm with remainder
and exponentiation.

\newpage

%\schemedisplayspace
\begin{schemedisplay}
(define logo
  (lambda-e (n b q r)
    (`((1) __ () ()) (poso b))
    (`(__ __ () __) (<o n b) (pluso r '(1) n))
    (`(__ __ (1) __) (>1o b) (=lo n b) (pluso r b n))
    (`(__ (1) __ __) (poso q) (pluso r '(1) n))
    (`(__ () __ __) (poso q) (== r n))
    (`((,a ,b^ . ,dd) (0 1) __ __) (poso dd)
     (exp2o n '() q)
     (exist (s) (splito n dd r s)))
    (`(__ __ __ __)
     (exist (a b^ add ddd)
       (conde
         ((== '(1 1) b))
         ((== `(,a ,b^ ,add . ,ddd) b))))
     (<lo b n)
     (exist (bw1 bw nw nw1 ql1 ql s)
       (exp2o b '() bw1)
       (pluso bw1 '(1) bw)
       (<lo q n)
       (exist (q^ bwq1)
         (pluso q '(1) q^)
         (mulo bw q^ bwq1)
         (<o nw1 bwq1))
       (exp2o n '() nw1)
       (pluso nw1 '(1) nw)
       (divo nw bw ql1 s)
       (pluso ql '(1) ql1)
       (<=lo ql q)
       (exist (bql qh s qdh qd)
         (repeated-mulo b ql bql)
         (divo nw bw1 qh s)
         (pluso ql qdh qh)
         (pluso ql qd q)
         (<=o qd qdh)
         (exist (bqd bq1 bq)
           (repeated-mulo b qd bqd)
           (mulo bql bqd bq)
           (mulo b bq bq1)
           (pluso bq r n)
           (<o n bq1)))))))
\end{schemedisplay}

Given numbers \scheme|n|, \scheme|b|, \scheme|q|, and \scheme|r|,
\scheme|logo| satisfies $n = b^q + r$, where $0 \leq n$ and where $q$
is the largest number that satisfies the equation.  The \scheme|logo|
definition is similar to \scheme|divo|, but uses exponentiation rather
than multiplication\footnote{A line-by-line description of the Prolog
version of \scheme|logo| and its helper relations can be found at
\url{http://okmij.org/ftp/Prolog/Arithm/pure-bin-arithm.prl}}.

%% From Oleg:
% When b = 2, exponentiation and discrete logarithm are easier to obtain
% n = 2^q + r, 0<= 2*r < n
% Here, we just relate n and q.
%    exp2 n b q
% holds if: n = (|b|+1)^q + r, q is the largest such number, and
% (|b|+1) is a power of two.
%
% Side condition: (|b|+1) is a power of two and b is L-instantiated.
% To obtain the binary exp/log relation, invoke the relation as
%  (exp2 n '() q)
% Properties: if n is L-instantiated, one solution, q is fully instantiated.
% If q is fully instantiated: one solution, n is L-instantiated.
% In any event, q is always fully instantiated in any solution
% and n is L-instantiated.
% We depend on the properties of split.

\scheme|logo| relies on helpers \scheme|exp2o| and
\scheme|repeated-mulo|.  \scheme|exp2o| is a simplified version of
exponentiation; given our binary representation of numbers,
exponentiation using base two is particularly simple.
\mbox{\scheme|(exp2o n '() q)|} satisfies $n = 2^q$; the more general
\mbox{\scheme|(exp2o n b q)|} satisfies $n = (\norm{b}+1)^q + r$ for
some \scheme|r|, where \scheme|q| is the largest such number and $0
\leq 2 \cdot r < n$, provided that \scheme|b| is length-instantiated
and $\norm{b}+1$ is a power of two.

\schemedisplayspace
\begin{schemedisplay}
(define exp2o
  (lambda (n b q)
    (match-e `(,n ,q)
      (`((1) ()))
      (`(__ (1))
       (>1o n)
       (exist (s)
         (splito n b s '(1))))
      (`(__ (0 . ,q^))
       (exist (b^)
         (poso q^)
         (<lo b n)
         (appendo b `(1 . ,b) b^)
         (exp2o n b^ q^)))
      (`(__ (1 . ,q^))
       (exist (nh b^ s)
         (poso q^)
         (poso nh)
         (splito n b s nh)
         (appendo b `(1 . ,b) b^)
         (exp2o nh b^ q^))))))
\end{schemedisplay}

%% From Oleg:
% nq = n^q where n is L-instantiated and q is fully instantiated

\mbox{\scheme|(repeated-mulo n q nq)|} satisfies $nq = n^q$ provided
\scheme|n| is length-instantiated and \scheme|q| is fully
instantiated.

\schemedisplayspace
\begin{schemedisplay}
(define repeated-mulo
  (lambda (n q nq)
    (match-e q
      (`() (== (1) nq) (poso n))
      (`(1) (== n nq))
      (__
        (>1o q)
        (exist (q^ nq1)
          (pluso q^ '(1) q)
          (repeated-mulo n q^ nq1)
          (mulo nq1 n nq))))))
\end{schemedisplay}

This simple \scheme|logo| example shows that $14 = 2^3 + 6$.

\wspace

\noindent\scheme|(run* (q) (logo '(0 1 1 1) '(0 1) '(1 1) q))| $\Rightarrow$ \scheme|`(0 1 1)|

\wspace

A more sophisticated example of \scheme|logo| is

\schemedisplayspace
\begin{schemedisplay}
(run9 (s)
  (exist (b q r)
    (logo '(0 0 1 0 0 0 1) b q r)
    (>1o q)
    (== `(,b ,q ,r) s))) $\Rightarrow$
\end{schemedisplay}
\nspace
\begin{schemeresponse}
((() (_.0 _.1 . _.2) (0 0 1 0 0 0 1))
 ((1) (_.0  _.1 . _.2) (1 1 0 0 0 0 1))
 ((0 1) (0 1 1) (0 0 1))
 ((1 1) (1 1) (1 0 0 1 0 1))
 ((0 0 1) (1 1) (0 0 1))
 ((0 0 0 1) (0 1) (0 0 1))
 ((1 0 1) (0 1) (1 1 0 1 0 1))
 ((0 1 1) (0 1) (0 0 0 0 0 1))
 ((1 1 1) (0 1) (1 1 0 0 1))),
\end{schemeresponse}

\noindent which shows that:

\noindent$68 = 0^n + 68$ where $n$ is greater than one,

\nspace

\noindent$68 = 1^n + 67$ where $n$ is greater than one,

\nspace

\noindent$68 = 2^6 + 4$,

\nspace

\noindent$68 = 3^3 + 59$,

\nspace

\noindent$68 = 4^3 + 4$,

\nspace

\noindent$68 = 8^2 + 4$,

\nspace

\noindent$68 = 5^2 + 43$,

\nspace

\noindent$68 = 6^2 + 32$, and

\nspace

\noindent$68 = 7^2 + 19$.

We can define the exponentiation relation in terms of \scheme|logo|.

\schemedisplayspace
\begin{schemedisplay}
(define expo (lambda (b q n) (logo n b q '())))
\end{schemedisplay}

\noindent We can use \scheme|expo| to show that three to the fifth
power is $243$:

\wspace

\noindent \scheme|(run* (q) (expo '(1 1) '(1 0 1) q))| $\Rightarrow$ \scheme|`(1 1 0 0 1 1 1 1)|.

\wspace

The code in this chapter demonstrates the difficulty of achieving
refutational completeness, even for relatively simple relations.
Bounding the sizes of terms is a very powerful technique for ensuring
termination, but can be tricky to apply.  The definitions in this
chapter were derived from equations defining arithmetic operators, and
from the design of hardware half-adders and full-adders.  It would
have been extremely difficult to write this code from first
principles.


\lab{Facial Recognition}{Facial Recognition}
\label{lab:FacialRecognition}

\objective{
Facial recognition algorithms attempt to match a person's portrait to a database of many portraits.
Facial recognition is becoming increasingly important in security, law enforcement, artificial intelligence, and other areas.
Though humans can easily match pictures to people, computers are beginning to surpass humans at facial recognition.
In this lab, we implement a basic facial recognition system that relies on eigenvectors and the SVD to efficiently determine the difference between faces.
}

\section*{Preparing an Image Database} % ======================================

The \texttt{faces94} face image dataset\footnote{See \url{http://cswww.essex.ac.uk/mv/allfaces/faces94.html}.} contains several photographs of 153 people, organized into folders by person.
To perform facial recognition on this dataset, select one image per person and convert these images into a database.
For this particular facial recognition algorithm, the entire database can be stored in just a few NumPy arrays.

Digital images are stored on computers as arrays of pixels.
Therefore, an $m \times n$ image can be stored in memory as an $m\times n$ matrix or, equivalently, as an $mn$-vector by concatenating the rows of the matrix.
Then a collection of $k$ images can be stored as a single $mn\times k$ matrix $F$, where each column of $F$ represents a single image.
That is, if
\[
F = \left[\begin{array}{c|c|c|c}
\arrayrulecolor{lightgray}
& & & \\
\f_1 & \f_2 & \cdots & \f_k
\\
& & &
\end{array}\right],
\]
then each $\f_i$ is a $mn$-vector representing a single image.

The following function obtains one image for each person in the \texttt{faces94} dataset and converts the collection of images into an $mn \times k$ matrix $F$ described above.

\begin{lstlisting}
import os
import numpy as np
from imageio import imread

def get_faces(path="./faces94"):
    # Traverse the directory and get one image per subdirectory.
    faces = []
    for (dirpath, dirnames, filenames) in os.walk(path):
        for fname in filenames:
            if fname[-3:]=="jpg":       # Only get jpg images.
                # Load the image, convert it to grayscale,
                # and flatten it into a vector.
                faces.append(np.ravel(imread(dirpath+"/"+fname, as_gray=True)))
                break
    # Put all the face vectors column-wise into a matrix.
    return np.transpose(faces)
\end{lstlisting}

\begin{problem} % show_face() function.
Write a function that accepts an image as a flattened $mn$-vector, along with its original dimensions $m$ and $n$.
Use \li{np.reshape()} to convert the flattened image into its original $m\times n$ shape and display the result with \li{plt.imshow()}.
\\ (Hint: use \li{cmap="gray"} in \li{plt.imshow()} to display images in grayscale.)

Unzip the \texttt{faces94.zip} archive and use \li{get_faces()} to construct $F$.
Each \texttt{faces94} image is $200 \times 180$, and there are $153$ people in the dataset, so $F$ should be $36000 \times 153$.
Use your function to display one of the images stored in $F$.
\label{prob:visualize-faces}
\end{problem}

\section*{The Eigenfaces Method} % ============================================

With the image database $F$, we could construct a simple facial recognition system with the following strategy.
Let $\g$ be an $mn$-vector representing an unknown face that is not part of the database $F$.
Then the $\f_i$ that minimizes $\|\g - \f_i\|_2$ is the matching face.
Unfortunately, computing $\|\g - \f_i\|_2$ for each $i$ is very computationally expensive, especially if the images are high-resolution and/or the database contains a large number of images.
The \emph{eigenfaces method} is a way to reduce the computational cost of finding the closest matching face by focusing on only the most important features of each face.
Because the method ignores less significant facial features, it is also usually more accurate than the na\"ive method.

The first step of the algorithm is to shift the images by the \emph{mean face}.
Shifting a set of data by the mean exaggerates the distinguishing features of each entry.
In the context of facial recognition, shifting by the mean accentuates the unique features of each face.
For the images vectors stored in $F$, the mean face $\boldsymbol{\mu}$ is the defined to be the element-wise average of the $\f_i$:
\[
\boldsymbol{\mu} = \frac{1}{k}\sum_{i=1}^k \f_i.
\]
Hence, the $i$th mean-shifted face vector $\bar{\f}_i$ is given by
\[
\bar{\f}_i = \f_i - \boldsymbol{\mu}.
\]
Next, define $\bar{F}$ as the $mn \times k$ matrix whose columns are given by the mean-shifted face vectors,
\[
\bar{F} = \left[\begin{array}{c|c|c|c}
\arrayrulecolor{lightgray}
& & & \\
\bar{\f}_1 & \bar{\f}_2 & \cdots & \bar{\f}_k
\\
& & &
\end{array}\right].
\]

\begin{figure}[H] % Mean face + shifted faces.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/meanFace.png}
    \caption{The mean face.}
    \label{fig:facerec-meanface}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/originalFace0.png}
    \caption{An original face.}
    \label{fig:facerec-original}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/differenceFace0.png}
    \caption{A mean-shifted face.}
    \label{fig:facerec-shifted}
\end{subfigure}
\caption{}
\end{figure}

\begin{problem} % Shift by the mean.
\label{prob:meanFace}
Write a class called \li{FacialRec} whose constructor accepts a path to a directory of images.
In the constructor, use \li{get_faces()} to construct $F$, then compute the mean face $\boldsymbol\mu$ and the shifted faces $\bar{F}$.
Store each array as an attribute.
\\(Hint: Both $\boldsymbol\mu$ and $\bar{F}$ can be computed in a single line of code by using NumPy functions and/or array broadcasting.)

Use your function from Problem \ref{prob:visualize-faces} to visualize the mean face, and compare it to Figure \ref{fig:facerec-meanface}.
Also display an original face and its corresponding mean-shifted face.
Compare your results with Figures \ref{fig:facerec-original} and \ref{fig:facerec-shifted}.
\end{problem}

To increase computational efficiency and minimize storage, the face vectors can be represented with fewer values by projecting $\bar{F}$ onto a lower-dimensional subspace.
Let $s$ be a natural number such that $s < r$, where $r$ is the rank of $\bar{F}$.
By projecting $\bar{F}$ onto an $s$-dimensional subspace, each face can be stored with only $s$ values.

Specifically, let $U \Sigma V\hrm$ be the compact SVD of $\bar{F}$ with rank $r$, which can also be represented by
 \[
\bar{F}= \sum_{i=1}^r \sigma_i\u_i\v_i\hrm.
\]
The first $r$ columns of $U$ form a basis for the range of $\bar{F}$.
Recall that the Schmidt, Mirsky, Eckart-Young Theorem states that the matrix
\[
\bar{F}_s = \sum_{i=1}^s \sigma_i\u_i\v_i\hrm
\]
is the best rank-$s$ approximation of $\bar{F}$ for each $s < r$.
This means that $\| \bar{F} - \bar{F_s} \|$ is minimized against all other $\| \bar{F} - B \|$ where $B$ has rank $s$.
As a consequence of this theorem, the first $s$ columns of $U$ form a basis that provides the ``best'' $s$-dimensional subspace for approximating $\bar{F}$.

The $s$ basis vectors $\u_1, \ldots, \u_s$ are are commonly called the \emph{eigenfaces} because they are eigenvectors of $\bar{F}\bar{F}\trp$ and because they resemble face images.
Each original face image can be efficiently represented in terms of these eigenfaces.
See Figure \ref{fig:facerec-eigenfaces} for visualizations of the some of the eigenfaces for the \texttt{facesd94} data set.

\begin{figure}[H] % Eigenfaces.
\centering
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/eigenface0.png}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/eigenface49.png}
\end{subfigure}
%
\begin{subfigure}{.32\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/eigenface99.png}
\end{subfigure}
\caption{The first, 50th, and 100th eigenfaces.}
\label{fig:facerec-eigenfaces}
\end{figure}

In general, the lower eigenfaces provide a more general information of a face and higher-ordered eigenfaces provide the details necessary to distinguish particular faces \cite{muller2004singular}.
These eigenfaces will be used to construct the face images in the dataset.
The more eigenfaces used, the more detailed the resulting image will be.

Next, let $U_s$ be the matrix with the first $s$ eigenfaces as columns.
Since the eigenfaces $\{\u_i\}_{i=1}^s$ form an orthonormal set, $U_s$ is an orthonormal matrix (independent of $s$) and hence $U_s\trp U_s = I$.
The matrix $P_s = U_sU_s\trp$ projects vectors in $\mathbb{R}^{mn}$ to the subspace spanned by the orthonormal basis $\{\u_i\}_{i=1}^s$, and the change of basis matrix $U_s\trp$ puts the projection in terms of the basis of eigenfaces.
Thus the projection $\widehat{\f}_i$ of $\bar{\f}_i$ in terms of the basis of eigenfaces is given by
%
\begin{equation}
\widehat{\f}_i = U_s\trp P_s\bar{\f}_i = U_s\trp U_s U_s\trp \bar{\f}_i = U_s\trp\bar{\f}_i.
\label{eq:facialrec-projection}
\end{equation}
%
Note carefully that though the shifted image $\bar{\f_i}$ has $mn$ entries, the projection $\widehat{\f}_i$ has only $s$ entries since $U_s$ is $mn\times s$.
Likewise, the matrix $\widehat{F}$ that has the projections $\widehat{\f}_i$ as columns is $s \times k$, and
\begin{equation}
\widehat{F} = U_s\trp F.
\label{eq:facialrec-projectall}
\end{equation}

\begin{problem} % Compute U, implement project().
In the constructor of \li{FacialRec}, calculate the compact SVD of $\bar{F}$ and save the matrix $U$ as an attribute.
Compare the computed eigenfaces (the columns of $U$) to Figure \ref{fig:facerec-eigenfaces}.

Also write a method that accepts a vector of length $mn$ or an $mn\times \ell$ matrix, as well as an integer $s < mn$.
Construct $U_s$ by taking the first $s$ columns of $U$, then use (\ref{eq:facialrec-projection}) or (\ref{eq:facialrec-projectall}) to calculate the projection of the input vector or matrix onto the span of the first $s$ eigenfaces.
\\(Hint: this method should be implemented with a single line of code.)
\label{prob:facialrec-project}
\end{problem}

Reducing the mean-shifted face image $\bar{\f}_i$ to the lower-dimensional projection $\widehat{\f}_i$ drastically reduces the computational cost of the facial recognition algorithm, but this efficiency gain comes at a price.
A projection image only approximates the corresponding original image, but as long as $s$ isn't too small, the approximation is usually good enough for the algorithm to work well.
Before completing the facial recognition system, we reconstruct some of these projections to visualize the amount of information lost.

From (\ref{eq:facialrec-projection}), since $U_s\trp$ projects $\bar{\f}_i$ and performs a change of basis to get $\widehat{\f}_i$, its transpose $U_s$ puts $\widehat{\f}_i$ back into the original basis with as little error as possible.
That is,
\[
U_s \widehat{\f}_i \approx \bar{\f}_i = \f_i - \boldsymbol{\mu},
\]
so that we have the approximation
\begin{equation}
\widetilde{\f}_i = U_s \widehat{\f}_i + \boldsymbol{\mu} \approx \f_i.
\label{eq:facialrec-reconstruct}
\end{equation}
This $\widetilde{\f}_i$ is called the \emph{reconstruction} of $\f_i$.

\begin{figure}[H]
\begin{subfigure}{0.32\textwidth}
    \includegraphics[width=\textwidth]{figures/rebuiltThirtySecond.png}
    \caption{A reconstruction with $s=5$.}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
    \includegraphics[width=\textwidth]{figures/rebuiltEighth.png}
    \caption{A reconstruction with $s=19$.}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
    \includegraphics[width=\textwidth]{figures/rebuiltHalf.png}
    \caption{A reconstruction with $s=75$.}
\end{subfigure}
\caption{An image rebuilt with various numbers of eigenfaces. The image is already recognizable when it is reconstructed with only 19 eigenfaces, less than an eighth of the 153 eigenfaces corresponding to nonzero eigenvalues or $\bar{F}\bar{F}\trp$.
Note the similarities between this method and regular image compression via the truncated SVD.}
\label{fig:rebuiltImage}
\end{figure}

\begin{problem}
Instantiate a \li{FacialRec} object that draws from the \texttt{faces94} dataset.
Select one of the shifted images $\bar{\f}_i$.
For at least 4 values of $s$, use your method from Problem \ref{prob:facialrec-project} to compute the corresponding $s$-projection $\widehat{\f}_i$, then use (\ref{eq:facialrec-reconstruct}) to compute the reconstruction $\widetilde{\f}_i$.
Display the various reconstructions and the original image.
Compare your results to Figure \ref{fig:rebuiltImage}
\end{problem}

\section*{Matching Faces} % ===================================================

Let $\g$ be a vector representing an unknown face that is not part of the database.
We determine which image in the database is most like $\g$ by comparing $\widehat{\g}$ to each of the $\widehat{\f}_i$.
First, shift $\g$ by the mean to obtain $\bar{\g}$, then project $\bar{\g}$ using a given number of eigenfaces:
\begin{equation}
    \widehat{\g} = U_s\trp\bar{\g} = U_s\trp(\g - \boldsymbol{\mu})
    \label{eq:facialrec-unknownface}
\end{equation}

Next, determine which $\widehat{\f}_i$ is closest to $\widehat{\g}$.
Since the columns of $U_s$ are an orthonormal basis, the computation in this basis yields the same result as computing in the standard Euclidean basis would.
Then setting
\begin{equation}
j = \mathop{{\text{argmin}}\vphantom{\sim}}\limits_{\displaystyle _{i}} \|\widehat{\f}_i - \widehat{\g}\|_2, % This makes i below the argmin
\label{eq:facialrec-findnearest}
\end{equation}
we have that the $j$th face image $\f_j$ is the best match for $\g$.
Again, since $\widehat{\f}_i$ and $\widehat{\g}_i$ only have $s$ entries, the computation in (\ref{eq:facialrec-findnearest}) is much cheaper than comparing the raw $\f_i$ to $\g$.

\begin{problem} % Find nearest.
Write a method for the \li{FacialRec} class that accepts an image vector $\g$ and an integer $s$.
Use your method from Problem \ref{prob:facialrec-project} to compute $\widehat{F}$ and $\widehat{\g}$ for the given $s$, then use (\ref{eq:facialrec-findnearest}) to determine the best matching face in the database.
Return the index of the matching face.
\\(Hint: \li{scipy.linalg.norm()} and \li{np.argmin()} may be useful.)
\label{prob:facialrec-nearest}
\end{problem}

\begin{info}
This facial recognition system works by solving a \emph{nearest neighbor search}, since the goal is to find the $\f_i$ that is ``nearest'' to the input image $\g$.
Nearest neighbor searches can be performed more efficiently with the use of a \emph{$k$-d tree}, a binary search tree for storing vectors.
The system could also be called a \emph{$k$-neighbors classifier} with $k=1$.
% In this context, a $k$-neighbors classifier searches for the $k$ faces that are closest to $\g$ and use these ``neighboring'' faces to decide how to classify $\g$.
\end{info}

\begin{problem}
Write a method for the \li{FacialRec} class that accepts an flat image vector $\g$, an integer $s$, and the original dimensions of $\g$.
Use your method from Problem \ref{prob:facialrec-nearest} to find the index $j$ of the best matching face, then display the original face $\g$ alongside the best match $\f_j$.

The following generator yields random faces from \texttt{faces94} that can be used as test cases.
%
\begin{lstlisting}
def sample_faces(num_faces, path="./faces94"):
    # Get the list of possible images.
    files = []
    for (dirpath, dirnames, filenames) in os.walk(path):
        for fname in filenames:
            if fname[-3:]=="jpg":       # Only get jpg images.
                files.append(dirpath+"/"+fname)

    # Get a subset of the image names and yield the images one at a time.
    test_files = np.random.choice(files, num_faces, replace=False)
    for fname in test_files:
        yield np.ravel(imread(fname, as_gray=True))
\end{lstlisting}
%
The \li{yield} keyword is like a \li{return} statement, but the next time the generator is called, it will resume immediately after the last \li{yield} statement.%
\footnote{See the Python Essentials lab on Profiling for more on generators.}

Use \li{sample_faces()} to get at least 5 random faces from \texttt{faces94}, and match each random face to the database with $s=38$.
Iterate through the random faces with the following syntax.
%
\begin{lstlisting}
for test_image in sample_faces(5):
    # 'test_image' is a now flattened face vector.
\end{lstlisting}
\label{prob:facialrec-match}
\end{problem}

Although there are other approaches to facial recognition that utilize more complex techniques, the method of eigenfaces remains a wonderfully simple and effective solution.

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Improvements on the Facial Recognition System with Eigenfaces} % -

The \li{FacialRec} class does its job well, but it could be improved in several ways.
Here are a few ideas.
\begin{itemize}
    \item The most computationally intensive part of the algorithm is computing $\widehat{F}$.
    Instead of recomputing $\widehat{F}$ every time the method from Problem \ref{prob:facialrec-nearest} is called, store $\widehat{F}$ and $s$ as attributes the first time the method is called.
    In subsequent calls, only recompute $\widehat{F}$ if the user specifies a different value for $s$.

    \item Load a \li{scipy.spatial.KDTree} object with $\widehat{F}$ and use its \li{query()} method to compute (\ref{eq:facialrec-findnearest}).
    Building a $k$d-tree is expensive, so be sure to only build a new tree when necessary (i.e., the user specifies a new value for $s$).

    \item Include an error tolerance $\epsilon$ in the method for Problem \ref{prob:facialrec-nearest}.
    If $\|\f_j - \g\| > \epsilon$, print a message or raise an exception to indicate that there is no suitable match for $\g$ in the database.
    In this case, add $\g$ to the database for future reference.

    \item Generalize the system by turning it into a $k$-neighbors classifier.
    In the constructor, add several faces per person to the database (this requires modifying \li{get_faces()}).
    Assign each individual a unique ID so that the system knows which faces correspond to the same person.
    Modify the method from Problem \ref{prob:facialrec-nearest} so that it also accepts an integer $k$, then use \li{scipy.spatial.KDTree} to find the $k$ nearest images to $\g$.
    Choose the ID that belongs to the most nearest neighbors, then return an index that corresponds to an individual with that ID.

    In other words, choose the $k$ faces $\f_i$ that give the smallest values of $\|\f_i - \widehat{\g}\|_2$.
    These faces then get to vote on which person $\g$ belongs to.

    \item Improve the user interface of the class by modifying the method from Problem \ref{prob:facialrec-match} so that it accepts a file name to read from instead of an array.
    A few lines of code from \li{get_faces()} or \li{sample_faces()} might be helpful for this.
\end{itemize}

\subsection*{Other Methods for Facial Recognition} % --------------------------

The method of facial recognition presented here is more formally called \emph{principal component analysis (PCA) using eigenfaces}.
Several other machine learning and optimization techniques, such as linear discriminant analysis (LDA), elastic matching, dynamic link matching, and hidden Markov models (HMMs) have also been applied to the facial recognition problem.
Other techniques focus on getting better information about the faces in the first place, the most prevalent being 3-dimensional recognition and thermal imaging.
See \url{https://en.wikipedia.org/wiki/Facial_recognition_system} for a good survey of different approaches to the facial recognition problem.

\begin{comment}
% This proof is not needed, the Schmidt-Eckart-Young-Mirsky Theorem is sufficient to show that the first s columns of U in SVD provides the best basis for the s-dimensional subspace.
\subsection*{The Proof: SVD as a Least Squares Solution} % --------------------

\begin{theorem}
Let $\f_1, \ldots, \f_k$ be vectors in $\mathbb{R}^{mn}$, and let $\bar{F} = [\bar{f}_1 \; \ldots \; \bar{f}_k]$. Suppose $U\Sigma V\trp$ is the SVD for $\bar{F}$. Then the $s$-dimensional subspace that solves the least squares problem for $\f_1, \ldots, \f_k$ is the span of the first $s$ columns of $U$. If $U_s$ is the first $s$ columns of $U$, then the matrix $U_sU_s\trp$ is the projection onto this subspace.
\end{theorem}
\begin{proof}
We seek a rank-$s$ projection matrix $P_s$ so that $\sum_{i=1}^k \|P_s\bar{\f}_i - \bar{\f}_i\|_2^2$ is minimized---i.e.,
 the sum of the squares of the ``errors'' is minimal when we project $\bar{\f}_i$ via $P_s$.
 But minimizing this quantity is the same as minimizing its square, which happens to equal the Frobenius norm of $P_s\bar{F} - \bar{F}$.
Written mathematically,
 \begin{align*}
\inf_{\text{rank}(P_s)=s} \sum_{i=1}^k \|P_s\bar{\f}_i - \bar{\f}_i\|_2^2 &=  \inf_{\text{rank}(P_s)=s} \left( \sum_{i=1}^k \|P_s\bar{\f}_i - \bar{\f}_i\|_2^2 \right) ^2\\
 & =  \inf_{\text{rank}(P_s)=s} \| P_s\bar{F}-\bar{F}\|_F.
 \end{align*}

Now let $U \Sigma V\trp$ be an SVD of $\bar{F}$ with $\u_i$ the columns of $U$, $\v_i$ the columns of $V$, and $\sigma_i$ the singular values of $\bar{F}$.
If $P_s = \sum_{i=1}^s \u_i \u_i\trp$, then
\begin{align*}
P_s\bar{F} &=  \left( \sum_{i=1}^s \u_i \u_i\trp \right)\left(  \sum_{j=1}^k \sigma_j \u_i \v_i\trp \right)
= \sum_{i=1}^s\sum_{j=1}^k \sigma_j \u_i\u_i\trp\u_j\v_j\trp\\
&=  \sum_{i=1}^s\sum_{j=1}^k \sigma_j \u_i\delta_{ij}\v_j\trp
= \sum_{i=1}^s \sigma_i \u_i\v_i\trp.
\end{align*}

In fact, the Schmidt-Eckart-Young-Mirsky Theorem tells us that $X = \sum_{i=1}^s \sigma_i \u_i\v_i\trp$ is exactly the rank-$s$ matrix that minimizes $\|X - \bar{F}\|_F$.
Since $P_s \bar{F}$ will always have rank $s$ or less, the projection $P_s =  \sum_{i=1}^s \u_i \u_i\trp$ is the one we seek.
If we let $U_s = [ \u_1\; \ldots \; \u_s]$, then we may write $P_s = U_sU_s\trp$. Notice that $P_s$ is projection onto the subspace spanned by the columns of $U_s$.
\end{proof}

\end{comment}


\problemname{Interview Queue}

A very popular position has just opened up at Poodle Inc. Candidates have formed
a queue while they wait for their turn to be interviewed.

Competition is fierce and each candidate knows that they will not be selected if another candidate is strictly better than them. Every minute, each candidate looks at the resum\'{e} of candidates who are currently adjacent to them in the queue (ahead and behind). If at least one of the neighbour's resum\'{e}'s perceived value is strictly greater than their resum\'{e}'s perceived value, they leave the queue since they do not want to waste their time. The candidates all look at their neighbor's resum\'{e} simultaneously, and then some candidates leave the queue simultaneously.

This process continues until no more candidates leave the queue. Determine the number of minutes that pass while this process takes place. Report the values of the candidates that leave the queue in each round. Also report the final state of the queue.

\section*{Input}

The first line of input contains a single integer $N$ ($1 \leq N \leq 100\,000$), which is the number of candidates.

The second line contains $N$ integers $v_1, \ldots, v_N$ ($0 \leq v_i \leq 10^9$ for each $1 \leq i \leq N$), where $v_i$ is the perceived value of the $i^\textrm{th}$ candidate.

\section*{Output}
Display $M$, the number of minutes taken by this process. Then display $M$ lines. The $i^\textrm{th}$ line should contain the perceived values of the candidates who left the queue in the $i^\textrm{th}$ minute in the same relative order that they appeared in the queue. Finally display a line indicating the final list of perceived values in the queue after candidates no longer leave it.