\chapter{Substitution Commands}\index{Substitution}
An important class of commands in {\REDUCE} define
substitutions for variables and expressions to be made during the
evaluation of expressions.  Such substitutions use the prefix operator
{\tt SUB}, various forms of the command {\tt LET}, and rule sets.

\section{SUB Operator}\ttindex{SUB}

Syntax:
\begin{verbatim}
        SUB(<substitution_list>,EXPRN1:algebraic):algebraic
\end{verbatim}
where {\tt <substitution\_list>} is a list of one or more equations of the
form
\begin{verbatim}
        VAR:kernel=EXPRN:algebraic
\end{verbatim}
or a kernel that evaluates to such a list.

The {\tt SUB} operator gives the algebraic result of replacing every
occurrence of the variable {\tt VAR} in the expression {\tt EXPRN1} by the
expression {\tt EXPRN}.  Specifically, {\tt EXPRN1} is first evaluated
using all available rules.  Next the substitutions are made, and finally
the substituted expression is reevaluated.  When more than one variable
occurs in the substitution list, the substitution is performed by
recursively walking down the tree representing {\tt EXPRN1}, and replacing
every {\tt VAR} found by the appropriate {\tt EXPRN}.  The {\tt EXPRN} are
not themselves searched for any occurrences of the various {\tt VAR}s.
The trivial case {\tt SUB(EXPRN1)} returns the algebraic value of
{\tt EXPRN1}.

{\it Examples:}
\begin{verbatim}
				    2              2
     sub({x=a+y,y=y+1},x^2+y^2) -> A  + 2*A*Y + 2*Y  + 2*Y + 1
\end{verbatim}
and with {\tt s := \{x=a+y,y=y+1\}},
\begin{verbatim}
				    2              2
     sub(s,x^2+y^2)             -> A  + 2*A*Y + 2*Y  + 2*Y + 1
\end{verbatim}

Note that the global assignments {\tt x:=a+y}, etc., do not take place.

{\tt EXPRN1} can be any valid algebraic expression whose type is such that
a substitution process is defined for it (e.g., scalar expressions, lists
and matrices).  An error will occur if an expression of an invalid type
for substitution occurs either in {\tt EXPRN} or {\tt EXPRN1}.

The braces around the substitution list may also be omitted, as in:

\begin{verbatim}
				    2              2
     sub(x=a+y,y=y+1,x^2+y^2)   -> A  + 2*A*Y + 2*Y  + 2*Y + 1
\end{verbatim}

\section{LET Rules}\ttindex{LET}
Unlike substitutions introduced via {\tt SUB}, {\tt LET}
rules are global in scope and stay in effect until replaced or {\tt CLEAR}ed.

The simplest use of the {\tt LET} statement is in the form
\begin{verbatim}
	LET <substitution list>
\end{verbatim}
where {\tt <substitution list>} is a list of rules separated by commas, each
of the form:
\begin{verbatim}
	<variable> = <expression>
\end{verbatim}
or
\begin{verbatim}
    <prefix operator>(<argument>,...,<argument>) = <expression>
\end{verbatim}
or
\begin{verbatim}
    <argument> <infix operator>,..., <argument> = <expression>
\end{verbatim}
For example,
\begin{verbatim}
	let {x => y^2,
	     h(u,v) => u - v,
	     cos(pi/3) => 1/2,
	     a*b => c,
	     l+m => n,
	     w^3 => 2*z - 3,
	     z^10 => 0}
\end{verbatim}
The list brackets can be left out if preferred.  The above rules could
also have been entered as seven separate {\tt LET} statements.

After such {\tt LET} rules have been input, {\tt X} will always be
evaluated as the square of {\tt Y}, and so on.  This is so even if at the
time the {\tt LET} rule was input, the variable {\tt Y} had a value other
than {\tt Y}. (In contrast, the assignment {\tt x:=y\verb|^|2} will set {\tt X}
equal to the square of the current value of {\tt Y}, which could be quite
different.)

The rule {\tt let a*b=c} means that whenever {\tt A} and {\tt B} are both
factors in an expression their product will be replaced by {\tt C}.  For
example, {\tt a\verb|^|5*b\verb|^|7*w} would be replaced by
{\tt c\verb|^|5*b\verb|^|2*w}.

The rule for {\tt l+m} will not only replace all occurrences of {\tt l+m}
by {\tt N}, but will also normally replace {\tt L} by {\tt n-m}, but not
{\tt M} by {\tt n-l}.  A more complete description of this case is given
in Section~\ref{sec-gensubs}.

The rule pertaining to {\tt w\verb|^|3} will apply to any power of {\tt W}
greater than or equal to the third.

Note especially the last example, {\tt let z\verb|^|10=0}.  This declaration
means, in effect: ignore the tenth or any higher power of {\tt Z}.  Such
declarations, when appropriate, often speed up a computation to a
considerable degree. (See\index{Asymptotic command}
Section~\ref{sec-asymp} for more details.)

Any new operators occurring in such {\tt LET} rules will be automatically
declared {\tt OPERATOR} by the system, if the rules are being read from a
file.  If they are being entered interactively, the system will ask
{\tt DECLARE} ... {\tt OPERATOR?} .  Answer {\tt Y} or {\tt N} and hit
\key{Return}.

In each of these examples, substitutions are only made for the explicit
expressions given; i.e., none of the variables may be considered arbitrary
in any sense. For example, the command
\begin{verbatim}
        let h(u,v) = u - v;
\end{verbatim}
will cause {\tt h(u,v)} to evaluate to {\tt U - V}, but will not affect
{\tt h(u,z)} or {\tt H} with any arguments other than precisely the
symbols {\tt U,V}.

These simple {\tt LET} rules are on the same logical level as assignments
made with the := operator.  An assignment {\tt x := p+q} cancels a rule
{\tt let x = y\verb|^|2} made earlier, and vice versa.

{\it CAUTION:} A recursive rule such as
\begin{verbatim}
        let x = x + 1;
\end{verbatim}
is erroneous, since any subsequent evaluation of {\tt X} would lead to a
non-terminating chain of substitutions:
\begin{verbatim}
      x -> x + 1 -> (x + 1) + 1 -> ((x + 1) + 1) + 1 -> ...
\end{verbatim}
Similarly, coupled substitutions such as
\begin{verbatim}
        let l = m + n, n = l + r;
\end{verbatim}
would lead to the same error. As a result, if you try to evaluate an {\tt X},
{\tt L} or {\tt N} defined as above, you will get an error such as
\begin{verbatim}
        X improperly defined in terms of itself
\end{verbatim}

Array and matrix elements can appear on the left-hand side of a {\tt LET}
statement. However, because of their {\em instant evaluation\/}
\index{Instant evaluation} property, it is the value of the element that
is substituted for, rather than the element itself.  E.g.,
\begin{verbatim}
        array a(5);
        a(2) := b;
        let a(2) = c;
\end{verbatim}
results in {\tt B} being substituted by {\tt C}; the assignment for
{\tt a(2)} does not change.

Finally, if an error occurs in any equation in a {\tt LET} statement
(including generalized statements involving {\tt FOR ALL} and {\tt SUCH
THAT)}, the remaining rules are not evaluated.

\subsection{FOR ALL \ldots LET}\ttindex{FOR ALL}
If a substitution for all possible values of a given argument of an
operator is required, the declaration {\tt FOR ALL} may be used. The
syntax of such a command is
\begin{verbatim}
        FOR ALL <variable>,...,<variable>
                <LET statement> <terminator>
\end{verbatim}
e.g.,
\begin{verbatim}
        for all x,y let h(x,y) = x-y;
        for all x let k(x,y) = x^y;
\end{verbatim}
The first of these declarations would cause {\tt h(a,b)} to be evaluated
as {\tt A-B}, {\tt h(u+v,u+w)} to be {\tt V-W}, etc.  If the operator
symbol {\tt H} is used with more or fewer argument places, not two, the
{\tt LET} would have no effect, and no error would result.

The second declaration would cause {\tt k(a,y)} to be evaluated as
{\tt a\verb|^|y}, but would have no effect on {\tt k(a,z)} since the rule
didn't say {\tt FOR ALL Y} ... .

Where we used {\tt X} and {\tt Y} in the examples, any variables could
have been used.  This use of a variable doesn't affect the value it may
have outside the {\tt LET} statement.  However, you should remember what
variables you actually used.  If you want to delete the rule subsequently,
you must use the same variables in the {\tt CLEAR} command.

It is possible to use more complicated expressions as a template for a
{\tt LET} statement, as explained in the section on substitutions for
general expressions.  In nearly all cases, the rule will be accepted, and
a consistent application made by the system.  However, if there is a sole
constant or a sole free variable on the left-hand side of a rule (e.g.,
{\tt let 2=3} or {\tt for all x let x=2)}, then the system is unable to
handle the rule, and the error message
\begin{verbatim}
        Substitution for ... not allowed
\end{verbatim}
will be issued.  Any variable listed in the {\tt FOR ALL} part will have
its symbol preceded by an equal sign: {\tt X} in the above example will
appear as {\tt =X}.  An error will also occur if a variable in the
{\tt FOR ALL} part is not properly matched on both sides of the {\tt LET}
equation.

\subsection{FOR ALL \ldots SUCH THAT \ldots LET}
\ttindex{FOR ALL}\ttindex{SUCH THAT}

If a substitution is desired for more than a single value of a variable in
an operator or other expression, but not all values, a conditional form of
the {\tt FOR ALL \ldots LET} declaration can be used.

{\it Example:}
\begin{verbatim}
        for all x such that numberp x and x<0 let h(x)=0;
\end{verbatim}
will cause {\tt h(-5)} to be evaluated as 0, but {\tt H} of a positive
integer, or of an argument that is not an integer at all, would not be
affected.  Any boolean expression can follow the {\tt SUCH THAT} keywords.

\subsection{Removing Assignments and Substitution Rules}\ttindex{CLEAR}

The user may remove all assignments and substitution rules from any
expression by the command {\tt CLEAR}, in the form
\begin{verbatim}
        CLEAR <expression>,...,<expression><terminator>
\end{verbatim}
e.g.
\begin{verbatim}
        clear x, h(x,y);
\end{verbatim}
Because of their {\em instant evaluation\/} property, array and matrix elements
cannot be cleared with {\tt CLEAR}.  For example, if {\tt A} is an array,
you must say
\begin{verbatim}
        a(3) := 0;
\end{verbatim}
rather than
\begin{verbatim}
        clear a(3);
\end{verbatim}
to ``clear'' element {\tt a(3)}.

On the other hand, a whole array (or matrix) {\tt A} can be cleared by the
command {\tt clear a};  This means much more than resetting to 0 all the
elements of {\tt A}.  The fact that {\tt A} is an array, and what its
dimensions are, are forgotten, so {\tt A} can be redefined as another type
of object, for example an operator.

The more general types of {\tt LET} declarations can also be deleted by
using {\tt CLEAR}.  Simply repeat the {\tt LET} rule to be deleted, using
{\tt CLEAR} in place of {\tt LET}, and omitting the equal sign and
right-hand part.  The same dummy variables must be used in the {\tt FOR
ALL} part, and the boolean expression in the {\tt SUCH THAT} part must be
written the same way. (The placing of blanks doesn't have to be
identical.)

{\it Example:} The {\tt LET} rule
\begin{verbatim}
        for all x such that numberp x and x<0 let h(x)=0;
\end{verbatim}
can be erased by the command
\begin{verbatim}
        for all x such that numberp x and x<0 clear h(x);
\end{verbatim}

\subsection{Overlapping LET Rules}
{\tt CLEAR} is not the only way to delete a {\tt LET} rule.  A new {\tt
LET} rule identical to the first, but with a different expression after
the equal sign, replaces the first.  Replacements are also made in other
cases where the existing rule would be in conflict with the new rule.  For
example, a rule for {\tt x\verb|^|4} would replace a rule for {\tt x\verb|^|5}.
The user should however be cautioned against having several {\tt LET}
rules in effect that relate to the same expression.  No guarantee can be
given as to which rules will be applied by {\REDUCE} or in what order.  It
is best to {\tt CLEAR} an old rule before entering a new related {\tt LET}
rule.

\subsection{Substitutions for General Expressions}
\label{sec-gensubs}
The examples of substitutions discussed in other sections have involved
very simple rules. However, the substitution mechanism used in {\REDUCE} is
very general, and can handle arbitrarily complicated rules without
difficulty.

The general substitution mechanism used in {\REDUCE} is discussed in Hearn, A.
C., ``{\REDUCE}, A User-Oriented Interactive System for Algebraic
Simplification,'' Interactive Systems for Experimental Applied Mathematics,
(edited by M. Klerer and J. Reinfelds), Academic Press, New York (1968),
79-90, and Hearn. A. C., ``The Problem of Substitution,'' Proc. 1968 Summer
Institute on Symbolic Mathematical Computation, IBM Programming Laboratory
Report FSC 69-0312 (1969). For the reasons given in these
references, {\REDUCE} does not attempt to implement a general pattern
matching algorithm.  However, the present system uses far more sophisticated
techniques than those discussed in the above papers.  It is now possible for
the rules appearing in arguments of {\tt LET} to have the form
\begin{verbatim}
        <substitution expression> = <expression>
\end{verbatim}
where any rule to which a sensible meaning can be assigned is permitted.
However, this meaning can vary according to the form of {\tt <substitution
expression>}. The semantic rules associated with the application of the
substitution are completely consistent, but somewhat complicated by the
pragmatic need to perform such substitutions as efficiently as possible.
The following rules explain how the majority of the cases are handled.

To begin with, the {\tt <substitution expression>} is first partly
simplified by collecting like terms and putting identifiers (and kernels)
in the system order.  However, no substitutions are performed on any part
of the expression with the exception of expressions with the {\em instant
evaluation\/} property, such as array and matrix elements, whose actual
values are used.  It should also be noted that the system order used is
not changeable by the user, even with the {\tt KORDER} command.  Specific
cases are then handled as follows:
\begin{enumerate}
\item If the resulting simplified rule has a left-hand side that is an
identifier, an expression with a top-level algebraic operator or a power,
then the rule is added without further change to the appropriate table.

\item If the operator * appears at the top level of the simplified left-hand
side, then any constant arguments in that expression are moved to the
right-hand side of the rule.  The remaining left-hand side is then added
to the appropriate table.  For example,
\begin{verbatim}
        let 2*x*y=3
\end{verbatim}
becomes
\begin{verbatim}
        let x*y=3/2
\end{verbatim}
so that {\tt x*y} is added to the product substitution table, and when
this rule is applied, the expression {\tt x*y} becomes 3/2, but {\tt X} or
{\tt Y} by themselves are not replaced.

\item If the operators {\tt +}, {\tt -} or {\tt /} appear at the top level
of the simplified left-hand side, all but the first term is moved to the
right-hand side of the rule.  Thus the rules
\begin{verbatim}
        let l+m=n, x/2=y, a-b=c
\end{verbatim}
become
\begin{verbatim}
        let l=n-m, x=2*y, a=c+b.
\end{verbatim}
\end{enumerate}
One problem that can occur in this case is that if a quantified expression
is moved to the right-hand side, a given free variable might no longer
appear on the left-hand side, resulting in an error because of the
unmatched free variable. E.g.,
\begin{verbatim}
        for all x,y let f(x)+f(y)=x*y
\end{verbatim}
would become
\begin{verbatim}
        for all x,y let f(x)=x*y-f(y)
\end{verbatim}
which no longer has {\tt Y} on both sides.

The fact that array and matrix elements are evaluated in the left-hand side
of rules can lead to confusion at times. Consider for example the
statements
\begin{verbatim}
        array a(5); let x+a(2)=3; let a(3)=4;
\end{verbatim}
The left-hand side of the first rule will become {\tt X}, and the second
0.  Thus the first rule will be instantiated as a substitution for
{\tt X}, and the second will result in an error.

The order in which a list of rules is applied is not easily understandable
without a detailed knowledge of the system simplification protocol. It is
also possible for this order to change from release to release, as improved
substitution techniques are implemented. Users should therefore assume
that the order of application of rules is arbitrary, and program
accordingly.

After a substitution has been made, the expression being evaluated is
reexamined in case a new allowed substitution has been generated. This
process is continued until no more substitutions can be made.

As mentioned elsewhere, when a substitution expression appears in a
product, the substitution is made if that expression divides the product.
For example, the rule
\begin{verbatim}
	let a^2*c = 3*z;
\end{verbatim}
would cause {\tt a\verb|^|2*c*x} to be replaced by {\tt 3*Z*X} and
{\tt a\verb|^|2*c\verb|^|2} by {\tt 3*Z*C}.  If the substitution is desired only
when the substitution expression appears in a product with the explicit
powers supplied in the rule, the command {\tt MATCH} should be used
instead.\ttindex{MATCH}

For example,
\begin{verbatim}
        match a^2*c = 3*z;
\end{verbatim}
would cause {\tt a\verb|^|2*c*x} to be replaced by {\tt 3*Z*X}, but
{\tt a\verb|^|2*c\verb|^|2} would not be replaced. {\tt MATCH} can also be used
with the {\tt FOR ALL} constructions described above.

To remove substitution rules of the type discussed in this section, the
{\tt CLEAR}\ttindex{CLEAR} command can be used, combined, if necessary,
with the same {\tt FOR ALL} clause with which the rule was defined, for
example:
\begin{verbatim}
        for all x clear log(e^x),e^log(x),cos(w*t+theta(x));
\end{verbatim}
Note, however, that the arbitrary variable names in this case {\em must\/}
be the same as those used in defining the substitution.

\section{Rule Lists} \index{Rule lists}

Rule lists offer an alternative approach to defining substitutions that is
different from either {\tt SUB} or {\tt LET}.  In fact, they provide the
best features of both, since they have all the capabilities of {\tt LET},
but the rules can also be applied locally as is possible with {\tt SUB}.
In time, they will be used more and more in {\REDUCE}.  However, since they
are relatively new, much of the {\REDUCE} code you see uses the older
constructs.

A rule list is a list of {\em rules\/} that have the syntax
\begin{verbatim}
     <expression> => <expression> (WHEN <boolean expression>)
\end{verbatim}
For example,
\begin{verbatim}
	{cos(~x)*cos(~y) => (cos(x+y)+cos(x-y))/2,
	 cos(~n*pi)      => (-1)^n when remainder(n,2)=0}
\end{verbatim}

The tilde preceding a variable marks that variable as {\em free\/} for that
rule, much as a variable in a {\tt FOR ALL} clause in a {\tt LET}
statement.  The first occurrence of that variable in each relevant rule
must be so marked on input, otherwise inconsistent results can occur.
For example, the rule list
\begin{verbatim}
        {cos(~x)*cos(~y) => (cos(x+y)+cos(x-y))/2,
	 cos(x)^2        => (1+cos(2x))/2}
\end{verbatim}
designed to replace products of cosines, would not be correct, since the
second rule would only apply to the explicit argument {\tt X}.  Later
occurrences in the same rule may also be marked, but this is optional
(internally, all such rules are stored with each relevant variable
explicitly marked).  The optional {\tt WHEN}\ttindex{WHEN} clause allows
constraints to be placed on the application of the rule, much as the {\tt
SUCH THAT} clause in a {\tt LET} statement.

A rule list may be named, for example
\begin{verbatim}
        trig1 := {cos(~x)*cos(~y) => (cos(x+y)+cos(x-y))/2,
                  cos(~x)*sin(~y) => (sin(x+y)-sin(x-y))/2,
                  sin(~x)*sin(~y) => (cos(x-y)-cos(x+y))/2,
                  cos(~x)^2       => (1+cos(2*x))/2,
                  sin(~x)^2       => (1-cos(2*x))/2};
\end{verbatim}

Such named rule lists may be inspected as needed. E.g., the command
{\tt trig1;} would cause the above list to be printed.

Rule lists may be used in two ways.  They can be globally instantiated by
means of the command {\tt LET}.\ttindex{LET} For example,
\begin{verbatim}
        let trig1;
\end{verbatim}
would cause the above list of rules to be globally active from then on until
cancelled by the command {\tt CLEARRULES},\ttindex{CLEARRULES} as in
\begin{verbatim}
        clearrules trig1;
\end{verbatim}
{\tt CLEARRULES} has the syntax
\begin{verbatim}
        CLEARRULES <rule list>|<name of rule list>(,...) .
\end{verbatim}

The second way to use rule lists is to invoke them locally by means of a
{\tt WHERE}\ttindex{WHERE} clause.  For example
\begin{verbatim}
        cos(a)*cos(b+c)
	   where {cos(~x)*cos(~y) => (cos(x+y)+cos(x-y))/2};
\end{verbatim}
or
\begin{verbatim}
        cos(a)*sin(b) where trigrules;
\end{verbatim}

The syntax of an expression with a {\tt WHERE} clause is:
\begin{verbatim}
        <expression>
            WHERE <rule>|<rule list>(,<rule>|<rule list> ...)
\end{verbatim}
so the first example above could also be written
\begin{verbatim}
        cos(a)*cos(b+c)
           where cos(~x)*cos(~y) => (cos(x+y)+cos(x-y))/2;
\end{verbatim}

The effect of this construct is that the rule list(s) in the {\tt WHERE}
clause only apply to the expression on the left of {\tt WHERE}.  They have
no effect outside the expression.  In particular, they do not affect
previously defined {\tt WHERE} clauses or {\tt LET} statements.  For
example, the sequence
\begin{verbatim}
     let a=2;
     a where a=>4;
     a;
\end{verbatim}
would result in the output
\begin{verbatim}
     4

     2
\end{verbatim}

Although {\tt WHERE} has a precedence less than any other infix operator,
it still binds higher than keywords such as {\tt ELSE}, {\tt THEN},
{\tt DO}, {\tt REPEAT} and so on.  Thus the expression
\begin{verbatim}
        if a=2 then 3 else a+2 where a=3
\end{verbatim}
will parse as
\begin{verbatim}
        if a=2 then 3 else (a+2 where a=3)
\end{verbatim}

{\tt WHERE} may be used to introduce auxiliary variables in symbolic mode
expressions, as described in Section~\ref{sec-lambda}.  However, the
symbolic mode use has different semantics, so expressions do not carry
from one mode to the other.

\COMPATNOTE In order to provide compatibility with older versions of rule
lists released through the Network Library, it is currently possible to use
an equal sign interchangeably with the replacement sign {\tt =>} in rules
and {\tt LET} statements.  However, since this will change in future
versions, the replacement sign is preferable in rules and the equal sign
in non-rule-based {\tt LET} statements.

\subsection*{Advanced Use of Rule Lists}

Some advanced features of the rule list mechanism make it possible to
write more complicated rules than those discussed so far, and in many
cases to write more compact rule lists.  These features are:

\begin{itemize}
\item Free operators
\item Double slash operator
\item Double tilde variables.
\end{itemize}
A {\bf free operator} in the left hand side of a pattern will match any
operator with the same number of arguments.  The free operator is written
in the same style as a variable.  For example, the implementation of the
product rule of differentiation can be written as:
\begin{verbatim}
operator diff, !~f, !~g;

prule := {diff(~f(~x) * ~g(~x),x) =>
	     diff(f(x),x) * g(x) + diff(g(x),x) * f(x)};

let prule;

diff(sin(z)*cos(z),z);

         cos(z)*diff(sin(z),z) + diff(cos(z),z)*sin(z)
\end{verbatim}

The {\bf double slash operator} may be used as an alternative to a single
slash (quotient) in order to match quotients properly.  E.g., in the
example of the Gamma function above, one can use:
\begin{verbatim}
gammarule :=
   {gamma(~z)//(~c*gamma(~zz))  => gamma(z)/(c*gamma(zz-1)*zz)
		  when fixp(zz -z) and (zz -z) >0,
    gamma(~z)//gamma(~zz) => gamma(z)/(gamma(zz-1)*zz)
		  when fixp(zz -z) and (zz -z) >0};

let gammarule;

gamma(z)/gamma(z+3);

          1
----------------------
  3      2
 z  + 6*z  + 11*z + 6
\end{verbatim}
The above example suffers from the fact that two rules had to be
written in order to perform the required operation. This can be simplified
by the use of {\bf double tilde variables}. E.g. the rule list
\begin{verbatim}
 GGrule :=  {
    gamma(~z)//(~~c*gamma(~zz))  => gamma(z)/(c*gamma(zz-1)*zz)
     when fixp(zz -z) and (zz -z) >0};
\end{verbatim}
will implement the same operation in a much more compact way.
In general, double tilde variables are bound to the neutral element
with respect to the operation in which they are used.

\begin{tabular}{lll}

Pattern given & Argument used & Binding  \\
\\
\symbol{126}z + \symbol{126}\symbol{126}y  &   x   &  z=x; y=0  \\
\symbol{126}z + \symbol{126}\symbol{126}y  &   x+3 &  z=x; y=3  or  z=3; y=x \\
\\
\symbol{126}z * \symbol{126}\symbol{126}y  &   x   &  z=x; y=1\\
\symbol{126}z * \symbol{126}\symbol{126}y  &   x*3 &  z=x; y=3  or  z=3; y=x\\
\\
\symbol{126}z / \symbol{126}\symbol{126}y  &    x   &  z=x; y=1\\
\symbol{126}z / \symbol{126}\symbol{126}y  &    x/3 &  z=x; y=3  \\
\\
\end{tabular}

Remarks: A double tilde variable as the numerator of a pattern is not allowed.
Also, using double tilde variables may lead to recursion errors when the
zero case is not handled properly.
\begin{verbatim}
let f(~~a * ~x,x)  => a * f(x,x) when freeof (a,x);

f(z,z);

***** f(z,z) improperly defined in terms of itself

% BUT:

let ff(~~a * ~x,x)
       => a * ff(x,x) when freeof (a,x) and a neq 1;

ff(z,z);
                 ff(z,z)

ff(3*z,z);
                 3*ff(z,z)
\end{verbatim}

\subsection*{Displaying Rules Associated with an Operator}

The operator {\tt SHOWRULES}\ttindex{SHOWRULES} takes a single identifier
as argument, and returns in rule-list form the operator rules associated
with that argument.  For example:
\begin{verbatim}
showrules log;

{LOG(E) => 1,

 LOG(1) => 0,

      ~X
 LOG(E  ) => ~X,

		    1
 DF(LOG(~X),~X) => ----}
		    ~X
\end{verbatim}

Such rules can then be manipulated further as with any list.  For example
{\tt rhs first ws;} has the value {\tt 1}.  Note that an operator may
have other properties that cannot be displayed in such a form, such as the
fact it is an odd function, or has a definition defined as a procedure.

\subsection*{Order of Application of Rules}

If rules have overlapping domains, their order of application is
important.  In general, it is very difficult to specify this order
precisely, so that it is best to assume that the order is arbitrary.
However, if only one operator is involved, the order of application of the
rules for this operator can be determined from the following:

\begin{enumerate}
\item Rules containing at least one free variable apply before all rules
without free variables.
\item Rules activated in the most recent {\tt LET}
command are applied first.
\item {\tt LET} with several entries generate
the same order of application as a corresponding sequence of commands with
one rule or rule set each.
\item Within a rule set, the rules containing at least
one free variable are applied in their given order.
In other words, the first member of the list is applied first.
\item Consistent with the first item, any rule in a rule list that
contains no free variables is applied after all rules containing free
variables.
\end{enumerate}
{\it Example:} The following rule set enables the computation of exact
values of the Gamma function:
\begin{verbatim}
        operator gamma,gamma_error;
        gamma_rules :=
        {gamma(~x)=>sqrt(pi)/2 when x=1/2,
         gamma(~n)=>factorial(n-1) when fixp n and n>0,
         gamma(~n)=>gamma_error(n) when fixp n,
         gamma(~x)=>(x-1)*gamma(x-1) when fixp(2*x) and x>1,
         gamma(~x)=>gamma(x+1)/x when fixp(2*x)};
\end{verbatim}
Here, rule by rule, cases of known or definitely uncomputable values
are sorted out; e.g. the rule leading to the error expression
will be applied for negative integers only, since the positive
integers are caught by the preceding rule, and the
last rule will apply for negative odd multiples of $1/2$ only.
Alternatively the first rule could have been written as
\begin{verbatim}
        gamma(1/2) => sqrt(pi)/2,
\end{verbatim}
but then the case $x=1/2$ should be excluded in the {\tt WHEN} part of the
last rule explicitly because a rule without free variables cannot take
precedence over the other rules.

\section{Asymptotic Commands} \index{Asymptotic command}
\label{sec-asymp}
In expansions of polynomials involving variables that are known to be
small, it is often desirable to throw away all powers of these variables
beyond a certain point to avoid unnecessary computation.  The command {\tt
LET} may be used to do this.  For example, if only powers of {\tt X} up to
{\tt x\verb|^|7} are needed, the command
\begin{verbatim}
        let x^8 = 0;
\end{verbatim}
will cause the system to delete all powers of {\tt X} higher than 7.

{\it CAUTION:}  This particular simplification works differently from most
substitution mechanisms in {\REDUCE} in that it is applied during
polynomial manipulation rather than to the whole evaluated expression.
Thus, with the above rule in effect, {\tt x\verb|^|10/x\verb|^|5} would give the
result zero, since the numerator would simplify to zero.  Similarly
{\tt x\verb|^|20/x\verb|^|10} would give a {\tt Zero divisor} error message,
since both numerator and denominator would first simplify to zero.

The method just described is not adequate when expressions involve several
variables having different degrees of smallness. In this case, it is
necessary to supply an asymptotic weight to each variable and count up the
total weight of each product in an expanded expression before deciding
whether to keep the term or not. There are two associated commands in the
system to permit this type of asymptotic constraint. The command {\tt WEIGHT}
\ttindex{WEIGHT}
takes a list of equations of the form
\begin{verbatim}
        <kernel form> = <number>
\end{verbatim}
where {\tt <number>} must be a positive integer (not just evaluate to a
positive integer).  This command assigns the weight {\tt <number>} to the
relevant kernel form.  A check is then made in all algebraic evaluations
to see if the total weight of the term is greater than the weight level
assigned to the calculation.  If it is, the term is deleted.  To compute
the total weight of a product, the individual weights of each kernel form
are multiplied by their corresponding powers and then added.

The weight level of the system is initially set to 1. The user may change
this setting by the command\ttindex{WTLEVEL}
\begin{verbatim}
        wtlevel <number>;
\end{verbatim}
which sets {\tt <number>} as the new weight level of the system.
{\tt <number>} must evaluate to a positive integer.  WTLEVEL will also
allow NIL as an argument, in which case the current weight level is returned.


\documentclass[]{report}   % list options between brackets
\usepackage{graphicx, listings, color}      % list packages between braces
\graphicspath{ {Images/} }

% type user-defined commands here

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codepurple},
    basicstyle=\footnotesize,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=5pt,
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    tabsize=2
}

\lstset{style=mystyle}

\begin{document}

\title{CAB420 Assignment 1}   % type title between braces
\author{Jarrod Williams (n9722068) and Madeline Miller (n9342401)}         % type author(s) between braces
\maketitle

\chapter{Theory}

Given the following equation,

$$L(w)=-\sum_{i=1}^{N}log(\frac{1}{1+e^{y_i(w^Tx_i+b)}})+\lambda||w||_2^2$$

\section{Finding the Partial Derivative}
Finding the partial derivative $$\frac{\partial L}{\partial w_j}$$

$$\frac{\partial L}{\partial w_j}(-\sum_{i=1}^{N}log(\frac{1}{1+e^{y_i(w^Tx_i+b)}})+\lambda||w||_2^2)$$

As the regulator at the end of the function is a squared L2 norm, it can be derived trivially.

$$\frac{\partial L}{\partial w_j}(-\sum_{i=1}^{N}log(\frac{1}{1+e^{y_i(w^Tx_i+b)}}))+2\lambda w_j$$

Simplify,

$$\frac{\partial L}{\partial w_j}(\sum_{i=1}^{N}log(1+e^{y_i(w^Tx_i+b)}))+2\lambda w_j$$

Using the chain rule,

$$\sum_{i=1}^{N}\frac{\frac{\partial L}{\partial w_j}(1+e^{y_i(w^Tx_i+b)})}{1+e^{y_i(w^Tx_{i,j}+b)}}+2\lambda w_j$$

$$\sum_{i=1}^{N}\frac{\frac{\partial L}{\partial w_j}(1)+\frac{\partial L}{\partial w_j}(e^{y_i(w^Tx_i+b)})}{1+e^{y_i(w^Tx_{i,j}+b)}}+2\lambda w_j$$

$$\sum_{i=1}^{N}\frac{\frac{\partial L}{\partial w_j}(e^{y_i(w^Tx_i+b)})}{1+e^{y_i(w^Tx_{i,j}+b)}}+2\lambda w_j$$

Using the chain rule again,

$$\sum_{i=1}^{N}\frac{y_ix_ie^{y_i(w^Tx_i+b)}}{1+e^{y_i(w^Tx_{i,j}+b)}}+2\lambda w_j$$

\section{Finding the Second Partial Derivative}
Finding the second partial derivative $$\frac{\partial^2 L}{\partial w_j\partial w_k}$$

$$\frac{\partial^2 L}{\partial w_j\partial w_k}=\frac{\partial}{\partial w_k}(\sum_{i=1}^{N}\frac{y_ix_ie^{y_i(w^Tx_i+b)}}{1+e^{y_i(w^Tx_{i,j}+b)}}+2\lambda w_j)$$

$$\sum_{i=1}^{N}\frac{y_ix_i(\frac{\partial}{\partial w_k}e^{y_i(w^Tx_i+b)})}{1+e^{y_i(w^Tx_{i,j}+b)}}$$

Using the chain rule,

$$\sum_{i=1}^{N}\frac{e^{y_i(w_kx_i+b)}y_i^2x_i(\frac{\partial}{\partial w_k}(w^Tx_i+b))}{1+e^{y_i(w^Tx_{i,j}+b)}}$$

$$\sum_{i=1}^{N}\frac{e^{y_i(w^Tx_i+b)}y_i^2x_iw_kx_i+b}{1+e^{y_i(w^Tx_{i,j}+b)}}$$

$$\sum_{i=1}^{N}\frac{e^{y_i(w^Tx_i+b)}y_i^2x_i^2w_k+b}{1+e^{y_i(w^Tx_{i,j}+b)}}$$

\section{Proving it's a convex function}

A function can be defined as convex if the Hessian matrix is positive semi-definite. This is defined as,

$$a^THa\equiv\sum_{j,k}a_ja_kH_{j,k}\geq0$$

Where,

$$H_{j,k}=\frac{\partial^2L}{\partial w_j\partial w_k}$$

This could be shown using a proof by contradiction.

\chapter{Practice}

\section{Features, Classes, and Linear Regression}
%%%%%%%%%%%%%%%%%%%%
{\bf (a) Plot the training data in a scatter plot.}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_1_Figure_1.png}\\
	{Figure 2.1.1: Training data in a scatter plot.}
\end{center}
~\\
{\bf (b) Create a linear predictor (assuming quadratic features). Plot it on the same plot as the training data.}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_1_Figure_2.png}\\
	{Figure 2.1.2: Quadratic linear predictor with training data scatter plot.}
\end{center}.
~\\
{\bf (c) Create another plot with the data and a fifth-degree polynomial.}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_1_Figure_3.png}\\
	{Figure 2.1.3: Quintic linear predictor with training data scatter plot.}
\end{center}
~\\
{\bf (d) Calculate the mean squared error associated with each of your learned models on the training.}
data.
\begin{lstlisting}[caption=Matlab output for MSE calculations on training data.]
The MSE for the quadratic linear predictor on training data was: 2178.55
The MSE for the quintic linear predictor on training data was: 1254.61
\end{lstlisting}
~\\
{\bf (e) Calculate the MSE for each model on the test data (in mcycleTest.txt).}
\begin{lstlisting}[caption=Matlab output for MSE calculations on test data.]
The MSE for the quadratic linear predictor on test data was: 1573.27
The MSE for the quintic linear predictor on test data was: 999.65
\end{lstlisting}

\section{kNN Regression}
%%%%%%%%%%%%%%%%%%%%
{\bf (a) Using the knnRegress class, implement (add code to) the predict function to make it functional.}
\\~\\
\begin{lstlisting}[language=Matlab, caption=predict() Implementation]
% Test function: predict on Xtest
function Yte = predict(obj,Xte)
  % get size of training, test data
  [Ntr,Mtr] = size(obj.Xtrain);
  [Nte,Mte] = size(Xte);
  % figure out how many classes & their labels
  classes = unique(obj.Ytrain);
  % make Ytest the same data type as Ytrain
  Yte = repmat(obj.Ytrain(1), [Nte,1]);
  % can't have more than Ntrain neighbors
  K = min(obj.K, Ntr);
  % For each test example:
  for i=1:Nte,
    % compute sum of squared differences
    dist = sum( bsxfun( @minus, obj.Xtrain, Xte(i,:) ).^2 , 2);
    % find nearest neighbors over Xtrain (dimension 2)
    [tmp,idx] = sort(dist);
    % Set y to the mean of the K nearest neighbours
    Yte(i) = mean(obj.Ytrain(idx(1:K)));
  end;
end
\end{lstlisting}
{\bf (b) Using the same technique as in Problem 1a, plot the predicted function for several values of k: 1, 2, 3, 5, 10, 50. (You can just use a for-loop to do this.) How does the choice of k relate to the “complexity” of the regression function?}
\\~\\
\begin{center}
	\includegraphics[width=35em]{2_2_Figure_1.png}
	{Figure 2.2.1: kNN Regression model at various k values.}
\end{center}
The closer to 1 that the k value is, the more complex the function appears. This is due to the line being influenced by more separate clusters of nodes when k is a lower value. When k is higher, individual nodes have less of an impact as it uses an average of the k nearest neighbours to determine the current y value.
\\~\\
{\bf (c) We discussed in class that the k-nearest-neighbor classifier’s decision boundary can be shown to be piecewise linear. What kind of functions can be output by a nearest neighbor regression function? Briefly justify your conclusion. (You do not need to discuss the general case – just the 1-dimensional regression picture such as your plots.)}
\\~\\
In this case, as well as most cases, the output function will also be piecewise linear. Linear because there will only ever be a single y value for each x value, and piecewise because it may not necessarily be a single plot-able function. In this specific case it's much too complicated to be plotted as a single function and must be plotted as a piecewise.
\section{Hold-out and Cross-validation}
{\bf (a) Similarly to Problem 1 and 2, compute the MSE of the test data on a model trained on only the first 20 training data examples for k = 1, 2, 3, . . . , 100. Plot the MSE versus k on a log-log scale (see help loglog).}
\\~\\
See part c below.
\\~\\
{\bf (b) Repeat, but use all the training data. What happened? Contrast with your results from problem 1}
\\~\\
For plot, see part c below.
\\~\\
With all of the training data, k nearest neighbours works better. Due to the lack of data points in the 20 training data set, the lower k values performed better than the higher ones.
\\~\\
When k nearest neighbours has more data points available, higher k values can provide for a more accurate model. This is seen at around $10^1$ in the in the plot.
\\~\\
{\bf (c) We discussed in class that the k-nearest-neighbor classifier’s decision boundary can be shown to be piecewise linear. What kind of functions can be output by a nearest neighbor regression function? Briefly justify your conclusion. (You do not need to discuss the general case – just the 1-dimensional regression picture such as your plots.)}
\\~\\
The cross validation allows us to get an estimated accuracy by averaging out the accuracy from multiple different models. It also allows for better validation of each k value without requiring more data, by using existing data over and over in different ways.
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_3_Figure_1.png}\\
	{Figure 2.3.1: kNN Regression model Mean Squared Error and Cross Validation.}
\end{center}

\section{Nearest Neighbor Classifiers}
%%%%%%%%%%%%%%%%%%%%
{\bf (a) Plot the data by their feature values, using the class value to select the color.}
\begin{center}
	\includegraphics[width=35em]{2_4_Figure_1.png}
	{Figure 2.4.1: Data plotted by feature value with class value determining colour.}
\end{center}
~\\
{\bf (b) Use the provided $knnClassify$ class to learn a 1-nearest-neighbour predictor. Use the function $class2DPlot(learner,X,Y)$ to plot the decision regions and training data together.}
\begin{lstlisting}[language=Matlab, caption=Teaching the 1-nearest-neighbour predictor.]
learner = knnClassify(1, X, Y);
class2DPlot(learner,X,Y);
title('1-nearest-neighbour predictor');
\end{lstlisting}
\begin{center}
	\includegraphics[width=35em]{2_4_Figure_2.png}
	{Figure 2.4.2: Plot of 1-nearest-neighbour predictor. Generated with the code in Listing 2.4.}
\end{center}
~\\
{\bf (c) Do the same thing for several values of k (say, [1, 3, 10, 30]) and comment on their appearance.}
\begin{lstlisting}[language=Matlab, caption=Teaching k-nearest-neighbour predictor for multiple values of k. {\bf Note:} figure for 1-nearest-neighbour predictor is above in part (b).]
kValues = [1,3,10,30]; % k=1 is already plotted above in (B).
for index = 1:length(kValues)
    learner = knnClassify(kValues(index), X, Y);
    class2DPlot(learner,X,Y);
    title(strcat(int2str(kValues(index)), '-nearest-neighbour predictor'));
end
\end{lstlisting}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_4_Figure_3.png}\\
	{Figure 2.4.3: Plot of 3-nearest-neighbour predictor. Generated with the code in Listing 2.4.}
\end{center}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_4_Figure_4.png}\\
	{Figure 2.4.4: Plot of 10-nearest-neighbour predictor. Generated with the code in Listing 2.4.}
\end{center}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_4_Figure_5.png}\\
	{Figure 2.4.5: Plot of 30-nearest-neighbour predictor. Generated with the code in Listing 2.4.}
\end{center}
{From visual inspection of figures 2.4.[2-5], as the value of $k$ increases, so does the definition of the decision regions. For small values of $k$ (such as 1), clear under-fitting can be observed. The opposite is also true for large values (such as 30) at which point over-fitting occurs.}
\\~\\
{\bf (d) Now split the data into an 80/20 training/validation split. For k = [1, 2, 5, 10, 50, 100, 200], learn a model on the 80$\%$ and calculate its performance ($\#$ of data classified incorrectly) on the validation data.}
\begin{lstlisting}[language=Matlab, caption=Training models on 80\% training data and calculating its performance on 20\% validation data.]
Xtrain = X(1:118, 1:end);
Xvalid = X(119:end, 1:end);
Ytrain = Y(1:118);
Yvalid = Y(119:end);
kValues = [1, 2, 5, 10, 50, 100, 200];
errors = [];
for index = 1:length(kValues)
    learner = knnClassify(kValues(index), Xtrain, Ytrain); % train model on X/Ytrain
    Yhat = predict(learner, Xvalid); % predict results on Xtrain/Yvalid
    errors = [errors, numel(find(Yhat~=Yvalid))]; % count what fraction of predictions are wrong
    hold on;
end
\end{lstlisting}
\begin{center}
	\includegraphics[width=35em]{2_4_Figure_6.png}
	{Figure 2.4.6: Plot of incorrectly classified validation data points against k-nearest-neighbour predictors. Data generated with the code in Listing 2.5.}
\end{center}
{\bf What value of k appears to generalize best given your training data? Comment on the performance at the two endpoints, in terms of over- or under-fitting.} \\~\\
{For small values of $k$ (1,3,5), under-fitting is present, while larger values (200) provide over-fitting models. As a result, $k$ appears to generalise best between these two values. For Figure 2.4.6 specifically, $k = 10$ appears to generalise best.}

\section{Perceptrons and Logistic Regression}
{\bf (a) Show the two classes in a scatter plot and verify that one is linearly separable while the other is not.}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_5_Figure_1.png}\\
	{Figure 2.5.1: Class A. Data {\bf is} linearly separable.}
\end{center}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_5_Figure_2.png}\\
	{Figure 2.5.2: Class B. Data {\bf is not} linearly separable.}
\end{center}
~\\
{\bf (b) Write the function @logisticClassify2/plot2DLinear.m so that it plots the two
classes of data in different colors, along with the decision boundary. Include the listing of your code in your report.}

\begin{lstlisting}[language=Matlab, caption=plot2DLinear() Implementation]
function plot2DLinear(obj, X, Y)
% plot2DLinear(obj, X,Y)
%   plot a linear classifier (data and decision boundary) when features X are 2-dim
%   wts are 1x3,  wts(1)+wts(2)*X(1)+wts(3)*X(2)
%
[n,d] = size(X);
if (d~=2) error('Sorry -- plot2DLogistic only works on 2D data...'); end;

figure('Name','Linear Plot');
hold on;

% Plot class data.
classes = unique(Y);

classAIndicies = find(Y==classes(1));
xPointsClassA = X(classAIndicies, 1:end);
scatter(xPointsClassA(:, 1), xPointsClassA(:, 2));

classBIndicies = find(Y==classes(2));
xPointsClassB = X(classBIndicies, 1:end);
scatter(xPointsClassB(:, 1), xPointsClassB(:, 2));

% Plot decision boundary.
wts = getWeights(obj);
f = @(x1, x2) wts(1) + wts(2)*x1 + wts(3)*x2 ;
ezplot(f,[-3.5,3.5])

legend('Class 0','Class 1', 'Decision Bondary');
hold off;
\end{lstlisting}
{\bf To demo your function plot the decision boundary corresponding to the classifier: $sign( .5 + 1x_{1} − .25x_{2} )$ along with the A data, and again with the B data.}
\begin{lstlisting}[language=Matlab, caption=Demoing plot2DLinear()]
%% (B) Write the function @logisticClassify2/plot2DLinear.m such that it
%      can Plot the two classes of data in different colors, along with the
%      decision boundary (a line). To demo your function plot the decision
%       boundary corresponding to the classifier sign( .5 + 1x1 − .25x2 )
%       along with the A data, and again with the B data.

learner=logisticClassify2(); % create "blank" learner
learner=setClasses(learner, unique(YA)); % define class labels using YA or YB
wts = [0.5 1 -0.25];
learner=setWeights(learner, wts); % set the learner's parameters
plot2DLinear(learner, XA, YA);
title('Class A with Decision Boundary');
plot2DLinear(learner, XB, YB);
title('Class B with Decision Boundary');
\end{lstlisting}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_5_Figure_3.png}\\
	{Figure 2.5.3: Class A with decision boundary plotted.}
\end{center}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_5_Figure_4.png}\\
	{Figure 2.5.4: Class B with decision boundary plotted.}
\end{center}
~\\
{\bf (c) Complete the predict.m function to make predictions for your linear classifier.}
\begin{lstlisting}[language=Matlab, caption=predict() Implementation]
function Yte = predict(obj,Xte)
% Yhat = predict(obj, X)  : make predictions on test data X

% (1) make predictions based on the sign of wts(1) + wts(2)*x(:,1) + ...
% (2) convert predictions to saved classes: Yte = obj.classes( [1 or 2] );
wts = getWeights(obj);
yhat = zeros(size(Xte,1),1);
f = @(x1, x2) wts(1) + wts(2)*x1 + wts(3)*x2;

for i=1:size(Xte,1)
    x = sign(f(Xte(i,1),Xte(i,2)));
    yhat(i) = obj.classes(ceil((x+3)/2));
end;

Yte = yhat;
\end{lstlisting}
{\bf Again, verify that your function works by computing \& reporting the error rate of the classifier in the previous part on both data sets A and B. (The error rate on data set A should be ≈ 0.0505.)}
\begin{lstlisting}[language=Matlab, caption=Verifying predict() works]
% Data set A
yte = predict(learner,XA);
classError = 0;
for i=1:size(YA,1);
    if(YA(i) ~= yte(i));
        classError = classError + 1;
    end;
end;
finalClassError = classError/size(YA,1); % = 0.0505
disp(strcat({'The error for class A is:'},{' '},{num2str(finalClassError,' %.4f')}));
% Data set B
yte = predict(learner,XB);
classError = 0;
for i=1:size(YB,1);
    if(YB(i) ~= yte(i));
        classError = classError + 1;
    end;
end;
finalClassError = classError/size(YB,1); % = 0.54555
disp(strcat({'The error for class B is:'},{' '},{num2str(finalClassError,' %.4f')}));
\end{lstlisting}
\begin{lstlisting}[caption=Matlab output for error rate of Class A and B.]
'The error rate for class A is: 0.0505'
'The error rate for class B is: 0.5455'
\end{lstlisting}
~\\
{\bf (d) In my provided code, I first transform the classes in the data Y into "class 0" (negative) and "class 1" (positive). In our notation, let $z = \theta x^{(i)}$ is the linear response of the perceptron, and $\sigma$ is the standard logistic function.
$$\sigma (z) = (1 + exp(-z))^{-1}$$
The (regularized) logistic negative log likelihood loss for a single data point j is then
$$ J_{j}(\theta) = -y^{(j)} log \sigma (\theta x^{(j)T}) - (1 - y^{(j)})log(1-\sigma (\theta x^{(j)T})) + \alpha \sum_{i} \theta^{2}_{i} $$
where $y(j)$ is either 0 or 1. Derive the gradient of the regularized negative log likelihood $J_{j}$
for logistic regression, and give it in your report.}
\\~\\
{The derivative of the gradient can be found by computing $\frac{\partial J_{j}(\theta)}{\partial \theta_{i}}$ over the surrogate loss for point j, as given by $x^{(j)}$, $y^{(i)}$. The derivative then is as follows:
$$\frac{\partial J_{j}(\theta)}{\partial \theta_{i}} = x^{j}((1+exp(z))^{-1} - y^{(j)}) + 2.\alpha.\theta_{i} $$}
\\~\\
{\bf (e) Complete your train.m function to perform stochastic gradient descent on the logistic loss function. This will require that you fill in:}
\\~\\
{\bf (1) computing the surrogate loss function at each iteration ($J = 1/m\sum J_{j}$)}
\begin{lstlisting}[language=Matlab, caption=Calculating surrogate loss at each point.]
iter=1; Jsur=zeros(1,stopIter); J01=zeros(1,stopIter); done=0;
while (~done)
  step = stepsize/iter;               % update step-size and evaluate current loss values
  Jsur(iter) = inf;
  wtsTrans = (obj.wts * obj.wts')';
  Jsur(iter) = mean(-Y .* log(logistic(obj, X)) - (1 - Y) .* log(1 - logistic(obj, X)) + reg * sum(wtsTrans));
  J01(iter) = err(obj, X, Yin);
\end{lstlisting}
~\\
{\bf (2) computing the prediction and gradient associated with each data point $x^{(i)}, j^{(i)}$;}
\begin{lstlisting}[language=Matlab, caption=Compute linear responses and activation for data point j.]
    % Compute linear responses and activation for data point j
    y = logistic(obj,X(j,:));
\end{lstlisting}
~\\
{\bf (3) a gradient step on the parameters θ;}
\begin{lstlisting}[language=Matlab, caption=Compute gradient.]
    % Compute gradient:
    gradient = X1(j,:) * (y - Y(j)) + 2 * reg * obj.wts;

    obj.wts = obj.wts - step * gradient;      % take a step down the gradient
\end{lstlisting}
~\\
{\bf (4) a stopping criterion (usually either stopIter iterations or that J has not changed by more than stopTol since the last iteration through all the data).}
\begin{lstlisting}[language=Matlab, caption=Implement stopping criterion.]
  %   done = false;
  deltaJ = mean(-Y .* log(logistic(obj, X)) - (1 - Y) .* log(1 - logistic(obj, X)) + reg * obj.wts * obj.wts');
  if (iter == stopIter || abs(deltaJ - Jsur(iter)) < stopTol)
    done = true;
  end;
\end{lstlisting}
~\\
{\bf (f) Run your logistic regression classifier on both data sets (A and B); for this problem, use no regularization ($\alpha = 0$). Describe your parameter choices (stepsize, etc.) and show a plot of both the convergence of the surrogate loss and error rate, and a plot of the final converged classifier with the data (using e.g. plotClassify2D). In your report, please also include the functions that you wrote (at minimum, train.m, but possibly a few small helper functions as well).}

\begin{lstlisting}[language=Matlab, caption=Full train.m implementation.]
function obj = train(obj, X, Y, varargin)
% obj = train(obj, Xtrain, Ytrain [, option,val, ...])  : train logistic classifier
%     Xtrain = [n x d] training data features (constant feature not included)
%     Ytrain = [n x 1] training data classes
%     'stepsize', val  => step size for gradient descent [default 1]
%     'stopTol',  val  => tolerance for stopping criterion [0.0]
%     'stopIter', val  => maximum number of iterations through data before stopping [1000]
%     'reg', val       => L2 regularization value [0.0]
%     'init', method   => 0: init to all zeros;  1: init to random weights;
% Output:
%   obj.wts = [1 x d+1] vector of weights; wts(1) + wts(2)*X(:,1) + wts(3)*X(:,2) + ...

  [n,d] = size(X);            % d = dimension of data; n = number of training data

  % default options:
  plotFlag = true;
  init     = [];
  stopIter = 100; % Lowered to improve performance. Initially 1000.
  stopTol  = -1;
  reg      = 0.0;
  stepsize = 1;

  i=1;                                       % parse through various options
  while (i<=length(varargin)),
    switch(lower(varargin{i}))
    case 'plot',      plotFlag = varargin{i+1}; i=i+1;   % plots on (true/false)
    case 'init',      init     = varargin{i+1}; i=i+1;   % init method
    case 'stopiter',  stopIter = varargin{i+1}; i=i+1;   % max # of iterations
    case 'stoptol',   stopTol  = varargin{i+1}; i=i+1;   % stopping tolerance on surrogate loss
    case 'reg',       reg      = varargin{i+1}; i=i+1;   % L2 regularization
    case 'stepsize',  stepsize = varargin{i+1}; i=i+1;   % initial stepsize
    end;
    i=i+1;
  end;

  X1    = [ones(n,1), X];     % make a version of training data with the constant feature

  Yin = Y;                              % save original Y in case needed later
  obj.classes = unique(Yin);
  if (length(obj.classes) ~= 2) error('This logistic classifier requires a binary classification problem.'); end;
  Y(Yin==obj.classes(1)) = 0;
  Y(Yin==obj.classes(2)) = 1;           % convert to classic binary labels (0/1)

  if (~isempty(init) || isempty(obj.wts))   % initialize weights and check for correct size
    obj.wts = randn(1,d+1);
  end;
  if (any( size(obj.wts) ~= [1 d+1]) ) error('Weights are not sized correctly for these data'); end;
  wtsold = 0*obj.wts+inf;

% Training loop (SGD):
iter=1; Jsur=zeros(1,stopIter); J01=zeros(1,stopIter); done=0;
while (~done)
  step = stepsize/iter;               % update step-size and evaluate current loss values
  Jsur(iter) = inf;
  wtsTrans = (obj.wts * obj.wts')';
  Jsur(iter) = mean(-Y .* log(logistic(obj, X)) - (1 - Y) .* log(1 - logistic(obj, X)) + reg * sum(wtsTrans));
  J01(iter) = err(obj, X, Yin);

  if (plotFlag), switch d,            % Plots to help with visualization
    case 1, fig(2); plot1DLinear(obj,X,Yin);  %  for 1D data we can display the data and the function
    case 2, fig(2); plot2DLinear(obj,X,Yin);  %  for 2D data, just the data and decision boundary
    otherwise, % no plot for higher dimensions... %  higher dimensions visualization is hard
  end; end;
  fig(1); semilogx(1:iter, Jsur(1:iter),'b-',1:iter,J01(1:iter),'g-'); drawnow;

  for j=1:n,
    % Compute linear responses and activation for data point j
    y = logistic(obj,X(j,:));

    % Compute gradient:
    gradient = X1(j,:) * (y - Y(j)) + 2 * reg * obj.wts;

    obj.wts = obj.wts - step * gradient;      % take a step down the gradient
  end;

  %   done = false;
  deltaJ = mean(-Y .* log(logistic(obj, X)) - (1 - Y) .* log(1 - logistic(obj, X)) + reg * obj.wts * obj.wts');
  if (iter == stopIter || abs(deltaJ - Jsur(iter)) < stopTol)
    done = true;
  end;

  wtsold = obj.wts;
  iter = iter + 1;
end;
\end{lstlisting}
{ As $\alpha$ was set to 0, $stepsize$ was set to 1 and number of iterations was set to 100. Beyond 100 iterations, computation time became too high for minimal improvements to the error  rate and surrogate loss. The logistic regression classifier was then run on both classes:}
\begin{lstlisting}[language=Matlab, caption=Code to run logistic regression classifier on both data sets.]
% Train Class A
train(learner, XA, YA);
legend('Error Rate', 'Surrogate Loss');
% Plot final converged classifier decision boundaries.
figure();
plotClassify2D(learner, XA, YA);

% Train Class B
train(learner, XB, YB);
legend('Error Rate', 'Surrogate Loss');
% Plot final converged classifier decision boundaries.
figure();
plotClassify2D(learner, XB, YB);
\end{lstlisting}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_1.png}\\
	{Figure 2.6.1: Plot of convergence of error rate and surrogate loss for Class A.}
\end{center}
\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_2.png}\\
	{Figure 2.6.2: Plot of final converged classifier for Class A.}
\end{center}

\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_3.png}\\
	{Figure 2.6.3: Plot of final converged classifier decision boundaries for Class A.}
\end{center}

\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_4.png}\\
	{Figure 2.6.4: Plot of convergence of error rate and surrogate loss for Class B.}
\end{center}

\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_5.png}\\
	{Figure 2.6.5: Plot of final converged classifier for Class B.}
\end{center}

\begin{center}
	\includegraphics[width=30em,keepaspectratio]{2_6_Figure_6.png}\\
	{Figure 2.6.6: Plot of final converged classifier decision boundaries for Class B.}
\end{center}

\end{document}


\documentclass{article}
\usepackage{tocloft}
\include{common_symbols_and_format}
\renewcommand{\cfttoctitlefont}{\Large\bfseries}

\begin{document}
\logo
\rulename{Regression of Technical Signal} %Argument is name of rule
\tblofcontents

\ruledescription{Technical indicators are metrics used by traders to determine opportune times to enter and exit trades. To use a technical indicator for trading you both need to know how to calculate the indicator and the interpretation of how the resultant signal should be used. For a given technical indicator, different traders may agree on how to calculate the signal and the indicator's definition, but disagree on how to use the signal for trading. As a security's price history is widely available to all market participants, popular technical strategies can become crowded. \\ \\ This strategy is a regression based approach to forecasting price change in assets using a signal generated from a technical indicator. This particular strategy, Regression of Technical Signal, uses a simple mathematical regression (120 step length) to size trade positioning based on the strength of the recent relationship between the technical indicator and future price changes. The first stage is to calculate the technical indicator: for this InferTrade uses the versions of technical indicators defined by the \href{https://pypi.org/project/ta/}{TA} open source library. For the second stage - interpretation of when to trade - InferTrade regresses the history of returns in the price series against the 1-step lagged level of the technical indicator. The trade size taken then uses a simplified version of the Kelly Criterion, calculating investment size as a ratio of the expected return for the next time step to a fixed assumption of volatility. The resultant trading rule is therefore sensitive to the type of market conditions evaluated by the technical signal, but uses a statistical approach for implementation and sizing, rather than more conventional interpretations. As a single input signal is used, this also means technical indicators with multiple signal time series - such as Bollinger Bands, with lower and upper bands - must have one of the signals selected for this strategy. As the window length used for the regression is fixed (120 steps) and the Kthe strategy will have the same number of free parameters as the signal creation function itself. \\ \\ This form of regression strategy provides a way to evaluate the raw signals different technical indicators for information content in a consistent way, using them as potential predictive features rather than following their usual implementation recipe. InferTrade also supports the usual implementations for some technical indicators but these would be separate strategies. The rationale for also offering these regression-based approach is from two benefits: 1) regression sized technical rules give continuous size recommendations, rather than discrete entry/exit points. This makes them easier to optimize, as small changes in the parameters result in small change in positions and thus performance. Some technical indicator interpretations take discontinuous positions, whereby a small change results in a large position taken, which can increase overfitting during optimisation. 2) regression sized technical rules embed automatic adjustment for regimes where the strategy is not performing - if the technical indicator stops providing a significant prediction of price changes then the sized positions will be minimal.}

\howtotrade
{Any of the technical indicators supported by InferTrade can be used as signal to forecast price change in asset. \\ \\ As an example we can consider Money Flow Index. MFI indicator is a momentum indicator which varies between 0 and 100 and is used to define an overbought and oversold conditions. Usually MFI value above 80 is considered overbought and a sell signal is generated while a value below 20 is considered oversold and a buy signal is generated.\\ \\
By using this trading strategy we compute the MFI of underlying asset using it's historical High, Low, Close \& Volume. Instead of using the MFI value to generate buy/sell signal as explained above, a forecast is made for expected change in price using MFI as a signal governed by Equation \ref{eqn:SignalRegression}. $\beta$ coefficient determines the relationship and impact of Signal $S$ on change in price of asset $\Delta \price$.\\ \\
Forecast generated as described above is then used to generate fractional portfolio investment using formulation mentioned in Equation \ref{eqn:portfolioallocation}.
}

\ruleparameters %You can include however many arguments (in groups of 4) as you want!
{Regression Period}{120}{Previous data points used to fit a regression line.}{\lookbacklength}
{Kelly fraction}{1.0}{Amplitude weighting. $1.0$ is maximum growth if regression is exact.
$<1.0$ scales down positions taken.}{$\kellyfraction$}
{Volatility}{0.1}{Volatility used to compute the Kelly recommended optimum.}{$\sigma$}
\stoptable %must be included or Tex engine runs infinitely

\section{Equation}
Below is the equations which govern how this specific trading rule calculates a trading position. Equation 1 is the regression equation used to determine $\beta$ and $c$. Equation 2 is the subsequent calculation of position sizing.

\begin{equation}
\label{eqn:SignalRegression}
\Delta \price_{\currenttime} = \beta \, S_{\currenttime-1} + c + \varepsilon_{\currenttime}
\end{equation}
\begin{equation}
\label{eqn:portfolioallocation}
    \position_{\currenttime} = \kellyfraction \, \frac{E[\Delta \price_{\currenttime}]}{\sigma^{2}} = \kellyfraction \, \frac{S_{\currenttime-1} + c}{\sigma^{2}}
\end{equation}
\\
with:

$\Delta \price_{\currenttime}:$ is the change in asset price at time $\currenttime$.

$E[\Delta \price_{\currenttime}]$: is the expected change in price at time $\currenttime$.

$S_{\currenttime-1}$: is the signal value at time $\currenttime-1$.

$\beta$: is the relationship coefficient between $S_{\currenttime-1}$ and $\Delta \price_{\currenttime}$.

$c$: is a constant bias.

$\varepsilon_{\currenttime}$: is the error term.

$\kellyfraction$: is the Kelly fraction.

$\position_{\currenttime}$: is the resultant fractional portfolio investment at time \currenttime.

\keyterms
\furtherlinks %The footer
\end{document}


\documentclass{beamer}

\usepackage{beamerthemevictor,comment,verbatim,graphicx,amssymb}
\usepackage[noeepic]{qtree}

\input{tutmacs}
\input{slidemacs}
\input idxmacs

\begin{document}

\title{Hashing}
\author{Victor Eijkhout}
\date{Notes for CS 594 -- Fall 2004}

\frame{\titlepage}

\section{Introduction}

\frame{
  \frametitle{The basic problem}
Storing names and information about them:\\
associative storage
}

\frame{
\frametitle{Issues}
\begin{itemize}
\item<2-> Insertion
\item<3-> Retrieval
\item<4-> Deletion
\end{itemize}
}

\frame{
  \frametitle{Simple strategies}
\begin{itemize}
\item List in order of creation
\item<2-> $\Rightarrow$ Cheap to create, linear search time, linear deletion
\item<3-> Sorted list
\item<4-> $\Rightarrow$ Creation in linear time, search logarithmic,
  deletion linear
\item<5-> Linear list
\item<6-> $\Rightarrow$ all linear time
\end{itemize}
}

\frame{
  \frametitle{one more strategy}
\begin{quote}
\Tree [.$\bullet$ [.B ART [.E GIN LL ] ] [.E LSE ND ] ]
\end{quote}
\begin{itemize}
\item<2-> $\Rightarrow$ all linear in length of string
\end{itemize}
}

\sectionframe{Hash functions}

\frame[containsverbatim]{
  \frametitle{}
\begin{itemize}
\item Mapping from space of words to space of indices
\item Source: unbounded; in practice not extremely large
\item Target: array (static/dynamic)
\end{itemize}
}

\frame{
  \frametitle{Requirements}
\begin{itemize}
\item<2-> Function determined only by input data
\item<3-> Determined by as much of the data as possible\\
\n{key1}, \n{key2},\dots
\item<4-> Uniform distribution (clustering bad, collisions really bad)
\item<5-> Similar data, mapped far apart
\end{itemize}
}

\subsection{Modulo operations}

\frame[containsverbatim]{
  \frametitle{Good idea: prime numbers}
With $M$ size of the hash table:
\begin{equation}
    h(K) = K\mod M, \end{equation}
or:
\begin{equation}
    h(K) = aK\mod M,\end{equation}
}

\frame[containsverbatim]{
  \frametitle{Bad examples:}
\begin{itemize}
\item $M$~is even, say~$M=2M'$,\\ $r=K\mod M$ say~$K=nM+r$
then
\begin{eqnarray*}
K=2K'&\Rightarrow& r=2(nM'-K')\\
K=2K'+1&\Rightarrow& r=2(nM'-K')+1
\end{eqnarray*}
so key even iff number $\Rightarrow$~dependence on last digit
\item $M$ multiple of three: anagrams map to same key (sum of digits)
\item $\Rightarrow$ $M$ prime, far away from powers of 2
\end{itemize}
}

\frame[containsverbatim]{
  \frametitle{Multiplication instead of division}
\begin{itemize}
\item $r=K\mod M = M\big((K/M)\mod 1\big)$
\item $A\approx w/M$, where $w$ maxint
\item Then $1/M=A/w$, ($A$ with decimal point to its left).
\item from %K\mod M=M(K/M\mod 1)$:
\[ h(K)=\lfloor M\left(\left({A\over w}K\right)\mod
1\right)\rfloor. \]
\end{itemize}
}

\frame[containsverbatim]{
  \frametitle{Example: Bible}
\begin{itemize}
\item 42,829 unique words,
\item into a hash table with 30,241 elements (prime): 76.6\% used
\item table of size: 30,240 (divisible by 2--9): 60.7\% used
\item (collisions discussed later)
\end{itemize}
}

\subsection{Character hashing}

\frame[containsverbatim]{
  \frametitle{Two-step hashing}
\begin{itemize}
\item Mix up characters of the key
\item then modulo with table size
\end{itemize}
}

\frame[containsverbatim]{
  \frametitle{Character based hashing}
\begin{verbatim}
  h = <some value>
  for (i=0; i<len(var); i++)
    h = h + <byte i of string>;
\end{verbatim}
prevent anagram problem:
\begin{verbatim}
  h = <some value>
  for (i=0; i<len(var); i++)
    h = Rand( h + <byte i of string> );
\end{verbatim}
with table of random numbers; also function possible
}

\frame[containsverbatim]{
  \frametitle{ELF hash}
\begin{verbatim}
/* UNIX ELF hash
 * Published hash algorithm used in the UNIX ELF format
 * for object files
 */
unsigned long hash(char *name)
{
    unsigned long h = 0, g;

    while ( *name ) {
        h = ( h << 4 ) + *name++;
        if ( g = h & 0xF0000000 )
          h ^= g >> 24;
        h &= ~g;
    }
}
\end{verbatim}
}

\frame[containsverbatim]{
  \frametitle{Another hash function}
\begin{verbatim}
/* djb2
 * This algorithm was first reported by Dan Bernstein
 * many years ago in comp.lang.c
 */
unsigned long hash(unsigned char *str)
{
    unsigned long hash = 5381;
    int c;
    while (c = *str++) hash = ((hash << 5) + hash) + c;
    return hash;
}
\end{verbatim}
}

\sectionframe{Hash tables: collisions}

\frame[containsverbatim]{
  \frametitle{So far so good}
\pgfimage[height=2in]{hash-direct}
}

\frame[containsverbatim]{
  \frametitle{Collisions}
\begin{itemize}
\item $k_1\not=k_2$, $h(k_1)=h(k_2)$
\item several strategies; all analysis statistical in nature
\item open hash table: solve conflict outside the table
\item closed hash table: solve by moving around in the table
\end{itemize}
}

\frame[containsverbatim]{
  \frametitle{Separate chaining}
\pgfimage[height=2in]{hash-separate}
}

\subsection{Open hash table}

\frame[containsverbatim]{
  \frametitle{}
\begin{itemize}
\item Pro: no need for searching through hash table
\item Con: dynamic storage
\item Also: $M$ large to prevent collisions $\Rightarrow$~wasted space
\end{itemize}
}

\subsection{Closed hash table}

\frame[containsverbatim]{
  \frametitle{Linear probing}
\pgfimage[height=2in]{hash-linear}

Location occupied: search linearly from first hash
}

\frame[containsverbatim]{
\footnotesize
\begin{verbatim}
addr = Hash(K);
if (IsEmpty(addr)) Insert(K,addr);
else {
    /* see if already stored */
  test:
    if (Table[addr].key == K) return;
    else {
      addr = Table[addr].link; goto test;}
    /* find free cell */
    Free = addr;
    do { Free--; if (Free<0) Free=M-1; }
    while (!IsEmpty(Free) && Free!=addr)
    if (!IsEmpty(Free)) abort;
    else {
      Insert(K,Free); Table[addr].link = Free;}
}
\end{verbatim}
}

\frame{
  \frametitle{Merging blocks in linear probing}
\pgfimage[height=1in]{probing1}
\pgfimage[height=1in]{probing2}
\pgfimage[height=1in]{probing3}
}

\frame[containsverbatim]{
  \frametitle{Linear probing analysis}
\begin{itemize}
\item  Clusters forming
\item Particularly bad: merging clusters
\item Ratio occupied/total: $\alpha=N/M$\\
expected search time
\[ T\approx\begin{cases}
    {1\over 2}\left(1+\left({1\over 1-\alpha}\right)^2\right)&
        unsuccessful\\
    {1\over 2}\left(1+{1\over 1-\alpha}\right)&
        successful
    \end{cases}
\]
\item $\Rightarrow$ increasing as table fills up
\end{itemize}
}

\subsection{Chaining}

\frame[containsverbatim]{
\frametitle{Chaining}
\pgfimage[height=1in]{hash-chain0}
\pgfimage[height=1in]{hash-chain}
\pgfimage[height=1in]{hash-hash2}

If location occupied, search from top of table
}

\frame[containsverbatim]{
\footnotesize
\begin{verbatim}
addr = Hash(K); Free = M-1;
if (IsEmpty(addr)) Insert(K,addr);
else {
    /* see if already stored */
  test:
    if (Table[addr].key == K) return;
    else {
      addr = Table[addr].link; goto test;}
    /* find free cell */
    do { Free--; }
    while (!IsEmpty(Free)
    if (Free<0) abort;
    else {
      Insert(K,Free); Table[addr].link = Free;}
}
\end{verbatim}
}

\frame[containsverbatim]{
  \frametitle{Chaining analysis}
\begin{itemize}
\item No clusters merging
\item Coalescing lists
\item Search time ($\alpha$~occupied fraction)
\[ T\approx\begin{cases}
        1+(e^{2\alpha}-1-2\alpha)/4&unsuccessful\\
        1+(e^{2\alpha}-1-2\alpha)/8\alpha+\alpha/4&successful
           \end{cases}
\]
\end{itemize}
}

\frame[containsverbatim]{
  \frametitle{Nonlinear rehashing}
\begin{itemize}
\item `Random probing': Try $(h(m)+p_i)\mod s$, where $p_i$~is a
  sequence of random numbers (stored)\\
  prevent secondary collisions
\item `Add the hash': Try $(i\times h(m))\mod s$. ($s$~prime)
\item Pro: scattered hash keys
\item Con: more calculations, worse memory locality
\end{itemize}
}

\section{Other}
\subsection{Deletion}

\frame[containsverbatim]{
  \frametitle{Deleting keys}
\begin{itemize}
\item Simple in direct chaining
\item Very hard in closed hash table methods: can only mark `unused'
\end{itemize}
}

\subsection{Examples}

\frame[containsverbatim]{
  \frametitle{Search in chess programs}
\begin{itemize}
\item Problem: evaluation board positions
\item if position arrived in two ways, no two calculations
\item Solution: hash the board, use as key in table of evaluations
\item Collisions?
\end{itemize}
}

\frame{
  \frametitle{String searching}
\begin{itemize}
\item Problem: does string (length~$M$) occur in document (length~$N$)
\item naive: $N$ comparisons, giving $O(MN)$ complexity
\item solution: hash the strings, compare hash values
\item (hash function does not distinguish between anagrams)
\[ h(k)=\left\{\sum_ik[i]\right\}\mathop{\mathrm{mod}} K\]
\item<2-> cheap updating of the document hash key:
\[ h(t[2\ldots n+1]) = h(t[1\ldots n])+t[n+1]-t[1] \]
(with addition/subtraction modulo~$K$)
\item string comparison in $O(1)$, $\Rightarrow$~total cost~$O(M+N)$
\end{itemize}
}

\subsectionframe{Discussion}

\frame{
  \frametitle{Hash table vs trees}
\begin{itemize}
\item<2-> Best case search time can be equal: harder to implement in
  trees
\item<3-> Trees can become unbalanced: considerable time and effort to
  balance
\item<4-> Threes have dynamic storage: harder to code optimally; worse
  memory locality
\end{itemize}
}

\frame{
  \frametitle{Open vs closed hash tables}
\begin{itemize}
\item<2-> Approximately equal performance until the table fills up
\item<3-> Open: much simpler storage management, especially deletion
\end{itemize}
}

\end{document}


\section{Miscellaneous}
\subsection{VS Code Config}
\lstinputlisting{./source/01_Miscellaneous/00_VS Code Config.txt}
\subsection{Day of Date}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/01_Day of Date.cpp}
\subsection{Number of Days since 1-1-1}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/02_Number of Days since 1-1-1.cpp}
\subsection{Enumerate Subsets of a Bitmask}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/03_Enumerate Subsets of a Bitmask.cpp}
\subsection{Fast IO}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/04_Fast IO.cpp}
\subsection{Int to Roman}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/05_Int to Roman.cpp}
\subsection{Josephus Problem}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/06_Josephus Problem.cpp}
\subsection{Random Primes}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/07_Random Primes.cpp}
\subsection{RNG}
\lstinputlisting[language=c++]{./source/01_Miscellaneous/08_RNG.cpp}
\section{Data Structures}
\subsection{2D Segment Tree}
\lstinputlisting[language=c++]{./source/02_Data Structures/01_2D Segment Tree.cpp}
\subsection{Fenwick RU-RQ}
\lstinputlisting[language=c++]{./source/02_Data Structures/02_Fenwick RU-RQ.cpp}
\subsection{Heavy-Light Decomposition}
\lstinputlisting[language=c++]{./source/02_Data Structures/03_Heavy-Light Decomposition.cpp}
\subsection{Li-Chao Tree}
\lstinputlisting[language=c++]{./source/02_Data Structures/04_Li-Chao Tree.cpp}
\subsection{Persistent Segment Tree}
\lstinputlisting[language=c++]{./source/02_Data Structures/05_Persistent Segment Tree.cpp}
\subsection{STL PBDS}
\lstinputlisting[language=c++]{./source/02_Data Structures/06_STL PBDS.cpp}
\subsection{Treap}
\lstinputlisting[language=c++]{./source/02_Data Structures/07_Treap.cpp}
\subsection{Unordered Map Custom Hash}
\lstinputlisting[language=c++]{./source/02_Data Structures/08_Unordered Map Custom Hash.cpp}
\subsection{Mo's on Tree}
\input{./source/02_Data Structures/09_Mo's on Tree.tex}
\subsection{Link-Cut Tree}
\lstinputlisting[language=c++]{./source/02_Data Structures/10_Link-Cut Tree.cpp}
\section{Dynamic Programming}
\subsection{DP Convex Hull}
\lstinputlisting[language=c++]{./source/03_Dynamic Programming/01_DP Convex Hull.cpp}
\subsection{DP DNC}
\lstinputlisting[language=c++]{./source/03_Dynamic Programming/02_DP DNC.cpp}
\subsection{DP Knuth-Yao}
\lstinputlisting[language=c++]{./source/03_Dynamic Programming/03_DP Knuth-Yao.cpp}
\section{Geometry}
\subsection{Geometry Template}
\lstinputlisting[language=c++]{./source/04_Geometry/01_Geometry Template.cpp}
\subsection{Convex Hull}
\lstinputlisting[language=c++]{./source/04_Geometry/02_Convex Hull.cpp}
\subsection{Closest Pair of Points}
\lstinputlisting[language=c++]{./source/04_Geometry/03_Closest Pair of Points.cpp}
\subsection{Smallest Enclosing Circle}
\lstinputlisting[language=c++]{./source/04_Geometry/04_Smallest Enclosing Circle.cpp}
\subsection{Sutherland-Hodgman Algorithm}
\lstinputlisting[language=c++]{./source/04_Geometry/05_Sutherland-Hodgman Algorithm.cpp}
\subsection{Centroid of Polygon}
\input{./source/04_Geometry/06_Centroid of Polygon.tex}
\subsection{Pick Theorem}
\input{./source/04_Geometry/07_Pick Theorem.tex}
\section{Graphs}
\subsection{Articulation Point and Bridge}
\lstinputlisting[language=c++]{./source/05_Graphs/01_Articulation Point and Bridge.cpp}
\subsection{SCC and Strong Orientation}
\lstinputlisting[language=c++]{./source/05_Graphs/02_SCC and Strong Orientation.cpp}
\subsection{Centroid Decomposition}
\lstinputlisting[language=c++]{./source/05_Graphs/03_Centroid Decomposition.cpp}
\subsection{Dinic's Maximum Flow}
\lstinputlisting[language=c++]{./source/05_Graphs/04_Dinic's Maximum Flow.cpp}
\subsection{Minimum Cost Maximum Flow}
\lstinputlisting[language=c++]{./source/05_Graphs/05_Minimum Cost Maximum Flow.cpp}
\subsection{Flows with Demands}
\input{./source/05_Graphs/06_Flows with Demands.tex}
\subsection{Hungarian}
\lstinputlisting[language=c++]{./source/05_Graphs/07_Hungarian.cpp}
\subsection{Edmonds' Blossom}
\lstinputlisting[language=c++]{./source/05_Graphs/08_Edmonds' Blossom.cpp}
\subsection{Eulerian Path or Cycle}
\lstinputlisting[language=c++]{./source/05_Graphs/09_Eulerian Path or Cycle.cpp}
\subsection{Hierholzer's Algorithm}
\lstinputlisting[language=c++]{./source/05_Graphs/10_Hierholzer's Algorithm.cpp}
\subsection{2-SAT}
\lstinputlisting[language=c++]{./source/05_Graphs/11_2-SAT.cpp}
\section{Math}
\subsection{Extended Euclidean GCD}
\lstinputlisting[language=c++]{./source/06_Math/01_Extended Euclidean GCD.cpp}
\subsection{Generalized CRT}
\lstinputlisting[language=c++]{./source/06_Math/02_Generalized CRT.cpp}
\subsection{Generalized Lucas Theorem}
\lstinputlisting[language=c++]{./source/06_Math/03_Generalized Lucas Theorem.cpp}
\subsection{Linear Diophantine}
\lstinputlisting[language=c++]{./source/06_Math/04_Linear Diophantine.cpp}
\subsection{Modular Linear Equation}
\lstinputlisting[language=c++]{./source/06_Math/05_Modular Linear Equation.cpp}
\subsection{Miller-Rabin and Pollard's Rho}
\lstinputlisting[language=c++]{./source/06_Math/06_Miller-Rabin and Pollard's Rho.cpp}
\subsection{Berlekamp-Massey}
\lstinputlisting[language=c++]{./source/06_Math/07_Berlekamp-Massey.cpp}
\subsection{Fast Fourier Transform}
\lstinputlisting[language=c++]{./source/06_Math/08_Fast Fourier Transform.cpp}
\subsection{Number Theoretic Transform}
\lstinputlisting[language=c++]{./source/06_Math/09_Number Theoretic Transform.cpp}
\subsection{Gauss-Jordan}
\lstinputlisting[language=c++]{./source/06_Math/10_Gauss-Jordan.cpp}
\subsection{Derangement}
\lstinputlisting[language=c++]{./source/06_Math/12_Derangement.cpp}
\subsection{Bernoulli Number}
\input{./source/06_Math/13_Bernoulli Number.tex}
\subsection{Forbenius Number}
\input{./source/06_Math/14_Forbenius Number.tex}
\subsection{Stars and Bars with Upper Bound}
\input{./source/06_Math/15_Stars and Bars with Upper Bound.tex}
\subsection{Arithmetic Sequences}
\input{./source/06_Math/16_Arithmetic Sequences.tex}
\subsection{FWHT}
\lstinputlisting[language=c++]{./source/06_Math/17_FWHT.cpp}
\section{Strings}
\subsection{Aho-Corasick}
\lstinputlisting[language=c++]{./source/07_Strings/01_Aho-Corasick.cpp}
\subsection{Eertree}
\lstinputlisting[language=c++]{./source/07_Strings/02_Eertree.cpp}
\subsection{Manacher's Algorithm}
\lstinputlisting[language=c++]{./source/07_Strings/03_Manacher's Algorithm.cpp}
\subsection{Suffix Array}
\lstinputlisting[language=c++]{./source/07_Strings/04_Suffix Array.cpp}
\subsection{Suffix Automaton}
\lstinputlisting[language=c++]{./source/07_Strings/05_Suffix Automaton.cpp}
\section{OEIS}
\subsection{A000108 (Catalan)}
\lstinputlisting{./source/08_OEIS/01_A000108 (Catalan).txt}
\subsection{A000127}
\lstinputlisting{./source/08_OEIS/02_A000127.txt}
\subsection{A001434}
\lstinputlisting{./source/08_OEIS/04_A001434.txt}
\subsection{A018819}
\lstinputlisting{./source/08_OEIS/05_A018819.txt}
\subsection{A092098}
\lstinputlisting{./source/08_OEIS/06_A092098.txt}