\title{Ideals, Varieties and \Mtwo}
\titlerunning{Ideals, Varieties and \Mtwo}
\toctitle{Ideals, Varieties and \Mtwo}
\author{Bernd Sturmfels\thanks{Partially supported by
the National Science Foundation (DMS-9970254).}}
\authorrunning{B. Sturmfels}
% \institute{University of California at Berkeley, Department of Mathematics,
%   Berkeley, CA 94720, USA}
\maketitle

\begin{abstract}
This chapter introduces \Mtwo commands for
some elementary computations in algebraic geometry.
Familiarity with Gr\"obner bases is assumed.
\end{abstract}

Many students and researchers alike have their first encounter with
Gr\"ob\-ner bases through the delightful text books \cite{CLO1} and \cite{CLO2}
by David Cox, John Little and Donal O'Shea. This chapter illustrates
the use of \Mtwo for some computations discussed in these books.
It can be used as a supplement for an advanced undergraduate course or
first-year graduate course in computational algebraic geometry. The
mathematically advanced reader will find this chapter a useful summary
of some basic \Mtwo commands.

\section{A Curve in Affine Three-Space}

Our first example concerns geometric objects in
(complex) affine 3-space. We start by
setting up the ring of polynomial functions with rational coefficients.
\beginOutput
i1 : R = QQ[x,y,z]\\
\emptyLine
o1 = R\\
\emptyLine
o1 : PolynomialRing\\
\endOutput
Various monomial orderings are available in \Mtwo; since we did not specify
one explicitly, the monomials in the ring ${\tt R}$ will be sorted in
graded reverse lexicographic order  \cite[\S I.2, Definition 6]{CLO1}.
We define an ideal generated by two polynomials
in this ring and assign it to the variable named
{\tt curve}.

\beginOutput
i2 : curve = ideal( x^4-y^5, x^3-y^7 )\\
\emptyLine
\               5    4     7    3\\
o2 = ideal (- y  + x , - y  + x )\\
\emptyLine
o2 : Ideal of R\\
\endOutput
We compute the reduced Gr\"obner basis of our ideal:
\beginOutput
i3 : gb curve\\
\emptyLine
o3 = | y5-x4 x4y2-x3 x8-x3y3 |\\
\emptyLine
o3 : GroebnerBasis\\
\endOutput
By inspecting leading terms (and using \cite[\S 9.3, Theorem 8]{CLO1}),
we see that our ideal {\tt curve} does indeed
define a one-dimensional affine variety. This can be tested directly
with the following commands in \Mtwo:
\beginOutput
i4 : dim curve\\
\emptyLine
o4 = 1\\
\endOutput
\beginOutput
i5 : codim curve\\
\emptyLine
o5 = 2\\
\endOutput
The {\it degree} of a curve in complex affine $3$-space is the
number of intersection points with a general plane. It coincides
with the degree  \cite[\S 6.4]{CLO2} of the projective closure
\cite[\S 8.4]{CLO1} of our curve, which we compute as follows:
\beginOutput
i6 : degree curve\\
\emptyLine
o6 = 28\\
\endOutput
The Gr\"obner basis in {\tt o3} contains two polynomials which are not
irreducible: they contain a factor of $x^3$. This shows that our curve
is not irreducible over ${\bf Q}$. We first extract the components
which are transverse to the plane $x=0$:
\beginOutput
i7 : curve1 = saturate(curve,ideal(x))\\
\emptyLine
\               2       5    4   5    3\\
o7 = ideal (x*y  - 1, y  - x , x  - y )\\
\emptyLine
o7 : Ideal of R\\
\endOutput
And next we extract the component which lies in the plane $x=0$:
\beginOutput
i8 : curve2 = saturate(curve,curve1)\\
\emptyLine
\             3   5\\
o8 = ideal (x , y )\\
\emptyLine
o8 : Ideal of R\\
\endOutput
The second component is a multiple line. Hence our input ideal was not radical.
To test equality of ideals we use the command {\tt ==}\indexcmd{==} .
\beginOutput
i9 : curve == radical curve\\
\emptyLine
o9 = false\\
\endOutput
We now replace our curve by its first component:
\beginOutput
i10 : curve = curve1\\
\emptyLine
\                2       5    4   5    3\\
o10 = ideal (x*y  - 1, y  - x , x  - y )\\
\emptyLine
o10 : Ideal of R\\
\endOutput
\beginOutput
i11 : degree curve\\
\emptyLine
o11 = 13\\
\endOutput
The ideal of this curve is radical:
\beginOutput
i12 : curve == radical curve\\
\emptyLine
o12 = true\\
\endOutput
Notice that the variable ${\bf z}$ does not appear
among the generators of the ideal. Our curve consists of
$13$ straight lines (over {\bf C}) parallel to the {\tt z}-axis.

\section{Intersecting Our Curve With a Surface}

In this section we explore basic operations on ideals,
starting with those described in \cite[\S 4.3]{CLO1}.
Consider the following surface in affine $3$-space:
\beginOutput
i13 : surface = ideal( x^5 + y^5 + z^5 - 1)\\
\emptyLine
\             5    5    5\\
o13 = ideal(x  + y  + z  - 1)\\
\emptyLine
o13 : Ideal of R\\
\endOutput
The union of the curve and the surface is represented by the
intersection of their ideals:
\beginOutput
i14 : theirunion = intersect(curve,surface)\\
\emptyLine
\              6 2      7      2 5    5    5    5      2       5 5    1 $\cdot\cdot\cdot$\\
o14 = ideal (x y  + x*y  + x*y z  - x  - y  - z  - x*y  + 1, x y  + y  $\cdot\cdot\cdot$\\
\emptyLine
o14 : Ideal of R\\
\endOutput
In this example this coincides with the product of the two ideals:
\beginOutput
i15 : curve*surface == theirunion\\
\emptyLine
o15 = true\\
\endOutput
The intersection of the curve and the surface is represented by the
sum of their ideals. We get a finite set of points:
\beginOutput
i16 : ourpoints = curve + surface\\
\emptyLine
\                2       5    4   5    3   5    5    5\\
o16 = ideal (x*y  - 1, y  - x , x  - y , x  + y  + z  - 1)\\
\emptyLine
o16 : Ideal of R\\
\endOutput
\beginOutput
i17 : dim ourpoints\\
\emptyLine
o17 = 0\\
\endOutput
The number of points is sixty five:
\beginOutput
i18 : degree ourpoints\\
\emptyLine
o18 = 65\\
\endOutput
Each of the points is multiplicity-free:
\beginOutput
i19 : degree radical ourpoints\\
\emptyLine
o19 = 65\\
\endOutput
The number of points coincides with the number of
monomials not in the initial ideal \cite[\S 2.2]{CLO2}.
These are called the {\it standard monomials}.
\beginOutput
i20 : staircase = ideal leadTerm ourpoints\\
\emptyLine
\                2   5   5   5\\
o20 = ideal (x*y , z , y , x )\\
\emptyLine
o20 : Ideal of R\\
\endOutput
The {\tt basis} command can be used to list all the standard monomials
\beginOutput
i21 : T = R/staircase;\\
\endOutput
\beginOutput
i22 : basis T\\
\emptyLine
o22 = | 1 x x2 x3 x4 x4y x4yz x4yz2 x4yz3 x4yz4 x4z x4z2 x4z3 x4z4 x3y $\cdot\cdot\cdot$\\
\emptyLine
\              1       65\\
o22 : Matrix T  <--- T\\
\endOutput

The assignment of the quotient ring to the global variable {\tt T} had a side
effect: the variables {\tt x}, {\tt y}, and {\tt z} now have values in that
ring.
To bring the variables of {\tt R} to the fore again, we must say:
\beginOutput
i23 : use R;\\
\endOutput
Every polynomial function on our 65 points can be written uniquely
as a linear combination of these standard monomials. This
representation can be computed using the normal form command {\tt \%}\indexcmd{\%}.

\beginOutput
i24 : anyOldPolynomial = y^5*x^5-x^9-y^8+y^3*x^5\\
\emptyLine
\       5 5    9    5 3    8\\
o24 = x y  - x  + x y  - y\\
\emptyLine
o24 : R\\
\endOutput
\beginOutput
i25 : anyOldPolynomial {\char`\%} ourpoints\\
\emptyLine
\       4     3\\
o25 = x y - x y\\
\emptyLine
o25 : R\\
\endOutput
Clearly, the normal form is zero if and only the polynomial is in the ideal.
\beginOutput
i26 : anotherPolynomial = y^5*x^5-x^9-y^8+y^3*x^4\\
\emptyLine
\       5 5    9    8    4 3\\
o26 = x y  - x  - y  + x y\\
\emptyLine
o26 : R\\
\endOutput
\beginOutput
i27 : anotherPolynomial {\char`\%} ourpoints\\
\emptyLine
o27 = 0\\
\emptyLine
o27 : R\\
\endOutput

\section{Changing the Ambient Polynomial Ring}

During a \Mtwo session it sometimes becomes necessary to change the
ambient ring in which the computations takes place. Our original
ring, defined in {\tt i1}, is the polynomial ring in three variables
over the field  {\bf Q} of rational numbers
with the graded reverse lexicographic order. In this section
two modifications are made: first we replace the field of coefficients
by a finite field, and later we replace the  monomial order
by an elimination order.

An important operation in algebraic geometry is
the decomposition of algebraic varieties
into irreducible components \cite[\S 4.6]{CLO1}.
Algebraic algorithms for this purpose are based on the
{\it primary decomposition} of ideals \cite[\S 4.7]{CLO1}.
A future version of \Mtwo will have an implementation of
primary decomposition over any polynomial ring.
The current version of \Mtwo has a command
{\tt decompose} for finding all the minimal primes of an ideal,
but, as it stands, this works only over a finite field.

Let us change our coefficient field to the field with $101$ elements:
\beginOutput
i28 : R' = ZZ/101[x,y,z];\\
\endOutput

We next move our ideal from the previous section into the new ring
(fortunately, none of the coefficients of its generators have 101 in the
denominator):

\beginOutput
i29 : ourpoints' = substitute(ourpoints,R')\\
\emptyLine
\                2       5    4   5    3   5    5    5\\
o29 = ideal (x*y  - 1, y  - x , x  - y , x  + y  + z  - 1)\\
\emptyLine
o29 : Ideal of R'\\
\endOutput
\beginOutput
i30 : decompose ourpoints'\\
\emptyLine
\                                                                       $\cdot\cdot\cdot$\\
o30 = \{ideal (z + 36, y - 1, x - 1), ideal (z + 1, y - 1, x - 1), idea $\cdot\cdot\cdot$\\
\emptyLine
o30 : List\\
\endOutput
Oops, that didn't fit on the display, so let's print them out one per line.
\beginOutput
i31 : oo / print @@ print;\\
ideal (z + 36, y - 1, x - 1)\\
\emptyLine
ideal (z + 1, y - 1, x - 1)\\
\emptyLine
ideal (z - 6, y - 1, x - 1)\\
\emptyLine
ideal (z - 14, y - 1, x - 1)\\
\emptyLine
ideal (z - 17, y - 1, x - 1)\\
\emptyLine
\        3      2              2                      3    2     2      $\cdot\cdot\cdot$\\
ideal (x  - 46x  + 28x*y - 27y  + 46x + y + 27, - 16x  + x y + x  - 15 $\cdot\cdot\cdot$\\
\emptyLine
\            2                                            2             $\cdot\cdot\cdot$\\
ideal (- 32x  - 16x*y + x*z - 16x - 27y - 30z - 14, - 34x  - 14x*y + y $\cdot\cdot\cdot$\\
\emptyLine
\          2                                         2            2     $\cdot\cdot\cdot$\\
ideal (44x  + 22x*y + x*z + 22x - 26y - 30z - 6, 18x  + 12x*y + y  + 1 $\cdot\cdot\cdot$\\
\emptyLine
\            2                                           2            2 $\cdot\cdot\cdot$\\
ideal (- 41x  + 30x*y + x*z + 30x + 38y - 30z + 1, - 26x  - 10x*y + y  $\cdot\cdot\cdot$\\
\emptyLine
\          2                                            2            2  $\cdot\cdot\cdot$\\
ideal (39x  - 31x*y + x*z - 31x - 46y - 30z + 36, - 32x  - 13x*y + y   $\cdot\cdot\cdot$\\
\emptyLine
\            2                                          2            2  $\cdot\cdot\cdot$\\
ideal (- 10x  - 5x*y + x*z - 5x - 40y - 30z - 17, - 37x  + 35x*y + y   $\cdot\cdot\cdot$\\
\emptyLine
\endOutput
If we just want to see the degrees of the irreducible components, then
we say:
\beginOutput
i32 : ooo / degree\\
\emptyLine
o32 = \{1, 1, 1, 1, 1, 30, 6, 6, 6, 6, 6\}\\
\emptyLine
o32 : List\\
\endOutput
Note that the expressions ${\tt oo}$
and ${\tt ooo}$ refer to the previous and
prior-to-previous output lines respectively.

\medskip

Suppose we wish to compute the $x$-coordinates of our sixty five points.
Then we must use an elimination order, for instance, the
one described in \cite[\S 3.2, Exercise 6.a]{CLO1}.
We define a  new polynomial ring with the elimination order
for $\{y,z\} > \{x\}$ as follows:
\beginOutput
i33 : S = QQ[z,y,x, MonomialOrder => Eliminate 2]\\
\emptyLine
o33 = S\\
\emptyLine
o33 : PolynomialRing\\
\endOutput
We move our ideal into the new ring,
\beginOutput
i34 : ourpoints'' = substitute(ourpoints,S)\\
\emptyLine
\              2        5    4     3    5   5    5    5\\
o34 = ideal (y x - 1, y  - x , - y  + x , z  + y  + x  - 1)\\
\emptyLine
o34 : Ideal of S\\
\endOutput
and we compute the reduced Gr\"obner basis in this new order:
\beginOutput
i35 : G = gens gb ourpoints''\\
\emptyLine
o35 = | x13-1 y-x6 z5+x5+x4-1 |\\
\emptyLine
\              1       3\\
o35 : Matrix S  <--- S\\
\endOutput
To compute the elimination ideal we use the following command:
\beginOutput
i36 : ideal selectInSubring(1,G)\\
\emptyLine
\             13\\
o36 = ideal(x   - 1)\\
\emptyLine
o36 : Ideal of S\\
\endOutput

\section{Monomials Under the Staircase}

Invariants of an algebraic variety, such as its dimension
and degree, are computed from an initial monomial ideal.
This computation amounts to the combinatorial task
of analyzing the collection of standard monomials,
that is, the monomials under the staircase \cite[Chapter 9]{CLO1}.
In this section we demonstrate some basic operations on
monomial ideals in \Mtwo.

Let us create a non-trivial staircase in three dimensions
by taking the third power of the initial monomial from line {\tt i20}.
\beginOutput
i37 : M = staircase^3\\
\emptyLine
\              3 6   2 4 5   2 9   7 4     2 10     7 5   6 2 5     12  $\cdot\cdot\cdot$\\
o37 = ideal (x y , x y z , x y , x y , x*y z  , x*y z , x y z , x*y  , $\cdot\cdot\cdot$\\
\emptyLine
o37 : Ideal of R\\
\endOutput
The number of current generators of this ideal equals
\beginOutput
i38 : numgens M\\
\emptyLine
o38 = 20\\
\endOutput
To see all generators we can transpose the matrix of minimal generators:
\beginOutput
i39 : transpose gens M\\
\emptyLine
o39 = \{-9\}  | x3y6   |\\
\      \{-11\} | x2y4z5 |\\
\      \{-11\} | x2y9   |\\
\      \{-11\} | x7y4   |\\
\      \{-13\} | xy2z10 |\\
\      \{-13\} | xy7z5  |\\
\      \{-13\} | x6y2z5 |\\
\      \{-13\} | xy12   |\\
\      \{-13\} | x6y7   |\\
\      \{-13\} | x11y2  |\\
\      \{-15\} | z15    |\\
\      \{-15\} | y5z10  |\\
\      \{-15\} | x5z10  |\\
\      \{-15\} | y10z5  |\\
\      \{-15\} | x5y5z5 |\\
\      \{-15\} | x10z5  |\\
\      \{-15\} | y15    |\\
\      \{-15\} | x5y10  |\\
\      \{-15\} | x10y5  |\\
\      \{-15\} | x15    |\\
\emptyLine
\              20       1\\
o39 : Matrix R   <--- R\\
\endOutput
Note that this generating set is not minimal; see {\tt o48} below.
The number of standard monomials equals
\beginOutput
i40 : degree M\\
\emptyLine
o40 = 690\\
\endOutput
To list all the standard monomials we first create the residue ring
\beginOutput
i41 : S = R/M\\
\emptyLine
o41 = S\\
\emptyLine
o41 : QuotientRing\\
\endOutput
and then we ask for a vector space basis of the residue ring:
\beginOutput
i42 : basis S\\
\emptyLine
o42 = | 1 x x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x14y x14yz x14 $\cdot\cdot\cdot$\\
\emptyLine
\              1       690\\
o42 : Matrix S  <--- S\\
\endOutput
Let us count how many standard monomials there are of a given degree.
The following table represents the Hilbert function
of the residue ring.
\beginOutput
i43 : tally apply(flatten entries basis(S),degree)\\
\emptyLine
o43 = Tally\{\{0\} => 1  \}\\
\            \{1\} => 3\\
\            \{10\} => 63\\
\            \{11\} => 69\\
\            \{12\} => 73\\
\            \{13\} => 71\\
\            \{14\} => 66\\
\            \{15\} => 53\\
\            \{16\} => 38\\
\            \{17\} => 23\\
\            \{18\} => 12\\
\            \{19\} => 3\\
\            \{2\} => 6\\
\            \{3\} => 10\\
\            \{4\} => 15\\
\            \{5\} => 21\\
\            \{6\} => 28\\
\            \{7\} => 36\\
\            \{8\} => 45\\
\            \{9\} => 54\\
\emptyLine
o43 : Tally\\
\endOutput
Thus the largest degree of a standard monomial is nineteen,
and there are three standard monomials of that degree:
\beginOutput
i44 : basis(19,S)\\
\emptyLine
o44 = | x14yz4 x9yz9 x4yz14 |\\
\emptyLine
\              1       3\\
o44 : Matrix S  <--- S\\
\endOutput
The most recently defined ring involving {\tt x}, {\tt y}, and {\tt z} was
{\tt S}, so all computations involving those variables are done in the
residue ring {\tt S}.
%% The current ring {\tt S} is the residue ring. All calculations
%% are done in {\tt S}.
For instance, we can also obtain the
standard monomials of  degree nineteen as follows:
\beginOutput
i45 : (x+y+z)^19\\
\emptyLine
\            14   4          9   9         4   14\\
o45 = 58140x  y*z  + 923780x y*z  + 58140x y*z\\
\emptyLine
o45 : S\\
\endOutput
An operation on ideals which will occur frequently throughout this
book is the computation of minimal free resolutions. This is done as follows:
\beginOutput
i46 : C = res M\\
\emptyLine
\       1      16      27      12\\
o46 = R  <-- R   <-- R   <-- R   <-- 0\\
\                                      \\
\      0      1       2       3       4\\
\emptyLine
o46 : ChainComplex\\
\endOutput
This shows that our ideal {\tt M} has sixteen minimal generators.
They are the entries in the leftmost matrix of the chain complex {\tt C}:
\beginOutput
i47 : C.dd_1\\
\emptyLine
o47 = | x3y6 x7y4 x2y9 x2y4z5 x11y2 xy12 x6y2z5 xy7z5 xy2z10 x15 y15 x $\cdot\cdot\cdot$\\
\emptyLine
\              1       16\\
o47 : Matrix R  <--- R\\
\endOutput
This means that four of the twenty generators in {\tt o39} were redundant.
We construct the set consisting of the four redundant generators
as follows:
\beginOutput
i48 : set flatten entries gens M - set flatten entries C.dd_1\\
\emptyLine
\            6 7   10 5   5 10   5 5 5\\
o48 = Set \{x y , x  y , x y  , x y z \}\\
\emptyLine
o48 : Set\\
\endOutput
Here {\tt flatten entries} turns the matrix ${\tt M}$ into a single list.
The command {\tt set} turns that list into a set, to which we
can apply the difference operation for sets.

Let us now take a look at the first syzygies
(or {\it minimal S-pairs} \cite[\S 2.9]{CLO1})
among  the sixteen minimal generators.
They correspond to the columns of the second matrix in our resolution {\tt C}:
\beginOutput
i49 : C.dd_2\\
\emptyLine
o49 = \{9\}  | -y3 -x4 0   -z5 0   0   0   0   0   0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{11\} | 0   y2  0   0   0   -x4 0   0   -z5 0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{11\} | x   0   -y3 0   0   0   0   0   0   -z5 0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{11\} | 0   0   0   xy2 -y3 0   -x4 0   x5  y5  0   -z5 0   0   0 $\cdot\cdot\cdot$\\
\      \{13\} | 0   0   0   0   0   y2  0   0   0   0   0   0   0   -x4 0 $\cdot\cdot\cdot$\\
\      \{13\} | 0   0   x   0   0   0   0   -y3 0   0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{13\} | 0   0   0   0   0   0   y2  0   0   0   0   0   0   0   - $\cdot\cdot\cdot$\\
\      \{13\} | 0   0   0   0   x   0   0   0   0   0   -y3 0   0   0   0 $\cdot\cdot\cdot$\\
\      \{13\} | 0   0   0   0   0   0   0   0   0   0   0   xy2 -y3 0   0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   0   0   0   y2  0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   x   0   0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   0   0   0   0   y $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   x   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   0   0   x   0   0 $\cdot\cdot\cdot$\\
\      \{15\} | 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 $\cdot\cdot\cdot$\\
\emptyLine
\              16       27\\
o49 : Matrix R   <--- R\\
\endOutput
The first column represents the S-pair between the
first generator $x^3 y^6 $ and the third generator $x^2 y^9$.
It is natural to form the {\it S-pair graph} with $16$ vertices and
$27$ edges represented by  this matrix. According to the
general theory described in \cite{MS}, this is a planar graph
with $12$ regions. The regions correspond to the $12$ second syzygies,
that is, to the columns of the matrix
\beginOutput
i50 : C.dd_3\\
\emptyLine
o50 = \{12\} | z5  0   0   0   0   0   0   0   0   0   0   0   |\\
\      \{13\} | 0   z5  0   0   0   0   0   0   0   0   0   0   |\\
\      \{14\} | 0   0   z5  0   0   0   0   0   0   0   0   0   |\\
\      \{14\} | -y3 -x4 0   0   0   0   0   0   0   0   0   0   |\\
\      \{14\} | 0   0   -y5 z5  0   0   0   0   0   0   0   0   |\\
\      \{15\} | 0   0   0   0   z5  0   0   0   0   0   0   0   |\\
\      \{15\} | 0   0   0   0   -x5 z5  0   0   0   0   0   0   |\\
\      \{16\} | 0   0   0   0   0   0   z5  0   0   0   0   0   |\\
\      \{16\} | 0   y2  0   0   -x4 0   0   0   0   0   0   0   |\\
\      \{16\} | x   0   -y3 0   0   0   0   0   0   0   0   0   |\\
\      \{16\} | 0   0   0   0   0   0   -y5 z5  0   0   0   0   |\\
\      \{16\} | 0   0   0   -y3 0   -x4 0   0   0   0   0   0   |\\
\      \{16\} | 0   0   0   0   0   0   0   -y5 z5  0   0   0   |\\
\      \{17\} | 0   0   0   0   0   0   0   0   0   z5  0   0   |\\
\      \{17\} | 0   0   0   0   0   0   0   0   0   -x5 z5  0   |\\
\      \{17\} | 0   0   0   0   0   0   0   0   0   0   -x5 z5  |\\
\      \{18\} | 0   0   0   0   y2  0   0   0   0   -x4 0   0   |\\
\      \{18\} | 0   0   x   0   0   0   -y3 0   0   0   0   0   |\\
\      \{18\} | 0   0   0   0   0   y2  0   0   0   0   -x4 0   |\\
\      \{18\} | 0   0   0   x   0   0   0   -y3 0   0   0   0   |\\
\      \{18\} | 0   0   0   0   0   0   0   0   -y3 0   0   -x4 |\\
\      \{20\} | 0   0   0   0   0   0   0   0   0   y2  0   0   |\\
\      \{20\} | 0   0   0   0   0   0   x   0   0   0   0   0   |\\
\      \{20\} | 0   0   0   0   0   0   0   0   0   0   y2  0   |\\
\      \{20\} | 0   0   0   0   0   0   0   x   0   0   0   0   |\\
\      \{20\} | 0   0   0   0   0   0   0   0   0   0   0   y2  |\\
\      \{20\} | 0   0   0   0   0   0   0   0   x   0   0   0   |\\
\emptyLine
\              27       12\\
o50 : Matrix R   <--- R\\
\endOutput
But we are getting ahead of ourselves. Homological algebra and resolutions
will be covered in the next chapter, and monomial ideals
will appear in the chapter of Ho\c{s}ten and Smith.
Let us return to Cox, Little and O'Shea \cite{CLO2}.

\section{Pennies, Nickels, Dimes and Quarters}

We now come to an application of Gr\"obner bases which appears in
\cite[Section 8.1]{CLO2}: {\sl Integer Programming}. This is the problem of
 minimizing a linear objective function over the set of non-negative
integer solutions of a system of linear equations.  We demonstrate
some techniques for doing this in \Mtwo. Along the way, we learn about
multigraded polynomial rings and how to compute
Gr\"obner bases with respect to monomial orders defined by weights.
Our running example is the linear system defined  by the matrix:
\beginOutput
i51 : A = \{\{1, 1, 1, 1\},\\
\           \{1, 5,10,25\}\}\\
\emptyLine
o51 = \{\{1, 1, 1, 1\}, \{1, 5, 10, 25\}\}\\
\emptyLine
o51 : List\\
\endOutput
For the algebraic study of integer programming problems, a good starting
point is to work in a multigraded polynomial ring, here in four variables:
\beginOutput
i52 : R = QQ[p,n,d,q, Degrees => transpose A]\\
\emptyLine
o52 = R\\
\emptyLine
o52 : PolynomialRing\\
\endOutput
The degree of each variable is the corresponding column vector of the matrix
Each variable represents one of the four coins in the U.S. currency system:
\beginOutput
i53 : degree d\\
\emptyLine
o53 = \{1, 10\}\\
\emptyLine
o53 : List\\
\endOutput
\beginOutput
i54 : degree q\\
\emptyLine
o54 = \{1, 25\}\\
\emptyLine
o54 : List\\
\endOutput
Each monomial represents a collection of coins. For instance, suppose
you own four  pennies, eight nickels, ten dimes, and three quarters:
\beginOutput
i55 : degree(p^4*n^8*d^10*q^3)\\
\emptyLine
o55 = \{25, 219\}\\
\emptyLine
o55 : List\\
\endOutput
Then you have a total of 25 coins worth two dollars and nineteen cents.
There are nine other possible ways of having 25 coins of the same value:
\beginOutput
i56 : h = basis(\{25,219\}, R)\\
\emptyLine
o56 = | p14n2d2q7 p9n8d2q6 p9n5d6q5 p9n2d10q4 p4n14d2q5 p4n11d6q4 p4n8 $\cdot\cdot\cdot$\\
\emptyLine
\              1       9\\
o56 : Matrix R  <--- R\\
\endOutput
For just counting the number of columns of this matrix
we can use the command
\beginOutput
i57 : rank source h\\
\emptyLine
o57 = 9\\
\endOutput
How many ways can you make change for ten dollars using $100$ coins?
\beginOutput
i58 : rank source basis(\{100,1000\}, R)\\
\emptyLine
o58 = 182\\
\endOutput
A typical integer programming problem is this: among all 182 ways of
expressing ten dollars using 100 coins, which one uses the fewest dimes?
We set up the  Conti-Traverso algorithm \cite[\S 8.1]{CLO2} for
answering this question. We use the following ring with the lexicographic
order and with the variable order:
dimes (d) before pennies (p) before nickels (n) before quarters (q).
\beginOutput
i59 : S = QQ[x, y, d, p, n, q, \\
\          MonomialOrder => Lex, MonomialSize => 16]\\
\emptyLine
o59 = S\\
\emptyLine
o59 : PolynomialRing\\
\endOutput
The option {\tt MonomialSize} advises \Mtwo to use more space to store the
exponents of monomials, thereby avoiding a potential overflow.

We define an ideal with one generator for each column of the matrix A.
\beginOutput
i60 : I = ideal( p - x*y, n - x*y^5, d - x*y^10, q - x*y^25)\\
\emptyLine
\                             5           10           25\\
o60 = ideal (- x*y + p, - x*y  + n, - x*y   + d, - x*y   + q)\\
\emptyLine
o60 : Ideal of S\\
\endOutput
The integer program is solved by normal form reduction with respect
to the following Gr\"obner basis consisting of binomials.
\beginOutput
i61 : transpose gens gb I\\
\emptyLine
o61 = \{-6\}  | p5q-n6     |\\
\      \{-4\}  | d4-n3q     |\\
\      \{-3\}  | yn2-dp     |\\
\      \{-6\}  | yp4q-dn4   |\\
\      \{-4\}  | yd3-pnq    |\\
\      \{-6\}  | y2p3q-d2n2 |\\
\      \{-5\}  | y2d2n-p2q  |\\
\      \{-7\}  | y2d2p3-n5  |\\
\      \{-6\}  | y3p2q-d3   |\\
\      \{-6\}  | y3dp2-n3   |\\
\      \{-5\}  | y4p-n      |\\
\      \{-6\}  | y5n-d      |\\
\      \{-8\}  | y6d2-pq    |\\
\      \{-16\} | y15d-q     |\\
\      \{-7\}  | xq-y5d2    |\\
\      \{-5\}  | xn-y3p2    |\\
\      \{-2\}  | xd-n2      |\\
\      \{-2\}  | xy-p       |\\
\emptyLine
\              18       1\\
o61 : Matrix S   <--- S\\
\endOutput
We fix the quotient ring, so the reduction to normal form
will happen automatically.
\beginOutput
i62 : S' = S/I\\
\emptyLine
o62 = S'\\
\emptyLine
o62 : QuotientRing\\
\endOutput
You need at least two dimes to express one dollar with ten coins.
\beginOutput
i63 : x^10 * y^100\\
\emptyLine
\       2 6 2\\
o63 = d n q\\
\emptyLine
o63 : S'\\
\endOutput
But you can express ten dollars with a hundred coins none of which is a dime.
\beginOutput
i64 : x^100 * y^1000\\
\emptyLine
\       75 25\\
o64 = n  q\\
\emptyLine
o64 : S'\\
\endOutput
The integer program is infeasible if and only if the normal form still
contains the variable $x$ or the variable $y$. For instance, you cannot
express ten dollars with less than forty coins:
\beginOutput
i65 : x^39 * y^1000\\
\emptyLine
\       25 39\\
o65 = y  q\\
\emptyLine
o65 : S'\\
\endOutput
We now introduce a new term order on the polynomial ring, defined
by assigning a weight to each variable. Specifically, we assign
weights for each of the coins. For instance,
let pennies have weight 5, nickels weight 7,
dimes weight 13 and quarters weight 17.
\beginOutput
i66 : weight = (5,7,13,17)\\
\emptyLine
o66 = (5, 7, 13, 17)\\
\emptyLine
o66 : Sequence\\
\endOutput
We set up a new ring with the resulting weight term order, and work modulo
the same ideal as before in this new ring.
\beginOutput
i67 : T = QQ[x, y, p, n, d, q, \\
\                Weights => \{\{1,1,0,0,0,0\},\{0,0,weight\}\},\\
\                MonomialSize => 16]/\\
\            (p - x*y, n - x*y^5, d - x*y^10, q - x*y^25);\\
\endOutput
One dollar with ten coins:
\beginOutput
i68 : x^10 * y^100\\
\emptyLine
\       5 2 3\\
o68 = p d q\\
\emptyLine
o68 : T\\
\endOutput
Ten dollars with one hundred coins:
\beginOutput
i69 : x^100 * y^1000\\
\emptyLine
\       60 3 37\\
o69 = p  n q\\
\emptyLine
o69 : T\\
\endOutput
Here is an optimal solution which involves all four types of coins:
\beginOutput
i70 : x^234 * y^5677\\
\emptyLine
\       2 4 3 225\\
o70 = p n d q\\
\emptyLine
o70 : T\\
\endOutput


%!TEX program = xelatex

\documentclass[12pt,a4paper]{article}
\usepackage{xeCJK}
\usepackage{amsmath}
\setmainfont{Times New Roman}
\usepackage{setspace}
\usepackage{caption}
\usepackage{graphicx, subfig}
\usepackage{float}
\usepackage{listings}
\usepackage{booktabs}
\usepackage{setspace}%使用间距宏包
\usepackage{mathtools}
\usepackage{amsfonts}
\usepackage{amsmath}
    \newcommand{\dd}{\mathrm{d}}
\usepackage{tcolorbox}
    \tcbuselibrary{xparse}
        \DeclareTotalTCBox{\verbbox}{ O{green} v !O{} }
            {fontupper=\ttfamily,nobeforeafter,tcbox raise base,
             arc=0pt,outer arc=0pt,top=0pt,bottom=0pt,left=0mm,
             right=0mm,leftrule=0pt,rightrule=0pt,toprule=0.3mm,
             bottomrule=0.3mm,boxsep=0.5mm,bottomrule=0.3mm,boxsep=0.5mm,
             colback=#1!10!white,colframe=#1!50!black,#3}{#2}
\usepackage{color}
\usepackage{textcomp}
\definecolor{listinggray}{gray}{0.9}
\definecolor{lbcolor}{rgb}{0.9,0.9,0.9}
\lstset{
	backgroundcolor=\color{lbcolor},
	tabsize=4,
	rulecolor=,
	language=matlab,
        basicstyle=\scriptsize,
        upquote=true,
        aboveskip={1.5\baselineskip},
        columns=fixed,
        showstringspaces=false,
        extendedchars=true,
        breaklines=true,
        prebreak = \raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}},
        frame=single,
        showtabs=false,
        showspaces=false,
        showstringspaces=false,
        identifierstyle=\ttfamily,
        keywordstyle=\color[rgb]{0,0,1},
        commentstyle=\color[rgb]{0.133,0.545,0.133},
        stringstyle=\color[rgb]{0.627,0.126,0.941},
}

\begin{document}
\title{homework12}
	\author{11611118 郭思源}
\begin{spacing}{1.5}%%行间距变为double-space

\section{Problem 1}

Problem: Maximize $f(x_1,x_2)$.
\[
f(x_1,x_2) = x_1(P_1-a_1x_1-b_1x_2)+x_2(P_2-a_2x_1-b_2x_2)-F-C_1x_1-C_2x_2
\]
1. Slove the problem by calculus.
\begin{equation*}
	\begin{aligned}
		\frac{\partial f(x_1,x_2)}{\partial x_1}
        &= P_1-C_1-2a_1x_1-a_2x_2-b_1x_2 \\
        \frac{\partial f(x_1,x_2)}{\partial x_2}
        &= P_2-C_2-a_2x_1-b_1x_1-2b_2x_2 \\
        \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}
        &= -2a_1 \\
        \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2}
        &= -a_2-b_1 \\
        \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}
        &= -2b_2 \\
	\end{aligned}
\end{equation*}

There should be :
\begin{equation*}
	\begin{aligned}
		\frac{\partial f(x_1,x_2)}{\partial x_1} &= 0 \\
        \frac{\partial f(x_1,x_2)}{\partial x_2} &= 0 \\
        \frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2} &< 0 \\
        (\frac{\partial^2 f(x_1,x_2)}{\partial x_1 \partial x_2})^2 &
        - \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2} \times
        \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2} < 0
	\end{aligned}
\end{equation*}

\newpage
So :
\begin{equation*}
    \begin{aligned}
        x_2 &=
        \frac{2a_1(P_2-C_2)-(a_2+b_1)(P_1-C_1)}{4a_1b_2-(a_2+b_1)^2}  \\
        x_1 &= \frac{P_1-C_1}{2a_1} - \frac{a_2+b_1}{2a_1} \times
        \frac{2a_1(P_2-C_2)-(a_2+b_1)(P_1-C_1)}{4a_1b_2-(a_2+b_1)^2}  \\
        & \qquad \qquad a_2 + b_1 > 0 \\
        & \qquad \qquad (a_2 + b_1)^2-4a_1b_2 < 0 \\
    \end{aligned}
\end{equation*}
2. Slove the problem by via MATLAB.

\begin{lstlisting}[language=matlab]
a1 = 1; a2 = 2; b1 = 1; b2 = 3;
c1 = 1; c2 = 2; P1 = 5; P2 = 3; F = 3;
f = @(x)(-1*(x(1)*(P1-a1*x(1)-b1*x(2)) + ...
          x(2)*(P2-a2*x(1)-b2*x(2)) ...
          - F - c1*x(1) - c2*x(2)));
opts = optimoptions(@fmincon,'Algorithm','interior-point');
% problem = createOptimProblem('fmincon', ...
%     'x0',[2;3],'objective',sixmin, ...
%     'Aineq',A,'bineq',b,'options',opts);
problem = createOptimProblem('fmincon','x0',[0;0],'objective',f,'options',opts);
[x,fval,eflag,output] = fmincon(problem)
\end{lstlisting}

\newpage
\section{Problem 2}

Problem: Minimize $f(x_1,x_2)$, such that $x_1+2x_2=12$.
\[
f(x_1,x_2) = \frac{23}{x_1} + x_1 + \frac{30}{x_2} - 6x_2
\]
1. Please present a short review of the Lagrange Multiplier Method. \\\\
Lagrange multiplication method is an optimization algorithm, which is mainly used to solve the optimization problem under the condition of constraint. The basic idea is to transform the constraint optimization problem with $N$ variables and $K$ constraints into unconstrained optimization problem containing $(N+K)$ variables by introducing Lagrange multiplication. \\\\
The standard form of constrained optimization problem can be expressed in the following formula:
\begin{equation}
    \begin{aligned}
        min \quad & f(x)\\
        s.t.: \quad & g_i(x) \leq 0, i=1,...,m\\
              \quad & h_j(x) = 0, j=1,...,t\\
    \end{aligned}
\end{equation}
The basic idea of Lagrangian multiplication method is to transform the optimization problem with constraints in formula (1) into an unconstrained optimization problem. Its concrete implementation is as follows:
\begin{equation}
    \begin{aligned}
        L(x;\alpha,\beta) = f(x) + \sum_{i=1}^m \alpha_i g_i(x) + \sum_{j=1}^t \beta_j h_j(x)
    \end{aligned}
\end{equation}
Where $L$ is the Lagrange function, $\alpha$, $\beta$ is the Lagrange boarding, and satisfies ${\alpha}_i \geq 0$. $x$ is the primal variable, $\alpha$ and $\beta$ are the dual variables.

\newpage
\noindent For problem:
\begin{equation*}
    \begin{aligned}
        min \quad & f(x)\\
        s.t.: \quad & g(x) = 0
    \end{aligned}
\end{equation*}

\noindent From a geometric point of view, the goal of this problem is to look for points on the dimensional curvature surface determined by the equation that can minimize the objective function. \\\\
It is not difficult for us to draw the following conclusion: \\
1. For any point on the constrained surface, the gradient of the point $\nabla g(x)$ is orthogonal to the constraint surface \\
2. For the best solution $x^*$, the gradient of the objective function in this point
$\nabla f(x^*)$ is also orthogonal to the constrained surface \\\\
So that, in the most advantages, the gradient and the inevitable direction are the same or the opposite. That is: Existing $\lambda\not=0$, such that:
\[
    \nabla f(x^*) + \lambda \nabla g(x^*) = 0
\]

\noindent 2. Slove the problem by the Lagrange Multiplier Method (LMM).

\begin{equation*}
    \begin{aligned}
        \nabla f(\vec{x}) &= (1-\frac{23}{x_1^2},-6-\frac{30}{x_2^2}) \\
        g(\vec{x}) &= x_1 + 2x_2 - 12 \\
        \nabla g(\vec{x}) &= (1,2)
    \end{aligned}
\end{equation*}
for $\nabla f(x^*) + \lambda \nabla g(x^*) = 0$:
\[
\vec{x} = (2.229, 4.885),\quad f(\vec{x}) = -10.62
\]

\newpage
\section{Problem 3}
$A\ Least\ Squares\ Plane$ --- Given the four points $(x_k,y_k,z_k)$
\[
(0,0,0),\quad (0,1,1),\quad (1,1,1),\quad (1,0,-1)
\]
Find the values of A,B and C to minimize the sum of squared errors
\[
\sum_{k=1}^4 (Ax_k + By_k + C - z_k)^2
\]
If the points must line in the plane
\[
z=Ax+By+C
\]
\begin{equation*}
    \begin{aligned}
        W &= \sum_{k=1}^4 (Ax_k + By_k + C - z_k)^2 \\
          &= 2A^2+2B^2+4C^2+2AB+4AC+4BC-4B-2C+3 \\
        \frac{\partial W}{\partial A}
          &= 4A+2B+4C = 0 \\
        \frac{\partial W}{\partial B}
          &= 4B+2A+4C-4 = 0 \\
        \frac{\partial W}{\partial C}
          &= 8C+4A+4B-2 = 0 \\
    \end{aligned}
\end{equation*}
Get $A=-0.5,B=1.5,C=-0.25$ \\
Then we have:
\[
\mathbf{H(\vec{x})} =
\begin{bmatrix}
4 & 2 & 4 \\
2 & 4 & 4 \\
4 & 4 & 8
\end{bmatrix},
\mathbf{det(H(\vec{x}))} > 0
\]
So we have the $min$ W.

\end{spacing}

\end{document}


\section{Quadric Surfaces}
\noindent
Quadric surfaces extend parabolas and other shapes composed of at most squared terms into 3D.

\input{./differentialMultivariableCalculus/paraboloids}
\input{./differentialMultivariableCalculus/hyperboloids}
\input{./differentialMultivariableCalculus/hyperbolicParaboloids}
\input{./differentialMultivariableCalculus/ellipsoids}
\input{./differentialMultivariableCalculus/3DCylinders}


\documentclass[main.tex]{subfiles}
\begin{document}

\section{The multivariate gaussian}

\marginpar{Monday\\ 2020-10-19, \\ compiled \\ \today}

This will be a once-in-a-lifetime calculation, we need the result practically but it is good to have seen the derivation once.

The Multi-Variate Normal is in general
%
\begin{align}
\mathcal{N} (\vec{x}, \vec{\mu}, C) =
\frac{1}{(2 \pi )^{n/2} \sqrt{\det C}}
\exp(- \frac{1}{2} \qty(\vec{x} - \vec{\mu})^{\top} C^{-1} (\vec{x} - \vec{\mu}))
\,.
\end{align}

Gaussians have many applications: with lots of data points, both the likelihood and the posterior converge to Gaussians with the same mean.
We have many analytical results about Gaussians.

\paragraph{Normalization}

We define \(V = C^{-1}\), the precision matrix.
This is diagonalized as \(O^{\top} V O\), where \(\Lambda \) is diagonal with eigenvalues \(\lambda _k\) and \(O\) is orthogonal.
Then, the integral of the exponential \(\exp( -\vec{y}^{\top} V \vec{y}/2)\) can be expressed as
%
\begin{align}
\int \dd[n]{y} \exp(- \frac{1}{2} \vec{y}^{\top} O^{\top} V O \vec{y}) \det O &=
\int \dd[n]{y} \exp(- \frac{1}{2} \lambda _k y_k^2)
 \\
&= \prod_{k=1}^{n} \int \dd{y}_k \exp(- \frac{1}{2} \lambda _k y_k^2)   \\
&= \prod_{k=1}^{n} \sqrt{\frac{2 \pi }{\lambda _k}} = \frac{(2\pi)^{n/2}}{\sqrt{\det \Lambda }} = (2 \pi )^{n/2} \sqrt{\det C}
\,.
\end{align}

\paragraph{Marginalization}

We have an \(n\)-variate MVN, which we want to marginalize over \(M\) parameters: the integral we want to perform is
%
\begin{align}
\int \dd{x_{n-M+1}} \dots \dd{x_n} \frac{1}{(2 \pi )^{n/2} \sqrt{\det C}} \exp(- \frac{1}{2} \vec{y}^{\top} V \vec{y})
\,.
\end{align}

We partition the precision matrix into \(M\) and \(n-M\)-dimensional blocks:
%
\begin{align}
V = \left[\begin{array}{cc}
V_{aa} & V_{ab} \\
V_{ab} & V_{bb}
\end{array}\right]
\,.
\end{align}

Note that there is no one-to-one correspondence with the inverse matrix: in general \(V_{aa} \neq C^{-1}_{aa}\).
The argument of the exponential can be written, with the notation \(\vec{y} = \vec{x} - \vec{\mu}\):
%
\begin{align}
- \frac{1}{2} \vec{y}^{\top} V \vec{y} &=
\vec{y}_a^{\top} V_{aa} \vec{y}_a +
\vec{y}_b^{\top} V_{bb} \vec{y}_b +
2\vec{y}_a^{\top} V_{ab} \vec{y}_b
\,.
\end{align}

We then use the square completion formula:
%
\begin{align}
\frac{1}{2} \vec{z}^{\top} A \vec{z} + \vec{b}^{\top} \vec{z} + c
= \frac{1}{2} \qty(\vec{z} + A^{-1} \vec{b})^{\top} A \qty(\vec{z} + A^{-1} \vec{b}) - \frac{1}{2} \vec{b}^{\top} A^{-1} \vec{b} + c
\,,
\end{align}
%
which generalizes the procedure used to solve second-degree equations.
We identify \(A = V_{bb}\), \(\vec{z} = \vec{y}_b\), \(\vec{b} = V_{ab}^{\top} \vec{y}_a\),  \(c = \vec{y}_a V_{aa} \vec{y}_a / 2\).

\todo[inline]{Check calculation! something is wrong here}

Finally, we get that the marginalized distribution is still Gaussian, and is written as
%
\begin{align}
\mathcal{N}(\vec{x}_a | \vec{\mu}_a, (V_{aa} - V_{ab} V_{bb}^{-1} V_{ab})^{-1})
\,.
\end{align}

The new covariance can be expressed more simply after some matrix algebra (using the Woodbury formula for block matrix inversion):
we can show that
%
\begin{align}
C_{aa} = \qty(V^{-1})_{aa} = \qty(V_{aa} - V_{ab} V_{bb}^{-1} V_{ab})^{-1}
\,,
\end{align}
%
so the marginal PDF is just \(\mathcal{N}(\vec{x}_a| \mu_a, C_{aa})\).
Marginalizing for Gaussians can then be done fully analytically in \(\mathcal{O}(1)\) time: we just discard the unnecessary parts of the covariance and mean.

The Hessian (calculated at the mean) is the inverse of the opposite of the covariance matrix: \(H = - V = - C^{-1}\).

% \todo[inline]{Hold on\dots is this right?}

Then, \(\sigma _m = \sqrt{(- H)^{-1}_{mm}}\).

\paragraph{Conditioning}

Now we want to compute the conditional probability of a certain part of the parameter vector, \(\vec{x}_b\), if we fix the rest of the vector to values \(\vec{x}_a\).
This can also be done analytically: now, the mean will be \(\vec{\mu}_{b | a} = \vec{\mu}_b - V^{-1}_{bb} V_{ba} \qty(\vec{x}_a - \vec{\mu}_a)\), while the covariance is \(C_{b | a} = V^{-1}_{bb}\).
% This almost looks like the wrong thing, \(C_{bb} = V^{-1}_{bb}\), but in this case it is actually correct.

If we marginalize over all the other parameters, the error looks like \(\sigma _m = \sqrt{(-H_{mm})^{-1}}\). This time we invert the single matrix element.
This will usually be much smaller than what we would get when marginalizing.

\paragraph{Summing}

If both \(\vec{x}\) and \(\vec{y}\) are MVN distributed and independent, then
%
\begin{align}
\mathbb{P}(\vec{z}) = \mathcal{N} (\vec{z} | \vec{\mu}_x + \vec{\mu}_y, C_x + C_y)
\,.
\end{align}

The way to prove this is to start from  the fact that
%
\begin{align}
\mathbb{P}(z) &= \int \dd{x} \dd{y} \mathbb{P}(x) \mathbb{P}(y) \delta (z - (x + y))  \\
&= \int \dd{x} \mathbb{P}(x) \mathbb{P}(z-x)
\,,
\end{align}
%
a convolution. The convolution of two Gaussians is a Gaussian, so we can just calculate the mean, covariance, and we are done:
%
\begin{align}
\expval{x + y} = \expval{x} + \expval{y}
\,,
\end{align}
%
while the covariance is (implying a tensor product between the vectors):
%
\begin{align}
\expval{(\vec{x} + \vec{y})(\vec{x} + \vec{y})} - (\vec{x}+\vec{y})(\vec{x}+\vec{y})
&= \expval{\vec{x} \vec{x}} - \expval{\vec{x}}^2 + \expval{\vec{y} \vec{y}} - \expval{\vec{y}}^2
\,,
\end{align}
%
using the property that \(\expval{\vec{x} \vec{y}} - \expval{\vec{x}} \expval{\vec{y}} = 0\).

Note that independence \(\implies\) uncorrelation, but not the other way around. For example, if \(x\) is normally distributed with mean zero, \(x^2\) is \emph{uncorrelated} to it (the correlation would be given by \(\expval{x^3} = 0\)); however \(x\) and \(x^2\) are related two-to-one, certainly not independent.

\end{document}


\documentclass[aps,prd,amsmath,amssymb,superscriptaddress,onecolumn,
nofootinbib,showpacs,preprintnumbers]{revtex4-1}

\usepackage{hyperref,color,subfigure,soul}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}% Include figure files
\usepackage{dcolumn}% Align table columns on decimal point
\usepackage{bm}% bold math
\usepackage{physics}
\usepackage{cleveref}
\usepackage{tikz}
\usepackage{mathtools}

\newcommand{\pp}{\pi\pi }
\newcommand{\ma}{\mathcal{A}}
\newcommand{\maI}[1]{\mathcal{A}^{(#1)}}
\newcommand{\mvI}[1]{\mathcal{V}^{(#1)}}

\newcommand{\az}{\alpha_0}
\newcommand{\apr}{\alpha^\prime}
\newcommand{\as}{\alpha_s}
\newcommand{\at}{\alpha_t}

\begin{document}

\section{General $\pp$ Scattering Amplitude}
We are interested in the structure of the the isospin invariant $\pp$ scattering amplitude:
%
	\begin{equation}
	\frac{1}{32\pi}\mel{\pi_k(p_3)\pi_l(p_4)}{T}{\pi_i(p_1)\pi_j(p_2)} \equiv i (2 \pi)^4 \delta^4(p_1 + p_2 - p_3 - p_4) \;  \mathcal{M}_{ijkl}(s,t,u)
	\end{equation}
%
where the Cartesian indices denote isospin indices and $s + t + u = 16 \; m_\pi^2$ are the standard Mandelstam variables.
We can factor out the isospin dependence and define three scalar amplitudes representing the scattering amplitdes in the s-, t-, and u-channels respectively:
%
	\begin{equation} \label{scalaramps}
	\mathcal{M}_{ijkl}(s,t,u) = \delta_{ij}\delta_{kl} \; A(s,t,u) + \delta_{ik}\delta_{jl} \; B(s,t,u) + \delta_{il}\delta_{jk} \; C(s,t,u).
	\end{equation}
%
The three scalar amplitudes in \cref{scalaramps} are not independent thanks to symmetries of the initial and final state pions. Bose symmetry lets us freely interchange pions in the final state, i.e. amplitudes are invariant from exchange $t \leftrightarrow u$. Additionally, crossing symmetry relates cross-channel amplitudes must by the interchange of a initial and final state pion, ($i\leftrightarrow k$ or $i \leftrightarrow j$ or similarly $s \leftrightarrow t$ and $s \leftrightarrow u$). This gives us relations:
%
	\begin{equation} \label{symmetries}
	 B(s,t,u) = A(t,s,u) \quad \textrm{and} \quad C(s,t,u) = A(u,t,s).
	\end{equation}
%

Thus we can write \cref{scalaramps} in terms of a single amplitude:
%
	\begin{equation}\label{matrixelement}
	\mathcal{M}_{ijkl}(s,t,u) =  \delta_{ij}\delta_{kl} \; A(s, t,u) + \delta_{ik}\delta_{jl} \; A(t, s, u) + \delta_{il}\delta_{jk} \; A(u, t,s).
	\end{equation}
	\subsection{Isospin Amplitudes}

Similar to \cite{JPACCollaboration2018} we want to look at amplitudes which have a well-defined isopin in the $s$-channel which we can project out. We define two-pion isospin states by:
%
	\begin{equation} \label{isospin}
	\ket{\pi^i \pi^j}_I = P^{I}_{ijkl} \ket{\pi^k \pi^l}
	\end{equation}
%
where
%
	\begin{equation} \label{projectors}
	P^{(0)}_{ijkl} = \frac{1}{3}\delta_{ij}\delta_{kl},  \quad P^{(1)}_{ijkl} = \frac{1}{2}(\delta_{ik}\delta_{jl}-\delta_{il}\delta_{jk}),  \quad \textrm{and} \quad   P^{(2)}_{ijkl} = \frac{1}{2}
	(\delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk}) - \frac{1}{3} \delta_{ij}\delta_{kl}.
	\end{equation}
%

Combining \cref{matrixelement} and \cref{projectors} we get:
%
	\begin{equation}\label{iso-decomp}
	\mathcal{M}_{ijkl}(s,t,u) = P^{(0)}_{ijkl} \; A^{(0)}(s,t,u) + P^{(1)}_{ijkl}  \; A^{(1)}(s,t,u) +  P^{(2)}_{ijkl} \; A^{(2)}(s,t,u).
	\end{equation}
%
Also comparing \cref{matrixelement,iso-decomp} we can write our isospin amplitudes in terms of the original scalar amplitude:
%
	\begin{align} \label{matrix}
		\begin{bmatrix}
		A^{(0)}(s,t,u) \\ A^{(1)} (s,t,u) \\ A^{(2)}(s,t,u)
		\end{bmatrix}
	=
		\begin{bmatrix*}[r]
			3 & 1 & 1 \\ 	0 & 1 & -1 \\ 0 & 1 & 1
		\end{bmatrix*}
		\begin{bmatrix}
		A(s,t,u) \\ A(t,s,u) \\ A(u,t,s)
		\end{bmatrix}
	\end{align}
%
By projecting out specific partial-waves of the isospine definite amplitudes \cref{matrix} we can study the resonance content of the scattering amplitude.

%%%%%%%%%%%%%%%%%%%%%%%%% VENEZIANO AMPLITUDE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Veneziano Model}

Historically a dual resonance formalism for $\pp$ scattering takes the form of decomposing the scalar amplitude $A(s,t,u)$ into symmetric meromorphic functions of two variables and imposing Bose and crossing symmetries. Specifically this means:
	\begin{equation}
		A(s,t,u) = [ \ma(t,u) - \ma(s,u) - \ma(s,t) ]
		\label{dual_decomp}
	\end{equation}
for $\ma(s,t) = \ma(t,s)$ being chosen to have the correct analytic (i.e. dynamical resonance) structure in the energy variables.
The original Veneziano amplitude selected:
	\begin{equation}
		\ma(s,t) = \frac{\Gamma( 1 - \alpha(s) ) \Gamma(1- \alpha(t))}{\Gamma(1- \alpha(s) - \alpha(t))}
		\label{eq:orig_venez}
	\end{equation}
which has infinitely many simple pole singularities when $\alpha(s)$ (or $\alpha(t)$) are integers and the correct Regge behavior ( $\ma(s,t) \sim s^{\alpha(t)}$) at high energies. Analogous decompositions for the cross channel amplitudes are defined by simple interchange of Mandelstam variables in \cref{dual_decomp}
	\begin{equation}
		\text{t-channel:} \quad A(t,s,u) \qquad \text{u-channel:} \quad A(u,t,s)
	\end{equation}.
Then isospin definite scattering amplitudes can be assembled by combinations of \cref{eq:orig_venez}, using \cref{dual_decomp} in \cref{matrix}.
\begin{align} \label{isoamps}
		\begin{bmatrix}
		A^{(0)}(s,t,u) \\
		A^{(1)}(s,t,u) \\
		A^{(2)}(s,t,u)
		\end{bmatrix}
	=
		\begin{bmatrix*}[r]
			-3 & -3 & 1 \\
			 -2 & \;2 & \;0 \\
				\; 0 & \; 0 & -2
		\end{bmatrix*}
		\begin{bmatrix}
		\ma(s,t) \\
		\ma(s,u) \\
		\ma(t,u)
		\end{bmatrix}
	\end{align}

	 This approach, however, requires the assumption of exact exchange degeneracy between trajectories, so that the couplings and masses of isospin-0 trajectories depend on the parameterization of the isospin-1 trajectory. It also provides very little freedom in the changing the features predicted by the model.

\subsection{Phenomenological Approach}
We wish to construct a model that takes advantage of the appealing features of the Veneziano amplitude, most notably the Regge behavior at high energies, but with more flexibility in describing resonance structures in scattering data.
To incorporate multiple trajectories (i.e. $\sigma, \rho, f_2, ...$) we take a bottom-up approach, noticing that the direct-channel resonance content of the isospin amplitudes should come from the Bose-symmetric combinations:
	\begin{equation}
		[ \ma^{(I)}(s,t) + (-1)^I \ma^{(I)}(s,u) ]
		\label{direct}
	\end{equation}
where we now allow our (at this point completely general) symmetric functions to be different for each isospin channel. This freedom will allow us to parameterize each channel independently (a trajectory $\alpha^{(I)}$(s) for each isospin channel) and more accurately describe resonance masses and widths.

In order to also satisfy crossing symmetry, we must add background terms containing poles in $t$ and $u$ in other isospin channels. To this end, we consider the most general sum of background terms that are Bose-symmetric:
	\begin{align}
		\text{Even Isospin (symmetric): }& \qquad \sum_{I^\prime} C_{II^\prime} \; \ma^{(I^\prime)}(t,u) \nonumber \\
		\text{Odd Isospin (antisymmetric): }& \qquad \sum_{I^\prime} C_{II^\prime} \; [ \ma^{(I^\prime)}(s,t) - \ma^{(I^\prime)}(s,u) ].
		\label{background}
		\end{align}
We immediately see that the isospin-1 poses as problem as the only way to construct a background term that is antisymmetric under $t \leftrightarrow u$ with symmetric scalar functions introduces $s$ dependence and therefore the possibility of unwanted direct-channel resonance contributions from the background terms. Our choice for the function form of $\maI{I}$ must have a way to eliminate these unwanted poles.

Imposing crossing symmettry on $A^{(I)}(s,t,u)$ as a sum of \cref{direct,background} we can derive the most general form of a dual-resonance model  for $\pp$ scattering using different isospin-definite scalar functions:
    \begin{align} \label{isospin!}
          A^{(I)}(s,t,u) &= (-1)^I \big[ \maI{1}(s,t) + \maI{I}(s,t) \big ] + \big [ \maI{1}(s,u) +  (-1)^I \; \maI{I}(s,u) \big ] + \big[ \maI{1}(t,u) - \maI{I}(t,u)\big] \nonumber \\
          &+ \frac{1}{2}  \big[1-(-1)^I\big]  \sum_{I^\prime}  \; C_{II^\prime} \bigg [ \maI{I^\prime}(s,t) + (-1)^{I+I^\prime} \maI{I^\prime}(s,u)\bigg]   \\
          &+ \frac{1}{2} \big[1 + (-1)^I] \sum_{I^\prime} C_{II^\prime}\;\maI{I^\prime}(t,u) \nonumber.
    \end{align}
Here the coefficients $C_{II^\prime}$ are the elements of the isospin crossing matrix
	\begin{align}
	C=
	\setlength\arraycolsep{6pt}
	 	\begin{bmatrix*}[r]
	 	\dfrac{2}{3} & 2 & \dfrac{10}{3} \\[1.2em]
	 	\dfrac{2}{3} & 1 & -\dfrac{5}{3} \\[1.2em]
	 	\dfrac{2}{3} & -1 & \dfrac{1}{3}
	 	\end{bmatrix*},
	\end{align}
and allow the isospin-projections of the $t$ and $u$ channels to be related to \cref{isospin!} by permutations of Mandelstam variables.
 The extra factors of $\maI{1}$ are required to maintain crossing symmetry and come from the necessary $s$-dependence of anti-symmetric background terms.

 For our choice of scalar function we use an infinite sum of Euler function-like terms:
 	\begin{equation}
 		\maI{I}(s,t) = \sum_{n=1}^{n_{\text{max}}} \sum_{m=n}^{\infty} \sum_{k=0}^{m} \; c^{(I)}_{m,k} \frac{\Gamma( m- \alpha^{(I)}(s) ) \Gamma( m - 	\alpha^{(I)}(t))}{\Gamma(m + k - \alpha^{(I)}(s) - \alpha^{(I)}(t))}
 	\end{equation}
With appropriate choices of $c^{(I)}_{m,k}$ (c.f. charmonium paper) we can define:
	\begin{equation} \label{single-pole}
    	\mathcal{V}^{(I)}_n(s,t) = \bigg [\frac{2n - \as^{(I)} - \at^{(I)}}{(n- \as^{(I)})(n-\at^{(I)})} \bigg] \sum_{i=0}^n a^{(I)}_{n,i} \; (\as^{(I)} + \at^{(I)} )^{i} \; \bigg [ \frac{\Gamma(N + 1 - \as^{(I)}) \; \Gamma (N + 1 - \at^{(I)})}{\Gamma ( N + 1 - n) \; \Gamma (N+ n + 1 - \as^{(I)} - \at^{(I)})} \bigg ]
	\end{equation}
where $\as^{(I)} \equiv \az^{(I)} + {\apr}^{(I)} \; s$ is the linear Regge trajectory of isospin-$I$.
We see that \cref{single-pole} retains the usual behavior of the Veneziano amplitude \cref{eq:orig_venez} at when $\as \geq N$, and thus at high energies the Regge behavior is preserved. The low enegy behavior (in the resonance region), is a simple summation of simple poles in $s$ and $t$ channels when $\as$ or $\at = n$ respectively. The pole at $\as = n$ has a polynomial residue of $\mathcal{O}(n)$ in both $s$ and $t$ and will thus have partial waves of spin $0 \leq \ell \leq n$. The coefficients $a_{n,i}^{(I)}$ allow us to control the strengths of the couplings to individual resonant intermediate states and/or decouple unwanted resonances (discussed below).

Thus the full model becomes \cref{isospin!} with
	\begin{equation}
	\maI{I}(s,t) = \sum_{n=1}^{n_{\text{max}}} \mvI{I}_n(s,t).
	\end{equation}
The inputs then become parameterizations for Regge trajectories of each channel to reproduce physical resonance masses, the ``resonant region cutoff'', $N$, above which Regge behavior dominates. Fitting to data scattering data, we can extract couplings (and therefore partial widths) to individual resonant intermediate states.

 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Resonances}
From \cref{direct} we see that $A^{(I)}(s,t,u)$ will have direct-channel poles of isospin-$I$ at every $\as = n$. It will have ``wrong parity'' contributions from the background as discussed above.
Looking first at the physical poles with the correct Bose symmetry (surpressing isospin indices for ease of notation):
	\begin{align} \label{pole}
	A^{(I)}(m_n^2,t,u) \sim& \frac{1}{s -  m_n^2} \sum_{i=0}^n a^{(I)}_{n,i} \; \bigg [ \big(\alpha(m_n^2) + \alpha(t) \big)^i + (-1)^{I} \big(\alpha(m_n^2) + \alpha(u) \big)^i \bigg]
	\end{align}
where
	\begin{equation}
	m_n^2 \equiv {m_n^{(I)}}^2 = \frac{n - \az^{(I)}}{{\apr}^{(I)}}
	\end{equation}
is the mass of the resonance of isospin-$I$ at $\as = n$.
Looking at the residue of \cref{pole}, we can redefine the coefficients $a_{n,i}^{(I)}$ such that
	\begin{equation} \label{residue}
	\text{Res }[ A^{(I)}(m_n^2,t,u)] = \sum_{\ell=0}^n \gamma_{n,\ell}^{(I)} \; [1 + (-1)^{\ell+I}] \; P_\ell(z_s)
	\end{equation}
where we've used:
	\begin{equation}
	t(s,z_s) = -2k^2(s)(1-z_s) \quad \text{ and } \quad u(s, z_s) = -2k^2(s)(1+z_s)
	\end{equation}
	with $z_s$, the $s$-channel scattering angle, $k(s) = \sqrt{s - 4m^2_p}/2$ and Legendre polynomials $P_\ell$. We see Bose symmetry ($I+\ell$ is even) is explicitly enforced in \cref{residue}. We additionally have $n$ coefficients $\gamma_{n.\ell}^{(I)}$ related to the partial decay widths of each partial wave by
	\begin{equation}
	\Gamma^{(I)}_\ell = \frac{ - k(m^2_n)}{2m^2_n} \int^1_{-1} dz_s P_\ell \; \text{Res }A^{(I)}(m^2_n, z_s) = (2\ell +1) \frac{ - k(m^2_n)}{2m^2_n} \gamma^{(I)}_{n,\ell}.
	\end{equation}
Here we can address the problem of unwanted poles of wrong parity. These partial widths are completely undetermined, instead require being fit to data. However, by making the distinction of different Regge trajectories, couplings to resonances not the natural Bose symmetry of that trajectory (i.e. $I+\ell$ is even) are unphysical. For example the $\rho$ trajectory containing isospin-1 resonances ($\rho(770), \rho(1450), \rho_3(1690)$, etc) should not contain spin-even resonances.  We can set these couplings identically to zero, leaving $\lceil n/2 \rceil$ free couplings for the physical poles:
		\begin{equation}
		\gamma_{n,\ell}^{(I)} \equiv 0 \qquad \text{for odd} \; \ell+I.
		\end{equation}

We note that the unwanted background poles are exactly of this form and by setting these unphysical couplings to zero eliminates them. For example, taking $I = 0$ in \cref{isospin!},
    \begin{align} \label{Azero}
    A^{(0)}(s,t,u) &= \maI{0}(s,t) + \maI{0}(s,u) - \frac{1}{3} \, \maI{0}(t,u) \nonumber \\[0.5em]
                        &\; + \maI{1}(s,t) + \maI{1}(s,u) + 3 \, \maI{1}(t,u)  \\[0.5em]
                        &\; + \frac{10}{3} \, \maI{2}(t,u) \nonumber,
    \end{align}
the contributing poles come from terms of the form
	\begin{equation}
			[ \ma^{(I^\prime)}(s,t) + (-1)^I \ma^{(I^\prime)}(s,u) ] \sim \sum_{\ell=0}^n \gamma_{n,\ell}^{(I^\prime)} \; [1 + (-1)^{\ell+I}] \; P_\ell(z_s)
	\end{equation}
where for even (odd) $I$, the background $I^\prime$ is odd (even). Thus, Bose symmetry for $I=0$ will have contributions from even resonances on the isospin-1 trajectory which are identically zero. Similarly we can in general always decouple odd partial waves on the isospin-0 and 2 trajectories to prevent unphysical contributions to the isospin-1 amplitude.

\end{document}


% Part: first-order-logic
% Chapter: model-theory
% Section: partial-iso

\documentclass[../../../include/open-logic-section]{subfiles}

\begin{document}

\olfileid{mod}{bas}{pis}
\section{Partial Isomorphisms}

\begin{defn}
  Given two !!{structure}s $\Struct{M}$ and $\Struct{N}$, a
  \emph{partial isomorphism} from $\Struct{M}$ to $\Struct{N}$ is a
  finite partial function $p$ taking arguments in $\Domain M$ and returning
  values in $\Domain N$, which satisfies the isomorphism conditions from
  \olref[iso]{defn:isomorphism} on its domain:
  \begin{enumerate}
  \item $p$ is !!{injective};
  \item for every !!{constant}~$c$: if $p(\Assign{c}{M})$ is defined,
    then $p(\Assign{c}{M}) = \Assign{c}{N}$;
  \item for every $n$-place !!{predicate} $P$: if $a_1$, \dots, $a_n$
    are in the domain of $p$, then $\langle a_1, \dots, a_n\rangle \in
    \Assign P M$ if and only if $\langle p(a_1), \dots, p(a_n) \rangle
    \in \Assign P N$;
  \item for every $n$-place !!{function} $f$: if $a_1$, \dots, $a_n$
    are in the domain of $p$, then $p(\Assign f M (a_1, \dots,a_n))
    = \Assign f N (p(a_1), \ dots, p(a_n))$.
  \end{enumerate}
  That $p$ is finite means that $\dom{p}$ is finite.
\end{defn}

Notice that the empty function~$\emptyset$ is always a partial
isomorphism between any two !!{structure}s.

\begin{defn}\ollabel{defn:partialisom}
  Two !!{structure}s $\Struct{M}$ and $\Struct{N}$, are
  \emph{partially isomorphic}, written $\Struct{M} \iso[p]
  \Struct{N}$, if and only if there is a non-empty set $I$
  of partial isomorphisms between $\Struct{M}$ and $\Struct{N}$
  satisfying the \emph{back-and-forth} property:
  \begin{enumerate}
  \item (\emph{Forth}) For every $p \in I$ and $a \in \Domain M$
    there is $q \in I$ such that $p \subseteq q$ and $a$ is
    in the domain of $q$;
  \item (\emph{Back}) For every $p \in I$ and $b \in \Domain N$
    there is $q \in I$ such that $p \subseteq q$ and $b$ is
    in the range of $q$.
  \end{enumerate}
\end{defn}

\begin{thm}\ollabel{thm:p-isom1}
  If $\Struct{M} \iso[p] \Struct{N}$ and $\Struct{M}$ and
  $\Struct{N}$ are !!{enumerable}, then $\Struct{M} \iso
  \Struct{N}$.
\end{thm}

\begin{proof}
  Since $\Struct{M}$ and $\Struct{N}$ are !!{enumerable}, let $\Domain{M} =
  \{a_0, a_1, \ldots \}$ and $\Domain{N} = \{b_0, b_1, \ldots \}$. Starting
  with an arbitrary $p_0 \in I$, we define an increasing
  sequence of partial isomorphisms $p_0 \subseteq p_1 \subseteq p_2
  \subseteq \cdots$ as follows:
  \begin{enumerate}
  \item if $n+1$ is odd, say $n = 2r$, then using the Forth property
    find a $p_{n+1} \in I$ such that $p_n \subseteq p_{n+1}$
    and $a_r$ is in the domain of $p_{n+1}$;
  \item if $n+1$ is even, say $n+1 =2r$, then using the Back property
    find a $p_{n+1} \in I$ such that $p_n \subseteq p_{n+1}$
    and $b_r$ is in the range of $p_{n+1}$.
  \end{enumerate}
If we now put:
\[
p = \bigcup_{n\ge 0} p_n,
\]
we have that $p$ is a an isomorphism between $\Struct{M}$ and
$\Struct{N}$.
\end{proof}

\begin{prob}
  Show in detail that $p$ as defined in
  \olref[mod][bas][pis]{thm:p-isom1} is in fact an isomorphism.
\end{prob}

\begin{thm}\ollabel{thm:p-isom2}
  Suppose $\Struct{M}$ and $\Struct{N}$ are !!{structure}s for a
  purely relational !!{language} (!!a{language} containing only
  !!{predicate}s, and no !!{function}s or constants). Then if
  $\Struct{M} \iso[p] \Struct{N}$, also $\Struct{M} \elemequiv
  \Struct{N}$.
\end{thm}

\begin{proof}
  By induction on !!{formula}s, one shows that if $a_1$, \dots, $a_n$ and
  $b_1$, \dots, $b_n$ are such that there is a partial isomorphism $p$
  mapping each $a_i$ to $b_i$ and $s_1(x_i) =a_i$ and $s_2(x_i) =b_i$
  (for $i =1$, \dots,~$n$), then $\Sat{M}{!A}[s_1]$ if
  and only if $\Sat{N}{!A}[s_2]$. The case for $n=0$
  gives $\Struct{M} \elemequiv \Struct{N}$.
\end{proof}

\begin{rem}
If !!{function}s are present, the previous result is still true, but
one needs to consider the isomorphism induced by $p$ between the
sub!!{structure} of $\Struct{M}$ generated by $a_1$, \dots, $a_n$ and the
sub!!{structure} of $\Struct{N}$ generated by $b_1$, \dots, $b_n$.
\end{rem}

The previous result can be ``broken down'' into stages by establishing a
connection between the number of nested quantifiers in a !!{formula} and
how many times the relevant partial isomorphisms can be extended.

\begin{defn}
  For any !!{formula} $!A$, the \emph{quantifier rank} of $!A$, denoted
  by $\QuantRank{!A} \in \Nat$, is recursively defined as
  the highest number of nested quantifiers in $!A$.  Two
  !!{structure}s $\Struct{M}$ and $\Struct{N}$ are \emph{$n$-equivalent},
  written $\Struct{M} \elemequiv[n] \Struct{N}$, if they agree on all
  sentences of quantifier rank less than or equal to~$n$.
\end{defn}

\begin{prop}\ollabel{prop:qr-finite}
  Let $\Lang{L}$ be a finite purely relational !!{language}, i.e., a
  !!{language} containing finitely many !!{predicate}s and !!{constant}s,
  and no !!{function}s. Then for each $n \in \Nat$ there are
  only finitely many first-order !!{sentence}s in the !!{language}
  $\Lang{L}$ that have quantifier rank no greater than $n$, up to
  logical equivalence.
\end{prop}

\begin{proof}
  By induction on $n$.
\end{proof}

\begin{defn}
  Given a !!{structure} $\Struct{M}$, let $\Domain M^{<\omega}$ be the set of
  all finite sequences over $\Domain{M}$. We use $\mathbf{a},
  \mathbf{b}, \mathbf{c}, \ldots$ to range over finite sequences of
  elements. If $\mathbf{a} \in \Domain{M}^{<\omega}$ and $a \in \Domain{M}$, then
  $\mathbf{a}a$ represents the \emph{concatenation} of $\mathbf{a}$ with $a$.
\end{defn}

\begin{defn}
  Given !!{structure}s $\Struct{M}$ and $\Struct{N}$, we define
  relations $I_n \subseteq \Domain M^{<\omega} \times \Domain N^{<\omega}$ between
  sequences of equal length, by recursion on $n$ as follows:
   \begin{enumerate}
   \item $I_0(\mathbf{a},\mathbf{b})$ if and only if $\mathbf{a}$ and
     $\mathbf{b}$ satisfy the same atomic !!{formula}s in  $\Struct{M}$
     and  $\Struct{N}$; i.e., if $s_1(x_i) = a_i$ and $s_2(x_i) =
     b_i$ and $!A$ is atomic with all !!{variable}s among
     $x_1$, \dots,~$x_n$, then $\Sat{M}{!A}[s_1]$ if and
     only if~$\Sat{N}{!A}[s_2]$.
   \item $I_{n+1} (\mathbf{a},\mathbf{b})$ if and only if for every
     $a\in A$ there is a $b\in B$ such that $I_n
     (\mathbf{a}a,\mathbf{b}b)$, and vice-versa.
   \end{enumerate}
\end{defn}

\begin{defn}
  Write $\Struct{M} \approx_n \Struct{N}$ if
  $I_n(\emptyseq,\emptyseq)$ holds of $\Struct{M}$ and
  $\Struct{N}$ (where $\emptyseq$ is the empty sequence).
\end{defn}

\begin{thm}\ollabel{thm:b-n-f}
  Let $\Lang{L}$ be a purely relational !!{language}. Then $I_n
  (\mathbf{a},\mathbf{b})$ implies that for every $!A$ such that
  $\QuantRank{!A} \le n$, we have $\Sat{M}{!A}[\mathbf{a}]$ if and
  only if $\Sat{N}{!A}[\mathbf{b}]$ (where again $\mathbf{a}$
  satisfies $!A$ if any $s$ such that $s(x_i) = a_i$ satisfies
  $!A$). Moreover, if $\Lang{L}$ is finite, the converse also holds.
\end{thm}

\begin{proof}
  The proof that $I_n(\mathbf{a},\mathbf{b})$ implies that
  $\mathbf{a}$ and $\mathbf{b}$ satisfy the same !!{formula}s of
  quantifier rank no greater than $n$ is by an easy induction on
  $!A$. For the converse we proceed by induction on $n$, using
  \olref{prop:qr-finite}, which ensures that for each $n$
  there are at most finitely many non-equivalent !!{formula}s of that
  quantifier rank.

  For $n=0$ the hypothesis that $\mathbf{a}$ and $\mathbf{b}$ satisfy
  the same quantifier-free !!{formula}s gives that they satisfy the same
  atomic ones, so that $I_0(\mathbf{a},\mathbf{b})$.

  For the $n+1$ case, suppose that $\mathbf{a}$ and $\mathbf{b}$
  satisfy the same !!{formula}s of quantifier rank no greater than
  $n+1$; in order to show that $I_{n+1}(\mathbf{a},\mathbf{b})$
  suffices to show that for each $a \in \Domain M$ there is a $b \in
  \Domain N$ such that $I_n(\mathbf{a}a,\mathbf{b}b)$, and by the
  inductive hypothesis again suffices to show that for each $a \in
  \Domain M$ there is a $b \in \Domain N$ such that $\mathbf{a}a$ and
  $\mathbf{b}b$ satisfy the same !!{formula}s of quantifier rank no
  greater than $n$.

  Given $a \in \Domain M$, let $!T^a_n$ be set of !!{formula}s
  $!B(x,\mathbf{y})$ of rank no greater than $n$ satisfied by
  $\mathbf{a}a$ in $\Struct{M}$; $\tau^a_n$ is finite, so we can
  assume it is a single first-order !!{formula}. It follows that
  $\mathbf{a}$ satisfies $\lexists[x][!T^a_n(x,\mathbf{y})]$, which
  has quantifier rank no greater than $n+1$. By hypothesis
  $\mathbf{b}$ satisfies the same !!{formula} in $\Struct{N}$, so that
  there is a $b \in \Domain N$ such that $\mathbf{b}b$ satisfies
  $!T^a_n$; in particular, $\mathbf{b}b$ satisfies the same
  !!{formula}s of quantifier rank no greater than $n$ as
  $\mathbf{a}a$. Similarly one shows that for every $b \in \Domain N$
  there is $a\in \Domain M$ such that $\mathbf{a}a$ and $\mathbf{b}b$
  satisfy the same !!{formula}s of quantifier rank no greater than $n$,
  which completes the proof.
\end{proof}

\begin{cor}\ollabel{cor:b-n-f}
  If $\Struct{M}$ and $\Struct{N}$ are purely relational !!{structure}s
  in a finite !!{language}, then $\Struct{M} \approx_n\Struct{N}$ if and
  only if $\Struct{M} \elemequiv[n] \Struct{N}$. In particular
  $\Struct{M} \elemequiv \Struct{N}$ if and only if for each $n$,
  $\Struct{M} \approx_n \Struct{N}$ .
\end{cor}

\end{document}


\section{Results}
Results for two different cases are being presented, one the one-dimensional Ising Model and another for the the two-dimensional Ising model. We begin with looking at the. 1D Ising model.

\subsection{1D Ising model}
\subsubsection{Fitting with linear regression}
For linear regression we got coefficients of $\bm{J}$ in \eqref{eq:1d-ising-linreg} as the following presented in figure \ref{fig:reg-coef-heatmap},
\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.9\textwidth}
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=1]{../fig/{regression_ising_1d_heatmap_lambda0.001}.pdf}
        \caption{$\lambda=10^{-3}$}
        \label{fig:linreg-hm-1e-3}
    \end{subfigure} \\
    \begin{subfigure}[b]{0.9\textwidth}
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=1]{../fig/{regression_ising_1d_heatmap_lambda0.1}.pdf}
        \caption{$\lambda=10^{-1}$}
        \label{fig:linreg-hm-1e-1}
    \end{subfigure} \\
    \begin{subfigure}[b]{0.9\textwidth}
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=1]{../fig/{regression_ising_1d_heatmap_lambda10.0}.pdf}
    \caption{$\lambda=10^{1}$}
        \label{fig:linreg-hm-1e2}
    \end{subfigure}
    \caption{Heat map plots of the $\bm{J}$ in \eqref{eq:1d-ising-linreg} retrieved from OLS, Ridge and Lasso. Gathered using $N_\mathrm{train}=5000$.}
    \label{fig:reg-coef-heatmap}
\end{figure}

The $R^2$ score of the OLS, Ridge and Lasso can be seen in figure \ref{fig:linreg-r2},
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/r2_ols_ridge_lasso.pdf}
    \caption{$R^2$ score for different Ordinary Least Squares(OLS), Ridge and Lasso regression. Retrieved $N_\mathrm{train}=5000$ and $N_\mathrm{test}=5000$ on a 1D Ising model of size $L=20$.}
    \label{fig:linreg-r2}
\end{figure}

The bias-variance decomposition for Ridge and Lasso using bootstrap and cross validation can be viewed in figure \ref{fig:linreg-bias-variance-decomp-ridge} and \ref{fig:linreg-bias-variance-decomp-lasso}.

\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/ridge_bs_bias_variance_analysis.pdf}
        \caption{Bootstrap.}
        \label{fig:linreg-bias-variance-decomp-bs-ridge}
    \end{subfigure}%
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/ridge_cv_bias_variance_analysis.pdf}
        \caption{$k$-fold Cross Validation.}
        \label{fig:linreg-bias-variance-decomp-cv-ridge}
    \end{subfigure}
    \caption{A bias-variance decomposition of Ridge regression using bootstrapping\ref{fig:linreg-bias-variance-decomp-bs-ridge} and cross-validation\ref{fig:linreg-bias-variance-decomp-cv-ridge}.}
    \label{fig:linreg-bias-variance-decomp-ridge}
\end{figure}

\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/lasso_bs_bias_variance_analysis.pdf}
        \caption{Bootstrap.}
        \label{fig:linreg-bias-variance-decomp-bs-lasso}
    \end{subfigure}%
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/lasso_cv_bias_variance_analysis.pdf}
        \caption{$k$-fold Cross Validation.}
        \label{fig:linreg-bias-variance-decomp-cv-lasso}
    \end{subfigure}
    \caption{A bias-variance decomposition of Lasso regression using bootstrapping\ref{fig:linreg-bias-variance-decomp-bs-lasso} and cross-validation\ref{fig:linreg-bias-variance-decomp-cv-lasso}.}
    \label{fig:linreg-bias-variance-decomp-lasso}
\end{figure}

\subsubsection{Fitting with a neural network}
By setting the output activation function to the identity and by having zero hidden layers, we are essentially performing a regression analysis on the 1D Ising model. We generate the same amount of data by inputing the same RNG(random number generator) seed. A fit using $N_\mathrm{train}=400$, $N_\mathrm{train}=5000$ and $N_\mathrm{test}=5000$ for $\lambda=10^{-3}, 10^{-1}, 10^1$ can be seen in figure \ref{fig:mlp_coefs}.

\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda0.0001_N800}.pdf}
        \caption{$N_\mathrm{train}=400$, $\lambda=10^{-3}$}
        \label{fig:mlp-reg-heatmap400-lmb-3}
    \end{subfigure} \qquad \qquad \qquad
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda0.0001_N100000}.pdf}
        \caption{$N_\mathrm{train}=5000$, $\lambda=10^{-3}$}
        \label{fig:mlp-reg-heatmap5000-lmb-3}
    \end{subfigure} \\
        \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda0.1_N800}.pdf}
        \caption{$N_\mathrm{train}=400$, $\lambda=10^{-1}$}
        \label{fig:mlp-reg-heatmap400-lmb-1}
    \end{subfigure} \qquad \qquad \qquad
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda0.1_N100000}.pdf}
        \caption{$N_\mathrm{train}=5000$, $\lambda=10^{-1}$}
        \label{fig:mlp-reg-heatmap5000-lmb-1}
    \end{subfigure} \\
        \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda10.0_N800}.pdf}
        \caption{$N_\mathrm{train}=400$, $\lambda=10^1$}
        \label{fig:mlp-reg-heatmap400-lmb1}
    \end{subfigure} \qquad \qquad \qquad
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[trim={1.5cm 3.5cm 0 3.5cm},clip, scale=0.6]{../fig/{mlp_ising_1d_heatmap_lambda10.0_N100000}.pdf}
        \caption{$N_\mathrm{train}=5000$, $\lambda=10^1$}
        \label{fig:mlp-reg-heatmap5000-lmb1}
    \end{subfigure} \\
    \caption{Heat map plot of the coefficients of $\bm{J}$ in \eqref{eq:1d-ising-linreg} using neural networks with different regularizations for $\lambda=10^{-3}, 10^{-1}, 10^1$.}
    \label{fig:mlp-coefs}
\end{figure}

The $R^2$ score of the neural network using L$^1$, L$^2$ and no regularization can be seen in figure \ref{fig:mlp-r2},
\begin{figure}[H]
    \centering
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/{mlp_r2_ols_ridge_lasso800}.pdf}
        \caption{$N_\mathrm{train}=400$}
        \label{fig:mlp-r2-800}
    \end{subfigure}%
    \begin{subfigure}[b]{0.5\textwidth}
        \centering
        \includegraphics[scale=0.5]{../fig/{mlp_r2_ols_ridge_lasso100000}.pdf}
        \caption{$N_\mathrm{train}=5000$}
        \label{fig:mlp-r2-5000}
    \end{subfigure}
    \caption{$R^2$ score for the neural network using L$^1$ (Lasso), L$^2$ (Ridge) and no regularization (OLS). Retrieved $N_\mathrm{train}=400$ on the left and $N_\mathrm{train}=5000$ on the right, for a 1D Ising model of size $L=20$.}
    \label{fig:mlp-r2}
\end{figure}

% TODO: rerun mlp regression with N_samples = 10000 as I run for too much :|

\subsection{2D Ising model}
As stated in the section about the 2D Ising model \ref{sec:2d-ising-model}, the classification will focus on evaluating the phases of different lattice configurations, and wetter or not it is below or above a critical temperature. We begin by listing the results from the logistic regression.
\subsubsection{Classification through logistic regression}
In logistic regression we investigated the behavior of the classification and compared it to that of SciKit Learn\cite{scikit-learn}, using the standard logistic regression method\footnote{See \href{https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html}{Logistic Regression documentation}} and
SciKit Learn's SGD(Stochastic Gradient Descent) implementation \footnote{See \href{https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html}{SGD documentation}}. This gave the results found in figure \ref{fig:logreg-accuracy-sklearn-comparison}.

\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/logistic_accuracy_sklearn_comparison.pdf}
    \caption{The accuracy for our implementation of logistic regression versus that of SciKit learn.}
    \label{fig:logreg-accuracy-sklearn-comparison}
\end{figure}

\subsubsection{Classification through neural networks}
For classifying the states through a neural network, we looked at several different hyper parameters. All runs were made using $N_{samples}=10000$ except stated other wise. The training percent was 0.5. We start by comparing two different cost functions and their layer outputs,
\begin{itemize}
    \item Cross entropy with softmax layer output\eqref{eq:ce-mlp-cost}
    \item MSE with sigmoidal layer output.\eqref{eq:mse-mlp-cost}
\end{itemize}
These cost functions following behavior for epochs seen in following figure \ref{fig:mlp-cost-function-comparison},
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_cost_functions.pdf}
    \caption{A comparison in accuracy scores between the MSE and (CE) Cross Entropy loss functions over 500 epochs. The output layer for MSE is sigmoidal, the output layer for CE is softmax. The learning parameter was $\eta=0.001$ and we used the inverse learning rate\eqref{eq:inverse-eta}.}
    \label{fig:mlp-cost-function-comparison}
\end{figure}

We then wish to to investigate the effects of having different initial weights. Given the initial weights \textit{large} and \textit{default} as listed in section \ref{sec:nn-weights}, we get the results as seen in figure \ref{fig:mlp-epoch-init-weights},
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_weight_inits.pdf}
    \caption{A comparison in accuracy scores between the initial weights \textit{large} and \textit{default} as listed in section \ref{sec:nn-weights}. The run was for 500 epochs. The cost function was set to cross entropy and had softmax output activation. The learning parameter was $\eta=0.001$ and we used the inverse learning rate\eqref{eq:inverse-eta}.}
    \label{fig:mlp-epoch-init-weights}
\end{figure}

% skriv inn lambda ting her !=")#(097 21841 \\\\}∆∑◊∂"

An investigation into different layer activations\ref{sec:layer-acts} was performed for both the MSE- and the CE-cost function. The results from MSE can be seen in figure
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_mse.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for MSE as cost function. The run was for 500 epochs. The learning rate was set with the inverse learning rate \eqref{eq:inverse-eta} with an $\eta_0=0.001$ and $\lambda=0.0$.}
    \label{fig:mlp-epoch-activations-mse}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_log_loss.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for cross entropy as cost function. The run was for 500 epochs. The learning rate was set with the inverse learning rate \eqref{eq:inverse-eta} with an $\eta_0=0.001$ and $\lambda=0.0$.}
    \label{fig:mlp-epoch-activations-log-loss}
\end{figure}

We then move on to an investigation for different L$^2$ regularization strengths $\lambda$ versus different constant learning rates $\eta$. A run with 500 epochs, cross entropy and sigmoidal hidden layer activation can be seen in figure \ref{fig:mlp-eta-lambda},
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_lambda_eta.pdf}
    \caption{The accuracy as function of the L$^2$ regularization parameter $\lambda$ and constant training rate $\eta$. The run was for 500 epochs and with cross entropy as cost function, softmax output and sigmoidal hidden layer activation. The hidden layer was 10 neurons large.}
    \label{fig:mlp-eta-lambda}
\end{figure}

A comparison of the accuracy score\eqref{eq:mlp-accuracy} as a function of L$^2$ regularization parameter and hidden layer size(the neurons) can be viewed in figure \ref{fig:mlp-lambda-neurons}.
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_lambda_neurons.pdf}
    \caption{The accuracy score as function of the L$^2$ regularization parameter $\lambda$ and the number of neurons. The run was for 500 epochs and with cross entropy as cost function, softmax output and sigmoidal hidden layer activation. The learning rate was set with the inverse learning rate \eqref{eq:inverse-eta} with an $\eta_0=0.001$.}
    \label{fig:mlp-lambda-neurons}
\end{figure}

The accuracy score\eqref{eq:mlp-accuracy} as a function of the hidden layer size(the neurons) and the training data size as percentage of of a $N_\mathrm{samples}=10000$ training data, can be viewed in figure \ref{fig:mlp-neurons-ts}.
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_neurons_training_size.pdf}
    \caption{The accuracy score as function of the number of neurons and the training data size percentage of $N_\mathrm{samples}=10000$. The run was for 500 epochs and with cross entropy as cost function, softmax output and sigmoidal hidden layer activation. The learning rate was set with the inverse learning rate \eqref{eq:inverse-eta} with an $\eta_0=0.001$ and $\lambda=0.0$. }
    \label{fig:mlp-neurons-ts}
\end{figure}

The accuracy score\eqref{eq:mlp-accuracy} as a function of the hidden layer size(the neurons) and the learning rate $\eta$, can be viewed in figure \ref{fig:mlp-neurons-ts}.
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_neurons_eta.pdf}
    \caption{The accuracy score as function of the number of neurons and the learning rate $\eta$. The run was for 500 epochs and with cross entropy as cost function, softmax output and sigmoidal hidden layer activation. The regularization strength was set to $\lambda=0.0$}.
    \label{fig:mlp-neurons-eta}
\end{figure}

The accuracy score\eqref{eq:mlp-accuracy} as a function of L$^2$ regularization strength $\lambda$ and the mini batch size in the SGD\ref{alg:sgd}, can be viewed in figure \ref{fig:mlp-lambda-mb}.
\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_lambda_mini_batch_size.pdf}
    \caption{The accuracy score as function of the  L$^2$ regularization strength $\lambda$ and the mini batch size in the SGD\ref{alg:sgd}. The run was for 500 epochs and with cross entropy as cost function, softmax output and sigmoidal hidden layer activation and the inverse learning rate \eqref{eq:inverse-eta} with $\eta_0=0.001$.}
    \label{fig:mlp-lambda-mb}
\end{figure}

After choosing a set of optimal parameters seen in table \ref{tab:opt_ols_result}, we get the results that is seen in figure \ref{fig:mlp-epoch-activations-mse-optimal} with the MSE cost function, figure \ref{fig:mlp-epoch-activations-log-loss-optimal} with the CE cost function.
\begin{table}[H]
    \centering
    \caption{A set of optimal parameters}
    \begin{tabular}{l l} % 6 columns
        \specialrule{.1em}{.05em}{.05em}
        Parameters & Values \\ \hline
        $N_\mathrm{neurons}$    & 10 \\
        $\lambda$               & 0.1 \\
        $N_\mathrm{mb}$         & 20 \\
        $N_\mathrm{epochs}$     & 500 \\
        $\eta$ (constant)       & 0.001 \\
        \specialrule{.1em}{.05em}{.05em}
    \end{tabular}
    \label{tab:opt_ols_result}
\end{table}

\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_mse2.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for MSE as cost function using \textit{optimal parameters}. The optimal parameters can be viewed in table \ref{tab:opt_ols_result}.}
    \label{fig:mlp-epoch-activations-mse-optimal}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_log_loss2.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for cross entropy as cost function using \textit{optimal parameters}. The optimal parameters can be viewed in table \ref{tab:opt_ols_result}.}
    \label{fig:mlp-epoch-activations-log-loss-optimal}
\end{figure}

Another set of optimal parameters were chosen a listed in table \ref{tab:opt_ols_result2} provided us with the figures seen in \ref{fig:mlp-epoch-activations-mse-optimal} and \ref{fig:mlp-epoch-activations-log-loss-optimal}.
% Taking the last 50 accuracy scores gives as with tanh activation function, Accuracy$=0.999(21)$.
\begin{table}[H]
    \centering
    \begin{tabular}{l l} % 6 columns
        \specialrule{.1em}{.05em}{.05em}
        Parameters & Values \\ \hline
        $N_\mathrm{neurons}$    & 20 \\
        $\lambda$               & 0.1 \\
        $N_\mathrm{mb}$         & 30 \\
        $N_\mathrm{epochs}$     & 500 \\
        $\eta$ (constant)       & 0.001 \\
        \specialrule{.1em}{.05em}{.05em}
    \end{tabular}
    \label{tab:opt_ols_result2}
\end{table}

\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_mse_optimal3.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for MSE as cost function using \textit{optimal parameters}. The optimal parameters can be viewed in table \ref{tab:opt_ols_result2}.}
    \label{fig:mlp-epoch-activations-mse-optimal}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=1.0]{../fig/mlp_epoch_activations_log_loss_optimal3.pdf}
    \caption{A comparison in accuracy scores between the hidden layer activation functions(see section \ref{sec:layer-acts}) for cross entropy as cost function using \textit{optimal parameters}. The optimal parameters can be viewed in table \ref{tab:opt_ols_result2}.}
    \label{fig:mlp-epoch-activations-log-loss-optimal}
\end{figure}