\section{Center of Mass}\label{sec:centerofmass}

Suppose a beam is 10 meters long, and that there are three weights on
the beam: a 10 kilogram weight 3 meters from the left end, a 5
kilogram weight 6 meters from the left end, and a 4 kilogram weight 8
meters from the left end. Where should a fulcrum be placed so that the
beam balances? Let's assign a scale to the beam, from 0 at the left
end to 10 at the right, so that we can denote locations on the beam
simply as $x$ coordinates; the weights are at $x=3$, $x=6$, and $x=8$,
as in Figure~\ref{fig:massesonbeam}.

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <1truecm,1truecm>
\setplotarea x from 0 to 10, y from -0.5 to 1
\linethickness1pt
\axis bottom shiftedto y=0 ticks length <2pt>
  withvalues {3} {6} {8} / at 3 6 8 / /
\setlinear
\plot 5 0 5.2 -0.4 4.8 -0.4 5 0 /
\plot 3.4 0 3.4 0.8 2.6 0.8 2.6 0 /
\put {$10$} at 3 0.4
\plot 6.4 0 6.4 0.8 5.6 0.8 5.6 0 /
\put {$5$} at 6 0.4
\plot 8.4 0 8.4 0.8 7.6 0.8 7.6 0 /
\put {$4$} at 8 0.4
\endpicture}}
\caption{A beam with three masses.}
\label{fig:massesonbeam}
\end{figure}

Suppose to begin with that the fulcrum is placed at $x=5$. What will
happen? Each weight applies a force to the beam that tends to rotate
it around the fulcrum; this effect is measured by a quantity called
\dfont{torque}, proportional to the mass times the
distance from the fulcrum. Of course, weights on different sides of
the fulcrum rotate the beam in opposite directions. We can distinguish
this by using a signed distance in the formula for torque. So with the
fulcrum at 5, the torques induced by the three weights will be
proportional to $(3-5)10=-20$, $(6-5)5=5$, and $(8-5)4=12$. For the
beam to balance, the sum of the torques must be zero; since the sum is
$-20+5+12=-3$, the beam rotates counter-clockwise, and to get the beam
to balance we need to move the fulcrum to the left. To calculate
exactly where the fulcrum should be, we let $\ds \bar x$ denote the
location of the fulcrum when the beam is in balance. The total torque
on the beam is then $\ds (3-\bar x)10+(6-\bar x)5+(8-\bar x)4=92-19\bar
x$. Since the beam balances at $\ds \bar x$ it must be that
$\ds 92-19\bar x=0$ or $\ds \bar x=92/19\approx 4.84$, that is, the fulcrum
should be placed at $x=92/19$ to balance the beam.

Now suppose that we have a beam with varying density---some portions
of the beam contain more mass than other portions of the same size. We
want to figure out where to put the fulcrum so that the beam balances.

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <1truecm,1truecm>
\setplotarea x from 0 to 10, y from 0 to 1
\putrule from 0 0 to 10 0
\putrule from 10 1 to 10 0
\putrule from 0 1 to 10 1
\putrule from 0 0 to 0 1
\putrule from 1 0 to 1 1
\putrule from 2 0 to 2 1
\putrule from 3 0 to 3 1
\putrule from 4 0 to 4 1
\putrule from 5 0 to 5 1
\putrule from 6 0 to 6 1
\putrule from 7 0 to 7 1
\putrule from 8 0 to 8 1
\putrule from 9 0 to 9 1
\put {$m_0$} at 0.5 0.5
\put {$m_1$} at 1.5 0.5
\put {$m_2$} at 2.5 0.5
\put {$m_3$} at 3.5 0.5
\put {$m_4$} at 4.5 0.5
\put {$m_5$} at 5.5 0.5
\put {$m_6$} at 6.5 0.5
\put {$m_7$} at 7.5 0.5
\put {$m_8$} at 8.5 0.5
\put {$m_9$} at 9.5 0.5
\endpicture}}
\caption{A solid beam.}
\label{fig:solidbeam}
\end{figure}

\begin{example}{Balance Point of a Beam}{BalancePointBeam}
Find the balance point of a solid beam, illustrated in Figure~\ref{fig:solidbeam},
assuming the beam is 10 meters long and that the density is $1+x$ kilograms
per meter at location $x$ on the beam.
\end{example}
\begin{solution}
To approximate the solution, we can think of the beam as a sequence of weights ``on''
a beam. For example, we can think of the portion of the beam between
$x=0$ and $x=1$ as a weight sitting at $x=0$, the portion between
$x=1$ and $x=2$ as a weight sitting at $x=1$, and so on, as indicated
in Figure~\ref{fig:solidbeam}. We then
approximate the mass of the weights by assuming that each portion of
the beam has constant density. So the mass of the first weight is
approximately $\ds m_0=(1+0)1=1$ kilograms, namely, $(1+0)$
kilograms per meter times 1 meter. The second weight is $\ds
m_1=(1+1)1=2$ kilograms, and so on to the tenth weight with $\ds
m_9=(1+9)1=10$ kilograms.  So in this case the total torque is
\[(0-\bar x)m_0+(1-\bar x)m_1+\cdots+(9-\bar x)m_9=
  (0-\bar x)1+(1-\bar x)2+\cdots+(9-\bar x)10.\]
If we set this to zero and solve for $\ds \bar x$ we get $\ds \bar x=6$.
In general, if we divide the beam into $n$ portions, the mass
of weight number $i$ will be $\ds m_i=(1+x_i)(x_{i+1}-x_i)=(1+x_i)\Delta x$
and the torque induced by weight number $i$ will be
$\ds (x_i-\bar x)m_i=(x_i-\bar x)(1+x_i)\Delta x$. The total torque is then
\begin{align*}
(x_0-\bar x)(1+x_0)\Delta x&+(x_1-\bar x)(1+x_1)\Delta x+\cdots+(x_{n-1}-\bar x)(1+x_{n-1})\Delta x	\\
  &=\sum_{i=0}^{n-1} x_i(1+x_i)\Delta x-\sum_{i=0}^{n-1}\bar x(1+x_i)\Delta x	\\
  &=\sum_{i=0}^{n-1} x_i(1+x_i)\Delta x-\bar x\sum_{i=0}^{n-1}(1+x_i)\Delta x.
\end{align*}
If we set this equal to zero and solve for $\ds \bar x$ we get an
approximation to the balance point of the beam:
\begin{align*}
  0&=\sum_{i=0}^{n-1} x_i(1+x_i)\Delta x-
    \bar x\sum_{i=0}^{n-1}(1+x_i)\Delta x	\\
  \bar x\sum_{i=0}^{n-1}(1+x_i)\Delta x&=\sum_{i=0}^{n-1} x_i(1+x_i)\Delta x	\\
  \bar x&={\ds\sum_{i=0}^{n-1} x_i(1+x_i)\Delta x\over
     \ds\sum_{i=0}^{n-1}(1+x_i)\Delta x}.
\end{align*}
The denominator of this fraction has a very familiar
interpretation. Consider one term of the sum in the denominator: $\ds
(1+x_i)\Delta x$. This is the density near $\ds x_i$ times a short
length, $\Delta x$, which in other words is approximately the mass of
the beam between $\ds x_i$ and $\ds x_{i+1}$. When we add these up we
get approximately the mass of the beam.

Now each of the sums in the fraction has the right form to turn into
an integral, which in turn gives us the exact value of $\ds \bar x$:
\[
  \bar x={\ds\int_0^{10} x(1+x)\,dx\over\ds\int_{0}^{10}(1+x)\,dx}.
\]
The numerator of this fraction is called the \dfont{moment} of the system around zero:
\[\int_0^{10} x(1+x)\,dx=\int_0^{10} x+x^2\,dx={1150\over 3},\]
and the denominator is the mass of the beam:
\[\int_0^{10} (1+x)\,dx=60,\]
and the balance point, officially called the \dfont{center of
mass} is
\[\bar x={1150\over 3}{1\over 60} = {115\over 18}\approx 6.39.\]
\end{solution}

It should be apparent that there was nothing special about the density
function $\sigma(x)=1+x$ or the length of the beam, or even that the
left end of the beam is at the origin. In general, if the density of
the beam is $\sigma(x)$ and the beam covers the interval $[a,b]$, the
moment of the beam around zero is
\[M_0=\int_a^b x\sigma(x)\,dx\]
and the total mass of the beam is
\[M=\int_a^b \sigma(x)\,dx\]
and the center of mass is at
\[\bar x={M_0\over M}.\]
\begin{example}{Center of Mass of a Beam}{CenterMassBeam}
Suppose a beam lies on the $x$-axis between 20 and 30, and
has density function $\sigma(x)=x-19$. Find the center of mass.
\end{example}
\begin{solution}
This is the same as the previous example except that the beam has been
moved. Note that the density at the left end is $20-19=1$ and at the
right end is $30-19=11$, as before. Hence the center of mass must be
at approximately $20+6.39=26.39$. Let's see how the calculation works
out.
\begin{align*}
  M_0&=\int_{20}^{30} x(x-19)\,dx=\int_{20}^{30} x^2-19x\,dx=
    \left.{x^3\over3}-{19x^2\over2}\right|_{20}^{30}={4750\over3}	\\
  M&=\int_{20}^{30} x-19\,dx=\left.{x^2\over2}-19x\right|_{20}^{30}=60	\\
  {M_0\over M}&={4750\over3}{1\over60}={475\over18}\approx 26.39.
\end{align*}
\end{solution}

\begin{example}{Centroid of a Flat Plate}{CentroidFlatPlate}
Suppose a flat plate of uniform density has the shape
contained by $\ds y=x^2$, $y=1$, and $x=0$, in the first quadrant. Find
the center of mass. (Since the density is constant, the center of mass
depends only on the shape of the plate, not the density, or in other
words, this is a purely geometric quantity. In such a case the center
of mass is called the \dfont{centroid}.)
\end{example}
\begin{solution}
This is a two dimensional problem, but it can be solved as if it were
two one dimensional problems: we need to find the $x$ and $y$
coordinates of the center of mass, $\ds \bar x$ and $\ds \bar y$, and
fortunately we can do these independently. Imagine looking at the
plate edge on, from below the $x$-axis. The plate will appear to be a
beam, and the mass of a short section of the ``beam'', say between
$\ds x_i$ and $\ds x_{i+1}$, is the mass of a strip of the plate
between $\ds x_i$ and $\ds x_{i+1}$. See Figure~\ref{fig:twodcenterofmass}
showing the plate from above and as it appears edge on.

\begin{figure}[H]
\centerline{
\hbox to \hsize{\hfill
\begin{tikzpicture}[baseline=0,x=3cm,y=3cm]
\fill[fill=red!10] (0.5,0.25) rectangle (0.6,1);
\draw (0.5,0.25) rectangle (0.6,1);
\draw (0,0) -- (1.1,0) ;
\draw (0,0) -- (0,1.1) ;
\foreach \x in {0,1} \draw (\x,0) -- (\x,-2pt) node[anchor=north] {$\x$};
\foreach \y in {0,1} \draw (0,\y) -- (-2pt,\y) node[anchor=east]{$\y$};
\draw (0.5,0) -- (0.5,-2pt) node [anchor=north] {$x_i$};
\node [anchor=south east] at (0.375,0.6) {$(\bar x,\bar y)$};
\node at (0.375,0.6) {$\bullet$};
\draw[color=black] (0,0) parabola (1,1);
\draw (0,1) -- (1,1);
\end{tikzpicture}
\hskip2cm
\begin{tikzpicture}[domain=-1.2:1.2,baseline=0,x=3cm,y=3cm]
\fill[fill=red!10] (0.5,0) rectangle (0.6,0.1);
\draw (0,0) -- (1.1,0) ;
\foreach \x in {0,1} \draw (\x,0) -- (\x,-2pt) node[anchor=north] {$\x$};
\draw (0.5,0) -- (0.5,-2pt) node[anchor=north] {$x_i$};
\node[anchor=south] at (0.55,0.1) {$m_i$};
\draw (0,0) rectangle (1,0.1);
\draw (0.5,0) rectangle (0.6,0.1);
\end{tikzpicture}\hfill}}
\caption{Center of mass for a two dimensional plate.}
\label{fig:twodcenterofmass}
\end{figure}

Since the plate has uniform density we may as well assume that
$\sigma=1$. Then the mass of the plate between $\ds x_i$ and $\ds
x_{i+1}$ is approximately $\ds m_i=\sigma(1-x_i^2)\Delta
x=(1-x_i^2)\Delta x$. Now we can compute the moment around the
$y$-axis:
\[M_y=\int_0^1 x(1-x^2)\,dx={1\over4}\]
and the total mass
\[M=\int_0^1 (1-x^2)\,dx={2\over3}\]
and finally
\[\bar x = {1\over4}{3\over2}={3\over8}.\]
Next we do the same thing to find $\ds \bar y$. The mass of the plate
between $\ds y_i$ and $\ds y_{i+1}$ is approximately $\ds
n_i=\sqrt{y}\Delta y$, so
\[M_x=\int_0^1 y\sqrt{y}\,dy={2\over5}\]
and
\[\bar y={2\over5}{3\over2}={3\over5},\]
since the total mass $M$ is the same. The center of mass is shown in
Figure~\ref{fig:twodcenterofmass}.
\end{solution}

\begin{example}{Center of Mass under Cosine}{centerofmassundercos}
Find the center of mass of a thin, uniform plate whose shape
is the region between $y=\cos x$ and the $x$-axis between $x=-\pi/2$
and $x=\pi/2$.
\end{example}
\begin{solution}
It is clear that $\ds \bar x=0$, but for practice let's
compute it anyway. We will need the total mass, so we compute it
first:
\[M=\int_{-\pi/2}^{\pi/2} \cos x\,dx=\sin x\Big|_{-\pi/2}^{\pi/2}=2.\]
The moment around the $y$-axis is
\[M_y=\int_{-\pi/2}^{\pi/2} x\cos x\,dx=\cos x+x\sin x\Big|_{-\pi/2}^{\pi/2}=0\]
and the moment around the $x$-axis is
\[M_x=\int_{0}^{1} y\cdot2\arccos y\,dy=\left.y^2\arccos y-{y\sqrt{1-y^2}\over2}+{\arcsin y\over 2}\right|_{0}^{1}={\pi\over4}.\]
Thus
\[\bar x={0\over2},\quad \bar y={\pi\over8}\approx 0.393.\]
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:centerofmass}}

\begin{enumialphparenastyle}

\begin{ex}
A beam 10 meters long has density $\ds \sigma(x)=x^2$ at
distance $x$ from the left end of the beam. Find the center of mass
$\ds \bar x$.
\begin{sol}
$15/2$
\end{sol}
\end{ex}

\begin{ex}
A beam 10 meters long has density $\sigma(x)=\sin(\pi x/10)$ at
distance $x$ from the left end of the beam. Find the center of mass
$\ds \bar x$.
\begin{sol}
$5$
\end{sol}
\end{ex}

\begin{ex}
A beam 4 meters long has density $\ds \sigma(x)=x^3$ at
distance $x$ from the left end of the beam. Find the center of mass
$\ds \bar x$.
\begin{sol}
$16/5$
\end{sol}
\end{ex}

\begin{ex}
Verify that $\ds\int 2x\arccos x\,dx=
x^2\arccos x-{x\sqrt{1-x^2}\over2}+{\arcsin x\over 2}+C$.
\end{ex}

\begin{ex}
A thin plate lies in the region between $\ds y=x^2$ and the $x$-axis
between $x=1$ and $x=2$. Find the centroid.
\begin{sol}
$\ds \bar x=45/28$, $\ds \bar y = 93/70$
\end{sol}
\end{ex}

\begin{ex}
A thin plate fills the upper half of the unit circle
$\ds x^2+y^2=1$. Find the centroid.
\begin{sol}
$\ds \bar x=0$, $\ds \bar y=4/(3\pi)$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region contained by $y=x$ and
$\ds y=x^2$. Find the centroid.
\begin{sol}
$\ds \bar x=1/2$, $\ds \bar y=2/5$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region contained by $\ds y=4-x^2$ and
the $x$-axis. Find the centroid.
\begin{sol}
$\ds \bar x=0$, $\ds \bar y=8/5$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region contained by $\ds y=x^{1/3}$ and
the $x$-axis between $x=0$ and $x=1$. Find the centroid.
\begin{sol}
$\ds \bar x=4/7$, $\ds \bar y=2/5$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region contained by
$\ds \sqrt{x}+\sqrt{y}=1$ and the axes in the first quadrant.
Find the centroid.
\begin{sol}
$\bar x=\bar y=1/5$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region between
the circle $\ds x^2+y^2=4$ and the circle $\ds x^2+y^2=1$, above the $x$-axis.
Find the centroid.
\begin{sol}
$\ds \bar x=0$, $\ds \bar y=28/(9\pi)$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region between
the circle $\ds x^2+y^2=4$ and the circle $\ds x^2+y^2=1$ in the first quadrant.
Find the centroid.
\begin{sol}
$\ds \bar x=\bar y=28/(9\pi)$
\end{sol}
\end{ex}

\begin{ex}
A thin plate lies in the region between
the circle $\ds x^2+y^2=25$ and the circle $\ds x^2+y^2=16$
above the $x$-axis.
Find the centroid.
\begin{sol}
$\ds \bar x=0$, $\ds\bar y=244/(27\pi)\approx 2.88$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\documentclass[11pt,a4paper]{article}

\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{bm}
\usepackage{calligra}
\usepackage{wrapfig}
\usepackage{comment}
\usepackage{subcaption}
\usepackage{color}

%\DeclareMathAlphabet{\mathcalligra}{T1}{calligra}{m}{n}
%\DeclareFontShape{T1}{calligra}{m}{n}{<->s*[2.2]callig15}{}

\setlength{\parindent}{0pt}

\DeclarePairedDelimiter{\avg}{\langle}{\rangle}

\title{The Lau/Ostoji\'c Ising model simulation}
\date{}
\author{Bernie Lau and Oliver Ostoji\'c}

\begin{document}

\maketitle

\section{Introduction}

This project is in essence an investigation of the Ising Hamiltonian with no external field;\footnote{As always,
$\avg{i,j}$ indicates a sum over nearest neighbors}

\begin{equation}\label{eq:Hamiltonian}
    H = -J \sum_{\avg*{i,j}} s_i s_j
\end{equation}

As this apparently simple Hamiltonian is known to result in very interesting and rich dynamical behavior on a
 macroscopic scale, we endeavor to find, through computational methods, some interesting thermodynamical quantities
 and qualitative behaviors. Of primary interest is the search for signs of a phase transistion, which the two and three
 dimensional Ising model is known to exhibit.

 In order to be less vague, we shall from now on talk about the Ising Hamiltonian
 in the context of ferromagnetism, where the quantites $s_i$ are associated with electron spins which can point in one of two
 directions. We will also set the coupling energy and the Boltzmann canstant equal to one,
 $J = k_B = 1$, for all simulations. As a consequence, the inverse temperature $\beta = 1/k_BT$ is measured in unphysical
 units. This is justified by the
 fact that  we are primarily interested in the qualitative behavior of the system and all actual numbers in the following
 simulations can still be compared to theoretical calculations and similar simulations.

 Phase transitions are typically characterized by rapid changes in macroscopic quantities near the transition temperature, which
 is called the critical temperature, $T_c$. The macroscopic quantities we will be investigating are the net magnetization,
 $M = \sum_i s_i$, the magnetic susceptibility (a measure for how the magnetization reacts to external magnetic fields) and
 the correlation length (a measure for the length scale at which the spins influence each other). These quantities and their
 connection to phase transitions will be made more concrete in the relevant sections.

 The actual methods we employ are two Monte Carlo type algorithms, with most simulations carried out using a
 Metropolis algorithm and a couple of measurements using a Wolff algorithm. It seems appropriate therefore, to summarize the
 principles of those algorithms here, in a minimalistic fashion.

\subsection{Metropolis algorithm}
As the focus of this assignment is on the \textit{implementation} of the Metropolis algorithm rather than its theoretical
 aspects, we have chosen to omit the technical details and just outline the procedure used in its implementation here.

 Given a
 lattice with a spin variable $s_i \in \{-1,1\}$ on each lattice point $i$, we define a "state" to be a particular
 configuration of all the ${s_i}$. The energy $E_\mu$ of the system in a state $\mu$ is determined by the Hamiltonian
 (Equation \ref{eq:Hamiltonian}). Time evolution of the system is achieved by randomly choosing a single site $j$ and determining
 the energy $E_\nu$ of the system if the spin on that site were flipped, $s_j \rightarrow -s_j$. Weather or not we actually flip
 the spin is determined by the the number $A(\mu \rightarrow \nu)$ given by;

\begin{equation}\label{eq:A-ratio}
    A(\mu \rightarrow \nu) = \begin{cases}
        e^{-\beta (E_\nu - E_\mu)} & \mbox{if} \,\,  E_\nu - E_\mu > 0 \\
        1 & \mbox{else}
    \end{cases}
\end{equation}

Once the spin flip is accepted or rejected, the process is repeated and each iteration is called a "Monte Carlo step". It is actually
 this particular definiton of the so called
 "acceptance ratio", $A(\mu \rightarrow \nu)$ that makes the algorihm a Metropolis algorithm,
 the rest of the procedure is common to all single spin-flip dynamics Monte Carlo algorithms. The conditions
 of detailed balance and ergodicity, which ensure that the
 equilibrium state of the system will behave according to Boltzmann statistics and that each state can be reached from each other state, are
 statisfied by this acceptance ratio, as must be the case with all Monte Carlo algorithms.

\subsection{Wolff algorithm}
An example of a cluster-flipping algorithm, the Wolff algorithm works by iteratively picking out
 an entire cluster of spins pointing in the same direction and then flipping them all at the same time. The actual physics content of this
 procedure resides entirely in the generation of the cluster, which is done based on a temperature dependent probability chosen such that the
 conditions of detailed balance and ergodicity are again satisfied.
 More specifically, the procedure which is iterated is as follows;
 \begin{enumerate}
 \item Pick a spin at random and look at all its neighbors.
 \item If a neighbor is pointing in the same direction as the current one, add it to the cluster with a probability $P_{add}$ given by

\begin{equation}\label{eq:A-ratio}
    P_{add} = 1 - e^{-2\beta J}
\end{equation}

 \item For each spin that was added, look at all \textit{its} neighbors and add them with the same probability. Do this until no spins are left
 in the cluster whose neighbors have not been considered for addition.
 \item Flip the cluster

 \end{enumerate}

The Wolff algorithm, although less intuitive than Metropolis has the advantage of being much faster near the critical temperature, thereby
 allowing us to obtain more accurate results in the region of interest.

\subsection{Technicalities}
In what follows, we will set the coupling energy and the Boltzmann canstant equal to one,
 $J = k_B = 1$, for all simulations. As a consequence, the inverse temperature $\beta = 1/k_BT$ is measured in unphysical
 units. This is justified by the
 fact that  we are primarily interested in the qualitative behavior of the system and all actual numbers in the following
 simulations can still be compared to theoretical calculations and similar simulations.

\section{Thermalization and autocorrelation for the 40x40 lattice}

In this section we present some results of a simulation of a 40x40 two dimensional Ising model, using a Metropolis algorithm.
 We present plots of energy and magnetization as a function of time and report the thermalization times found for each plot.
 We also plot autocorrelation functions for both enery and magnetization
 for five values of the inverse temperature $\beta$ between $\beta = 0.3$ and $\beta = 0.5$.
 Both high temperature (fully disordered) and temperature (all spins aligned) initial conditions are simulated.
 Time is measured in Monte-Carlo steps per spin for all simulations. Each simlutation "takes" $32\cdot 10^6$
 Monte-Carlo steps.

\subsection{Energy functions}
To find the thermalization times, we plot the energy as a function of time (measured in Monte-Carlo steps per site)
 and fit a general exponential curve and read off the characteristic time $\tau$:

\begin{equation}\label{eq:exp_decay}
    f(t)=Ae^{t/\tau} + C
\end{equation}

After a single characteristic time $\tau$, the value of the energy will be at about $37\%$ of its initial value and
 the system will not yet be thermalized. We therefore define the
 thermalization time as five times the characteristic time, just to be safe:

\begin{equation*}
    t_{therm} = 5\tau
\end{equation*}

Figure \ref{fig:Evt} shows the results of this procedure for the values $\beta = 0.3$; $\beta = 0.35$;
 $\beta = 0.4$; $\beta = 0.45$ and $\beta = 0.5$. Included are the acquired values of $t_{therm}$. Both
 high temperature and low temperature initial conditions are covered.
\\
\\
{\color{red}REVIZE SECTION AFTER ADDING PLOTS AND WRITING INTRO}

\begin{figure}[h!]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=\linewidth]{metro_40x40_iter_32M_unaligned_E_vs_time.png}
  \caption{}
  \label{fig:Evt_highT}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=\linewidth]{metro_40x40_iter_32M_aligned_E_vs_time.png}
  \caption{}
  \label{fig:Evt_lowT}
\end{subfigure}
\caption{Energy vs ierations per spin for the values $\beta = 0.3$; $\beta = 0.35$;
         $\beta = 0.4$; $\beta = 0.45$ and $\beta = 0.5$. a) High temperature starting condition.
         {\color{red}To add: the (calculated) values of the thermalization times per $\beta$}.
         b) Low temperature starting condition.
         {\color{red}To add: the (calculated) values of the thermalization times per $\beta$)}}
\label{fig:Evt}
\end{figure}

\subsection{Magnetization functions}
As we are still looking for thermalization times, just now for magnetization, we use the same procedure as in
 the previous section.

Figure \ref{fig:Mvt} shows plots of magnetization versus time for the values $\beta = 0.3$; $\beta = 0.35$;
 $\beta = 0.4$; $\beta = 0.45$ and $\beta = 0.5$ for high and low temperature initial conditions.

\begin{figure}[h!]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=\linewidth]{metro_40x40_iter_32M_unaligned_net_M_vs_time.png}
  \caption{}
  \label{fig:Mvt_highT}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=\linewidth]{metro_40x40_iter_32M_aligned_net_M_vs_time.png}
  \caption{}
  \label{fig:Mvt_lowT}
\end{subfigure}
\caption{Magnetization vs ierations per spin for the values $\beta = 0.3$; $\beta = 0.35$;
         $\beta = 0.4$; $\beta = 0.45$ and $\beta = 0.5$. a) High temperature starting condition
         {\color{red}To add: the (calculated) values of the thermalization times per $\beta$}.
         b) Low temperature starting condition.
         {\color{red}To add: the (calculated) values of the thermalization times per $\beta$}.}
\label{fig:Mvt}
\end{figure}

\subsection{Autocorrelation functions}
In addition to the previous results, the autocorrelation times for both energy and magnetization can be
 extracted from the simulations. We implement the formulas

\begin{equation*}
    c_e(\Delta t) = \avg{(E(t+\Delta t) - E_{avg})\cdot (E(t)-E_{avg})}_t
\end{equation*}

and

\begin{equation*}
    c_m(\Delta t) = \avg{(M(t+\Delta t) - M_{avg})\cdot (M(t)-M_{avg})}_t
\end{equation*}

where, $E_{avg}$ and $M_{avg}$ indicate averaging over the \textit{last} million values of the energy and magnetization
 respectively. This in contrast to the averaging indicated by the $\avg{}_t$, which means we average over the number of
 iterations. We again fit the general exponential decay given by equation \ref{eq:exp_decay}, and report the characteristic
 time, which is by definition the autocorrelation time. The results are given below (Figure \ref{fig:autocorr})
 for each simulation, i.e for the same values of $\beta$ as listed above.

\begin{figure}[h!]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.4\linewidth]{Boltzmann.jpg}
  \caption{}
  \label{fig:autocorr_energy}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.4\linewidth]{Boltzmann.jpg}
  \caption{}
  \label{fig:autocorr_mag}
\end{subfigure}
\caption{Autocorrelation times vs $\Delta t$ for the values $\beta = 0.3$; $\beta = 0.35$;
         $\beta = 0.4$; $\beta = 0.45$ and $\beta = 0.5$. a) Energy autocorrelation function.
         b) Magnetization autocorrelation function.}
\label{fig:autocorr}
\end{figure}

\section{Magnetization and magnetic suscepibility with metropolis}

\section{The Wolff}

\section{The 3D Ising model}
\end{document}


% A simple template for LaTeX documents
%
% To produce pdf run:
%   $ pdflatex paper.tex
%

\documentclass[10pt, twocolumn]{article}
%\documentclass[12pt]{article}

% Begin paragraphs with new line
\usepackage{parskip}

% Change margin size
\usepackage[margin=0.5in]{geometry}

% Graphics Example:  (PDF's make for good plots)
\usepackage{graphicx}
% \centerline{\includegraphics{figure.pdf}}

% Allows hyperlinks
\usepackage{hyperref}

% Blocks of code
\usepackage{listings}
\lstset{basicstyle=\ttfamily, title=\lstname}
% Insert code like this. replace `plot.R` with file name.
% \lstinputlisting{plot.R}

% Monospaced fonts
%\usepackage{inconsolata}
% GNU \texttt{make} is a nice tool.

% Supports proof environment
\usepackage{amsthm}

% Allows writing \implies and align*
\usepackage{amsmath}

% Allows mathbb{R}
\usepackage{amsfonts}

% Numbers in scientific notation
%\usepackage{siunitx}

% Use tables generated by pandas
\usepackage{booktabs}

% norm and infinity norm
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inorm}[1]{\left\lVert#1\right\rVert_\infty}

% Statistics essentials
\newcommand{\iid}{\text{ iid }}
\newcommand{\Expect}{\operatorname{E}}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\Cov}{\operatorname{Cov}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\textbf{Chi square}
Special case: $X \sim \chi^2_n \equiv \text{Gamma}(\frac{n}{2}, \frac{1}{2})$

$
f(x) \propto x^{\frac{n}{2} - 1} e^{\frac{-x}{2}},
\quad x > 0
\qquad \Expect X = n,
\quad \Var X = 2n
$

Let $Z_i$ be iid $N(0, 1)$.
$\sum_{i=1}^n Z_i^2 \sim \chi^2_n$

Noncentral $\chi^2$. Let $Y \sim N(\mu, I)$ be an $n$ vector. Then
\[
    \norm{Y}^2 \sim \chi^2_n(\norm{\mu}^2)
\]
\textbf{F} if num. and den. independent then
\[
    F(m, n) \equiv \frac{\frac{\chi^2_m}{m}}
        {\frac{\chi^2_n}{n}}
\]
\textbf{T} if num. and den. independent then
\[
    t(n) = \frac{N(0, 1)}
    {\sqrt{\frac{\chi^2_n}{n}}}
\]
Conditional pdf:
\[
    f_{X|Y}(x | y) \equiv \frac{f_{X, Y}(x, y)}{f_Y(y)}
\]
Iterated expectation:
\[
    E(Y) = E(E(Y | X))
\]
Conditional variance formula:
\[
    \Var(Y) = \Var(E(Y | X)) + E(\Var(Y | X))
\]
Singular Value Decompostion (SVD) Any matrix $X$ can be written
\[
    X = UDV^T
\]
with $U, V$ orthogonal, and $D$ diagonal.

Moore Penrose Psuedoinverse $A^+$ exists uniquely for every matrix $A$.
Properties: $AA^+A = A, \quad A^+AA^+ = A^+$ and $AA^+, A^+A$ are symmetric.

Projection matrix $P$ are symmetric and idempotent. They have eigenvalues
either 0 or 1.
\[
    P = P^T \qquad P^2 = P
\]
Covariance of linear transformations
\[
    Cov(Ay, Bx) = A Cov(y, x) B^T
\]
Invert $2 \times 2$ matrix:
$
    A =
    [\begin{smallmatrix}
        a & b \\
        c & d \\
    \end{smallmatrix}]
$
\[
    A^{-1} =
    \frac{1}{\det (A)}
    \begin{bmatrix}
        d & -b \\
        -c & a \\
    \end{bmatrix}
\]
Sum identities:
\[
    \sum_{k=0}^{\infty} p^k = \frac{p}{1 - p} \qquad
    \sum_{k=0}^{\infty} k p^k = \frac{p}{(1 - p)^2} \qquad |p| < 1
\]
Integration by parts:
\[
    \int uv' = uv - \int u'v
\]
Matrix / Vector differentiation

$\frac{\partial A^T \beta}{\partial \beta} = A$,
$\frac{\partial \beta^T A \beta}{\partial \beta} = (A + A^t) \beta =
2A\beta$ for $A$ symmetric.

$\frac{\partial}{\partial \theta_i} \log (|A|) =
tr( A^{-1} \frac{\partial A}{\partial \theta_i}$)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

General linear tests under $H_0$:
\[
    \frac{\frac{SSE_r - SSE_f}{p - r}}
         {\frac{SSE_f}{n - p}}
         \sim F_{p-r, n-p}
\]
Three principles of experimental design: 1) Replication 2) Randomization 3)
Blocking

One way ANOVA with $n$ total observations, $K$ groups:

{
\centering
\begin{tabular}{lll}
    SS   &  & DF     \\
    SSTR & $\sum_{j=1}^K n_j (\bar{y_{j \cdot}} - \bar{y_{\cdot \cdot}})^2$  & K - 1 \\
    SSE  & $\sum_{i=1}^n (y_{ij} - \bar{y_{j \cdot}})^2$  & n - K \\
    SSTO & $\sum_{i=1}^n (y_{ij} - \bar{y_{\cdot \cdot}})^2$  & n - 1
\end{tabular}
}

\subsection*{Multivariate Normal}

$X \sim N(\mu, \Sigma)$, $\Sigma$ positive definite
\[
    f(x) = \frac{\exp\{ - \frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu) \}}
        {(2\pi)^{\frac{k}{2}} \sqrt{\det(\Sigma)}}
        = \frac{1}{\sqrt{2 \pi} \sigma} e^{-\frac{(x - \mu)^2}{2
        \sigma^2}}
\]
mgf: $M_X (t) = \exp (\mu' t + \frac{1}{2} t' \Sigma t)$

log likelihood for $k$ vector $x \sim N(\mu, \Sigma)$
\[
    l_x = -\frac{k}{2} \log 2 \pi - \frac{1}{2}
    \{ \log \det \Sigma + (x - \mu)^T \Sigma^{-1} (x - \mu) \}
\]
Stein's formula: $X \sim N(\mu, \sigma)$
\[
    \Expect (g(X) (X - \mu)) = \sigma^2 \Expect(g'(X))
\]
assuming these expectations are finite.

$X \sim N(\mu, \Sigma)$, $A$ an $m \times n$ matrix,
then
\[
    AX \sim N(A \mu, A \Sigma A^t)
\]
For $\Sigma$ full rank it's possible to transform between $Z \sim
N(0, I)$ and $X$:
\[
    X = \Sigma^{1/2} Z + \mu \qquad Z = \Sigma^{-1/2} (X - \mu)
\]
In block matrix form:
\[
    X =
    \begin{bmatrix}
        X_1 \\
        X_2 \\
    \end{bmatrix}
    \sim N \left(
    \begin{bmatrix}
        \mu_1 \\
        \mu_2 \\
    \end{bmatrix}
    ,
    \begin{bmatrix}
        \Sigma_{11} & \Sigma_{12} \\
        \Sigma_{21} & \Sigma_{22} \\
    \end{bmatrix}
\right)
\]
Assuming $\Sigma_{11}$ is positive definite then the conditional
distribution
\[
    X_2 | X_1 \sim N(\mu_2 + \Sigma_{21} \Sigma_{11}^{-1} (X_1 - \mu_1),
    \Sigma_{22} - \Sigma_{21} \Sigma_{11}^{-1} \Sigma_{12})
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}


\SecDef{zerosum}{Degree-\texorpdfstring{$d$}{d} Zero-Sum Sets and Sum-Invariant Matrices}

A natural question to ask is which other linear mappings have a similar property as given in \EqRef{invariant_preserving_linear}. To answer this question, we study \emph{degree-$d$ zero-sum sets} as a generalization of the above problem.

\begin{definition}[Degree-$d$ Zero-Sum Set]
Let $S \subseteq \F_2^n$ and let $d \in \mathbb{N}$. We call $S$ \emph{degree-$d$ zero-sum} if, for all $f \in \BF{n}{d}$,
\eql{zerosum}{
\bigoplus_{s \in S} f(s) = 0.
}
We define $\rank(S)$ to be the maximum number of linearly independent elements in $S$ and denote by $\ZS{n}{m}{d}$ the set of degree-$d$ zero-sum sets with $m$ elements and rank $n$.
\end{definition}

We first show the following equivalent characterizations of degree-$d$ zero-sum sets.
\begin{proposition}
\PropLabel{zero_sum}
Let $S = \{s_1,\dots,s_{k}\} \subseteq \F_2^n$ and let $d \in \ZZplus$. Let $\matzs{S} \in \F_2^{n \times k}$ be any matrix (up to a permutation of the columns) the columns of which correspond to the elements of $S$, i.e., $$
\matzs{S} = \rowthree{s_1^\top}{\dots}{s_{k}^\top}.
$$
Then the following statements are equivalent:
\enumroman
\begin{enumerate}
    \item $S$ is a degree-$d$ zero-sum set.
    \item $k$ is even and, for any choice of $d$ (not necessarily distinct) rows $r_1,\dots,r_d$ of $\matzs{S}$, it is $\inprod{ r_1,\dots,r_d} = 0$.
    \item in every $d \times k$ submatrix of $\matzs{S}$, each column occurs an even number of times.
    \item $\deg(\Ind_S) \leq n-d-1$.
    \item for all $t \geq 1$ and all $f \in \BF{t}{d}$, $\forall X \in \F_2^{t \times n}\colon \bigoplus_{s \in S} f(sX^\top) = 0$.
\end{enumerate}
In particular, the degree-$d$ zero-sum sets in $\F_2^n$ are exactly the supports of the $n$-bit Boolean functions of degree at most $n-d-1$. Therefore, any non-empty degree-$d$ zero-sum set must contain at least $2^{d+1}$ elements.
\end{proposition}
\begin{proof}
To prove $(i) \Rightarrow (ii)$, let
$$
\matzs{S} = \colthree{r_1}{\vdots}{r_n}
$$
with $r_i \in \F_2^{k}$. Let $l_1,\dots,l_d$ be $d$ (not necessarily distinct) row indices and consider the monomial function $f \in \BF{n}{d}, \ x \mapsto \prod_{i=1}^{d}x_{l_i}$, which has degree $d$. From \EqRef{zerosum}, it must be
$$
0
= \bigoplus_{s \in S} f(s)
= \bigoplus_{s \in S} \prod_{i=1}^{d} s_{l_i}
= \inprod{ r_{l_1},\dots,r_{l_d}}.
$$
Clearly, $k$ must be even because $\bigoplus_{s \in S} 1 = 0$.

$(ii) \Rightarrow (iii)$: We first see that any $1 \times k$ submatrix of $\matzs{S}$ contains each element in $\F_2$ an even number of times. Indeed, let $r$ be any row in $\matzs{S}$. From $(ii)$ we know that $\wt(r) \mod 2 = \langle r \rangle = 0$ and thus $r$ contains an even number of $1$'s. Because $k$ is even, it must also contain an even number of $0$'s. We now use induction on the number of rows. Let $d' < d$ such that $(ii) \Rightarrow (iii)$ holds for $d'$. Let us choose an arbitrary $(d'+1) \times k$ submatrix $H = [m_{i,j}]_{1\leq i \leq d'+1, 1 \leq j \leq k}$ of $\matzs{S}$. We define $H^{(0)} \coloneqq [m^{(0)}_{i,j}]$ to be the submatrix of $H$ that is obtained by selecting exactly the columns $m_{\star,j}$ of $H$ for which $m_{d'+1,j} = 0$. Similarly, let $H^{(1)} \coloneqq [m^{(1)}_{i,j}]$ be the submatrix of $H$ that is obtained by selecting exactly the columns $m_{\star,j}$ of $H$ for which $m_{d'+1,j} = 1$. We have already seen from the initial step that both $H^{(0)}$ and $H^{(1)}$ must contain an even number of columns (otherwise the row $m_{d'+1,\star}$ would have an odd weight). From $(ii)$, we know that
\begin{align*}
    0 &= \inprod{ m_{1,\star},\dots,m_{d',\star},m_{d'+1,\star} } =
    \inprod{ m^{(0)}_{1,\star},\dots,m^{(0)}_{d'+1,\star} } + \inprod{ m^{(1)}_{1,\star},\dots,m^{(1)}_{d'+1,\star} } \\
    &= \inprod{ m^{(1)}_{1,\star},\dots,m^{(1)}_{d',\star} } = \inprod{ m^{(0)}_{1,\star},\dots,m^{(0)}_{d',\star} }.
\end{align*}
Because of the induction hypothesis, $H^{(0)}$ and $H^{(1)}$ contain each column an even number of times and therefore, every column of $H$ occurs an even number of times.

$(iii) \Rightarrow (iv)$: Let $u \in \F_2^n$ with $\wt(u) \geq n-d$. Because of $(iii)$,
$$
|\pset{s \in S \mid s \preceq u}|
$$
is even (because $d$ zeroes in positions $i$ where $u_i = 0$ occur an even number of times among elements of $S$). It follows that
$$
|\pset{s \in S \mid s \preceq u}| \mod 2 = \bigoplus_{s \preceq u}\Ind_S(s) = 0
$$
and thus, the monomial $x^u$ doesn't occur in the ANF of $\Ind_S$. Since this holds for all $u$ with $\wt(u) \geq n-d$, the algebraic degree of $\Ind_S$ is at most $n-d-1$.

$(iv) \Rightarrow (v)$: Let $f \in \BF{t}{d}$ be an arbitrary function of degree at most $d$. Observe that
\eql{indicator_sum}{
    \forall X \in \field{t \times n} \quad \bigoplus_{s \in \field{n}} \Ind_{S} \cdot f(sX^{\top}) = 0,
}
because $\deg{\Ind_{S} \cdot (f \circ X)} \le \deg{\Ind_{S}} + \deg{f}\le n - 1$. Here, $f \circ X$ denotes the $n$-bit Boolean function $s \mapsto f(sX^\top)$.
\EqRef{indicator_sum} can equivalently be written as
\eq{
    \forall X \in \field{t \times n} \quad \bigoplus_{s \in S}  f(sX^\top) = 0,
}
which proves $(v)$. The implication $(v) \Rightarrow (i)$ follows by letting $t=n$ and $X = \idmat{n}$.

To see that any non-empty degree-$d$ zero-sum set contains at least $2^{d+1}$ elements, we use the fact that any non-zero Boolean function of degree at most $n-d-1$ has a weight at least $2^{n-(n-d-1)} = 2^{d+1}$.
\end{proof}

It is worth remarking that the property of being degree-$d$ zero-sum  is invariant under the application of an injective linear mapping. Indeed, if $\varphi\colon \Span(S) \rightarrow \F_2^{n'}$ is an injective linear function on the subspace $\Span(S)$ of dimension $\rank(S)$, then $|\varphi(S)|=|S|$ and if $S$ is degree-$d$ zero-sum, so is $\varphi(S)$. Further, $\rank(\varphi(S)) = \rank(S)$. Therefore, without loss of generality, we can represent a zero-sum set $S \in \ZS{n}{m}{d}$ as a subset of $\F_2^n$ and given by the columns of an $n \times m$ matrix $\matzs{S}$ of the form
\eql{systematic_form}{
\matzs{S} = \rowtwo{\idmat{n}}{L}
}
for an $L \in \F_2^{n \times (m-n)}$. We say that a zero-sum set (resp. a matrix $\matzs{S}$) given in the representation of \EqRef{systematic_form} is in \emph{systematic form}. We are in particular interested in the properties of such matrices $L$ that define zero-sum sets in $\ZS{n}{m}{d}$ in the above way. For instance, such an $L$ can only exist if $m$ is even. We generalize this by introducing the notion of a \emph{degree-$d$ sum-invariant matrix} as follows.

\begin{definition}[Degree-$d$ Sum-Invariant Matrix]
A matrix $L \in \F_2^{n \times m}$ is called \emph{degree-$d$ sum-invariant} if, for all $t \geq 1$ and all $f \in \BF{t}{d}$,
\eql{linear_layer}{
\forall X \in \F_2^{t \times n} \colon
\bigoplus_{i=1}^n f\big( (X^\top)_i \big) =
\bigoplus_{j=1}^m f\big( ((XL)^\top)_j \big) + \varepsilon_{m+n}f(0),
}
where $\varepsilon_{m+n} = (m+n) \mod 2$.
\end{definition}

\begin{proposition}
\PropLabel{inner_product}
Let $L \in \F_2^{n \times m}$ be a linear mapping and let $d \in \mathbb{N}$.
Then the following statements are equivalent:
\enumroman
\begin{enumerate}
    \item $L$ is degree-$d$ sum-invariant.

    \item The columns of the matrix $\matl{L}$ occurring with odd multiplicity define a degree-$d$ zero-sum set, where
    \begin{equation}
    \begin{cases}
    \arraycolsep=4pt\def\arraystretch{1}
    \matl{L} \coloneqq \rowtwo{\idmat{n}}{L} \in \F_2^{n\times(m+n)},
    & \text{~if~} m+n \text{~is even}\;; \\
    \arraycolsep=4pt\def\arraystretch{1}
    \matl{L} \coloneqq \rowthree{\idmat{n}}{L}{0} \in \F_2^{n\times(m+n+1)},
    & \text{~if~} m+n \text{~is odd}\;.
    \end{cases}
    \end{equation}

    \item For all $x_1,\dots x_d \in \F_2^{n}$ it is $\inprod{ x_1,\dots,x_d} = \inprod{x_1 L,\dots,x_d L}$.
\end{enumerate}
Moreover, if $L$ fulfills $(i)$ and if $d \geq 2$, then $n \le m$, $LL^{\top} = I_{n}$ and $L$ must have full rank $n$.
\end{proposition}
\begin{proof}

We first prove $(i) \Rightarrow (ii)$.
If $m+n$ is even, then \EqRef{linear_layer} is equivalent to
\eql{linear_layer_rewrite}{
\forall X \in \F_2^{t \times n} \colon
\bigoplus_{i=1}^n f\big(e_iX^\top\big) +
\bigoplus_{j=1}^m f\big( (L^\top)_j X^\top \big) = 0,
}
where $e_i$ denotes the $i$-th unit vector. If there is a $j$ for which $(L^\top)_j$ is equal to a unit vector $e_k$, then $f((L^\top)_j X^\top) = f(e_k X^\top)$ and the two terms cancel in \EqRef{linear_layer_rewrite}. Similarly, if there exist two different $j_1,j_2$ such that $(L^\top)_{j_1} = (L^\top)_{j_2}$, then $f( (L^\top)_{j_1} X^\top)$ and  $f( (L^\top)_{j_2}X^\top)$ cancel out. This is another way of saying that the columns of the matrix $\matl{L} = \rowtwo{\idmat{n}}{L}$ occurring with odd multiplicity define a degree-$d$ zero-sum set.

If $m+n$ is odd, then $\varepsilon_{m+n} = 1$ and \EqRef{linear_layer} can be written as
\eq{
\forall X \in \F_2^{t \times n} \colon
\bigoplus_{i=1}^n f\big(e_i X^\top \big) +
\bigoplus_{j=1}^m f\big( (L^\top)_{j} X^{\top}\big) + f(0X^\top) = 0.
}
This is equivalent to say that the columns of the $n\times(m+n+1)$ matrix $\matl{L} = \rowthree{\idmat{n}}{L}{0}$ occurring with odd multiplicity define a degree-$d$ zero-sum set.

$(ii) \Rightarrow (iii)$. If the columns of $\matl{L}$ occurring with odd multiplicity define a degree-$d$ zero sum set, then, because of \PropRef{zero_sum}, any $d$ (not necessarily distinct) rows $\rowtwo{e_{l_1}}{L_{l_1}},\ldots,\rowtwo{e_{l_d}}{L_{l_d}}$ of $\matl{L}$ fulfill
\begin{align*}
    \inprod{ \rowtwo{e_{l_1}}{L_{l_1}}, \ldots, \rowtwo{e_{l_d}}{L_{l_d}} } = 0\;,
\end{align*}
which is equivalent to
$$
\inprod{e_{l_1},\dots,e_{l_d}} = \inprod{ e_{l_1}L,\dots,e_{l_d}L }.
$$
Because of the linearity of the inner product, i.e.,
$$
\inprod{x_1+x_1',x_2,\dots,x_d} = \inprod{x_1,x_2,\dots,x_d} +
                                  \inprod{x_1',x_2,\dots,x_d},
$$
the statement follows.

$(iii) \Rightarrow (i)$. If there are $f_1,f_2 \in \BF{t}{d}$ such that \EqRef{linear_layer} holds for both $f_1$ and $f_2$, then it clearly holds for $f_1+1$ and for $f_1+f_2$ as well. Therefore, without loss of generality, let $f \in \BF{t}{d}$ be a monomial function, i.e., $f(z) = \prod_{k=1}^{d}z_{l_k}$ for $1 \leq l_1 \leq \dots \leq l_{d} \leq t$. Let $X \in \F_2^{t \times n}$. Then,
\begin{align*}
    \bigoplus_{i=1}^n f((X^\top)_i) =
    \bigoplus_{i=1}^n\prod_{k=1}^d (X^\top)_{i,l_k} =
    \langle X_{l_1},\dots,X_{l_d} \rangle
\end{align*}
and
\begin{align*}
    \bigoplus_{j=1}^m f( ((XL)^\top)_j) + \varepsilon_{m+n}f(0) =
    \bigoplus_{j=1}^m\prod_{k=1}^d ((XL)^\top)_{j,l_k} = \langle X_{l_1}L,\dots,X_{l_d}L \rangle \;.
\end{align*}
It follows that if $L$ preserves all generalized inner products of $d$ elements, then $L$ is degree-$d$ sum-invariant.

If $L$ fulfills the equivalent statements $(i)$ - $(iii)$,  then, for all $x,y \in \F_2^{n}$, it is
\[ xy^\top = \inprod{ x,y} = \inprod{xL,yL} = xL(yL)^\top =  xLL^{\top}y\;.\]
It follows that $LL^\top$ must be the identity and thus, $L$ must have full rank $n$.
\end{proof}

This result shows a relation between degree-$d$ sum-invariant matrices and semi-orthogonal matrices. A matrix $L \in \F_2^{n \times m}$ with $n \leq m$ is called \emph{semi-orthogonal} if $LL^\top = \idmat{n}$. Indeed, we have shown that a matrix is degree-$2$ sum-invariant if and only if it is semi-orthogonal.\footnote{We only consider matrices with $n \leq m$. If $L \in \F_2^{n\times m}$ with $n > m$, $L$ would be defined to be semi-orthogonal if $L^\top L = I_m$. Then, $L$ is semi-orthogonal if and only if $L^\top$ is degree-$2$ sum-invariant.} Because of the above relation, the degree-$(d+1)$ sum-invariant matrices might also be called \emph{$d$-th order semi-orthogonal}.

The invertible semi-orthogonal matrices are exactly the \emph{orthogonal} matrices and the orthogonal matrices in dimension $n$ form a multiplicative group, called the \emph{orthogonal group}. With the above equivalences, we obtain an interesting characterization of the orthogonal groups over $\F_2$.

\begin{corollary}
A matrix $L \in \F_2^{n \times n}$ is orthogonal if and only if in each $2 \times 2n$ submatrix of $\rowtwo{\idmat{n}}{L}$, each column occurs an even number of times.
\end{corollary}

\subsection{Relation to Orthogonal Arrays}
\PropRef{zero_sum} points out a relation between degree-$d$ zero-sum sets and orthogonal arrays.

\begin{definition}[Orthogonal Array~\cite{hedayat1999orthogonal}]
An $m \times n$ matrix $M$ with entries from a finite set of cardinality $k$ is said to be an \emph{orthogonal
array with $k$ levels, strength $d$ and index $\lambda$}, denoted $OA(m,n,k,d)$, if every $m \times d$ submatrix of $M$ contains each $d$-tuple exactly $\lambda$ times as a row. Without loss of generality, we will assume that $M$ is a matrix with elements in $\mathbb{Z}_k$.
\end{definition}
% \lambda = m / s^d

For our purposes we are only interested in the  case of $k = 2$. We directly obtain the following.

\begin{corollary}
Let $S \subseteq \F_2^n$. If $\matzs{S}^{\top}$ is an $OA(|S|,n,2,d)$ such that $2^{d+1}$ divides $|S|$ (i.e., if the index $\lambda$ is even), then $S$ is a degree-$d$ zero-sum set.
\end{corollary}

As an example, for $d = 3$, there is a well-known construction of orthogonal arrays from Hadamard matrices (see~\cite[pp.\@ 145--148]{hedayat1999orthogonal}). A \emph{Hadamard matrix} of order $n$ is a matrix $H \in \mathbb{Z}^{n \times n}$ which can only take values in $\{-1,1\}$ and which fulfills $H^{\top}H = n \idmat{n}$. For a  matrix $M$ with elements in $\{-1,1\}$, we denote by $\widetilde{M}$ the $\F_2$ matrix obtained from $M$ by replacing $-1$ with $0$, i.e., we define $\widetilde{M}$ to be the result of $\frac{1}{2}(M+1)$, interpreted in $\F_2$.

If $H$ is a Hadamard matrix of order $8k$ for $k \in \ZZplus$, it is well known that
$$
\widetilde{\coltwo{H}{-H}}
$$
is an $OA(16k,8k,2,3)$ of even index (see~\cite[Theorem 4.16]{hedayat_hadamard}). Therefore, it defines a degree-$3$ zero-sum set $S \subseteq \F_2^{8k}$ with $16k$ elements. However, its rank can be at most $4k$ (see~\cite[Proposition 2]{hadamard_codes}) and we are interested in the zero-sum sets of full rank.


 \documentclass[../thesis.tex]{subfiles}
 \begin{document}

This appendix contains proofs for several theorems extending
the well-known Data Processing Inequality in information theory
to configurations of random variables beyond a triplet Markov Chain.
The motivation for these theorems is the desire to model information
flow through a world modeled as a Bayesian network, where information
flow is determine by causal flows and nomic associations.
Nomic association here is measured as mutual information between two variables.
The Chain rule for mutual information and the Markov properties of
a Bayesian network make it possible to prove several theorems that are
as far as we know new.

\section{Triplet Structures}

The Data Processing Inequality is a standard theorem in information theory.
It concerns the mutual information of three variables arranged in a
Markov Chain.

\begin{dfn}
  (\citet{cover2012elements})
  Random variables $X, Y, Z$ are said to \emph{form a Markov chain in that order}
  (denoted $X \rightarrow Y \rightarrow Z$) if the conditional distribution
  of $Z$ depends only on $Y$ and is conditionally independent of $X$.
  Specifically, $X,Y,Z$ form a Markov chain $X \rightarrow Y \rightarrow Z$
  if the joint probability mass function can be written as
  \begin{equation}
    p(x,y,z) = p(x)p(y \vert x)p(z \vert y)
  \end{equation}
\end{dfn}

\begin{thm}[Data Processing Inequality]
  Given a probability model defined by the following (Markov Chain):
  \begin{center}
    \begin{tikzcd}
      X \arrow[r] & Y \arrow[r] & Z \\
    \end{tikzcd}
  \end{center}
  where $X \independent Z \vert Y$, then it must be that $I(X;Y) \geq I(X;Z)$.
\end{thm}
\begin{proof}
  From \cite{cover2012elements}. By the Chain Rule, mutual information
  can be expanded in two different ways:
  \begin{equation}
    \begin{split}
    I(X;Y,Z) & = I(X;Z) + I(X;Y \vert Z) \\
    & = I(X;Y) + I(X;Z \vert Y)
    \end{split}
  \end{equation}
  Since X and Z are conditionally independent given Y, we have
  $I(X;Z \vert Y) = 0$. Since $I(X;Y \vert Z) \geq 0$, we have
  \begin{equation}
    I(X;Y) \geq I(X;Z)
  \end{equation}
  We have equality if and only if $I(X;Y \vert Z) = 0$
  (i.e. $X \rightarrow Z \rightarrow Y$ forms a Markov Chain).
  Similarly, one can prove that $I(Y;Z) \geq I(X;Z)$.
\end{proof}

Bayesian networks are a generalization of Markov chains.
A Bayesian network models the probability distribution
between many random variables as a directed acyclic
graph. The conditional probability distribution of
each variable is defined in terms of the graphical
parents $pa(\dot)$ of each variable, i.e. $P(X_i) = P(X_i \vert pa(X_u))$.
The joint distribution is

\begin{equation}
  P(X_1, X_2, ..., X_n) = \prod_{i=1}^{n}P(X_i \vert pa(X_i))
\end{equation}

We can now prove several theorems that are similar to the
Data Processing Inequality but for other probabilistic
structures besides Markov chains.

\begin{thm}[Data Sourcing Inequality]
  Given a probability model defined by the following (Common Cause):
  \begin{center}
    \begin{tikzcd}
         & Y \arrow[dl] \arrow[dr] & \\
      X  &                         & Z
    \end{tikzcd}
  \end{center}
  then it must be that $I(X;Y) \geq I(X;Z)$.
\end{thm}
\begin{proof}
  The implication of the common cause structure is that
  \begin{equation}
    p(x,y,z) = p(x \vert y )p(y)p(z \vert y).
  \end{equation}
  It follows that $X \independent Z \vert Y$.
  The rest of the proof is identifical to the previous proof.
\end{proof}

\begin{thm}[Unobserved common effect inequality]
  Given variables $X,Y,Z$ with
the common cause structure
\begin{center}
  \begin{tikzcd}
    X \arrow[dr] &   & \arrow[dl] Z \\
    & Y &                        &
  \end{tikzcd}
\end{center}
  then it must be that $I(X,Y) \geq I(X,Z) = 0$.
\end{thm}
\begin{proof}
  The implication of the structure is that
  \begin{equation}
    p(x,y,z) = p(x) p(y \vert x,z) p(z).
  \end{equation}
  It follows that $X \independent Z$, therefore $I(X;Z) = 0$.
  Because the mutual information of two variables is always
  nonnegative,
  $$I(X,Y) \geq I(X;Z)$$
\end{proof}

We note that while a similar property holds for a
``collider'' or common effect structure, it's proof is different
from the chain and common cause cases because, in general,
it is not the case that $X \independent Z \vert Y$ for
a common effect structure.
For example, when $X$ and $Z$ are both fair coin tosses
and $Y = X \oplus Z$, $X$ and $Z$ are independent from each other
but not when conditioned on $Y$.

When a common effect is in the conditioning set,
the two causes depend probabilistically on each other.
The extent to which these dependencies are limited can
be characterized by a few equations.

\begin{lem}
  \label{lemma:common-effect-1}
  Given variables $X_1,X_2,Y$ with
  the common effect structure $X_1 \rightarrow Y \leftarrow X_2$,
  then $I(X_1;X_2,Y) = I(X_1,Y \vert X_2)$.
\end{lem}
\begin{proof}
  By the Chain Rule for mutual information,
  $$I(X_1;X_2,Y) = I(X_1;X_2) + I(X_1;Y \vert X_2)$$
  Because of the common effect structure, $I(X_1;X_2) = 0$.
  Therefore, $I(X_1;X_2,Y) = I(X_1;Y \vert X_2)$.
\end{proof}

\begin{lem}
  \label{lemma:common-effect-2}
  Given variables $X_1,X_2,Y$ with
  the common effect structure $X_1 \rightarrow Y \leftarrow X_2$,
  then
  \begin{equation}
    \begin{split}
      I(Y; X_1, X_2) & = I(X_1;X_2,Y) + I(X_2;Y) \\
      & = I(X_2;X_1,Y) + I(X_1;Y)
    \end{split}
  \end{equation}
\end{lem}
\begin{proof}
  \begin{equation}
    \begin{split}
      & I(X_1;X_2,Y) \\
      & = I(X_1;Y \vert X_2) \\
      & = H(Y \vert X_2) - H(Y \vert X_1, X_2)\\
      & = H(Y \vert X_2) - H(Y) + I(Y ; X_1, X_2)\\
      & = I(Y ; X_1, X_2) - I(X_2;Y)
    \end{split}
  \end{equation}
  which implies that
  $$I(X_1;X_2,Y) + I(X_2;Y) = I(Y ; X_1, X_2)$$
  The proof works symmetrically for
  $I(X_2;X_1,Y) + I(X_1;Y) = I(Y ; X_1, X_2)$
\end{proof}

\begin{lem}
  \label{lemma:common-effect-3}
  Given variables $X_1,X_2,Y$ with
  the common effect structure $X_1 \rightarrow Y \leftarrow X_2$,
  then $I(X_1,X_2 \vert Y) \leq I(X_1;X_2,Y)$.
\end{lem}
\begin{proof}
  \begin{equation}
    \begin{split}
      & I(X_1,X_2 \vert Y) \\
      & = H(X_1 \vert Y) - H(X_1 \vert X_2, Y) \\
      & = H(X_1) - I(X_1;Y) - H(X_1) + I(X_1;X_2,Y)\\
      & = I(X_1;X_2,Y) - I(X_1;Y) \\
      & \leq I(X_1;X_2,Y) \\
    \end{split}
  \end{equation}
\end{proof}

\begin{thm}
  \label{thm:common-effect-inequality}
  Given variables $X_1,X_2,Y$ with
  the common effect structure $X_1 \rightarrow Y \leftarrow X_2$, then
  $$I(X_1,X_2 ; Y) \geq I(X_1;X_2,Y) = I(X_1 ; Y \vert X_2) \geq I(X_1;X_2 \vert Y)$$
\end{thm}
\begin{proof}
  Follows from Lemmas \ref{lemma:common-effect-1},
  \ref{lemma:common-effect-2}, and \ref{lemma:common-effect-3}.
\end{proof}

\section{Quartet Structures}

While triplet structures (chain, common effect, and common cause)
are the building blocks of larger paths in Bayesian networks,
an analysis of larger, quarter structures will help us develop
general theorems about the mutual information along paths.

Recall that a both with a common effect is not blocked
if \emph{either} the common effect \emph{or} a descendant of
the effect is in the conditioning set.
Let's look at the following structure,
which we will call a \emph{wishbone} structure.

\begin{center}
  \begin{tikzcd}
    X_0 \arrow[dr] &   & \arrow[dl] X_1 \\
    & Y \arrow[d] & \\
    & Z &
  \end{tikzcd}
\end{center}

Here, $Y$ is a common effect of $X_0$ and $X_1$,
and $Z$ is a descendant of $Y$.
How much information flows from $X_0$ to $X_1$
when $Z$ is known?

\begin{thm}
  \label{thm:wishbone}
  For variables $X_0, X_1, Y, Z$ in a wishbone structure,
  $$I(X_0;X_1 \vert Z) \leq I(Y;Z)$$
\end{thm}
\begin{proof}
  Consider the quantity $I(X_0, Y; X_1, Z)$,
  expanded by the Chain Rule.
  One expansion is:
  \begin{equation}
    \begin{split}
      & I(X_0, Y; X_1, Z) \\
      & = I(Y;Z) + I(X_0;Z \vert Y) + I(Y; X_1 \vert Z) + I(X_0, X_1 \vert Y,Z) \\
      & = I(Y;Z) + I(X_0,Y;X_1)
    \end{split}
  \end{equation}
  Another expansion is:
  \begin{equation}
    \begin{split}
      & I(X_0, Y; X_1, Z) \\
      & = I(X_0;Z) + I(Y;X_1 \vert X_0) +
      I(X_0; X_1 \vert Z) + I(Y; X_1 \vert X_0,Z) \\
      & \geq I(Y; X_1 \vert X_0) + I(X_0; X_1 \vert Z)
    \end{split}
  \end{equation}

  By Theorem \ref{thm:common-effect-inequality},
  we know that $I(Y;X_1 \vert X_0) = I(X_0,Y;X_1)$
  for three variables in a common effect structure,
  as they are for these variables in the wishbone structure.

  So we can set the two expansions equal to each other and reduce:
  \begin{equation}
    \begin{split}
      I(Y; X_1 \vert X_0) + I(X_0; X_1 \vert Z) & \leq I(Y;Z) + I(X_0,Y;X_1) \\
      I(X_0, Y; X_1) + I(X_0; X_1 \vert Z) & \leq I(Y;Z) + I(X_0,Y;X_1) \\
      I(X_0; X_1 \vert Z) & \leq I(Y;Z)
    \end{split}
  \end{equation}
\end{proof}

\section{Paths}

We can now look at mutual information of nodes connected
by longer paths.
We start with an arbitrariliy long Markov chain.

\begin{center}
  \begin{tikzcd}
    X_0  \arrow[r] & X_1 \arrow[r] & ... \arrow[r] & X_{n-1} \arrow[r] & X_n
  \end{tikzcd}
\end{center}

\begin{thm}[Chain Data Processing Inequality]
  \label{cdpi-thm}
  Given a Markov chain of variables $$X_1, ..., X_n$$
  such that $X_1 \rightarrow ... \rightarrow X_n$.
  It must be the case that
  $$I(X_1,X_n) \leq \min_i I(X_i,X_{i+1}).$$
\end{thm}
\begin{proof}
  \label{cdpi-prf}
  For all $i$, by the Chain rule for mutual information
  and the independence properties of the Markov chain,
  \begin{equation}
    \label{cdpi-prf-eq1}
    \begin{split}
      I(X_0, ..., X_{i} ; X_{i+1},...,X_n) = \\
      \sum_{j=i+1}^{n} I(X_0,...,X_{i}; X_j \vert X_{i+1},...,X_j) = \\
      I(X_0,...,X_{i}; X_{i+1}) = \\
      \sum_{j=i}^{1} I(X_{i+1}; X_{j} \vert X_i,...X_j) = \\
      I(X_i;X_{i+1}) + \sum_{j=i-1}^{1} I(X_{i+1}; X_{j} \vert X_i,...X_j) = \\
      I(X_i;X_{i+1})
    \end{split}
  \end{equation}
  The Chain rule can expand the variables in arbitary order.
  So we can also derive (using the fact that mutual information
  is always nonnegative):
  \begin{equation}
    \label{cdpi-prf-eq2}
    \begin{split}
      & I(X_0, ..., X_{i} ; X_{i+1},...,X_n) \\
      &= I(X_0, .., X_{i} ; X_n) + \sum_{j=n-1}^{i+1} I(X_{0}..,X_{i}; X_j \vert X_{j+1}..,X_j) \\
      &\geq I(X_0, ..., X_{i} ; X_n) \\
      &= \sum_{j = 0}^{n-1} I(X_n ; X_j \vert X_{j-1}, ... , X_0) \\
      &= I(X_n; X_0) + \sum_{j=1}^{n-1} I(X_n; X_j \vert X_{j-1}, ..., X_0) \\
      &\geq I(X_n; X_0)
    \end{split}
  \end{equation}
  Combining these two results and generalizing across all $i$,
  \begin{equation}
    \forall i, I(X_0;X_n) \leq I(X_i,X_{i+1})
  \end{equation}
  which entails that which is to be proven,
  \begin{equation}
    I(X_0;X_n) \leq \min_i I(X_i,X_{i+1})
  \end{equation}
\end{proof}

Our goal is to generalize this theorem to Bayesian paths
with other structures, just found as in the previous section
we found equivalents to the Data Processing Inequality in
other triplet structures.

\begin{dfn}[Path]
A \emph{path} between two nodes \(X_1\) and \(X_2\) in a graph
to be a sequence of nodes starting with \(X_1\) and ending with \(X_2\)
such that successive nodes are connected by an edge (traversing
in either direction).
\end{dfn}

In this section, we will only consider paths isolated from
any other variables.
We are interested in how to derive useful bounds on the
mutual information of a path based on the mutual information
of links within the path.

\begin{dfn}[Mutual information of a path]
  The \emph{mutual information of a path} between two nodes \(X\) and \(Y\)
  is $I(X,Y)$.
\end{dfn}

\begin{thm}[Unobserved Path Data Processing Inequality]
  \label{thm:updpi}
  Given a path between $X_0$ and $X_n$
  of variables $X_0, ..., X_n$, with no other connected variables.
  It must be the case that
  $$I(X_1,X_n) \leq \min_{i} I(X_i,X_{i+1}).$$
\end{thm}
\begin{proof}
  This proof mirrors the proof of Theorem \ref{cdpi-thm}.

  For any $i$, consider $I(X_0,...X_i;X_{i+1},...,X_n)$.

  By the logic of Equation \ref{cdpi-prf-eq1},
  $I(X_0,...X_i;X_{i+1},...,X_n) = I(X_i,X_{i+1})$.

  By the logic of Equation \ref{cdpi-prf-eq2},
  $I(X_0,...X_i;X_{i+1},...,X_n) \geq I(X_0,X_n)$.

  Therefore, $\forall i, I(X_0;X_n) \leq I(X_i,X_{i+1})$
  and $I(X_0;X_n) \leq \min_i I(X_i,X_{i+1})$.
\end{proof}

Theorem \ref{thm:updpi} applies to any paths on the condition
that none of the variables are observed.
Its proof is identical to the proof for Markov chains because
isolated, unobserved paths are Markov equivalent to Markov chains.

Some proofs extending this result follow from theory of Bayesian
networks. Recall that there are two conditions under which a
path betweent two variables is blocked.
First, an unobserved head-to-head connection on the
path blocks the path and makes the terminal nodes conditionally
independent. Second, an observation of a head-to-tail or tail-to-tail
node blocks the path and makes the terminal nodes conditionally
independent.
If the only paths between two variables are blocked, then they
are d-separated and therefore independent, with zero mutual information.

\begin{thm}[Blocked Path Mutual Information]
  \label{thm:bpmi}
  For any blocked paths between $X_0$ and $X_n$
  of variables $X_0, ..., X_n$ with no other connected
  variables, $I(X_0,X_n) = 0$.
\end{thm}
\begin{proof}
  If the only path between $X_0$ and $X_n$ is blocked,
  then $X_0$ and $X_n$ are d-separated and
  conditionally independent. If $X_0$ and $X_n$ are
  conditionally independent,
  then $I(X_0, X_n) = 0$.
\end{proof}

The difficult case for determining the mutual information
of a path is the case where there are observed common
effects on the paths.
This breaks the conditions for the proof of Theorem \ref{thm:updpi}.
It is possible for $I(X_i,X_{i+1}) = 0$ but
$I(X_{i-1},X_{i+1} \vert X_i) > 0$.
As a simple example, consider again the case where
$X_{i-1}$ and $X_{i+1}$ are fair coin tosses and
$X_i = X_{i-1} \oplus X_{x+1}$.

If there are many common effect nodes on the path and
only some of them are observed, then the path is
blocked and the mutual information is solved using
Theorem \ref{thm:bpmi}; the mutual information of the
path is zero.
Similarly, if there are common cause or chain triplets
on the path and the central node of the triplet is observed,
the mutual information of the path is trivially ze
So we need consider only the case where there's
a path where \emph{all and only} the common effect
nodes are observed.

\begin{thm}[Path Mutual Information Theorem (PMIT)]
  \label{thm:path-mutual-information}
  Given a path between $X_0$ and $X_n$
  of variables $\{X_0, ..., X_n\} = \mathcal{X}$ with no other connected
  variables.
  Let $\mathcal{X}_E$ be the common effect nodes, meaning only those
  nodes $X_i$ such that the edge structure of the path is
  $X_{i-1} \rightarrow X_i \leftarrow X_{i+1}$.
  The mutual information of the path when all the common effects
  are observed is is:

  $$I(X_0; X_n \vert \mathcal{X}_E) \leq min_{i}
  \begin{cases}
    I(X_i,X_{i+1}) & \text{if} X_i,X_{i+1} \notin \mathcal{X}_E \\
    I(X_{i-1},X_{i+1}) & \text{if} X_i \in \mathcal{X}_E\\
  \end{cases}$$

\end{thm}
\begin{proof}
  For any $i$, consider
  $$I(X_0,..., X_i;X_{i+1},..., X_n \vert \mathcal{X}_E)$$
  By the Chain Rule for mutual information, this can be expanded as
  $$\sum_{j=0}^i I(X_j;X_{i+1},..., X_n \vert X_0,..., X_{j-1},\mathcal{X}_E)$$

  Consider two cases.

  In the first case, $X_i \notin \mathcal{X}_E$
  and $X_{i+1} \notin \mathcal{X}_E$.

  By logic similar to Equation \ref{cdpi-prf-eq1}
  and Equation \ref{cdpi-prf-eq2}, then as before
  $I(X_0;X_n) \leq I(X_i,X_{i+1})$.

  In the second case, $X_i$ is a common effect node,
  i.e $X_i \in \mathcal{X}_E$.
  It is not possible to have two common
  efffect nodes adjacent on a path.
  So in any case where either $X_{i-1}$ or $X_{i+1}$ is
  in the conditioning set, the path is blocked.
  We can therefore compute the mutual information and its Chain Rule
  expansion as:

  \begin{equation}
    \begin{split}
      I(X_0,..., X_{i-1};X_{i+1},..., X_n \vert \mathcal{X}_E) \\
      = \sum_{j=i-1}^0 I(X_j;X_{i+1},..., X_n \vert X_{i-1},..., X_{j-1},\mathcal{X}_E) \\
      = I(X_{i-1};X_{i+1},..., X_n \vert \mathcal{X}_E)\\
      = I(X_{i-1};X_{i+1} \vert \mathcal{X}_E)\\
      = I(X_{i-1};X_{i+1} \vert X+i)
    \end{split}
  \end{equation}

  Since once again by the logic of Equation \ref{cdpi-prf-eq2}
  this value is greater than or equal to the mutual information of
  the path, we have
  $$I(X_0;X_n) \leq I(X_{i-1},X_{i+1} \vert X_i)$$
  for the cases when $X_i \in \mathcal{X}_E$.

  Combining these results, we get the bound on the mutual
  information of the path.
\end{proof}

Note that Theorem \ref{thm:updpi} is a special case of PMIT,
or Theorem \ref{thm:path-mutual-information}, where the
set of common effects on the path $\mathcal{X}_E$ is empty.

%*** What about if there is a common effect node,
%and a descendent of it is observed? ***
%
%\begin{center}
%  \begin{tikzcd}
%    X_0  \arrow[dr] & & & &  \\
%      & Y_0 \arrow[r] & ... \arrow[r] & Y_n \arrow[r] & Z \\
%    X_1 \arrow[ur] & & & &
%  \end{tikzcd}
%\end{center}
%
%\emph{TODO: Proof for this case}
%
%
%Note that Theorem \ref{thm:wishbone} generalizes to any
%variables such that the Markov
%properties implies by the wishbone structure hold.
%This can be the case even when the variables are
%part of a larger, more complex structure.
%For example, consider the structure:
%
%
%The inequality $I(X_0;X_1 \vert Z) \leq I(Y;Z)$
%still holds in this case.
%The difference is that $I(Y;Z)$ depends
%on the mutual information of the intermediate
%connections between $Y$ and $Z$.
%This brings us to our study of the mutual information
%along Bayesian network paths in general.
%

\end{document}


\chapter{Exercises}
\label{chap:Exercises}

% TODO
%
% All exercises should be group with a chapter
%
% ▶   Recommended
% M   Matematically oriented
% HM  Higher matematical education required
% MP  Matematical problem solving skills required,
%     double the rating otherwise; difficult is
%     very personal
%
% 00  Immediate
% 10  Simple
% 20  Medium
% 30  Moderately hard
% 40  Term project
% 50  Research project
%
% ⁺   High risk of undershoot difficulty

\begin{enumerate}[label=\textbf{\arabic*}.]

\item {[$\RHD$\textit{02}]} \textbf{Saturated subtraction}

Implement the function

\vspace{-1em}
\begin{alltt}
   void monus(z_t r, z_t a, z_t b);
\end{alltt}
\vspace{-1em}

\noindent
which calculates $r = a \dotminus b = \max \{ 0,~ a - b \}$.

\item {[$\RHD$\textit{10}]} \textbf{Modular powers of 2}

What is the advantage of using \texttt{zmodpow}
over \texttt{zbset} or \texttt{zlsh} in combination
with \texttt{zmod}?

\item {[\textit{M15}]} \textbf{Convergence of the Lucas Number ratios}

Find an approximation for
$\displaystyle{ \lim_{n \to \infty} \frac{L_{n + 1}}{L_n}}$,
where $L_n$ is the $n^{\text{th}}$
Lucas Number \psecref{sec:Lucas numbers}.

\( \displaystyle{
    L_n \stackrel{\text{\tiny{def}}}{\text{=}} \left \{ \begin{array}{ll}
      2 & \text{if} ~ n = 0 \\
      1 & \text{if} ~ n = 1 \\
      L_{n - 1} + L_{n + 1} & \text{otherwise}
    \end{array} \right .
}\)

\item {[\textit{MP12}]} \textbf{Factorisation of factorials}

Implement the function

\vspace{-1em}
\begin{alltt}
   void factor_fact(z_t n);
\end{alltt}
\vspace{-1em}

\noindent
which prints the prime factorisation of $n!$
(the $n^{\text{th}}$ factorial). The function shall
be efficient for all $n$ where all primes $p \le n$
can be found efficiently. You can assume that
$n \ge 2$. You should not evaluate $n!$.

\item {[\textit{M20}]} \textbf{Reverse factorisation of factorials}

{\small\textit{You should already have solved
``Factorisation of factorials'' before you solve
this problem.}}

Implement the function

\vspace{-1em}
\begin{alltt}
   void unfactor_fact(z_t x, z_t *P,
        unsigned long long int *K, size_t n);
\end{alltt}
\vspace{-1em}

\noindent
which given the factorsation of $x!$ determines $x$.
The factorisation of $x!$ is
$\displaystyle{\prod_{i = 1}^{n} P_i^{K_i}}$, where
$P_i$ is \texttt{P[i - 1]} and $K_i$ is \texttt{K[i - 1]}.

\item {[$\RHD$\textit{MP17}]} \textbf{Factorials inverted}

Implement the function

\vspace{-1em}
\begin{alltt}
   void unfact(z_t x, z_t n);
\end{alltt}
\vspace{-1em}

\noindent
which given a factorial number $n$, i.e. on the form
$x! = 1 \cdot 2 \cdot 3 \cdot \ldots \cdot x$,
calculates $x = n!^{-1}$. You can assume that
$n$ is a perfect factorial number and that $x \ge 1$.
Extra credit if you can detect when the input, $n$,
is not a factorial number. Such function would of
course return an \texttt{int} 1 if the input is a
factorial and 0 otherwise, or alternatively 0
on success and $-1$ with \texttt{errno} set to
\texttt{EDOM} if the input is not a factorial.

\item {[\textit{05}]} \textbf{Fast primality test}

$(x + y)^p \equiv x^p + y^p ~(\text{Mod}~p)$
for all primes $p$ and for a few composites $p$,
which are know as pseudoprimes. Use this to implement
a fast primality tester.

\item {[\textit{10}]} \textbf{Fermat primality test}

$a^{p - 1} \equiv 1 ~(\text{Mod}~p) ~\forall~ 1 < a < p$
for all primes $p$ and for a few composites $p$,
which are know as pseudoprimes\footnote{If $p$ is composite
but passes the test for all $a$, $p$ is a Carmichael
number.}. Use this to implement a heuristic primality
tester. Try to mimic \texttt{zptest} as much as possible.
GNU~MP uses $a = 210$, but you don't have to. ($a$ is
called a base.)

\item {[\textit{11}]} \textbf{Lucas–Lehmer primality test}

The Lucas–Lehmer primality test can be used to determine
whether a Mersenne numbers $M_n = 2^n - 1$ is a prime (a
Mersenne prime). $M_n$ is a prime if and only if
$s_{n - 1} \equiv 0 ~(\text{Mod}~M_n)$, where

\( \displaystyle{
    s_i = \left \{ \begin{array}{ll}
      4 & \text{if} ~ i = 0 \\
      s_{i - 1}^2 - 2 & \text{otherwise}.
    \end{array} \right .
}\)

\noindent
The Lucas–Lehmer primality test requires that $n \ge 3$,
however $M_2 = 2^2 - 1 = 3$ is a prime. Implement a version
of the Lucas–Lehmer primality test that takes $n$ as the
input. For some more fun, when you are done, you can
implement a version that takes $M_n$ as the input.

For improved performance, instead of using \texttt{zmod},
you can use the recursive function
%
\( \displaystyle{
    k \text{ mod } (2^n - 1) =
    \left (
      (k \text{ mod } 2^n) + \lfloor k \div 2^n \rfloor
    \right ) \text{ mod } (2^n - 1),
}\)
%
where $k \mod 2^n$ is efficiently calculated
using \texttt{zand($k$, $2^n - 1$)}. (This optimisation
is not part of the difficulty rating of this problem.)

\item {[\textit{20}]} \textbf{Fast primality test with bounded perfection}

Implement a primality test that is both very fast and
never returns \texttt{PROBABLY\_PRIME} for input less
than or equal to a preselected number.

\item {[\textit{30}]} \textbf{Powers of the golden ratio}

Implement function that returns $\varphi^n$ rounded
to the nearest integer, where $n$ is the input and
$\varphi$ is the golden ratio.

\item {[\textit{$\RHD$05}]} \textbf{zlshu and zrshu}

Why does libzahl have

\vspace{-1em}
\begin{alltt}
   void zlsh(z_t, z_t, size_t);
   void zrsh(z_t, z_t, size_t);
\end{alltt}
\vspace{-1em}

\noindent
rather than

\vspace{-1em}
\begin{alltt}
   void zlsh(z_t, z_t, z_t);
   void zrsh(z_t, z_t, z_t);
   void zlshu(z_t, z_t, size_t);
   void zrshu(z_t, z_t, size_t);
\end{alltt}
\vspace{-1em}

\item {[\textit{$\RHD$M15}]} \textbf{Modular left-shift}

Implement a function that calculates
$2^a \text{ mod } b$, using \texttt{zmod} and
only cheap functions. You can assume $a \ge 0$,
 $b \ge 1$. You can also assume that all
parameters are unique pointers.

\item {[\textit{$\RHD$08}]} \textbf{Modular left-shift, extended}

{\small\textit{You should already have solved
``Modular left-shift'' before you solve this
problem.}}

Extend the function you wrote in ``Modular left-shift''
to accept negative $b$ and non-unique pointers.

\item {[\textit{13}]} \textbf{The totient}

The totient of $n$ is the number of integer $a$,
$0 < a < n$ that are relatively prime to $n$.
Implement Euler's totient function $\varphi(n)$
which calculates the totient of $n$. Its
formula is

\( \displaystyle{
    \varphi(n) = |n| \prod_{p \in \textbf{P} : p | n}
    \left ( 1 - \frac{1}{p} \right ).
}\)

Note that $\varphi(-n) = \varphi(n)$, $\varphi(0) = 0$,
and $\varphi(1) = 1$.

\item {[\textit{M13}]} \textbf{The totient from factorisation}

Implement the function

\vspace{-1em}
\begin{alltt}
   void totient_fact(z_t t, z_t *P,
                     unsigned long long int *K, size_t n);
\end{alltt}
\vspace{-1em}

\noindent
which calculates the totient $t = \varphi(n)$, where
$n = \displaystyle{\prod_{i = 1}^n P_i^{K_i}} > 0$,
and $P_i = \texttt{P[i - 1]} \in \textbf{P}$,
$K_i = \texttt{K[i - 1]} \ge 1$. All values \texttt{P}
are mutually unique. \texttt{P} and \texttt{K} make up
the prime factorisation of $n$.

You can use the following rules:

\( \displaystyle{
  \begin{array}{ll}
      \varphi(1) = 1                      & \\
      \varphi(p) = p - 1                  & \text{if } p \in \textbf{P} \\
      \varphi(nm) = \varphi(n)\varphi(m)  & \text{if } \gcd(n, m) = 1   \\
      n^a\varphi(n) = \varphi(n^{a + 1})  &
  \end{array}
}\)

\item {[\textit{HMP32}]} \textbf{Modular tetration}

Implement the function

\vspace{-1em}
\begin{alltt}
   void modtetra(z_t r, z_t b, unsigned long n, z_t m);
\end{alltt}
\vspace{-1em}

\noindent
which calculates $r = {}^n{}b \text{ mod } m$, where
${}^0{}b = 1$, ${}^1{}b = b$, ${}^2{}b = b^b$,
${}^3{}b = b^{b^b}$, ${}^4{}b = b^{b^{b^b}}$, and so on.
You can assume $b > 0$ and $m > 0$. You can also assume
\texttt{r}, \texttt{b}, and \texttt{m} are mutually
unique pointers.

\item {[\textit{13}]} \textbf{Modular generalised power towers}

{\small\textit{This problem requires a working
solution for ``Modular tetration''.}}

Modify your solution for ``Modular tetration'' to
evaluate any expression on the forms
$a^b,~a^{b^c},~a^{b^{c^d}},~\ldots \text{ mod } m$.

\end{enumerate}

\chapter{Solutions}
\label{chap:Solutions}

\begin{enumerate}[label=\textbf{\arabic*}.]

\item \textbf{Saturated subtraction}

\vspace{-1em}
\begin{alltt}
void monus(z_t r, z_t a, z_t b)
\{
    zsub(r, a, b);
    if (zsignum(r) < 0)
        zsetu(r, 0);
\}
\end{alltt}
\vspace{-1em}

\item \textbf{Modular powers of 2}

\texttt{zbset} and \texttt{zbit} requires $\Theta(n)$
memory to calculate $2^n$. \texttt{zmodpow} only
requires $\mathcal{O}(\min \{n, \log m\})$ memory
to calculate $2^n \text{ mod } m$. $\Theta(n)$
memory complexity becomes problematic for very
large $n$.

\item \textbf{Convergence of the Lucas Number ratios}

It would be a mistake to use bignum, and bigint in particular,
to solve this problem. Good old mathematics is a much better solution.

$$ \lim_{n \to \infty} \frac{L_{n + 1}}{L_n} = \lim_{n \to \infty} \frac{L_{n}}{L_{n - 1}} = \lim_{n \to \infty} \frac{L_{n - 1}}{L_{n - 2}} $$

$$ \frac{L_{n}}{L_{n - 1}} = \frac{L_{n - 1}}{L_{n - 2}} $$

$$ \frac{L_{n - 1} + L_{n - 2}}{L_{n - 1}} = \frac{L_{n - 1}}{L_{n - 2}} $$

$$ 1 + \frac{L_{n - 2}}{L_{n - 1}} = \frac{L_{n - 1}}{L_{n - 2}} $$

$$ 1 + \varphi = \frac{1}{\varphi} $$

So the ratio tends toward the golden ratio.

\item \textbf{Factorisation of factorials}

Base your implementation on

\( \displaystyle{
    n! = \prod_{p~\in~\textbf{P}}^{n} p^{k_p}, ~\text{where}~
    k_p = \sum_{a = 1}^{\lfloor \log_p n \rfloor} \lfloor np^{-a} \rfloor.
}\)

There is no need to calculate $\lfloor \log_p n \rfloor$,
you will see when $p^a > n$.

\item \textbf{Reverse factorisation of factorials}

$\displaystyle{x = \max_{p ~\in~ P} ~ p \cdot f(p, k_p)}$,
where $k_p$ is the power of $p$ in the factorisation
of $x!$. $f(p, k)$ is defined as:

\vspace{1em}
\hspace{-2.8ex}
\begin{minipage}{\linewidth}
\begin{algorithmic}
    \STATE $k^\prime \gets 0$
    \WHILE{$k > 0$}
      \STATE $a \gets 0$
      \WHILE{$p^a \le k$}
        \STATE $k \gets k - p^a$
        \STATE $a \gets a + 1$
      \ENDWHILE
      \STATE $k^\prime \gets k^\prime + p^{a - 1}$
    \ENDWHILE
    \RETURN $k^\prime$
\end{algorithmic}
\end{minipage}
\vspace{1em}

\item \textbf{Factorials inverted}

Use \texttt{zlsb} to get the power of 2 in the
prime factorisation of $n$, that is, the number
of times $n$ is divisible by 2. If we write $n$ on
the form $1 \cdot 2 \cdot 3 \cdot \ldots \cdot x$,
every $2^\text{nd}$ factor is divisible by 2, every
$4^\text{th}$ factor is divisible by $2^2$, and so on.
From calling \texttt{zlsb} we know how many times,
$n$ is divisible by 2, but know how many of the factors
are divisible by 2, but this can be calculated with
the following algorithm, where $k$ is the number
times $n$ is divisible by 2:

\vspace{1em}
\hspace{-2.8ex}
\begin{minipage}{\linewidth}
\begin{algorithmic}
    \STATE $k^\prime \gets 0$
    \WHILE{$k > 0$}
      \STATE $a \gets 0$
      \WHILE{$2^a \le k$}
        \STATE $k \gets k - 2^a$
        \STATE $a \gets a + 1$
      \ENDWHILE
      \STATE $k^\prime \gets k^\prime + 2^{a - 1}$
    \ENDWHILE
    \RETURN $k^\prime$
\end{algorithmic}
\end{minipage}
\vspace{1em}

\noindent
Note that $2^a$ is efficiently calculated with,
\texttt{zlsh}, but it is more efficient to use
\texttt{zbset}.

Now that we know $k^\prime$, the number of
factors ni $1 \cdot \ldots \cdot x$ that are
divisible by 2, we have two choices for $x$:
$k^\prime$ and $k^\prime + 1$. To check which, we
calculate $(k^\prime - 1)!!$ (the semifactoral, i.e.
$1 \cdot 3 \cdot 5 \cdot \ldots \cdot (k^\prime - 1)$)
naïvely and shift the result $k$ steps to the left.
This gives us $k^\prime!$. If $x! \neq k^\prime!$, then
$x = k^\prime + 1$ unless $n$ is not factorial number.
Of course, if $x! = k^\prime!$, then $x = k^\prime$.

\item \textbf{Fast primality test}

If we select $x = y = 1$ we get
$2^p \equiv 2 ~(\text{Mod}~p)$. This gives us

\vspace{-1em}
\begin{alltt}
enum zprimality
ptest_fast(z_t p)
\{
    z_t a;
    int c = zcmpu(p, 2);
    if (c <= 0)
        return c ? NONPRIME : PRIME;
    zinit(a);
    zsetu(a, 1);
    zlsh(a, a, p);
    zmod(a, a, p);
    c = zcmpu(a, 2);
    zfree(a);
    return c ? NONPRIME : PROBABLY_PRIME;
\}
\end{alltt}
\vspace{-1em}

\item \textbf{Fermat primality test}

Below is a simple implementation. It can be improved by
just testing against a fix base, such as $a = 210$, it
$t = 0$. It could also do an exhaustive test (all $a$
such that $1 < a < p$) if $t < 0$.

\vspace{-1em}
\begin{alltt}
enum zprimality
ptest_fermat(z_t witness, z_t p, int t)
\{
    enum zprimality rc = PROBABLY_PRIME;
    z_t a, p1, p3, temp;
    int c;

    if ((c = zcmpu(p, 2)) <= 0) \{
        if (!c)
            return PRIME;
        if (witness && witness != p)
            zset(witness, p);
        return NONPRIME;
    \}

    zinit(a), zinit(p1), zinit(p3), zinit(temp);
    zsetu(temp, 3), zsub(p3, p, temp);
    zsetu(temp, 1), zsub(p1, p, temp);

    zsetu(temp, 2);
    while (t--) \{
        zrand(a, DEFAULT_RANDOM, UNIFORM, p3);
        zadd(a, a, temp);
        zmodpow(a, a, p1, p);
        if (zcmpu(a, 1)) \{
            if (witness)
                zswap(witness, a);
            rc = NONPRIME;
            break;
        \}
    \}

    zfree(temp), zfree(p3), zfree(p1), zfree(a);
    return rc;
\}
\end{alltt}
\vspace{-1em}

\item \textbf{Lucas–Lehmer primality test}

\vspace{-1em}
\begin{alltt}
enum zprimality
ptest_llt(z_t n)
\{
    z_t s, M;
    int c;
    size_t p;

    if ((c = zcmpu(n, 2)) <= 0)
        return c ? NONPRIME : PRIME;

    if (n->used > 1) \{
        \textcolor{c}{/* \textrm{An optimised implementation would not need this} */}
        errno = ENOMEM;
        return (enum zprimality)(-1);
    \}

    zinit(s), zinit(M), zinit(2);

    p = (size_t)(n->chars[0]);
    zsetu(s, 1), zsetu(M, 0);
    zbset(M, M, p, 1), zsub(M, M, s);
    zsetu(s, 4);
    zsetu(two, 2);

    p -= 2;
    while (p--) \{
        zsqr(s, s);
        zsub(s, s, two);
        zmod(s, s, M);
    \}
    c = zzero(s);

    zfree(two), zfree(M), zfree(s);
    return c ? PRIME : NONPRIME;
\}
\end{alltt}

$M_n$ is composite if $n$ is composite, therefore,
if you do not expect prime-only values on $n$, the
performance can be improved by using some other
primality test (or this same test if $n$ is a
Mersenne number) to first check that $n$ is prime.

\item \textbf{Fast primality test with bounded perfection}

First we select a fast primality test. We can use
$2^p \equiv 2 ~(\texttt{Mod}~ p) ~\forall~ p \in \textbf{P}$,
as describe in the solution for the problem
\textit{Fast primality test}.

Next, we use this to generate a large list of primes and
pseudoprimes. Use a perfect primality test, such as a
naïve test or the AKS primality test, to filter out all
primes and retain only the pseudoprimes. This is not in
runtime so it does not matter that this is slow, but to
speed it up, we can use a probabilistic test such the
Miller–Rabin primality test (\texttt{zptest}) before we
use the perfect test.

Now that we have a quite large — but not humongous — list
of pseudoprimes, we can incorporate it into our fast
primality test. For any input that passes the test, and
is less or equal to the largest pseudoprime we found,
binary search our list of pseudoprime for the input.

For input, larger than our limit, that passes the test,
we can run it through \texttt{zptest} to reduce the
number of false positives.

As an alternative solution, instead of comparing against
known pseudoprimes. Find a minimal set of primes that
includes divisors for all known pseudoprimes, and do
trail division with these primes when a number passes
the test. No pseudoprime need to have more than one divisor
included in the set, so this set cannot be larger than
the set of pseudoprimes.

\item \textbf{Powers of the golden ratio}

This was an information gathering exercise.
For $n \ge 2$, $L_n = [\varphi^n]$, where
$L_n$ is the $n^\text{th}$ Lucas number.

\( \displaystyle{
    L_n \stackrel{\text{\tiny{def}}}{\text{=}} \left \{ \begin{array}{ll}
      2 & \text{if} ~ n = 0 \\
      1 & \text{if} ~ n = 1 \\
      L_{n - 1} + L_{n + 1} & \text{otherwise}
    \end{array} \right .
}\)

\noindent
but for efficiency and briefness, we will use
\texttt{lucas} from \secref{sec:Lucas numbers}.

\vspace{-1em}
\begin{alltt}
void
golden_pow(z_t r, z_t n)
\{
    if (zsignum(n) <= 0)
        zsetu(r, zcmpi(n, -1) >= 0);
    else if (!zcmpu(n, 1))
        zsetu(r, 2);
    else
        lucas(r, n);
\}
\end{alltt}
\vspace{-1em}

\item \textbf{zlshu and zrshu}

You are in big trouble, memorywise, of you
need to evaluate $2^{2^{64}}$.

\item \textbf{Modular left-shift}

\vspace{-1em}
\begin{alltt}
void
modlsh(z_t r, z_t a, z_t b)
\{
    z_t t, at;
    size_t s = zbits(b);

    zinit(t), zinit(at);
    zset(at, a);
    zsetu(r, 1);
    zsetu(t, s);

    while (zcmp(at, t) > 0) \{
        zsub(at, at, t);
        zlsh(r, r, t);
        zmod(r, r, b);
        if (zzero(r))
            break;
    \}
    if (!zzero(a) && !zzero(b)) \{
        zlsh(r, r, a);
        zmod(r, r, b);
    \}

    zfree(at), zfree(t);
\}
\end{alltt}
\vspace{-1em}

It is worth noting that this function is
not necessarily faster than \texttt{zmodpow}.

\item \textbf{Modular left-shift, extended}

The sign of \texttt{b} shall not effect the
result. Use \texttt{zabs} to create a copy
of \texttt{b} with its absolute value.

\item \textbf{The totient}

\( \displaystyle{
    \varphi(n)
    = n \prod_{p \in \textbf{P} : p | n} \left ( 1 - \frac{1}{p} \right )
    = n \prod_{p \in \textbf{P} : p | n} \left ( \frac{p - 1}{p} \right )
}\)

\noindent
So, if we set $a = n$ and $b = 1$, then we iterate
of all integers $p$, $2 \le p \le n$. For which $p$
that is prime, we set $a \gets a \cdot (p - 1)$ and
$b \gets b \cdot p$. After the iteration, $b | a$,
and $\varphi(n) = \frac{a}{b}$. However, if $n < 0$,
then, $\varphi(n) = \varphi|n|$.

\item \textbf{The totient from factorisation}

\vspace{-1em}
\begin{alltt}
void
totient_fact(z_t t, z_t *P,
             unsigned long long *K, size_t n)
\{
    z_t a, one;
    zinit(a), zinit(one);
    zseti(t, 1);
    zseti(one, 1);
    while (n--) \{
        zpowu(a, P[n], K[n] - 1);
        zmul(t, t, a);
        zsub(a, P[n], one);
        zmul(t, t, a);
    \}
    zfree(a), zfree(one);
\}
\end{alltt}
\vspace{-1em}

\item \textbf{Modular tetration}

Let \texttt{totient} be Euler's totient function.
It is described in the problem ``The totient''.

We need two help function: \texttt{tetra(r, b, n)}
which calculated $r = {}^n{}b$, and \texttt{cmp\_tetra(a, b, n)}
which is compares $a$ to ${}^n{}b$.

\vspace{-1em}
\begin{alltt}
void
tetra(z_t r, z_t b, unsigned long n)
\{
    zsetu(r, 1);
    while (n--)
        zpow(r, b, r);
\}
\end{alltt}
\vspace{-1em}

\vspace{-1em}
\begin{alltt}
int
cmp_tetra(z_t a, z_t b, unsigned long n)
\{
    z_t B;
    int cmp;

    if (!n || !zcmpu(b, 1))
        return zcmpu(a, 1);
    if (n == 1)
        return zcmp(a, b);
    if (zcmp(a, b) >= 0)
        return +1;

    zinit(B);
    zsetu(B, 1);
    while (n) \{
        zpow(B, b, B);
        if (zcmp(a, B) < 0) \{
            zfree(B);
            return -1;
        \}
    \}
    cmp = zcmp(a, B);
    zfree(B);
    return cmp;

\}
\end{alltt}
\vspace{-1em}

\texttt{tetra} can generate unmaintainably huge
numbers. Will however only call \texttt{tetra}
when this is not the case.

\vspace{-1em}
\begin{alltt}
void
modtetra(z_t r, z_t b, unsigned long n, z_t m)
\{
    z_t t, temp;

    if (n <= 1) \{
        if (!n)
            zsetu(r, zcmpu(m, 1));
        else
            zmod(r, b, m);
        return;
    \}

    zmod(r, b, m);
    if (zcmpu(r, 1) <= 0)
        return;

    zinit(t);
    zinit(temp);

    t = totient(m);
    zgcd(temp, b, m);

    if (!zcmpu(temp, 1)) \{
        modtetra(temp, b, n - 1, t);
        zmodpow(r, r, temp, m);
    \} else if (cmp_tetra(t, b, n - 1) > 0) \{
        temp = tetra(b, n - 1);
        zpowmod(r, r, temp, m);
    \} else \{
        modtetra(temp, b, n - 1, t);
        zmodpow(temp, r, temp, m);
        zmodpow(r, r, t, m);
        zmodmul(r, r, temp, m);
    \}

    zfree(temp);
    zfree(t);
\}
\end{alltt}
\vspace{-1em}

\item \textbf{Modular generalised power towers}

Instead of the signature

\vspace{-1em}
\begin{alltt}
   void modtetra(z_t r, z_t b, unsigned long n, z_t m);
\end{alltt}
\vspace{-1em}

\noindent
you want to use the signature

\vspace{-1em}
\begin{alltt}
   void modtower_(z_t r, z_t *a, size_t n, z_t m);
\end{alltt}
\vspace{-1em}

Instead of using, \texttt{b} (in \texttt{modtetra}),
use \texttt{*a}. At every recursion, instead of
passing on \texttt{a}, pass on \texttt{a + 1}.

The function \texttt{tetra} is modified into

\vspace{-1em}
\begin{alltt}
   void tower(z_t r, z_t *a, size_t n)
   \{
       zsetu(r, 1);
       while (n--);
           zpow(r, a[n], r);
   \}
\end{alltt}
\vspace{-1em}

\noindent
\texttt{cmp\_tetra} is changed analogously.

To avoid problems in the evaluation, define

\vspace{-1em}
\begin{alltt}
   void modtower(z_t r, z_t *a, size_t n, z_t m);
\end{alltt}
\vspace{-1em}

\noindent
which cuts the power short at the first element
of of \texttt{a} that equals 1. For example, if
$a = \{2, 3, 4, 5, 1, 2, 3\}$, and $n = 7$
($n = |a|$), then \texttt{modtower}, sets $n = 4$,
and effectively $a = \{2, 3, 4, 5\}$. After this
\texttt{modtower} calls \texttt{modtower\_}.

\end{enumerate}


\section{Introduction}

%\paragraph{(Frame the problem)}

Today’s social, behavioral and medical scientists have access to large multidimensional datasets that can be
used to investigate the complex relationships between social, psychological and biological factors in
shaping individual and societal outcomes.
Large social science datasets, such as the World Values Survey, or the European Values Study (EVS),
are easily available to researchers and initiatives have been undertaken to link and extend these datasets
into a system of linked open data.
Making use of the full potential of these data sets requires dealing with the crucial problem of multivariate
missing data.

Rubin's Multiple Imputation (MI) approach \citep{rubin:1987} was developed to specifically address the issue of missing
responses in surveys.
MI is a three-step process that entails an imputation, analysis, and pooling phase.
The fundamental idea of the imputation phase is to replace each missing data point with $m$ plausible values sampled from
their posterior predictive distributions given the observed data.
This procedure leads to the definition of $m$ complete versions of the original data that can be analyzed separately
using standard complete-data analysis models (analysis phase).
Finally, the $m$ estimates of any parameter of interest can be pooled following Rubin's rules \citep{rubin:1987}
(pooling phase).

Since Rubin's seminal work, two main strategies have become popular for multiple imputation of multivariate
missing data: joint modelling (JM) \citep[ch. 4]{schafer:1997} and full conditional specification (FCS), also known
as Multiple Imputation by Chained Equation (MICE) \citep{vanBuurenEtAl:2006}.
The first one relies on defining a multivariate distribution for the missing data, deriving conditional
distributions for each missing data pattern, and obtaining samples from it by means of a Markov Chain Monte Carlo
algorithm.
The second method defines conditional densities for each incomplete variable and performs iterative imputations on
a variable-by-variable basis.
Compared to the JM approach, FCS imputation can easily accommodate for the different distributions of variables and it
can preserve unique features in the data, such as variables interactions or questionnaires skip patterns.

Both the JM and the FCS approaches rely on the crucial \emph{missing at random} (MAR) assumption.
Meeting this assumption requires specifying imputation models for the MI procedure that include all observed
variables that are correlates of missingness.
Omitting from the imputation models an observed predictor related to both the missingness and the imputed variables
creates a \emph{missing not at random} (MNAR) problem.
MI under MNAR leads to substantial bias in parameter estimation in the analysis and pooling phases, and
invalidates hypothesis testing involving the imputed variables.

As a result, when it comes to defining the set of auxiliary variables for the imputation models within an MI procedure,
an inclusive strategy (i.e. including numerous auxiliary variables) is generally preferred to restrictive approach
(i.e. including few or no auxiliary variables).
An inclusive approach reduces the chances of omitting important correlates of missingness, making the MAR assumption
more plausible.
Furthermore, the inclusive strategy has been shown to reduce estimation bias and increase efficiency \citep{collinsEtAl:2001},
as well as reducing the chances of specifying uncongenial imputation and analysis models \citep{meng:1994}.

Specifying the imputation models for a FCS MI procedure remains one of the most challenging steps in dealing
with missing values for large multidimensional data sets.
In practice, the inclusive strategy faces identification and computational limitations.
One serious risk of an inclusive strategy is the occurrence of singular matrices within the imputation algorithm.
When data is high-dimensional (i.e. the number of recorded units $n$ is not substantially larger than the number of recorded
variables $p$) or afflicted by high collinearity (i.e. one or more of the variables is equal to a linear
combination of the others) the data covariance matrix is singular.
Singular matrices are not invertible, an operation that is fundamental in the estimation of the imputation
models in any parametric imputation procedure.
As a result, the possible high dimensionality of the observed data matrix, resulting from an inclusive strategy,
can prevent a straightforward application of MI algorithms or force researchers to make arbitrary choices
regarding which variables to include in the imputation models.

Recent developments in high-dimensional data MI techniques represent interesting opportunities to embrace an
inclusive strategy, without facing its downsides.
Some statisticians and machine learning experts have focused on high-dimensional Single Imputation (SI) methods in
an effort to improve the accuracy of individual imputations \citep[e.g.][]{kimEtAl:2005, stekhovenBuhlmann:2011,
d'ambrosioEtAl:2012}.
However, the main task of social scientists is to make inference about a population based on a sample of observed
data points, and SI is simply inadequate for this purpose: it does not provide statistically valid
inference \citep{rubin:1996}.
The concept of statistical validity as defined by \citep{rubin:1996} is meant to capture two features of
estimation.
First, the point estimate of a parameter of interest must be unbiased, and second, the actual
confidence interval coverage (CIC) of the true parameter value must be equal or greater than nominal
coverage rates.
SI strategies might meet the first requirement, but cannot meet the second as they
do not take into account the uncertainty regarding the imputed values.
MI, on the other hand, was designed to provide statistically valid inference and therefore is
more suitable for social scientific research.

The combination of MI with high-dimensional prediction models has been directly tackled by algorithms combining
full conditional specification of imputation models with shrinkage methods \citep{zhaoLong:2016, dengEtAl:2016},
but their application has been studied only for biomedical sciences.
Other researchers have proposed FCS strategies using dimensionality reduction to avoid the obstacles of an inclusive
strategy in high-dimensional data imputation.
However, these solutions were either limited to the Joint Modeling approach \citep{songBelin:2004},
or tested exclusively on particularly low-dimensional settings \citep{howardEtAl:2015}.
Finally, tree-based FCS strategies also have the potential to overcome the limitations of inclusive strategies.
The non-parametric nature of decision trees bypasses the identification issues most parametric methods face
in high-dimensional contexts.
However, these methods have been either used to deal with different issues, such as imputation in the presence
of interaction effects \citep{dooveEtAl:2014}, or have been tested exclusively on biomedical datasets
\citep{shahEtAl:2014}.
%
\paragraph{Scope}
The inclusion of shrinkage methods, principal component analysis and non-parametric decision trees within the FCS framework
has the potential of simplifying the decisions social scientists need to make when dealing with missing values.
The lack of comparative research on the performance of these methods makes it difficult for social scientists working with
large multidimensional data sets to decide which imputation method to adopt.
With this article, we provide a comparison of these state-of-the-art high-dimensional imputation algorithms.
We compared the imputation methods based on the statistical validity of the complete-data analyses they allow to perform.
The comparison was based on three numerical experiments: two simulation studies and a resampling study using real survey
data.
%
\paragraph{Outline}
In what follows, we first present the general MICE framework, how the high-dimensional MI methods fit within it, and some
single data missing data strategies considered for reference.
Then we present the three numerical experiments, their design, and the results.
We then discus the implications of the results and provides recommendations for applied researchers.
Finally, we provide a description of the limitations of the study, and possible future research directions.


\problemname{Train Boarding}

Punctual City is well known for the punctuality of its citizens and
its public transportation system.  It is particularly famous for its
train system.  It is always on time, and never too late (or even too
early).  Statistics about train boarding is regularly collected to
keep things running smoothly.

A train has cars numbered $1$ to $N$ (from front to back), each of
length $L$ meters. Each car has exactly one door for boarding located
at the center ($L/2$ meters from each end of the car). There are no gaps
between cars.

When the train stops at the boarding platform, each passenger waiting
for the train walks to the door of the car which is closest to them,
taking the higher numbered car in the case of a tie.

Given the location of the passengers relative to the train, help the
city by reporting the longest distance that any passenger has
to walk and the maximum number of passengers boarding any single car.

\begin{center}
  \includegraphics[width=0.9\textwidth]{Diagram1.png}
\end{center}

\section*{Input}

The first line of input contains three integers $N$ ($1 \leq N \leq 100$),
which is the number of cars of the train, $L$ ($2 \leq L \leq 100$), which
is the length of each car, and $P$ ($1 \leq P \leq 1\,000$), which is the
number of passengers waiting for the train. It is guaranteed that $L$
is an even number.

The next $P$ lines describe the location of the passengers relative to
the train. Each line contains a single integer $x$
($0 \leq x \leq 10\,000$), which is the distance the passenger is
behind the front-end of the train.

\section*{Output}

Display the longest distance that any passenger has to walk on one
line.  On the next line, display the maximum number of passengers boarding
any single car.