\chapter{Set Theory II}

\epigraph{Algebra is the offer made by the devil to the mathematician. The devil says: I will give you this powerful machine, it will answer any question you like. All you need to do is give me your soul: give up geometry and you will have this marvelous machine.}{Michael Francis Atiyah}

Watch the Art of Problem Solving: Venn Diagrams with \href{https://www.youtube.com/watch?v=c8VHzEFVmLA}{two categories} and \href{https://www.youtube.com/watch?v=LIzIhmKlYPk}{three categories}!

We will use {\bf Venn diagrams} to visualise operations between sets. I think they are pretty self explanitory, so I will mostly go through a few examples here and move on. For example, below the left circle represents the $A$ set, and the right circle represents the $B$ set. The coloured portion of the graphic represents the intersection (shared elements) between $A$ and $B$. So for $A \cap B$:

\begin{venndiagram2sets}[shade=skyblue,showframe=false]
	\fillACapB
\end{venndiagram2sets}

\begin{boxexample}{}{}
	Suppose $A=\{a,b,c\}$ and $B=\{a,b,x,y\}$, then $A \cap B=\{a,b\}$ is visualised with:

	\begin{venndiagram2sets}[shade=skyblue,showframe=false,labelOnlyA={c},labelOnlyB={x,y},labelAB={a,b}]
		\fillACapB
	\end{venndiagram2sets}
\end{boxexample}

This time we will visualise the union of two sets. Again, the coloured area represents everything withing the union. So for $A \cup B$:

\begin{venndiagram2sets}[shade=skyblue,showframe=false]
	\fillA \fillB
\end{venndiagram2sets}

\begin{boxexample}{}{}
	Suppose $A=\{a,b,c\}$ and $B=\{a,b,x,y\}$, then $A \cup B=\{a,b,c,x,y\}$  is visualised with:

	\begin{venndiagram2sets}[shade=skyblue,showframe=false,labelOnlyA={c},labelOnlyB={x,y},labelAB={a,b}]
		\fillA \fillB
	\end{venndiagram2sets}
\end{boxexample}

And now we look at the complement of the set. So everything within the \emph{universe of discourse} that is not in the set is coloured. Consider the Venn Diagram for $A^\complement$:

\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
	\fillNotA
\end{venndiagram2sets}

\begin{boxexample}{}{}
	Suppose $A=\{a,b,c\}$ and $\mathcal{U}=\{a,b,c,d,e,f,g\}$, then $A^\complement=\{d,e,f,g\}$ can be visualised with:

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={\quad $d,e,f,g$},labelOnlyA={$a,b,c$}]
		\fillNotA
	\end{venndiagram2sets}
\end{boxexample}

And when we have two disjoint sets (two sets that do not share elements), no elements are coloured in their intersection. So for $A \cap B = \varnothing$:

\begin{venndiagram2sets}[shade=skyblue,showframe=false,overlap=-.5cm]
\end{venndiagram2sets}

\begin{boxexample}{}{}
	For example, given the two sets $A=\{a,b,c\}$, and $B=\{1,2,3\}$, we can see that they share no elements and are disjoint.

	\begin{venndiagram2sets}[shade=skyblue,showframe=false,overlap=-.5cm,labelOnlyA={a,b,c},labelOnlyB={1,2,3}]
	\end{venndiagram2sets}
\end{boxexample}

Now for the set-difference. We colour in the area that is in $A$, but not in $B$ So for $A \setminus B$:

\begin{venndiagram2sets}[shade=skyblue,showframe=false]
	\fillOnlyA
\end{venndiagram2sets}

\begin{boxexample}{}{}
	For example, if $A=\{a,b,c,d\}$ and $B=\{c,d\}$, than $A \setminus B=\{c,d\}$.

	\begin{venndiagram2sets}[shade=skyblue,showframe=false,labelOnlyA={c,d},labelAB={a,b}]
		\fillOnlyA
	\end{venndiagram2sets}
\end{boxexample}

\section{Applications for Venn Diagrams}

We'll go over some examples of filling out more complicated Venn Diagrams. In general, you can use simpler Venn Diagrams to build more complicated ones. You'll see that in the below examples.

\begin{boxexample}{}{}
	Suppose we want to visualise $(A \cap B)^\complement \cup B$. Thats a little long so lets break it up. Recall what $A \cap B$ looks like.

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillACapB
	\end{venndiagram2sets}

	Now, we'll invert that to get $(A \cap B)^\complement$:

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillNotAorNotB
	\end{venndiagram2sets}

	And now lets add $B$. So, $(A \cap B)^\complement \cup B$ is:

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillAll
	\end{venndiagram2sets}

	And we see that $(A \cap B)^\complement \cup B$ is acutally the universal set $\mathcal{U}$!
\end{boxexample}

\begin{boxexample}{}{}
	Let's try another one. $(A \cap B) \cup (A \cup B^\complement)^\complement$. Again, let's break this down. $A \cap B$ is simple enough.

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillACapB
	\end{venndiagram2sets}

	The next one is $A \cup B^\complement$, a little bit more tricky. That can be read as the area of A, plus the area of not B. Think about it for a bit, its a logic puzzle.
	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillA \fillNotAorB
	\end{venndiagram2sets}

	We take the complement of above (inverting it) to get $(A \cup B^\complement)^\complement$:

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillOnlyB
	\end{venndiagram2sets}

	And now we glue them together to get to get our union $(A \cap B) \cup (A \cup B^\complement)^\complement$:

	\begin{venndiagram2sets}[shade=skyblue,labelNotAB={$\mathcal{U}$}]
		\fillB
	\end{venndiagram2sets}

	From this digram, we can see there is another way of writting $(A \cap B) \cup (A \cup B^\complement)^\complement$. It's just the set $B$!
\end{boxexample}

\section{Applications for Sets}

Now we'll look at how sets and Venn Diagrams can be used to solve problems. In this class, we solve these intuitively. Algebra is another way to solve these problem though. These feel like Suduko puzzles to me.

\begin{boxexample}{}{}
	Suppose we surveyed 25 farms. 18 farms had potatoes, and 12 farms had corn. How many farms grow both potatoes and corn?\\

	If we add the number of farms with potatoes to the number of farms with corn, then the farms that grow both will be counted twice. In other words,

	\[
		|P|+|C|=|P \cup C| + |P \cap C|
	\]

	Now, if we slightly rearrange this equation, we get

	\[
		|P|+|C| - |P \cup C|=|P \cap C|
	\]

	So, we can find the intercept by plugging in the values that we know.

	\[
		18+12-25=|P \cap C| \implies |P \cap C|=5
	\]

	Our answer is 5, and this is our Venn Diagram:

	\begin{venndiagram2sets}[shade=skyblue,showframe=false,labelA={P},labelB={C},labelOnlyA={13},labelOnlyB={7},labelAB={5}]
		\fillACapB
	\end{venndiagram2sets}

\end{boxexample}

\begin{boxexample}{}{}
	Suppose 150 children were surveyed. It was found that 35 played hockey, 71 played baseball, 30 played soccer, 10 played all three, 3 played only soccer, 17 played only hockey, 6 played only soccer and hockey, 48 played only baseball, and 53 played no sports.\\

	First, lets draw the Venn Diagram. Start by drawing the three sets.

	\begin{venndiagram3sets}[shade=skyblue,labelA={H},labelB={B},labelC={S},labelNotABC={$\mathcal{U}$}]
	\end{venndiagram3sets}

	Now, let's fill some of this in. Since 53 didn't play anything, they are outside of the three sets. 3 Played only soccer, 17 played only hockey and 48 played only baseball. 10 Played all three, and 6 played only soccer and hockey. So we have:

	\begin{venndiagram3sets}[shade=skyblue,labelA={H},labelB={B},labelC={S},labelNotABC={\quad $\mathcal{U} \quad 53$},labelOnlyA={17},labelOnlyB={48},labelOnlyC={3},labelABC={10},labelOnlyAC={6}]
	\end{venndiagram3sets}

	We'll fill in some more intersections now. How many played only hockey and baseball? There are 35 hockey players, take away 17 who played only hockey, take away 10 who played all three, and take away 6 who played hockey and soccer. So $35-17-10-6=2$. How many played only soccer and baseball? 71 baseball players, take away 48 who played baseball only, take away 10 who played all three, take away 2 who played hockey and baseball. So, $71-48-10-2=11$. The diagram looks like:

	\begin{venndiagram3sets}[shade=skyblue,labelA={H},labelB={B},labelC={S},labelNotABC={\quad $\mathcal{U} \quad 53$},labelOnlyA={17},labelOnlyB={48},labelOnlyC={3},labelABC={10},labelOnlyAB={2},labelOnlyBC={11},labelOnlyAC={6}]
	\end{venndiagram3sets}

	Now we can use this to answer some questions.
	\begin{enumerate}
		\item How many played soccer and baseball only? $11$
		\item How many played hockey and baseball only? $2$
		\item How many played soccer and baseball? $10+11=21$
		\item How many played soccer {\bf or} baseball but not hockey? $48+11+3=62$
		\item How many played exactly 2 games? $6+2+11=19$
		\item How many played exactly 1 game? $17+48+3=68$
		\item How many played only soccer? $3$
		\item How many played soccer {\bf and} baseball but not hockey? $3+48=51$
	\end{enumerate}
\end{boxexample}

\begin{boxexample}{}{}
	Fourty students who play board games was surveyed.

	\medskip
	\begin{tabular}{r|l}
		\hline
		Number of students & Game(s) played\\
		\hline
		18 & Chess\\
		20 & Scrabble\\
		27 & Carrom\\
		7  & Chess \& Scrabble\\
		12 & Scrabble \& Carrom\\
		4  & Chess \& Carrom \& Scrabble\\
		\hline
		\multicolumn{2}{c}{40 Total}\\
		\hline
	\end{tabular}
	\medskip

	What is the number of people who play both Chess \& Carrom? What is the number of people who play Chess and Carrom but not Scrabble?\\

	We begin by drawing our Venn diagram. We are given the Chess \& Carrom \& Scrabble intersection as $4$. We can use that to find our Chess \& Scrabble only ($7-4=3$) and Scrabble \& Charrom only ($12-4=8$) intersections. Finally, we can calculate our scrabble-only section ($20-3-4-8=5$).

	\begin{venndiagram3sets}[shade=skyblue,showframe=false,labelA={Chess},labelB={Scrabble},labelC={Carrom},labelABC={4},labelOnlyAB={3},labelOnlyBC={8},labelOnlyB={5}]
	\end{venndiagram3sets}

	Now it gets slightly more tricky. Consider the following highlighted section ($\text{Chess} \cup \text{Carrom}$):

	\begin{venndiagram3sets}[shade=skyblue,showframe=false,labelA={Chess},labelB={Scrabble},labelC={Carrom},labelABC={4},labelOnlyAB={3},labelOnlyBC={8},labelOnlyB={5}]
		\fillA \fillC
	\end{venndiagram3sets}

	There must be $40-5=35$ students in there. We know that $27$ of them play Carrom, and that $18$ of them play chess. So the Chess \& Carrom intersection must be $27+18-35=10$, and the Chess \& Carrom intersection is $10-4=6$. Thus the chess-only section is $18-6-4-3=5$, and the carrom-only section is $27-6-4-8=9$.

	\begin{venndiagram3sets}[shade=skyblue,showframe=false,labelA={Chess},labelB={Scrabble},labelC={Carrom},labelABC={4},labelOnlyAB={3},labelOnlyBC={8},labelOnlyB={5},labelOnlyAC={6},labelOnlyA={5},labelOnlyC={9}]
	\end{venndiagram3sets}

	The number of people who play both Chess \& Carrom is $6+4=10$, and the number of people who play Chess and Carrom but not Scrabble is $6$.
\end{boxexample}


\section{Random Number Generation}
\label{sec:random-number-generation}

The generation of pseudo-random numbers is a fundamental building-block in any next-event simulation.
%
In fact, a sequence of pseudo-random numbers uniformly distributed in $(0,1)$ can be used to generate stochastic variates, e.g. the exponential distribution, that can be used in turn to generate streams of random events, e.g. requests to the system with random occurrence time and computational demand.
%
There exist many techniques for random number generation, a lot of which are comprehensively presented in \cite{l1994uniform}.
The most notable algorithmic generators are \textit{linear congruential generators}, \textit{multiple recursive generators}, \textit{composite generators}, and \textit{shift-register generators}.

In this work we adopted a custom implementation of a multi-stream Lehmer generator $(a,m,s)$, which belongs to the family of linear congruential generators and it is defined by the following equation:

\begin{equation}
\label{eqn:lehmer}
x_{i+1} = (a^{j} \mathrm{mod}m) x_{i} \mathrm{mod}m \qquad\forall j=0,...,s-1
\end{equation}

where $m$ is the modulus, $a$ is the multiplier, $s$ is the number of streams and $(a^{j} \mathrm{mod}m)$ is the jump multiplier.

We have chosen this solution because
(i) it provides a great degree of randomness with the appropriate parameters
(ii) the multi-streaming is required by simulations with multiple stochastic components,
(iii) it has a simple implementation and a smaller computational complexity with respect to others, and
(iv) it is a de-facto standard, hence it is easy to compare our experimental results with the ones provided in literature.

We propose a generator with the following parameters:

\begin{itemize}
	\item \textbf{modulus $\mathbf{2^{31}-1}$:} the modulus should be the maximum prime number that can be represented in the target system.
	Although all modern computers have a 64-bit architecture, we considered a 32-bit one because the algorithm to find the right multiplier for a 64-bit modulus can be very slow.
	For this reason we have chosen $2^{31}-1$ as our modulus.

	\item \textbf{multiplier $\mathbf{50812}$:} the multiplier should be \textit{full-period modulus-compatible} with respect to the chosen modulus. The chosen modulus has 23093 of such multipliers. Among these there are also multipliers such as $16807$, widely used in the past, and $48271$, that is currently the most widely adopted.
	We have chosen $50812$ as our multiplier because we wanted to study a suitable multiplier that is different from the de-facto standard.

	\item \textbf{$\mathbf{256}$ streams:} the original periodic random sequence can be partitioned in different disjoint periodic random sub-sequences, one for each stream.
	The number of streams should be no more than the number of required disjoint sub-sequences, because streams come with the cost of reducing the size of the random sequence.
	We have chosen 256 streams, even if it is a lot more than the strictly required for our simulations, because it is a de-facto standard hence it is useful for comparisons between our evaluation and the one proposed in literature \cite{leemis2006discrete}.

	\item \textbf{jump multiplier $\mathbf{29872}$:} the jump multiplier is used to partition the random sequence in disjoint sub-sequences, one for each stream, whose length is often called jump size. The jump multiplier should be \textit{modulus compatible} with the chosen modulus.
	We have chosen $29872$ as our jump multiplier because it is the value that maximizes the jump size.

	\item \textbf{initial seed $\mathbf{123456789}$:} the initial seed is the starting point of the finite sequence of generated values. Even if the initial seed does not impact the randomness degree of a generator in a single run (it only has to be changed in different replication of the same ensemble), we decide to indicate it here for completeness.
\end{itemize}

The randomness degree of such a generator has been assessed by the usage of \textit{spectral test}, \textit{test of extremes} and the \textit{analysis of Kolomogorv-Smirnov}.
%
The experimental results are reported in Section~\ref{sec:evaluation}.


\section{Line Integrals}
Under what circumstances is the line integral
$\int_{A}^{B} f(z) \, dz$
independent of the path from $A$ to $B$?

\begin{theorem}[Cauchy]
    If $f(z)$ is analytic in a simply connected,
    bounded domain $D$,
    for every simple closed path $C$ in $D$,
    \begin{align}
        \oint_C f(z) \, dz = 0
        \label{eqn:1}
    \end{align}
\end{theorem}
To prove this, consider
\begin{align}
    \oint f(z)\, dz
    &= \oint_C (u + iv) (dx + i\, dy)\\
    &= \oint_C (u\,dx - v\,dy) +
    i \oint_C (v\,dx + u\,dy)
\end{align}
Then use Stokes theorm
\begin{align}
    \oint\vec{A}\cdot d\vec{r}
    = \int(\vec{\nabla}\times\vec{A})\cdot d\vec{s}.
\end{align}
Let the line integral Lie in hte $xy$ plane.
\begin{align}
    d\vec{r}
    &= dx \hat{e}_x + dy\hat{e}_y\\
    \vec{A} &= A_x \hat{e}_x + A_y \hat{e}_y.
\end{align}
Then
\begin{align}
    \oint_C \left(
        A_x\,dx + A_y\,dy
    \right)
    = \int\left(
    \frac{\partial A_y}{\partial x}
    - \frac{\partial A_x}{\partial y}
    \right)\,dx\,dy
\end{align}
if we go around the path counterclockwise.
Notice this is the same form as before.

Let us apply this to the two line integrals in Equation~\ref{eqn:1}.
For the first integral,
set $A_x = v$ and $A_y = -v$.
Notice that by the Cauchy-Riemann condition,
\begin{align}
    \oint\left(
        u\,dx - v\,dy
    \right)
    =
    \int\left(
    -\frac{\partial v}{\partial x}
    - \frac{\partial u}{\partial y}
    \right)\,dx\,dy
    =0.
\end{align}
For the second integral, set
$A_x = v$ and $A_y = u$.
Then by the other Cauchy-Riemann condition,
\begin{align}
    \oint\left(
        v\,dx + u\,dy
    \right)
    =
    \int\left(
    \frac{\partial u}{\partial x}
    - \frac{\partial v}{\partial y}
    \right)\,dx\,dy
    =0.
\end{align}
Hence
\begin{align}
    \oint_C f(z)\,dz = 0
\end{align}
Suppose $C$ is made up of two paths $C_1$ and $C_2$.
\begin{align}
    \int_{C_1} f(z)\,dz
    - \int_{C_2} f(z)\,dz
    = \oint_C f(z)\,dz
    = 0.
\end{align}
Hence
\begin{align}
    \int_{C_1}f(z)\,dz
    = \int_{C_2}f(z)\,dz
\end{align}
What Cauchy is saying is that this integral is actually a function of the final
point for fixed $A$
\begin{align}
    \int_{A}^{P}f(z)\,dz = F(P).
\end{align}

\section{Cauchy's Integral Formula}
Cauchy is saying if I know the values of an analytic function on the boundary of
a simply connected region in $\mathbf{C}$,
then I can tell you the value of the function anywhere inside the region.

I was shocked when I first saw this theorem.
There's no analogue of this for real functions.

If you consider a complex function
\begin{align}
    f(z) = u(x, y) + i v(x, y).
\end{align}
Then you can solve Laplace's equation in 2D if you know the boundary.
\begin{align}
    \frac{\partial^2 u}{\partial x^2}
     + \frac{\partial^2 v}{\partial y^2}
     = 0.
\end{align}
That's the application we're interested in.

Cauchy's integral formula states that if $f(z)$ is analytic in a simply
connected domain $D$,
then for any point $z=a$ in $D$ and any closed path $C$ in $D$ which encloses
the point $a$,
we have
\begin{align}
    \oint_C \frac{f(z)}{z - a} \, dz = 2\pi i f(a)
\end{align}
where the integration is being taken in the counterclockwise sense.
If you know the values on the boundary then you know the value of any point
inside the region.
The reason it's possible is because $f$ is composed of harmonic functions.
In principle,
if you have the values on the boundary for Laplace's equation,
then in principle you have enough information to solve Laplace's equation in the
bulk.

Define
\begin{align}
    \phi(z) := \frac{f(z)}{z - a}.
\end{align}
This is analytic everywhere in $D$ except at $z=a$.
We need to evaluate
\begin{align}
    \oint_C \phi(z)\, dz.
\end{align}
Let us go counterclockwise.
Then around $a$ draw a small circle,
and I call that circle $C'$.
Then I make a cut between $C$ and $C'$.
Let there be a point $A$ on $C$ and $B$ on $C'$.
Make a cut along $AB$.
Integrate $\phi(z)$ along the closed path from $A$ to $C$ to $B$
around $C'$ and then back to $A$.
The contribution from the cut vanishes since we integrate first in one
direction and then again in the opposite direction.
\begin{align}
    \oint_C \phi(z)\, dz + \oint_{C'}\phi(z)\, dz = 0
\end{align}
where we are going counterclockwise in $C$ and clockwise in $C'$.
Then
\begin{align}
    \oint_{C}\phi(z)\,dz = \oint_{C'}\phi(z)\,dz
\end{align}
when both integrals are performed counterclockwise.
Along the circle $C'$,
we have
$z=a + \rho e^{i\theta}$.
Remember this is a circle and the differential is
\begin{align}
    dz = i\rho e^{i\theta} \,d\theta
\end{align}
So the integral is
\begin{align}
    \oint_{C}\phi(z)\,dz &= \oint_{C'}\phi(z)\,dz\\
    &=
    \oint_{C'}\frac{f(z)}{z - a} \,dz\\
    &= \int_{0}^{2\pi}\frac{f(z)}{\rho e^{i\theta}}\left(
        i\rho e^{i\theta}
    \right) \,d\theta\\
    &= i\int_{0}^{2\pi} f(z)\, d\theta
\end{align}
but because the circle is so small $f(z)\to f(a)$ and so
\begin{align}
    \oint_C \phi(z)\, dz
    = \oint \frac{f(z)}{z - a}\,dz
    = 2\pi i f(a).
\end{align}

The following is useful for integer $n$.
\begin{align}
    \oint \frac{dz}{{(z - z_0)}^n}
    =
    \begin{cases}
        2\pi i & \text{if } n = 1\\
        0 & \text{otherwise}
    \end{cases}
\end{align}
You can do this by using $z = z_0 + \rho e^{i\theta}$
and doing the integral explicitly on a circle.
If it's not a circle,
use the same trick with the cut like how we proved the Cauchy integral formula.

\section{Liouville's theorem}
Every analytic function is either constant or blows up somewhere in the complex
plane.
\begin{theorem}
    If $f(z)$ is analytic and bounded in absolute value in the entire complex
    plane,
    then it must be a constant.
\end{theorem}
\begin{proof}
    Assume $|f(z)|$ is bounded,
    so that $|f(z)| < M$
    for all $z\in\mathbb{C}$.
    Since $f(z)$ is analytic,
    it can be expanded like
    \begin{align}
        f(z) &= \sum_{n=0}^{\infty} a_n z^{n}.
    \end{align}
    By the Cauchy integral formula,
    \begin{align}
        |a_n|
        &= \left|\frac{1}{2\pi i} \oint \frac{f(z)}{z^{n + 1}}\, dz\right|\\
        &=
        \left|
            \sum_\gamma \frac{1}{2\pi i}
            \frac{f(z)}{z_{\gamma}^{n + 1}}
            \Dlta z_\gamma
        \right|\\
        &\le
        \sum_{\gamma}
        \left|\frac{1}{2\pi i}\right|
        \left|\frac{f(z_\gamma)}{z_\gamma^{n + 1}}\right|
        \left|\Delta z_\gamma \right|\\
        &\le
        \sum_{\gamma}
        \frac{1}{2\pi}
        \frac{f(z)|_{\max}}{R^{n + 1}} |\Delta z_\gamma|\\
        &\le \frac{1}{2\pi} \frac{f(z)|_{\max}}{R^{n + 1}} 2\pi R
    \end{align}
    where the contour is a circle centred at the origin and
    we have used the identities
    \begin{align}
        |ab| &= |a||b|\\
        |a + b| &\le |a| + |b|
    \end{align}
    Hence
    \begin{align}
        |a_n| \le \frac{M}{R^n}
    \end{align}
    for some constant $M$.
    We can take $R$ to be arbitrarily large.
    As $R\to\infty$, $a_n\to 0$ for $n\ge 1$.
    Hence $f(z)=a_0$,
    which is a constant.
\end{proof}


\section{ReLU Fourier representation}\label{sec:error2}
Rather than using general Fourier transform  to represent
$e^{i\omega\cdot x}$ in terms of $\sigma(\omega\cdot x+b)$,
\cite{klusowski2016uniform} gave a different method to represent
$e^{i\omega\cdot x}$ in terms of $(\omega\cdot x+b)_+^k$  for $k=1$
and $2$.   The following lemma gives a generalization of this
representation for all $k\ge 0$.
\begin{lemma}\label{lm:talorcomplex}
For any $k\ge0$ and $x\in \Omega$,
\begin{equation}
e^{i\omega\cdot x} =\sum_{j=0}^k{(i\omega\cdot x)^{j}\over j!}
+
{i^{k+1}\over k!} \|\omega\|^{k+1}\int_{0}^T\left[(\bar \omega\cdot x - t)_+^ke^{i\|\omega\|t}
+(-1)^{k-1}(-\bar \omega\cdot x - t)_+^ke^{-i\|\omega\|t} \right]dt.
\end{equation}
\end{lemma}
\begin{proof}
For $|z|\leq c$, by the Taylor expansion with integral remainder,
\begin{equation}
e^{iz} = \sum_{j=0}^k {(iz)^j\over j!} + {i^{k+1}\over k!} \int_0^z e^{iu}(z-u)^kdu.
\end{equation}
Note that
$$
(z-u)^k=(z-u)^k_+ - (u-z)^k_+.
$$
It follows that
\begin{equation}
\begin{split}
\int_{0}^z (z-u)^ke^{iu} du=&\int_{0}^z (z-u)_+^ke^{iu} du + \int_{0}^z (-1)^k(u-z)_+^ke^{iu} du
\\
=&\int_{0}^z (z-u)_+^ke^{iu} du + \int_{0}^{-z} (-1)^{k-1}(-u-z)_+^ke^{-iu} du
\\
=&\int_{0}^c (z-u)_+^ke^{iu} du + (-1)^{k-1}(-u-z)_+^ke^{-iu} du.
\end{split}
\end{equation}
Thus,
\begin{equation}
e^{iz} - \sum_{j=0}^k{(iz)^{j}\over j!}
=
{i^{k+1}\over k!}\int_{0}^c\left[(z - u)_+^ke^{iu} + (-1)^{k-1}(-z - u)_+^ke^{-iu} \right]du.
\end{equation}
Let
\begin{equation}\label{baromega}
z=\omega\cdot x,\quad u=\|\omega\|t,\quad \bar \omega={\omega\over \|\omega\|}.
\end{equation}
Since $\|x\| \le T$ and $|\bar \omega \cdot x|\le T$, we obtain
\begin{equation}
e^{i\omega\cdot x} - \sum_{j=0}^k{(i\omega\cdot x)^{j}\over j!}
=
{i^{k+1}\over k!} \|\omega\|^{k+1}\int_{0}^T\left[(\bar \omega\cdot x - t)_+^ke^{i\|\omega\|t}
+(-1)^{k-1}(-\bar \omega\cdot x - t)_+^ke^{-i\|\omega\|t} \right]dt,
\end{equation}
which completes the proof.
\end{proof}

Since $
u(x) = {1\over (2\pi)^d}\int_{\mathbb{R}^d} e^{i\omega\cdot x}\hat{u}(\omega)d\omega
$
and
$
 \partial^\alpha u(x)=\int_{\mathbb{R}^d} i^{|\alpha|}\omega^\alpha e^{i\omega\cdot x}\hat{u}(\omega)d\omega,
$
\begin{eqnarray}
 \partial^\alpha u(0)x^\alpha=\int_{\mathbb{R}^d} i^{|\alpha|}\omega^\alpha x^\alpha\hat{u}(\omega)d\omega.
\end{eqnarray}
Note that $\displaystyle (\omega\cdot x)^j=\sum_{|\alpha|=j}{j!\over \alpha !}\omega^\alpha x^\alpha $. It follows that
\begin{equation}
\sum_{|\alpha|=j}{1\over \alpha!} \partial^\alpha u(0) x^\alpha=i^j\sum_{|\alpha|=j}{1\over \alpha!} \int_{\mathbb{R}^d} \omega^\alpha x^\alpha\hat{u}(\omega)d\omega
={1\over j!}  \int_{\mathbb{R}^d} (i\omega\cdot x)^j \hat{u}(\omega)d\omega.
\end{equation}
Let $\hat{u}(\omega)=|\hat{u}(\omega)|e^{ib(\omega)}$. Then, $e^{i\|\omega\|t}\hat{u}(\omega) = |\hat{u}(\omega)|e^{i(\|\omega\|t + b(\omega))}$.
By Lemma \ref{lm:talorcomplex},
\begin{equation}\label{eq:fftaylor}
\begin{split}
&u(x) - \sum_{|\alpha|\le k}{1\over \alpha!} \partial^\alpha u(0) x^\alpha
\\
= &\int_{\mathbb{R}^d} \big (e^{i\omega\cdot x}-\sum_{j=0}^k{1\over j!}(i\omega\cdot x)^j\big )\hat{u}(\omega)d\omega.
\\
=&{\rm Re} \bigg ({i^{k+1}\over k!}\int_{\mathbb{R}^d} \int_{0}^T\left[(\bar \omega\cdot x - t)_+^ke^{i\|\omega\|t}
+(-1)^{k-1}(-\bar \omega\cdot x - t)_+^ke^{-i\|\omega\|_{\ell_1}t} \right]\hat{u}(\omega)\|\omega\|^{k+1}dt d\omega\bigg )
\\
=& {1\over k!}\int_{\{-1,1\}}\int_{\mathbb{R}^d} \int_{0}^T (z\bar \omega\cdot x - t)_+^k s(zt,\omega)  |\hat{u}(\omega)|\|\omega\|^{k+1}dtd\omega dz
\end{split}
\end{equation}
with $\int_{\{-1, 1\}} r(z) dz = r(-1) + r(1)$ and
\begin{equation}
s(zt,\omega)=
\begin{cases}
(-1)^{k+1\over 2}\cos(z\|\omega\|t + b(\omega)) & k \text{ is odd},
\\
(-1)^{k+2\over 2}\sin(z\|\omega\|t + b(\omega)) & k \text{ is even}.
\end{cases}
\end{equation}
Define $G=\{-1,1\}\times [0,T]\times \mathbb{R}^{d}$, $\theta=(z, t, \omega)\in G$,
\begin{equation}\label{eq:straglam}
g(x,\theta)= (z\bar \omega\cdot x - t)_+^k {\rm sgn} s(zt,\omega),\qquad  \rho(\theta) = {1\over (2\pi)^d}|s(zt,\omega)||\hat{u}(\omega)|\|\omega\|^{k+1},\quad \lambda(\theta)={\rho(\theta)\over
\|\rho\|_{L^1(G)}}.
\end{equation}

Then \eqref{eq:fftaylor} can be written as
\begin{equation}
u(x) = \sum_{|\alpha|\le k}{1\over \alpha!}D^\alpha u(0) x^\alpha
+ {\nu\over k!}\int_G  g(x, \theta)\lambda(\theta)d\theta,
\end{equation}
with $\nu=\int_G \rho(\theta)d\theta$. In summary, we have the following lemma.

\begin{lemma}\label{lm:probabilityexpan}
It holds that
\begin{equation}\label{ReLUm}
u(x) = \sum_{|\alpha|\le k}{1\over \alpha!} \partial^\alpha u(0) x^\alpha
+ {\nu\over k!}r_k(x),\qquad x\in \Omega
\end{equation}
with $\nu=\int_G \rho(\theta)d\theta$ and
\begin{equation}\label{ReLUrm}
r_k(x) = \int_G  g(x, \theta)\lambda(\theta)d\theta,\qquad G=\{-1,1\}\times [0,T]\times \mathbb{R}^{d},
\end{equation}
and  $g(x,\theta)$, $\rho(\theta)$  and $\lambda(\theta)$ defined in \eqref{eq:straglam}.
\end{lemma}

According to \eqref{eq:straglam}, the main ingredient $(z\bar
\omega\cdot x - t)_+^k$ of $g(x,\theta)$ only includes the direction
$\bar\omega$ of $\omega$ which belongs to a bounded domain
$\mathbb{S}^{d-1}$. Thanks to the continuity of $(z\bar \omega\cdot x
- t)_+^k$ with respect to $(z, \bar\omega, t)$ and the boundedness of
$\mathbb{S}^{d-1}$, the application of the stratified sampling to the
residual term of the Taylor expansion leads to the
approximation property in Theorem \ref{est:stratify}.

\begin{theorem}\label{est:stratify}
Assume $u\in B^{k+1}(\Omega)$
%$$
% \int_{\mathbb{R}^{d}} |\hat{f}(\omega)|\|\omega\|^{m+1} d\omega<\infty.
%$$
There exist $\beta_j\in [-1, 1]$, $\|\bar \omega_j\|=1$, $t_j\in [0,T]$ such that
\begin{equation}
u_N(x)= \sum_{|\alpha|\le k}{1\over \alpha!} \partial^\alpha u(0) x^\alpha + {2\nu\over k!N}\sum_{j=1}^{N}\beta_j (\bar \omega_j\cdot x - t_j)_+^k
\end{equation}
with $\nu=\int_G \rho(\theta)d\theta$ and $\rho(\theta)$  defined in \eqref{eq:straglam}
satisfies the following estimate
\begin{equation}
\|u - u_N \|_{H^m(\Omega)} \lesssim
\begin{cases}
 N^{-{1\over 2}-{1\over d}}\|u\|_{B^{k+1}(\Omega)},&m< k,
\\
N^{-{1\over 2}}\|u\|_{B^{k+1}(\Omega)}& m=k.
\end{cases}
\end{equation}
%Especially,
%\begin{equation}
%\|u - u_N \|_{L^2(\Omega)} \leq {(2T)^k|\Omega|^{1\over 2}\over (k-1)!} N^{-{1\over 2}-{1\over d}}\|u\|_{\mathcal B^{k+1, q}(\Omega)}.
%\end{equation}
%\begin{equation}
%\|D^\beta (f(x)- f_n(x))\|_{L^2(\Omega)}\le \sqrt{2^{m-k-2}(2m-k)\over k!(m-k)!}|\Omega|^{1/2} n^{-{1\over 2}-{1\over d}},\quad |\beta|=k\le m.
%\end{equation}
\end{theorem}

\begin{proof}
Let
$$
u_N(x)=  \sum_{|\alpha|\le k}{1\over \alpha!} \partial^\alpha u(0) x^\alpha + {\nu\over k!} r_{k,N}(x), \qquad r_{k,N}(x)={1\over N}\sum_{j=1}^{N}\beta_j (\bar \omega_j\cdot x - t_j)_+^k.
$$
Recall  the representation  of $u(x)$ in \eqref{ReLUm} and $r_k(x)$ in \eqref{ReLUrm}. It holds that
\begin{equation}
u(x) - u_N(x)={2\nu\over k!} (r_k(x) - r_{k,N}(x)).
\end{equation}
By Lemma \ref{lem:stratifiedapprox}, for any decomposition $\displaystyle G=\cup_{i=1}^N G_i$, there exist $\{\theta_i\}_{i=1}^N$ and $\{\beta_i\}_{i=1}^N\in [0, 1]$ such that
\begin{equation}
\| \partial_x^\alpha (u - u_N)\|_{L^2(\Omega)} = {\nu\over k!}\|  \partial_x^\alpha (r_k - r_{k,N})\|_{L^2(\Omega)} \leq {1\over k!N^{1/2}}\max_{1\le j\le n}\sup_{\theta_{j},\theta_{j}'\in G_j} \|  \partial_x^\alpha \big(g(x,\theta_j) - g(x,\theta_j')\big)\|_{L^2(\Omega)}.
\end{equation}
\iffalse
Consider a $\epsilon$-covering decomposition $G=\cup_{i=1}^M G_i$ as follows.
The variable $z$ is in the set $\{-1,1\}$, which can be divided into two subsets $\{-1\}$ and $\{1\}$.
Given a positive integer $N$, for the random variable $t$, the interval  $ [0,T]$ can be divided into $n_t$ subintervals $\{G_i^t\}_{i=1}^{n_t}$ such that
$$
|t-t'|<{1\over 2}N^{-{1\over d}}\quad t,t'\in G_i^t,\quad 1\leq i\leq n_t
$$
for $n_t>2\lceil T N^{1\over d}\rceil$.
For variable $\bar \omega=\omega/\|\omega\|\in \mathbb{S}^{d-1}$ where $\mathbb{S}^{d-1}=\{\bar \omega\in \mathbb{R}^d: \|\bar \omega\|=1\}$. Note that $\mathbb{S}^{d-1}$ can be divided into $n_\alpha$ subdomains $\{G_i^s \}_{i=1}^{n_s}$ such that
$$
\|\bar \omega- \bar \omega'\|\leq {1\over 2}N^{-{1\over d}}\qquad \bar \omega, \bar \omega' \in G_i^s,\quad 1\leq i\leq n_s
$$
for $(2N^{1\over d})^{d-1}\leq n_s\leq \lceil (5N^{1\over d})^{d-1}\rceil$ \cite{klusowski2016uniform}.
Then
$$
G=\displaystyle \cup \{G_{ijk\ell}: 1\leq i\leq 2,\ 1\leq j\leq n_t,\ 1\leq k\leq n_s,\ 1\le \ell\le 2\}
$$
with
\begin{equation}
G_{ijk\ell} = \{(z, t, \omega): z=(-1)^i,\ t\in G_j^t, \bar \omega \in G_k^s,\ {\rm sgn} s(zt,\omega)=(-1)^\ell\}.
\end{equation}
Denote this decomposition of $G$ by $G=\cup_{i=1}^{M} G_i$ with $M=4n_sn_t\le 2^{d}N$. For each $G_i$,
\fi
Consider a $\epsilon$-covering decomposition $G=\cup_{i=1}^N G_i$  such that
\begin{equation}
z=z',\ |t-t'|<\epsilon,\ \|\bar \omega  - \bar \omega'\|_{\ell^1}<\epsilon\qquad \forall \theta=(z, t, \omega),\ \theta'=(z', t', \omega')\in G_i
\end{equation}
where $\bar\omega$ is defined in \eqref{baromega}.
For any $\theta_i, \theta'_i\in G_i$,
$$
| \partial_x^\alpha \big (g(x,\theta_i) - g(x,\theta_i')\big )| = {k!\over (k-|\alpha|)!} | g_\alpha(x, \bar\omega, t) -  g_\alpha(x, \bar\omega', t')|
$$
with
\begin{equation}
 g_\alpha(x, \bar\omega, t)  = (z\bar \omega\cdot x-t)^{k-|\alpha|}_+\bar \omega^\alpha.
 \end{equation}
 Since
$$
|\partial_{\bar\omega_i}  g_\alpha|\le (2T)^{m-|\alpha|-1}\big ((k-|\alpha|)x_i + 2T\alpha_i\big ), \qquad |\partial_t  g_\alpha|\le (k-|\alpha|)(2T)^{k-|\alpha|-1},
$$
it follows that
\begin{equation}
\big | \partial_x^\alpha \big (g(x,\theta_i) - g(x,\theta_i')\big )\big | \le {k!\over (k-|\alpha|)!}(2T)^{k-|\alpha|-1}   \bigg ( (k-|\alpha|)(|x|_{\ell_1}+1) + 2T|\alpha |\bigg ) \epsilon.
\end{equation}
Thus, by Lemma \ref{lem:stratifiedapprox}, if $m=|\alpha|<k$,
\begin{equation}
\|  \partial_x^\alpha (u - u_N)\|_{L^2(\Omega)} \le {|\Omega|^{1/2}\over (k-|\alpha|)!}(2T)^{k-|\alpha|-1}   \bigg ( (k-|\alpha|)(T+1) + 2T|\alpha |\bigg )N^{-{1\over 2}}\epsilon.
\end{equation}
Note that $\epsilon \sim N^{-{1\over d}}$. There exist $\theta_{i,j}$ such that for any $0\le k< m$,
\begin{equation}
\| u - u_N\|_{H^k(\Omega)} \le  C(m,k,\Omega)\nu N^{-{1\over 2}-{1\over d}}
\end{equation}
with $\nu\le \|u\|_{B^{k+1}(\Omega)}$ and
\begin{equation}\label{equ:defcmko}
C(m,k,\Omega)=|\Omega|^{1/2}\bigg (\sum_{|\alpha|\le k}{1\over (k-|\alpha|)!}(2T)^{k-|\alpha|-1}   \big ( (k-|\alpha|)(T+1) + 2T|\alpha |\big )\bigg )^{1/2}.
\end{equation}
If $m=|\alpha|=k$,
$$
\max_{1\le j\le M}\sup_{\theta_{j},\theta_{j}'\in G_j} \| D_x^\alpha \big(g(x,\theta_j) - g(x,\theta_j')\big)\|_{L^2(\Omega)}\lesssim 1.
$$
This leads to
\begin{equation}
\| u - u_N\|_{H^m(\Omega)} \le  C(m,k,\Omega)\nu N^{-{1\over 2}}\quad \mbox{for }\ k=m.
\end{equation}
Note that $u_N$ defined above can be written as
$$
u_N(x)=  \sum_{|\alpha|\le k}{1\over \alpha!} \partial^\alpha u(0) x^\alpha + {1\over k!N}\sum_{j=1}^{N}\beta_j (\bar \omega_j\cdot x - t_j)_+^k
$$
with $\beta_j\in [-1, 1]$,
which completes the proof.
\end{proof}

\begin{lemma}
There exist $\alpha_i$, $\omega_i$, $b_i$ and $N\le 2\begin{pmatrix} k+d\\k\end{pmatrix}$
such that
$$
 \sum_{|\alpha|\le m}{1\over \alpha!} \partial^\alpha u(0) x^\alpha = \sum_{i=1}^N\alpha_i (\omega_i\cdot x + b_i)_+^k
$$
with $
x^\alpha = x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d},\quad \alpha!=\alpha_1!\alpha_2!\cdots \alpha_d!.
$
\end{lemma}
The above result can be found in \cite{he2020preprint}

A combination of Theorem \ref{est:stratify} and the above the lemma gives the following estimate in Theorem \ref{th:stra}.
\begin{theorem} \label{th:stra}
Suppose $u\in B^{k+1}(\Omega)$.
There exist $\beta_j, t\in \mathbb{R}$, $\omega_j \in \mathbb{R}^d$ such that
\begin{equation}
u_N(x)= \sum_{j=1}^{N}\beta_j (\bar \omega_j\cdot x - t_j)_+^k
\end{equation}
satisfies the following estimate
\begin{equation}\label{d}
\|u- u_N \|_{H^m(\Omega)} \lesssim
\begin{cases}
N^{-{1\over 2}-{1\over d}}\|u\|_{B^{k+1}(\Omega)},\qquad k> m,
\\
N^{-{1\over 2}}\|u\|_{B^{k+1}(\Omega)},\qquad k= m,
\end{cases}
\end{equation}
where $\bar\omega$ is defined in \eqref{baromega}.
\end{theorem}

\begin{remark}
We make the following comparisons:
\begin{enumerate}
\item The results in \ref{sec:Bsplines} are for activation functions $\sigma=b_k$, while the results in Section \ref{sec:error2} are for activation functions $\sigma={\rm ReLU}^k$.
\item By \eqref{splinetorelu}, the following relation obviously holds
$$
V_N(b_k)\subset V_{N+k}({\rm ReLU}^k),
$$
where
\begin{equation}
\label{VkN}
V_{N+k}({\rm ReLU}^k)=\left\{\sum_{i=1}^Na_i(w_i\cdot x+b_i)_+^k, a_i, b_i\in\mathbb R^1, w_i\in \mathbb R^{1\times d}\right\},
\end{equation}
and $V_N(b_k)$ is the one hidden layer neuron network
function class with activation function $b_k$.  Thus, asymptotically
speaking, the results that hold for $\sigma=b_k$ also hold for
$\sigma={\rm ReLU}^k$.
\end{enumerate}
\end{remark}


%!TEX root = ../main.tex
\documentclass[../main.tex]{subfiles}
\begin{document}
\subsection{First and Second Order Conditions of the Utility Maximization Problem}\label{app:foc_soc_utility}
Starting from the utility function in equation (\ref{eq:two_task_utility}), this section presents the first and second derivative of the utility function $u_{i,t}$ w.r.t. choice parameter $\lambda_{i,t}$. Based on these results, the first and second order conditions for a global maximum of the utility function can be identified.\\
\begin{align}
	\intertext{Starting from the utility function in a setting with $J=2$ tasks.}
	u_{i,t} &= \lambda_{i,t} \tilde{w}_{i,t} - 2\theta|b_i - \lambda_{i,t}|^\phi \tag{\ref{eq:two_task_utility}}\\
	\intertext{Calculate the first derivative of utility w.r.t. task choice parameter $\lambda_{i,t}$ by application of the chain rule.}
	\frac{\partial u_{i,t}}{\partial \lambda_{i,t}} &= \tilde{w}_{i,t} - 2\theta \phi |b_i - \lambda_{i,t}|^{\phi-1} \frac{\partial}{\partial \lambda_{i,t}} [|b_i - \lambda_{i,t}|^\phi] \nonumber \\
	\intertext{Applying the derivation rules for absolute value functions:}
	{} &= \tilde{w}_{i,t} + 2\theta \phi |b_i - \lambda_{i,t}|^{\phi-1} \frac{b_i - \lambda_{i,t}}{|b_i - \lambda_{i,t}|} \nonumber \\
	\intertext{Finally, after some rearrangements:}
	{} &= \tilde{w}_{i,t} + 2\theta \phi \frac{|b_i - \lambda_{i,t}|^\phi}{b_i - \lambda_{i,t}} \label{eq:first_derivative}
	\intertext{Notice that this derivative is not defined at $b_i = \lambda_{i,t}$.}
	\intertext{Continuing from equation (\ref{eq:first_derivative}), now calculate the second derivative of the utility function w.r.t. task choice parameter $\lambda_{i,t}$ by application of the division rule.}
	\frac{\partial^2 u_{i,t}}{\partial \lambda_{i,t}^2} &= 2\theta \phi \left(\frac{\frac{\partial }{\partial \lambda_{i,t}} [|b_i - \lambda_{t,i}|^\phi] (b_i - \lambda_{i,t}) - |b_i - \lambda_{i,t}|^\phi \frac{\partial}{\partial \lambda_{i,t}} [b_i - \lambda_{i,t}]}{(b_i - \lambda_{i,t})^2} \right) \nonumber \\
	\intertext{By application of the chain rule and derivation rules for absolute value functions:}
	{} &= 2 \theta \phi \left(\frac{- \phi |b_i - \lambda_{i,t}|^\phi + |b_i - \lambda_{i,t}|^\phi}{(b_i - \lambda_{i,t})^2} \right) \nonumber \\
	\intertext{Rearranging:}
	{} &= 2 \theta \phi (1 - \phi) \frac{|b_i- \lambda_{i,t}|^\phi}{(b_i - \lambda_{i,t})^2} \label{eq:second_derivative}
\end{align}
From equation (\ref{eq:second_derivative}) it can be seen immediately that the second derivative of the utility function w.r.t. task choices $\lambda_{i,t}$ is (strictly) positive for penalty exponents $\phi$ (strictly) larger than one, resulting in a (strictly) concave utility function. Analogously, penalty exponents (strictly) smaller than one will result in a (strictly) convex utility function.\\
Similarly to the first derivative, the second derivative is not defined at $b_i = \lambda_{i,t}$.

\end{document}


\section{Entropy Regularization of Optimal Transport}
\label{sec:3}
\subsection{Background and Formulation}
\subsubsection{Kantorovich formulation of optimal transport}
\cite{COT} introduces the Kantorovich formulation of optimal transport (Kantorovich's relaxation). For two vectors $\mu\in\mbR^m$ and $\nu\in\mbR^n$, which satisfies $\mbone_n^T\mu=\mbone_m^T\nu$, we consider the following set of coupling matrices
\begin{equation}
U(\mu, \nu) \triangleq \left\{\pi\in\mbR^{m\times n}_+: \pi\mbone_n = \mu, \pi^T\mbone_m=\nu \right\}
\end{equation}
where we use the following matrix-vector notation
$$
\pi\mbone_n  = (\sum_j \pi_{i,j})_i\in\mbR^m, \quad \pi\mbone_m  = (\sum_i \pi_{i,j})_j\in\mbR^n,
$$

The Kantorovich's optimal transport problem reads
\begin{equation}
\label{equ:kot}
L_\mfc(\mu,\nu)\triangleq \min\limits_{\pi\in U(\mu, \nu)}\la C,\pi\ra\triangleq\sum_{i,j}C_{i,j}\pi_{i,j}
\end{equation}
Therefore, problem \ref{equ:kot} is equivalent to the standard form of LP \ref{lp}.

\subsubsection{Entropy regularization}
For a coupling matrix $\pi \in U(\mu, \nu)$, the discrete entropy of $\pi$ is defined as
\begin{equation}
H(\pi)\triangleq-\sum_{i,j}\pi_{i,j}(\log(\pi_{i,j})-1)
\end{equation}
with an analogous definition for vectors, with the convention that $H(\pi) = -\infty$ if one of the entries $\pi_{i,j}$ is negative. Although discrete entropy is defined on matrices with positive entry, with $\lim\limits_{x\to0+}x(\log x-1)=0$, we can extend its definition to  matrices with non-negative entry. And we have
$$
H(\pi)=-\sum_{\pi_{i,j}\geqslant0}\pi_{i,j}(\log(\pi_{i,j})-1)
$$
Suppose $M=\min\{\|\mu\|_\infty, \|\nu\|_\infty\}$, then we have $\pi_{i,j}\leqslant\min\{\mu_i, \nu_j\}\leqslant M$. The function $H(\pi)$ is then $1/M$-strongly convex, because its Hessian is $\p^2H(\pi)=-\diag(1/\pi_{i,j})$.

The idea of entropic regularization of optimal transport is to use $-H$ as a regularizing function to obtain approximate solutions to the Kantorovich's optimal transport problem \ref{equ:kot}.

\begin{equation}
\label{equ:erot}
L^\epsilon_C(\mu,\nu)\triangleq \min\limits_{\pi\in U(\mu, \nu)}\la C,\pi\ra-\epsilon H(\pi)
\end{equation}

The convergence of the solution of that regularized problem towards an optimal solution of the original linear program has been studied by \cite{aaote}.
\begin{proposition}[Convergence with $\epsilon$]
\label{sh_cvg}
The unique solution $\pi^\epsilon$ of \ref{equ:erot} converges to the optimal solution with maximal entropy within the set of all optimal solutions of the Kantorovich problem, namely
\begin{equation}
\label{sh_limit}
\pi^\epsilon\to\arg\min\limits_\pi\{-H(\pi): \pi\in U(\mu,\nu), \la\pi, C\ra=L_C(\mu,\nu)\}, \quad \epsilon\to0
\end{equation}
so that in particular
$$
L^\epsilon_C(\mu,\nu)\to L_C(\mu,\nu), \quad \epsilon\to0
$$
\end{proposition}
\begin{proof}
We consider a sequence $\{\epsilon_l\}$ such that $\epsilon_l\to0$ and $\epsilon_l>0$. We denote $\pi^l$ the solution to \ref{equ:erot} for $\epsilon=\epsilon_l$. Since $U(\mu,\nu)$ is bounded, we can extract a sequence (that we do not relabel for sake of simplicity) such that $\pi^l\to\pi^*$. Since $U(\mu, \nu)$ is closed, $\pi^*\in U(\mu,\nu)$. We consider any $\pi$ such that $\la C,\pi\ra=L_C(\mu,\nu)$. By the optimality of $\pi$ and $\pi^l$ for their respective optimization problem, we have
\begin{equation}
0\leqslant\la C, \pi^l\ra-\la C, \pi\ra\leqslant\epsilon_l(H(\pi^l)-H(\pi))
\end{equation}
Since $H$ is continuous, taking the limit $l\to\infty$ in this expression shows that $\la C, \pi^*\ra=\la C, \pi\ra$ so that $\pi^*$ is a feasible point of \ref{sh_limit}. Furthermore, we have $H(P^*)\geqslant H(P)$, which shows that $P^*$ is the solution to \ref{sh_limit}. By strict convexity of $-H$, the solution is unique and the whole sequence is converging.
\end{proof}

The following figures \ref{gmm1} and \ref{gmm2} illustrates the impact of $\epsilon$ on the coupling $\pi$ between two Gaussian mixture models.
\begin{figure}[!htb]
\centering
\includegraphics[width=12cm]{final/gmm1.png}
\caption{The probability distribution function of two Gaussian mixture models $G_1$ and $G_2$. $G_1$ (blue one) has mean $[0.3,0.5]$, variance $[0.05^2, 0.03^2]$ and component proportion $[0.5, 0.5]$; $G_2$ (red one) has mean $[0.6,0.7]$, variance $[0.03^2, 0.05^2]$ and component proportion $[0.6, 0.4]$. }
\label{gmm1}
\end{figure}

\begin{figure}[!htb]
\centering
\includegraphics[width=12cm]{final/gmm2.png}
\caption{Impact of $\epsilon$ on the coupling between two gaussian mixture models \ref{gmm1}. From left to right, $\epsilon=1,0.1,0.01,0.001$. The purple one is the solution to the entropy regularized LP. The green one is the solution to the original LP.}
\label{gmm2}
\end{figure}

Define the Kullback-Leibler divergence between couplings as
$$
\mathrm{KL}(\pi|K)\triangleq\sum_{i,j}\pi_{i,j}\log(\frac{\pi_{i,j}}{K_{i,j}})-\pi_{i,j}+K_{i,j}
$$
We take $K^\epsilon_{i,j}\triangleq e^{-\frac{C_{i,j}}{\epsilon}}$, the Gibbs kernel associated to the cost matrix $C$. The unique solution $\pi^\epsilon$ of \ref{equ:erot} is a projection onto $U(\mu,\nu)$, namely
$$
\pi^\epsilon\triangleq\arg\min\limits_{\pi\in U(\mu,\nu)}\mathrm{KL}(\pi|K^\epsilon)
$$

\subsubsection{Formulation of the problem}
The optimization problem of the entropic regularization of optimal transport with parameter $\epsilon$ is given by:
\begin{equation}
\label{equ:kl}
\left\{
\begin{aligned}
&\min \, \la C,\pi\ra-\epsilon H(\pi)=\mathrm{KL}(\pi|K^\epsilon)\\
&\mathrm{s.t.}\, \pi\in U(\mu, \nu)
\end{aligned}\right.
\end{equation}
where $K^\epsilon_{i,j}= e^{-\frac{C_{i,j}}{\epsilon}}$.

\subsection{Sinkhorn's Algorithm}
We start with the following proposition.

\begin{proposition}
\label{sh_basic}
The solution to \ref{equ:kl} is unique and has the form
\begin{equation}
\label{equ:para}
\forall 1\leqslant i\leqslant m, 1\leqslant j\leqslant n, \quad \pi^\epsilon_{i,j}=a_iK^\epsilon_{i,j}b_j
\end{equation}
for two (unknown) scaling variable $(a,b)\in \mbR_+^m\times\mbR_+^n$.
\end{proposition}

\begin{proof}
Introducing two dual variable $f\in\mbR^m, g\in\mbR^n$ for each marginal constraint. The Lagrangian of \ref{equ:kl} reads
$$
L(\pi,f,g)=\la\pi,C\ra-\epsilon H(P)-\la f, \pi\mbone_n-\mu\ra-\la g, \pi^T\mbone_m-\nu\ra
$$
The first order conditions yield
$$
\frac{\p L}{\p \pi_{i,j}}=C_{i,j}+\epsilon log(P_{i,j})-f_i-g_j=0
$$
This means that an optimal $\pi$ coupling to the regularized problem can be written as $P_{i,j}=e^{f_i/\epsilon}e^{-C_{i,j}/\epsilon}e^{g_j/\epsilon}$. Let $a=e^{f_i/\epsilon}, b= e^{g_j/\epsilon}$, then we have $$
\pi_{i,j}=a_iK^\epsilon_{i,j}b_j
$$
\end{proof}

This shows that the solution of \ref{equ:kl} has a specific form, which can be parameterized using $m+n$ variables.

The parameterization of the optimal solution in equation \ref{equ:para} can be written in the matrix form $\pi^\epsilon = \diag(a)K^\epsilon\diag(b)$. Then, $a,b$ must satisfies:
\begin{equation}
\diag(a)K^\epsilon\diag(b)\mbone_m=\mu, \quad \diag(a)K^T\diag(b)\mbone_n=\nu
\end{equation}
These two equations can be further simplified as
\begin{equation}
a\odot (K^\epsilon b)=\mu, \quad b\odot\left((K^\epsilon)^Ta\right)=\nu
\end{equation}
where $\odot$ corresponds to entry-wise multiplication of vectors. The Sinkhorn's algorithm iteratively update $a^{(l+1)}$ and $b^{(l+1)}$ by
\begin{equation}
\label{upd_sh}
a^{(l+1)}=\frac{\mu}{K^\epsilon b^{(l)}}, \quad b^{l+1}=\frac{\nu}{(K^\epsilon)^Ta^{(l+1)}}
\end{equation}
initialized with an arbitrary positive vector $b^{(0)}=\mbone_n$. The division operator used above between two vectors is to be understood entry-wise. The whole algorithm is given by:

\begin{algorithm}
\caption{Sinkhorn's algorithm}
\label{alg:sa}
\begin{algorithmic}[1]
\REQUIRE $\epsilon$, $C\in\mbR^{m\times n}$, $\mu\in\mbR^m$, $\nu\in\mbR^n$.
\STATE Start with $b^{(0)}=\mbone_n$, $l=0$
\STATE Calculate $K^\epsilon_{i,j}= e^{-\frac{C_{i,j}}{\epsilon}}$
\WHILE {\text{not converge}}
	\STATE Update $a^{(l+1)}$ and $b^{(l+1)}$ by \ref{upd_sh}, $l=l+1$
\ENDWHILE
\STATE $a=a^{(l)}, b=b^{(l)}$
\RETURN $\pi=\diag(a)K^\epsilon \diag(b)$
\end{algorithmic}
\end{algorithm}

This algorithm was originally introduced with a proof of convergence by \cite{sinkhorn}. For simplicity, we assume $m=n$. \cite{nltaa} showed that by setting $\epsilon=\frac{\tau}{4\log(n)}$, $O(\|C\|_\infty^3\log(n)\tau^{-3})$ Sinkhorn iterations (with an additional
rounding step to compute a valid coupling $\hat\pi\in U(\mu,\nu)$) are enough to ensure $\la \hat \pi, C\ra\leqslant L_C(\mu,\nu)+\tau$. This implies that Sinkhorn computes a $\tau$-approximate solution of the unregularized OT problem in $O(n^2\log(n)\tau^{-3})$ operations. The rounding scheme consists in, given two vectors $a\in\mbR^m, \, b\in\mbR^n$ to carry out the following updates:
$$
a'=a\odot \min\left(\frac{\mu}{a\odot(K^\epsilon b)},\mbone_m\right), \quad b'=b\odot \min \left(\frac{\nu}{b\odot((K^\epsilon)^Ta')},\mbone_n\right)
$$
$$
\Delta_\mu = \mu-a'\odot(K^\epsilon b'), \quad \Delta_\nu=\nu-b'\odot\left((K^\epsilon)^Ta'\right)
$$
$$
\hat \pi = \diag(a') K^\epsilon\diag (b')+\Delta_\mu\Delta_\nu^T/\|\Delta_\mu\|_1
$$
This yields a matrix $\hat\pi\in U(\mu,\nu)$ such that the $1$-norm of $\hat \pi -\diag(a) K^\epsilon\diag (b) $ is controlled by the marginal violations of $\diag(a)K^\epsilon\diag(b)$, namely
$$
\|\hat \pi -\diag(a) K^\epsilon\diag (b')\|_1\leqslant\|\mu-a\odot(K^\epsilon b)\|_1+\|\nu-b\odot\left((K^\epsilon)^Ta\right)\|_1
$$
In actual implementation, this step is crucial to decrease the violation of constraints.

\subsection{Our proposed algorithm}
\subsubsection{Sinkhorn's algorithm with numerical stability and continuation strategy}
We denote $\|C\|_\infty=\max\limits_{i,j}|C_{i,j}|$. In actual implementation of Sinkhorn's algorithm \ref{alg:sa}, because we have $K^\epsilon_{i,j}=e^{-\frac{C_{i,j}}{\epsilon}}$, when $\epsilon$ is small, namely $\epsilon<\frac{\|C\|_\infty}{806}$, the largest entry of $K$ is smaller than $e^{-806}\approx 10^{-350}$. Most of the entry of $K^{\epsilon}$ would be rounded to $0$ because they are smaller than the smallest positive number that the computer can restore. If we want to calculate $K^\epsilon$, we would have a positive lower bound $\frac{\|C\|_\infty}{806}$ for $\epsilon$. This is not desired and it would make the result from proposition \ref{sh_cvg} meaningless.

We propose a numerical stable version of Sinkhorn's algorithm with no lower bound of $\epsilon$. A possible way to remove this lower bound of $\epsilon$ is to avoid directly calculating $K^\epsilon$ to update $a^{(l+1)}$ and $b^{(l+1)}$. Therefore, from the proof of proposition \ref{sh_basic}, we denote $a^{(l)}=e^{f^{(l)}/\epsilon}$, $b^{(l)}=e^{g^{(l)}/\epsilon}$, where $f^{(l)}\in R^m, g^{(l)}\in R^n$. Then, the update rule for $f^{(l+1)}$ and $g^{(l+1)}$ reads
\begin{equation}
\label{upd_sh_log}
f^{(l+1)}=\epsilon(\log(\mu)-\log(K^\epsilon e^{g^{(l)}/\epsilon})), \quad g^{(l+1)}=\epsilon(\log(\nu)-\log((K^\epsilon)^T e^{f^{(l+1)}/\epsilon})),
\end{equation}
Our motivation is to caculate $\log(K^\epsilon e^{g^{(l)}/\epsilon})$ and $\log((K^\epsilon)^T e^{f^{(l+1)}/\epsilon})$ in update rule \ref{upd_sh_log} in a numerically stable way. We start with the update of $f^{(l+1)}$. It is easy to verify that
$$
f^{(l+1)}_i = \epsilon \log(\mu_i)-\epsilon \log(\sum_{j=1}^ne^{(-C_{i,j}+g^{(l)}_j)/\epsilon})
$$
Directly calculating $\sum_{j=1}^ne^{(-C_{i,j}+g^{(l)}_j)/\epsilon}$ is dangerous because it might be rounded to $0$. Let $\hat f^{(l+1)}_i=\max_j\{-C_{i,j}+g^{(l)}_j\}$. Then, to update $f$ with numerical stability, we can update $f^{(l+1)}_i$ by
\begin{equation}
\label{sh_f_upd}
f^{(l+1)}_i = \epsilon \log(\mu_i)-\hat f^{(l)}_i-\epsilon \log(\sum_{j=1}^ne^{ (-C_{i,j}+g^{(l)}_j-\hat f^{(l)}_i)/\epsilon})
\end{equation}
Because $\sum_{j=1}^ne^{ (-C_{i,j}+g^{(l)}_j-\hat f^{(l)}_i)/\epsilon}\geqslant1$, to update $f^{(l+1)}$ in this way is safe. Actually, we use $\hat f_i^{(l)}/\epsilon+\log(\sum_{j=1}^ne^{ (-C_{i,j}+g^{(l)}_j-\hat f^{(l)}_i)/\epsilon})$ to calculate $\log(K^\epsilon e^{g^{(l)}/\epsilon})$.  Similarly, we let $\hat g^{(l+1)}_j=\max_i\{-C_{i,j}+f^{(l+1)}_i\}$, and update $g^{(l+1)}_j$ by
\begin{equation}
\label{sh_g_upd}
g^{(l+1)}_j = \epsilon \log(\nu_j)-\hat g^{(l+1)}_j-\epsilon \log(\sum_{i=1}^ne^{ (-C_{i,j}+f^{(l+1)}_i-\hat g^{(l+1)}_j)/\epsilon})
\end{equation}
In the end of the algorithm, we also use an additional rounding step to compute a valid coupling $\hat\pi$. Given two vectors $f\in\mbR^m, g\in\mbR^n$ and coupling matrix $\pi\in\mbR^{m\times n}$ with $\pi_{i,j}=e^{(f_i-C_{i,j}+g_j)/\epsilon}$, we compute $\hat\pi$ in the following way:
\begin{equation}
\label{add_coup}
\begin{aligned}
&\hat f_i=\max_j\{-C_{i,j}+g_j\}, \quad f'_i=f_i+\min\{\epsilon(\log(\mu_i)-\sum_{j=1}^ne^{ (-C_{i,j}+g_j-\hat f_i)/\epsilon})-\hat f_i-f_i,0\}\\
&\hat g_j=\max_i\{-C_{i,j}+f'_i\}, \quad g'_j=g_j+\min\{\epsilon(\log(\nu_j)-\sum_{i=1}^me^{ (-C_{i,j}+f'_i-\hat g_j)/\epsilon})-\hat g_j-g_j,0\}\\
&\hat f'_i=\max_j\{-C_{i,j}+g'_j\}, \quad (\Delta_\mu)_i=\mu_i-(\sum_{j=1}^ne^{ (-C_{i,j}+g'_j-\hat f'_i)/\epsilon})e^{(f'_i+\hat f'_i)/\epsilon}\\
&(\Delta_\nu)_j=\nu_j-(\sum_{i=1}^ne^{ (-C_{i,j}+f'_i-\hat g'_j)/\epsilon})e^{(g'_j+\hat g'_j)/\epsilon}, \quad \hat\pi=\pi+\Delta_\mu\Delta_\nu^T/\|\Delta_\mu\|_1
\end{aligned}
\end{equation}
Then, we get the following Sinkhorn's algorithm with numerical stability. It shares the same idea with Log-domain Sinkhorn's algorithm \cite{COT}.
\begin{algorithm}
\caption{Sinkhorn's algorithm with numerical stability}
\label{alg:sans}
\begin{algorithmic}[1]
\REQUIRE $\epsilon$, $C\in\mbR^{m\times n}$, $\mu\in\mbR^m$, $\nu\in\mbR^n$.
\STATE Start with $g^{(0)}=\mathbf{0}_n$, $l=0$
\WHILE {\text{not converge}}
	\STATE Calculate $\hat f^{(l+1)}_i=\max_j\{-C_{i,j}+g^{(l)}_j\}$ and update $f^{(l+1)}$ by \ref{sh_f_upd}
	\STATE Calculate $\hat g^{(l+1)}_i=\max_i\{-C_{i,j}+f^{(l+1)}_i\}$ and update $g^{(l+1)}$ by \ref{sh_g_upd}, $l=l+1$
\ENDWHILE
\STATE Let $f=f^{(l)}, g=g^{(l)}$, calculate $\pi_{i,j}=e^{(f_i-C_{i,j}+g_j)/\epsilon}$, and compute $\hat\pi$ by \ref{add_coup}
\RETURN $\hat \pi$
\end{algorithmic}
\end{algorithm}

We shall point out that $\epsilon>0$ is vital in the iteration \ref{sh_f_upd} and \ref{sh_g_upd} because $f^{(l)}$ and $g^{(l)}$ will converge to the optimal solution to the dual problem of \ref{equ:kl} by Proposition \ref{sh_basic}. If we let $\epsilon\to 0$ in \ref{sh_f_upd} and \ref{sh_g_upd}, we would find that these updates become as follows:
\begin{equation}
\label{upd_div}
f_i^{(l+1)}=-\hat f_i^{(l)}=\min_j\{C_{i,j}-g^{(l)}_j\}, \quad g_j^{(l+1)}=-\hat g^{(l+1)}_j = \max_i\{C_{i,j}-f^{(l+1)}_i\}
\end{equation}
Nevertheless, $f^{(l)}$ and $g^{(l)}$ in this iteration \ref{upd_div} will not converge to the solution to the dual problem of \ref{lp}. With $\epsilon=0$, we lose the strong convexity of the objective function and, therefore, the solution to the dual problem of \ref{lp} is not unique. As a result,  $f^{(l)}$ and $g^{(l)}$ may stuck in a plateau. This is also pointed out in Section 3.2 of \cite{COT}.

We can also apply continuation strategy in Sinkhorn's algorithm with numerical stability. We have three additional parameters $\epsilon_0, \alpha, M$ for continuation. Namely, we start with a large $\epsilon_0$ and $k=0$. Then, until $\epsilon_k=\epsilon$, we gradually decrease the value of $\epsilon_{k+1}=\max\{\alpha\epsilon_k,\epsilon \}$ and set $k=k+1$ after $M$ iterations. In each iteration, $f_i^{(l+1)}$ and $g_j^{(l+1)}$ are updated as follows:
\begin{equation}
\label{sh_f_upd_c}
f^{(l+1)}_i = \epsilon_k \log(\mu_i)-\hat f^{(l)}_i-\epsilon_k \log(\sum_{j=1}^ne^{ (-C_{i,j}+g^{(l)}_j-\hat f^{(l)}_i)/\epsilon_k})
\end{equation}
\begin{equation}
\label{sh_g_upd_c}
g^{(l+1)}_j = \epsilon_k \log(\nu_j)-\hat g^{(l+1)}_j-\epsilon_k \log(\sum_{i=1}^ne^{ (-C_{i,j}+f^{(l+1)}_i-\hat g^{(l+1)}_j)/\epsilon_k})
\end{equation}
In the end, we still use an additional rounding step to compute a valid coupling $\hat\pi$. The algorithm is given below:

\begin{algorithm}
\caption{Sinkhorn's algorithm with numerical stability and continuation strategy}
\label{alg:sansc}
\begin{algorithmic}[1]
\REQUIRE $\epsilon$, $C\in\mbR^{m\times n}$, $\mu\in\mbR^m$, $\nu\in\mbR^n$, continuation parameter $\epsilon_0, \alpha, M$.
\STATE Start with $g^{(0)}=\mathbf{0}_n$, $k=0$.
\WHILE{$\epsilon_k>\epsilon$}
	\FOR{$l=0:M-1$}
	\STATE Calculate $\hat f^{(l+1)}_i=\max_j\{-C_{i,j}+g^{(l)}_j\}$ and update $f^{(l+1)}$ by \ref{sh_f_upd_c}
	\STATE Calculate $\hat g^{(l+1)}_i=\max_i\{-C_{i,j}+f^{(l+1)}_i\}$ and update $g^{(l+1)}$ by \ref{sh_g_upd_c}
	\ENDFOR
	\STATE Set $\epsilon_{k+1}=\max\{\alpha\epsilon_k,\epsilon \}$, $f^{(0)}=f^{(M)}$, $g^{(0)}=g^{(M)}$ and $k=k+1$
\ENDWHILE
\FOR{$l=0:M-1$}
	\STATE Calculate $\hat f^{(l+1)}_i=\max_j\{-C_{i,j}+g^{(l)}_j\}$ and update $f^{(l+1)}$ by \ref{sh_f_upd_c}
	\STATE Calculate $\hat g^{(l+1)}_i=\max_i\{-C_{i,j}+f^{(l+1)}_i\}$ and update $g^{(l+1)}$ by \ref{sh_g_upd_c}
\ENDFOR
\STATE Let $f=f^{(l)}, g=g^{(l)}$, calculate $\pi_{i,j}=e^{(f_i-C_{i,j}+g_j)/\epsilon}$, and compute $\hat\pi$ by \ref{add_coup}
\RETURN $\hat \pi$
\end{algorithmic}
\end{algorithm}

\subsection{First-Order Method: ADMM}
To implement a first-order algorithm to solve Entropy Regularization of OT \ref{equ:kl}, we again choose ADMM. We reformulate the problem as follows,
\begin{eqt}
\min_{\pi, \tilde\pi}&\ \la C, \pi\ra - \epsilon H(\tilde{\pi}) \\
\text{s.t.}&\ \pi = \tilde{\pi}\in U(\mu, \nu)
\end{eqt}
which is, equivalently,
\begin{eqt}
\min_{\pi, \tilde\pi}\ &\sum_{i, j}C_{i, j}\pi_{i, j} + \epsilon \sum_{i, j}\tilde\pi_{i, j}(\log(\tilde\pi_{i, j}) - 1) \\
\text{s.t.}\ &\sum_{j}\pi_{i, j} = \mu_i,\ \sum_{i}\pi_{i, j} = \nu_j,\ \pi = \tilde{\pi}\geqslant 0.
\end{eqt}
The augmented Lagrangian function is
\begin{eqt}
& L_t(\pi, \tilde\pi, \gamma, \lambda, \omega) \\
=& \sum_{i, j}C_{i, j}\pi_{i, j} + \sum_{i}\gamma_i\left(\mu_i - \sum_j\pi_{i, j}\right) + \sum_j\lambda_j\left(\nu_j-\sum_i\pi_{i, j}\right) + \sum_{i, j}\omega_{i, j}(\tilde\pi_{i, j} - \pi_{i, j})\\
& +\frac{t}{2}\left(\sum_i\left(\mu_i-\sum_{j}\pi_{i, j}\right)^2+\sum_j\left(\nu_j-\sum_{i}\pi_{i, j}\right)^2+\sum_{i, j}(\pi_{i, j}-\tilde\pi_{i, j})^2\right) \\
&+\epsilon\sum_{i, j}\tilde\pi_{i, j}(\log(\tilde\pi_{i, j})-1)
\end{eqt}

For the update of $\pi$, which is similar to \ref{upd_pi} in the previous sections, we have
\begin{eqt}
\label{piupdate}
\pi_{i,j} \gets x_{i,j} & - \frac{1}{m+1}\left(\sum_kx_{k,j} - \frac{1}{m+n+1}\sum_{k, l}x_{k,l}\right)
\\& - \frac{1}{n+1}\left(\sum_kx_{i,k}-\frac{1}{m+n+1}\sum_{k, l}x_{k,l}\right)
\end{eqt}
where
\begin{eqt}
\label{xcompute}
x_{i, j} = \frac{1}{t}(\gamma_{i}+\lambda_{j}+\omega_{i,j}-c_{i,j}) + \mu_{i} + \nu_{j} + \tilde{\pi}_{i,j}
\end{eqt}

For the update of $\tilde{\pi}$, setting $\frac{\partial L_t}{\partial\tilde\pi}=0$, we need to solve the following problem for each $\tilde\pi_{i, j}$,
\begin{eqt}
\omega_{i, j} + t(\tilde\pi_{i, j} - \pi_{i, j}) + \epsilon\log\tilde{\pi}_{i, j} = 0
\end{eqt}
Such equation has no closed form solution, though the left hand side is strictly increasing with $\tilde{\pi}_{i, j}$ and its range is $\mathbb{R}$($t$ and $\epsilon$ are both positive numbers). Therefore, we seek to update one step ahead with Newton Method in each iteration as well as sustain the nonnegativity, i.e.,
\begin{eqt}
\label{tildepiupdate}
\tilde\pi_{i, j}\gets \max\left\{\tilde\pi_{i, j} - \frac{\omega_{i, j}+t(\tilde\pi_{i, j}-\pi_{i, j})+\epsilon\log\tilde\pi_{i, j}}{t+\epsilon/\tilde\pi_{i, j}}, 0\right\}
\end{eqt}

However, $\tilde\pi_{i, j}$ may possibly be zero, which may lead to numerical explosion in \ref{tildepiupdate}. Thus, every time before \ref{tildepiupdate}, we let
\begin{eqt}
\label{add}
\tilde\pi_{i, j} \gets \tilde\pi_{i, j} + \tilde\epsilon,
\end{eqt}
where $\tilde\epsilon$ is a very small number. In our implementation, we let $\tilde\epsilon$ be 1e-16.

Update of multipliers are as follows,
\begin{eqt}
\label{multupdate}
& \gamma_{i} \gets \gamma_i + t(\mu_i - \sum_k\pi_{i,k}) \\
& \lambda_{j} \gets \lambda_j + t(\nu_j - \sum_k\pi_{k,j}) \\
& \omega_{i, j} \gets \omega_{i, j} + t(\tilde{\pi}_{i, j} - \pi_{i, j})
\end{eqt}

We thus have the following Algorithm \ref{alg:ADMM-ER}.
\begin{algorithm}
\caption{ADMM for Entropy Regularization}
\label{alg:ADMM-ER}
\begin{algorithmic}[1]
\REQUIRE {$C\in\mbR^{m\times n}$, $\mu\in\mbR^m$, $\nu\in\mbR^n$, $t, \epsilon, \tilde\epsilon\in \mathbb{R}^+$}
\STATE $\pi, \tilde{\pi}, \gamma, \lambda, \omega\gets 0$
\WHILE {\text{not converge}}
	\STATE Update $\pi$ according to \ref{piupdate}, \ref{xcompute}
	\STATE Update $\tilde\pi$ according to \ref{add}, \ref{tildepiupdate}
	\STATE Update $\gamma$, $\lambda$ and $\omega$ by \ref{multupdate}
\ENDWHILE
\RETURN $\pi$
\end{algorithmic}
\end{algorithm}


\subsection{Step 1}
Let's start by rewriting our ADI equation for the Step 1 (given in class 13 slide 13), for an staggered mesh for the horizontal velocity $u$:
\begin{align*}
-d_1u_{i-1/2,j}^{n+1/2}+(1+2d_1)u_{i+1/2,j}^{n+1/2}-d_1u_{i+3/2,j}^{n+1/2}=d_2u_{i+1/2,j+1}^{n}+(1-2d_2)u_{i+1/2,j}^{n}+d_2u_{i+1/2,j-1}^{n},
\end{align*}
which has the form
\begin{align*}
au_{i-1/2,j}^{n+1/2}+bu_{i+1/2,j}^{n+1/2}+cu_{i+3/2,j}^{n+1/2}=d,
\end{align*}
a tridiagonal system. The staggered mesh for $u$ gives us a matrix $u$ that is $(M+1)\times(N+2)$. We only need to solve the previous equation for the interior with two $for$ loops in \textsl{Matlab},
\begin{verbatim}
for j=2:N+1
for i=2:M
		...
	end
end
\end{verbatim}.
We will now include the boundary conditions for each case.
\subsubsection*{Case $j=2$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1u_{3/2,2}^{n+1/2}+(1+2d_1)u_{5/2,2}^{n+1/2}-d_1u_{7/2,2}^{n+1/2}=d_2u_{5/2,3}^{n}+(1-2d_2)u_{5/2,2}^{n}+d_2u_{5/2,1}^{n}.
\end{align*}
Note that I will be using \textsl{Maltab} indices, so the first index is $1$. Since we know that $u_{3/2,2}^{n+1/2}=0$ since it corresponds to the left wall, we can then impose that $a(1)=0$. The value $u_{5/2,1}$ will also be determined by the boundary conditions. Since it is a cell-like boundary conditions in this case, we can impose it by updating the ghost cells. Thus, $d(1)$ is the right hand side of the previous equation.

\subsubsection*{Case $j=2$ and $i\in [3,M-1]$}
In this case the general equation becomes
\begin{align*}
-d_1u_{i-1/2,2}^{n+1/2}+(1+2d_1)u_{i+1/2,2}^{n+1/2}-d_1u_{i+3/2,2}^{n+1/2}=d_2u_{i+1/2,3}^{n}+(1-2d_2)u_{i+1/2,2}^{n}+d_2u_{i+1/2,1}^{n}.
\end{align*}
Like in the previous case, the value $u_{i+1/2,1}$ will also be determined by the boundary conditions. Since it is a cell-like boundary conditions in this case, we can impose it by updating the ghost cells. Thus, $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j=2$ and $i=M$}
In this case the general equation becomes
\begin{align*}
-d_1u_{M-1/2,2}^{n+1/2}+(1+2d_1)u_{M+1/2,2}^{n+1/2}-d_1u_{M+3/2,2}^{n+1/2}=d_2u_{M+1/2,3}^{n}+(1-2d_2)u_{M+1/2,2}^{n}+d_2u_{M+1/2,1}^{n}.
\end{align*}
In this case we can see that the value $u_{M+3/2,2}^{n+1/2}$ corresponds to the horizontal velocity at the right wall, hence it is zero and so it is $c$ in this case. Like in the previous case, the value $u_{M+1/2,1}$ will also be determined by the boundary conditions. Since it is a cell-like boundary condition in this case, we can impose it by updating the ghost cells. Thus, $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j\in[3,N]$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1u_{3/2,j}^{n+1/2}+(1+2d_1)u_{5/2,j}^{n+1/2}-d_1u_{7/2,j}^{n+1/2}=d_2u_{5/2,j+1}^{n}+(1-2d_2)u_{5/2,j}^{n}+d_2u_{5/2,j-1}^{n}.
\end{align*}
In this case the value $u_{3/2,j}^{n+1/2}$ corresponds to the horizontal velocity at the left wall. Since its value will depend on $j$ because of the inlet 1, it will be moved to the right hand side, getting
\begin{align*}
(1+2d_1)u_{5/2,j}^{n+1/2}-d_1u_{7/2,j}^{n+1/2}=d_2u_{5/2,j+1}^{n}+(1-2d_2)u_{5/2,j}^{n}+d_2u_{5/2,j-1}^{n}+d_1u_{3/2,j}^{n+1/2},
\end{align*}
which makes $a=0$ for this case and $d$ being the right hand side of the previous equation.

\subsubsection*{Case $j\in[3,N]$ and $i\in[3,M-1]$}
In this case the general equation does not include any boundary conditions, therefore it is unaltered.
\subsubsection*{Case $j\in[3,N]$ and $i=M$}
In this case the general equation becomes
\begin{align*}
-d_1u_{M-1/2,j}^{n+1/2}+(1+2d_1)u_{M+1/2,j}^{n+1/2}-d_1u_{M+3/2,j}^{n+1/2}=d_2u_{M+1/2,j+1}^{n}+(1-2d_2)u_{M+1/2,j}^{n}+d_2u_{M+1/2,j-1}^{n}.
\end{align*}
Like before the value of $u_{M+3/2,j}^{n+1/2}$ will depend on the value of $j$ because of the inlet 2. Therefore we will move it to the right hand side and get
\begin{align*}
-d_1u_{M-1/2,j}^{n+1/2}+(1+2d_1)u_{M+1/2,j}^{n+1/2}=d_2u_{M+1/2,j+1}^{n}+(1-2d_2)u_{M+1/2,j}^{n}+d_2u_{M+1/2,j-1}^{n}+d_1u_{M+3/2,j}^{n+1/2},
\end{align*}
which gives us, for this case, $c=0$ and $d$ being the right hand side of the previous equation.

\subsubsection*{Case $j=N+1$ and $i=2$}
In this case the general equation becomes
\begin{align*}
-d_1u_{3/2,N+1}^{n+1/2}+(1+2d_1)u_{5/2,N+1}^{n+1/2}-d_1u_{7/2,N+1}^{n+1/2}=d_2u_{5/2,N+2}^{n}+(1-2d_2)u_{5/2,N+1}^{n}+d_2u_{5/2,N}^{n}.
\end{align*}
In this case the value of $u_{3/2,N+1}^{n+1/2}$ corresponds to the horizontal velocity at the left wall, which is zero imposed by the boundary conditions. This implies that $a=0$ for this case. The value of $u_{5/2,N+2}^{n}$ will be determined by the boundary conditions as well. Since it is a cell-like boundary condition in this case, we can impose it by updating the ghost cells. Thus, $d$ is the right hand side of the previous equation.

\subsubsection*{Case $j=N+1$ and $i\in[3,M-1]$}
\begin{align*}
-d_1u_{i-1/2,N+1}^{n+1/2}+(1+2d_1)u_{i+1/2,N+1}^{n+1/2}-d_1u_{i+3/2,N+1}^{n+1/2}=d_2u_{i+1/2,N+2}^{n}&+(1-2d_2)u_{i+1/2,N+1}^{n}\\&+d_2u_{i+1/2,N}^{n},
\end{align*}
Like in the previous case, the value of $u_{i+1/2,N+2}^{n}$ will be determined by the boundary conditions as well. Since it is a cell-like boundary condition in this case, we can impose it by updating the ghost cells. Thus, $d$ is the right hand side of the previous equation. We will see how its value does not depend on the value of $i$ since the horizontal velocity in the inlet 3 is zero.

\subsubsection*{Case $j=N+1$ and $i=M$}
\begin{align*}
-d_1u_{M-1/2,N+1}^{n+1/2}+(1+2d_1)u_{M+1/2,N+1}^{n+1/2}-d_1u_{M+3/2,N+1}^{n+1/2}=d_2u_{M+1/2,N+2}^{n}&+(1-2d_2)u_{M+1/2,N+1}^{n}\\&+d_2u_{M+1/2,N}^{n},
\end{align*}
In this case the value of $u_{M+3/2,N+1}^{n+1/2}$ corresponds to the horizontal velocity at the right wall, imposed to be zero by the boundary conditions. Therefore $c=0$ for this case. The value of $u_{M+1/2,N+2}^{n}$ will be determined by the boundary conditions as well. Since it is a cell-like boundary condition in this case, we can impose it by updating the ghost cells.

\subsection{Step 2}
We start by rewriting our ADI equation for the Step 2 (given in class 13 slide 14), for an staggered mesh for the horizontal velocity $u$:
\begin{align*}
-d_2u_{i+1/2,j-1}^{n+1}+(1+2d_2)u_{i+1/2,j}^{n+1}-d_2u_{i+1/2,j+1}^{n+1}=d_1u_{i+3/2,j}^{n+1/2}+(1-2d_1)u_{i+1/2,j}^{n+1/2}+d_1u_{i-1/2,j}^{n+1/2}
\end{align*}
which has the form
\begin{align*}
au_{i+1/2,j-1}^{n+1}+bu_{i+1/2,j}^{n+1}+cu_{i+1/2,j+1}^{n+1}=d
\end{align*}
a tridiagonal system. The staggered mesh for $u$ gives us a matrix $u$ that is $(M+1)\times(N+2)$. We only need to solve the previous equation for the interior with two $for$ loops in \textsl{Matlab},
\begin{verbatim}
for j=2:M
for i=2:N+1
		...
	end
end
\end{verbatim}.
We will now include the boundary conditions for each case.
\subsubsection*{Case $i=2$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2u_{5/2,1}^{n+1}+(1+2d_2)u_{5/2,2}^{n+1}-d_2u_{5/2,3}^{n+1}=d_1u_{7/2,2}^{n+1/2}+(1-2d_1)u_{5/2,2}^{n+1/2}+d_1u_{3/2,2}^{n+1/2}.
\end{align*}
The value of $u_{5/2,1}^{n+1}$ does not correspond to the interior and will be determined by the boundary conditions to be $-u_{5/2,2}^{n+1}$. Since it is a cell-like boundary condition in this case, we can impose it by updating the ghost cells. The value of $u_{3/2,2}^{n+1/2}$ corresponds to the left wall and it is then imposed to be zero. The previous equation then gives us
\begin{align*}
(1+3d_2)u_{5/2,2}^{n+1}-d_2u_{5/2,3}^{n+1}=d_1u_{7/2,2}^{n+1/2}+(1-2d_1)u_{5/2,2}^{n+1/2}.
\end{align*}

Thus, for this case $a=0$, $b=1+3d_2$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=2$ and $j\in[3,N]$}
In this case the general equation becomes
\begin{align*}
-d_2u_{5/2,j-1}^{n+1}+(1+2d_2)u_{5/2,j}^{n+1}-d_2u_{5/2,j+1}^{n+1}=d_1u_{7/2,j}^{n+1/2}+(1-2d_1)u_{5/2,j}^{n+1/2}+d_1u_{3/2,j}^{n+1/2}.
\end{align*}
The value of $u_{3/2,j}^{n+1/2}$ corresponds to the left wall and it is then imposed to be zero and $d$ is given by the right hand side.

\subsubsection*{Case $i=2$ and $j=N+1$}
In this case the general equation becomes
\begin{align*}
-d_2u_{5/2,N}^{n+1}+(1+2d_2)u_{5/2,N+1}^{n+1}-d_2u_{5/2,N+2}^{n+1}=d_1u_{7/2,N+1}^{n+1/2}+(1-2d_1)u_{5/2,N+1}^{n+1/2}+d_1u_{3/2,N+1}^{n+1/2}.
\end{align*}
The value of $u_{3/2,N+1}^{n+1/2}$ corresponds to the left wall and it is then imposed to be zero. The value of $u_{5/2,N+2}^{n+1}=-u_{5/2,N+1}^{n+1}$ by the boundary conditions. Thus, for this case $c=0$, $b=1+3d_2$ and $d$ is given by the right hand side.

\subsubsection*{Case $i\in[3,M-1]$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2u_{i+1/2,1}^{n+1}+(1+2d_2)u_{i+1/2,2}^{n+1}-d_2u_{i+1/2,3}^{n+1}=d_1u_{i+3/2,2}^{n+1/2}+(1-2d_1)u_{i+1/2,2}^{n+1/2}+d_1u_{i-1/2,2}^{n+1/2}.
\end{align*}
By the boundary conditions, $u_{i+1/2,1}^{n+1}=-u_{i+1/2,2}^{n+1}$ and the previous equation yields
\begin{align*}
(1+3d_2)u_{i+1/2,2}^{n+1}-d_2u_{i+1/2,3}^{n+1}=d_1u_{i+3/2,2}^{n+1/2}+(1-2d_1)u_{i+1/2,2}^{n+1/2}+d_1u_{i-1/2,2}^{n+1/2}.
\end{align*}
Thus, for this case $a=0$, $b=1+3d_2$ and $d$ is given by the right hand side.
\subsubsection*{Case $i\in[3,M-1]$ and $j\in[3,N]$}
In this case the general equation does not include any boundary conditions, therefore it is unaltered.

\subsubsection*{Case $i\in[3,M-1]$ and $j=N+1$}
In this case the general equation becomes
\begin{align*}
-d_2u_{i+1/2,N}^{n+1}+(1+2d_2)u_{i+1/2,N+1}^{n+1}-d_2u_{i+1/2,N+2}^{n+1}=d_1u_{i+3/2,N+1}^{n+1/2}&+(1-2d_1)u_{i+1/2,N+1}^{n+1/2}\\&+d_1u_{i-1/2,N+1}^{n+1/2}.
\end{align*}
By the boundary conditions, $u_{i+1/2,N+2}^{n+1}=-u_{i+1/2,N+1}^{n+1}$ and the previous equation yields
\begin{align*}
-d_2u_{i+1/2,N}^{n+1}+(1+3d_2)u_{i+1/2,N+1}^{n+1}=d_1u_{i+3/2,N+1}^{n+1/2}+(1-2d_1)u_{i+1/2,N+1}^{n+1/2}+d_1u_{i-1/2,N+1}^{n+1/2}.
\end{align*}
Thus, for this case $c=0$, $b=1+3d_2$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=M$ and $j=2$}
In this case the general equation becomes
\begin{align*}
-d_2u_{M+1/2,1}^{n+1}+(1+2d_2)u_{M+1/2,2}^{n+1}-d_2u_{M+1/2,3}^{n+1}=d_1u_{M+3/2,2}^{n+1/2}+(1-2d_1)u_{M+1/2,2}^{n+1/2}+d_1u_{M-1/2,2}^{n+1/2}.
\end{align*}
By the boundary conditions, $u_{M+1/2,1}^{n+1}=-u_{M+1/2,2}^{n+1}$ and $u_{M+3/2,2}^{n+1/2}=0$ since it corresponds to the right wall. The previous equation yields
\begin{align*}
(1+3d_2)u_{M+1/2,2}^{n+1}-d_2u_{M+1/2,3}^{n+1}=(1-2d_1)u_{M+1/2,2}^{n+1/2}+d_1u_{M-1/2,2}^{n+1/2}.
\end{align*}
Thus, for this case $a=0$, $b=1+3d_2$ and $d$ is given by the right hand side.

\subsubsection*{Case $i=M$ and $j\in[3,N]$}
In this case the general equation becomes
\begin{align*}
-d_2u_{M+1/2,j-1}^{n+1}+(1+2d_2)u_{M+1/2,j}^{n+1}-d_2u_{M+1/2,j+1}^{n+1}=d_1u_{M+3/2,j}^{n+1/2}+(1-2d_1)u_{M+1/2,j}^{n+1/2}+d_1u_{M-1/2,j}^{n+1/2}
\end{align*}
The value of $u_{M+3/2,j}^{n+1/2}$ depends on $j$ because of inlet 2. It will be imposed by the corresponding boundary condition in the code. In this case, $d$ is given by the right hand side.

\subsubsection*{Case $i=M$ and $j=N+1$}
In this case the general equation becomes
\begin{align*}
-d_2u_{M+1/2,N}^{n+1}+(1+2d_2)u_{M+1/2,N+1}^{n+1}-d_2u_{M+1/2,N+2}^{n+1}=d_1u_{M+3/2,N+1}^{n+1/2}&+(1-2d_1)u_{M+1/2,N+1}^{n+1/2}\\&+d_1u_{M-1/2,N+1}^{n+1/2}
\end{align*}
By the boundary conditions, $u_{M+1/2,N+2}^{n+1}=-u_{M+1/2,N+1}^{n+1}$ and $u_{M+3/2,N+1}^{n+1/2}=0$ since it corresponds to the right wall. The previous equation yields
\begin{align*}
-d_2u_{M+1/2,N}^{n+1}+(1+3d_2)u_{M+1/2,N+1}^{n+1}=(1-2d_1)u_{M+1/2,N+1}^{n+1/2}+d_1u_{M-1/2,N+1}^{n+1/2}.
\end{align*}
Thus, for this case $c=0$, $b=1+3d_2$ and $d$ is given by the right hand side.


\section{Initial Value Problems}
\noindent
We can see that since solving a differential equation will mean integrating to get rid of derivatives, the $+ C$ from integration will gives us multiple solutions. We call these sets of solutions that differ only in these constants ``solution families''. If we want to find one specific solution, we need more information about the value of the function and it's derivatives. This type of problem where a differential equation is coupled with function values is called an initial value problem (IVP).\\

\begin{definition}
	An initial value problem has the general form.
	\begin{equation*}
		\begin{cases}
			a_ny^{(n)} + a_{n-1}y^{(n-1)} + \ldots + a_0y = f(x) \\
			y(x_0) = y_0 \\
			y'(x_1) = y_1 \\
			\vdots \\
			y^{(n)}(x_n) = y_n
		\end{cases}.
	\end{equation*}
	Often, each $x_i$ is 0.
\end{definition}

\begin{example}
	The general solution to the IVP
	\begin{equation*}
		\begin{cases}
			y^{\prime\prime} - 5y^\prime + 6y = 0 \\
			y(0) = 3 \\
			y^\prime(0) = 1
		\end{cases}
	\end{equation*}
	is $y = c_1e^{2x} + c_2e^{3x}$. Find the specific solution.
\end{example}
\noindent
Evaluating $y$ at $x = 0$,
\begin{equation*}
	y(0) = c_1 + c_2 = 3.
\end{equation*}
Evaluating $y^\prime$ at $x = 0$,
\begin{equation*}
	y^\prime(0) = 2c_1 + 3c_2 = 1.
\end{equation*}
To find $c_1$ and $c_2$, we need to solve the system of linear equations
\begin{equation*}
	\begin{cases}
		c_1 + c_2 = 3 \\
		2c_1 + 3c_2 = 1
	\end{cases} \implies \begin{cases}
		c_1 = 8 \\
		c_2 = -5
	\end{cases}.
\end{equation*}
So, our specific solution to the IVP is
\begin{equation*}
	y = 8e^{2x} - 5e^{3x}.
\end{equation*}

\begin{theorem}[Existence and Uniqueness of Solutions to 1st Order IVPs]
	Consider the IVP
	\begin{equation*}
		\begin{cases}
			\dd{x}{y} = f(x,y) \\
			y(x_0) = y_0
		\end{cases}.
	\end{equation*}
	If $f(x,y)$ and $\frac{\partial}{\partial y}f$ are both continuous on some rectangular region containing the point $(x_0, y_0)$, then the IVP has a unique solution $y = y(x)$ on some open interval containing $x_0$.
\end{theorem}

\begin{example}
	Does a solution to the following IVP exist? Is it unique?
	\begin{equation*}
		\begin{cases}
			\dd{y}{x} = x^2 - xy^3 \\
			y(1) = 6
		\end{cases}
	\end{equation*}
\end{example}
\noindent
$f(x,y) = x^2 - xy^3$ and $\frac{\partial f}{\partial y} = -3xy^2$ are continuous on all of $\R^2$. So, the existence and uniqueness theorem tells us that the IVP has a unique solution on an open interval containing $x_0 = 1$.

\begin{example}
	Does a solution to the following IVP exist? Is it unique?
	\begin{equation*}
		\begin{cases}
			\dd{y}{x} = 3y^{2/3} \\
			y(2) = 0
		\end{cases}
	\end{equation*}
\end{example}
\noindent
$f(x,y) = 3y^{2/3}$ is continuous on $y \in \R$, and $\frac{\partial f}{\partial y} = 2y^{-1/3}$ is continuous on $x \in \left(-\infty, 0\right) \cup \left(0, \infty\right)$. Since $\frac{\partial f}{\partial y}$ is not continuous on a domain containing $(2,0)$, the existence and uniqueness theorem does not guarantee a solution.


    \filetitle{movavg}{Moving average pseudofunction}{modellang/movavg}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
movavg(Expr)
movavg(Expr,K)
\end{verbatim}

\paragraph{Description}\label{description}

If the second input argument, \texttt{K}, is negative, this function
expands to the moving average of the last K periods (including the
current period), i.e.

\begin{verbatim}
(((Expr)+(Expr{-1})+ ... +(Expr{-(K-1)})/-K)
\end{verbatim}

where \texttt{Expr\{-N\}} derives from \texttt{Expr} and has all its
time subscripts shifted by \texttt{-N} (if specified).

If the second input argument, \texttt{K}, is positive, this function
expands to the moving average of the next K periods ahead (including the
current period), i.e.

\begin{verbatim}
(((Expr)+(Expr{1})+ ... +(Expr{K-1})/K)
\end{verbatim}

If the second input argument, \texttt{K}, is not specified, the default
value -4 is used (based on the fact that most of the macroeconomic
models are quarterly).

\paragraph{Example}\label{example}

The following three lines

\begin{verbatim}
movavg(Z)
movavg(Z,-3)
movavg(X+Y{-1},2)
\end{verbatim}

will expand to

\begin{verbatim}
(((Z)+(Z{-1})+(Z{-2})+(Z{-3}))/4)
(((Z)+(Z{-1})+(Z{-2}))/3)
(((X+Y{-1})+(X{1}+Y))/2)
\end{verbatim}


\subsection{Convolutional layers}

Can connect each  node in first hidden layer to a subset of the input layer, eg one node for each 5x5 pixels

We also share weights for each of the first layer. Much fewer parameters, and can learn all good stuff

This also uses windows. Instead of max we multiply the window by a matrix elementwise and sum the values

Each matrix can represent some feature, like a curve.

We can use multiply convolution matrices to create multiple output matrices.

Matrices are called kernels. they are trained. start off random

\subsubsection{Training convolutional layers}


\section{Multiple Synthetic Risky Assets}
In this section we present an application of the learning algorithms considered above to a multi-asset allocation problem. Given the difficulties of learning a profitable strategy for historical data, we consider once again synthetic price series that can be traded profitably. To define the generative model, we start from a continuous-time formulation and then derive a discrete-time model using standard discretization techniques. In particular, we assume that the market consists of two risky asset $\{S_t^1, S_t^2\}$, whose prices evolve according to the following dynamics
\begin{equation}
	\begin{cases}
		\frac{dS_t^1}{S_t^1} &= \sigma_1 dW_t^1\\
		\frac{dS_t^2}{S_t^2} &= \frac{1}{2} \sigma_\chi^2 dt + \sigma_2 dW_t^2 + d\chi_t\\
		d\chi_t &= -\lambda \chi_t dt + \sigma_\chi dW_t^\chi\\
	\end{cases}
\end{equation}
where $\{W_t^1, W_t^2, W_t^\chi\}$ are standard Brownian motions such that $\E {dW_t^1 dW_t^2} = \rho dt$, with $-1 \leq \rho \leq 1$, $W_t^\chi \independent (W_t^1, W_t^2)$,
$\sigma_1$, $\sigma_2$, $\sigma_\chi$, $\lambda >0$ and $\chi_0 = 0$. The Ornstein-Uhlenbeck process $\chi_t$ represents a mean-reverting spread between the two risky assets. Let $\widetilde{S}_t^i = \log S_t^i$, $i \in \{1,2\}$ denote the log-price. A simple application of Itô's lemma yields
\begin{equation}
	\label{eq:sde}
	\begin{cases}
		d\widetilde{S}_t^1 &= -\frac{1}{2} \sigma_1^2 dt + \sigma_1 dW_t^1\\
		d\widetilde{S}_t^2 &= -\frac{1}{2} \sigma_2^2 dt + \sigma_2 dW_t^2 + d\chi_t\\
		d(e^{\lambda t} \chi_t) &= \sigma_\chi e^{\lambda t} dW_t^\chi\\
	\end{cases}
\end{equation}
Integrating between $0$ and $t$ and rearranging the various terms, we obtain
\begin{equation}
	\label{eq:sol_sde}
	\begin{cases}
		S_t^1 &= S_0^1 e^{-\frac{1}{2} \sigma_1^2 t + \sigma_1 W_t^1}\\
		S_t^2 &= S_0^2 e^{-\frac{1}{2} \sigma_2^2 t + \sigma_2 W_t^2 + \chi_t}\\
		\chi_t &= \sigma_\chi \int_0^t e^{-\lambda (t-u)} dW_u^\chi\\
	\end{cases}
\end{equation}
We notice that the spread is a Gaussian process and $\forall t > 0$
\begin{equation}
	\chi_t \sim \calN\left(0, \frac{\sigma_\chi^2}{2\lambda}\left(1-e^{-2\lambda t}\right) \right)
\end{equation}
To better understand the role of the spread, let us remind that $W_t^2$ can be decomposed in the following way
\begin{equation}
	W_t^2 = \rho W_t^1 + \sqrt{1-\rho^2} W_t^{\independent}
\end{equation}
where $W_t^{\independent} \independent W_t^1$ is a standard Brownian motion. Thus, we have
\begin{equation}
	\frac{S_t^2}{S_t^1} = \frac{S_0^2}{S_0^1} e^{-\frac{1}{2}(\sigma_2^2 - \sigma_1^2) t + (\rho \sigma_2 - \sigma_1) W_t^1 + \sigma_2 \sqrt{1-\rho^2} W_t^{\independent}} e^{\chi_t}
\end{equation}
Taking the expected value, we obtain
\begin{equation}
	\E{\frac{S_t^2}{S_t^1}} = \frac{S_0^2}{S_0^1} e^{\sigma_1 (\sigma_1 - \rho \sigma_2) t} e^{\frac{\sigma_\chi^2}{4\lambda} (1-e^{-2\lambda t})}
\end{equation}
The term coming from the stochastic spread disappears in the long term. In the limit case where $\sigma_1 = \sigma_2$ and $\rho = 1$. Then
\begin{equation}
	\E{\frac{S_t^2}{S_t^1}} = \frac{S_0^2}{S_0^1} e^{\frac{\sigma_\chi^2}{4\lambda} (1-e^{-2\lambda t})} \underset{t \to \infty}{\to} \frac{S_0^2}{S_0^1}
\end{equation}
Therefore, the expected value of the ratio between the prices of the two risky assets mean-reverts to the initial ratio. It is easy to understand that this feature can be traded profitably by betting on the convergence of the two assets to their long-term regime.\\
The solutions of the system (\ref{eq:sol_sde}) can be easily used to simulate the risky assets prices, which can then be used as inputs of the asset allocation problem. An alternative approach is to obtain the discrete-time dynamics of the system. Let us consider a uniform time-grid $t_k = k \Delta t$, $k \in \mathbb{N}$, and let us integrate the system (\ref{eq:sde}) between $t_k$ and $t_{k+1}$. After some simple algebraic manipulations, we obtain the following equations
\begin{equation}
	\label{eq:dt_dynamics}
	\begin{cases}
		\widetilde{S}_{k+1}^1 &= \widetilde{S}_{k}^1 -\frac{1}{2} \sigma_1^2 \Delta t + \sigma_1 \sqrt{t} \epsilon_k^1\\
		\widetilde{S}_{k+1}^2 &= \widetilde{S}_{k}^2 -\frac{1}{2} \sigma_2^2 \Delta t + \sigma_2 \sqrt{t} \epsilon_k^2 + \chi_{k+1} - \chi_k\\
		\chi_{k+1} &= e^{-\lambda \Delta t} \chi_k + \sigma_\chi \sqrt{\frac{1-e^{-2\lambda \Delta t}}{2\lambda}} \epsilon_k^\chi
	\end{cases}
\end{equation}
where the noises are a Gaussain white noise with the following structure
\begin{equation}
	\begin{bmatrix}
	  	\epsilon_k^1\\
	  	\epsilon_k^2\\
	  	\epsilon_k^\chi\\
	\end{bmatrix} \sim \calN\left(
	\begin{bmatrix}
		  	0\\
		  	0\\
		  	0\\
	\end{bmatrix},
	\begin{bmatrix}
		  	1, \rho, 0\\
		  	\rho, 1, 0\\
		  	0, 0, 1\\
	\end{bmatrix}
	\right)
\end{equation}
We expect to be possible to trade the two risky assets profitably using an approach similar to the well-known pairs trading technique for cointegrated assets. Intuitively, because of the mean-reversion of the stochastic spread, we should be able to generate a profit by betting on the convergence of the prices of the two assets when they are far apart by selling the more expensive asset and buying the cheaper one. For these considerations, we expect the RL algorithms discussed in the previous chapters to be able to spot this pattern and exploit it to generate a profit.

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.8\textwidth]{Images/8_cointegrated_series}
	\caption[Sample paths for the risky assets with a mean-reverting spread.]{Sample paths for the risky assets with a mean-reverting spread.}
	\label{fig:cointegrated_series}
\end{figure}

\subsection{Specifications of the Learning Algorithms}
The parametric policies used in Section \ref{sec:synthetic_risky_asset} can be directly used in this setting to select the weight of one of the assets. Given the poor results obtained with ARAC, we only focus on the PGPE and NPGPE algorithms. The weight on the first asset is selected using The following controller
\begin{equation*}
	a^1 = F_\theta(s) = \sign(\theta \cdot s)
\end{equation*}
where in PGPE the parameters are sampled from a multi-variate Gaussian distribution
\begin{equation*}
	\theta \sim \calN(\mu, \diag(\sigma))
\end{equation*}
while in NPGPE the controller parameters are sampled from a Gaussian distribution parameterized by its mean and Cholesky factor
\begin{equation*}
	\theta \sim \calN(\mu, C^T C)
\end{equation*}
The long-short strategy is completed by selecting $a^2 = - a^1$. In this case we notice that $a^1 + a^2 = 0$, which means that the long position on one asset is entirely financed by the short position on the other asset. Clearly, this assumption is simplistic as it neglects all the practical constraints on short-selling. Given that we will always be short on one of the two assets, we will also neglect short-selling fees, i.e. $\delta_s = 0$.


\section{Definition and examples}

\begin{outcome}
  \begin{enumerate}
  \item Understand the definition of a linear transformation in the
    context of vector spaces.
  \item Determine whether a function is linear or not.
  \end{enumerate}
\end{outcome}

In Chapter~\ref{cha:linear-transformation}, we defined a linear
transformation $T:\R^n\to\R^m$ to be a function that preserves
addition and scalar multiplication. We now revisit this concept in the
more general setting of vector spaces $V$ and $W$.

\begin{definition}{Linear transformation}{linear-transformation-vector-space}
  Let $V$ and $W$ be vector spaces over some field $K$. A function
  $T: V \to W$ is called a \textbf{linear transformation}%
  \index{linear transformation}%
  \index{linear transformation!on vector spaces} from $V$ to $W$ if
  it satisfies the following two conditions:
  \begin{enumerate}
  \item $T$ preserves addition, i.e., for all\/
    $\vect{v},\vect{w}\in V$, we have
    $T(\vect{v}+\vect{w}) = T(\vect{v}) + T(\vect{w})$;
  \item $T$ preserves scalar multiplication, i.e, for all\/
    $\vect{v}\in V$ and $k\in K$, we have
    $T(k\vect{v}) = kT(\vect{v})$.
  \end{enumerate}
  A linear transformation is also sometimes called a \textbf{linear
    function} or a \textbf{linear map}. A linear transformation
  $T:V\to V$ (i.e., when $V=W$) is also sometimes called an
  \textbf{operator}%
  \index{operator}%
  \index{operator|seealso{linear transformation}}.
\end{definition}

Our first example of a linear transformation is a matrix
transformation. We have already seen this in
Section~\ref{sec:matrix-of-transformation}.

\begin{example}{Matrix transformation}{matrix-transformation}
  Let $A$ be an $m\times n$-matrix. Then the function $T:\R^m\to\R^n$
  defined by $T(\vect{v})=A\vect{v}$ is a linear transformation,
  called a \textbf{matrix transformation}%
  \index{matrix transformation}. This was proved in
  Proposition~\ref{prop:matrix-are-linear}.
\end{example}

There are many interesting examples of linear transformations on
vector spaces other than $\R^n$. We will consider a few such examples.

\begin{example}{Derivative operator}{derivative-operator}
  Let $\Poly_n$ be the vector space of real polynomials of degree at
  most $n$. The function $D:\Poly_n\to\Poly_{n-1}$ is defined by
  \begin{equation*}
    D(p(x)) = p'(x),
  \end{equation*}
  where $p'(x)$ denotes the derivative of the polynomial $p(x)$. The
  function $D$ is called the \textbf{derivative operator}.
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item Compute $D(x^3)$, $D(2x^2+x)$, and $D(ax^3+bx^2+cx+d)$.
    \item Show that $D$ is a linear transformation.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{example}

\begin{solution}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item In each case, we simply take the derivative:
      \begin{eqnarray*}
        D(x^3) &=& 3x^2, \\
        D(2x^2+x) &=& 4x+1, \\
        D(ax^3+bx^2+cx+d) &=& 3ax^2 + 2bx + c.
      \end{eqnarray*}
    \item First, we note that if $p(x)$ is a polynomial of degree at
      most $n$, then its derivative $p'(x)$ is a polynomial of degree
      at most $n-1$. Therefore, the derivative operator $D$ is a
      well-defined function from $\Poly_n$ to $\Poly_{n-1}$. To show that
      it preserves addition, consider any two polynomials
      $p(x),q(x)\in\Poly_n$. From calculus, we know that the
      derivative of $p(x)+q(x)$ is $p'(x)+q'(x)$. Therefore,
      \begin{equation*}
        D(p(x)+q(x)) ~=~ (p(x)+q(x))' ~=~ p'(x)+q'(x) ~=~ D(p(x)) + D(q(x)),
      \end{equation*}
      and so $D$ preserves addition. To show that it preserves scalar
      multiplication, consider $p(x)\in\Poly_n$ and $k\in\R$. From
      calculus, we know that the derivative of $kp(x)$ is $kp'(x)$, and
      therefore
      \begin{equation*}
        D(kp(x)) ~=~ (kp(x))' ~=~ kp'(x) ~=~ kD(p(x)).
      \end{equation*}
      Hence, $D$ preserves scalar multiplication. It follows that $D$ is a
      linear transformation.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{solution}

It is important to understand that we are not claiming that the
derivative $p'(x)$ of a polynomial $p(x)$ is a linear function. It is
of course a polynomial. Rather, what the above example shows is that
the act of {\em taking} the derivative is a linear operation, i.e.,
the derivative of a sum is the sum of the derivatives, and the
derivative of a constant times a function is a constant times the
derivative.

\begin{example}{A differential equation}{differential-equation-derivative-operator}
  Solve the equation $p(x) = x^3 + D(p(x))$, where
  $D:\Poly_3\to\Poly_2$ is the derivative operator of
  Example~\ref{exa:derivative-operator}.
\end{example}

\begin{solution}
  Every element of $P_3$ is of the form $p(x) = ax^3 + bx^2 + cx + d$.
  We can write the equation $p(x) = x^3 + D(p(x))$ as
  \begin{equation*}
    (ax^3 + bx^2 + cx + d) = x^3 + (3ax^2 + 2bx + c).
  \end{equation*}
  For the left-hand side and right-hand side to be equal, we must have
  $a=1$, $b=3a$, $c=2b$, and $d=c$. This yields the unique solution
  $(a,b,c,d) = (1,3,6,6)$, or $p(x) = x^3+3x^2+6x+6$.
\end{solution}

\begin{example}{The shift and unshift operators}{shift-unshift}
  Consider the vector space $\Seq_K$ of infinite sequences of elements
  of $K$. The function $\shift:\Seq_K\to\Seq_K$ is defined by shifting
  the entire sequence to the left and dropping the first element:
  \begin{equation*}
    \shift(a_0,a_1,a_2,a_3,\ldots) = (a_1,a_2,a_3,a_4,\ldots).
  \end{equation*}
  The function $\unshift:\Seq_K\to\Seq_K$ is defined by shifting the
  entire sequence to the right and adding $0$ as the new first
  element:
  \begin{equation*}
    \unshift(a_0,a_1,a_2,a_3,\ldots) = (0,a_0,a_1,a_2,\ldots).
  \end{equation*}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item Compute $\shift(1,2,3,\ldots)$,
      $\unshift(\shift(1,1,1,\ldots))$, and
      $\shift(\unshift(1,1,1,\ldots))$.
    \item Show that $\shift$ and $\unshift$ are linear transformations.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{example}

\begin{solution}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item We have
      \begin{eqnarray*}
        \shift(1,2,3,4,\ldots)
        &=& (2,3,4,5,\ldots), \\
        \unshift(\shift(1,1,1,1,\ldots))
        &=& \unshift(1,1,1,1,\ldots)
        ~~=~~ (0,1,1,1,\ldots), \\
        \shift(\unshift(1,1,1,1,\ldots))
        &=& \shift(0,1,1,1,\ldots)
        ~~=~~ (1,1,1,1,\ldots).
      \end{eqnarray*}
    \item To show that $\shift$ is a linear transformation, we show
      that it preserves addition and scalar multiplication. Let
      $a=(a_0,a_1,a_2,\ldots)$ and $b=(b_0,b_1,b_2,\ldots)$. Then
      \begin{eqnarray*}
        \shift(a+b) &=& \shift(a_0+b_0,~a_1+b_1,~a_2+b_2,~\ldots) \\
                    &=& (a_1+b_1,~a_2+b_2,~a_3+b_3,~\ldots) \\
                    &=& (a_1,~a_2,~a_3,~\ldots) + (b_1,~b_2,~b_3,~\ldots) \\
                    &=& \shift(a) + \shift(b),~ \\
        \shift(ka)  &=& \shift(ka_0,~ka_1,~ka_2,~\ldots) \\
                    &=& (ka_1,~ka_2,~ka_3,~\ldots) \\
                    &=& k(a_1,~a_2,~a_3,~\ldots) \\
                    &=& k\shift(a).
      \end{eqnarray*}
      Therefore, $\shift$ is a linear transformation. The proof for
      $\unshift$ is similar.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{solution}

\begin{example}{Properties of shift and unshift}{shift-unshift-properties}
  Show that for all sequences $a\in\Seq_K$, we have
  $\shift(\unshift(a))=a$. On the other hand, show that in general,
  $\unshift(\shift(a))\neq a$.
\end{example}

\begin{solution}
  For $a=(a_0,a_1,a_2,\ldots)$, we have
  \begin{equation*}
    \shift(\unshift(a))
    ~=~ \shift(\unshift(a_0,a_1,a_2,\ldots))
    ~=~ \shift(0,a_0,a_1,\ldots)
    ~=~ (a_0,a_1,a_2,\ldots)
    ~=~ a.
  \end{equation*}
  On the other hand, we have
  \begin{equation*}
    \unshift(\shift(a))
    ~=~ \unshift(\shift(a_0,a_1,a_2,\ldots))
    ~=~ \unshift(a_1,a_2,a_3,\ldots)
    ~=~ (0,a_1,a_2,\ldots).
  \end{equation*}
  The latter is not equal to $a$ unless $a_0=0$.
\end{solution}

\begin{example}{Recurrence as a linear equation}{recurrence-as-equation}
  Find the general solution to the following equation, where
  $a\in\Seq_K$:
  \begin{equation*}
    \shift(\shift(a)) = \shift(a) + a.
  \end{equation*}
  Where have you seen this equation before?
\end{example}

\begin{solution}
  For $a=(a_0,a_1,a_2,\ldots)$, we have
  \begin{eqnarray*}
    \shift(\shift(a)) &=& (a_2,a_3,a_4,\ldots), \\
    \shift(a) &=& (a_1,a_2,a_3,\ldots), \\
    a &=& (a_0,a_1,a_2,\ldots).
  \end{eqnarray*}
  Therefore $a$ is a solution of the equation $\shift(\shift(a)) =
  \shift(a) + a$ if and only if
  \begin{eqnarray*}
    a_2 &=& a_1 + a_0, \\
    a_3 &=& a_2 + a_1, \\
    a_4 &=& a_3 + a_2,
  \end{eqnarray*}
  and so on. In other words, $a$ is a solution if and only if
  $a_{n+2} = a_{n+1} + a_n$ holds for all $n\geq 0$. This is nothing
  but the recurrence relation of
  Example~\ref{exa:subspace-recurrence}. We already calculated the
  general solution in
  Example~{\ref{exa:subspace-recurrence-dimension}}. The general
  solution is
  \begin{equation*}
    a = (x,~y,~x+y,~x+2y,~2x+3y,~3x+5y,~\ldots),
  \end{equation*}
  and a basis for the solution space is
  \begin{equation*}
    \set{(1,0,1,1,2,3,\ldots),~ (0,1,1,2,3,5,\ldots)}.
  \end{equation*}
\end{solution}

We conclude this section by stating some elementary properties of
linear transformations. ``Elementary'' means that these properties
follow directly from the definition, i.e., from the fact that linear
transformations preserve addition and scalar multiplication.

\begin{proposition}{Properties of linear transformations}{properties-linear-transformations}
  Let $V$ and $W$ be vector spaces over a field $K$, and let
  $T:V \to W$ be a linear transformation.  Then
  \begin{itemize}
  \item $T$ preserves the zero vector:
    $T(\vect{0})=\vect{0}$.
  \item $T$ preserves additive inverses:
    $T(-\vect{v})= -T(\vect{v})$.
  \item $T$ preserves linear combinations:
    \begin{equation*}
      T(a_1\vect{v}_1 + a_2\vect{v}_2 + \ldots + a_k\vect{v}_k)
      ~=~
      a_1T(\vect{v}_1) + a_2T(\vect{v}_2) + \ldots + a_kT(\vect{v}_k).
    \end{equation*}
  \end{itemize}
\end{proposition}

\begin{proof}
  To prove the first property, let $k=0$ in the equation
  $T(k\vect{v}) = kT(\vect{v})$. Since $0\vect{v}=\vect{0}$ and
  $0T(\vect{v})=\vect{0}$ by
  Proposition~\ref{prop:vector-space-elementary}, we therefore have
  $T(\vect{0}) = \vect{0}$.  Similarly, to prove the second property,
  let $k=-1$ in the equation $T(k\vect{v}) = kT(\vect{v})$.
  Finally, the third property is a direct consequence of the fact that
  $T$ preserves addition and scalar multiplication:
  \begin{eqnarray*}
    T(a_1\vect{v}_1 + a_2\vect{v}_2 + \ldots + a_k\vect{v}_k)
    &=& T(a_1\vect{v}_1) + T(a_2\vect{v}_2) + \ldots + T(a_k\vect{v}_k) \\
    &=& a_1T(\vect{v}_1) + a_2T(\vect{v}_2) + \ldots + a_kT(\vect{v}_k).
  \end{eqnarray*}
\end{proof}


%---------------------------Oddy---------------------------
\section{Oddy}

First we define the Oddy $O$ in terms of the Jacobian matrices $A_i$ from \S\ref{s:hex}:
\[
  O(A_i) = \frac{\left| A_i^t A_i \right|^2 - \frac {1}{3}\left|A_i\right|^4}{\alpha_i^{\frac{4}{3}}}.
\]
The metric value is then the maximum Oddy over all the corners and the element center
\[
  q = \max_{i\in\{0,1,\ldots,8\}}\left\{ O(A_i) \right\}.
\]
This can be interpreted as the maximum deviation of
the metric tensor ($A_i^tA_i$) from the identity matrix, evaluated at the corners and element center.

Note that if $\alpha_i \leq DBL\_MIN$ for any $i$, we set $q = DBL\_MAX$.

\hexmetrictable{Oddy}%
{$1$}%                                        Dimension
{$[0,0.5]$}%                                  Acceptable range
{$[0,DBL\_MAX]$}%                             Normal range
{$[0,DBL\_MAX]$}%                             Full range
{$0$}%                                        Cube
{Adapted from \cite{odd:88}}%                 Citation
{v\_hex\_oddy}%                               Verdict function name


\subsection{Randomly Controlled Trials (RCTs)}

If the model is:

\(y_i=D_i\theta +g(X) +\epsilon_i\)

And \(D\) is randomly assigned, then we can estimate

\(y_i=D_i\theta +\epsilon_i\)

To get an estimate for \(\theta \) without collecting data on \(X\).