%!TEX root = /home/renaud/Documents/EPL/tfe/latex/tfe.tex
\section{Extension to continuous time}

From a general viewpoint, the discrete process can be interpreted as an approximation of its continuous counterpart: whereas the state of the discrete-time random walker can only change at unit-time intervals, the continuous-time random walkers undergo a waiting time between each change of state which is itself a random variable. More precisely, the waiting time is a continuous memoryless random variable distributed exponentially. Obviously, the transition probabilities from one node to the other are the same for both discrete- and continuous-time processes, only the time at which the jump occurs may vary. The continuous-time process corresponding to~\eqref{eq:discreteMP} is governed by the following dynamics:
\begin{equation} \label{eq:continuousMP_general}
    	\dot{\b p} = \b p \diag\left\{\bs \lambda(\b q)\right\} \b Q^{-1} \b A - \b p \diag\left\{\bs \lambda(\b q)\right\} = -\b p \b L,
\end{equation}
where $\lambda_i(\b q)$ is the rate at which random walkers leave node $i$, and
\begin{equation}
	\b L = \diag\{\bs \lambda(\b q)\}[-\b Q^{-1} \b A + \b I].
\end{equation}
Two particular cases of this process are implemented by the stability software and are thus examined here, depending on the choice of $\bs \lambda(\b q)$: the so-called \textit{normalized Laplacian dynamics} and \textit{standard (combinatorial) Laplacian dynamics}. Their names come from the similarity that arises between $\b L$ and the normalized/standard Laplacian matrix. Each of those two dynamics represent best different physical processes. The former corresponds to the choice $\bs \lambda_{norm}(\b q) = \b 1$. Hence, the expected waiting time is $1$ at every node, and $\b L = -\b Q^{-1} \b A + \b I = -\b M + \b I$. The latter corresponds to $\bs \lambda_{combi}(\b q) = \b q/\langle \b q \rangle$. In that case, $\b L = (-\b A + \b Q)/\langle \b q \rangle$ and the average waiting time at node $i$ is $\langle \b q \rangle/q_i$. Hence, the expected waiting time at a given node is smaller (resp. larger) than $1$ if the total weight of the outgoing edges from that node is larger (resp. smaller) than the average total weight of the outgoing edges on the network. However, the expected waiting time over the whole network is $\langle \langle \b q \rangle/\b q \rangle = 1$.\label{expectedtimebetweenjumps} The corresponding governing equations are respectively
\begin{equation} \label{eq:continuousMP_norm}
	\dot{\b p} = \b p \b Q^{-1} \b A - \b p = \b p \b M - \b p
\end{equation}
for the normalized Laplacian and
\begin{equation} \label{eq:continuousMP_combi}
    	\dot{\b p} = \b p \frac{\b A}{\langle \b q \rangle} - \b p\frac{\b Q}{\langle \b q \rangle}
\end{equation}
for the combinatorial Laplacian.

The clustered autocovariance matrix for partition $\P$ at time $t$ is easily generalized to
\begin{equation}
	\b R(t;\P) = \b H_{\P}^{\t}(\b \Pi\b P(t) - \bs \pi^{\t}\bs \pi)\b H_{\P},
\end{equation}
where $\b P(t)$ is the the transition matrix of the process at time $t$: $\b P(t) = \Exp^{-t\b L}$. The continuous-time definition of the stability of a partition $\P$ follows almost straightforwardly:
\begin{equation}
	r(t;\P) = \trace \left[ \b R(t; \P) \right].
\end{equation}
Notice that it is not necessary to minimize over the time interval $[0,t]$: indeed, it can be shown that $\trace \left[ \b R(t;\P) \right]$ is monotonically decreasing with time. The interpretation in terms of a random walk is similar to the discrete case: let $P(\C,t)$ be the probability that a random walker is in community $\C$ at time $t$ if it was initially in $\C$, when the system is at stationarity. Discounting the probability of such an event to take place by chance at stationarity and summing over all communities of $\P$ leads to the definition of the stability of the partition $\P$:
\begin{equation} \label{eq:generalstability}
	r(t;\P) = \sum_{\C \in \P} P(\C,t) - P(\C,\infty).
\end{equation}
By ergodicity, the memory of the initial condition is lost at infinity and $P(\C,\infty)$ is thus equal to the probability that two independent walkers are in $\C$ at stationarity. Equation~\eqref{eq:generalstability} tells us that only the communities in which a random walker is likely to stay bring a positive contribution to stability, where \textit{likely to stay} means that the probability for a walker to be in its initial community at time $t$ is larger than the probability of that event occurring by chance at stationarity. The stability curve of the graph can now be expressed as a continuous function of $t$:
\begin{equation}
	r(t) = \max_{\P} r(t;\P).
\end{equation}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tikz}
\setlength{\parindent}{0pt}

\newtheorem*{theorem}{Theorem}
\newtheorem*{definition}{Definition}
\newtheorem*{lemma}{Lemma}
\newtheorem*{corollary}{Corollary}
\newtheorem{example}{Example}
\newtheorem*{trick}{Trick}
\newtheorem*{question}{Question}

\title{Lecture 18: Change of Variables}
\author{}
\date{}

\begin{document}

\maketitle

\section{Introduction to Change of Variables in Double Integrals}

\begin{example}
  Calculate the area of the ellipse with semiaxes $a$ and $b$.

  We can use double integrals to calculate the area. By definition of double
  integrals, the area of the described ellipse
  \begin{equation*}
    \begin{split}
      A &= \iint_R 1 dA \\
        &= \iint_{(\frac{x}{a})^2 + (\frac{y}{b})^2 = 1} dx dy \\
    \end{split}
  \end{equation*}

  To simplify the equation, we can set
  \begin{gather*}
    u = \frac{x}{a} \\
    v = \frac{y}{b} \\
  \end{gather*}
  Then
  \begin{gather*}
    du = \frac{dx}{a} \\
    dv = \frac{dy}{b} \\
    du dv = \frac{dx dy}{ab} \\
    dx dy = ab du dv \\
  \end{gather*}

  Therefore, the equation of the double integral is equivalent to
  \begin{equation*}
    \begin{split}
      A &= \iint_{(\frac{x}{a})^2 + (\frac{y}{b})^2 = 1} dx dy \\
        &= \iint_{u^2 + v^2 = 1} ab du dv \\
        &= ab \iint_{u^2 + v^2 = 1} du dv \\
        &= \pi ab \\
    \end{split}
  \end{equation*}

  Therefore, the area of the ellipse with semiaxes $a$ and $b$ is $\pi ab$.
\end{example}

From the above example we can see, in general when we change the variables in a
double integral, we need to find the scaling factor between $dA = dx dy$ and
$dA' = du dv$. In order to ensure the correctness of the double integral after
changing the variables, we need to multiply the scaling factor with
differentials of changed variables, to ensure the differential of area is still
$dA$ rather than $dA'$. Also note that the scaling factor between $dA$ and $dA'$
might not be constant in the region $R$.

\begin{question}
  Why when we make sure $dA$ is correctly expressed by $du dv$ the double
  integral with change of variables yields the correct result?

  Answer:

  Probably think about it from the definition of double integral, and the
  geometric interpretation of change of variables.
\end{question}

\section{Change of Variables with Linear Transformation}

\begin{example}
  Find the scaling factor between $dA = dx dy$ and $dA' = du dv$ with the
  following linear transformation:
  \begin{equation*}
    \begin{cases}
      u = 3x - 2y \\
      v = x + y \\
    \end{cases}
  \end{equation*}

  In the $xy$-coordinates, $dA = dx dy$ represents a small rectangle. Regarding
  the geometric interpretation of the linear transformation, we have the
  following two observations:
  \begin{itemize}
    \item A rectangle in the $xy$-coordinates corresponds to a parallelogram in
    the $uv$-coordinates. The ratio of the area of the parallelogram in the
    $uv$-coordinates to the area of the rectangle in the $xy$-coordinates is the
    area scaling factor.
    \item The area scaling factor of the linear transformation doesn't depend on
    the choice of rectangles, either the position or the area.
  \end{itemize}

  Based on the above two observations, we can study a special case to find the
  scaling factor of the linear transformation. The special case we choose is a
  unit rectangle with the left bottom corner on the origin.

  \begin{tikzpicture}
    [help line/.style={dashed}]
    \draw[->] (-1, 0) -- (2, 0) node[right] {x};
    \draw[->] (0, -1) -- (0, 2) node[right] {y};
    \draw (0, 0) node[below right] {O};
    \draw[-] (0, 0) -- (0, 1) -- (1, 1) -- (1, 0) -- (0, 0);
  \end{tikzpicture}

  After the linear transformation,
  \begin{gather*}
    (0, 0) \rightarrow (0, 0) \\
    (1, 0) \rightarrow (3, 1) \\
    (1, 1) \rightarrow (1, 2) \\
    (0, 1) \rightarrow (-2, 1) \\
  \end{gather*}

  \begin{tikzpicture}
    [help line/.style={dashed}]
    \draw[->] (-2, 0) -- (3, 0) node[right] {u};
    \draw[->] (0, -1) -- (0, 3) node[right] {v};
    \draw (0, 0) node[below right] {O};
    \draw[-] (0, 0) -- (3, 1) -- (1, 2) -- (-2, 1) -- (0, 0);
  \end{tikzpicture}

  We can use the determinant to calculate the area of a parallelogram in a plane
  \begin{equation*}
    \begin{split}
      A &= \det(<3, 1>, <-2, 1>) \\
        &= \begin{vmatrix}
             3 & 1 \\
             -2 & 1 \\
           \end{vmatrix} \\
        &= 5 \\
    \end{split}
  \end{equation*}

  Therefore, the scaling factor of the linear transformation is $5$:
  \begin{gather*}
    dA' = 5dA \\
    dA = \frac{1}{5} dA' \\
    dx dy = \frac{1}{5} du dv \\
  \end{gather*}

\end{example}

\begin{question}
  Why the scaling factor in the above example is $5$ rather than $\frac{1}{5}$?

  Answer:
\end{question}

\section{Change of Variables in General}

For general cases of change of variables:
\begin{gather*}
  u = u(x, y) \\
  v = v(x, y) \\
\end{gather*}

By linear approximation, we have
\begin{gather*}
  \Delta u \approx u_x \Delta x + u_y \Delta y \\
  \Delta v \approx v_x \Delta x + v_y \Delta y \\
  \begin{bmatrix}
    \Delta u \\
    \Delta v \\
  \end{bmatrix} \approx
  \begin{bmatrix}
    u_x & u_y \\
    v_x & v_y \\
  \end{bmatrix}
  \begin{bmatrix}
    \Delta x \\
    \Delta y \\
  \end{bmatrix} \\
\end{gather*}

Therefore, for any transformation or change of variables, a small rectangle in
the original coordinate system corresponds to a parallelogram in the new
coordinate system.

The correspondence of the vertices between the small rectange in the original
coordinate system and the parallelogram in the new coordinate system is
\begin{gather*}
  (0, 0) \rightarrow (0, 0) \\
  (\Delta x, 0) \rightarrow (u_x \Delta x, v_x \Delta x) \\
  (0, \Delta y) \rightarrow (u_y \Delta y, v_y \Delta y) \\
  (\Delta x, \Delta y) \rightarrow (u_x \Delta x + u_y \Delta y, v_x \Delta x + v_y \Delta y) \\
\end{gather*}

Therefore, the area of the parallelogram is
\begin{equation*}
  \begin{split}
    A &= \det(<u_x \Delta x, v_x \Delta x>, <u_y \Delta y, v_y \Delta y>) \\
      &= \begin{vmatrix}
           u_x \Delta x & v_x \Delta x \\
           u_y \Delta y & v_y \Delta y \\
         \end{vmatrix} \\
      &= (u_x v_y - u_y v_x) \Delta x \Delta y \\
      &= \begin{vmatrix}
           u_x & u_y \\
           v_x & v_y \\
         \end{vmatrix} \Delta x \Delta y \\
  \end{split}
\end{equation*}

Therefore, the scaling factor between two coordinates is
$\begin{vmatrix}
   u_x & u_y \\
   v_x & v_y \\
\end{vmatrix}$. Since the scaling factor consists of partial derivatives, it
varies with different positions of the differential of area.

Here we define the Jacobian determinant as
\begin{equation*}
  J = \frac{\partial(u, v)}{\partial(x, y)} = \begin{vmatrix}
                                                u_x & u_y \\
                                                v_x & v_y \\
                                              \end{vmatrix}
\end{equation*}
Then
\begin{equation*}
  du dv = |J| dx dy = |\frac{\partial(u, v)}{\partial(x, y)}| dx dy
\end{equation*}

\begin{example}
  Find the scaling factor between the Cartesian coordinates and the polar
  coordinates.

  According to the definition of the polar coordinates
  \begin{gather*}
    x = r \cos\theta \\
    y = r \sin\theta \\
  \end{gather*}

  Then we can find the Jacobian determinant as
  \begin{equation*}
    \begin{split}
      J &= \frac{\partial(x, y)}{\partial(r, \theta)} \\
        &= \begin{vmatrix}
             \frac{\partial x}{\partial r} & \frac{\partial x}{\partial \theta} \\
             \frac{\partial y}{\partial r} & \frac{\partial y}{\partial \theta} \\
           \end{vmatrix} \\
        &= \begin{vmatrix}
             \cos\theta & -r \sin\theta \\
             \sin\theta & r \cos\theta \\
           \end{vmatrix} \\
        &= r \cos^2\theta + r \sin^2\theta \\
        &= r \\
    \end{split}
  \end{equation*}

  Therefore,
  \begin{equation*}
    dx dy = |\frac{\partial(x, y)}{\partial(r, \theta)}| dr d\theta = r dr d\theta
  \end{equation*}
\end{example}

Note that the Jacobian determinant of a change of variables works in both ways,
which means
\begin{gather*}
  J_1 = \frac{\partial(x, y)}{\partial(u, v)} = \frac{dx dy}{du dv} \\
  J_2 = \frac{\partial(u, v)}{\partial(x, y)} = \frac{du dv}{dx dy} \\
  \frac{dx dy}{du dv} \cdot \frac{du dv}{dx dy} = 1 \\
  \frac{\partial(x, y)}{\partial(u, v)} \cdot \frac{\partial(u, v)}{\partial(x, y)} = 1 \\
\end{gather*}

Therefore, the two Jacobian determinants are inverse to each other, and we can
choose the easier one to calculate.

\begin{example}
  Calculate $\int_0^1 \int_0^1 x^2 y dx dy$ by changing the variables as
  \begin{gather*}
    u = x \\
    v = xy \\
  \end{gather*}

  First we need to calculate the differential of area.
  \begin{gather*}
    \begin{split}
      J &= \frac{\partial(u, v)}{\partial(x, y)} \\
        &= \begin{vmatrix}
             1 & 0 \\
             y & x \\
           \end{vmatrix} \\
        &= x \\
    \end{split} \\
    \begin{split}
      du dv &= |J| dx dy \\
            &= |x| dx dy \\
            &= x dx dy \ (x \geq 0)
    \end{split} \\
  \end{gather*}

  Then we need to calculate the integrand
  \begin{equation*}
    \begin{split}
      x^2 y dx dy &= x^2 y \frac{1}{x} du dv \\
                  &= xy du dv \\
                  &= v du dv \\
    \end{split}
  \end{equation*}

  Then we need to determine the region of the double integral in the new
  coordinates.

  In the $xy$-coordinates, the region of the double integral is a rectangle as
  shown in the below diagram.

  \begin{tikzpicture}
    [help line/.style={dashed}]
    \draw[->] (-1, 0) -- (2, 0) node[right] {x};
    \draw[->] (0, -1) -- (0, 2) node[right] {y};
    \draw (0, 0) node[below right] {O};
    \draw[-] (0, 0) -- (1, 0) -- (1, 1) -- (0, 1) -- (0, 0);
  \end{tikzpicture}

  After changing the variables, if we calculate the double integral in the order
  of $du dv$, it means for each possible value of $v$, we will keep it constant,
  and vary the value of $u$.

  Therefore, the way we determine the boundary of the region is to find the
  range of $v$, and for each value of $v$, find the possible values of $u$ based
  on the region in $xy$-coordinates.

  According to the diagram, we can find that the range of $v$ is $(0, 1)$, and
  for each value of $v$, the range of $u$ is $(v, 1)$.

  Therefore, the double integral after changing the variables is
  \begin{equation*}
    \begin{split}
      \int_0^1 \int_0^1 x^2 y dx dy &= \int_0^1 \int_v^1 v du dv \\
                                    &= \int_0^1 (uv)|_{u = v}^{u = 1} dv \\
                                    &= \int_0^1 v - v^2 dv \\
                                    &= (\frac{v^2}{2} - \frac{v^3}{3})|_0^1 \\
                                    &= \frac{1}{6} \\
    \end{split}
  \end{equation*}
\end{example}

\end{document}


% !TEX root = main.tex

\chapter[Holomorphic Functions]{Differentiability and Holomorphic Functions}
\section{The Derivative of a Complex Function}
Throughout this section, let $U$ be an open subset of $\C$.  The definition of differentiability for a function $f: U \to \C$ looks almost identical to the corresponding definition for real functions.

\begin{definition}
Let $f: U \to \C$ be a function, and let $z \in U$.  We say that $f$ is \emph{differentiable at $z$} if the limit
\begin{equation}
\label{e:diff}
\lim_{\substack{h \to 0\\ h \in \C \backslash \set{0}}} \frac{f(z+h)-f(z)}{h}
\end{equation}
exists.  When it does, we denote its value by $f'(z)$.

%If $f$ is differentiable at $w$ for all $w \in U$, we say that $f$ is \emph{holomorphic on $U$}.
\end{definition}

Since $U$ is open, for a fixed $z \in U$, the quantity (the \emph{difference quotient})
\[
\frac{f(z+h)-f(z)}{h}
\]
is defined whenever $h$ is `sufficiently small' (to ensure that $z+h \in U$ and so $f(z+h)$ is defined) but not zero (to avoid division by zero). In other words, $0$ is a limit point of the domain of the difference quotient (regarded as a function of $h$).

 For this reason, we can again omit the second subscript and write~\eqref{e:diff} as
\[
\lim_{h \to 0} \frac{f(z+h)-f(z)}{h}
\]
without ambiguity.

\begin{example}
\label{e:diff1}
Let us investigate whether or not the function
\[
f:\C \to \C, \quad f(z)=-i\conj{z}
\]
is differentiable at any points $z \in \C$.
\end{example}

%1 page handwritten.
\begin{solution}
Fix $z \in \C$. To evaluate the limit
\[
\lim_{h \to 0} \frac{f(z+h)-f(z)}{h},
\]
we first try to simplify the expression
\[
\frac{f(z+h)-f(z)}{h}.
\]
For any $z \in \C \backslash \set{0}$,
\[
\frac{f(z+h)-f(z)}{h} = \frac{-i\ \conj{(z+h)}-(-i)\conj{z}}{h} = -i \frac{\conj{h}}{h}.
\]
%\vspace*{8cm}
But then the limit
\[
\lim_{h \to 0} \frac{f(z+h)-f(z)}{h} = \lim_{h \to 0} -i \frac{\conj{h}}{h},
\]
is the same as the limit that we considered in Example~\ref{e:rlim}, except scaled by a factor of $-i$.  Hence this limit does not exist for the same reason; that is
\[
\rlim{h \to 0}{h \in \R \backslash \set{0}} -i \frac{\conj{h}}{h} = -i, \text{ while } \rlim{h \to 0}{h \in i\R \backslash \set{0}} -i \frac{\conj{h}}{h} = i.
\]
So $f$ is not differentiable at any point $z \in \C$.
\end{solution}

\begin{example}
\label{e:diff2}
The function $\mathbf{g}: \R^2 \to \R^2$ is defined by
\[
\mathbf{g}(x,y)=(1,-x^2-y^2)\quad (x,y) \in \R^2.
\]
Find the corresponding complex function $g:\C \to \C$ and investigate where it is differentiable.
\end{example}

\begin{solution}
The corresponding function $g: \C \to \C$ is
\[
g(x+iy) = 1+i(-x^2-y^2),
\]
or equivalently,
\[
g(z) = 1-i z \conj{z}.
\]
Again, we fix $z \in \C$ and look at the difference quotient
\[
\frac{g(z+h)-g(z)}{h} = -i(z \frac{\conj{h}}{h}+\conj{z}+\conj{h})
\]
for $h \in \C \backslash \set{0}$.\\

As with the previous example, we have $\frac{\conj{h}}{h}$ appearing in the difference quotient, so it looks like our limit will not exist.  Again, we shall check by evaluating the restricted limits along the real and imaginary axes.

\begin{align*}
\rlim{h \to 0}{h \in \R \backslash \set{0}} -i \left( z \frac{\conj{h}}{h} + \conj{z} + \conj{h} \right) & = -i(z + \conj{z} ) \\
\rlim{h \to 0}{h \in i\R \backslash \set{0}} -i \left( z \frac{\conj{h}}{h} + \conj{z} + \conj{h} \right) & = -i (-z+\conj{z} ).
\end{align*}
These restricted limits are equal if and only if $z+\conj{z} = -z + \conj{z}$, which occurs if and only if $z=0$.  It follows that the unrestricted limit
\[
\lim_{h \to 0} \frac{g(z+h)-g(z)}{h}
\]
does not exist at any $z \in \C \backslash \set{0}$, hence $g$ is not differentiable at any of these points.

What about when $z=0$?  While the restricted limits along the real and imaginary axes are equal, this is \emph{not} enough to conclude that the unrestricted limit exists.  Indeed, we must verify this directly:
\[
\lim_{h \to 0} \frac{g(0+h)-g(0)}{h} = \lim_{h \to 0} -i \conj{h} = 0.
\]
(Here we have used the fact that $z \mapsto \conj{z}$ is continuous (Example~\ref{e:cts}), and so limit can be found by substitution of $z=0$.)  In other words, $g$ is differentiable at $0$ (and nowhere else), with $g'(0)=0$.
%\vspace*{15cm}
\end{solution}
\begin{note}
In Example~\ref{e:diff2}, both the real and the imaginary parts of $g$ are differentiable with respect to both $x$ and $y$ everywhere in $\R^2$.
\end{note}

\begin{example}
\label{e:diff3}
Consider the function $f$ defined by $f(z)=\dfrac{1}{z}$.  At which points $z \in \C$ is $f$ differentiable?
\end{example}

\begin{solution}
Since the domain of $f$ is the (open) set $\C \backslash \set{0}$, we must necessarily restrict our attention to points $z$ in this set.  Then with $h \neq 0$, we have
\[
\frac{f(z+h)-f(z)}{h} = \frac{\left( \frac{1}{z+h} \right) - \frac{1}{z}}{h} = \frac{-1}{z(z+h)}.
\]
Since by assumption $z \neq 0$, the algebra of limits (Proposition~\ref{p:alglimits}) tells us that $-\dfrac{1}{z(z+h)} \to - \dfrac{1}{z^2}$ as $h \to 0$.  Thus $f$ is differentiable at every $z \in \C \backslash \set{0}$, with $f'(z) = -\dfrac{1}{z^2}$.
%\vspace*{6cm}
\end{solution}

\begin{example}
\label{e:diff4}
Fix $\alpha \in \C$ and let $f$ be the constant function $f(z) = \alpha$ for all $z \in \C$.  Then $f$ is differentiable at every point $z \in \C$, and satisfies $f'(z)=0$ for all $z \in \C$.
\end{example}
\begin{solution}
For any $z\in \C$, and $h \in \C \backslash \set{0}$,
\[
\frac{f(z+h)-f(z)}{h} = \frac{\alpha-\alpha}{h}=0 \to 0 \text{ as } h \to 0.
\]
\end{solution}

\begin{question}
If $f:U \to \C$ is differentiable at every point $z$ in $U$, and has $f'(z)=0$ for all $z$, is $f$ necessarily constant?
\end{question}

\begin{answer}
Not necessarily.  If $U=U_1 \cup U_2$ where $U_1$ and $U_2$ are disjoint open subsets of $\C$, and $f:U \to \C$ is defined by
\[
f(z) = \begin{cases}
0 & \text{ if } z \in U_1 \\
1& \text{ if } z \in U_2,
\end{cases}
\]
then $f'(z)$ exists and is equal to $0$ at every $z \in U$, but $f$ is non-constant.
\end{answer}
%\vspace*{5cm}

\begin{example}
\label{e:diff5}
Fix $\beta \in \C$ and let $f(z)=\beta z$.  Then $f'(z_0) = \beta$ for all $z_0 \in \C$.
\end{example}

\begin{comment}
\begin{solution}
\[
\frac{f(z+h)-f(z)}{h} = \frac{\beta(z+h)-\beta z}{h}=\beta  \to 0 \text{ as } h \to 0.
\]
\end{solution}
\end{comment}

\section{Holomorphic Functions}

\begin{definition}
Let $U \subseteq \C$ be an open set and $f:U \to \C$ a complex function.  If $f$ is differentiable at $z$ for all $z \in U$, we say that $f$ is \emph{holomorphic on $U$}.
\end{definition}
Why \emph{holomorphic} and not simply \emph{differentiable} on $U$?  One reason for this is that there are many functions $f$ that are differentiable on the real line, but fail to be differentiable throughout any open subset $U \subseteq \C$.

\begin{definition}
Let $f:U \to \C$ be a complex function and $z \in U$.  If there is some $r>0$ such that $f$ is differentiable at every point in the open disc $D(z,r)$, then $f$ is said to be \emph{holomorphic at $z$}.
\end{definition}

If a function $f$ is holomorphic at $z$, then it is differentiable at $z$, but the converse is not necessarily true.

\begin{example}
Let us examine whether or not the functions from Examples~\ref{e:diff1},~\ref{e:diff2} and~\ref{e:diff3} are holomorphic on any subsets of $\C$.
\begin{blankbox}
\begin{itemize}
\item $f(z)=-i\conj{z}$ is not differentiable at any $z \in \C$ and therefore not holomorphic on any $U \subseteq \C$.
%\vspace*{2cm}
\item $f(z) = 1-i z \conj{z}$ is differentiable at $0$ and nowhere else.  Since the one-point set $\set{0}$ is not an open set, there is no open subset $U \subseteq \C$ with $f$ holomorphic on $U$.
%\vspace*{3cm}
\item $f(z) = \dfrac{1}{z}$ is differentiable at every point of the (open) set $\C \backslash \set{0}$ and is therefore holomorphic on this set.
%\vspace*{3cm}
\end{itemize}
\end{blankbox}
\end{example}

If $f:U \to \C$ is holomorphic on $U$, then we get a new function $f':U \to \C, \ z \mapsto f'(z)$, called the \emph{derivative of $f$}.  In other words, for $z \in U$, $f'(z)$ is defined as the limit
\[
f'(z) = \lim_{\substack{h \to 0 \\ h \in \C \backslash \set{0}}} \frac{f(z+h)-f(z)}{h}.
\]

As for differentiable functions in $\R$, we have sum, product, chain and quotient rules for holomorphic functions defined on open subsets of $\C$.

\begin{theorem}[Rules of Differentiation; proof non-examinable]
\label{t:diffrules}
Let $U$ and $V$ be open subsets of $\C$ and let $f:U \to \C$ and $g:V \to \C$ be holomorphic on $U$ and $V$ respectively.  Then
\begin{enumerate}
\item (Sum rule) $f+g$ is homomorphic on $U \cap V$ and $(f+g)'(z)=f'(z)+g'(z)$
\item (Scalar Multiples) For any $\alpha \in \C$, $(\alpha f )$ is holomorphic on $U$ and $(\alpha f)' (z) = \alpha f'(z)$
\item (Product Rule) $fg$ is holomorphic on $U \cap V$ and $(fg)'(z)=f'(z)g(z)+g'(z)f(z)$
\item (Quotient Rule) The quotient $f/g$ is holomorphic on $U \cap \set{ z \in V: g(z) \neq 0}$ and
\[
\left( \frac{f}{g} \right) '(z) = \frac{f'(z)g(z)-f(z)g'(z)}{g(z)^2}
\]
\item (Chain Rule) $f \circ g$ is holomophic on $V \cap g^{-1}(U)$ and $(f\circ g)'(z) = f'(g(z))g'(z).$
\end{enumerate}
\end{theorem}

%\vspace*{7cm}
\begin{comment}
Since we only ever speak of a function being holomorphic on an \emph{open} subset $U \subseteq \C$, the statement of Theorem~\ref{t:diffrules} implicitly relies on the assumption that $U \cap V$, $U \cap \set{ z \in V: g(z) \neq 0 }$ and $V \cap g^{-1} (U)$ are all open whenever $U$ and $V$ are open and $f$ and $g$ are holomorphic.  These are all relatively easy to prove (the second and third rely on Proposition~\ref{p:diffimpliescontinuous}).

The proof of Theorem~\ref{t:diffrules} is very similar to that of the corresponding result for functions of a real variable, and is thus omitted.
\end{comment}
\begin{example}
 Let $p:\C \to \C$ be a complex polynomial
\[
p(z) = \alpha_0 + \alpha_1z + \ldots + \alpha_n z^n.
\]
Then $p$ is holomorphic on $\C$.
\end{example}

\begin{solution}
We saw in Examples~\ref{e:diff4} and~\ref{e:diff5} that the functions $f(z) = \beta z$ and $g(z)= \alpha$ (where $\alpha, \beta \in \C$ are fixed), are holomorphic on $\C$ with derivatives $f'(z) = \beta$ and $g'(z)=0$ for all $z \in \C$.  Together with Theorem~\ref{t:diffrules}, this shows that $p(z)$ is holomorphic on $\C$ with derivative
\[
p'(z) = \alpha_1 + 2\alpha_2 z + \ldots + n \alpha_n z^{n-1}.
\]
Thus complex polynomials can be differentiated using exactly the same rules as for real polynomials
\end{solution}
\begin{note}
A similar argument shows that if $g(z) = \dfrac{1}{z^n}$ where $n >0$, then $g'(z) = -\dfrac{n}{z^{n+1}}$.
\end{note}
\begin{example}
The complex functions $\exp,\ \sin$ and $\cos$ defined in Chapter 1 are also holomorphic on $\C$, with derivatives
\[
\left( \exp (z) \right)'=\exp(z) \quad \left( \sin (z) \right)' = \cos (z) \quad\text{and}\quad \left( \cos(z) \right)' = - \sin(z).
\]
\end{example}

%\vspace*{6cm}

\begin{proposition}[Proof non-examinable]
\label{p:diffimpliescontinuous}
Let $f: U \to \C$ be differentiable at a point $z \in U$.  Then $f$ is continuous at $z$.
\end{proposition}

\begin{question} If $f$ is holomophic on $U$, is $f'$ holomorphic on $U$?
\end{question}
\begin{example}
In real analysis, consider $g:\R \to \R$ defined by
\[
g(x) = \begin{cases}
-x^2 & x<0 \\
x^2 & x\geq 0.
\end{cases}
\]
\begin{blankbox}
It is easily shown that $g$ is differentiable on $\R$ with $g'(x) = 2 \abs{x}$ for all $x \in \R$.  However, we know that $x \mapsto 2 \abs{x}$ fails to be differentiable at $0$.

In fact for holomorphic functions, the answer is yes.  This gives us an even stronger result: if $f:U \to \C$ is holomorphic on $U$, then $f$ is infinitely differentiable on $U$.  We shall return to this later on in the module.
\end{blankbox}
\end{example}

\section{The Cauchy Riemann Equations}
Again, let $U \subseteq \C$ be open.  Suppose we are given $f:U \to \C$, then we have seen how to write $f$ as a sum of its real and imaginary parts:

\[ f(x+iy)=u(x,y)+iv(x,y) \]
for all $z=x+iy \in U$.  \

\begin{comment}
Written in terms of $z$, we have
\[
f(z) = \underbrace{\frac{f(z)+\conj{f(z)}}{2}}_{\text{Real}} + i \underbrace{\left(\frac{f(z)-\conj{f(z)}}{2i} \right)}_{\text{Real}}
\]
\end{comment}

\begin{question}
If $u$ and $v$ are differentiable with respect to both $x$ and $y$, does it follow that $f$ is holomorphic?
\end{question}

%\vspace*{2cm}
\begin{answer}
No.  We saw in Example~\ref{e:diff1} that
\[
f(x+iy) = 1-i (x^2+y^2),
\]
whose real and imaginary parts are differentiable everywhere in $\R^2$ with respect to both $x$ and $y$, is not differentiable at any point $z \in \C \backslash \set{0}$.
\end{answer}

For clarity, let us recall the definition of partial derivatives.

\begin{definition}
Suppose that $u: \R^2 \to \R$ is a function.  Then the \emph{partial derivatives} of $u$ with respect to $x$ and $y$ are the functions $\frac{\partial u}{\partial x}$ and $\frac{\partial u}{\partial y}$ respectively, defined via
\begin{align*}
\frac{\partial u}{\partial x} (x_0,y_0):&= \lim_{\substack{h\to 0\\h \in \R \backslash \set{0}}} \frac{u(x_0+h,y_0)-u(x_0,y_0)}{h}\\
\frac{\partial u}{\partial y} (x_0,y_0):&= \lim_{\substack{h\to 0\\h \in \R \backslash \set{0}}} \frac{u(x_0,y_0+h)-u(x_0,y_0)}{h}
\end{align*}
at the points $(x_0,y_0)$ where the limits exist.
\end{definition}

\begin{theorem}[Differentiability Implies the Cauchy-Riemann Equations]
\label{t:cr1}

Let $f$ be a complex-valued function defined on some open set $U$ and let $z_0=x_0+iy_0 \in U$.  Write
\[
f(x+iy) = u(x,y)+iv(x,y).
\]
Then if $f$ is differentiable at the point $z_0$ we have the following:

\begin{enumerate}
\item[(i)] The partial derivatives $\pd{u}{x}, \pd{u}{y}, \pd{v}{x}, \pd{v}{y}$ all exist at the point $(x_0,y_0) \in \R^2$ corresponding to $z_0$.
\item[(ii)] The partial derivatives satisfy the Cauchy-Riemann Equations at $(x_0,y_0)$:
\begin{equation}
\label{eq:cr}
\pd{u}{x} (x_0,y_0) = \pd{v}{y} (x_0,y_0),\quad \pd{u}{y} (x_0,y_0) = - \pd{v}{x} (x_0,y_0)
\end{equation}
\item[(iii)] At this point  the derivative of $f$ satisfies
\[
f'(z_0)= \pd{u}{x} (x_0,y_0) + i \pd{v}{x} (x_0,y_0) = \pd{v}{y} (x_0,y_0) - i \pd{u}{y} (x_0,y_0).
\]
\end{enumerate}
\end{theorem}

Before proving Theorem~\ref{t:cr1}, let us look at some examples that demonstrate how it is a powerful result.

\begin{example}
 Verify that the Cauchy-Riemann equations are satisfied by the function  $f(z)=z^2$.\\
\end{example}
\begin{solution}
Here we have
\[
f(x+iy) = \underbrace{x^2-y^2}_{u(x,y)} + i \underbrace{2xy}_{v(x,y)},
\]
and hence
\[
\pd{u}{x} = 2x = \pd{v}{y} \text{ and } \pd{u}{y} = -2y = - \pd{v}{x}.
\]
Thus~\eqref{eq:cr} holds at every $(x,y) \in \R^2$.  Moreover, we have
\[
\pd{u}{x} + i \pd{v}{x} = 2x+i2y = 2(x+iy) = 2z = f'(z),
\]
as expected.
\end{solution}
\begin{example} Use the Cauchy-Riemann equations to investigate the differentiability of $f(z)=\conj{z}$.
\end{example}

\begin{solution}
This time
\[
f(x+iy) = \underbrace{x}_{u(x,y)} + i \underbrace{-y}_{v(x,y)},
\]
so that $\pd{u}{x} = 1$ while $\pd{v}{y}=-1$ and~\eqref{eq:cr} does not hold at any point $(x,y) \in \R^2$ (note that we need both of the equations in~\eqref{eq:cr} to hold).
 Thus we conclude that $f(z) = \conj{z}$ is not differentiable at any point $ z \in \C$.
 \end{solution}

\begin{exercise}
Before we cover it in lectures, try to prove Theorem~\ref{t:cr1} yourself as follows:
\begin{enumerate}
\item[(i)] Assuming that $f$ is differentiable at $z_0$, what can be said about the restricted limits
\[
\rlim{h \to 0}{h \in \R \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h} \quad\text{and}\quad \rlim{h \to 0}{h \in \C \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h} \quad ?
\]
\item[(ii)] Write the restricted limits from (i) in terms of (the partial derivatives of) $u$ and $v$.  The result should follow easily.
\end{enumerate}
\end{exercise}
\begin{proof}[Proof of Theorem~\ref{t:cr1}]
Since $f$ is differentiable at $z_0$ we know that
\[
\rlim{h\to 0}{h \in \C \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h}
\]
exists and is equal to $f'(z_0)$.  In particular, all of corresponding restricted limits exist, and are also equal to $f'(z_0)$.  As before, we shall examine the restricted limits along the real and imaginary axes.

If we restrict $h$ to the nonzero real axis, then
\[
z_0+h = (x_0+h)+iy_0,
\]
which corresponds to the point $(x_0+h,y_0) \in \R^2$.  Thus the restricted limit satisfies:

%\vspace*{10cm}
\begin{align*}
f'(z_0) & = \rlim{h \to 0}{h \in \R \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h} \\
& = \rlim{h \to 0}{h \in \R \backslash \set{0}} \frac{\left[ u(x_0+h,y_0)+iv(x_0+h,y_0) \right] - \left[ u(x_0,y_0)+iv(x_0,y_0) \right]}{h} \\
& \vspace*{2cm} \\
& = \left( \rlim{h \to 0}{h \in \R \backslash \set{0}} \frac{u(x_0+h,y_0)-u(x_0,y_0)}{h}  \right) +i \left(  \rlim{h \to 0}{h \in \R \backslash \set{0}} \frac{v(x_0+h,y_0)-v(x_0,y_0)}{h} \right)  \\
& = \pd{u}{x} (x_0,y_0)+i \pd{v}{x} (x_0,y_0).
\end{align*}
Hence both $\pd{u}{x}$ and $\pd{v}{x}$ exist at the point $(x_0,y_0)$, and the derivative of $f$ at $z_0$ satisfies
\begin{equation}
\label{eq:cr1}
\tag{$\dagger$}
f'(z_0) = f'(x_0+iy_0) = \pd{u}{x} (x_0,y_0)+i \pd{v}{x} (x_0,y_0).
\end{equation}
%\vspace*{7cm}

We shall now examine the restricted limit along the nonzero imaginary axis. This limit must also satisfy
\[
f'(z_0) = \rlim{h\to 0}{h \in i \R \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h}.
\]
%\vspace*{3cm}
If $h \in i \R \backslash \set{0}$, then $h=0+ik$ for some $k \in \R \backslash \set{0}$, and so
\[
z_0+h=x_0+i(y_0+k),
\]
which corresponds to the point  $(x_0,y_0+k) \in \R^2$.

Thus
\begin{align*}
f'(z_0) & = \rlim{h \to 0}{h \in i\R \backslash \set{0}} \frac{f(z_0+h)-f(z_0)}{h} \\
& = \rlim{k \to 0}{k \in \R \backslash \set{0}} \frac{\left[ u(x_0,y_0+k)+iv(x_0,y_0+k) \right] - \left[ u(x_0,y_0)+iv(x_0,y_0) \right]}{ik} \\
& \vspace*{2cm} \\
& =   \rlim{k \to 0}{k \in \R \backslash \set{0}} \frac{i\ \left[ v(x_0,y_0+k)-v(x_0,y_0) \right]}{ik} +
 \rlim{k \to 0}{k \in \R \backslash \set{0}} \frac{u(x_0,y_0+k)-u(x_0,y_0)}{ik}     \\[5ex]
 & =   \rlim{k \to 0}{k \in \R \backslash \set{0}} \frac{v(x_0,y_0+k)-v(x_0,y_0)}{k}
 -i \left( \rlim{k \to 0}{k \in \R \backslash \set{0}} \frac{u(x_0,y_0+k)-u(x_0,y_0)}{k} \right)     \\
 & = \pd{v}{y} (x_0,y_0)-i \pd{u}{y} (x_0,y_0).
\end{align*}
So $\pd{u}{y}$ and $\pd{v}{y}$ also exist at $(x_0,y_0)$, and the derivative of $f$ at $z_0$ also satisfies
\begin{equation}
\label{eq:cr2}
\tag{$\ddagger$}
f'(z_0) = f'(x_0+iy_0) = \pd{v}{y} (x_0,y_0) - i \pd{u}{y} (x_0,y_0).
\end{equation}
Equating the real and imaginary parts of the two expressions~\eqref{eq:cr1} and~\eqref{eq:cr2} for $f'(z_0)$ gives
\[
\pd{u}{x} (x_0,y_0) = \pd{v}{y} (x_0,y_0) \text{ and } \pd{v}{x}(x_0,y_0) = - \pd {u}{y} (x_0,y_0),
\]
which completes the proof.
%\vspace*{15cm}
\end{proof}
The Cauchy Riemann Equations provide a very useful way of showing that a function is \emph{not} holomorphic.  We cannot use Theorem~\ref{t:cr1} to show that a function is holomorphic.
\begin{example}
For the function $f: \C \to \C$ defined by
\[
f(z) =
\begin{cases}
\exp(-z^{-4}) & z \neq 0 \\
0 & z=0
\end{cases}
\]
the partial derivatives of $u$ and $v$ satisfy the Cauchy-Riemann equations everywhere, but $f$ is not differentiable (nor even continuous) at $z=0$.  (This is difficult to prove).
\end{example}
\begin{comment}
\begin{remark}
A partial converse does exist: if $f:U \to \C$ is given by
\[
f(x+iy)=u(x,y)+iv(x,y)
\]
and if at all points of $U$, the partial derivatives of $u$ and $v$ (a) exist (b) satisfy the Cauchy Riemann equations and (c) are continuous, then $f$ is holomophic on $U$.  This is far more difficult to prove (and you are forbidden from using it in this module!).
\end{remark}
\begin{remark} (Note: completely irrelevant for this module)
Let $f:U \to \C$ be given and let $\mathbf{f}:U \to \R^2$ denote the corresponding real function.  If $f$ satisfies the Cauchy-Riemann equations, then $\mathbf{f}$ satisfies the \emph{Laplace Equation}
\[
\pd{{}^2\mathbf{f}}{x^2} + \pd{{}^2\mathbf{f}}{y^2}=0,
\]
and important partial differential equation that shows up in many branches of mathematics and physics.  Thus the study of holomorphic functions is closely related to the study of the Laplace equation.  Indeed, solutions to this PDE are known as \emph{harmonic functions}, and there is an entire branch of mathematics dedicated to their study.
\end{remark}
\end{comment}

\section{Geometry of Derivatives for Complex-Valued Functions}
When working with differentiable functions in $\R$, it is useful to have the geometric picture of the derivative of a function $g: \R \to \R$ - for example, by considering $g'(x)$ as the slope of the tangent to the graph of $g$ at the point $(x,g(x))$.  In this section we will try to give a geometric description of the derivative of a holomorphic function $f:U \to \C$ at a point $z_0 \in U$.  Since we cannot draw the graph of such a function, some care is needed.

Returning to the real case, consider the following graph of a differentiable function $g:\R \to \R$:
\begin{center}
\begin{tabular}{ccc}
\altgraphics[scale=0.5]{ch2_realderivative1_full}{ch2_realderivative1} & \qquad & \altgraphics[scale=0.5]{ch2_realderivative2_full}{ch2_realderivative2}
\end{tabular}
\end{center}

For sufficiently small $h$ we have
\[
\frac{g(a+h)-g(a)}{h} \approx g'(a).
\]
\begin{blankbox}
We can rewrite this approximation as
\[
g(a+h)-g(a) \approx g'(a) h.
\]
Now, at least when $g'(a) \neq 0$, we know that $g$ maps the interval $[a,a+h]$, of length $h$, to the interval $[g(a),g(a+h)]$ of length $g(a+h)-g(a)$  i.e. length approximately $g'(a)h$.

In other words, $g$ moves $[a,a+h]$ to an interval from $g(a)$, and (approximately) scales it by a factor of $g'(a)$.
%\vspace*{3cm}
\end{blankbox}
If $g'(a)$ is negative, then $[a,a+h]$ is approximately sent to $[g(a+h),g(a)]$.
\begin{blankbox}
Then the mapping reverses the direction of the interval, i.e., sends $[a,a+h]$ to $[g(a+h),g(a)]$.  Again, the interval is scaled by a factor of $\abs{g'(a)}$, but this time, also rotated by an angle of $\pi$.  We can represent these mappings using one-dimensional figures as shown.

{
\centering
%\altgraphics[scale=0.4]{ch2_4lines_full}{ch2_4lines}
\hidegraphics[width=0.8\textwidth]{ch2_4lines}
\showgraphics[width=\textwidth]{ch2_4lines_full}
}
\end{blankbox}

For a complex function $f:U \to \C$, its graph is the set of points
\[
\set{ (z,f(z)): z \in U },
\]
as subset of $\C^2$.  We would need 4 coordinates to draw such a graph, which is impossible.

\begin{question}
How do we describe the geometry of derivatives of complex functions?
\end{question}

\begin{answer}
To describe the derivative of $f'(z_0)$ for a point $z_0 \in U$ geometrically, we will look at an analogy of the real case we have described above.   Indeed, rather than looking at a small interval of the form $[a,a+h]$, we look at a small disk $D(z_0,r)$ centred at  $z_0$.  Again, let us restrict to the case where $g'(z_0) \neq 0$.

If $r$ is `small' and $z=z_0+h \in D(z_0,r)$ then again
\[
\frac{f(z_0+h)-f(z_0)}{h} \approx f'(z_0).
\]

This can be rewritten as
\[
f(z_0+h) - f(z_0)\approx hf'(z_0), \quad\text{i.e.}\quad f(z)-f(z_0) \approx hf'(z_0).
\]
If we think of $h$ as the vector from $z_0$ to $z$, and $f(z)-f(z_0)$ as the vector from $f(z_0)$ to $f(z)$, then roughly speaking, $f$  sends $h$ to $hf'(z_0)$.  In other words, the vector $h$ gets moved to a vector from $f(z_0)$, and approximately gets multiplied by $f'(z_0)$, i.e.
\begin{itemize}
\item scaled by a factor of $\abs{f'(z_0)}$ and
\item rotated by angle $\arg (f'(z_0))$ anticlockwise about $f(z_0)$.
\end{itemize}
Again, we are using the fact that for $z_1,z_2 \in \C$, $\abs{z_1z_2} = \abs{z_1} \abs{z_2}$ and $\arg (z_1z_2) = \arg(z_1)+\arg(z_2)$.

\begin{center}
\begin{tabular}{cc}
\hidegraphics[scale=1]{ch2_deriv3} & \qquad \hidegraphics[scale=1]{ch2_deriv4} \\
\end{tabular}
\showgraphics[width=\textwidth]{ch2_complexderiv_full}
\end{center}
\end{answer}
Since the above remarks hold for any $z \in D(z_0,r)$, the mapping $f$ approximately transforms all points in $D(z_0,r)$ in the same way.
\begin{summary}
Let $f$ be differentiable at a point $z_0 \in \C$ with $f'(z_0) \neq 0$, then $f$ approximately maps small disks centred at $z_0$ to small disks centred at $f(z_0)$ as follows:
\begin{center}
\emph{scaling by a factor of $\abs{f'(z_0)}$ and rotating by an angle of $\arg (f'(z_0))$ anticlockwise about $f(z_0)$.}
\end{center}
\end{summary}
\begin{figure}[H]
\centering
\altgraphics[width=\textwidth]{ch2_geometry_deriv_full}{ch2_geometry_deriv}
\caption{The image under $f$ of a small disc centred at $z_0$.}
\end{figure}

\begin{example}
Find the geometric effect of applying the function
\[
f(z)=z^2-\frac{i}{z^2}
\]
to a small disk centred at $i$.
\end{example}
%\vspace*{12cm}
\begin{solution}
Since $f$ is differentiable at $i$, the previous remarks indicate that a small disc centred at $i$ gets sent to a small disc centred at $f(i)$, where
\[
f(i) = i^2-\frac{i}{i^2} = -i+1.
\]
To determine the geometric effect of $f$ applied to this disc, we need to calculate $\abs{f'(i)}$ and $\arg (f'(i))$.  The rules of differentiation tell us that
\[
f'(z) = 2z-i \left( \frac{-2}{z^3} \right) = 2z+ \frac{2i}{z^3},
\]
and hence $f'(i) = 2i-2$, which has modulus $\abs{2i-2}=2\sqrt{2}$ and argument $3\pi/4$.

Hence a small disc at $i$ is approximately mapped to a small disc at $-1+i$, and is scaled by a factor of $2\sqrt{2}$ and rotated by an angle of $3\pi/4$  in the anticlockwise direction about $-1+i$.

\end{solution}

\begin{figure}[H]
\centering
\hidegraphics[scale=1]{ch2_parametric3} \quad \hidegraphics[scale=1]{ch2_parametric4}
\showgraphics[scale=0.5]{ch2_parametric3_full}
\caption{The geometric effect of applying $f(z)=z^2-\dfrac{i}{z^2}$ to small circles centred at $i$. The transformed circles are centred at $f(i)$.  As Figure~\ref{f:circles} shows, the images of the smaller circles are almost circular, but the larger ones less so.  The horizontal line from $i$ is rotated by $\arg(f'(i))$.}
\label{f:circles}
\end{figure}

\begin{example}
Determine the geometric effect of applying the function $f$, where $f(z)=z^3$, to a small disc centred at $z_0 = \sqrt{3}-i$.
\end{example}
\begin{solution}
This time, the disc gets sent to a disc centred at $f(z_0) = (\sqrt{3}-i)^3 = -4i$.  Since $f'(z)=3z^2$, we have
\[
f'(\sqrt{3}-i) = 3 ( \sqrt{3}-i)^2 = 3 (2-i2\sqrt{3}) = 6-i6\sqrt{3}.
\]
Thus
\[
\abs{f'(\sqrt{3}-i) } = \sqrt{(6)^2+\left(6\sqrt{3}\right)^2 } = 12,
\]
and
\[
\arg \left( f' (\sqrt{3}-i ) \right) = \arg \left( 6-i6\sqrt{3} \right) = - \pi/3.
\]

Hence a small disc at $\sqrt{3}-i$ is approximately mapped to a small disc at $-4i$, and is scaled by a factor of $12$ and rotated by an angle of $-\pi/3$  in the \emph{clockwise} direction about $-4i$ (note: this time the rotation is clockwise since $\arg (f'(z_0))$ is negative).
\end{solution}


Many people (and textbooks) incorrectly refer to \java{\%} as the ``modulus operator''.
In mathematics, however, {\bf modulus} is the number you're dividing by.
In the previous example, the modulus is 12.
The Java language specification refers to  \java{\%} as the ``remainder operator''.

The remainder operator looks like a percent sign, but you might find it helpful to think of it as a division sign ($\div$) rotated to the left.

%Note that both \java{/} and \java{\%} perform {\em integer division}, so the result always rounds down.
%The reason why integer division ``rounds down'' is that the hardware computes the quotient and remainder separately.

\index{divisible}
\index{extract digits}

Modular arithmetic turns out to be surprisingly useful.
For example, you can check whether one number is divisible by another: if \java{x \% y} is zero, then \java{x} is divisible by \java{y}.
You can use remainder to ``extract'' digits from a number: \java{x \% 10} yields the rightmost digit of \java{x}, and \java{x \% 100} yields the last two digits.
And many encryption algorithms use the remainder operator extensively.

\section{Putting it all together}

At this point, you have seen enough Java to write useful programs that solve everyday problems.
You can (1) import Java library classes, (2) create a \java{Scanner}, (3) get input from the keyboard, (4) format output with \java{printf}, and (5) divide and mod integers.
Now we will put everything together in a complete program:

%Since we've looked at each of these topics in isolation, it's important to see how they fit together in a complete program.
%If you've been working through the examples on your computer as you've been reading (like we recommended in Section~\ref{sec:examples}), then good job!

Using division and modulo, we can convert to feet and inches like this:

\begin{code}
    feet = 76 / 12;    // quotient
    inches = 76 % 12;  // remainder
\end{code}

The first line yields 6.
The second line, which is pronounced ``76 mod 12'', yields 4.
So 76 inches is 6 feet, 4 inches.

Addition, subtraction, and multiplication all do what you expect, but you might be surprised by division.
For example, the following fragment tries to compute the fraction of an hour that has elapsed:%, but it has a logic error:

At this point, you have seen enough Java to write useful programs that solve everyday problems.
You can (1) import Java library classes, (2) create a \java{Scanner}, (3) get input from the keyboard, (4) format output with \java{printf}, and (5) divide and mod integers.
Now we will put everything together in a complete program:

%Since we've looked at each of these topics in isolation, it's important to see how they fit together in a complete program.
%If you've been working through the examples on your computer as you've been reading (like we recommended in Section~\ref{sec:examples}), then good job!

\index{Convert.java}

\begin{trinket}{Convert.java}
import java.util.Scanner;

/**
* Converts centimeters to feet and inches.
*/
public class Convert {

    public static void main(String[] args) {
        double cm;
        int feet, inches, remainder;
        final double CM_PER_INCH = 2.54;
        final int IN_PER_FOOT = 12;
        Scanner in = new Scanner(System.in);

        // prompt the user and get the value
        System.out.print("Exactly how many cm? ");
        cm = in.nextDouble();

        // convert and output the result
        inches = (int) (cm / CM_PER_INCH);
        feet = inches / IN_PER_FOOT;
        remainder = inches % IN_PER_FOOT;
        System.out.printf("%.2f cm = %d ft, %d in\n",
        cm, feet, remainder);
    }
}
\end{trinket}

Although not required, all variables and constants are declared at the top of \java{main}.
This practice makes it easier to find their types later on, and it helps the reader know what data is involved in the algorithm.


\documentclass[12pt]{article}

\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{float}
\usepackage{ulem}
\usepackage{bm}

\usepackage{anysize}

\marginsize{2cm}{2cm}{0.9cm}{1.8cm}
\usepackage[framed,numbered,autolinebreaks,useliterate]{mcode}
\usepackage{listings}
\lstset{language=Matlab}
\lstset{breaklines}
\lstset{extendedchars=false}

\title{Machine Learning/Pattern Recognition\\  \begin{Large} Homework \#4 \end{Large} }
\author{Jiyu Tian}
\date{}

\begin{document}

\maketitle
%%---------------------------------------------------------------
%% Problem 1
%%---------------------------------------------------------------
\section{Problem 4.1}
\textbf{\begin{large}Solution:\end{large}}\\
\\
(a) The expectation maximization algorithm for a given Gaussian mixture model is:\\
\\
\textbf{1.} Initialize the means $\bm{\mu}_k$, covariances $\bm{\Sigma}_k$ and mixing coefficients $\pi_k$, and evaluate the initial value of the likelihood.\\
\textbf{2. E step}. Evaluate the responsibilities using the current parameter values
\begin{equation*}
\gamma(z_{nk})=\frac{\pi_k \mathcal{ N}(\bm{x}_n|\bm{\mu}_k,\bm{\Sigma}_k)}{\sum_{j=1}^K\pi_j \mathcal{ N}(\bm{x}_n|\bm{\mu}_j,\bm{\Sigma}_j)}
\end{equation*}
\textbf{3. M step}. Re-estimate the parameters using the current responsibilities
\begin{equation*}
\bm{\mu}_k^{new}=\frac{1}{N_k}\sum_{n=1}^N\gamma(z_{nk})\bm{x}_n
\end{equation*}
\begin{equation*}
\bm{\Sigma}_k^{new}=\frac{1}{N_k}\sum_{n=1}^N\gamma(z_{nk})(\bm{x}_n-\bm{\mu}_k^{new})(\bm{x}_n-\bm{\mu}_k^{new})^T
\end{equation*}
\begin{equation*}
\pi_k^{new}=\frac{N_k}{N}
\end{equation*}
where $N_k = \sum_{n=1}^N\gamma(z_{nk})$.\\
\textbf{4.} Evaluate the log likelihood
\begin{equation*}
\text{ln }p(\bm{X}|\bm{\mu},\bm{\Sigma},\bm{\pi})= \sum^N_{n=1} \text{ln } \{\sum_{j=1}^K\pi_k \mathcal{ N}(\bm{x}_n|\bm{\mu}_k,\bm{\Sigma}_k)\}
\end{equation*}
and check for convergence of either the parameters or the log likelihood. If the convergence criterion is not satisfied return to step 2.\\
\\
(b) See Figure 1. Comparing to K-means, EM performed better because it takes into consideration the corresponding covariance matrix rather than uniformly identity covariance matrix.\\
\\
(c) See Figure 2. EM is able to find the true clusters and the origin covariance matrices. However, it performs not quite well in the intersection.
\vfill
\clearpage

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{41b.png}
\caption{Result\ of\ clustering\ with\ EM-GMM\ on\ dataset1.mat}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{41c.png}
\caption{Result of clustering with EM-GMM on dataset2.mat}
\end{figure}

\vfill
\clearpage

\noindent(d) See Figure 3. Since the true clusters are radial while EM tries to fit the data points with Gaussian mixture, EM does not find the true clusters. However, EM still performs better than K-means.\\
\\
(e) See Figure 4. EM is able to capture the distribution of this non-Gaussian dataset better than K-means.

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{41d.png}
\caption{Result of clustering with EM-GMM on dataset3.mat}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{41e.png}
\caption{Result of clustering with EM-GMM on dataset4.mat}
\end{figure}
\vfill
\clearpage
\section{Problem 4.2}
\textbf{\begin{large}Solution:\end{large}}\\
\\
(a) See Figure 5. The maximum matches the true number of clusters.\\
\\
(b) See Figure 6. The maximum matches the true number of clusters.\\

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{42a.png}
\caption{BIC result of clustering with EM-GMM on dataset1.mat}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{42b.png}
\caption{BIC result of clustering with EM-GMM on dataset2.mat}
\end{figure}
\vfill
\clearpage

\noindent(c) See Figure 7. BIC does not find the true number of clusters.\\
\\
(d) See Figure 8. The maximum matches the true number of clusters. However, there would be only two clusters in the result, which I believe depends on the initial conditions for EM algorithm.

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{42c.png}
\caption{BIC result of clustering with EM-GMM on dataset3.mat}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{42d.png}
\caption{BIC result of clustering with EM-GMM on dataset4.mat}
\end{figure}
\vfill
\clearpage

\noindent(e) Both K-means and EM-GMM give the result of four clusters, but EM-GMM has a better explanation for the dataset.
\begin{figure}[H]
\centering
\includegraphics[width = 1\textwidth]{42e.png}
\caption{BIC result of clustering with EM-GMM on dataset5.mat}
\end{figure}

\end{document}


\section{General continuous piecewise linear functions as a DNN}
\subsection{Lattice representation of CPWL}

We say a function $f:\mathbb{R}^n \to \mathbb{R}$ is continuous piecewise linear (CPWL) if there exists a finite set of polyhedra whose union is $\mathbb{R}^n$, and $f$ is affine linear over each polyhedron (note that the definition automatically implies continuity of the function because the affine regions are closed and cover $\mathbb{R}^n$, and affine functions are continuous). The number of pieces of $f$ is the number of maximal connected subsets of $\mathbb{R}^n$ over which $f$ is affine linear (which is finite).

\begin{figure}[!ht]
\centering
	\includegraphics[width=0.6\textwidth]{6DL/figures/latticePWL.png}
	\caption{Unique order regions.}
	\label{fig:latticePWL}
\end{figure}
Figure \ref{fig:latticePWL} shows a CPWL function with unidimensional domain that has a domain partition generated by $4$ boundaries. The result is a set of $5$ regions and the corresponding set of $5$ local functions, two of which are identical, $f_2$ and $f_5$. In the case of a model described in terms of the local functions.

The arrangements in ascending order of the linear functions in the points $a$ and $b$ of region $R_4$ are respectively, ($f_2 = f_5 < f_1 < f_4 < f_3$) and ($f_2= f_5 < f_4 < f_1 < f_3$). We note that the intersection of the linear functions fland f4occurs at point x5, an inner point of $R_4$. Given the importance of the order arrangement when dealing with models based on the local functions, the division of region $R_4$ into two subregions is practically meaningful.

Any intersection of linear functions produces a change of order arrangement. So, if we consider every intersection of local functions, the resulting domain partition will be formed by connected, and convex regions with a unique order arrangement. It should be noted that some of those intersections might not correspond to any boundary of the PWL function.

We introduce a domain partition that will have unique order on each subdomain \cite{tarela1999region}.
\begin{definition}
	Let $f:\mathbb{R}^n\to\mathbb{R}$ be a continuous piecewise linear function with m distinct local functions $l_i$, $1\le i\le m$. Consider the intersections between local functions, those intersections with some noticed segments in $f$ will be called visible boundaries, and those boundaries corresponds to the set of functions $\{\phi_{\lambda_v}\}$. The rest of intersections occurring inside the domain will be called hidden boundaries and noted as $\{\phi'_{\lambda_h}\}$. A \textbf{unique-order region} is each region of the domain partition produced by the boundary configuration $\{\phi_{\lambda_v}\}\bigcup\{\phi'_{\lambda_h}\}$.
\end{definition}

\bigskip
The unique-order regions have the following characterizations:
\begin{itemize}
	\item The domain of $f$ is divided into $M$ unique-order regions.
	\item The local function associated to two unique-order regions may be identical.
	\item The unique-order regions are convex.
	\item A unique-order region has the same rearrangement in ascending order of the values of the $m$ local functions in all its points.
\end{itemize}

\begin{remark}
	If $m$ is the number of distinct linear components of $f$, then the maximum number of rearrangements in ascending order is $m!$. However, the number of unique-order regions is in general lower than $m!$, because of the geometric constraints of the problem \cite{wilkinson1963method}. Thus, the different arrangements in $\mathbb{R}^n$ is $Q$, then:
	\begin{equation*}
	\begin{aligned}
	&\mathrm{if}\ 1\le m\le n+1\quad\Rightarrow Q=m!\\
	&\mathrm{if}\ m\ge n+2\quad\Rightarrow (n+1)!<Q<m!
	\end{aligned}
	\end{equation*}
	The number of unique-regions is $M$, then we have $M\le Q$, since only some of these arrangements constitute unique-order regions.
\end{remark}

\begin{lemma}\label{lem:1dlattice}
	If $p(t)$ is a continuous piecewise linear function. And the unique-order region partition is
	$$0 = t_0 < t_1 < ... < t_{r+1} = 1$$
	and the linear function in $[t_i,t_{i+1}]$ is $l_i(t) = k_i t+b_i$. And parameter satisfied
	$$ b_0 > b_r,\qquad k_0 + b_0 > k_r + b_r $$
	which means
	$$l_0(t)> l_r(t)\quad \mbox{in}\quad [t_0,t_1]$$
	$$l_0(t)> l_r(t)\quad \mbox{in}\quad [t_r,t_{r+1}]$$
	Then, exist $l_p(t) = k_p t+ b_p$, such that
	$$b_p \ge b_0,\qquad k_p+b_p\le k_r+b_r$$
	That's to say
	$$l_0(t)\le l_p(t)\quad \mbox{in}\quad [t_0,t_1]$$
	$$l_p(t)\le l_r(t)\quad \mbox{in}\quad [t_r,t_{r+1}]$$
\end{lemma}
\begin{figure}[th]
	\centering
	\includegraphics[width=0.7\linewidth]{6DL/pic/fig1}
	\caption{}
\end{figure}

\begin{proof}

	Let $k_p=\min\{k_i\}$, $\Delta t_i=t_{i+1}-t_i$.
	Since here we only have linear functions, so we can represent each point $(t_i,y_i)$ by using $k_i$'s and $\Delta t_i$'s.
	Then the point $(t_p,b_0+\sum_{i=0}^{p-1}k_i\Delta t_i)$ is on $y=k_p t+b_p$, so we can represent $b_p$ as following:
	$$b_p=b_0+\sum_{i=0}^{p-1}k_i\Delta t_i-k_p t_p=b_0+\sum_{i=0}^{p-1}(k_i-k_p)\Delta t_i$$
	Since here $k_p$ is the minimum, we have:
	$$b_p=b_0+\sum_{i=0}^{p-1}(k_i-k_p)\Delta t_i\ge b_0$$
	And
	\begin{equation*}
	\begin{aligned}
	k_p+b_p&=k_p+b_0+\sum_{i=0}^{p-1}(k_i-k_p)\Delta t_i\\
	k_r+b_r&=k_r+b_0+\sum_{i=0}^{r-1}(k_i-k_p)\Delta t_i\\
	(k_r+b_r)-(k_p+b_p)&=k_r-k_p+\sum_{i=p}^{r-1}(k_i-k_p)\Delta t_i\ge 0
	\end{aligned}
	\end{equation*}
	which means we find the desired pair of $k$ and $p$.
	Notice here $k_p\ne k_0$ and $k_p\ne k_r$ by the assumptions in the lemma. This completes the proof.

\end{proof}

\bigskip
\begin{theorem}\label{PWLtoRelu}
	For every continuous piecewise linear function $f:\mathbb{R}^n\to\mathbb{R}$ with finite pieces defined by the distinct local linear functions $l_i$, $1\le i\le m$ and $\{\Omega_k\}_{k=1}^M$ be the unique-order subdomains. Then there exist finite non-empty subsets of $\{1,2,\dots,m\}$, say $s_k$, $1\le k\le M$, such that
	\begin{equation}
	f(x)=\max_{1\le k\le M}\{\min_{i\in s_k} l_i\},
	\end{equation}
	here $s_k=\{i: l_i\ge l_k\ \mathrm{on}\ \Omega_{k}\}$.
\end{theorem}

\begin{proof}
	Let $l_i(x)=f(x)|_{\Omega_k}$. In each $\Omega_{k}$, consider the functions lie completely above $l_k$, and define the convex polynomial
	$$\Phi_{k}=\min_{i\in s_k} l_i.$$
	Define
	$$\Phi(x)=\max_{k} \Phi_{k}(x)$$
	and next we show that $\Phi_{k}(x)\le f(x)$ for all x and every $k$.

	For any fixed $k$, if $x_0\in\Omega_{k}$,
	$$
	\Phi_{k}(x_0)=l_k(x_0)= f(x_0).
	$$
	If $x_0\notin\Omega_{k}$, then we suppose $x_0\in \Omega_{k'}$, so
	$$
	f(x_0)|_{\Omega_{k'}}=l_{k'}(x_0).
	$$
	Notice that here we have unique-order region, thus in each $\Omega_i$, the order of $l_k$ and $l_{k'}$ is fixed. There're several situations:
	\begin{enumerate}
		\item  If $l_{k'}(x)\ge l_{k}(x)$ for $x\in \Omega_{k}$, then $l_{k'}$ is a lattice variable in the convex polynomial $\Phi_k$, and so
		$$\Phi_{k}(x_0)\le l_{k'}(x_0)=f(x_0).$$
		\item  If $l_{k'}(x)< l_{k}(x)$ for $x\in \Omega_{k}$, then we consider the domain $\Omega_{k'}$:
		\begin{enumerate}
			\item
			If $l_{k'}(x)\ge l_k(x)$ for $x\in \Omega_{k'}$. Then on $\Omega_{k'}$,
			\begin{equation*}
			\Phi_{k}(x_0)= l_k(x_0)\le l_{k'}(x_0)=f(x_0)
			\end{equation*}
			\item
			If $l_{k'}(x)< l_k(x)$ for $x\in \Omega_{k'}$. We take $x\in \Omega_{k}^\circ, x' \in \Omega_{k'}^\circ$. Then we have a path $L(\theta)$, the coordinate of the path is defined as
			$$
			(x + \theta(x'-x),f(x + \theta(x'-x)))
			$$
			with $\theta\in[0,1]$(see Figure~\ref{fig:sec:lattice}). It is just a piecewise linear function with the parameter $\theta$. Notice that the domain partition is unique-order. So if we want to compare the order of the linear function,  we just compare one point value in that region.
		\begin{figure}[th]
			\centering
			\includegraphics[width=0.7\linewidth]{6DL/pic/fig2}
			\caption{figure}
			\label{fig:sec:lattice}
		\end{figure}
		Then by Lemma \ref{lem:1dlattice}, there must exist $l_t$ with $t\ne k,k'$ and
			\begin{equation*}
			\begin{aligned}
			&l_t\le l_{k'}\quad \mathrm{on}\ \Omega_{k'}\\
			&l_t\ge l_{k}\quad \mathrm{on}\ \Omega_{k}
			\end{aligned}
			\end{equation*}
			Then we should have:
			$$\Phi_{k}(x_0)\le l_t(x_0)\le l_{k'}(x_0)=f(x_0).$$
		\end{enumerate}
	\end{enumerate}
Thus for every $\Phi_{k}$, we have $\Phi_{k}(x)\le f(x)$ for all $x$. It is obvious that  $f(x)\le \Phi_{k}(x)$ for all $x$ since
$$
f(x)=l_k(x)\le \max_k l_k=\max \Phi_k=\Phi(x),\quad \forall x\in \Omega_k.
$$
So
			$$f(x)=\max_{k}\min_{i\in s_k}\{l_i\}$$
			This is exactly the desired form. Here $|s_k|\le m$, and the number of $\Phi_{k}$ depends on the partition we do.
\end{proof}

\subsection{General CPWL as a DNN}

Assume that $f:\mathbb{R}^d\to\mathbb{R}$ is a continuous function
that are piecewise linear on $m$ subdomains
$$
\Omega_i, \quad i=1:m.
$$
Namely, on each $\Omega_i$, $f$ is a linear function:
$$
f(x)=f_i(x)=a_i\cdot x+b_i, \quad x\in \Omega_i,
$$
with some $a_i \in \mathbb{R}^d$ and $b_i \in \mathbb{R}$.

Combining Theorem \ref{PWLtoRelu} with Lemma \ref{linearcombine2Relu}, we have the following result.
\begin{theorem}
Every $\mathbb{R}^{d} \rightarrow \mathbb{R}$ ReLU DNN represents a piecewise linear function, and every piecewise linear function $\mathbb{R}^{d} \rightarrow \mathbb{R}$ can be represented by a ReLU DNN with at most $\left\lceil\log _{2}(d+1) \right\rceil+1$ depth.
\end{theorem}
\begin{proof}
It is clear that any function represented by a ReLU DNN is a PWL function. To see the converse, we first note that any PWL function can be represented as a linear combination of piecewise linear convex functions. More formally, by Theorem \ref{PWLtoRelu}, for every piecewise linear function $f: \mathbb{R}^{d} \rightarrow \mathbb{R},$ there exists a finite set of affine linear functions $\ell_{1}, \ldots, \ell_{m}$ and subsets $s_{1}, \ldots, s_{M} \subseteq\{1, \ldots, m\}$ (not necessarily disjoint) where each $s_{i}$ is of cardinality at most $d+1,$ such that
$$
f=\sum_{k=1}^{p} v_{k}\left(\max _{i \in s_{k}} \ell_{i}\right)
$$
where $v_{j} \in\{-1,+1\}$ for all $j=1, \ldots, p .$ Since a function of the form $\max _{i \in s_{j}} \ell_{i}$ is a piecewise linear convex function with at most $n+1$ pieces (because $\left|s_{j}\right| \leq d+1$ ),   any continuous piecewise linear function (not necessarily convex) can be obtained as a linear combination of piecewise linear convex functions each of which has at most $d+1$ affine pieces. Note that $\max \{x, y\}=\frac{x+y}{2}+\frac{|x-y|}{2}$ is implementable by a two layer ReLU network and use this construction in an inductive manner to show that maximum of $d+1$ numbers can be computed using a ReLU DNN with depth at most $\left\lceil\log _{2}(d+1)\right\rceil$.
\end{proof}

For the relationship between ReLU DNNs and general CPWL functions, we have the next theorem with some estimation~\cite{arora2016understanding}.
\begin{theorem}\label{main0}
	A continuous function $f:\mathbb{R}^d\to\mathbb{R}$ that are piecewise linear on $m$ subdomains $\{\Omega_j\}_{j=1}^m$ can be represented by a ReLU DNN.
	Furthermore,
	\begin{enumerate}
		\item the number of  hidden layers  is bounded by
		\begin{equation}
		\label{layer}
		N_{\rm layer}\le \lceil \log_2(d+1)\rceil.
		\end{equation}
		\item the number of neurons
		\begin{equation}
		\label{neurons}
		N_{\rm neuron}=
		\left\{
		\begin{array}{ll}
		\mathcal  O\left(d2^{mM+(d+1)(m-d-1)}\right) & \mbox{ if } m\ge d+1,\\
		\mathcal O\left(d2^{mM}\right) & \mbox{ if } m< d+1.
		\end{array}
		\right.
		\end{equation}
		here $M$, satisfying $m\le M\le m!$, is the number of subdomains in which $f_i-f_j$ does not change sign.
	\end{enumerate}
\end{theorem}
Combining Theorem~\ref{main0} with Theorem~\ref{lowerbound}, we have the following corollary regarding the minimal number of layers needed to recover all piecewise linear functions.

\begin{corollary}
	\begin{equation}
	2 \le J_d \le \lceil\log_2(d+1)\rceil.
	\end{equation}
	This also indicates that $\lceil\log_2(d+1)\rceil$ is ``optimal" for $d=2,3$.
\end{corollary}

\endinput

\section{$\lceil \log(d+1)\rceil$-depth dnn}
Given any $L>n+1$ linear functions, the maximum of these $L$ functions can be represented as the maximum of only $n+1$ linear functions. The next lemma shows that we can reduce the number of functions by one if it is larger than $n+1$.
Let $l(x,a)=[1\ x^T]a$.
\begin{lemma}\label{reduce}
	For any interger $L$ with $1\le n< L$, $c_0\in\mathbb{R}$ and arbitrary linear function $l_1(x),\dots,l_L(x)$ of $x\in\mathbb{R}^d$, there exist finite groups of $L-1$ linear functions, say $l(x,b_1(k))$, $\dots$,
	$l(x,b_{L-1}(k))$, $1\le k\le K$, and corresponding $c_k\in\mathbb{R}$, $\sigma_k\in\{1,-1\}$ such that

	\begin{equation}\label{goal}
	\max\{c_0,l_1,\dots,l_L\} = \sum_{k=1}^{K}\sigma_k\max\{c_k,l(x,b_1(k)),\dots,l(x,b_{L-1}(k))\},
	\end{equation}
	where $K=2^{n+1}-1$.
\end{lemma}

\begin{proof}
		Let $l_i(x)=l(x,a_i)=a_{i0}+x^T\bar{a}_i$ with $a_{i0}\in\mathbb{R}$ and $\bar{a}_i\in\mathbb{R}^n$ for $1\le i\le L$. Assume there are at most $\bar{n}$ linearly independent $\bar{a}_i$, $\bar{n}\le n$. Without loss of generality, assume $\bar{a}_1,\dots,\bar{a}_{\bar{n}}$ are linearly independent. Then by basic linear algebra, we know that
		\begin{equation}\label{0}
		l_L=\sum_{j=1}^{\bar{n}}\alpha_j l_j+\alpha_0.
		\end{equation}
		Denote
		$$
		\mu(x)=\max\{l_{\bar{n}+1},\dots,l_{L-1}\}.
		$$
		Then
		\begin{equation}\label{1}
		\max\{c_0,l_1,\dots,l_L\}=\max\{c_0,l_1,\dots,l_{\bar{n}},\mu(x),l_L\}.
		\end{equation}
		 If $\alpha_j = 0$ in \eqref{0} for each $1\le j\le \bar{n}$, then by taking $\max\{c_0,\alpha_0\}$, we already make the RHS of (\ref{1}) as the RHS of (\ref{goal}). Otherwise, $\exists \alpha_j\ne0$ for some $1\le j\le \bar{n}$, we can assume that $\alpha_{\eta} \neq 0, \alpha_{\eta+1}= ... = \alpha_{\bar{n}}=0$, so
		$$
		l_L=\sum_{j=1}^{\eta}\alpha_j l_j+\alpha_0,\qquad \eta\le \bar{n}.
		$$
		Next we show that it can be represented by
		\begin{equation}\label{latticereaarange1}
		\max\{c_0,l_1,\dots,l_L\}=\max\{c_0,l_1',\dots,l_{\bar{n}}',\mu(x),l_L'\}
		\end{equation}
		where $\displaystyle l_L'=\alpha_\eta l_\eta+\sum_{j=1}^{\eta-1}\alpha_jl_j'+\alpha_0$ with $\alpha_\eta\neq 1$.

		If $\alpha_i=1$ for each $1\le i\le\eta$, we can do the following linear transformation $(l'_1,\dots,l'_{\bar{n}})^T=A(l_1,\dots,l_{\bar{n}})^T$, where
		\begin{equation*}
		\begin{aligned}
		l'_i&=l_i,\qquad &\mathrm{for}\ i\ne\eta,\\
		l'_i&=\sum_{j=1}^{\eta} l_j+\alpha_0,\qquad &\mathrm{for}\ i=\eta.\\
		\end{aligned}
		\end{equation*}
		Then (\ref{1}) equals
		$$
		\max\{c_0,l'_1,\dots,l'_{\bar{n}},\mu(x),-\alpha_0-l'_1-\dots-l'_{\eta-1}+l'_{\eta}\}.
		$$
		in this case,
		$$
		l_L'=-\alpha_0-l'_1-\dots-l'_{\eta-1}+l'_{\eta}
		$$
		the coefficients of $l'_1,\dots,l'_{\eta-1}$ are not 1. So we can rearrange $l_i'$ and assume there is at least one $\alpha_i\ne1$ for $1\le i\le\eta$, say $\alpha_{\eta}\ne1$. This leads to the representation \eqref{latticereaarange1}. Let
		\begin{equation*}
		\begin{aligned}
		f&=\max\{c_0,\mu(x),l_1',\dots,l_{\eta-1}',l_{\eta+1}',\dots,l_{\bar{n}}'\},\\
		g&=l_{\eta},\\
		h&=\sum_{j=1}^{\eta-1}\alpha_j l_j'+\alpha_0.
		\end{aligned}
		\end{equation*}
		By (\ref{key-reduce}),
		\begin{eqnarray}\label{2}
		\max\{c_0,l_1',\dots,l_{\bar{n}}',\mu(x),l_L'\}&&=\max\{f,g,\alpha_{\eta} g+h  \}\\
		\label{3}
		&&=\sigma_1\max\{f,g,\frac{\sum_{j=1}^{\eta-1}\alpha_j l_j'+\alpha_0}{1-\alpha_{\eta}}  \}\\
		\label{4}
		&&+\sigma_2\max\{f,\alpha_{\eta} g+h,\frac{\sum_{j=1}^{\eta-1}\alpha_j l_j'+\alpha_0}{1-\alpha_{\eta}}   \}\\
		\label{5}
		&&+\sigma_3\max\{f,\bar{g} \}.
		\end{eqnarray}
		(\ref{5}) is already the desired form, because now we only take maximum  over $L-1$ linear functions and one constant.

		As for the (\ref{3}), notice that now we have eliminated $l_{\eta}$ in the third expression
		$$
		\alpha_\eta l_\eta'+\sum_{j=1}^{\eta-1}\alpha_j l_j'+\alpha_0\quad \Rightarrow \quad \frac{\sum_{j=1}^{\eta-1}\alpha_j l_j'+\alpha_0}{1-\alpha_{\eta}}.
		$$
		So continue this procedure, at last we will only have constant in the last expression, by taking maximum of this constant and $c_0$, we can reduce one term in the max expression.

		For (\ref{4}), consider the linear transformation $(l''_1,\dots,l''_{\bar{n}})^T=B(l_1',\dots,l_{\bar{n}}')^T$:
		\begin{equation*}
		\begin{aligned}
		l''_i&=l_i',\qquad &\mathrm{for}\ i\ne\eta,\\
		l''_i&=\sum_{j=1}^{\eta}\alpha_{j} l_j'+\alpha_0,\qquad &\mathrm{for}\ i=\eta.\\
		\end{aligned}
		\end{equation*}
		So (\ref{4}) becomes
		$$
		\max\{c_0,\mu,l''_1,\dots,l''_{\bar{n}},\sum_{j=1}^{\eta-1}\alpha_j l''_j+\alpha_0\}.
		$$
		Then it is the same as (\ref{3}). Follow the same steps as for \eqref{3}, we can achieve the desired result.
	\end{proof}

	\begin{figure}[ht]
		\begin{center}
			\begin{tikzpicture}[>=triangle 45,font=\sffamily]
			\node (X)  {(\ref{2})};
			\node (Y) [below left=1cm and 2cm of X]  { (\ref{3})};% 2cm below, 1cm to the left (optional)
			\node (Z) [below right=1cm and 2cm of X] {(\ref{4})};
			\node (O) [below =1 cm of X] {(\ref{5})};
			\node (U) [below left=1cm and 1.2cm of Y] { ...};
			\node (V) [below right=1cm and 1.2cm of Y] {... };
			\node (W) [below =1cm of Y] { };
			\node (U1)[below left=1cm and 1.2cm of Z] {...};
			\node (V1)[below right=1cm and 1.2cm of Z] {...};
			\node (W1) [below =1cm of Z] {};
			\node (T1)[below left=1cm and 0.5cm of U] { };
			\node (T2)[below right=1cm and 0.5cm of U] { };
			\node (T3)[ below =1cm of U] { };
			\node (T4)[ below left=1cm and 0.5cm of V] { };
			\node (T5)[ below right=1cm and 0.5cm of V] { };
			\node (T6)[ below =1cm of V] { };
			\node (T7)[below right=1cm and 0.5cm of U1] { };
			\node (T8)[ below left=1cm and 0.5cm of U1] { };
			\node (T9)[below =1cm of U1] { };
			\node (T10)[below right=1cm and 0.5cm of V1] { };
			\node (T11)[ below left=1cm and 0.5cm of V1] { };
			\node (T12)[below =1cm of V1] { };
			\draw [semithick,->] (X) -- (Y);
			\draw [semithick,->] (X) -- (Z);
			\draw [semithick,->] (X) -- (O);
			\draw [semithick,->] (Y) -- (U);
			\draw [semithick,->] (Y) -- (V);
			\draw [semithick,->] (Y) -- (W);
			\draw [semithick,->] (Z) -- (U1);
			\draw [semithick,->] (Z) -- (V1);
			\draw [semithick,->] (Z) -- (W1);
			\draw [semithick,->] (U) -- (T1);
			\draw [semithick,->] (U) -- (T2);
			\draw [semithick,->] (U) -- (T3);
			\draw [semithick,->] (V) -- (T4);
			\draw [semithick,->] (V) -- (T5);
			\draw [semithick,->] (V) -- (T6);
			\draw [semithick,->] (U1) -- (T7);
			\draw [semithick,->] (U1) -- (T8);
			\draw [semithick,->] (U1) -- (T9);
			\draw [semithick,->] (V1) -- (T10);
			\draw [semithick,->] (V1) -- (T11);
			\draw [semithick,->] (V1) -- (T12);
			\end{tikzpicture}
			\caption{The process of reducing one term.}
		\end{center}
		\label{fig:Redu}
	\end{figure}

	\begin{remark}
		\label{timesestimate}
		Whenever we eliminate one $l_i$ in the expression of $l_L$, we will gain 3 terms, which is (\ref{3}-\ref{5}). Among these three terms, (\ref{5}) is in desired form, and we need to continue to use (\ref{key-reduce}) for (\ref{3}) and (\ref{4}) until we only have constant. Note that in the proof, $\eta\le n$.  By this procedure, we will gain at most $2^{n+1}-1$ terms (see Figure~\ref{fig:Redu}).
	\end{remark}


\subsection{SUVAT}

\subsubsection{Introduction}

For a constant acceleration environment we want to find equations to link:
\begin{itemize}
\item Initial speed: \(v_{t_0}\)
\item End speed: \(v_{t_1}\)
\item Time: \(t_1-t_0\)
\item Acceleration: \(a\)
\item Displacement \(s_{t_1}-s_{t_0}\)
\end{itemize}

\subsubsection{The SUVAT equations}

These are the following, and are derived below.

\begin{itemize}
\item \(v_{t_1}=a(t_1-t_0)+v_{t_0}\)
\item \((s_{t_1}-s_{t_0})=v_{t_0}(t_1-t_0)+\dfrac{1}{2}a(t_1-t_0)^2\)
\item \((s_{t_1}-s_{t_0})=v_{t_1}(t_1-t_0)-\dfrac{1}{2}a(t_1-t_0)^2\)
\item \(v_{t_1}^2= v_{t_0}^2+2a(s_{t_1}-s_{t_0})\)
\item \((s_{t_1}-s_{t_0})=(t_1-t_0)\dfrac{v_{t_1}+v_{t_1}}{2}\)
\end{itemize}

\subsubsection{Equation 1: No displacement}

This equation is:

\(v_{t_1}=a(t_1-t_0)+v_{t_0}\)

To derive this start with:

\(v_t:=\dfrac{\delta s_t}{\delta t}\)

\(a:=\dfrac{\delta v_t}{\delta t}\)

If acceleration is constant, then

\(\dfrac{\delta v_t}{\delta t}=a\)

\(v_t=\int a dt +v_0\)

\(v_t=at+v_0\)
\subsubsection{Equation 2: No end velocity}

This equation is:

\((s_{t_1}-s_{t_0})=v_{t_0}(t_1-t_0)+\dfrac{1}{2}a(t_1-t_0)^2\)

To derive this start with:

\(v:=\dfrac{\delta s_t}{\delta t}\)

Then:

\(\dfrac{\delta s_t}{\delta t}=at+v_0\)

\(s_t=\dfrac{1}{2}at^2+v_0t+s_0\)

\((s_t-s_0)=v_0t+\dfrac{1}{2}at^2 \)

\subsubsection{Equation 3: No start velocity}

This equation is:

\((s_{t_1}-s_{t_0})=v_{t_1}(t_1-t_0)-\dfrac{1}{2}a(t_1-t_0)^2\)

To derive this start with:

\(v_t=at+v_0\)

\((s_t-s_0)=t\dfrac{v_t+v_0}{2}\)

So:

\(v_0=v_t-at\)

\(v_0=\dfrac{2}{t}(s_t-s_0)- v_t\)

\(v_t-at=\dfrac{2}{t}(s_t-s_0)- v_t\)

\((s_t-s_0)=v_tt-\dfrac{1}{2}at^2\)

\subsubsection{Equation 4: No time}

This equation is:

\(v_{t_1}^2= v_{t_0}^2+2a(s_{t_1}-s_{t_0})\)

To derive this start with:

\(v_t=at+v_0\)

\((s_t-s_0)=t\dfrac{v_t+v_0}{2}\)

So:

\(t=\dfrac{v_t-v_0}{a}\)

\(t=2\dfrac{s_t-s_0}{v_t+v_0}\)

\(\dfrac{v_t-v_0}{a}=2\dfrac{s_t-s_0}{v_t+v_0}\)

\((v_t-v_0)(v_t+v_0)=2a(s_t-s_0)\)

\(v^2_t= v^2_0+2a(s_t-s_0)\)
\subsubsection{Equation 5: No acceleration}

This equation is:

\((s_{t_1}-s_{t_0})=(t_1-t_0)\dfrac{v_{t_1}+v_{t_1}}{2}\)

To derive this start with:

\(v_t=at+v_0\)

\(s_t-s_0=\dfrac{1}{2}at^2+v_0t\)

So:

\(a=\dfrac{v_t-v_0}{t}\)

\(a=\dfrac{2[(s_t-s_0)-v_0t]}{t^2}\)

\(\dfrac{v_t-v_0}{t}=\dfrac{2[(s_t-s_0)-v_0t]}{t^2}\)

\(t(v_t-v_0)=2[(s_t-s_0)-v_0t]\)

\(t(v_t+v_0)=2(s_t-s_0)\)

\((s_t-s_0)=t\dfrac{v_t+v_0}{2}\)


\documentclass[12pt]{article}

\title{Important theorems}
\author{Abel Doñate}
\date{}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{graphicx}
\graphicspath{ {./images/} }
\usepackage{wrapfig}

\setcounter{page}{0}

%Geometry
\usepackage{geometry}
\geometry{a4paper, margin=1in}

\begin{document}

\maketitle
\tableofcontents
\newpage

\section{Fermat's little theorem}
If $a$ and $p$ are coprimes, then:
\[a^{p-1}\equiv 1 \ (mod \ p)\]
\\
\textbf{Proof}\\
Consider the set $S=\{ a, 2a, \cdots, (p-1)a\}$.\\
Now we pick 2 elements of the set, for instance $ka, la$, with $1\leq k<l\leq p-1$. We are going to show that necessarily $ka \not\equiv la\ (mod \ p)$. Let's prove it by contradiction:\\
\\
If we suppose that it is true, then:
\[ka \equiv la\ (mod \ p) \ \ \implies p|(l-k)a \ \ \implies \begin{cases}
p|a \ \ \ \text{(impossible, they are coprimes)} \\
\text{or} \\
p|(l-k) \ \ \ \text{(impossible, it is positive and less than $p$)}
\end{cases} \]
So, it is proven by contradiction.\\
Thus, the residues of the elements of $S$ must be different from each other, so the set of the elements of $S$ in modulo $p$ is $S_p=\{1, 2, \cdots, p-1\}$, as it has to have the same number of elements that $S$ (the set does not has to be necessarily in order from $S$).\\

Now we multiply the elements of $S$ and the elements of $S_p$. If we consider the residues modulo $p$ of the results, they must be the same, because each element of $S_p$ is the residue of one element of $S$. Then:
\[a\cdot 2a \cdots (p-1)a = 1\cdot 2 \cdots (p-1) \ (mod \ p) \implies a^{p-1}(p-1)! = (p-1)! \ (mod \ p)  \]
Trivially $(p-1)!$ is coprime with p (they do not share any factor), so we can divide the expression by $(p-1)!$. As desired we end up with
\[a^{p-1}\equiv 1 \ (mod \ p)\]
\qed

\newpage

\section{Wilson's theorem}
Let $p$ any prime. Then it holds:
\[(p-1)!\equiv -1 \ (mod \ p)\]
\\
\textbf{Proof} \\
First we are going to proof two Lemmas.\\ \\
\textbf{Lemma 1.} If $a^2\equiv 1 \ (mod \ p)$, then $a \equiv 1 \ (mod \ p)$ or $a \equiv 1 \ (mod \ p)$. \\
The proof of this lemma is following:
\[a^2\equiv 1 \ (mod \ p) \implies p|a^2-1 \implies p|(a+1)(a-1) \implies \begin{cases}
p|(a+1) \\
\text{or} \\
p|(a-1)
\end{cases}\]
so, a must be $1$ or $-1$ in modulo $p$. \\
\\
\textbf{Lemma 2.} Every number between $2$ and $p-2$ has a unique inverse that is not itself.\\
The proof is very simple. Using \textit{Lemma 1}, the only numbers that could be its own inverse are $1$ and $-1$. We know that, as every element must have an inverse ($\mathbf{Z_p}$ is a group), the inverse of the remaining elements must be different from themselves.\\
\\
Now we are ready for the proof. If we take $(p-1)!$, we can split it in this way
\[(p-1)!=1\cdot (2\cdot 3 \cdots (p-3) \cdot (p-2))\cdot (p-1)\]
Observe that in the middle remains the numbers between $2$ and $p-2$. Using \textit{Lemma 2} we can pair the elements in pairs formed by one element and its inverse (that is not itself) so that the product is $1 \ (mod \ p)$. Finally multiplying by $1$ and $(p-1)$ gives us \[(p-1)!\equiv -1 \ (mod \ p)\] as desired. \\
\qed

\end{document}


\section{Conclusion and future work}

We introduced a mathematical framework to design neural networks for data that live on simplicial complexes and provided preliminary results on their ability to impute missing data.
Future work might include:
(i) comparing SNNs with state-of-the-art imputation algorithms,
(ii) using SNNs to solve vector field problems,
(iii) generalizing coarsening and pooling to simplicial complexes,
(iv) using boundaries and coboundaries to mix data structured by relationships of different dimensions,
and (v) studying the expressive power of SNNs.
% Their expressive power remains to be fully understood on non-homogeneous spaces.

Unrelated to the simplicial nature of this work, we would like to emphasize how the spectral language was key to developing and even formulating our method.
% Fourier analysis was developed to exploit symmetries in solving PDEs, and later used for data analysis and processing.
On homogeneous spaces, convolutions are defined as inner-products with filters shifted by the actions of a symmetry group of the space.
They are the most general shift-invariant linear operators.
% Convolutions are a sufficient and necessary condition for shift-invariance~\cite{kondor2018groupnn}
On non-homogeneous spaces however, the spectral language yields generalized convolutions which are inner-products with \emph{localized} filters~\cite[Sec.~2.4]{perraudin2019deepsphere}. %~\cite[Sec.~2.2]{perraudin2017stationarity}.
Those too are invariant to any symmetry the space might have.
% generality of spaces: homogeneous ⊂ global symmetries ⊂ some automorphisms (local symmetries) ⊂ asymmetric
% (homogeneous == any point is moved to any other by an automorphism)
Convolutions exploit the space's structure to reduce learning complexity by sharing learnable weights through shifts and localizations of filters.


\documentclass[10pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\begin{document}

\section{Stochastic Process}
A stochastic process is of the form
\begin{align}
X_{T+1}-X_{T} = \theta(T) X_{T} +f(t) +r_{T} \sigma(T),
\end{align}
where $r_T$ is a random variable drawn from the distribution $P(r,\vec{\lambda})$, where $\vec{\lambda}$ are the parameters that describe the distribution.

\begin{align}
P(X_T) &=  \int dX_{T-\Delta T}  \int dr_T \ P(X_T,X_{T-\Delta T},r), \\
 &=  \int dX_{T-\Delta T}  \int dr_T \ P(X_T|X_{T-\Delta T},r)P(X_{T-\Delta T},r_T),
\end{align}
assuming that $X_{T-\Delta T}$ is independent of the random variable $r_T$, then this expression becomes
\begin{align}
P(X_T) &= \int dX_{T-\Delta T}  \int dr_T \ P(X_T|X_{T-\Delta T},r_T)P(X_{T-\Delta T})P(r_T) .
\end{align}
Next, we apply the constraint where
\begin{align}
r_{T} = \frac{X_{T+1}-X_T -\theta(T) X_{T}-f(T) }{\sigma(T)},
\end{align}
this constraint is enforced through the function
\begin{align}
P(X_T|X_{T-\Delta T},r_T) = \delta\left(r_{T} -\frac{X_{T}-X_{T-1} -\theta(T-\Delta T) X_{T-\Delta T}-f(T) }{\sigma(T-\Delta T)} \right).
\end{align}
Carrying out the integral over $r_T$ results in
\begin{align}
P(X_T)  &=  \int dX_{T-\Delta T} P(X_{T-\Delta T})P\left(r_T = \frac{X_{T}-X_{T-1} -\theta(T-\Delta T) X_{T-\Delta T}-f(T-\Delta T) }{\sigma(T-\Delta T)}\right),
\end{align}
this last expression relates the probability distribution of the current value $X_T$ to the value at a time infinitesimally in the past. Carrying out this expression recursively we obtain
\begin{align}
P(X_T)  &=  \int dX_{T-\Delta T}  \ P\left(\frac{X_{T}-X_{T-\Delta T}-f(T-\Delta T) -\theta(T-\Delta T) X_{T-\Delta T} }{\sigma(T-\Delta T)}\right)  \notag\\
& \cdots \int dX_{0} \ P\left(\frac{X_{1}-X_{0}-f(T_0) -\theta(T_0) X_{0} }{\sigma(T_0)}\right)\delta(X_0 -x_0),
\end{align}
where $x_0$ is the position at the initial point $T_0$.\\

\section{Gaussian Noise}
Now we have the general forms of the stochastic integrals that we want to work with

\subsection{Fokker-Plank Equation}

\subsection{Path integrals}
This probability can also be expressed as a path integral of the form
\begin{align}
P(X_T)  &=  \int \mathcal{D}X \ \text{Exp}\left(\int\limits_{T_0}^T S(\dot{X},X,t)  dt \right),
\end{align}
where the action is
\begin{align}
S(\dot{X},X,t) &= -\frac{1}{2\sigma^{2}(t)}\left( \dot{X}(t)-f(t)-\theta(t) X(t) \right)^2
\end{align}

\end{document}


\subsection{Linearity}
\noindent
Another useful way to classify differential equations is by linearity.
\begin{definition}
	A differential equation is linear if all terms in the equation involving the dependent variables and its derivatives are in linear terms.
\end{definition}
\noindent
``Linear terms'' means that dependent variables should all be of degree 1, not be multiplied by a derivative involving the same variable, and not be in other functions like $\sin$ or $\ln$.
However, this does not exclude differential equations from having parts that are functions of only independent variables, like in equation 4.\\

\noindent
Below is a table of linearity and equation numbers.
\begin{table}[H]
	\centering
	\begin{tabular}{c|c}
		Linearity & Equation Number \\
		\hline
		Linear &  2, 3, 4, 5, 8, 9 \\
		Nonlinear & 1, 6, 7, 10 \\
	\end{tabular}
\end{table}
\noindent
We'll focus a lot of time on linear equations because we have some mathematical tools that are good at dealing with them.


\documentclass[a4paper,12pt]{scrartcl}

% font packages
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% packages for mathematical type setting
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{bbm}

% packge for norm command
\usepackage{physics}

% references
\usepackage{cleveref}

\author{\normalsize Linus Böhm, Jurek Rostalsky}
\title{Benign Overfitting}
\date{}

% formatting
\setlength{\parindent}{0pt}
\pagestyle{empty}

% definition
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]

\begin{document}
\maketitle
\section{Basic definitions} \label{sec:basic_definitions}
Let \(x \in \mathbb{H}, y \in \mathbb{R}\) with zero mean. Where \(\mathbb{H}\) is a Hilbert space.

\begin{definition} [covariance matrix]
	\label{def:covarianvce_matrix}
	\begin{equation}
		\Sigma = \mathbb{E}\left(\left(x - \mathbb{E}(x)\right)\left(x - \mathbb{E}(x)\right)^T\right) =  \mathbb{E}(xx^T)
	\end{equation}
\end{definition}

\begin{definition} [linear regression]
	\label{def:linear_regression}
	The problem of finding a parameter vector \(\theta^\ast \in \mathbb{H}\) with
	\begin{equation}
		\theta^\ast = arg \min\limits_\theta \mathbb{E}\left((y - x^T \theta)^2\right)
	\end{equation}
	is called \textbf{linear regression}.
\end{definition}

Let \(\left((x_1, y_1), ..., (x_n, y_n)\right) \in (\mathbb{H} \times \mathbb{R})^n\) a list of \(n\) sampled data points. Now we define the matrix \(X = \big(x_1 \, x_2 \, ... \, x_n\big)\) and the vector \(y = (y_1 \, y_2 \, ... \, y_n)^T\). If there is a \(\theta \in \mathbb{H}\) with \(y - X^T \theta = 0\) that \(\theta\) is a minimum of the linear regression problem sind the expectation of a square is non negative. Usually such a \(\theta\) isn't unique, so we are interested in the minimum norm \(\theta\) with that property.

\begin{definition}[minimum norm estimator]
	For given samples \(X \in \mathbb{H}^n, y \in \mathbb{R}^n\). The \textbf{minimum norm estimator} \(\theta\) is the solution of the QQP:
		\begin{align*}
			\label{eq:QQP}
			\hat{\theta} = arg \min\limits_{\theta} &\norm{\theta}^2 & \text{subject to: } \norm{X^T \theta - y}^2 = \min\limits_\beta \norm{X^T \beta - y}^2 && \text{(QQP)}\\
		\end{align*}
\end{definition}

The minimum norm estimator can be obtained by solving the normal equation:
\begin{equation}
\label{eq:normal_equation}
	XX^T \theta = X y,
\end{equation}
which can be done by numerical stable with QR-decomposition.

\newpage

\begin{definition} [Excess risk]
	\label{def:Excess risk}\ \\

	$\mathbb{E}_{x,y}$ denotes the conditional expectation , then define:

	\begin{equation}
	R:= \mathbb{E}_{x,y}[(y - x^T\theta)^2 - (y - x^T\theta^*)^2]
	\end{equation}
\end{definition}

\begin{definition} [Effective Ranks]
	\label{def:Effective Ranks}\ \\

	For the covariance operator $\sum$, define $\lambda_i = \mu_i(\sum)$ for $i = 1,2,...$ . Whereby \newline $\mu_1(\sum) \geq \mu_1(\sum) \geq ...$ . If $\sum\limits_{i=1}^\infty \lambda_i < \infty$ and $\lambda_{k+1} > 0$ for $k \geq 0,$ define:

	\begin{equation}
	r_k(\sum) = \frac{\sum_{i>k}\lambda_i}{\lambda_{k+1}} ,\hspace*{2cm}
    R_k(\sum) = \frac{(\sum_{i>k}\lambda_i)^2}{R_k(\sum_{i>k})\lambda_i^2}
	\end{equation}
\end{definition}

\newpage
\section{Theorems} \label{sec:Theorems}

\newtheorem{thm}{Theorem}

\begin{thm}\ \\
	For any $\sigma_x$ there are $b,c,c_1 > 1$, for which the following holds. Consider a linear regression problem from definition~\ref{def:linear_regression}. Define:
\begin{equation}
\begin{aligned}
k^* = \min \{k \geq 0: r_k(\sum) \leq bn\},
\end{aligned}
\end{equation}

Where the minimum of the empty set is defined as $\infty$. Suppose $\delta < 1$ with $\log(\frac{1}{\delta}) < n/c$. If $k^* \leq n/c_1$, then $\mathbb{E}R(\hat{\theta}) \leq \delta^2/c.$ Otherwise,

\begin{equation}
\begin{aligned}
R(\hat{\theta}) \leq c(\norm{\theta^*}^2\norm{\sum}\max{\sqrt{\frac{r_0(\sum)}{n}},\frac{r_0(\sum)}{n},\sqrt{\frac{\log(1/\delta)}{n}}}) + c\log(\frac{1}{\delta})\sigma_y^2\left(\frac{k^*}{n} + \frac{n}{R_{k^*(\sum)}}\right)\\
\end{aligned}
\end{equation}

with probability at least $1 - \delta$, and

\begin{equation}
\begin{aligned}
\mathbb{E}R(\hat{\theta}) \geq \frac{\sigma^2}{c} \left(\frac{k^*}{n} + \frac{n}{R_{k^*(\sum)}}\right)
\end{aligned}
\end{equation}
Moreover there are universal constants $a_1.a_2,n_0$ such that $\forall n \geq n_0, \forall \sum, \forall t \geq 0$ there is a $\theta^*$ with $\norm{\theta^*} = t$ such that for $x \sim N(0,\sum)$ and $y|x \sim N(x^T\theta^*,\norm{\theta^*}^2\norm{\sum})$, with probability at least $1/4$,

\begin{equation}
\begin{aligned}
R(\hat{\theta})\geq \frac{1}{a_1}\norm{\theta^*}^2\norm{\sum}\mathbbm{1}_{\left[\frac{r_0(\sum)}{n\log(1+r_0(\sum))} \geq a_2\right]}
\end{aligned}
\end{equation}

\end{thm}

\begin{thm}\ \\
The excess risk of the minimum norm estimator satisfies:
\begin{equation}
\begin{aligned}
R(\hat{\theta)}) & \leq 2(\theta^*)^TB\theta^* + c\delta^2\log(\frac{1}{\delta})tr(C) \text{ and} \\
\mathbb{E}_\epsilon R(\hat{\theta}) & \geq (\theta^*)^TB\theta* + \delta^2tr(C) \text{ where}\\\\
B & = (I - X^T(XX^T)^{-1}X)\sum(I - X^T(XX^T)^{-1}X),  \\
C & = (XX^T)^{-1}X\sum X^T(XX^T)^{-1}
\end{aligned}
\end{equation}

\end{thm}

\end{document}


\chapter{Formulae}

{\color{red} This chapter may be not required}

Plane wave basis $b_{\alpha}(\mathbf{r})$:
\begin{equation}
b_{\alpha}(\mathbf{r}) = \frac{1}{\sqrt{\Omega}} e^{\mathbf{G}_{\alpha}\cdot\mathbf{r}}
\end{equation}

Lattice vectors of unit cell in reciprocal space:
\begin{equation}\label{eq:recvecs}
\mathbf{b} = 2\pi\left( a^{T} \right)^{-1}
\end{equation}

\textbf{G}-vectors:
\begin{equation}
\mathbf{G} = i \mathbf{b}_{1} + j \mathbf{b}_{2} + k \mathbf{b}_{3}
\end{equation}

Structure factor:
\begin{equation}
S_{I}(\mathbf{G}) = \sum_{\mathbf{G}} e^{ -\mathbf{G}\cdot\mathbf{X}_{I} }
\end{equation}


%Some d^i and s^i fixed to d_i and s_i.
%Rhetorical questions are okay in spoken lectures but I think they should be used more sparingly in text.

\section{Simplices, more about homology}
Previously we introduced the standard $n$-simplex $\Delta^n\subseteq\mathbf{R}^{n+1}$. Singular simplices in a space $X$ are maps $\sigma\colon\Delta^n\to X$ and constitute the set $\Sin_n(X)$. For example, $\Sin_0(X)$ consists of points of $X$. In addition to the \emph{face maps} $d_i\colon\Sin_n(X)\to\Sin_{n-1}(X)$ we described, there are also \emph{degeneracy maps} $s_i\colon\Sin_n(X)\to\Sin_{n+1}(X)$, and the collection $\{\Sin_n(X),d_i,s_i\}$ forms a \emph{simplicial set}. Simplicial sets are combinatorial models for topological spaces. In the language of category theory, which we will discuss shortly, we have a functor $\mathbf{Top}\to\{\text{simplicial sets}\}$.

To the semi-simplicial set $\{\Sin_n(X),d_i\}$ we then applied the free abelian group functor, obtaining a semi-simplicial abelian group. Using the $d_i$s, we constructed a boundary map $\partial$ which makes $S_\ast(X)$ a \emph{chain complex} because $\partial^2=0$ (see Exercise \ref{exer:simplicialidentities}). We capture this process in a diagram:
\begin{equation*}
\xymatrix{\mathbf{Top}\ar[d]\ar[r] & \{\text{semi-simplicial sets}\}\ar[r] & \{\text{semi-simplicial abelian groups}\}\ar[d]\\
    \{\text{simplicial sets}\}\ar[ur] & & \{\text{chain complexes}\}\ar[d]^{\text{take homology}}\\
 & &\{\text{graded abelian group}\}}
\end{equation*}
Given a chain complex $\partial\colon A_n\to A_{n-1}$, one can define its homology $H_n(A,\partial)=\ker\partial_n/\img\partial_n$.

Here's an example. Suppose we have $\sigma\colon \Delta^1\to X$. Define $\phi\colon\Delta^1\to\Delta^1$ which sends $(t,1-t)\mapsto (1-t,t)$. Precomposing $\sigma$ with $\phi$ gives another singular simplex $\overline{\sigma}$ which reverses the orientation of $\sigma$. It is \textit{not} true that $\overline{\sigma}=-\sigma$ in $S_1(X)$.

However, we show that $\overline{\sigma}\equiv -\sigma\bmod B_1(X)$, meaning there is a $2$-chain in $X$ whose boundary is $\overline{\sigma}+\sigma$. If $d_0\sigma=d_1\sigma$ so that $\sigma\in Z_1(X)$, then $\overline{\sigma}$ and $-\sigma$ are homologous: $[\overline{\sigma}]=-[\sigma]$ in $ H_1(X)$.

\begin{figure}
	\centering
	\includegraphics[width=0.6\linewidth]{assets/L02/02-pi-then-sigma}
	\caption{If $\sigma$ is a 1-simplex in $X$, then precomposing by $\pi$ gives a 2-simplex $\sigma \circ \pi$ in $X$.}
	\label{fig:02-pi-then-sigma}
\end{figure}

Let $c^n_{x}$ denote the constant $n$-simplex at $x$. Consider the 2-simplex in Figure \ref{fig:02-pi-then-sigma}. Its boundary is $\partial(\sigma\circ\pi)=\sigma\pi d^0-\sigma\pi d^1 +\sigma\pi d^2=\overline{\sigma}-c^1_{\sigma(0)}+\sigma$. To compensate for the unwanted $-c^1_{\sigma(0)}$ term, consider the constant $2$-simplex $c^2_{\sigma(0)}$; its boundary is $c^1_{\sigma(0)}-c^1_{\sigma(0)}+c^1_{\sigma(0)}=c^1_{\sigma(0)}$. So $\overline{\sigma}+\sigma=\partial(\sigma\circ\pi + c^2_{\sigma(0)})$ and $\overline{\sigma}\equiv -\sigma\bmod B_1(X)$ as claimed.

To give the simplest explicit examples, let's compute the homologies of $\emptyset$ and $\ast$. For the former, $\Sin_n(\emptyset)=\emptyset$, so $S_\ast(\emptyset)=0$. Hence $\cdots\to S_2\to S_1\to S_0$ is the zero chain complex. This means that $Z_\ast(\emptyset)=B_\ast(\emptyset)=0$. The homology in all dimensions is therefore $0$.

For $\ast$, we have $\Sin_n(\ast)=\{c^n_\ast\}$ for all $n\geq 0$. Consequently $S_n(\ast)=\mathbf{Z}$. The boundary maps $\partial\colon S_n(\ast)\to S_{n-1}(\ast)$ in the chain complex depend on the parity of $n$ as follows:
\[\partial(c^n_\ast)=\sum_{i=0}^{n}(-1)^i c^{n-1}_\ast=
\begin{cases}
    c^{n-1}_* & \text{for } n \text{ even, and}\\
    0 & \text{for } n \text{ odd.}
  \end{cases}
\]
This means that our chain complex is:
$$\cdots\to\mathbf{Z}\xrightarrow{0}\mathbf{Z}\xrightarrow{1}\mathbf{Z}\xrightarrow{0}\mathbf{Z}\to 0.$$
The boundaries coincide with the cycles except in dimension zero, where $B_0(\ast)=0$ while $Z_0(\ast)=\mathbf{Z}$. Therefore $ H_0(\ast)=\mathbf{Z}$ but $ H_i(\ast)=0$ for $i>0$.

We've defined homology groups for each space, but haven't considered what happens to maps between spaces. A continuous map $f\colon X\to Y$ induces a map $f_\ast\colon \Sin_n(X)\to\Sin_n(Y)$ by composition: $\sigma\mapsto f\circ \sigma=:f_\ast\sigma$. For $f_\ast$ to be a map of semi-simplicial sets, it needs to commute with face maps. Explicitly, we need $f_\ast \circ d_i = d_i \circ f_\ast$. A diagram is said to be \emph{commutative} if all composites with the same source and target are equal, so this is equivalent to commutativity of the below.
\begin{eqnarray*}
\xymatrix{\Sin_n(X)\ar[r]^{f_\ast}\ar[d]^{d_i} & \Sin_n(Y)\ar[d]^{d_i}\\
\Sin_{n-1}(X)\ar[r]^{f_\ast} & \Sin_{n-1}(Y)}
\end{eqnarray*}
We see that $d_if_\ast\sigma=(f_\ast\sigma)\circ d^i=f\circ\sigma\circ d^i$, and $f_\ast(d_i\sigma)=f_\ast(\sigma\circ d^i)=f\circ\sigma\circ d^i$ as desired. The diagram remains commutative when we pass to the free abelian groups of chains.

If $C_\ast$ and $D_\ast$ are chain complexes, a \emph{chain map} $f\colon C_\ast\to D_\ast$ is a collection of maps $f_n\colon C_n\to D_n$ such that the following diagram commutes for every $n$:
\begin{equation*}
    \xymatrix{
	C_n\ar[r]^{f_n}\ar[d]^{\partial_C} & D_n\ar[d]^{\partial_D}\\
	C_{n-1}\ar[r]^{f_{n-1}} & D_{n-1}
    }
\end{equation*}
For example, if $f\colon X\to Y$ is a continuous map, then $f_\ast \colon S_\ast(X)\to S_\ast(Y)$ is a chain map as discussed above.

A chain map induces a map in homology $f_\ast: H_n(C)\to H_n(D)$. The method of proof is a so-called ``diagram chase'' and it will be the first of many. We check that we get a map $Z_n(C)\to Z_n(D)$. Let $c\in Z_n(C)$, so that $\partial_C c = 0$. Then $\partial_D f_n(c) = f_{n-1}\partial_C c = f_{n-1}(0) = 0$, because $f$ is a chain map. This means that $f_n(c)$ is also an $n$-cycle, i.e., $f$ gives a map $Z_n(C)\to Z_n(D)$.

Similarly, we also get a map $B_n(C)\to B_n(D)$. Let $c\in B_n(C)$, so that there exists $c^\prime \in C_{n+1}$ such that $\partial_C c^\prime = c$. Then $f_n(c) = f_n\partial_C c^\prime = \partial_D f_{n+1}(c^\prime)$. Thus $f_n(c)$ is the boundary of $f_{n+1}(c^\prime)$, and $f$ gives a map $B_n(C)\to B_n(D)$.

The two maps $Z_n(C)\to Z_n(D)$ and $B_n(C)\to B_n(D)$ give a map on homology $f_\ast: H_n(X)\to H_n(Y)$, as desired.


\section{Linear Smoothers: Influence, Variance, and Degrees of Freedom}
All the smoothers we have discussed in this class are linear
smoothers. The
estimates of the regression function can be written as
\[
\hat{\f} = \bS \by.
\]

For some of the smoothers we have defined we can define a weight sequence
for any $x$ and define
\[
\hat{f}(x) = \sum_{i=1}^n W_i (x) y_i.
\]

How can we characterize the amount of smoothing being performed? The
smoothing parameters provide a characterization, but it is not ideal
because it does not permit us to compare between different smoothers
and for smoothers like loess it does not take into account the shape
of the weight function nor the degree of the polynomial being fit.

We now use the connections between smoothing and multivariate linear
regression (they are both linear smoothers) to characterize pointwise
criteria that characterize the amount of smoothing at a single point
and global criteria that characterize the global amount of smoothing.

We will define variance reduction, influence, and degrees of freedom
for linear smoothers.

The variance of the interpolation estimate is $\var[y_1] =
\sigma^2$. The variance of our smooth estimate is
\[
\var[\hat{f}(x)] = \sigma^2 \sum_{i=1}^n W_i^2(x)
\]
so we define $\sum_{i=1}^n W_i^2(x)$ as the variance reduction. Under
mild conditions one can show that this is less than 1.

\begin{figure}[htb]
\caption{Degrees of freedom for loess and smoothing splines as
functions of the smoothing parameter}
\begin{center}
\epsfig{figure=Plots/plot-06-01.ps,angle=270,width=.8\textwidth}
\end{center}
\end{figure}

Because
\[
\sum_{i=1}^n \var[\hat{f}(x_i)] = \tr(\bS\bS')\sigma^2,
\]
the total variance reduction from $\sum_{i=1}^n \var[y_i]$ is
$\tr(\bS\bS')/n$.

In linear regression the variance reduction is related to the degrees of
freedom, or number of parameters. For linear regression, $\sum_{i=1}^n
\var[\hat{f}(x_i)]
= p \sigma^2$. One widely used definition of degrees of freedoms for
smoothers is $df = \tr(\bS\bS')$.

The sensitivity of the fitted value, say $\hat{f}(x_i)$, to the data
point $y_i$ can be
measured by $W_i(x_i)/\sum_{i=1}^n W_n(x_i)$ or $\bS_{ii}$ (remember
the denominator is usually 1).

The total influence or sensitivity is $\sum_{i=1}^n W_i(x_i) =
\tr(\bS)$.

In linear regression $\tr(\bS)=p$ is also equivalent to the degrees of
freedom. This is also used as a definition of degrees of freedom.

\begin{figure}[htb]
\caption{Comparison of three definition of degrees of freedom}
\begin{center}
\epsfig{figure=Plots/plot-06-02.ps,angle=270,width=.8\textwidth}
\end{center}
\end{figure}

Finally we notice that
\[
\E[ (\by - \hat{\f})'(\by - \hat{\f}) ] = \{n - 2\tr(\bS) +
\tr(\bS\bS')\}\sigma^2
\]
In the linear regression case this is $(n-p)\sigma^2$. We therefore denote
$n - 2\tr(\bS) + \tr(\bS\bS')$ as the residual degrees of freedom. A
third definition of degrees of freedom of a smoother is then
$2\tr(\bS) - \tr(\bS\bS')$.

Under relatively mild assumptions we can show that
\[
1 \leq \tr(\bS\bS') \leq \tr(\bS) \leq 2\tr(\bS) - \tr(\bS\bS') \leq n
\]


\subsection{Graphs of Trigonometric Functions}
The graph of the functions $\sin x$ and $\cos x$ can be visually represented as:

$$\includegraphics[width=4.0in]{images/sine-cosine}$$

Both $\sin x$ and $\cos x$ have domain $(-\infty,\infty)$ and range $[-1,1]$.
That is,
$$-1\leq\sin x\leq 1\qquad -1\leq\cos x\leq 1.$$
The zeros of $\sin x$ occur at the integer multiples of $\pi$, that is, $\sin x=0$ whenever $x=n\pi$, where $n$ is an integer.
Similarly, $\cos x=0$ whenever $x=\pi/2+n\pi$, where $n$ is an integer.

The six basic trigonometric functions can be visually represented as:

$$\includegraphics[width=7in]{images/trig-functions-new}$$

Both tangent and cotangent have range $(-\infty,\infty)$, whereas cosecant and secant have range $(-\infty,-1]\cup[1,\infty)$.
Each of these functions is periodic. Tangent and cotangent have period $\pi$, whereas sine, cosine, cosecant and secant have period $2\pi$.