\lab{Numerical Derivatives}{Numerical Derivatives}
\label{lab:NumericalDerivatives}
\objective{Understand and implement finite difference approximations of the derivative in single and multiple dimensions. Evaluate the accuracy of these approximations.  Then use finite difference quotients to find edges in images via the Sobel filter.}

\section*{Derivative Approximations in One Dimension} % =======================

The derivative of a function $f$ at a point $x_0$ is

\begin{equation}
\label{eqn:deriv}
f'(x_0) = \lim_{h\rightarrow 0} \frac{f(x_0 + h)-f(x_0)}{h}.
\end{equation}
%
In this lab, we will investigate one way a computer can calculate $f'(x_0)$.

\subsection*{Forward Difference Quotient} % -----------------------------------

Suppose that in Equation \eqref{eqn:deriv}, instead of taking a limit, we just pick a small value for $h$.
Then we would expect $f'(x_0)$ to be close to the quantity
%
\begin{equation}\label{equ:forward_diff}
\frac{f(x_0 + h)-f(x_0)}{h}.
\end{equation}
%
This quotient is called the \emph{first order forward difference approximation} of the derivative.
Because $f'(x_0)$ is the limit of such quotients, we expect that when $h$ is small, this quotient is close to $f'(x_0)$.
We can use Taylor's formula to find just how close.

By Taylor's formula,
\[
f(x_0+h) = f(x_0) + f'(x_0)h + R_2(h),
\]
where $R_2(h) = \left( \int_0^1 (1-t) f''(x_0+th) dt \right) h^2$.
(This is called the \emph{integral form} of the remainder for Taylor's Theorem; see Volume 1 Chapter 6). When we solve this equation for $f'(x_0)$, we get
%
\begin{equation}\label{equ:forward_diff_with_remainder}
f'(x_0) = \frac{f(x_0+h)-f(x_0)}{h} - \frac{R_2(h)}{h}.
\end{equation}
%
Thus, the error in using the first order forward difference quotient to approximate $f'(x_0)$ is
\[
\left | \frac{R_2(h)}{h} \right | \leq |h| \int_0^1 |1 - t||f''(x_0+th)|dt.
\]
If we assume $f''$ is continuous, then for any $\delta$, set $M = \sup_{x \in (x_0-\delta, x_0+\delta)} f''(x)$. Then if $|h| < \delta$, we have
\[
\left | \frac{R_2(h)}{h} \right | \leq |h|\int_0^1 M dt = M|h|  {\in}  O(h).
\]
Therefore, the error in using \eqref{equ:forward_diff} to approximate $f'(x_0)$ grows like $h$.

\subsection*{Centered Difference Quotient} % ----------------------------------

In fact, we can approximate $f'(x_0)$ to the second order with another difference quotient, called the \emph{centered difference quotient}.  We begin by trying to find the \emph{backward difference quotient}.
Evaluate Taylor's formula at $x_0-h$ to derive
%
\begin{equation}\label{equ:backwards_diff_with_remainder}
f'(x_0) = \frac{f(x_0)-f(x_0-h)}{h} + \frac{R_2(-h)}{h}.
\end{equation}
%
The first term on the right hand side of \eqref{equ:backwards_diff_with_remainder} is called the \emph{backward difference quotient.}
This quotient also approximates $f'(x_0)$ to first order, so it is not the quotient we are looking for.
When we add \eqref{equ:forward_diff_with_remainder} and \eqref{equ:backwards_diff_with_remainder} and solve for $f'(x_0)$ (by dividing by 2), we get
%
\begin{equation}\label{equ:centered_diff}
f'(x_0) = \frac{\frac{1}{2}f(x_0+h) - \frac{1}{2}f(x_0-h)}{h} + \frac{R_2(-h) - R_2(h)}{2h}
\end{equation}
%
The \emph{centered difference quotient} is the first term of the right hand side of \eqref{equ:centered_diff}.
Let us investigate the remainder term to see how accurate this approximation is.
Recall from the proof of Taylor's theorem that $R_k = \frac{f^{(k)}(x_0)}{k!}h^k + R_{k+1}$. Therefore,
%
\begin{align*}
\frac{R_2(-h) - R_2(h)}{2h} &= \frac{1}{2h}\left(\frac{f''(x_0)}{2}h^2 + R_{3}(-h) - \frac{f''(x_0)}{2}h^2 - R_{3}(h) \right)\\
&= \frac{1}{2h} ( R_3(-h)-R_3(h))\\
&= \frac{1}{2h}\left(  \left( \int_0^1 \frac{(1-t)^2}{2} f'''(x_0+th) dt \right) h^3  -  \left(\int_0^1 \frac{(1-t)^2}{2} f'''(x_0-th) dt \right) h^3  \right)\\
&= \left(  \int_0^1 \frac{(1-t)^2}{4}( f'''(x_0+th)-f'''(x_0-th)) \right)h^2\\
&{\in}   O(h^2)
\end{align*}
%
once we restrict $h$ to some $\delta$-neighborhood of 0.
So the error in using the centered difference quotient to approximate $f'(x_0)$ grows like $h^2$, which is smaller than $h$ when $|h|<1$.

\subsection*{Accuracy of Approximations} % ------------------------------------

Let us discuss what step size $h$ we should plug into the difference quotients to get the best approximation to $f'(x_0)$.
Since $f'$ is defined as a limit as $h \rightarrow 0$, you may think that it is best to choose $h$ as small as possible, but this is not the case.
In fact, dividing by very small numbers causes errors in floating point arithmetic.
This means that as we decrease $|h|$, the error between $f'(x_0)$ and the difference quotient will first decrease, but then increase when $|h|$ gets too small because of floating point arithmetic.

Here is an example with the function $f(x) = e^x$. A quick way to write $f$ as a function in Python is with the \li{lambda} keyword.

\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt
>>> f = lambda x: np.exp(x)
\end{lstlisting}

In general, the line \li{f = lambda <params> : <expression>} is equivalent to defining a function \li{f} that accepts the parameters \li{params} and returns \li{expression}.

Next we fix a step size \li{h} and define an approximation to the derivative of \li{f} using the \emph{centered difference quotient}.

\begin{lstlisting}
>>> h = 1e-1
>>> Df_app = lambda x: .5*(f(x+h)-f(x-h))/h
\end{lstlisting}

Finally, we check the accuracy of this approximation at $x_0=1$ by computing the difference between \li{Df_app(1)} and the actual derivative evaluated at 1.

\begin{lstlisting}
# Since f(x) = e^x, the derivative of f(x) is f(x)
>>> np.abs( f(1)-Df_app(1) )
0.0045327354883726301
\end{lstlisting}

We note that our functions \li{f} and \li{Df_app} behave as expected when they are passed a NumPy array.

\begin{lstlisting}
>>> h = np.array([1e-1, 1e-3, 1e-5, 1e-7, 1e-9, 1e-11])
>>> np.abs( f(1)-Df_app(1) )
array([  4.53273549e-03,   4.53046679e-07,   5.85869131e-11,
         5.85873572e-11,   6.60275079e-09,   1.04294937e-05])
\end{lstlisting}

These results are summarized in Table \ref{table:approx_errors}.

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
    \hline
    $h$     & 1e-1 & 1e-3 & 1e-5  & 1e-7  & 1e-9 & 1e-11 \\ \hline
    Error   & 5e-3 & 5e-7 & 6e-11 & 6e-11 & 7e-9 & 1e-5  \\ \hline
\end{tabular}
\caption{This table shows that it is best not to choose $h$ too small when you approximate derivatives with difference quotients.
Here, ``Error'' equals the absolute value of $f'(1)-f_{app}(1)$ where $f(x) = e^x$ and $f_{app}$ is the centered difference approximation to $f'$.}
\label{table:approx_errors}
\end{center}
\end{table}

Thus, the optimal value of $h$ is one that is small, but not too small. A good choice is \li{h = 1e-5}.

\begin{problem}
Write a function that accepts as input a callable function object \li{f}, an array of points \li{pts}, and
a keyword argument \li{h} that defaults to \li{1e-5}.
Return an array of the \emph{centered difference quotients} of \li{f} at each point in \li{pts} with the specified value of \li{h}.
\end{problem}

\begin{comment}
\begin{problem}
Write a function that accepts as input a callable function object \li{f}, an array of points \li{pts}, and
a keyword argument \li{h} that defaults to \li{1e-5}.  Return an array of the errors for the \emph{centered difference quotients} at each point in \li{pts} with the specified value of \li{h}.
\end{problem}
\end{comment}

You may wonder if the forward or backward difference quotients are ever used, since the centered difference quotient is a more accurate approximation of the derivative.
In fact, there are some functions that in practice do not behave well under centered difference quotients.
In these cases, one must use the forward or backward difference quotient.

Finally, we remark that forward, backward, and centered difference quotients can be used to approximate higher-order derivatives of $f$.
However, taking derivatives is an \emph{unstable} operation.
This means that taking a derivative can amplify the arithmetic error in your computation.
For this reason, difference quotients are not generally used  to approximate derivatives higher than second order.

\begin{comment}
The derivative of a function at a point is formally defined as

\begin{equation}
\label{eqn:deriv}
f'(x) = \lim_{h\rightarrow 0} \frac{f(x + h)-f(x)}{h}.
\end{equation}

In most real world applications we will be solving problems using computers. How does a computer calculate a limit? In short it can't. Computers can only approximate functions at specific points, and the notion of a limit graces infinity in a way that a computer never can.

So how can we use a computer to find the derivative of a function, particularly when we can't differentiate the function by hand? We use methods known as finite difference methods. For example suppose that in equation \ref{eqn:deriv}, instead of taking a limit we just pick a particularly small value for h. Then we have

\begin{equation*}
f'(x) \approx \frac{f(x + h)-f(x)}{h}
\end{equation*}

This is known as the first order forward difference approximation of the derivative.

How do we know the quality of this approximation? We can use Taylor's formula to find

\begin{equation*}
f(x_0 + h) = f(x_0) + hf'(x_0) + h^2/2 f''(\xi),\hspace{5mm} \xi \in (x_0,x_0 + h)
\end{equation*}

Which can be also expressed as

\begin{equation*}
f'(x_0) = \frac{f(x_0 + h) - f(x)}{h} + \frac{h}{2}f''(\xi) = \frac{f(x_0 + h) - f(x)}{h} + O(h)
\end{equation*}

Here we use the big-O notation to denote that the errors are bounded by some constant multiplied by $h$.

We can use Taylor expansions to find approximations that have different big-O error bounds, up to any polynomial of arbitrary degree. Tables \ref{Table:CDiff} and \ref{Table:FDiff} offer the coefficients for centered and forward difference schemes.

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\hline
Derivative & Accuracy & -3 & -2 & -1 & 0 & 1 & 2 & 3 \\ \hline
 & 2 & & & -1/2 & 0 & 1/2 & & \\ \cline{2-9}
 1 & 4 & & 1/12 & -2/3 &  0 & 2/3 & -1/12 & \\ \cline{2-9}
  & 6 & -1/60 & 3/20 & -3/4 & 0 & 3/4 & -3/20 & 1/60 \\ \hline
  & 2 & & & 1 & -2 & 1 & & \\ \cline{2-9}
 2 & 4 & & -1/12 & 4/3 &  -5/2 & 4/3 & -1/12 & \\ \cline{2-9}
  & 6 & 1/90 & -3/20 & 3/2 & -49/18 & 3/2 & -3/20 & 1/90 \\ \hline
\end{tabular}
\caption{Centered Difference Coefficients}
\label{Table:CDiff}
\end{center}
\end{table}

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
Derivative & Accuracy & 0 & 1 & 2 & 3 & 4 \\ \hline
 & 1 & -1 & 1 &  & &  \\ \cline{2-7}
 1 & 2 & -3/2 & 2 & -1/2 & &  \\ \cline{2-7}
  & 3 & -11/6 & 3 & -3/2 & 1/3 &  \\ \hline
  & 1 & 1 & -2 & 1 &  & \\ \cline{2-7}
 2 & 2 & 2 & -5 & 4 &  -1 &  \\ \cline{2-7}
  & 3 & 35/12 & -26/3 & 19/2 & -14/3 & 11/12 \\ \hline
\end{tabular}
\caption{Forward Difference Coefficients}
\label{Table:FDiff}
\end{center}
\end{table}

These tables can be used by simply summing the function evaluations (the number at the top represents how many times $h$ is added to $x$), and then dividing by $h^n$, where $n$ is the degree of the derivative.

So, for example, the centered difference estimate of the second derivative that is $O(h^4)$ is
\begin{equation*}
f''(x) \approx \frac{-1/12(f(x-2h) + f(x+2h)) + 4/3(f(x-h) + f(x+h)) -5/2f(x)}{h^2}
\end{equation*}

Or, the forward difference estimate for the first derivative that is $O(h^2)$ is

\begin{equation*}
f'(x) \approx \frac{-3/2f(x) + 2f(x+h) - 1/2 f(x+2h)}{h}
\end{equation*}

It should be noted that we can convert a forward difference estimate to a backwards difference estimate by using $-h$. So the backwards difference estimate for the first derivative that is $O(h^2)$ is

\begin{equation*}
f'(x) \approx \frac{3/2f(x) - 2f(x-h) + 1/2 f(x-2h)}{h}
\end{equation*}

There are two important observations that you should make about these tables. First, in order to get higher order approximations we need to evaluate the function at more points. This should not be surprising. Second, you should notice that centered difference formulas require less function evaluations to get higher order approximations. However, in certain applications it is not possible to use centered difference formulas, so the backwards and forwards formulas are still very applicable.

One important aspect of this method is selecting an appropriate $h$. The natural temptation is to pick a very very small value. However, this is not always advisable. Note the values in table \ref{Table:FloatingError}, which approximates the derivative of $e(x)$ at $x = 1$:

\begin{table}[h!]
\begin{center}
\begin{tabular}{|cc|}
\hline
h & Error  = $|f'(1)-f'_{app}(1)|$ \\ \hline
1e-1 & 4.5e-3 \\
1e-3 & 4.5305e-7 \\
1e-7 & 5.8587e-11 \\
1e-10 & 6.7274e-7 \\ \hline
\end{tabular}
\caption{Error in numerical derivative, using double precision floating point arithmetic}
\label{Table:FloatingError}
\end{center}
\end{table}

As you can see, the error actually increases as $h$ becomes very small. Why is this? Division by small numbers causes errors in floating point arithmetic. So, be aware that usually the optimal $h$ is of moderately small size. However, in the framework of double floating point arithmetic, this is usually less of a concern.

As a matter of reference, calculating numerical derivatives is an unstable operation. An unstable operation, informally, is one where errors are magnified by the operation. This usually is not an issue, but it's important to know that taking derivatives can amplify errors.

\begin{problem}
Write a function \li{numDer1} that accepts as inputs: a callable function object \li{f} and
a keyword argument \li{h} giving the step size (default \li{h = 1e-5}). Have the function return an array of the approximate 1st order derivative with accuracy 1 of \li{f} at each of the points in \li{pts}, using the centered  coefficients.
\end{problem}

\begin{problem}
Write a function \li{numDer2} that accepts as inputs: a callable function object \li{f} and
a keyword argument \li{h} giving the step size (default \li{h = 1e-5}). Have the function return an array of the approximate 2nd order derivative with accuracy 1 of \li{f} at each of the points in \li{pts}, using the centered  coefficients.
\end{problem}

\begin{problem}
Write a function \li{numDer} that accepts as inputs: a callable function object \li{f}, an
array of numbers \li{pts}, a keyword argument \li{mode} (taking one of the values
\li{'centered'}, \li{'forward'}, or \li{'backward'}), a keyword argument \li{d} (taking one of
the values \li{1} or \li{2}), a keyword argument \li{o} (taking an integer value in
\li{[2, 4, 6]} if \li{mode = 'centered'}, and otherwise taking a value in \li{[1, 2, 3]}), and
a keyword argument \li{h} giving the step size. The default settings of the keyword
arguments should be \li{mode = 'centered', d = 1, o = 2, h = 1e-5}.

Have the function return an array of the approximate derivative of order \li{d} with accuracy
 \li{o} of \li{f} at each of the points in \li{pts}, using the coefficients indicated by \li{mode}.
\end{problem}

We note that higher order approximations of the derivative can be derived using the Taylor series and Lagrange polynomials, but generally higher-order approximations are not practically useful as they can often be ill-conditioned.

For any numerical approximation method, it is important to be able to empirically calculate
the order of convergence. We will do so for our numerical derivative approximation.
For an $m$-th order approximation of the first derivative, we have that the error is in
$O(h^m)$, so that
$$
err(h) \approx Ch^m
$$
for some constant $C$.
Hence, taking the log of both sides, we obtain
$$
\log err(h) \approx \log C + m\log h,
$$
which means that if we plot the log of the errors against the log of the $h$ values, we
ought to see a linear relationship whose slope gives the rate of convergence.

We do this in python as follows (The \li{numDer} function in this code can use any of the coefficients from \ref{Table:CDiff} and \ref{Table:FDiff}):

\begin{lstlisting}
# assume that the function numDer has already been written
import numpy as np
from matplotlib import pyplot as plt

# approximate the derivative of cosine at x = 3
# create a callable function object
def myCosine(x):
    return np.cos(x)
f = myCosine

# calculate the actual derivative
actual = -np.sin(3.0)

# initialize array of h values at which to calculate the error
hvals = np.linspace(1e-5, 1e-1)
err1 = np.zeros(hvals.shape)
err2 = np.zeros(hvals.shape)

# calculate the errors for order 1 and order 2 approximations for the forward coeffiects
for i in xrange(len(hvals)):
    err1[i] = np.abs(actual - numDer(f, np.array([3.0]), mode = 'forward', h = hvals[i], o=1))
    err2[i] = np.abs(actual - numDer(f, np.array([3.0]), mode = 'forward', h = hvals[i], o=2))

# plot the log of the h values against the log of the errors
plt.subplot(121)
plt.loglog(hvals, err1)
plt.ylim((1e-11, 1e-1))
plt.subplot(122)
plt.loglog(hvals, err2)
plt.ylim((1e-11, 1e-1))
plt.show()
\end{lstlisting}

The generated plot is shown in Figure \ref{fig:convergence}. Note that the slope
of the line in the left plot is about 1, and the slope of the line in the right
plot is about 2. Further, the log of the errors for the order 2 approximations
are much lower than the log of the errors of the order 1 approximations.

\begin{figure}[t]
    \includegraphics[width=0.8\textwidth]{figures/convergence.pdf}
    \caption{Convergence plots for our numerical derivative approximations.
    The left plot shows the convergence for order 1 approximations.
    The right plot shows the convergence for order 2 approximations.}
    \label{fig:convergence}
\end{figure}

\begin{problem}
Explore the convergence properties for different orders of approximation,
and for the second derivative as well. You may need to adjust your $h$
values, as they may be too small for some of the calculations.
\end{problem}

\end{comment}

\section*{Derivative Approximations in Multiple Dimensions} % =================

Finite difference methods can also be used to calculate derivatives in higher dimensions.
Recall that the Jacobian of a function $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$ at a point $x_0 \in \mathbb{R}^n$ is the $m \times n$ matrix $J = (J_{ij})$ defined component-wise by
\begin{equation*}
J_{ij} = \frac{\partial f_i}{\partial x_j}(x_0).
\end{equation*}
For example, the Jacobian for a function $f:\mathbb{R}^3 \rightarrow \mathbb{R}^2$ is defined by

\[
J = \begin{pmatrix}
\frac{\partial f}{\partial x_1}&\frac{\partial f}{\partial x_2}&\frac{\partial f}{\partial x_3}
\end{pmatrix}
= \begin{pmatrix}
\frac{\partial f_1}{\partial x_1}&\frac{\partial f_1}{\partial x_2}&\frac{\partial f_1}{\partial x_3}\\
\frac{\partial f_2}{\partial x_1}&\frac{\partial f_2}{\partial x_2}&\frac{\partial f_2}{\partial x_3}
\end{pmatrix}.
\]

The Jacobian is useful in many applications.  For example, the Jacobian can be used to find zeros of functions in multiple variables.

The forward difference quotient for approximating a partial derivative is
\begin{equation*}
\frac{\partial f}{\partial x_j} (x_0) \approx \frac{f(x_0+h e_j)-f(x_0)}{h},
\end{equation*}
where $e_j$ is the $j^{th}$ standard basis vector.
Similarly, the centered difference approximation is
\begin{equation*}
\frac{\partial f}{\partial x_j} (x_0) \approx \frac{\frac{1}{2}f(x_0+h e_j)-\frac{1}{2}f(x_0-h e_j)}{h}.
\end{equation*}

\begin{problem}
\leavevmode
Write a function that accepts
\begin{enumerate}
\item a function handle \li{f},
\item an integer \li{n} that is the dimension of the domain of \li{f},
\item an integer \li{m} that is the dimension of the range of \li{f},
\item an \li{1 x n}-dimensional NumPy array \li{pt} representing a point in $\mathbb{R}^n$, and
\item a keyword argument \li{h} that defaults to \li{1e-5}.
\end{enumerate}
Return the approximate Jacobian matrix of \li{f} at \li{pt} using the centered difference quotient.
\end{problem}
\begin{problem}
\item Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be defined by
%
\begin{equation*}
f(x, y) =
\left[\begin{array}{c}
e^{x} \sin(y) + y^3 \\
3y - \cos(x)
\end{array}\right]
\end{equation*}
%
Find the error between your Jacobian function and the analytically computed derivative on the square $[-1,1] \times [-1,1]$ using ten thousand grid points (100 per side).
You may apply your Jacobian function to the points one at a time using a double \li{for} loop.  Once you get the error matrix for a given point, calculate the Frobenius norm of this matrix (\li{la.norm} defaults to the Frobenius norm).  This norm will be your total error for that point.
What is the maximum error of your Jacobian function over all points in the square?

Hint: The following code defines the function
$f(x,y) = \left[\begin{array}{c} x^2 \\ x+y \end{array}\right]$.

\begin{lstlisting}
# f accepts a length-2 NumPy array
>>> f = lambda x: np.array([x[0]**2, x[0]+x[1]])
\end{lstlisting}
\end{problem}

\begin{comment}
Given a function from $\mathbb{R}^n \to \mathbb{R}$, sometimes the mixed partial derivatives are useful. In particular, the mixed partials will be useful when we study optimization in Volume 2. This information is contained in the Hessian matrix, which is defined as

\begin{equation*}
H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}
\end{equation*}

We can use the following formula to approximate mixed partial derivatives
\small
\begin{equation*}
\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{f(x + (e_i + e_j)h) - f(x + (e_i-e_j)h) -f(x + (e_j-e_i)h) + f(x - (e_i + e_j)h)}{4h^2}
\end{equation*}
\normalsize

\begin{problem}
Write a Python function that numerically calculates the Hessian of a given function.
The function should be named \li{Hessian}, and should accept as inputs a function handle,
an integer giving the dimension of the domain of the function, a NumPy
array giving the point at which to approximate the Hessian, and an optional argument giving the
step size.
Return the approximated Hessian matrix at the given point.

Test it on the following function
\begin{equation*}
f(x,y) = (1-x)^2 + 100(y-x^2)^2
\end{equation*}
This function is known as the Rosenbrock Banana function, or Rosenbrock's Valley. It is a common test function for optimization algorithms because it is non-convex and the global minimum is hard to find from certain starting points. A graph is shown in figure \ref{Fig:Rosenbrock}. Compare the output of your function with the analytic solution on the region $[-2,2] \times [0,2]$, using ten thousand points. What is the maximum error of your function?
\end{problem}
\begin{figure}
\begin{center}
\includegraphics[width = \textwidth]{figures/Rosenbrock.pdf}
\caption{The Rosenbrock Banana Function, a common test function in optimization algorithms}
\label{Fig:Rosenbrock}
\end{center}
\end{figure}
\end{comment}

\section*{Image Filters} % ====================================================

Recall that a computer stores an image as a 2-D array of pixel values (i.e., a matrix of intensities).
An image filter is a function that transforms an image by operating on it locally.
That is, to compute the $ij^{th}$ pixel value in the new image, an image filter uses only the pixels in a small neighborhood around the $ij^{th}$ pixel in the original image.

In this lab, we will use a filter derived from the gradient of an image to find edges in an image.

\subsection*{Convolutions} % --------------------------------------------------

One example of an image filter is to \emph{convolve} an image with a filter matrix.
A filter matrix is a matrix whose height and width are relatively small odd numbers.
If the filter matrix is
\[
F = \begin{pmatrix}
f_{-1,-1}&f_{-1,0}&f_{-1,1}\\
f_{0,-1}&f_{0,0}&f_{0,1}\\
f_{1,-1}&f_{1,0}&f_{1,1}
\end{pmatrix},
\]
then the convolution of an image $A$ with $F$ is $A \ast F = (C_{ij})$ where
\begin{equation}\label{equ:convolve}
C_{ij} = \sum_{k=-1}^1 \sum_{\ell=-1}^1 f_{k\ell}A_{i+k,j+\ell}.
\end{equation}
Say $A$ is an $m \times n$ matrix. Here, we take $A_{ij}=0$ when $i \not \in \{1, \ldots m\}$ or $j \not \in \{1, \ldots, n\}$.
The value of $C_{ij}$ is a linear combination of the nearby pixel values, with coefficients given by $F$ (see Figure \ref{fig:convolution}).
In fact, $C_{ij}$ equals the Frobenius inner product of $F$ with the $3 \times 3$ submatrix of $A$ centered at $ij$.

\begin{figure}
\centering
\begin{tikzpicture}
\node[draw, minimum width=12cm, minimum height=
    7cm, ultra thick](outer_rec)[]{};
\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, fill=black!10!](inner_rec)[]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square1)at(-4.75,2.25){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick,fill=black!30!](square2)at(-1.75,-.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square3)at(4.75,.75){\textbullet};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick, fill=black!30!](square4)at(3.25,-1.25){\textbullet};
\draw[step=.5, ultra thin, color=black!50!](-6,-3.5)grid(6,3.5);

%redraw borders
\node[draw, minimum width=10cm, minimum height=
    5cm, ultra thick, ][]{};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-4.75,2.25){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(-1.75,-.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(4.75,.75){};
\node[draw, minimum width=2.5cm, minimum height=2.5cm,
    ultra thick]at(3.25,-1.25){};
\end{tikzpicture}
\caption{This diagram illustrates how to convolve an image with a filter.
The light grey rectangle represents the original image $A$, and the dark grey squares are the filter $F$.
The larger rectangle is the image padded with zeros; i.e., all pixel values in the outer white band are 0.
To compute the entry of the convolution matrix $C$ located at a black dot, take the inner product of $F$ with the submatrix of the padded image centered at the dot.}
\label{fig:convolution}
\end{figure}

\subsubsection*{Implementation in NumPy} % - - - - - - - - - - - - - - - - - -

Let us write a function that convolves an image with a filter.
You can test this function on the image \li{cameraman.jpg}, which appears in Figure \ref{fig:cameraman}.
The following code loads this image and plots it with matplotlib.

\begin{lstlisting}
>>> image = plt.imread('cameraman.jpg')
>>> plt.imshow(image, cmap = 'gray')
>>> plt.show()
\end{lstlisting}

Here is the function definition and some setup.

\begin{lstlisting}
1. def Filter(image, F):
2.     m, n = image.shape
3.     h, k = F.shape
\end{lstlisting}

To convolve \li{image} with the filter \li{F}, we must first \emph{pad} the array \li{image} with zeros around the edges.
This is because in \eqref{equ:convolve}, entries $A_{ij}$ are set to zero when $i$ or $j$ is out of bounds.
We do this by creating a larger array of zeros, and then making the interior part of the array equal to the original image (see Figure \ref{fig:convolution}).

For example, if the filter is a $3 \times 3$ matrix, then the following code will pad the matrix with the appropriate number of zeros.

\begin{lstlisting}
 # Create a larger matrix of zeros
image_pad = np.zeros((m+2, n+2))
# Make the interior of image_pad equal to the original image
image_pad[1:1+m, 1:1+n] = image
\end{lstlisting}

We want to do this in general in our function.  Note that the number of zeros we need to pad our array depends on the size of the filter \li{F}.

\begin{lstlisting}
5.    image_pad = # Create an array of zeros of the appropriate size
6.   # Make the interior of image_pad equal to image
\end{lstlisting}

Finally, we iterate through the image to compute each entry of the convolution matrix.

\begin{lstlisting}
7.    C = np.zeros(image.shape)
8.    for i in range(m):
9.        for j in range(n):
10.            C[i,j] = # Compute C[i, j]
\end{lstlisting}

\subsubsection*{Gaussian Blur} % - - - - - - - - - - - - - - - - - - - - - - -

A \emph{Gaussian blur} is an image filter that operates on an image by convolving with the matrix
\[
G = \frac{1}{159}\begin{pmatrix}
2&4&5&4&2\\
4&9&12&9&4\\
5&12&15&12&5\\
4&9&12&9&4\\
2&4&5&4&2
\end{pmatrix}.
\]

Blurring an image can remove ``noise'', or random variation that is the visual analog of static in a radio signal (and equally undesirable).

\begin{problem}\label{prob:filter}
\leavevmode
Finish writing the function \li{Filter} by filling in lines 5, 6, and 10.  Hint: Note in \ref{equ:convolve}, $C_{ij}$ was calculated by summing from -1 to 1.  This is only the case if the filter \li{F} is $3 \times 3$. A slight modification is needed in the general case.  Test your function on the image \li{cameraman.jpg} using the Gaussian Blur. The result is in Figure \ref{fig:cameraman_blur}.
\end{problem}

% TODO: make pictures that aren't pdfs
\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/cameraman.jpg}
\caption{Unfiltered image.}
\label{fig:cameraman}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/cameramanBlur.pdf}
\caption{Image after Gaussian blur is applied.}
\label{fig:cameraman_blur}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/edges.pdf}
\caption{Image after the Sobel filter is applied.}
\label{fig:cameraman_edges}
\end{subfigure}
\caption{Here is an example of a Gaussian blur and the Sobel filter applied to an image.
This photo, known as ``cameraman,'' is a standard test image in image processing.
A database of such images can be downloaded from \url{http://www.imageprocessingplace.com/root_files_V3/image_databases.htm}.}
\label{fig:cameraman1}
\end{figure}

\subsection*{Edge Detection} % ------------------------------------------------

Automatic detection of edges in an image can be used to segment or sharpen the image.
We will find edges with the Sobel filter, which computes the gradient of the image at each pixel.
The magnitude of the gradient tells us the rate of change of the pixel values, and so large magnitudes should
correspond to edges within the image.
The Sobel filter is not a convolution, although it does use convolutions.

We can think of an image as a function from a $2 \times 2$ grid of points to $\mathbb{R}$.
The image maps a pixel location to an intensity.
It does not make sense to define the derivative of this function as a limit because the domain is discrete---a step size $h$ cannot take on arbitrarily small values.
Instead, we \emph{define} the derivative to be the centered difference quotient of the previous section.
That is, we define the derivative in the $x$-direction at the $ij^{th}$ pixel to be
\[
\frac{1}{2}A_{i+1, j} - \frac{1}{2}A_{i-1, j}.
\]

We can use a convolution to create a matrix $A_x$ whose $ij^{th}$ entry is the derivative of $A$ at the $ij^{th}$ entry, in the $x$-direction.
In fact, $A_x = A \ast S$, where
\[
S = \frac{1}{8}
\left[\begin{array}{ccc}
-1 & 0 & 1\\
-2 & 0 & 2\\
-1 & 0 & 1
\end{array}\right].
\]

Note that this convolution takes a weighted average of the $x$-derivatives at $(i, j)$, $(i, j+1)$, and $(i, j-1)$.
The derivative at $(i, j)$ is weighted by 2.
Using a weighted average instead of just the derivative at $(i, j)$ makes the derivative less affected by noise.

Now we can define the Sobel filter.
A Sobel filter applied to an image $A$ results in an array $B = (B_{ij})$ of 0's and 1's, where the 1's trace out the edges in the image.
By definition,
\[
B_{ij} = \left\{
     \begin{array}{ll}
       1 & \text{if}\; \;\|\nabla A(ij)\|_2 > M \\
       0 & \text{otherwise}.
     \end{array}
   \right.
\]
Here, $\nabla A(ij) = ((A \ast S)_{ij}, (A\ast S^T)_{ij})$ is the gradient of $A$ at the $ij^{th}$ pixel.
The constant $M$ should be ``sufficiently large'' enough to pick out those pixels with the largest gradient (i.e., those pixels that are part of an edge).
A good choice for $M$ is 4 times the average value of $\|\nabla A(ij)\|_2$ over the whole image $A$.

When the Sobel filter is applied to \li{cameraman.jpg}, we get the image in Figure \ref{fig:cameraman_edges}.
Here, the 1's in $B$ were mapped to ``white'' and the 0's were mapped to ``black.''

\begin{problem}
Write a function that accepts an image as input and applies the Sobel filter to the image.  Test your function on \li{cameraman.jpg}.  Hint: If you want to find the average of a matrix \li{A}, use the function \li{A.mean()}.
\end{problem}


\chapter{Proof of Lemma~\ref{lem:measuresets}}\label{app:measuresets}
\begin{proof}
Let
\begin{equation}
f_\nsamps (\param) = \sum_{\iparam=1}^{\nsamps} \frac{\eta_\nsamps (\VV^{(\iparam)}) }{\mu (\VV^{(\iparam)})} \Chi_{\VV^{(\iparam)}} (\param).
\end{equation}
Then, for any $A\in\BB_\pspace$, define
\begin{equation}
\eta (A) = \int_A f_\nsamps (\param) \, d\mu.
\end{equation}
We verify that $\eta$ is a probability measure on $(\pspace, \BB_\pspace)$ and that $\eta(A) = \eta_\nsamps(A) \; \forall \; A\in\BB_{\pspace, \nsamps}$ below:
\begin{itemize}
\item[(i)][Positive]
Let $A\in \BB_{\pspace}$.
\begin{equation*}
\begin{split}
\eta (A) &= \int_A f_\nsamps (\param) \, d\mu \\
&=  \int \Chi_A \sum_{\iparam=1}^{\nsamps} \frac{\eta_\nsamps (\VV^{(\iparam)}) }{\mu (\VV^{(\iparam)})} \Chi_{\VV^{(\iparam)}} (\param) \, d\mu \\
&= \sum_{\iparam=1}^{\nsamps} \left ( \frac{\eta_\nsamps (\VV^{(\iparam)}) }{\mu (\VV^{(\iparam)})} \int \Chi_{A\cap\VV^{(\iparam)}} (\param) \, d\mu \right ) \\
&= \sum_{\iparam=1}^{\nsamps} \left ( \frac{\eta_\nsamps (\VV^{(\iparam)}) }{\mu (\VV^{(\iparam)})} \mu\left (A\cap\VV^{(\iparam)}\right ) \right ) \geq 0
\end{split}
\end{equation*}

\item[(ii)][Definite]
\begin{equation*}
\eta (\nullset) = \int_\nullset f_\nsamps (\param) \, d\mu =\int \Chi_\nullset f_\nsamps (\param) \, d\mu = \mu(\nullset) = 0
\end{equation*}

\item[(iii)][Countably Additive]
Let $\set{A_k}_{k=1}^{\infty} \subset \BB_{\pspace}$.
\begin{equation*}
\begin{split}
\eta (\cup_k A_k) &= \int_{\cup_k A_k} f_\nsamps (\param) \, d\mu
= \int \Chi_{\cup_k A_k} f_\nsamps (\param) \, d\mu \\
&= \int \left( \sum_k \Chi_{A_k} \right ) f_\nsamps (\param) \, d\mu
=   \sum_k \int \Chi_{A_k} f_\nsamps (\param) \, d\mu \\
&=   \sum_k \int_{A_k} f_\nsamps (\param) \, d\mu = \sum_k \eta(A_k)
\end{split}
\end{equation*}

\noindent Finally, let $A\in\BB_{\pspace, \nsamps}\subset \BB_\pspace$.
Then there exists some $\iparam^* \in \set{1,2, \cdots, \nsamps}$ such that $\VV^{(\iparam^*)} = A$.
We have that
\begin{equation*}
\begin{split}
\eta (A) &= \int_A f_\nsamps (\param) \, d\mu
=  \int \sum_{\iparam=1}^{\nsamps} \left ( \frac{\eta_\nsamps (\VV^{(\iparam)}) }{\mu (\VV^{(\iparam)})} \Chi_{\VV^{(\iparam)}}\right ) \Chi_{\VV^{(\iparam^*)}} \, d\mu \\
&= \int \frac{\eta_\nsamps (\VV^{(\iparam^*)}) }{\mu (\VV^{(\iparam^*)})} \Chi_{\VV^{(\iparam^*)}} \, d\mu
= \frac{\eta_\nsamps (\VV^{(\iparam^*)}) }{\mu (\VV^{(\iparam^*)})} \mu (\VV^{(\iparam^*)}) \\
&= \eta_\nsamps (\VV^{(\iparam^*)}) = \eta_\nsamps (A).
\end{split}
\end{equation*}

\end{itemize}
\end{proof}


\lab{Conditioning and Stability}{Conditioning and Stability}
\objective{Explore the condition of problems and the stability of algorithms.}
\label{lab:conditioning_stability}

%\begin{equation*}
%\mathlarger{ \mathlarger{ \mathlarger{f:X \rightarrow Y}}}
%\end{equation*}
%\begin{center} vs. \end{center}
%\begin{equation*}
%\mathlarger{ \mathlarger{ \mathlarger{\hat{f}:\hat{X} \rightarrow \hat{Y}}}}
%\end{equation*}

%\begin{eqnarray}
%\mathlarger{\mathlarger{\mathlarger{f:X \rightarrow Y}}}\\ \mathlarger{\mathlarger{\mathlarger{ f:\hat{X} \rightarrow \hat{Y} }}}\\
%\mathlarger{\mathlarger{\mathlarger{ \hat{f}:X \rightarrow Y }}}%\\
%%\mathlarger{\mathlarger{\mathlarger{ \hat{f}:\hat{X} \rightarrow \hat{Y} }}}
% \end{eqnarray}
%

The \emph{condition number} of a function measures how sensitive that function is to changes in the input.
On the other hand, the \emph{stability} of an algorithm measures how well that algorithm computes the value of a function from exact input.

\section*{Condition Number of a Function}

The (absolute) condition number of a function $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ is
 \begin{equation}\label{equ:abs_cond}
J(\x) = \lim_{\delta \rightarrow 0} \sup_{\norm{\delta \x} \leq \delta} { \frac{\norm{\delta f}}{\norm{\delta \x}} }
\end{equation}
where $\delta f = f(\x+\delta\x)-f(\x)$.
In other words, the condition number of $f$ is (the limit of) the change in output over the change of input.

Similarly, the \emph{relative condition number} of $f$ is the limit of the relative change in output over the relative change in input, i.e.,
\begin{equation}\label{equ:rel_cond}
\kappa(\x) = \lim_{\delta \rightarrow 0} \sup_{\norm{\delta \x} \leq \delta} \left({ \frac{\norm{\delta f}}{\norm{f(\x)}} } \middle/ { \frac{\norm{\delta \x}}{\norm{\x}} }\right).
\end{equation}
In fact,
\[
\kappa(\x) = \frac{\norm{\x}}{\norm{f(\x)}} J(\x)
\]
A function is \emph{ill-conditioned} if its condition number is large.

Small changes to the input of an ill-conditioned function produce large changes in output.
In applications, it is important to know if a function is ill-conditioned because there is usually some error in the parameters passed to the function.

\subsection*{Example: the Wilkinson Polynomial}

Let $f:\mathbb{C}^{n+1} \rightarrow \mathbb{C}^n$ be the function that sends $(a_1, \ldots, a_{m+1})$ to the roots of $a_1x^n+a_2x^{n-1}+\ldots+a_nx+a_{n+1}$.
In other words, this function describes the problem of finding the roots of a polynomial.
Unfortunately, root finding is extremely ill-conditioned.

A classic example is the Wilkinson polynomial
\[
w(x) = \prod_{r=1}^{20}(x-r) = x^{20}-210x^{19}+20615x^{18}-\ldots.
\]
We will use NumPy to explore the condition number of $f$ at the coefficients of $w(x)$.
These coefficients are contained in the NumPy array \li{w_coeffs} below.
We also create an array \li{w_roots} containing the roots of $w(x)$.\footnote{
It is possible to create this array in NumPy by expanding the product $\prod_{r=1}^{20}(x-r)$, for example using \li{np.poly()}.
However, this approach is \emph{unstable} and will give you the wrong coefficients!}
\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la
>>> w_coeffs = np.array([1, -210, 20615, -1256850, 53327946, -1672280820,
                    40171771630, -756111184500, 11310276995381,
                    -135585182899530, 1307535010540395,
                    -10142299865511450, 63030812099294896,
                    -311333643161390640, 1206647803780373360,
                    -3599979517947607200, 8037811822645051776,
                    -12870931245150988800, 13803759753640704000,
                    -8752948036761600000, 2432902008176640000])
>>> w_roots = np.arange(1, 21)
\end{lstlisting}

Next we perturb the polynomial by changing the $x^{19}$-coefficient from $-210$ to $-210.0000001$.
\begin{lstlisting}
>>> perturb = np.zeros(21)
>>> perturb[1]=1e-7
>>> perturbed_coeffs = w_coeffs - perturb
\end{lstlisting}

Now we find the roots of the perturbed polynomial.
The function \li{np.poly1d()} creates a polynomial object using the array passed to it as coefficients, and \li{np.roots()} finds the roots of a polynomial.
\begin{lstlisting}
>>> perturbed_roots = np.roots(np.poly1d(perturbed_coeffs))
\end{lstlisting}
The new roots are plotted with the original roots in Figure \ref{fig:wilkinsonpolynomial}.

Finally we compute an approximation to the condition number.
We sort the roots before we compare them to ensure that they are in the same order.
\begin{lstlisting}
>>> w_roots = np.sort(w_roots)
>>> perturbed_roots = np.sort(perturbed_roots)
>>> la.norm(perturbed_roots-w_roots)/la.norm(perturb)
68214100.15878984
\end{lstlisting}
Thus, the condition number of this problem is something like $10^7$.

When we try to estimate the relative condition number, NumPy will not compute the 2-norm of \li{w_coeffs} because it is so large.
NumPy will not compute the square root of a very large number.
One easy solution is to use a different norm that does not require the square root.
\begin{lstlisting}
>>> # Estimate the absolute condition number in the infinity norm
>>> k = la.norm(perturbed_roots-w_roots, np.inf)/la.norm(perturb, np.inf)
>>> k
28260604.34345689
>>> # Estimate the relative condition number in the infinity norm
>>> k*la.norm(w_coeffs, np.inf)/la.norm(w_roots, np.inf)
1.9505129642488696e+25
\end{lstlisting}
As you can see, the order of magnitude of the absolute condition number is the same as when we computed it with the 2-norm.
The relative condition number for this problem is approximately $10^{24}$.

There are some caveats to this example.
First, when we compute the quotients in \eqref{equ:abs_cond} and \eqref{equ:rel_cond} for a fixed $\delta \x$, we are only \emph{approximating} the condition number.
The actual condition number is a limit of such quotients.
We hope that when $||\delta \x||$ is small, a random quotient is at least the same order of magnitude as the limit, but we have no way to be sure.

Second, this example assumes that NumPy's root-finding algorithm is \emph{stable}, so that the difference between the roots of \li{w_coeffs} and \li{perturbed_coeffs} is due to the difference in coefficients, and not the difference in roots.
We will return to this issue in the next section.

\begin{figure}
\centering
\includegraphics[height=2.25in]{figures/wilk2.png}
\caption{In these images, blue is associated with $w(x)$ and red is associated with $w(x)$ perturbed by 1e-7 in the $x^{19}$-coefficient. On the left is a plot of the two polynomials. On the right is a graphical representation of the roots of the polynomials. The blue dots are the roots of $w(x)$. The red x's are the roots of the perturbed polynomial. }
\label{fig:wilkinsonpolynomial}
\end{figure}

\begin{problem}\label{prob:wilk}
Write a Python function that investigates the condition number of the Wilkinson polynomial by doing the following.
\begin{enumerate}
\item Perform this experiment:

\begin{quote}
Randomly perturb $w(x)$ by replacing each coefficient $a_i$ with $a_i*r_i$, where $r_i$ is drawn from a normal distribution centered at 1 with standard deviation $1e-10$.
\end{quote}

Plot the results of 100 such experiments in a single graphic, along with the roots of the unperturbed polynomial $w(x)$. The plot should look something like Figure \ref{fig:wilkinsonpolynomial_many}. This exercise reproduces Figure 12.1 on p. 93 of \emph{Numerical Linear Algebra} by Lloyd N. Trefethen and David Bau III.
\item Using the final experiment only, estimate the relative and absolute condition number (in any norm you prefer). Print these numbers to the screen.
\end{enumerate}

\begin{figure}[H]
\includegraphics[height=3in]{figures/wilkinsonpolynomial_many.pdf}
\caption{Sample result of Problem \ref{prob:wilk}.
The blue dots are the roots of $w(x)$ and the black dots are roots of random perturbations.
This figure replicates Figure 12.1 on p. 93 of \emph{Numerical Linear Algebra} by Lloyd N. Trefethen and David Bau III.}
\label{fig:wilkinsonpolynomial_many}
\end{figure}

\end{problem}

%Notes on Problem 1:
% - Make something to copy-paste with the polynomial coefficients
% - You MUST copy paste the exact coefficients!!! Multiplying out the polynomial is not stable!!!
% - Also "compute" the roots of the wilkinson poly. They are close but not quite right. The difference is not visible on the graph. See that is probably useful.

% In introductory discussion, compute the condition number of the problem. Explain why we use the infinity norm.

\subsection*{Example: Calculating Eigenvalues}

Let $f:\mathbb{C}^{n^2} \rightarrow \mathbb{C}^n$ be the function that sends an $n \times n$ matrix to its $n$ eigenvalues.
This problem is well-conditioned for symmetric matrices, but can be extremely ill-conditioned for non-symmetric matrices.

Let us use NumPy to calculate the condition number of the eigenvalue problem at the identity matrix.
First we check that the eigenvalue solver is stable here.
\begin{lstlisting}
>>> M = np.array([[1,0],[0,1]])
>>> eigs = la.eig(M)[0]
>>> eigs
array([ 1.+0.j,  1.+0.j])
\end{lstlisting}
Now we perturb $M$ by adding a matrix drawn from a random normal distribution over the complex numbers.
We calculate the eigenvalues of the perturbed matrix.
\begin{lstlisting}
>>> perturb = np.random.normal(0, 1e-10, M.shape) + np.random.normal(0,1e-10, M.shape)*1j
>>> eigsp = la.eig(M+perturb)[0]
\end{lstlisting}
Finally we use this data to approximate the condition number.
\begin{lstlisting}
>>> k = la.norm(eigs-eigsp)/la.norm(perturb)	# Absolute condition number
0.62957336119253127
>>> k*la.norm(M)/la.norm(eigs)	# Relative condition number
0.62957336119253127
\end{lstlisting}
The absolute and relative condition number are the same because \li{la.norm(M)} and \li{la.norm(eigs)} are both $\sqrt{2}$.

\begin{problem}[Optional]\label{prob:eigenvalue}
Let us explore the condition number of the eigenvalue problem.
\begin{enumerate}
\item \begin{enumerate}
\item Write the following function.
\begin{lstlisting}
def eig_condit(M):
    '''
    Approximate the condition number of the eigenvalue problem at M.

    INPUT:
    M  - A 2-D NumPy array, representing a matrix.

    RETURN:
    A tuple containing approximations to the absolute and relative condition numbers of the eigenvalue problem at M.
    '''
\end{lstlisting}
\item Find an example of a $2\times2$ matrix with a very large condition number.
(Hint: Look at matrices whose off-diagonal entries are very different in magnitude.)
\item What is the order of magnitude of the condition number of a symmetric $2\times 2$ matrix?
\end{enumerate}
\item Write the following function.
\begin{lstlisting}
def plot_eig_condit(x0=-100, x1=100, y0=-100, y1=100, res=10):
    '''
    Plot the condition number of the eigenvalue problem on [x0, x1]x[y0,y1].

    Specifically, use plt.pcolormesh to plot the relative condition number of the eigenvalue problem at [[1,x],[y,1]] on this domain.
    The variable `res' should be the number of sample points taken along each axis, for a total of `res'**2 points in the plot.
    '''
\end{lstlisting}
\item Call your function for \li{res}=10, 50, 100, 200 and 400 (output for $res=200$ is pictured below). Recall that matplotlib scales the colorbar of the output to fit the largest and smallest output values.
What can you conclude about the condition number of the eigenvalue problem at a ``random'' $2\times2$ matrix?
\end{enumerate}
\begin{figure}[H]
\includegraphics[height=3in]{figures/eigenvalue_conditioning.png}
\caption{Output of \li{plot_eig_condit(res=200)}.}
\end{figure}
\end{problem}

%\begin{example}[Condition of a System of Equations]
%Consider the system of equations
%\[ Ax = b \]
%for an $n \times n$ matrix $A$ and $n \times 1$ vectors $x$ and $b$. If we hold $A$ fixed, consider the problem of computing $b$ with respect to small changes in $x$. Alternatively, hold $A$ fixed and consider the problem of computing $x = A^{-1} b$ with respect to small changes in $b$. Finally, we may hold $b$ fixed and consider the problem of computing $x = A^{-1}b$ with respect to small changes in $A$.
%
%It turns out that each of these problems has the \emph{same} relative condition number. This number is
%\[
%\mathcal{K} (A) = \norm{A}_2 \norm{A^{-1}}_2
%\]
%
%This is called the \emph{condition number of $A$}.
%The condition number of a matrix can be computed using \li{numpy.linalg.cond()}.
%
%Notice that the condition number of a matrix cannot be less that $1$.
%It is almost always best to work with orthonormal matrices (or operations that can be mathematically represented by orthonormal matrices).
%Since orthonormal matrices have a norm of $1$, as do their inverses, these transformations have the best possible condition number.
%The low condition number of orthonormal matrices is one of the primary reasons that Householder reflections and Givens rotations are used in so many algorithms.
%\end{example}

\section*{Stability of an Algorithm}

The stability of an algorithm is measured by the error in its output.
Suppose we have some algorithm to compute $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$.
Let $\tilde f(\x)$ represent the value computed by the algorithm at $\x$.
Then the \emph{forward error} of $f$ at $\x$ is $||f(\x)-\tilde f(\x)||$, and the \emph{relative forward error} of $f$ at $\x$ is
\[
\frac{||f(\x)-\tilde f(\x)||}{||f(\x)||}.
\]
An algorithm is \emph{stable} if this relative forward error is small.

As an example, let us examine the stability of NumPy's root finding algorithm that we used to investigate the Wilkinson polynomial.
We know the exact roots of $w(x)$, and we can also compute these roots using NumPy's \li{np.roots()} function.

\begin{lstlisting}
>>> roots = np.arange(1,21)
>>> w_coeffs = np.array([1, -210, 20615, -1256850, 53327946, -1672280820,
                    40171771630, -756111184500, 11310276995381,
                    -135585182899530, 1307535010540395,
                    -10142299865511450, 63030812099294896,
                    -311333643161390640, 1206647803780373360,
                    -3599979517947607200, 8037811822645051776,
                    -12870931245150988800, 13803759753640704000,
                    -8752948036761600000, 2432902008176640000])
>>> computed_roots = np.roots(np.poly1d(w_coeffs))
\end{lstlisting}

We sort the roots to ensure they are in the same order, then compute the absolute and relative forward error.

\begin{lstlisting}
>>> roots = np.sort(roots)
>>> computed_roots = np.sort(computed_roots)
>>> la.norm(roots-computed_roots)	# Forward error
0.020612653126379665
>>> la.norm(roots-computed_roots)/la.norm(roots)	# Relative forward error
0.00038476268486104599
\end{lstlisting}

This analysis gives us hope that questions of stability did not interfere too much with our experiments in Problem \ref{prob:wilk}.

\subsection*{Catastrophic Cancellation}
\emph{Catastrophic Cancellation} is a term for when a computer takes the difference of two very similar numbers, and the result is stored with a small number of significant digits.
Because of the way computers store and perform arithmetic on numbers, future computations can amplify a catastrophic cancellation into a huge error.

You are at risk for catastrophic cancellation whenever you subtract floats or large integers that are very close to each other.
You can avoid the problem either by rewriting your program to not use subtraction, or by increasing the number of significant digits that your computer tracks.

Here is an example of catastrophic cancellation.
Suppose we wish to compute $\sqrt{a}-\sqrt{b}$. We can either do this subtraction directly or perform the equivalent division
\[
\sqrt{a}-\sqrt{b} = (\sqrt{a}-\sqrt{b})\frac{\sqrt{a}+\sqrt{b}}{\sqrt{a}+\sqrt{b}} = \frac{a-b}{\sqrt{a}+\sqrt{b}}.
\]

Let us perform this computation both ways in NumPy with $a=10^{20}+1$ and $b=10^{20}$.
\begin{lstlisting}
>>> np.sqrt(1e20+1)-np.sqrt(1e20)
0.0
>>> 1/(np.sqrt(1e20+1)+np.sqrt(1e20))
5.0000000000000002e-11
\end{lstlisting}
Since $a \neq b$, clearly $\sqrt{a}-\sqrt{b}$ should be nonzero.

\begin{problem}
Let $I(n) = \int_0^1 x^n e^{x - 1} dx$.
\begin{enumerate}
\item Prove that $0 \leq I(n) \leq 1$ for all $n$.
\item It can be shown that for $n>1$,
\[
I(n) = \left(-1\right)^{n} !n + \left(-1\right)^{n + 1} \frac{n!}{e}
\]
where $!n$ is the \emph{subfactorial} of $n$.
Use this formula to write the following function.
\begin{lstlisting}
def integral(n):
    '''Return I(n).'''
\end{lstlisting}
Hint: The subfactorial function can be imported from SymPy with the line \li{from sympy import subfactorial}.
\item The actual values of $I(n)$ for many values of $n$ are listed in the table below.
Use your function \li{integral()} to compute $I(n)$ for these same values of $n$, and create a table comparing the data.
How can you explain what is happening?
\end{enumerate}

\begin{center}
\begin{tabular}{|l|l|}
\hline
$n$  & Actual value of $I(n)$ \\
\hline
$1$  & $0.367879441171$ \\
$5$  & $0.145532940573$ \\
$10$ & $0.0838770701034$ \\
$15$ & $0.0590175408793$ \\
$20$ & $0.0455448840758$ \\
$25$ & $0.0370862144237$ \\
$30$ & $0.0312796739322$ \\
$35$ & $0.0270462894091$ \\
$40$& $0.023822728669$ \\
$45$& $0.0212860390856$ \\
$50$ & $0.0192377544343$ \\
\hline
\end{tabular}
\end{center}

\end{problem}

%\begin{table}
%\centering
%\begin{tabular}{|l|l|l|}
%\hline
%Integrand & Computed Value & Actual Value \\
%\hline
%$x^{1}e^{x}$: & $0.367879441171$ & $0.367879441171$ \\
%$x^{5}e^{x}$: & $0.145532940573$ & $0.145532940573$ \\
%$x^{10}e^{x}$: & $0.0838770701084$ & $0.0838770701034$ \\
%$x^{15}e^{x}$: & $0.0590209960938$ & $0.0590175408793$ \\
%$x^{20}e^{x}$: & $0.0$ & $0.0455448840758$ \\
%$x^{25}e^{x}$: & $1073741824.0$ & $0.0370862144237$ \\
%$x^{30}e^{x}$: & $-1.80143985095 \cdot 10^{16}$ & $0.0312796739322$ \\
%$x^{35}e^{x}$: & $6.04462909807 \cdot 10^{23}$ & $0.0270462894091$ \\
%$x^{40}e^{x}$: & $0.0$ & $0.023822728669$ \\
%$x^{45}e^{x}$: & $0.0$ & $0.0212860390856$ \\
%$x^{50}e^{x}$: & $1.46150163733 \cdot 10^{48}$ & $0.0192377544343$ \\
%\hline
%\end{tabular}
%\caption{Inaccuracy of values computed using an unstable algorithm.}
%\label{table:unstable_computation}
%\end{table}

% For this section to be meaningful, it needs more explanation of when each algorithm is stable.
\begin{comment}
The algorithms that we have studied to solve linear systems have different levels of stability.
LU decomposition (with pivoting) is usually good enough, but there are some pathological examples of matrices that cause it to break down.
QR decomposition (with pivoting) is generally considered to be a better option than the LU decomposition.
Solving a linear system using the SVD is even more stable than the QR decomposition.
(Pivoting is a modification that is commonly made to the LU decomposition and QR decomposition algorithms we have discussed in earlier labs to make them more stable.)
Unfortunately, in this case, the algorithms that are more stable are also slower.
The LU decomposition is used by \li{scipy.linalg.solve()}.
The SVD is used by \li{scipy.linalg.lstsq()}.
Here is some code that uses the QR decomposition of a matrix $A$ to solve the linear system $A x = b$ for $x$.
It uses a lower-level function included in SciPy to perform the back substitution required to solve this system.
\begin{lstlisting}
from scipy import linalg as la
from scipy.linalg.flapack import dtrtrs
def qr_solve(A, b):
    Q, R = la.qr(A)
    return dtrtrs(R.T, Q.T.dot(b), lower=1, trans=1)[0]
\end{lstlisting}
A solution using a pivoted QR decomposition would be better, but this will be good enough for demonstration purposes.

The following are routines that generate matrices designed to show the relative benefits of each of these algorithms.
\begin{lstlisting}
from numpy.random import rand

def bad_arr_1(n):
    """ Construct a specific pathological example
    that breaks LU decomposition. These examples
    are very rare, but they do exist.
    Strictly speaking, the condition number
    for this matrix isn't terribly bad. """
    A = - np.ones((n, n))
    A[:,:-1] = np.tril(A[:,:-1])
    np.fill_diagonal(A, 1)
    A[:,-1] = 1
    return A

def bad_arr_2(n, peturbation = 1E-8):
    """ Construct another matrix that is nearly singular
    by computing A.dot(A.T) for a matrix A that is
    not square and then adding some small changes
    so it is not exactly singular. """
    A = rand(n, n // 2)
    return A.dot(A.T) + peturbation * rand(n, n)
\end{lstlisting}
\end{example}

\begin{problem}
For each of the array creation routines above, plot the error $\norm{\text{solve}\left(A, A b\right) - b}$ of each of the methods mentioned above for solving a linear system.
Use a log-scaled $y$-axis.
What do you observe?
\end{problem}
\end{comment}
%\subsection*{Stable vs. Backward Stable Algorithms}

%\subsection*{Table}
%
%\begin{table}[h]
%\begin{tabular}{|l|l|l|}
%\hline \textbf{Well-Conditioned} & \textbf{Ill-Conditioned} & \textbf{Systems of Equations} \\
%{\parbox{0.3\textwidth}{\raggedleft
%            \begin{itemize}[leftmargin=*]
%                \item finding eigenvalues of a symmetric (or normal) matrix
%                \item calculating $e^x$ for relatively small values of $x$
%                \item calculating $\ln(x)$ for $x$ not close to $1$
%            \end{itemize} }}           &
%
%{\parbox{0.3\textwidth}{\raggedleft
%            \begin{itemize}[leftmargin=*]
%                \item calculating $x_1 - x_2$ when $x_1 \approx x_2$
%                \item computing roots of a polynomial, given the coefficients
%                \item computing eigenvalues of a non-symmetric matrix
%            \end{itemize} }}              &
%
%{\parbox{0.3\textwidth}{
%            \begin{itemize}[leftmargin=*]
%                 \item relative condition number is \[ \mathcal{K} = ||A|| ||A^{-1}|| \]
%            \end{itemize} }}   \\ \hline
%\end{tabular}
%\end{table}
%
%
%\begin{table}[h]
%\begin{tabular}{|l|l|}
%\hline \textbf{Stable} & \textbf{Unstable} \\
%{\parbox{0.45\textwidth}{\raggedleft
%            \begin{itemize}[leftmargin=*]
%                \item finding eigenvalues of a symmetric (or normal) matrix
%                \item calculating $e^x$ for relatively small values of $x$
%                \item calculating $\ln(x)$ for $x$ not close to $1$
%            \end{itemize} }}           &
%
%{\parbox{0.45\textwidth}{\raggedleft
%            \begin{itemize}[leftmargin=*]
%                \item calculating $x_1 - x_2$ when $x_1 \approx x_2$
%                \item computing roots of a polynomial, given the coefficients
%                \item computing eigenvalues of a non-symmetric matrix
%            \end{itemize} }}    \\ \hline
%\end{tabular}
%\end{table}
%


% ------------------------------------------------------------------
\chapter{Implementation details}\label{s:impl}
% ------------------------------------------------------------------

This chapter contains calculations and details.

% ------------------------------------------------------------------
\section{Convolution}\label{s:impl-convolution}
% ------------------------------------------------------------------

It is often convenient to express the convolution operation in matrix form. To this end, let $\phi(\bx)$ the \verb!im2row! operator, extracting all $W' \times H'$ patches from the map $\bx$ and storing them as rows of a $(H''W'') \times (H'W'D)$ matrix. Formally, this operator is given by:
\[
   [\phi(\bx)]_{pq} \underset{(i,j,d)=t(p,q)}{=} x_{ijd}
\]
where the index mapping $(i,j,d) = t(p,q)$ is
\[
 i = i''+i'-1, \quad
 j = j''+j'-1, \quad
 p = i'' + H'' (j''-1), \quad
 q = i' + H'(j'-1) + H'W' (d-1).
\]
It is also useful to define the ``transposed'' operator \verb!row2im!:
\[
   [\phi^*(M)]_{ijd}
   =
   \sum_{(p,q) \in t^{-1}(i,j,d)}
   M_{pq}.
\]
Note that $\phi$ and $\phi^*$ are linear operators. Both can be expressed by a matrix $H\in\real^{(H''W''H'W'D) \times(HWD)}$ such that
\[
  \vv(\phi(\bx)) = H \vv(\bx), \qquad
  \vv(\phi^*(M)) = H^\top \vv(M).
\]
Hence we obtain the following expression for the vectorized output (see~\cite{kinghorn96integrals}):
\[
 \vv\by =
 \vv\left(\phi(\bx) F\right)
 =
 \begin{cases}
 (I \otimes \phi(\bx)) \vv F, & \text{or, equivalently,} \\
 (F^\top \otimes I) \vv \phi(\bx),
 \end{cases}
\]
where $F\in\mathbb{R}^{(H'W'D)\times K}$ is the matrix obtained by reshaping the array $\bff$ and $I$ is an identity matrix of suitable dimensions. This allows obtaining the following formulas for the derivatives:
\[
\frac{dz}{d(\vv F)^\top}
=
\frac{dz}{d(\vv\by)^\top}
(I \otimes \phi(\bx))
= \vv\left[
\phi(\bx)^\top
\frac{dz}{dY}
\right]^\top
\]
where $Y\in\real^{(H''W'')\times K}$ is the matrix obtained by reshaping the array $\by$. Likewise:
\[
\frac{dz}{d(\vv \bx)^\top}
=
\frac{dz}{d(\vv\by)^\top}
(F^\top \otimes I)
\frac{d\vv \phi(\bx)}{d(\vv \bx)^\top}
=
\vv\left[
\frac{dz}{dY}
F^\top
\right]^\top
H
\]
In summary, after reshaping these terms we obtain the formulas:
\[
\boxed{
\vv\by =
 \vv\left(\phi(\bx) F\right),
\qquad
\frac{dz}{dF}
=
\phi(\bx)^\top\frac{d z}{d Y},
\qquad
\frac{d z}{d X}
=
\phi^*\left(
\frac{d z}{d Y}F^\top
\right)
}
\]
where $X\in\real^{(H'W')\times D}$ is the matrix obtained by reshaping $\bx$. Notably, these expressions are used to implement the convolutional operator; while this may seem inefficient, it is instead a fast approach when the number of filters is large and it allows leveraging fast BLAS and GPU BLAS implementations.

% ------------------------------------------------------------------
\section{Convolution transpose}\label{s:impl-convolution-transpose}
% ------------------------------------------------------------------

In order to understand the definition of convolution transpose, let $\by$ obtained from $\bx$ by the convolution operator as defined in Section~\ref{s:convolution} (including padding and downsampling).  Since this is a linear operation, it can be rewritten as $\vv \by = M \vv\bx$ for a suitable matrix $M$; convolution transpose computes instead $\vv \bx = M^\top \vv \by$.  While this is simple to describe in term of matrices, what happens in term of indexes is tricky. In order to derive a formula for the convolution transpose, start from standard convolution (for a 1D signal):
\[
   y_{i''} = \sum_{i'=1}^{H'} f_{i'} x_{S (i''-1) + i' - P_h^-},
   \quad
    1 \leq i'' \leq 1 + \left\lfloor \frac{H - H' + P_h^- + P_h^+}{S} \right\rfloor,
\]
where $S$ is the downsampling factor, $P_h^-$ and $P_h^+$ the padding, $H$ the length of the input signal, $\bx$ and $H'$ the length of the filter $\bff$. Due to padding, the index of the input data $\bx$ may exceed the range $[1,H]$; we implicitly assume that the signal is zero padded outside this range.

In order to derive an expression of the convolution transpose,  we make use of the identity $\vv \by^\top (M \vv \bx) = (\vv \by^\top M) \vv\bx = \vv\bx^\top (M^\top \vv\by)$. Expanding this in formulas:
\begin{align*}
\sum_{i''=1}^b y_{i''}
\sum_{i'=1}^{W'} f_{i'} x_{S (i''-1) + i'  -P_h^-}
&=
\sum_{i''=-\infty}^{+\infty}
\sum_{i'=-\infty}^{+\infty}
y_{i''}\ f_{i'}\ x_{S (i''-1) + i'  -P_h^-}
\\
&=
\sum_{i''=-\infty}^{+\infty}
\sum_{k=-\infty}^{+\infty}
y_{i''}\ f_{k-S(i'' -1) + P_h^-}\ x_{k}
\\
&=
\sum_{i''=-\infty}^{+\infty}
\sum_{k=-\infty}^{+\infty}
y_{i''}%
\ %
f_{%
(k-1+ P_h^-) \bmod S +
S \left(1 -i''  + \left\lfloor \frac{k-1+ P_h^-}{S} \right\rfloor\right)+1
}\ x_{k}
\\
&=
\sum_{k=-\infty}^{+\infty}
x_{k}
\sum_{q=-\infty}^{+\infty}
y_{\left\lfloor \frac{k-1+ P_h^-}{S} \right\rfloor + 2 - q}
\ %
f_{(k-1+ P_h^-)\bmod S +S(q - 1)+1}.
\end{align*}
Summation ranges have been extended to infinity by assuming that all signals are zero padded as needed. In order to recover such ranges, note that $k \in [1,H]$ (since this is the range of elements of $\bx$ involved in the original convolution). Furthermore, $q\geq 1$ is the minimum value of $q$ for which the filter $\bff$ is non zero; likewise, $q\leq \lfloor (H'-1)/2\rfloor +1$ is a fairly tight upper bound on the maximum value (although, depending on $k$, there could be an element less). Hence
\begin{equation}\label{e:convt-step}
 x_k =
 \sum_{q=1}^{1 + \lfloor \frac{H'-1}{S} \rfloor}
y_{\left\lfloor \frac{k-1+ P_h^-}{S} \right\rfloor + 2 - q}\ %
f_{(k-1+ P_h^-)\bmod S +S(q - 1)+1},
\qquad k=1,\dots, H.
\end{equation}
Here $H$, $H'$ and $H''$ are related by:
\[
H'' = 1+ \left\lfloor \frac{H - H' + P_h^- + P_h^+}{S} \right\rfloor.
\]
If $H''$ is now given as input, it is not possible to recover $H$ uniquely; instead, all the following values are possible
\[
   S_h (H''-1) +H' -  P_h^- - P_h^+ \leq H < S_h H'' +H' -  P_h^- - P_h^+.
\]
We use the tighter definition and set $H =  S_h (H''-1) +H' -  P_h^- - P_h^+$.
Note that the summation extrema in \eqref{e:convt-step} can be refined slightly to account for the finite size of $\by$ and $\bw$:
\begin{multline*}
\max\left\{
1,
\left\lfloor \frac{k-1 + P_h^-}{S} \right\rfloor + 2 - H''
\right\}
\leq q \\
\leq
1 +\min\left\{
\left\lfloor \frac{H'-1-(k-1+ P_h^-)\bmod S}{S} \right\rfloor,
\left\lfloor \frac{k-1 + P_h^-}{S} \right\rfloor
\right\}.
\end{multline*}

% ------------------------------------------------------------------
\section{Spatial pooling}\label{s:impl-pooling}
% ------------------------------------------------------------------

Since max pooling simply select for each output element an input element, the relation can be expressed in matrix form as
$
    \vv\by = S(\bx) \vv \bx
$
for a suitable selector matrix $S(\bx)\in\{0,1\}^{(H''W''D) \times (HWD)}$. The derivatives can the be written as:
$
\frac{d z}{d (\vv \bx)^\top}
=
\frac{d z}{d (\vv \by)^\top}
S(\bx),
$
for all but a null set of points, where the operator is not differentiable (this usually does not pose problems in optimization by stochastic gradient). For max-pooling, similar relations exists with two differences: $S$ does not depend on the input $\bx$ and it is not binary, in order to account for the normalization factors. In summary, we have the expressions:
\begin{equation}\label{e:max-mat}
\boxed{
\vv\by = S(\bx) \vv \bx,
\qquad
\frac{d z}{d \vv \bx}
=
S(\bx)^\top
\frac{d z}{d \vv \by}.
}
\end{equation}

% ------------------------------------------------------------------
\section{Activation functions}\label{s:impl-activation}
% ------------------------------------------------------------------

% ------------------------------------------------------------------
\subsection{ReLU}\label{s:impl-relu}
% ------------------------------------------------------------------

The ReLU operator can be expressed in matrix notation as
\[
\vv\by = \diag\bfs \vv \bx,
\qquad
\frac{d z}{d \vv \bx}
=
\diag\bfs
\frac{d z}{d \vv \by}
\]
where $\bfs = [\vv \bx > 0] \in\{0,1\}^{HWD}$ is an indicator vector.

% ------------------------------------------------------------------
\subsection{Sigmoid}\label{s:impl-sigmoid}
% ------------------------------------------------------------------

The derivative of the sigmoid function is given by
\begin{align*}
\frac{dz}{dx_{ijk}}
&=
\frac{dz}{d y_{ijd}}
\frac{d y_{ijd}}{d x_{ijd}}
=
\frac{dz}{d y_{ijd}}
\frac{-1}{(1+e^{-x_{ijd}})^2} ( - e^{-x_{ijd}})
\\
&=
\frac{dz}{d y_{ijd}}
y_{ijd} (1 - y_{ijd}).
\end{align*}
In matrix notation:
\[
\frac{dz}{d\bx} = \frac{dz}{d\by} \odot
\by \odot
(\mathbf{1}\mathbf{1}^\top - \by).
\]

% ------------------------------------------------------------------
\section{Normalization}\label{s:normalization}
% ------------------------------------------------------------------

% ------------------------------------------------------------------
\subsection{Cross-channel normalization}\label{s:impl-ccnormalization}
% ------------------------------------------------------------------

The derivative is easily computed as:
\[
\frac{dz}{d x_{ijd}}
=
\frac{dz}{d y_{ijd}}
L(i,j,d|\bx)^{-\beta}
-2\alpha\beta x_{ijd}
\sum_{k:d\in G(k)}
\frac{dz}{d y_{ijk}}
L(i,j,k|\bx)^{-\beta-1} x_{ijk}
\]
where
\[
 L(i,j,k|\bx) = \kappa + \alpha \sum_{t\in G(k)} x_{ijt}^2.
\]

% ------------------------------------------------------------------
\subsection{Batch normalization}\label{s:impl-bnorm}
% ------------------------------------------------------------------

The derivative of the input with respect to the network output is computed as follows:
\[
\frac{dz}{dx_{ijkt}} = \sum_{i''j''k''t''}
\frac{dz}{d y_{i''j''k''t''}}
\frac{d y_{i''j''k''t''}}{d x_{ijkt}}.
\]
Since feature channels are processed independently, all terms with $k''\not=k$ are null. Hence
\[
\frac{dz}{dx_{ijkt}} = \sum_{i''j''t''}
\frac{dz}{d y_{i''j''kt''}}
\frac{d y_{i''j''kt''}}{d x_{ijkt}},
\]
where
\[
\frac{d y_{i''j''kt''}}{d x_{ijkt}}
=
w_k
\left(\delta_{i=i'',j=j'',t=t''} - \frac{d \mu_k}{d x_{ijkt}}\right)
\frac{1}{\sqrt{\sigma^2_k + \epsilon}}
-
\frac{w_k}{2}
\left(x_{i''j''kt''} - \mu_k\right)
\left(\sigma_k^2 + \epsilon \right)^{-\frac{3}{2}}
\frac{d \sigma_k^2}{d x_{ijkt}},
\]
the derivatives with respect to the mean and variance are computed as follows:
\begin{align*}
\frac{d \mu_k}{d x_{ijkt}} &= \frac{1}{HWT},
\\
\frac{d \sigma_k^2}{d x_{i'j'kt'}}
&=
\frac{2}{HWT}
\sum_{ijt}
\left(x_{ijkt} - \mu_k \right)
\left(\delta_{i=i',j=j',t=t'} - \frac{1}{HWT} \right)
=
\frac{2}{HWT} \left(x_{i'j'kt'} - \mu_k \right),
\end{align*}
and $\delta_E$ is the indicator function of the event $E$. Hence
\begin{align*}
\frac{dz}{dx_{ijkt}}
&=
\frac{w_k}{\sqrt{\sigma^2_k + \epsilon}}
\left(
\frac{dz}{d y_{ijkt}}
-
\frac{1}{HWT}\sum_{i''j''kt''}
\frac{dz}{d y_{i''j''kt''}}
\right)
\\
&-
\frac{w_k}{2(\sigma^2_k + \epsilon)^{\frac{3}{2}}}
\sum_{i''j''kt''}
\frac{dz}{d y_{i''j''kt''}}
\left(x_{i''j''kt''} - \mu_k\right)
\frac{2}{HWT} \left(x_{ijkt} - \mu_k \right)
\end{align*}
i.e.
\begin{align*}
\frac{dz}{dx_{ijkt}}
&=
\frac{w_k}{\sqrt{\sigma^2_k + \epsilon}}
\left(
\frac{dz}{d y_{ijkt}}
-
\frac{1}{HWT}\sum_{i''j''kt''}
\frac{dz}{d y_{i''j''kt''}}
\right)
\\
&-
\frac{w_k}{\sigma^2_k + \epsilon}
\,
\frac{x_{ijkt} - \mu_k}{\sqrt{\sigma^2_k + \epsilon}}
\,
\frac{1}{HWT}
\sum_{i''j''kt''}
\frac{dz}{d y_{i''j''kt''}}
\left(x_{i''j''kt''} - \mu_k\right).
\end{align*}

% ------------------------------------------------------------------
\subsection{Spatial normalization}\label{s:impl-spnorm}
% ------------------------------------------------------------------

The neighborhood norm $n^2_{i''j''d}$ can be computed by applying average pooling to $x_{ijd}^2$ using \verb!vl_nnpool! with a $W'\times H'$ pooling region, top padding $\lfloor \frac{H'-1}{2}\rfloor$, bottom padding $H'-\lfloor \frac{H-1}{2}\rfloor-1$, and similarly for the horizontal padding.

The derivative of spatial normalization can be obtained as follows:
\begin{align*}
\frac{dz}{dx_{ijd}}
&= \sum_{i''j''d}
\frac{dz}{d y_{i''j''d}}
\frac{d y_{i''j''d}}{d x_{ijd}}
\\
&=
\sum_{i''j''d}
\frac{dz}{d y_{i''j''d}}
(1 + \alpha n_{i''j''d}^2)^{-\beta}
\frac{dx_{i''j''d}}{d x_{ijd}}
-\alpha\beta
\frac{dz}{d y_{i''j''d}}
(1 + \alpha n_{i''j''d}^2)^{-\beta-1}
x_{i''j''d}
\frac{dn_{i''j''d}^2}{d (x^2_{ijd})}
\frac{dx^2_{ijd}}{d x_{ijd}}
\\
&=
\frac{dz}{d y_{ijd}}
(1 + \alpha n_{ijd}^2)^{-\beta}
-2\alpha\beta x_{ijd}
\left[
\sum_{i''j''d}
\frac{dz}{d y_{i''j''d}}
(1 + \alpha n_{i''j''d}^2)^{-\beta-1}
x_{i''j''d}
\frac{dn_{i''j''d}^2}{d (x_{ijd}^2)}
\right]
\\
&=
\frac{dz}{d y_{ijd}}
(1 + \alpha n_{ijd}^2)^{-\beta}
-2\alpha\beta x_{ijd}
\left[
\sum_{i''j''d}
\eta_{i''j''d}
\frac{dn_{i''j''d}^2}{d (x_{ijd}^2)}
\right],
\quad
\eta_{i''j''d}=
\frac{dz}{d y_{i''j''d}}
(1 + \alpha n_{i''j''d}^2)^{-\beta-1}
x_{i''j''d}
\end{align*}
Note that the summation can be computed as the derivative of the
\verb!vl_nnpool! block.

% ------------------------------------------------------------------
\subsection{Softmax}\label{s:impl-softmax}
% ------------------------------------------------------------------

Care must be taken in evaluating the exponential in order to avoid underflow or overflow. The simplest way to do so is to divide from numerator and denominator by the maximum value:
\[
 y_{ijk} = \frac{e^{x_{ijk} - \max_d x_{ijd}}}{\sum_{t=1}^D e^{x_{ijt}- \max_d x_{ijd}}}.
\]
The derivative is given by:
\[
\frac{dz}{d x_{ijd}}
=
\sum_{k}
\frac{dz}{d y_{ijk}}
\left(
e^{x_{ijd}} L(\bx)^{-1} \delta_{\{k=d\}}
-
e^{x_{ijd}}
e^{x_{ijk}} L(\bx)^{-2}
\right),
\quad
L(\bx) = \sum_{t=1}^D e^{x_{ijt}}.
\]
Simplifying:
\[
\frac{dz}{d x_{ijd}}
=
y_{ijd}
\left(
\frac{dz}{d y_{ijd}}
-
\sum_{k=1}^K
\frac{dz}{d y_{ijk}} y_{ijk}.
\right).
\]
In matrix for:
\[
  \frac{dz}{dX} = Y \odot \left(\frac{dz}{dY}
  - \left(\frac{dz}{dY} \odot Y\right) \bone\bone^\top\right)
\]
where $X,Y\in\real^{HW\times D}$ are the matrices obtained by reshaping the arrays
$\bx$ and $\by$. Note that the numerical implementation of this expression is straightforward once the output $Y$ has been computed with the caveats above.

% ------------------------------------------------------------------
\section{Losses and comparisons}\label{s:impl-losses}
% ------------------------------------------------------------------

% ------------------------------------------------------------------
\subsection{Log-loss}\label{s:impl-loss}
% ------------------------------------------------------------------

The derivative is
\[
\frac{dz}{dx_{ijd}} = - \frac{dz}{dy} \frac{1}{x_{ijc}} \delta_{\{d = c\}}.
\]

% ------------------------------------------------------------------
\subsection{Softmax log-loss}\label{s:impl-sfloss}
% ------------------------------------------------------------------

The derivative is given by
\[
\frac{dz}{dx_{ijd}}
= - \frac{dz}{dy} \left(\delta_{d=c} - y_{ijc}\right)
\]
where $y_{ijc}$ is the output of the softmax layer. In matrix form:
\[
\frac{dz}{dX}
= - \frac{dz}{dy} \left(\bone^\top \bfe_c - Y\right)
\]
where $X,Y\in\real^{HW\times D}$ are the matrices obtained by reshaping the arrays
$\bx$ and $\by$ and $\bfe_c$ is the indicator vector of class $c$.

% ------------------------------------------------------------------
\subsection{$p$-distance}\label{s:impl-pdistance}
% ------------------------------------------------------------------

The derivative of the operator without root is given by:
\begin{align*}
\frac{dz}{dx_{ijd}}
&=
\frac{dz}{dy_{ij}}
p |x_{ijd} - \bar x_{ijd}|^{p-1} \operatorname{sign} (x_{ijd} - \bar x_{ijd}).
\end{align*}
The derivative of the operator with root is given by:
\begin{align*}
\frac{dz}{dx_{ijd}}
&=
\frac{dz}{dy_{ij}}
\frac{1}{p}
\left(\sum_{d'} |x_{ijd'} - \bar x_{ijd'}|^p \right)^{\frac{1}{p}-1}
p |x_{ijd} - \bar x_{ijd}|^{p-1} \sign(x_{ijd} - \bar x_{ijd})
\\
&=
\frac{dz}{dy_{ij}}
\frac{|x_{ijd} - \bar x_{ijd}|^{p-1} \sign(x_{ijd} - \bar x_{ijd})}{y_{ij}^{p-1}}.
\end{align*}
The formulas simplify a little for $p=1,2$ which are therefore implemented as special cases.


\section{M/G/1 Queues}
\label{sec:M-G-1-queues}

A $M/G/1$ is a queue where
(i) the arrival process is Poissonian with rate $\lambda$,
(ii) the service process is generically distributed,
(iii) there is one servers,
(iv) the buffer has infinite capacity,
(v) the scheduling policy is FCFS.

%\begin{figure}[tp]
%\label{fig:M-G-1-queue}
%	\centering
%	\includegraphics{fig/M-G-1-Queue}
%	\caption{M/G/1 queue}
%\end{figure}

In general, we have that

\begin{equation}
\label{sec:M-G-1-Mean-Queue-Excess-Time}
\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\expected{S_{e}}
\end{equation}

where $\expected{T_{e}}$ is the \textit{excess of service time}, that is the remaining service time of the job in service, given that there is some job in service.

\subsection{P-K Formula}
\label{sec:PK-Formula}
The Pollaczek-Khinchin (P-K Formula) \cite{pollaczek1930aufgabe,khinchin1967mathematical} is written in several equivalent forms:

\begin{equation}
\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\cdot\frac{\expected{S^{2}}}{2\expected{S}}
\end{equation}

\begin{equation}
\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\cdot\frac{\expected{S}}{2}\cdot(C_{S}^{2}+1)
\end{equation}

\begin{equation}
\expected{T_{Q}}=\frac{\lambda\expected{S^{2}}}{2(1-\varrho)}
\end{equation}


\section{Homework}

\subsection{Non-Linear Case}

Given the conditions $x(0) = 0.5$ and $F(t) = F$, we
evaluate the Laplace transform of \ref{eq: system equation}
and obtain

\begin{equation}
  s \mathcal{X} - 0.5 + \frac{k}{b} \mathcal{X} =
\frac{1}{b} \frac{F}{s}
\end{equation}

Solving for $\mathcal{X}$.

\begin{equation}
 \mathcal{X} = \frac{F}{b}
\frac{1}{s(s+\frac{k}{b})} + \frac{0.5}{s+\frac{k}{b}}
\label{eq: non-linear system equation laplace}
\end{equation}

Using partial fractions, find an equivalent expression for
\ref{eq: non-linear system equation laplace} .

\begin{equation*}
 \begin{split}
  \frac{1}{s(s+\frac{k}{b})}  & = \frac{A}{s} +
\frac{B}{s+\frac{k}{b}}\\
A & = \frac{b}{k}\\
B &= -\frac{b}{k}\\
 \end{split}
\end{equation*}

\begin{equation}
 \frac{1}{s(s+\frac{k}{b})}   = \frac{1}{s} \frac{b}{k} -
\frac{b}{k}
\frac{1}{s+\frac{k}{b}}
\label{eq: partial fractions}
\end{equation}

Substitute \ref{eq: partial fractions} in \ref{eq:
non-linear system equation laplace} and then obtain the
inverse Laplace transform of the equation.

\begin{equation*}
 \begin{split}
   \mathcal{X} & = \frac{F}{b}
\Bigl ( \frac{1}{s} \frac{b}{k} -
\frac{b}{k}
\frac{1}{s+\frac{k}{b}} \Bigr) + \frac{0.5}{s+\frac{k}{b}}
\\
& = \frac{F}{k} \frac{1}{s}  -
\frac{F}{k}
\frac{1}{s+\frac{k}{b}} + \frac{0.5}{s+\frac{k}{b}}
\\
 \end{split}
\end{equation*}

\begin{equation}
 \mathfrak{L}^{-1} \Bigl \{ \mathcal{X} \Bigr \}
\xrightarrow{} x(t) = \frac{F}{k} - \frac{F}{k}
\mathrm{e}^{-\frac{k}{b}t} + 0.5 \mathrm{e}^{-\frac{k}{b}t}
\label{eq: non-linear system equation}
\end{equation}

Test the principles of proportionality and superposition on
\ref{eq: non-linear system equation}.

\begin{equation*}
 \begin{split}
 x(\alpha t) & = \frac{F}{k} - \frac{F}{k}
\mathrm{e}^{-\alpha \frac{k}{b}t} + 0.5
\mathrm{e}^{-\alpha \frac{k}{b}t}\\
 \alpha x(t) &= \alpha \frac{F}{k} - \alpha \frac{F}{k}
\mathrm{e}^{-\frac{k}{b}t} + 0.5 \alpha
\mathrm{e}^{-\frac{k}{b}t}\\
x(\alpha t) & \neq \alpha x(t)\\
  x( t_1 + t_2) & = \frac{F}{k} - \frac{F}{k}
\mathrm{e}^{-\frac{k}{b}(t_1+t_2)} + 0.5
\mathrm{e}^{-\frac{k}{b}(t_1+t_2)}\\
x(t_1) +  x(t_2) & = 2 \frac{F}{k} -
\frac{F}{k} \Bigl (
\mathrm{e}^{- \frac{k}{b}t_1} -\mathrm{e}^{- \frac{k}{b}t_2}
\Bigr ) + 0.5
\Bigl (\mathrm{e}^{- \frac{k}{b}t_1}
 +
\mathrm{e}^{- \frac{k}{b}t_2} \Bigr) \\
x(t_1 + t_2) & \neq x(t_1) + x(t_2)\\
 \end{split}
\end{equation*}

We conclude that the system in Non-linear for the given
initial conditions.