% corrected VD 89

\subsubsection{Feedforward Neural Network}~\\

Feedforward neural networks or multilayer perceptrons (MLPs) are very important
deep learning models. The objective of MLPs is to approximate a given function
$f^{*}$. A feedforward network defines a mapping $\bm{\hat{y}} = f(\bm{x};
\bm{\theta})$ and learns the values of the parameter $\bm{\theta}$ with a given
input vector $\bm{x}$ which yields the best approximation of $f^{*}$. This
function can be for example a classifier, $y = f^{*}(\bm{x})$ which maps an
input $\bm{x}$ to a class $y$.\\

These structures are called \textbf{feedforward} since there is a progress of
information in one direction: From input to output.\\

MLPs are considered \textbf{networks} because they are made up of composed
functions. Let $f_{1}$, $f_{2}$ and $f_{3}$ be three function forming a chain
$f(\bm{x}) = f_{3}(f_{2}(f_{1}(\bm{x})))$. The chain structures are commonly
used in ANNs. In this example, $f_{1}$ is called the first layer of the network,
$f_{2}$ is the second layer, etc. The lenght of the chain represents the depth
of the network. The final layer is also called the output layer.\\

During the training of the neural network, we try to find $f(\bm{x})$ to
approximate $f^{*}(\bm{x})$. The training data contains examples of
$f^{*}(\bm{x})$ obtained with different training points where every example
$\bm{x}$ is referring to a label $y \approx f^{*}(\bm{x})$. These examples
determine how the output layer should approximate a value close to $y$ with each
example $\bm{x}$. The other layers are called the \textbf{hidden layers} because
the training examples are not determining their behaviour and it does not
contain the desired output of each layer. Therefore, a learning algorithm has to
determine how to utilize the hidden layers to obtain a good approximation of
$f^{*}(\bm{x})$.\\

The \textbf{width} of the feedforward model is defined by the dimensionality of
each hidden layer which is usually a vector containing values. These vector
layers contain many \textbf{units} which process data in parallel. Each unit
refers to a vector-to-scalar function, by taking input from other units and
calculating its proper activation value.\\

We will present a simple MLP with one hidden layer which contains hidden units.
The hidden layer represents a vector of hidden units $\bm{h}$ which are
calculated by a function $f_{1}(\bm{x;\theta})$. $\bm{\theta}$ consists of
$\bm{W}$ and $\bm{b}$ which are the $weights$ and $biases$
repectively such that:

\begin{equation}
  \begin{split}
    h & = f_{1}(\bm{x;W,b}) \\
    \Leftrightarrow h & = \bm{W}^{T}\bm{x}+\bm{b}
  \end{split}
\end{equation}

$\bm{h}$ is then used as input for the second layer which is the output layer;
$f_{2}(\bm{h;V,c})$, with $V$ being the weigth matrix and $c$ the bias vector.

\begin{equation}
  \begin{split}
    y & = f_{2}(\bm{h;V,c}) \\
    \Leftrightarrow y & = \bm{V}^{T}\bm{h}+\bm{c}
  \end{split}
\end{equation}

The network can be illustrated as a chain of two functions:

\begin{equation}
  \begin{split}
  f(\bm{x;W,b,V,c}) \\
  = f_{2}(f_{1}(\bm{x}))
  \end{split}
\end{equation}

After using an affine transformation, the hidden units are computed by a
nonlinear function $g$ called an activation function:

\begin{equation}
  h = g(\bm{W}^{T}\bm{x+b})
\end{equation}

The common activation function in deep learning is the \textbf{rectified linear
unit} or ReLU defined as:

\begin{equation}
  g(z) = \max\{0,z\}
\end{equation}

\input{sections/scientific/fr2/relu.tex}

The complete feedforward network looks as follows:
\begin{equation}
  \begin{split}
  f(\bm{x;W,b,V,c}) \\
  = \bm{V}^{T}\max\{0,\bm{W}^{T}\bm{x+b}\}+\bm{c}
\end{split}
\end{equation}

A representation of a single perceptron and its position in an MLP is given in
Figure~\ref{ff}.\\

\input{sections/scientific/fr2/mlpfigure.tex}

\textbf{Gradient-Based Learning.} During the training of the model, deep
learning algorithms solve an optimization problem. The problem is to minimize
the error represented by an objective function (loss).  With the loss function
the deep learning model optimizes the approximation of $f^{*}$. An example of a
loss function for MLP can be the mean squared error (MSE):\\

\begin{equation}
  \frac{1}{n}\sum_{i=1}^n(f^*(\bm x_i)-f(\bm x_i;\bm{\theta}))^2
\end{equation}~\\

ANNs usually uses gradient-based optimizers to obtain a minimal loss function.\\

\textbf{Back-propagation.} Gradient descent is a back-propagation algorithm
which performs a backward pass through the network while adjusting the model's
parameters which are the weights and biases. It repeatedly modifies the
parameters of the units in the network to minimize the error between the
predicted and desired output vector. The algorithm passes information from the
loss function backwards through the network. It computes the gradient while
passing back the network. There are three types of gradient descent, namely
batch gradient descent, stochastic gradient descent (SGD) and mini-batch
gradient descent. Compared to batch gradient descent, SGD selects randomly a
data sample instead of a whole data batch for each backward iteration.


\providecommand{\main}{../..}
\documentclass[\main/thesis.tex]{subfiles}
\begin{document}

\section{The Addition Function}\label{addition}

The addition function on \lstinline|ℕ| can consequently be defined by
recursively ``moving'' the successor function \lstinline|suc| from one number
to the another.

\begin{lstlisting}
_+_ : ℕ → ℕ → ℕ
zero  + y = y
suc x + y = suc (x + y)
\end{lstlisting}

We may attempt to define addition on \lstinline|Numeral| with the same strategy.

\begin{lstlisting}
_+_ : ∀ {b d o}
    → Numeral b d o
    → Numeral b d o
    → Numeral b d o
x ∙      + ys = ? x ys
(x ∷ xs) + ys = ? x (xs + ys)
\end{lstlisting}

However, there is not a corresponding successor function on \lstinline|Numeral|
that allows us to recursively move a digit from one numeral to the another with.
What we need is some sort of ``one-sided'' addition function that takes
\textit{a digit} and \textit{a numeral} instead of two numerals.

\begin{lstlisting}
n+ : ∀ {b d o}
    → Digit d
    → Numeral b d o
    → Numeral b d o
\end{lstlisting}

Moreover, to ensure that a system is closed under these operations,
we require these systems to be \lstinline|Continuous|.

\begin{lstlisting}
∀ {b d o} → True (Continuous? b d o)
\end{lstlisting}

\subsection{Sum of Two Digits}

Before implementing \lstinline|n+|, we need to take a step back and figure out
how to add \textit{two digits} together.
Adding two digits should result in \textbf{a digit} and
possibly \textbf{a carry} when the sum overflows.

The carry is often a \textit{fixed digit} in most of the systems that we are
familiar with.
However, when it comes to redundant systems,
the carry is often \textit{indefinite},
because there is more than one way of representing a number.

\subsubsection{View for the Sum}

We capture these cases with a view called \lstinline|Sum|.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
data Sum : (b d o : ℕ) (x y : Digit (suc d)) → Set where
\end{lstlisting}

When the sum of two digits can still be represented by a single digit,
we compute the sum as \lstinline|leftover| and support it with a proof.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
    NoCarry : ∀ {b d o x y}
        → (leftover : Digit (suc d))
        → (property : Digit-toℕ leftover o ≡ sum o x y)
        → Sum b d o x y
\end{lstlisting}

When the sum of two digits exceeds the upper bound of a digit,
we move the exceeding part to \lstinline|carry| and leave the remainder to
\lstinline|leftover|. \lstinline|property| ensures the integrity of this
computation.

We opt for the constructor \lstinline|Fixed| when the exceeding part that goes
to \lstinline|carry| is fixed (equals to \lstinline|1 ⊔ o|) and
\lstinline|Floating| when it is indefinite.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
    Fixed Floating : ∀ {b d o x y}
        → (leftover carry : Digit (suc d))
        → (property : Digit-toℕ leftover o
                        + (Digit-toℕ carry o) * suc b ≡ sum o x y)
        → Sum b d o x y
\end{lstlisting}

We dispense with the implementation of the corresponding view function
\lstinline|sumView| for brevity.
\footnote{about 300 lines of code and reasoning.}

\begin{lstlisting}
sumView : ∀ b d o
    → (¬gapped : ¬ (Gapped#0 b d o))
    → (proper : 2 ≤ suc d + o)
    → (x y : Digit (suc d))
    → Sum b d o x y
\end{lstlisting}

\subsubsection{\lstinline|n+| on Proper Systems}

Implementing \lstinline|n+| on proper systems are relatively simple since most
of the hard work has been done by \lstinline|sumView|.
Carries are added back by recursively calling \lstinline|n+-Proper|.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
n+-Proper : ∀ {b d o}
    → (¬gapped : ¬ (Gapped#0 b d o))
    → (proper : suc d + o ≥ 2)
    → (x : Digit (suc d))
    → (xs : Numeral (suc b) (suc d) o)
    → Numeral (suc b) (suc d) o
n+-Proper {b} {d} {o} ¬gapped proper x xs
    with sumView b d o ¬gapped proper x (lsd xs)
n+-Proper ¬gapped proper x (_ ∙)    | NoCarry leftover property
    = leftover ∙
n+-Proper ¬gapped proper x (_ ∷ xs) | NoCarry leftover property
    = leftover ∷ xs
n+-Proper ¬gapped proper x (_ ∙)    | Fixed leftover carry property
    = leftover ∷ carry ∙
n+-Proper ¬gapped proper x (_ ∷ xs) | Fixed leftover carry property
    = leftover ∷ n+-Proper ¬gapped proper carry xs
n+-Proper ¬gapped proper x (_ ∙)    | Floating leftover carry property
    = leftover ∷ carry ∙
n+-Proper ¬gapped proper x (_ ∷ xs) | Floating leftover carry property
    = leftover ∷ n+-Proper ¬gapped proper carry xs
\end{lstlisting}

\subsubsection{Properties of \lstinline|n+-Proper|}

The property below verifies the correctness of \lstinline|n+-Proper|.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
n+-Proper-toℕ : ∀ {b d o}
    → (¬gapped : ¬ (Gapped#0 b d o))
    → (proper : suc d + o ≥ 2)
    → (x : Digit (suc d))
    → (xs : Numeral (suc b) (suc d) o)
    → ⟦ n+-Proper ¬gapped proper x xs ⟧ ≡ Digit-toℕ x o + ⟦ xs ⟧
n+-Proper-toℕ ¬gapped proper x xs with sumView b d o ¬gapped proper x (lsd xs)
n+-Proper-toℕ ¬gapped proper x (_ ∙)     | NoCarry _ property = property
n+-Proper-toℕ ¬gapped proper x (x' ∷ xs) | NoCarry leftover property =
    begin
        ⟦ leftover ∷ xs ⟧
    ≡⟨ ... property ... ⟩
        Digit-toℕ x o + ⟦ x' ∷ xs ⟧
    ∎
n+-Proper-toℕ ¬gapped proper x (_ ∙)     | Fixed _ _ property = property
n+-Proper-toℕ ¬gapped proper x (x' ∷ xs) | Fixed leftover carry property =
    begin
        ⟦ leftover ∷ n+-Proper ¬gapped proper carry xs ⟧
    ≡⟨ ... n+-Proper-toℕ ... ⟩
        Digit-toℕ leftover o + (Digit-toℕ carry o + ⟦ xs ⟧) * suc b
    ≡⟨ ... property ... ⟩
        Digit-toℕ x o + ⟦ x' ∷ xs ⟧
    ∎
n+-Proper-toℕ ¬gapped proper x (_ ∙)     | Floating _ _ property = property
n+-Proper-toℕ ¬gapped proper x (x' ∷ xs) | Floating leftover carry property =
    begin
        ⟦ leftover ∷ n+-Proper ¬gapped proper carry xs ⟧
    ≡⟨ ... n+-Proper-toℕ ... ⟩
        Digit-toℕ leftover o + (Digit-toℕ carry o + ⟦ xs ⟧) * suc b
    ≡⟨ ... property ... ⟩
        Digit-toℕ x o + ⟦ x' ∷ xs ⟧
    ∎
\end{lstlisting}

\subsubsection{Generalizing \lstinline|n+-Proper| to all Systems}

By imposing the condition of continuity, we exclude systems that are not
acceptable for \lstinline|n+-Proper|.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
n+ : ∀ {b d o}
    → {cont : True (Continuous? b d o)}
    → (n : Digit d)
    → (xs : Numeral b d o)
    → Numeral b d o
n+ {b} {d} {o}      n xs with numView b d o
n+ {_} {_} {_} {()} n xs | NullBase d o
n+ {_} {_} {_}      n xs | NoDigits b o = NoDigits-explode xs
n+ {_} {_} {_} {()} n xs | AllZeros b
n+ {_} {_} {_}      n xs | Proper b d o proper with Gapped#0? b d o
n+ {_} {_} {_} {()} n xs | Proper b d o proper | yes gapped#0
n+ {_} {_} {_}      n xs | Proper b d o proper | no ¬gapped#0
    = n+-Proper ¬gapped#0 proper n xs
\end{lstlisting}

The proof of the correctness of \lstinline|n+-toℕ| also follows the same pattern.

\begin{lstlisting}
n+-toℕ : ∀ {b d o}
    → {cont : True (Continuous? b d o)}
    → (n : Digit d)
    → (xs : Numeral b d o)
    → ⟦ n+ {cont = cont} n xs ⟧ ≡ Digit-toℕ n o + ⟦ xs ⟧
\end{lstlisting}

\subsubsection{Implmentation of the Addition Function}

First, we define the addition function on systems of \lstinline|Proper| with
the one-sided addition function \lstinline|n+-Proper|.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
+-Proper : ∀ {b d o}
    → (¬gapped : ¬ (Gapped#0 b d o))
    → (proper : suc d + o ≥ 2)
    → (xs ys : Numeral (suc b) (suc d) o)
    → Numeral (suc b) (suc d) o
+-Proper ¬gapped proper (x ∙)    ys       = n+-Proper ¬gapped proper x ys
+-Proper ¬gapped proper (x ∷ xs) (y ∙)    = n+-Proper ¬gapped proper y (x ∷ xs)
+-Proper {b} {d} {o} ¬gapped proper (x ∷ xs) (y ∷ ys)
    with sumView b d o ¬gapped proper x y
+-Proper ¬gapped proper (x ∷ xs) (y ∷ ys) | NoCarry leftover property
    = leftover ∷ +-Proper ¬gapped proper xs ys
+-Proper ¬gapped proper (x ∷ xs) (y ∷ ys) | Fixed leftover carry property
    = leftover ∷ n+-Proper ¬gapped proper carry (+-Proper ¬gapped proper xs ys)
+-Proper ¬gapped proper (x ∷ xs) (y ∷ ys) | Floating leftover carry property
    = leftover ∷ n+-Proper ¬gapped proper carry (+-Proper ¬gapped proper xs ys)
\end{lstlisting}

By generalizing \lstinline|+-Proper| to all continuous numeral systems,
the addition function we have been long for can be implemented as follows.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
_+_ : ∀ {b d o}
    → {cont : True (Continuous? b d o)}
    → (xs ys : Numeral b d o)
    → Numeral b d o
_+_ {b} {d} {o}  xs ys with numView b d o
_+_ {cont = ()}   xs ys | NullBase d o
_+_ {cont = cont} xs ys | NoDigits b o = NoDigits-explode xs
_+_ {cont = ()}   xs ys | AllZeros b
_+_ {cont = cont} xs ys | Proper b d o proper with Gapped#0? b d o
_+_ {cont = ()}   xs ys | Proper b d o proper | yes ¬gapped#0
_+_ {cont = cont} xs ys | Proper b d o proper | no ¬gapped#0
    = +-Proper ¬gapped#0 proper xs ys
\end{lstlisting}

\subsubsection{Properties of the Addition Function}

We can prove that the evaluation function \lstinline|⟦_⟧| is a homomorphism
between \lstinline|Numeral| and \lstinline|ℕ| that preserves their addition
functions.

\begin{lstlisting}[basicstyle=\ttfamily\scriptsize]
toℕ-+-homo : ∀ {b d o}
    → (cont : True (Continuous? b d o))
    → (xs ys : Numeral b d o)
    → ⟦ _+_ {cont = cont} xs ys ⟧ ≡ ⟦ xs ⟧ + ⟦ ys ⟧
\end{lstlisting}

The proofs are skipped for brevity.
\footnote{about 100 lines of code and reasoning.}

\end{document}


\subsection{Neuron Dynamics}
\label{subsec:neuron_dynamics}

\subsubsection{Current}

The original paper described the dynamics of neuron \(\nu_{i}\) as the change of its current \(c_{i}\) with time:

        \begin{equation}
            \tau \dot{c}_{i}(t) = - {c}_{i}(t) + \sum_{j=1}^{N} r_{j}(t) \cdot W_{i,j} + \xi_{i}(t)
        \label{eq:continuous_dynamics} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(\tau \in \R\) is the decay time, \\
            \(c \in \R\) is the synaptic current, \\
            \(N \in \N\) is the network number of neurons, \\
            \(W\) is the weight matrix, \\
            \(r \in \R\) is the firing rates, \\
            \(\xi \in \R\) is the Gaussian noise.
        \label{tab:conditions_continuous_dynamics} \end{tabular}

    The former equation can be discretized using the Euler method:

        \begin{equation}
            c_{i}(t+1) = -c_{i}(t) + \frac{dt}{\tau} \left[-c_{i}(t) + \sum_{j=1}^{N} r_{j}(t) \cdot W_{i,j} + \frac{\xi_{i}(t)}{\sqrt{dt}} \right]
        \label{eq:discrete_dynamics} \end{equation}

        where:

        \begin{tabular}{l}
            \(dt \in \R\) is the integration time step
        \label{tab:conditions_discrete_dynamics} \end{tabular}

\subsubsection{Firing Rates}

Firing rates \(r\) of each neuron are calculated by the gain function \(g(c)\), a step function with sublinear behavior or a value of zero:

        \begin{equation}
            g(c) =
            \begin{cases}
                (c + \theta)^{\gamma}   & \text{if~} (c + \theta) > 0 \\
                0                       & \text{if~} (c + \theta) \leq 0
            \end{cases}
        \label{eq:firing_rates} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(\theta \in \R\) is the gain function threshold, \\
            \(\gamma \in \R_{<1}\) is the gain function exponent. \\
        \label{tab:conditions_firing_rates} \end{tabular} \bigskip

\subsubsection{Memory Patterns}

Every memory is represented by a binary vector or pattern \(\boldsymbol{p}\) of size \(N\).
Each element of this vector corresponds to the state \(s\) of each neuron \(\nu\) for that memory pattern.
The value of this state is \(0\) if the neuron does not encode for that pattern and \(1\) if it does.

The different states are stored in a \(M \times N\) matrix, with shape:

    \[\bordermatrix{
        ~ & \nu_1 & \nu_2 & \cdots & \nu_n  \cr
        p_1 &  s_{1}^{1} & s_{2}^{1} & \cdots & s_{n}^{1}  \cr
        p_2 &  1 & 0 & \cdots & 1  \cr
        \vdots &  \vdots  & \vdots & \ddots & \vdots  \cr
        p_m &  s_{1}^{m} & s_{2}^{m} & \cdots & s_{n}^{m}
    }\]

\subsubsection{Inhibition}

    The network is subjected to periodic inhibition driven by a sine wave \(\phi(t)\), with the form:

        % \begin{equation}
        %      \begin{gathered}
        %         A = \frac{1}{2} (\phi_{max} - \phi_{min}) \\
        %         \phi(t) = A \left(1 + \frac{\phi_{min}}{A} + \sin\left[
        %         2 \pi \left(t + \frac{p_{\phi}}{dt}\right) \frac{1}{t_{\phi}} dt
        %         \right] \right)
        %      \end{gathered}
        % \label{eq:sine_wave} \end{equation}

        \begin{equation}
            \frac{\phi_{max} - \phi_{min}}{2} \left[1 +  \sin \left(2 \pi t + \frac{\pi}{2} \right) \right]
        \end{equation}

        where:

        \begin{tabular}{l} \\
            %\(A \in \R_{>0}\) is the amplitude, \\
            \(\phi_{min} \in \R\) is the minimum inhibition hyperparameter, \\
            \(\phi_{max} \in \R\) is the maximum inhibition hyperparameter, \\
            %\(p_{\phi} \in \R\) is the oscillation phase shift, \\
            %\(t_{\phi} \in \R\) is the oscillation time. \\
        \label{tab:conditions_sine_wave} \end{tabular} \bigskip

\subsubsection{Weights}

    Each neuron in the network is fully connected to all the other neurons.
    This gives a \(N \times N\) weight matrix \(\boldsymbol{W}_{i, j}\) representing the strength of connection or weight \(w\) between neurons \(\nu_{i}\) and \(\textit{}\nu_{j}\):

    \[\bordermatrix{
        ~ & \nu_1 & \nu_2 & \cdots & \nu_n  \cr
        \nu_1 &  w_{1, 1} & w_{1, 2} & \cdots & w_{1, n}  \cr
        \nu_2 &  w_{2, 1} & w_{2, 2} & \cdots & w_{2, n}  \cr
        \vdots &  \vdots  & \vdots & \ddots & \vdots  \cr
        \nu_m &  w_{m, 1} & w_{m, 2} & \cdots & w_{m, n}
    }\] \bigskip

    To calculate the weight matrix, the following Hebbian rule is used:

        \begin{equation}
            W_{i,j} = \frac{\kappa}{N} \left[ \sum_{p=1}^{M} (s_{i}^{p} - f) (s_{j}^{p} - f) - \phi(t) \right]
        \label{eq:weights} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(\kappa \in \R_{\geqslant 0}\) is the excitation, \\
            \(M \in \N\) is the number of memories, \\
            \(s \in \Z_{[0, 1]}\) is the neuron state for a memory, \\
            \(p\) is the memory pattern, \\
            \(f \in \R\) is the sparsity, \\
            \(\phi \in \R\) is the oscillatory inhibition.
        \label{tab:conditions_weights} \end{tabular} \bigskip

    To account for short term associations to the previous and next memories as in the SAM model, a new term \(\boldsymbol{W}_{i,j}^{*}\) is added to the original weight matrix:

        \begin{equation}
        W_{i, j}^{SAM} = W_{i, j} + W_{i, j}^{*} = W_{i, j} + \frac{\kappa}{N} \left[ \kappa_{f} \sum_{p=1}^{M-1} s_{i}^{p} \cdot s_{j}^{p+1} + \kappa_{b} \sum_{p=2}^{M} s_{i}^{p} \cdot s_{j}^{p-1} \right]
        \label{eq:weights_sam} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(\kappa_{f} \in \R\) is the forward contiguity, \\
            \(\kappa_{b} \in \R\) is the backward contiguity. \\
        \label{tab:conditions_weights_sam} \end{tabular} \bigskip

\subsubsection{Noise}

    Each neuron is subjected to Gaussian noise \(\xi\), following the probability density function:

        \begin{equation}
            p(z) = \frac{1}{\sigma \sqrt{ 2 \pi}} e^{ - \frac{ (z - \mu)^2 } {2 \sigma^2} }
        \label{eq:noise} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(\mu \in \R\) is the noise mean, \\
            \(\sigma \in \R_{\geqslant 0}\) is the noise standard deviation.
        \label{tab:conditions_noise} \end{tabular} \bigskip

\subsection{Population Dynamics}
\label{subsec:population_dynamics}

    Simulating the network with the original parameters is very computationally expensive, with the computation time depending primarily on the number of neurons.
    The system can be simplified, reducing the number of simulated units.
    All neurons that present the same activation state for the different memories will be considered that belong to the same population \(\pi\).
    Moreover, all these neurons have an identical weight matrix \(\boldsymbol{W}_{i, j}\)\footnote{\(i j\) notation refers to the simulated unit, which is from now on a population of neurons instead of the single neuron.}, which will be the weight matrix of the population.

\subsubsection{Current}

    A new term \(S_{\pi}\) is introduced in the calculation:

        \begin{equation}
            S_{\pi} = \frac{N_{\pi}}{N}
        \label{eq:fraction_neurons_pop} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(N_{\pi} \in \N\) is the number of neurons in a population. \\
        \label{tab:fraction_neurons_pop} \end{tabular} \bigskip

    The change of current of population \(\pi_{i}\) with time is:

        \begin{equation}
            \tau \dot{c}_{i}(t) = - {c}_{i}(t) + \sum_{j=1}^{U} r_{j}(t) \cdot W_{i,j} \cdot S_{\pi} + \xi_{i}(t)
        \label{eq:population_current} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(U \in \N\) is the number of unique populations. \\
        \label{tab:conditions_population_current} \end{tabular} \bigskip

    \medskip After discretizing by Euler:

        \begin{equation}
            c_{i}(t+1) = -c_{i}(t) + \frac{dt}{\tau} \left[- c_{i}(t) + \sum_{j=1}^{U} r_{j}(t) \cdot W_{i,j} \cdot S_{\pi} + \frac{\xi_{i}(t)}{\sqrt{dt}} \right]
        \label{eq:discrete_population_currents} \end{equation}

    % where:
    %
    % \begin{tabular}{l} \\
    %     \(\xi \in \R\) is the Gaussian noise adapted to populations.
    % \label{tab:conditions_discrete_population_currents} \end{tabular}

\subsubsection{Firing Rates}

    % An extra parameter \(\zeta\) was introduced in the gain function to achieve replication:
    Firing rates are calculated with the gain function as before. % with the following gain function:

        % \begin{equation}
        %     g(c) =
        %     \begin{cases}
        %         (c + \theta)^{\gamma}   & \text{if~} (c + \theta) > 0 \\
        %         0                       & \text{if~} (c + \theta) \leq 0
        %     \end{cases}
        %     %\begin{cases}  hopefully no replication param
        %     %    (c \cdot \zeta + \theta)^{\gamma}   & \text{if~} (c + \theta) > 0 \\
        %     %    0                       & \text{if~} (c + \theta) \leq 0
        %     %\end{cases}
        % \label{eq:population_firing_rates} \end{equation}
        %
        % where:
        %
        % \begin{tabular}{l} \\
        %     %\(\zeta \in \R\) is the current modulation parameter, \\  replication param
        %     \(\theta \in \R\) is the gain function threshold, \\
        %     \(\gamma \in \R_{<1}\) is the gain function exponent. \\
        % \label{tab:conditions_population_firing_rates} \end{tabular} \bigskip

\subsubsection{Memory Patterns}

The \(M \times N\) matrix containing \(s\) states is now a \(M \times U\) matrix.
This second matrix contains fewer elements than the original matrix, allowing for faster computation.

\subsubsection{Inhibition}

Inhibition calculation remains unchanged for the simulation with populations.

\subsubsection{Weights}

    The population weight matrix \(\boldsymbol{W}_{i, j}\) for neuron populations, with \(U \times U\) shape:

        \begin{equation}
            W_{i,j} = \frac{\kappa}{N} \left[\sum_{p=1}^{M} (u_{i}^{p} - f) (u_{j}^{p} - f) - \phi(t) \right]
        \label{eq:population_weights} \end{equation}

        \begin{equation}
            W_{i, j}^{SAM} = W_{i, j} + W_{i, j}^{*} = W_{i,j} + \frac{\kappa}{N} \left[ \kappa_{f} \sum_{p=1}^{M-1} u_{i}^{p} \cdot u_{j}^{p + 1} + \kappa_{b} \sum_{p=2}^{M} u_{i}^{p} \cdot u_{j}^{p - 1} \right]
        \label{eq:population_weights_sam} \end{equation}

        where:

        \begin{tabular}{l} \\
            \(u \in \Z_{[0, 1]}\) is the activity state of a population. \\
        \label{tab:conditions_population_weights} \end{tabular} \bigskip

\subsubsection{Noise}

    Unmodulated Gaussian noise \(\xi\) is computed as before.
    % Unmodulated Gaussian noise \(\xi^{*}\) is computed with different variance \(\sigma_{\pi}^{2}\):
%
    %     \begin{equation}
    %         \sigma_{\pi}^{2} = \frac{\sigma^{2}}{\sqrt{N_{\pi}}}
    %     \label{eq:population_noise_sd} \end{equation}
%
    %     where:
%
    %     \begin{tabular}{l} \\
    %         \(\mu \in \R\) is the noise mean, \\
    %         \(\sigma \in \R_{\geqslant 0}\) is the noise standard deviation.
    %     \label{tab:conditions_noise_sd} \end{tabular} \bigskip
%
    % Probability density function remains the same as in equation \ref{eq:noise}, with \(\sigma = \sigma_{\pi}\).

%         Unmodulated noise values are then scaled using according to \(N_{\pi}\), and an extra parameter \(\eta\) to achieve replication:
%
%         \begin{equation}
%             \xi = \frac{\xi^{*} \cdot \eta}{N_{\pi}}
%         \label{eq:population_noise} \end{equation}
%
%         where:
%
%         \begin{tabular}{l} \\
%             \(\xi^{*} \in \R\) is the unmodulated noise, \\
%             \(\eta \in \R\) is the noise modulation parameter.
%         \label{tab:conditions_population_noise} \end{tabular} \bigskip

\subsection{Simulation}

Simulation is carried away with population-level conditions (subsection \ref{subsec:population_dynamics}).
Calculations are then based on a \(M \times U\) matrix instead of a much larger \(M \times N\) matrix.
Computation time now scales with \(M\) instead of \(N\) but for very large values of \(N\), which would also increase the number of populations.
Recanatesi \textit{et al.} \parencite{recanatesi2015} estimated this method to be 99.9\% faster than simulating individual neurons.

Firing rates are initialized at \(r_{ini}\) for populations encoding a randomly chosen memory pattern.
Currents are set at \(r_{ini}^{\rfrac{1}{\gamma}}\).
All weights are also defined at this stage.
Values of noise and inhibition change per time step.
Neuron currents and firing rates are then calculated at each time step as well.

A memory is considered recalled if the average firing rate of all encoding neurons is above \(r_{recall}\).
The network is said to recall a certain memory \(p\) if the former condition is ever fulfilled for that memory during the simulation time.

Table \ref{table:hyperparameters} summarizes all hyperparameters used in the simulation.

\subsection{Recall Analysis}

The network was simulated first at a smaller scale and for one trial to observe detailed dynamics.
It was then scaled to computer clusters, allowing to reach a total of 10,000 simulations for each condition needed for memory recall analysis.
Each network is simulated for a total of 450 time cycles.

Several metrics are computed to assess the recall performance of the model.
In particular, inter-retrieval time (IRT) is calculated as the number of time cycles until the recall of a new memory item.
Other performance metrics such as memory size intersections or the average total recalls are also analyzed.

\subsection{Computational Tools}

Replication was carried out using Python 3.8.5 with packages NumPy 1.19.1, pandas 1.0.5, matplotlib 3.3.0, SciPy 1.5.2, and tqdm 4.48.0 on Parabola GNU/Linux-libre and CentOS GNU/Linux.


\lab{A Pseudospectral method for periodic functions}{A Pseudospectral method for periodic functions}
\label{lab:pseudospectral2}

\objective{
We look at a pseudospectral method with a Fourier basis, and numerically solve the advection equation using a pseudospectral discretization in space and a Runge-Kutta integration scheme in time.  }

Let $f$ be a periodic function on $[0,2\pi]$.
Let $x_1,\ldots,x_N$ be $N$ evenly spaced grid points on $[0,2\pi].
$Since $f$ is periodic on $[0,2\pi]$, we can ignore the grid point $x_N = 2\pi$.
We will further assume that $N$ is even; similar formulas can be derived for $N$ odd.
Let $h = 2\pi/N$; then $\{x_0,\ldots,x_{N-1}\} = \{0,h,2h,\ldots,2\pi-h\}$.

The discrete Fourier transform (DFT) of $f$, denoted by $\hat{f}$ or $\mathcal{F}(f)$, is given by
\[
\hat{f}(k) = h \sum_{j=0}^{N-1} e^{-ikx_j}f(x_j) \quad \text{ where } k = -N/2+1, \ldots,0,1,\ldots, N/2.
\]
The inverse DFT is then given by
\begin{align}
\begin{split}
f(x_j) &= \frac{1}{2\pi}\sum_{k=-N/2}^{N/2}\frac{e^{ikx_j}}{c_k}\hat{f}(k), \quad j = 0,\ldots, N-1,
\end{split}\label{inverse_dft}
\end{align}
where
\begin{align}
	c_k = \begin{cases} 2 & \text{if }k = -N/2 \text{ or }k = N/2, \\ 1 &  \text{otherwise.}
\end{cases}
\end{align}
The inverse DFT can then be used to define a natural interpolant (sometimes called a band-limited interpolant) by evaluating (\ref{inverse_dft}) at any $x$ rather than $x_j$:
\begin{align}
p(x) = \frac{1}{2\pi}\sum_{k=-N/2}^{N/2} e^{ikx}\hat{f}(k). \label{interpolant}
\end{align}
The interpolant for $f'$ is then given by
\begin{align}
p'(x) = \frac{1}{2\pi}\sum_{k=-N/2+1}^{N/2-1} ike^{ikx}\hat{f}(k). \label{spectral2:deriv}
\end{align}

Consider the function $u(x) = \sin^2 (x) \cos(x) +e^{2\sin(x+1)}$.
Using \eqref{spectral2:deriv}, the derivative $u'$ may be approximated with the following code.
\footnote{See \textit{Spectral Methods in MATLAB} by Lloyd N. Trefethen.
Another good reference is \textit{Chebyshev and Fourier Spectral Methods} by John P. Boyd.}
We note that although we only approximate $u'$ at the Fourier grid points, \eqref{spectral2:deriv} provides an analytic approximation of $u'$ in the form of a trigonometric polynomial.

\begin{lstlisting}
import numpy as np
from scipy.fftpack import fft, ifft
import matplotlib.pyplot as plt

N=24
x1 = (2.*np.pi/N)*np.arange(1,N+1)
f = np.sin(x1)**2.*np.cos(x1) + np.exp(2.*np.sin(x1+1))

# This array is reordered in Python to
# accomodate the ordering inside the fft function in scipy.
k = np.concatenate(( np.arange(0,N/2) ,
					 np.array([0])	, # Because hat{f}'(k) at k = N/2 is zero.
					 np.arange(-N/2+1,0,1)	))

# Approximates the derivative using the pseudospectral method
f_hat = fft(f)
fp_hat = ((1j*k)*f_hat)
fp = np.real(ifft(fp_hat))

# Calculates the derivative analytically
x2 = np.linspace(0,2*np.pi,200)
derivative = (2.*np.sin(x2)*np.cos(x2)**2. -
				np.sin(x2)**3. +
				2*np.cos(x2+1)*np.exp(2*np.sin(x2+1))
				)

plt.plot(x2,derivative,'-k',linewidth=2.)
plt.plot(x1,fp,'*b')
plt.savefig('spectral2_derivative.pdf')
plt.show()
\end{lstlisting}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{spectral2_derivative.pdf}
\caption{The derivative of $u(x) = \sin^2 (x) \cos(x) +e^{2\sin(x+1)}$.}
\label{fig:spectral:spectral2_derivative}
\end{figure}

\begin{problem}
Consider again the function $u(x) = \sin^2 (x) \cos(x) +e^{2\sin(x+1)}$.
Create a function that approximates $\frac{1}{2}u''-u'$ on the Fourier grid points for $N=24$.
\end{problem}

\section*{The advection equation}
Recall that the advection equation is given by
\begin{align}
&{ }u_t + cu_x = 0
\end{align}
where $c$ is the speed of the wave (the wave travels to the right for $c > 0$).
We will consider the solution of the advection equation on the circle; this essentially amounts to solving the advection equation on $[0,2\pi]$ and assuming periodic boundary conditions.

A common method for solving time-dependent PDEs is called the \textit{method of lines}.
To apply the method of lines to our problem, we use our Fourier grid points in $[0,\pi]$: given an even $N$, let $h = 2\pi/N$, so that $\{x_0,\ldots,x_{N-1}\} = \{0,h,2h,\ldots,2\pi-h\}$.
By using these grid points we obtain the collection of equations
\begin{align}
&{ }u_t(x_j,t) + cu_x(x_j,t) = 0, \quad t >0, \quad j = 0, \ldots N-1. \label{spectral2:method_oflines}
\end{align}

Let $U(t)$ be the vector valued function given by $U(t) = (u(x_j,t))_{j=0}^{N-1}$.
Let $\mathcal{F}(U)(t)$ denote the discrete Fourier transform of $u(x,t)$ (in space), so that
\[
\mathcal{F}(U)(t) = (\hat{u}(k,t))_{k=-N/2+1}^{N/2}.
\]
Define $\mathcal{F}^{-1}$ similarly.
Using the pseudospectral approximation in space leads to the system of ODEs
\begin{align}
	U_t +  \vec{c}\mathcal{F}^{-1}\left(i\vec{k}\mathcal{F}(U) \right) = 0
\end{align}
where $\vec{k}$ is a vector, and $\vec{k}\mathcal{F}(U) $ denotes element-wise multiplication.
Similarly $\vec{c}$ could also be a vector, if the wave speed $c$ is allowed to vary.

\begin{problem}
	Using a fourth order Runge-Kutta method (RK4), solve the initial value problem
	\begin{align}
		u_t +c(x) u_x = 0,
	\end{align}
where $c(x) = .2 + \sin^2(x-1)$, and $u(x,t=0) = e^{-100(x-1)^2}.$
Plot your numerical solution from $t = 0$ to $t = 8$ over 150 time steps and 100 $x$ steps.
Note that the initial data is nearly zero near $x = 0$ and $2 \pi$, and so we can use the pseudospectral method.
\footnote{This problem is solved in \textit{Spectral Methods in MATLAB} using a leapfrog discretization in time. }
\label{spectral2:advection_equation}
Use the following code to help graph.
\begin{lstlisting}
t_steps = 150    # Time steps
x_steps = 100     # x steps

'''
Your code here to set things up
'''

sol = # RK4 method. Should return a t_steps by x_steps array

X,Y = np.meshgrid(x_domain, t_domain)
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X,Y,sol)
ax.set_zlim(0,3)
plt.show()
\end{lstlisting}
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{advection.png}
\caption{The solution of the variable speed advection equation; see Problem \ref{spectral2:advection_equation}.}
\label{fig:spectral:spectral2_advection}
\end{figure}


\section{Nodal Basis and Nodal Space}
\label{sec:21nodalSpaces}

\minitoc{62mm}{4}

\mbox{}\vspace{-14mm}

\disableornamentsfornextheadingtrue
\subsection{Univariate Case}
\label{sec:211nodalUV}

\paragraph{Grid and basis functions}

In this thesis, we consider univariate functions
that are defined on the unit interval $\clint{0, 1}$.
\usenotation{l}
We discretize this domain by splitting it into $2^l$ equally-sized segments,
where $l \in \natz$ is the \term{level.}
\usenotation{i}
The resulting $2^l + 1$ \term{grid points} $\gp{l,i}$ are given by
\begin{equation}
  \gp{l,i} \ceq i \cdot \ms{l},\quad
  i = 0, \dotsc, 2^l,
\end{equation}
where $i$ is the \term{index} and $\ms{l} \ceq 2^{-l}$ is the \term{mesh size.}%
\footnote{%
  Note that from a strict formal perspective,
  this equation defined $\gp{l,i}$ only for $i = 0, \dotsc, 2^l$,
  but we will later need $\gp{l,i}$ also for $i < 0$ or $i > 2^l$.
  The convention in this thesis is that all definitions are
  implicitly generalized whenever needed.%
}
Every grid point is associated with a \term{basis function}
\begin{equation}
  \basis{l,i}\colon \clint{0, 1} \to \real.
\end{equation}
We assume $\basis{l,i}$ to be arbitrary,
satisfying required assumptions when needed and stated.
However, it helps for both the theory and the intuition to have a
specific example of basis functions in mind.
\usenotation{Ë1}
The so-called \term{hat functions} (linear B-splines)
are the most common choice for $\basis{l,i}$:
\begin{equation}
  \label{eq:hatFunctionUV}
  \bspl{l,i}{1}(x)
  \ceq \max(1 - \abs{\tfrac{x}{\ms{l}} - i}, 0).
\end{equation}
Here and in the following,
the superscript ``1'' stands for the degree of the linear B-spline and
is not to be read as an exponent.
We generalize this notation to B-splines $\bspl{l,i}{p}$ of
arbitrary degrees $p$ in \cref{chap:30BSplines}.

\paragraph{Nodal space}

The \term{nodal space} $\ns{l}$ of level $l$
is defined as the linear span of all basis functions
$\basis{l,i}$:
\begin{equation}
  \ns{l} \ceq \spn\{\basis{l,i} \mid i = 0, \dotsc, 2^l\}.
\end{equation}
We assume that the functions $\basis{l,i}$ form a basis of $\ns{l}$, i.e.,
they are linearly independent.
Consequently, every linear combination of these functions is unique.
This ensures that for every objective function $\objfun\colon \clint{0, 1} \to \real$,
there is a unique function $\fgintp{l}\colon \clint{0, 1} \to \real$ such that
\begin{equation}
  \label{eq:interpFullGridUV}
  \fgintp{l}
  = \sum_{i=0}^{2^l} \interpcoeff{l,i} \basis{l,i},\quad
  \falarge{i = 0, \dotsc, 2^l}{\fgintp{l}(\gp{l,i}) = \objfun(\gp{l,i})},
\end{equation}
for some $\interpcoeff{l,i} \in \real$.
In this case, $\fgintp{l}$ is called \term{interpolant} of $\objfun$ in $\ns{l}$.
The nodal space $\nsbspl{l}{1}$ is defined analogously to $\ns{l}$
as the span of the hat functions $\bspl{l,i}{1}$.
It is the space of all linear splines,
that is, the space of all continuous functions on $\clint{0, 1}$ that are
piecewise linear polynomials on $\clint{\gp{l,i}, \gp{l,i+1}}$ for
$i = 0, \dotsc, 2^l - 1$ \cite{Hoellig13Approximation}.
The nodal hat function basis of level~$l = 3$
and a linear combination are shown in \cref{fig:nodalHat}.

\begin{figure}
  \subcaptionbox{%
    Basis functions $\bspl{l,i}{1}$ ($i = 0, \dotsc, 2^l$)
    and grid points $\gp{l,i}$ \emph{(dots).}%
  }[72mm]{%
    \includegraphics{hierarchicalBasis_1}%
  }%
  \hfill%
  \subcaptionbox{%
    Piecewise linear interpolant $\fgintp{l}$
    of some function data $\objfun(\gp{l,i})$
    as a weighted sum of the nodal hat functions.%
  }[72mm]{%
    \includegraphics{interpolant_1}%
  }%
  \caption[%
    Univariate nodal hat functions%
  ]{%
    Univariate nodal hat functions of level $l = 3$.%
  }%
  \label{fig:nodalHat}%
\end{figure}

\subsection{Multivariate Case}
\label{sec:212nodalMV}

\paragraph{Cartesian and tensor products}

\usenotation{d}
For the multivariate case with $d \in \nat$ dimensions,
we employ a tensor product approach,
for which we replace all indices, points, and functions with
multi-indices, Cartesian products, and tensor products, respectively.
\usenotation{@0}
\usenotation{@1}
Therefore, the domain is now $\clint{\*0, \*1} \ceq \clint{0, 1}^d$,
which can be partitioned into
$\prod_{t=1}^d 2^{l_t} = 2^{\normone{\vec{l}}}$ equally-sized hyper-rectangles,
where $\*l = (l_1, \dotsc, l_d) \in \natz^d$ is the $d$-dimensional level
and $\normone{\vec{l}} \ceq \sum_{t=1}^d \abs{l_t}$ is the level sum.
The corners of the hyper-rectangles are given by the grid points
\begin{equation}
  \label{eq:gridPointMultivariate}
  \gp{\*l,\*i} \ceq \*i \cdot \ms{\*l},\quad
  \*i = \*0, \dotsc, \*2^{\*l}.
\end{equation}
Relations and operations with vectors in bold face
are to be read coordinate-wise in this thesis, unless stated otherwise.
Bold-faced numbers like $\*0$ are defined to be the vector $(0, \dotsc, 0)$
in which every entry is equal to that number.
This is to allow a somewhat intuitive and suggestive notation.
For example, \eqref{eq:gridPointMultivariate} is equivalent to
the much longer formula
\begin{equation}
  \gp{\*l,\*i}
  \ceq (i_1 \ms{l_1},\; \dotsc,\; i_d \ms{l_d}),\quad
  i_t = 0, \dotsc, 2^{l_t},\quad
  t = 1, \dotsc, d,
\end{equation}
with the $d$-dimensional mesh size
$\ms{\*l} \ceq \*2^{-\*l} = (\ms{l_1}, \dotsc, \ms{l_d})$.
Again, every grid point is associated with a basis function that is defined
as the tensor product of the univariate functions:%
\footnote{%
  Note that,
  although \cref{eq:tensorProduct} does not cover it,
  one could employ basis functions of different types in
  each dimension, for example B-splines of different degrees.
  All remaining considerations in this thesis
  regarding tensor product basis functions are independent
  of whether we use the same function type or
  different types in each dimension.%
}
\begin{equation}
  \label{eq:tensorProduct}
  \basis{\*l,\*i}\colon \clint{\*0, \*1} \to \real,\quad
  \basis{\*l,\*i}(\*x)
  \ceq \prod_{t=1}^d \basis{l_t,i_t}(x_t).
\end{equation}
\cref{fig:nodalHat2D} shows an example of a bivariate nodal hat function
$\bspl{\*l,\*i}{1}$.

\begin{SCfigure}
  \includegraphics{nodalHat2D_1}%
  \caption[%
    Bivariate nodal hat function%
  ]{%
    Bivariate nodal hat function of level $\*l = (2, 1)$ and
    index $i = (1, 1)$ as the tensor product of two univariate
    nodal hat functions.%
  }%
  \label{fig:nodalHat2D}%
\end{SCfigure}

\vspace*{\fill}
\pagebreak

\paragraph{Multivariate nodal space}

The multivariate nodal space $\ns{\*l}$ is defined analogously to
the univariate case:
\begin{equation}
  \ns{\*l}
  \ceq \spn\{\basis{\*l,\*i} \mid \*i = \*0, \dotsc, \*2^{\*l}\}.
\end{equation}
In the case of hat functions $\bspl{\*l,\*i}{1}$,
the nodal space $\nsbspl{\*l}{1}$ is the $d$-linear spline space
\cite{Hoellig13Approximation}, i.e.,
the space of all continuous functions
on $\clint{\*0, \*1}$ that are piecewise $d$-linear polynomials on
all hyper-rectangles
\begin{equation}
  \clint{\gp{\*l,\*i}, \gp{\*l,\*i+\*1}}
  \ceq \clint{\gp{l_1,i_1}, \gp{l_1,i_1+1}} \times \dotsb \times
  \clint{\gp{l_d,i_d}, \gp{l_d,i_d+1}},\quad
  \*i = \*0, \dotsc, \*2^\*l - \*1.
\end{equation}
Analogously to \eqref{eq:interpFullGridUV},
we can interpolate objective functions $\objfun\colon \clint{\*0, \*1} \to \real$
in the nodal space $\ns{\*l}$ with $\fgintp{\*l}\colon \clint{\*0, \*1} \to \real$ satisfying
\begin{equation}
  \label{eq:interpFullGridMV}
  \fgintp{\*l}
  = \sum_{\*i=\*0}^{\*2^\*l} \interpcoeff{\*l,\*i} \basis{\*l,\*i},\quad
  \falarge{\*i = \*0, \dotsc, \*2^\*l}{\fgintp{\*l}(\gp{\*l,\*i}) = \objfun(\gp{\*l,\*i})},
\end{equation}
where $\interpcoeff{\*l,\*i} \in \real$ and
the sum is over all $\*i = \*0, \dotsc, \*2^\*l$
(i.e., $i_t = 0, \dotsc, 2^{l_t}$, $t = 1, \dotsc, d$).
To ensure that the coefficients $\interpcoeff{\*l,\*i}$
exist for every objective function $\objfun$ and are uniquely determined by
the values at the grid points
\begin{equation}
\fgset{\*l}
\ceq \{\gp{\*l,\*i} \mid \*i = \*0, \dotsc, \*2^{\*l}\},
\end{equation}
we prove the following statement:

\vspace*{\fill}
\pagebreak

\begin{lemma}[linear independence of tensor products]
  \label{lemma:tensorProductLinearIndependence}
  The functions $\basis{\*l,\*i}$ ($\*i = \*0, \dotsc, \*2^\*l$)
  form a basis of $\ns{\*l}$, if the univariate functions
  $\basis{l_t,i_t}$ ($i_t = 0, \dotsc, 2^{l_t}$)
  form a basis of the univariate nodal space $\ns{l_t}$
  for $t = 1, \dotsc, d$.
\end{lemma}
\begin{proof}
  Assume that $\interpcoeff{\*l,\*i} \in \real$ are chosen in \eqref{eq:interpFullGridMV}
  such that $\fgintp{\*l} \equiv 0$.
  Then for all $\*i' = \*0, \dotsc, \*2^\*l$,
  we can evaluate \eqref{eq:interpFullGridMV} at $\gp{\*l,\*i'}$ to obtain
  \begin{equation}
    \sum_{i_1=0}^{2^{l_1}}
    \paren*{
      \sum_{i_2=0}^{2^{l_2}} \dotsb \paren*{
        \sum_{i_d=0}^{2^{l_d}}
        \interpcoeff{\*l,\*i} \basis{l_d,i_d}(\gp{l_d,i_d'})
      } \dotsb \basis{l_2,i_2}(\gp{l_2,i_2'})
    } \basis{l_1,i_1}(\gp{l_1,i_1'})
    = 0.
  \end{equation}
  We apply the univariate linear independence ($x_1$ direction) to infer
  that the sum over $i_2$ must vanish for all $i_1 = 0, \dotsc, 2^{l_1}$.
  Repeating this argument for all dimensions, we have
  $\interpcoeff{\*l,\*i} = 0$ for all~$\*i = \*0, \dotsc, \*2^\*l$,
  implying the linear independence of the functions $\basis{\*l,\*i}$.
\end{proof}

\usenotation{n10}
A common choice for the level $\*l$ is $n \cdot \*1$ for some $n \in \natz$.
\usenotation{Vnd}
In this case, we replace ``$\*l$'' in the subscripts with ``$n{,}d$''
(for example, $\ns{n,d} \ceq \ns{n \cdot \*1}$).
For the hat function basis $\bspl{\*l,\*i}{1}$,
it can be shown that the $\Ltwo$ interpolation error of the interpolant
$\fgintp{n,d} \in \ns{n,d}$ is given by
\begin{equation}
  \normLtwo{\objfun - \fgintp{n,d}} = \landauO{\ms{n}^2},
\end{equation}
i.e., the order of the interpolation error is quadratic in the mesh size
\multicite{Hoellig13Approximation,Bungartz04Sparse}.


%% Draft of probabilistic interpretation of quantization
\section{Probabilistic Interpretation of Quantization}
\begin{frame}{Probability distribution of the output of a quantizer}
\textbf{Question:} what's the probability density function (PDF) $p_{x_Q}(x)$ of the quantizer output $x_Q[n] \sim p_{x_Q}(x)$ when the input signal $x[n]$ has PDF $x[n] \sim p_{x}(x)$?
\begin{center}
	\resizebox{0.75\textwidth}{!}{\input{figs/quantizer_output_pdf.tex}}
\end{center}

\onslide<4|handout:1>{
The PDF of $x_Q[n] \sim p_{x_Q}(x)$ is formed by impulses at the quantization levels. The area of any given impulse is equal to the area under $p_x(x)$ in the quantization interval corresponding to that impulse. In the example of the figure, the area of the impulse at the origin is equal to the highlighted area under $p_x(x)$.
}
\end{frame}

%
\begin{frame}{Area sampling}
We can view $p_{x_Q}(x)$ as samples of the area of $p_x(x)$ under quantization intervals. This form of sampling is called \textbf{area sampling}.

\begin{equation*}
p_{x_Q}(x) = \underbrace{(p_x(x) \ast r(x))}_{\text{\normalsize area calculation}}\overbrace{\cdot s(x)}^{\text{\normalsize sampling}},
\end{equation*}
where
\begin{equation*}
r(x) = \begin{cases}
\frac{1}{\Delta}, & -\frac{\Delta}{2} \leq x \leq \frac{\Delta}{2} \\
0, & \text{otherwise}
\end{cases} \tag{rectangular window of area 1}
\end{equation*}

\begin{equation*}
s(x) = \sum_{n = -\infty}^{\infty} \delta(x - n\Delta) \tag{impulse train}
\end{equation*}
The period of the impulse train (\textit{sampling period}) is $\Delta$

\textbf{Interesting:} although quantization is an nonlinear operation on the signal, it is a linear operation on that signal PDF, since area sampling is linear.
\end{frame}

%
\begin{frame}{Area sampling: graphically}
\begin{center}
	\resizebox{0.9\textwidth}{!}{\input{figs/area_sampling_graphically.tex}}
\end{center}
\end{frame}

%
\begin{frame}{Random variables and convolution}
The area sampling interpretation of quantization gives us a simple equation for the PDF of $x_Q[n]$:
\begin{equation*}
p_{x_Q}(x) = \underbrace{(p_x(x) \ast r(x))}_{\text{\normalsize area calculation}}\overbrace{\cdot s(x)}^{\text{\normalsize sampling}},
\end{equation*}

\pause
\textbf{Another interpretation:} in probability theory, we use convolution to calculate the PDF of the \textbf{sum of two independent random variables}:

\begin{equation*}
Z = X + Y \Longrightarrow p_Z = p_X \ast p_Y
\end{equation*}

\pause
Applying to our problem:

\begin{equation*}
\tilde{x}_Q[n] = x[n] + q[n] \Longrightarrow p_{\tilde{x}_Q}(x) = \underbrace{(p_x(x) \ast r(x))}_{\text{\normalsize area calculation}}
\end{equation*}
where $q[n]$ is a random variable independent of $x[n]$ and with PDF $p_q(x) = r(x)$. Therefore, $q$ is an \textbf{uniform random variable} ($q\sim\mathcal{U}[-\Delta/2, \Delta/2]$).
\end{frame}

%
\begin{frame}
Comparing with the definition of quantization error:
\begin{align*}
x_Q[n] &= x[n] - e[n] \tag{from definition of quantization error} \\
\tilde{x}_Q[n] &= x[n] + q[n] \tag{from probabilistic interpreation of area sampling}
\end{align*}

The equations for $x_Q[n]$ and $\tilde{x}_Q[n]$ are very similar, but there's an important difference. The PDF $p_{x_Q}(x)$ is formed by \textbf{sampling} $p_{\tilde{x}_Q}(x)$.

\pause
\begin{center}
	\resizebox{0.55\textwidth}{!}{\input{figs/sampled_gaussian_pdf.tex}}
\end{center}

\pause
If the sampled PDF ($p_{x_Q}(x)$) perfectly describes $p_{\tilde{x}_Q}(x)$, we can say that quantization error is perfectly described by an uniform random process.

\pause
\textbf{Question:}  under what conditions can we perfectly reconstruct $p_{\tilde{x}_Q}(x)$ from its samples? \\
\end{frame}

%
\begin{frame}{Revisiting the Shannon-Nyquist theorem}
	Recall that the \textbf{Shannon-Nyquist theorem} guarantees that we can perfectly recover a signal from its samples if two conditions are met:
	\begin{enumerate}
		\item \textbf{No aliasing; spectrum replicas do not overlap}
		\item Use ideal reconstruction filter (ideal lowpass filter)
	\end{enumerate}
	\pause
	\begin{block}{Conditions for no aliasing}
		\begin{enumerate}
			\item The \textbf{characteristic function} (Fourier transform) of the area-sampled input PDF must satisfy:
			\begin{equation*}
				\Phi_{p_{x_Q}}(j\theta) = \mathcal{F}\{p_{x_Q}(x)\} = 0, |\theta| > \theta_N
			\end{equation*}
			This is analogous to the condition of a signal being \textit{band-limited}.
			\item\pause The \textit{sampling frequency} $\theta_s$ must be such that $\theta_s > 2\theta_N$.

			In this context, $\theta_s = 2\pi/\Delta$, and the quantization interval $\Delta$ is equivalent to the \textit{sampling period} (period of the impulse train).
		\end{enumerate}
	\end{block}
\end{frame}

%
\begin{frame}<beamer:0|handout:1>
Additional comments
\begin{itemize}
	\item When we used the Shannon-Nyquist theorem for sampling a continuous-time signal, we said that we can use an anti-aliasing filter to minimize aliasing, but we could not use the ideal lowpass filter, as it is unfeasible.
	\item In this discussion on quantization, we just want theoretical assurance that we can reconstruct the PDF from its samples. Therefore, for this purpose, we can use the ideal lowpass filter.
	\item On the other hand, we cannot guarantee that there won't be aliasing, as many probability distributions are not \textit{band-limited}.
\end{itemize}
\end{frame}

%
\begin{frame}
	\textbf{Conclusion:} if the conditions for no aliasing are met, we can perfectly reconstruct the original PDF from its samples, and therefore, quantization noise can be \underline{perfectly modeled} as an uniform random process.

	\pause
	\textbf{Bad news:} the conditions for no aliasing are generally not met in practice. \textbf{Example:} Gaussian distribution. The characteristic function of the Gaussian distribution is a Gaussian function. The Gaussian function is not \textit{band-limited}, as it is always non-zero.

	\pause
	\textbf{How can we mitigate aliasing?}
	\begin{enumerate}
		\item Anti-aliasing filter? This would be equivalent to modify the PDF of the input signal. But what would it do the signal?
		\item Oversampling $\implies$ shorter sampling period ($\Delta$) $\implies$ finer quantization
	\end{enumerate}
\end{frame}

%
\begin{frame}{Example for the Gaussian input distribution}
	\begin{itemize}
		\item A zero-mean Gaussian distributed signal with variance $\sigma^2$ is quantized with quantization resolution \tikz[baseline]{\node[fill=black!10,anchor=base] {$\Delta = 2.1\sigma$};}
		\item Overlap of spectrum replicas results in \underline{significant aliasing}
		\item The estimated quantization noise PDF differs from the uniform distribution
	\end{itemize}

	\begin{columns}[t]
		\begin{column}{0.5\textwidth}
			Fourier transform of sampled PDF
				\begin{center}
					\resizebox{\textwidth}{!}{\PlotGaussianCF{figs/aliased_gaussian_cf.tex}{1}{2.9920}{$\Delta = 2.1\sigma$}}
				\end{center}
		\end{column}
			\begin{column}{0.5\textwidth}
			Estimated quantization noise PDF
			\begin{center}
					\resizebox{0.9\textwidth}{!}{\input{figs/gaussian_quantiz_delta=2sigma.tex}}
			\end{center}
			\end{column}
	\end{columns}
\end{frame}

%
\begin{frame}{Example for the Gaussian input distribution}
\begin{itemize}
	\item A zero-mean Gaussian distributed signal with variance $\sigma^2$ is quantized with quantization resolution \tikz[baseline]{\node[fill=black!10,anchor=base] {$\Delta = \sigma$};}
	\item Spectrum replicas are sufficiently apart; \underline{aliasing is negligible}
	\item The estimated quantization noise PDF agrees well with the uniform distribution
\end{itemize}

\begin{columns}[t]
	\begin{column}{0.5\textwidth}
		Fourier transform of sampled PDF
		\begin{center}
			\resizebox{\textwidth}{!}{\PlotGaussianCF{figs/aliased_gaussian_cf.tex}{1}{6.2832}{$\Delta = \sigma$}}
		\end{center}
	\end{column}
	\begin{column}{0.5\textwidth}
		Estimated quantization noise PDF
		\begin{center}
			\resizebox{0.9\textwidth}{!}{\input{figs/gaussian_quantiz_delta=sigma.tex}}
		\end{center}
	\end{column}
\end{columns}
\end{frame}

%
\begin{frame}{Summary on the probabilistic interpretation of quantization}
	\begin{itemize}
		\item In the probability domain, quantization corresponds to area sampling of the input PDF (a linear operation).
		\item If there's no aliasing (spectrum replicas of area-sampled input PDF do not overlap)
		\begin{enumerate}
			\item Characteristic function of the area-sampled input PDF is \textit{band-limited} with maximum \textit{frequency} $\theta_N$
			\item Sampling period $\Delta$ satisfies $\Delta < \pi/\theta_N$
		\end{enumerate}
		the quantization error is \underline{perfectly modeled} by an uniform random process.
		\item In general aliasing cannot be avoided, but oversampling (finer quantization) makes aliasing negligible and consequently we can still accurately model quantization error as an uniform random process.
	\end{itemize}
\end{frame}


\section{Marciani's Rule}
\label{sec:marciani-rule}

It is well known that Arden's Rule permits the resolution of left-linear and
right-linear language equations \cite{Pettorossi13}.
In particular, by Arden's Rule, given the language equation $r=ar+s$ in the
unknown $r$, its least solution is $a^{*}s$. Likewise, given the language
equation $r=ra+s$ in the unknown $r$, its least solution is $sa^{*}$.

Now, we state and prove a theorem that permits the resolution of both-linear
language equations.

\begin{theorem}
	\label{thm:marciani-rule}

	Given the language equation $r=ar+rb+s$ in the unknown $r$, its least
	solution is $a^{*}sb^{*}$.

	\begin{proof}
		Let us divide the proof in the following three points.

		\begin{enumerate}[label=\alph*]

			\item We will first show that the language equation $r=ar+rb+s$ is
			equivalent to the language equation $r=a^{*}(rb+s)$.

			\item Then, we will show that $a^{*}sb^{*}$ is a solution for $r$ of
			the language equation $r=a^{*}(rb+s)$.

			\item Finally, we will show that $a^{*}sb^{*}$ is the minimal
			solution for $r$ of the language equation $r=a^{*}(rb+s)$; that is,
			for	any other solution $z$ we have that
			$L(a^{*}sb^{*})\subseteq L(z)$.

		\end{enumerate}

		\begin{description}

			\item [Proof (a)] By the application of Arden's Rule on $r=ar+rb+s$, we obtain $r=a^{*}(rb+s)$.
			So $ar+rb+s=a^{*}(rb+s)$\cite{Pettorossi13}.

			\item [Proof (b)] Notice that $a^{*}((a^{*}sb^{*})b+s)=a^{*}a^{*}sb^{*}b+a^{*}s=a^{*}sb^{*}b+a^{*}s$,
			so we have to show that
			(b.1) $a^{*}sb^{*}\subseteq a^{*}sb^{*}b+a^{*}s$ and
			(b.2) $\beta.2$ $a^{*}sb^{*}b+a^{*}s\subseteq a^{*}sb^{*}$.

			\begin{description}

				\item [Proof (b.1)] The following inclusion holds
				$a^{*}sb^{*}=a^{*}s(b^{+}+\varepsilon)\subseteq  a^{*}s(b^{*}b+\varepsilon)=a^{*}sb^{*}b+a^{*}s$.

				\item [Proof (b.2)] The following inclusions hold
				$a^{*}sb^{*}b\subseteq a^{*}sb^{*}$ and $a^{*}s\subseteq a^{*}sb^{*}$.

			\end{description}

			\item [Proof (c)] We assume that $z$ is a solution of $r=a^{*}(rb+s)$, that is $z=a^{*}(zb+s)$, and we show that $a^{*}sb^{*}\subseteq z$,
			that is $\bigcup_{i,j\geq0}a^{i}sb^{j}\subseteq z$. The proof can be done by induction on $i,j\geq0$.

			\begin{description}

				\item [(Basis:$i,j=0$)] $s\subseteq z$ holds because
				$z=a^{*}(zb+s)$.

				\item [(Step:$i\geq0,j=0$)] We have to show the following implication
				$\bigcup_{i\geq0}a^{i}s\subseteq z\rightarrow\bigcup_{i\geq0}a^{i+1}s\subseteq z$.
				This holds because $\bigcup_{i\geq0}a^{i+1}s\subseteq a\bigcup_{i\geq0}a^{i}s\subseteq az\subseteq z$.

				\item [(Step:$i,j\geq0$)] We have to show the following implication
				$\bigcup_{i,j\geq0}a^{i}sb^{j}\subseteq z\rightarrow\bigcup_{i,j\geq0}a^{i}sb^{j+1}\subseteq z$.
				This holds because $\bigcup_{i,j\geq0}a^{i}sb^{j+1}\subseteq\bigcup_{i,j\geq0}a^{i}sb^{j}b\subseteq zb\subseteq z$.

			\end{description}

		\end{description}
	\end{proof}
\end{theorem}


%\{Some Properties of Batch-Normalized Networks}
\section{Some other normalization methods in deep learning}
In this section, we will discuss some other normalization methods in deep neural networks.
We will try to propose a general framework for  these normalization methods which also include the batch normalization.

\subsection{A general framework for normalization in deep neural networks}
Following the idea and notation in the previous section. We consider the data set as
\begin{equation}\label{eq:trainingdata1}
(X,Y) := \{x^i, y^i\}_{i=1}^N.
\end{equation}
For simplicity, here we just focus on the normalization operator during one hidden layer like:
\begin{equation}
\begin{cases}
z &= Wx + b,\\
y &= \sigma(z)
\end{cases}
\end{equation}
where
\begin{equation}
x \in \mathbb{R}^d, \quad W \in \mathbb{R}^{d\times n}
\end{equation}
Based on these setup, we give the next general framework for normalization in deep neural network as
\begin{equation}\label{eq:norm-h}
\begin{cases}
z &= Wx + b,\\
\tilde  z &= h(z),\\
y &= \sigma(\tilde z).
\end{cases}
\end{equation}
Here $h: \mathbb{R}^{n} \mapsto \mathbb{R}^n$ has a general form as
$$
h(z) = \gamma_N \frac{z - \mu_N}{\sigma_N} + \beta_N,
$$
where $\gamma_N, \mu_N, \sigma_N, \beta_N \in \mathbb{R}^n$, and all operation will be applied element-wise.

For the above formulation, we have the next relation immediately.
\begin{itemize}
	\item If $h = {\rm id}$, it recover the original DNN models.
	\item If $h(z) = {\rm BN}_{X}(z)$, then it recovers the BN method with the ``modified" SGD training strategy.
\end{itemize}

\subsection{Some example of normalization in DNN}
Then, we will show that, they are many existing normalization methods which can be formulated with
special choice of $h(\cdot)$.

\paragraph{Layer Normalization (LN)}
The idea behind layer normalization is quite similar to batch normalization.
Batch normalization requires to do average or expectation with respect to the data direction.
For example, in batch normalization we have
$$
\mu_{BN} = \mu_X = \sum_{i=1}^N z(x^i) = \mathbb{E}_{s \sim X}[ z(x)].
$$
What layer normalization does is to take mean, (average, expectation) in the neuron (feature, space) direction.
That is to say, we can take those parameters in $h(\cdot)$ as:
\begin{equation}\label{eq:LN-mu}
\mu_{LN} = \frac{1}{n}\sum_{k=1}^n z_k, \quad \sigma_{LN}=\sqrt{\frac{1}{n}\sum_{k=1}^{n}(z_k-\mu_{LN})}.
\end{equation}
Here $\mu_{LN}  ,  \sigma_{LN} \in \mathbb{R}$, so we may put
$$
\mu_{LN} \bm 1,  \sigma_{LN} \bm 1 \in \mathbb{R}^n,
$$
with
$$
\bm 1 = (1, \cdots, 1) \in \mathbb{R}^n,
$$
for rigorous statement.
With the similar situation in batch normalization, we can take $\gamma_{LN}, \beta_{BN} \in \mathbb{R}^n$. This means that
we can also remove the bias $b$ in original DNN.
Thus to say, we have the layer normalization transformation as:
\begin{equation}\label{eq:LN-h}
h_{LN}(z) = \gamma_{LN} \frac{z - \mu_{LN}\bm 1}{\sigma_{LN} \bm 1} + \beta_{LN}.
\end{equation}
More details of this normalization can be found in \cite{ba2016layer} by G. Hinton for RNN models.

\paragraph{Weight Normalization (WN)}
Weight normalization  normalizes the
row norm of the matrix  $W$ in fully connected layer instead of
the output $z$.
To normalize $W_k$, i.e. keep the 2-norm of $W_k$ to $1$, we may
have a weight normalization transformation as
$$
[{\rm WN}(z(x)) ]_k= \gamma_k \frac{W_k \cdot x}{\|W_k\|_{2}} + b_k.
$$
This is a little different from the $h(\cdot)$ above, but we can still recover this
by choosing those parameters carefully.
Here, in weight normalization, we can take
\begin{equation}
\mu_{WN} = b,\quad  \sigma_{WN} = \left( \|W_1\|_{2}, \cdots \|W_n\|_{2}\right),\quad \beta_{WN} = b,
\end{equation}
with $\gamma_{WN}$ free.
More details can be found in \cite{salimans2016weight}.

\paragraph{Cosine Normalization (CN)}
Cosine normalization normalizes both $W_k$ and the data $z$.
It can be shown as
$$
[{\rm CN}(z(x)) ]_k= \gamma_k \frac{(W_k, b_k) \cdot (x,1) }{\|(W_k,b_k)\|_{2} \|(x,1)\|_{2}}.
$$
We can also recover this by choosing those parameters carefully.
Here, in cosine normalization, we can take
\begin{equation}
\mu_{CN} = 0,\quad  \sigma_{WN} =  \|(x,1)\|_{2}\left(\|(W_1,b_1)\|_{2}, \cdots \|(W_n,b_n)\|_{2}\right),\quad \beta_{WN} = 0,
\end{equation}
with $\gamma_{WN}$ free.
More details can be found in \cite{luo2018cosine}.

\paragraph{Spectral Normalization (SN)}
The original idea in spectral normalization is to control the ``robustness'' of the deep neural networks.
Thus to say, if you have a small perturbation in input data, the output should
also keep stable.
For ReLU DNN, the ``robustness'' means the Lipschitz  continuity because of the regularity of ReLU.

One hidden layer as example,
\begin{equation}
\| \sigma(Wx + b) - \sigma(W(x+\Delta x) + b)\|_2 \le \|W\Delta x\|_2,
\end{equation}
this means that the Lipschitz constant for $\sigma(Wx+b)$ equals to $\|W\|_2 = \sigma (W)$ where
\begin{equation}
\sigma(W)=\max_{x\neq0}\frac{\|Wx\|_2}{\|x\|_2}.
\end{equation}
For ReLU DNN case,
$$
\|f^J(x;\Theta)\|_{\rm Lip} \le \Pi_{\ell=0}^J \sigma(W).
$$
Spectral normalization devotes to normalize the$ \|f^J(x;\Theta)\|_{\rm Lip} $ to be 1.
This can hold if
$$
\sigma (W^\ell) = 1, \quad \forall \ell = 0:J.
$$
So, it is natural to add the next normalization
$$
{\rm SN}(z(x)) = \frac{ Wx }{\sigma(W)} + b.
$$
Then, it can be recovered from $h(\cdot)$ simply by choosing
\begin{equation}
\mu_{CN} = b,\quad  \sigma_{CN} =  \sigma(W),\quad \gamma_{CN} = 1, \quad \beta_{CN} = b.
\end{equation}

\begin{remark}
Generally speaking, $\sigma (W)$ is difficult to take gradient. They use some trick in the paper \cite{miyato2018spectral}.
\end{remark}

%The authors of SN argues that an advantage of smaller $\sigma(W)$ is that $Wx$ is less sensitive to the perturbation $\xi$ on $x$, because
%\begin{equation}
%	\frac{\|(W(x+\xi)+b)-(Wx+b)\|}{\|\xi\|}=\frac{\|W\xi\|}{\|\xi\|}\leq\sigma(W).
%\end{equation}
%$\sigma(W)$ is thus added to the loss function, and gives
%\begin{equation}
%	L_\text{SN}(X)=\frac{1}{m}\sum_{i=1}^m\|f(x_i)-y_i\|_2^2+\frac{\lambda}{2}\sum_{l=1}^{n_L}\sigma(W_l)^2.
%\end{equation}

\paragraph{Group Normalization (GN)}
This method is proposed by Kaiming He,
for the limitation of batch normalization with
\begin{itemize}
	\item If mini-batch is big, computation coast.
	\item if mini-batch is small, result is poor.
\end{itemize}
The idea is that, the feature in channel dimension (neuron dimension) should
group each together if they share similar features. That is to say, for example,
$$
(z_1, z_2, \cdots, z_n) = \left( (z_{1}, \cdots, z_{g}), (z_{g+1}, \cdots, z_{2g}), \cdots, (z_{n-g+1}, z_{n})\right),
$$
and in each group
$$
(z_{kg+1}, \cdots, z_{(k+1)g}), \quad k = 0:\frac{n}{g}-1,
$$
should share the similar distribution which means we should take average just in this group.
Thus to say, we have the next formula for $\mu_{GN}$ and $\sigma_{GN}$,
\begin{equation}\label{eq:GN-mu}
\mu_{GN} = (\mu_1 \bm 1_{g}, \cdots, \mu_{\frac{n}{g}} \bm 1_g), \quad \sigma_{GN} = (\sigma_1 \bm 1_{g}, \cdots, \sigma_{\frac{n}{g}} \bm 1_g),
\end{equation}
with
$$
\mu_{k} = \frac{1}{g}\sum_{i = (k-1)g+1}^{kg} z_i, \quad \sigma_k = \sqrt{\frac{1}{g} \sum_{i = (k-1)g+1}^{kg}(z_i - \mu_k)^2}.
$$

\begin{figure*}[!htb]
	\centering
	\includegraphics[width=.72\linewidth]{GN_all_norms}
	\vspace{.5em}
	\caption{\textbf{Normalization methods}. Each subplot shows a feature map tensor, with $N$ as the batch axis, $C$ as the channel axis, and $(H, W)$ as the spatial axes.
		The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels.
	}
	\label{fig:all_norms}
	\vspace{-.5em}
\end{figure*}
More details can be found in \cite{wu2018group}.

\section{Example: numerical results for different normalizations}
The methods are tested with a two-hidden-layer fully connected neural network, with Adam optimizer of learning rate 0.01.

The two hidden layers are both applied the same normalization methods. The first hidden layer has 128 neurons, and the second 64 neurons.

The dataset is MNIST. The training batch size is 64.

\begin{figure}[H]
	\center
	\includegraphics[width=0.8\textwidth]{Normal_res1.png}
	\caption{Comparison between different normalization methods. I means identity, i.e. without any normalization.}
\end{figure}

\begin{figure}[H]
	\center
	\includegraphics[width=0.8\textwidth]{Normal_res2.png}
	\caption{Use ReLU as activation.}
\end{figure}


\subsection{Mixed regression trees}

In classical trees all items in a leaf are assigned the same values. In this model, all are given \(\theta \) for a parametric model.

This makes the resulting trees smoother.

We have some \(\hat y_i = f(\mathbf x_i, \theta ) + \epsilon \)

The approach generalises classic regression trees. There the estimate was \(\bar y\). Here it's a regression.

\subsubsection{Training}

At each node we do OLS. If the \(R^2\) of the model is less than some constant, we find a split which maximises the minimum of the two new \(R^2\).


\subsection*{Exercise 2 (c)}

Either prove or show a counter example for the following statement. If the same
primes divide $m$ and $n$ then
\begin{equation}
m \phi (n) = n \phi (m)
\end{equation}

%---------------------------------------------------------------------------------------------------
%			SOLUTION 1
%---------------------------------------------------------------------------------------------------
\subsubsection*{Exercise 2 (c) Solution 1}

Let $n=p_1^{a_1}p_2^{a_2}...p_k^{a_k}$ and $m=p_1^{b_1}p_2^{a_2}...p_k^{b_k}$.
Then,
\begin{align}
	n \phi (m) 	&= p_1^{a_1}p_2^{a_2}...p_k^{a_k} \phi ( p_1^{b_1}p_2^{a_2}...p_k^{b_k} ) \\
				&= \Pi p_i^{a_i} \phi ( \Pi p_i^{b_i} )
\end{align}

%% TODO-Ve: finish this solution

%---------------------------------------------------------------------------------------------------
%			SOLUTION 2
%---------------------------------------------------------------------------------------------------
\subsubsection*{Exercise 2 (c) Solution 2}

See Greg Hurst's solutions \cite{hurst14solutions} for a similar approach starting with
\begin{equation}
	\frac{\phi (m)}{m} = \frac{\phi (n)}{n}.
\end{equation}


\section{Artificial Neural Network}
\label{sec:theoretical_background:ann}

An \acrlong{ann} is a model used in \acrshort{ml} which consists of artificial neurons and weighted connections between those neurons.
They usually feature an input layer, several hidden layers and an output layer.
The individual layers consist of an arbitrary number of artificial neurons, which are connected to one another \cite[p.~33-36]{nn}.

The simplest \acrshort{ann} is the feedforward neural network, where the information propagates in only one direction.
A common propagation function is the weighted sum of the neurons from the previous layer.
Therefore, the output of each neuron $i$ is multiplied with the weight $w_{i, j}$ associated to the connection between the two neurons.
These values are summed up and added to a bias term.
In a last step, an activation function is applied, which results in equation \ref{eq:artificial_neuron}.

\begin{equation}
  y_j = a\left(\sum\limits_{i} x_i \cdot w_{i, j} + b\right)
  \label{eq:artificial_neuron}
\end{equation}

where

\begin{tabular}{lll}
  $y_j$ & = & output of neuron $j$ \\
  $a(.)$ & = & activation function \\
  $x_i$ & = & output of neuron $i$ from the previous layer \\
  $w_{i, j}$ & = & weight $i, j$ connecting the output of neuron $i$ with the input of neuron $j$ \\
  $b$ & = & bias \\
\end{tabular}
\\

\subsection{Activation Function}
\label{subsec:theoretical_background:ann:activation_function}

An activation function is responsible for transforming the input of an artificial neuron to its output.
There exists a variety of different linear and non-linear activation functions.
However, non-linear activation functions are preferred due to their ability to learn more complex structures in the data.
The sigmoid and hyperbolic tangent activation functions were traditionally very popular, but they both suffer from a saturation problem.
Currently, the \acrfull{relu} is the most commonly used activation function in the field of deep learning.
It is a piecewise-defined function, which outputs the input if it positive and zero otherwise.
This is shown in equation \ref{eq:relu}.
Furthermore, the \acrshort{relu} activation function requires only a single comparison and is therefore not very computationally expensive \cite{relu}.

\begin{equation}
  a(x) = x^+ = \max(0, x)
  \label{eq:relu}
\end{equation}


\chapter{The Realization of a Vector Space or: Introduction to the Architecture of Spacetime}
Let's review\footnote{At this point, it might be useful to have watched a few of the lessons in the ``What is a Manifold?'' series.} what we have covered so far.
If we have a vector space $V$ over a field $F$, we can find a basis $\{\vec{e}_\mu\}$ for it.
The creation of that vector space automatically implies the creation of the dual space $V^*$ and its basis $\vec{e}^\nu$, chosen so that $\langle \vec{e}^\nu, \vec{e}_\mu \rangle = \delta^\nu_\mu$.
If we promote $V$ to a metric space by the inclusion of an inner product $(\cdot,\cdot)$, we automatically get a new object called the metric tensor, $g_{\mu\nu} \in \tps{T}^0_2$, defined so that
\begin{align*}
    (\vec{v},\vec{w}) = g_{\mu\nu}(\vec{v} \otimes \vec{w}).
\end{align*}
Up until this point, we've only discussed the vector inner product, which acts on elements of $V$ to produce elements of $\mathbb{F}$.
But as we've already shown, $V^*$ is \emph{also} a vector space in its own right, so when we promote $V$ to becoming a metric space, we should promote $V^*$ as well.
The question then becomes ``how do we define the inner product for $V^*$?''
Well, in a similar fashion to the definition of the basis vectors, we'll use $V$ as an inspiration.
In particular, we'll define the inner product on $V^*$ such that
\begin{align*}
    \underbrace{(A_\mu \vec{e}^\mu, B_\nu \vec{e}^\nu)}_{\text{$(\cdot, \cdot)$ on $V^*$}} = \underbrace{(A^\mu \vec{e}_\mu, B^\nu \vec{e}_\nu)}_{\text{$(\cdot, \cdot)$ on $V$}}.
\end{align*}
Just as there is a metric tensor for the vector inner product, we also get to define a metric tensor for the dual space inner product.
As you might guess, we call this rank $(2,0)$ tensor
\begin{align*}
    g^{\mu\nu} \vec{e}_\mu \otimes \vec{e}_\nu &: V \otimes V \to \mathbb{F} \\
               &: \tps{T}^2_0 \to \mathbb{F} \\
               &: \vb*{\alpha} \times \vb*{\beta} \mapsto (\vb*{\alpha},\vb*{\beta}).
\end{align*}
All of this, the basis vectors, the tensor product spaces, the dual space, notions of linear mapping, has been created \emph{de jure}, in a sense, by the creation of a vector space; once we instantiate $V$, everything else pops into existence without any additional work on our part.
If we promote $V$ to become an inner product space, we automatically get the metric tensors $g_{\mu\nu}$ and $g^{\mu\nu}$.
We get this because we've asserted that everything plays nicely; since we've created these vector spaces, \emph{of course} its logical that everything falls into place, because that's how we've defined it.
The objects we've been discussing have been the abstract mathematical ideals of what vectors and covectors and tensors are.
Now, in physics, we're going to look for \emph{realizations} of a vector space; these are specific objects which act as vector spaces (and are vector spaces in the fullest sense of the word), but which aren't generalized to the degree that we've been discussing.
A good example of this is the notion of vectors as arrows with magnitude and direction; although that's not what we defined a vector to be in this lesson, they certainly act as vectors in a vector space, since they can be added, have inverses and a zero vector, and are closed under these operations.
Likewise, the physical notions which we ascribe to these vectors, like forces and acceleration, also form realizations of a vector space.

\section{The Differential Operator}
Let's imagine a function $f$ in four variables, denoted $x_0$ through $x^3$, so
\[ f(x^0,x^1,x^2,x^3) = f(x^\mu) = f(x), \]
where we understand that $x = x^\mu$.
These elements will be known as \emph{coordinates}.
We say that $f$ is differentiable if we can write
\begin{align*}
    \bm{\partial}_\mu f \equiv \pdv{f}{x^\mu}
\end{align*}
where $\bm{\partial}_\mu f$ is a symbolic equivalent to the partial derivative of $f$ with respect to $x^\mu$.
As a quick side note, we choose this notation so that we can contract over it just like with tensors; if you think of $x^\mu$ as the basis and $\bm{\partial}_\mu$ as the component, then something like $\qty(\partial_\mu f)A^\mu$ could be summed over with Einstein summation.
Because derivatives are linear operators, we can write that
\[ \lambda\bm{\partial}_\mu f + \lambda\bm{\partial}_\nu f = \lambda\qty(\bm{\partial}_\mu + \bm{\partial}_\nu) f, \]
where $\lambda \in \mathbb{F}$ and $\qty(\bm{\partial}_\mu + \bm{\partial}_\nu)$ is in effect a new differential operator.
If we wanted to, we could create an operator $\mathcal{L} = a\bm{\partial}_0 + b\bm{\partial}_1 + c\bm{\partial}_2 + bm{\partial}_4$, so $\mathcal{L}$ is a linear combination of our original four partial differential operators.
Because of the linearity of these differential operators, if we created a set $\{\mathcal{L}\}$ of all possible differential (and therefore linear) operators on a function $f$ of four variables $x^\mu$, this set would be a vector space, with a basis $\{\bm{\partial}_\mu\}$.
Let's stipulate that $\lambda$ are all elements of $\mathbb{R}$; then we've just created a realization of a four dimensional, real vector space.
One interesting thing of note is that $f$ itself is kind of irrelevant here; it doesn't matter if $f$ is a linear function or not, since the differential operator $\bm{\partial}$ is.
We care about these operators\footnote{Notice that we've been writing $\bm{\partial}_\mu$ in \emph{bold}, to emphasize that the differential operators are ... you guessed it, vectors!} since they are tools which allow us to create a model of spacetime.

\section{Spacetime}
Throughout the rest of this lesson, we'll be using the following figure as a graphic model for spacetime.
This is a canonical way to represent points in spacetime, such as $P, Q, R, S$, in reference to coordinate lines, like $x^1$ and $x^2$.
\begin{figure}[h]
    \centering
    \begin{tikzpicture}
        \node (origin) at (0,0) {}; % Origin node, not shown

        %% These are the four corners of the "square"
        \node (a) at (0,0) {};
        \node (b) at (1,-0.5) {};
        \node (c) at (5.5,1) {};
        \node (d) at (4.5,0.5) {};
        \node (e) at (5,3.5) {};
        \node (f) at (4,4) {};
        \node (g) at (1,3.5) {};
        \node (h) at (0,3) {};

        %% Curves
        \draw [name path=line-1] (a) to [bend left=10] (f);
        \draw [name path=line-2] (b) to [bend left=10] (e);
        \draw [name path=line-3] (g) to [bend left=10] (c);
        \draw [name path=line-4] (h) to [bend left=10] (d);

        %% Points
        \path [name intersections={of=line-1 and line-3,by=P}];
        \node [circle, fill=black,inner sep=1.5pt,label=90:$P$] at (P) {};

        \path [name intersections={of=line-2 and line-3,by=Q}];
        \node [circle, fill=black,inner sep=1.5pt,label=0:$Q$] at (Q) {};

        \path [name intersections={of=line-2 and line-4,by=R}];
        \node [circle, fill=black,inner sep=1.5pt,label=-90:$R$] at (R) {};

        \path [name intersections={of=line-1 and line-4,by=S}];
        \node [circle, fill=black,inner sep=1.5pt,label={[shift={(-0.35,-0.4)}]$S$}] at (S) {};

        %% Line labels
        \node [label={[shift={(0.55,-0.5)}]$x^1$}] at (a) {};
        \node [label={[shift={(0.6,0)}]$x^2$}] at (h) {};
    \end{tikzpicture}
    \caption{Labeled diagram of our spacetime $\mathcal{S}$.}
    \label{fig:spacetime_plain}
\end{figure}
\noindent
We can think of the coordinate lines as being like latitude and longitude, or as $(t,x,y,z)$ coordinates.
In the diagram shown above, we're only depicting two coordinate lines, leaving the $x^0$ and $x^3$ coordinates of the points implicit.
Note that in reality, we would be giving names to an infinite number of points, not just the four shown here.
For each of the points shown, there are an associated set of coordinates.
If we take point $Q$ as an example, we might say that $Q$ has coordinates $(x^0,x^1,x^2,x^3)$, or more succinctly that $Q$ has coordinates\footnote{Without rigorously defining it at this point, it's good to recall that we usually think of $x^0$ as being time, while $x^1$ through $x^3$ represent various spatial coordinates. This corresponds nicely with the Minkowsky metric signature.} $x^\mu$.
Note that in this case, this doesn't mean that $x^\mu$ is a member of a vector space, since we haven't talked about what the basis vectors would be; we're just using the superscript notation as a shorthand for each of the separate coordinates.
It's also important to note that we're not talking about any particular coordinate systems here, just a generalized one; it could be spherical or Cartesian or what-not, but for now we really don't care.

Once we've defined a coordinate system, we can start to think of functions on those coordinates.
We could imagine such a function being given by
\[ f(x^\mu) = \mathrm{e}^{x^0}\sin x^1 + \frac{(3x^2)^3}{x^3}. \]
We call any such function a \emph{function on the spacetime}.

\section{Integrating Spacetime and Vector Spaces}
In the past, we've talked about an abstract vector space $V$ with a basis $\{\vec{e}_\mu\}$.
When discussing spacetime, we'll consider a realization of this vector space;
this realized vector space $V$ will be a four-dimensional, real vector space with basis $\{\bm{\partial}_\mu\}$.
Importantly, we won't be creating just one of these vector spaces.
We will create a separate vector space \emph{for each point in spacetime}.
We'll label the vector space by the point it's attached to, so the vector space at $Q$ might be $V_P$.
Because we're creating vector spaces at each point, we also get for free the dual space $V_P^*$ and the infinite number of tensor product spaces associated with $V_P$ and $V^*_P$.


\subsection{Limit of a sequence}

A sequence converges to a limit if

Can converge to a number (1/x)

Can converge to +/- infinity (x)

Otherwise, does not converge (1,-1,1,-1…)

Superior and inferior limits

A bounded increasing sequence converges to least upper bound

\subsubsection{Identifying the limit of a sequence}

Direct comparison test

Root test


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Disks and Washers}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises for the disk and washer methods.
\end{abstract}
\maketitle

\begin{exercise}
The region $0 \leq y \leq \sqrt{x}$ with $x \leq 1$, shown below, is revolved around the $x$-axis. Use the disk method to find the volume of the solid of revolution.
\begin{center}
\begin{image}
\includegraphics[width=4in]{diskwasher/disk01.png}
\end{image}
\end{center}
\begin{hint}
The radius $R(x)$ will be a difference of $y$-values because slices are indexed by the variable $x$.
Each slice will extend from $y=0$ to $y =  \sqrt{x}$, and so $R(x)$ must be the larger of these $y$-values minus the smaller of these $y$-values.
\end{hint}
\begin{prompt}
\[ R(x) = \answer{\sqrt{x}} \]
\[ V = \int_{\answer{0}}^{\answer{1}} \pi (R(x))^2 dx =  \answer{\frac{\pi}{2}} \]
\end{prompt}
\end{exercise}

\begin{exercise}
The region $0 \leq y \leq \sqrt{x}$ with $x \leq 1$, shown below, is revolved around the axis $x=1$. Use the disk method to find the volume of the solid of revolution.
\begin{center}
\begin{image}
\includegraphics[width=4in]{diskwasher/disk02.png}
\end{image}
\end{center}
\begin{hint}
The radius $R(y)$ will be a difference of $x$-values because slices are indexed by the variable $y$.
Each slice will extend from $x = y^2$ to $x = 1$, and so $R(y)$ must be the larger of these $x$-values minus the smaller of these $x$-values
\end{hint}
\begin{prompt}
\[ R(y) = \answer{1 - y^2} \]
\[ V = \int_{\answer{0}}^{\answer{1}} \pi (R(y))^2 dy = \answer{\frac{8\pi}{15}} \]
\end{prompt}
\end{exercise}

\begin{exercise}
The region $0 \leq y \leq \sqrt{x}$ with $x \leq 1$, shown below, is revolved around the axis $x=0$. Use the washer method to find the volume of the solid of revolution.
\begin{center}
\begin{image}
\includegraphics[width=4in]{diskwasher/disk04.png}
\end{image}
\end{center}
\begin{hint}
Each radius will be a difference of $x$-values because slices are indexed by the variable $y$.
The distance from the axis $x=0$ to the line $x=1$ is $1$, and the distance from the axis $x=0$ to $x = y^2$ is $y^2$.
\end{hint}
\begin{prompt}
\[ R_{\mathrm{outer}} (y) = \answer{1} \text{ and } r_{\mathrm{inner}}(y) = \answer{y^2} \]
\[ V = \int_{\answer{0}}^{\answer{1}} \pi  \left[ (R_{\mathrm{outer}}(y))^2 - (r_{\mathrm{inner}}(y))^2 \right] dy =  \answer{\frac{4\pi}{5}} \]
\end{prompt}

\end{exercise}

\begin{exercise}
The region $0 \leq y \leq \sqrt{x}$ with $x \leq 1$, shown below, is revolved around the axis $y=1$. Use the washer method to find the volume of the solid of revolution.
\begin{center}
\begin{image}
\includegraphics[width=4in]{diskwasher/disk03.png}
\end{image}
\end{center}
\begin{hint}
Each radius will be a difference of $y$-values because slices are indexed by the variable $x$.
The distance from the axis $y=1$ to the line $y=0$ is $1$, and the distance from the axis $y=1$ to $y = \sqrt{x}$ is $1 - \sqrt{x}$.
\end{hint}
\begin{prompt}
\[ R_{\mathrm{outer}} (x) = \answer{1} \text{ and } r_{\mathrm{inner}}(x) = \answer{1-\sqrt{x}} \]
\[ V = \int_{\answer{0}}^{\answer{1}} \pi  \left[ (R_{\mathrm{outer}}(x))^2 - (r_{\mathrm{inner}}(x))^2 \right] dx =  \answer{\frac{5\pi}{6}} \]
\end{prompt}
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[WasherQuad001]

The region in the plane bounded on the left by the curve \(x=-y^2\), on the right by the curve \(x=y^2+2y+2\), above by the line  \(y = 0\), and below by the line \(y = -2\) is revolved around the axis \(x = 2\). Compute the volume of the resulting solid.
(Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice{\(16\pi\)}
\choice{\(20\pi\)}
\choice[correct]{\(24\pi\)}
\choice{\(28\pi\)}
\choice{\(32\pi\)}
\choice{\(36\pi\)}
\end{multiplechoice}
\begin{feedback}
The axis \(x = 2\) is perpendicular to the direction of slices using the integration variable \(y\), which indicates the washer method.
 The region lies to the left of the axis. One way to see this is to evaluate \(x=-y^2\) at \(y = -2\), giving \(x = -4\), which is to the left of the axis \(x = 2\).
 \begin{hint}
The integral to compute equals \[ \begin{aligned} V &= \int_{-2}^{0}\pi \left((2-(-y^2))^2 - (2-(y^2+2y+2))^2\right)~ dy\\
& = \pi \int_{-2}^{0} (-4y^3+4)~ dy\\
& = \pi \left. \left(-y^4+4y\right) \right|_{-2}^{0} = 24\pi. \end{aligned}\]
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[WasherQuad008]

The region in the plane bounded below by the curve \(y=-2x^2+5x+2\), above by the curve \(y=-2x^2+2x+2\), on the right by the line  \(x = 0\), and on the left by the line \(x = -1\) is revolved around the axis \(y = 2\). Compute the volume of the resulting solid.
(Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(10\pi\)}
\choice{\(14\pi\)}
\choice{\(18\pi\)}
\choice{\(22\pi\)}
\choice{\(26\pi\)}
\choice{\(30\pi\)}
\end{multiplechoice}
\begin{feedback}
The axis \(y = 2\) is perpendicular to the direction of slices using the integration variable \(x\), which indicates the washer method.
 The region lies below the axis. One way to see this is to evaluate \(y=-2x^2+5x+2\) at \(x = -1\), giving \(y = -5\), which is below the axis \(y = 2\).
 \begin{hint}
The integral to compute equals \[ \begin{aligned} V &= \int_{-1}^{0}\pi \left((2-(-2x^2+5x+2))^2 - (2-(-2x^2+2x+2))^2\right)~ dx\\
& = \pi \int_{-1}^{0} (-12x^3+21x^2)~ dx\\
& = \pi \left. \left(-3x^4+7x^3\right) \right|_{-1}^{0} = 10\pi. \end{aligned}\]
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[washersqrtsub001]

The region in the plane given by \(\left|{- \frac{x}{2} + \frac{1}{2} \sqrt{9 - 6 x^{2}}}\right| \leq y \leq \frac{x}{2} + \frac{1}{2} \sqrt{9 - 6 x^{2}}\) and \(0 \leq x \leq \frac{2}{3} \sqrt{3}\) is revolved around the \(x\)-axis. Compute the volume of the resulting solid.
(Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\displaystyle \frac{13}{9} \pi\)}
\choice{\(\displaystyle \frac{19}{9} \pi\)}
\choice{\(\displaystyle \frac{26}{9} \pi\)}
\choice{\(\displaystyle \frac{28}{9} \pi\)}
\choice{\(\displaystyle \frac{37}{9} \pi\)}
\choice{\(\displaystyle \frac{49}{9} \pi\)}
\end{multiplechoice}
\begin{feedback}
If the variable \(x\) is used for slicing, then slices are perpendicular to the axis of rotation, which indicates the washer method should be used.
\begin{hint}
The inequalities for \(y\) give the outer and inner radii, and \[ \left(\frac{x}{2} + \frac{1}{2} \sqrt{9 - 6 x^{2}}\right)^2 - \left( \left|{- \frac{x}{2} + \frac{1}{2} \sqrt{9 - 6 x^{2}}}\right|\right)^2 = x \sqrt{9 - 6 x^{2}}. \](Note that the absolute values go away when the radius is squared.)
\begin{hint}
To compute the integral
\[ \int_{0}^{\frac{2}{3} \sqrt{3}} \pi x \sqrt{9 - 6 x^{2}}\, dx \]
 we can use the substitution \(u = 9 - 6 x^{2}\) which implies the equality \(du = \left(- 12 x\right)dx\) for the differentials. This gives the equality
\[ \begin{aligned} \int \pi x \sqrt{9 - 6 x^{2}}\, dx & = \int \left(- \frac{\pi}{12} \sqrt{u}\right)\, du \\
 & = - \frac{\pi}{18} u^{\frac{3}{2}}. \end{aligned} \]
Reversing the substitution gives
\[ \begin{aligned} \int_{0}^{\frac{2}{3} \sqrt{3}} \pi x \sqrt{9 - 6 x^{2}}\, dx & = \left. \left[- \frac{\pi}{18} \left(9 - 6 x^{2}\right)^{\frac{3}{2}} \right] \right|_{0}^{\frac{2}{3} \sqrt{3}}\\ & = \left(- \frac{\pi}{18} \right) - \left(- \frac{3}{2} \pi \right) = \frac{13}{9} \pi. \end{aligned} \]
\end{hint}
\end{hint}
\end{feedback}

\end{question}

\end{document}