\lab{Wavelet Denoising and Compression}{Denoising and Compression}

\objective{This lab presents applications of the Discrete Wavelet Transform in
image denoising and compression.}

\section*{The PyWavelets Module}
PyWavelets is a Python library for Wavelet Analysis. It provides convenient and
efficient methods to calculate the one and two-dimensional discrete Wavelet
transform, as well as much more. Assuming that the package has been installed on
your machine, type the following to get started:
\begin{lstlisting}
>>> import pywt
\end{lstlisting}
Performing the basic discrete Wavelet transform is very simple.
Below, we compute the one-dimensional transform for a sinusoidal signal.
\begin{lstlisting}
>>> import numpy as np
>>> f = np.sin(np.linspace(0,8*np.pi, 256))
>>> fw = pywt.wavedec(f, 'haar')
\end{lstlisting}
The variable \li{fw} is now a list of arrays, starting with the final approximation
frame, followed by the various levels of detail coefficients, just like the output
of the wavelet transform function that you coded in the previous lab.
Plot the level 2 detail and verify that it resembles a blocky sinusoid.
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> plt.plot(fw[-2], linestyle='steps')
>>> plt.show()
\end{lstlisting}
We can give alter the arguments to the \li{wavedec} function to use different
wavelets or obtain different levels of the wavelet transform. The second
positional argument, as you will notice, is a string that gives the name of the
wavelet to be used. We first used the Haar wavelet, with which you are already
familiar. PyWavelets supports a number of different Wavelets, however, which you can
list by executing the following code:
\begin{lstlisting}
>>> # list the available Wavelet families
>>> print pywt.families()
['haar', 'db', 'sym', 'coif', 'bior', 'rbio', 'dmey']
>>> # list the available wavelets in the coif family
>>> print pywt.wavelist('coif')
['coif1', 'coif2', 'coif3', 'coif4', 'coif5']
\end{lstlisting}
We can also include optional arguments \li{mode} and \li{level} when calling the
\li{wavedec} function. Using these arguments, you can adjust the mode for dealing
with border distortion and the level of the Wavelet decomposition, respectively.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{dwt2.pdf}
    \caption{Level 1 Wavelet decomposition of the Lena image.
    The upper left is the approximation frame, and the remaining
    plots are the detail coefficients.}
    \label{fig:dwt2}
\end{figure}

Now we illustrate how to perform a two-dimensional Wavelet transform using
PyWavelets. We will work with the traditional Lena image, performing a
two level wavelet transform using the Daubechies 4 Wavelet.
\begin{lstlisting}
>>> import scipy.misc
>>> lena = scipy.misc.lena()
>>> lw = pywt.wavedec2(lena, 'db4', level=2)
\end{lstlisting}
The variable \li{lw} is a list of tuples of arrays. The first entry of the list is
simply the level 2 approximation frame. The second entry of the list is a tuple of
the level 2 detail coefficients $LH$, $HL$, and $HH$ (in that order). The remaining
entries of the list are tuples containing the lower level detail coefficients.
Thus, to plot the level 1 $HL$ detail coefficients, we can execute the following code:
\begin{lstlisting}
>>> HL1 = lw[-1][1]
>>> plt.imshow(np.abs(HL1), cmap=plt.cm.Greys_r, interpolation='none')
>>> plt.show()
\end{lstlisting}
The output of this code should be a plot resembling the lower left plot given in Figure
\ref{fig:dwt2}.

We have only introduced a couple of the basic tools available in PyWavelets. There
are of course many more functions and methods that facilitate a more comprehensive
Wavelet analysis. In the remainder of this lab, we will explore three particular
applications of Wavelet analysis in the realm of image processing and compression.

%\section*{Edge Detection}
%It is often useful to identify the edges of objects and figures
%represented in images. The edge information can be used to classify images
%and group them with other similar images (this is part of a field called
%\textit{computer vision}), to segment the image into component parts, to
%sharpen blurry images, to filter out unnecessary details of the image,
%and so forth. Of course, our human eyes are very adept at recognizing edges,
%but enabling a computer to do the same is much more difficult. An edge can
%be thought of as a discontinuity in the image or a region of high contrast
%in either color or brightness. We can therefore leverage the high-frequency
%detail coefficients of the wavelet transform to detect the edges. Execute the
%following code:
%\begin{lstlisting}
%>>> # calculate one level of wavelet coefficients
%>>> coeffs = pywt.wavedec2(lena,'haar', level=1)
%\end{lstlisting}
%
%Note that the approximation coefficients are very close to the original
%image, while the detail coefficients are much more sparse, and roughly
%capture the edges in the image. In particular, the upper right coefficients
%emphasize the vertical edges, the lower left coefficients emphasize the
%horizontal edges, and the lower right coefficients emphasize the diagonal
%edges.
%
%\begin{problem}
%Now zero out the approximation coefficients and use your inverse DWT
%function to recreate the image. Plot its absolute value. This image is
%a fairly good representation of the edges. If we add this to the original
%image, we can increase the contrast at the edges (that is, make the dark
%side darker, and the light side lighter). Do this, and plot the original
%image side-by-side with the sharpened image. What do you notice? There
%are many image-sharpening techniques, and those based on wavelets
%are more sophisticated than what we have done here, but this gives the
%basic idea.
%\end{problem}
%the above section needs work, or maybe should be taken out completely.

\section*{Noise Removal}
Noise in an image can be defined as unwanted visual artifacts that
obscure the true image. Images can acquire noise from a variety of
sources, including the camera, transmission, and image processing
algorithms. Noise can be completely random and incoherent (as in
Figure \ref{fig:incoherent}), or it can be coherent and display
visual patterns (Figure \ref{fig:coherent}). In this section, we will
focus on reducing a particular type of random noise in images, called
\textit{Gaussian white noise}.

\begin{figure}[t]
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{phantom_random.pdf}
    \caption{The Phantom image with incoherent noise}
    \label{fig:incoherent}
\endminipage\hfill
\minipage{0.49\textwidth}
    \includegraphics[width=\linewidth]{phantom_coherent.pdf}
    \caption{The Phantom image with coherent noise}
    \label{fig:coherent}
\endminipage
\end{figure}

An image that is distorted by Gaussian white noise is one in which
every pixel has been perturbed by a small amount, such that the
perturbations are normally distributed. We can easily add such noise
to an image using the \li{np.random.normal} function.

\begin{lstlisting}
>>> noisyLena = lena + np.random.normal(scale=20, size=lena.shape)
>>> plt.imshow(noisyLena, cmap=plt.cm.Greys_r)
>>> plt.show()
\end{lstlisting}

Given an image with Gaussian white noise, how do we go about reducing
the noise level? Our approach will be based on the idea of thresholding.
It turns out that images are often sparse in the wavelet basis,
particularly in the high-frequency details. The Gaussian noise, however,
is very high frequency, and thus its wavelet transform will be
concentrated in high-frequency wavelet coefficients (of magnitude
roughly proportional to the variance of the noise). We can therefore
reduce the noise while preserving the true image by shrinking the
detail coefficients via hard or soft thresholding.

Given a positive threshold value $\tau$, hard thresholding sets
every wavelet coefficient whose magnitude is less than $\tau$ to
zero, while leaving the remaining coefficients untouched. Soft
thresholding also zeros out all coefficients of magnitude less than
$\tau$, but in addition maps every other coefficient $\beta$ to
$\beta - \tau$ if $\beta > 0$ or $\beta + \tau$ if $\beta < 0$.

Implementing these simple thresholding algorithms in Python is
straight-forward, but PyWavelets already provides this functionality.
The following code gives an example.

\begin{lstlisting}
>>> A = np.arange(-4,5).reshape(3,3)
>>> A
array([[-4, -3, -2],
       [-1,  0,  1],
       [ 2,  3,  4]])
>>> pywt.thresholding.hard(A,1.5)
array([[-4, -3, -2],
       [ 0,  0,  0],
       [ 2,  3,  4]])
>>> pywt.thresholding.soft(A,1.5)
array([[-2.5, -1.5, -0.5],
       [ 0. ,  0. ,  0. ],
       [ 0.5,  1.5,  2.5]])
\end{lstlisting}

Once the coefficients have been thresholded, we take the inverse
wavelet transform to recover the denoised image. This can be done
by calling the \li{waverec2} function, providing the list of Wavelet
coefficients as well as the name of the desired Wavelet as arguments.
The threshold value is generally a function of the variance of the noise,
and in real situations, we do not know what this variance is. In fact,
noise variance estimation in images is a research area in its own
right, but this goes beyond the scope of this lab, and so we will
assume that we already have a decent estimate of the variance.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{denoise.pdf}
    \caption{Noisy Lena (left), denoised using hard thresholding (center),
    and denoised using soft thresholding (right).}
    \label{fig:denoise}
\end{figure}

\begin{problem}
Write functions that implement the hard and soft thresholding
techniques. The inputs should be a list of wavelet coefficients
in the usual form, as well as the threshold value. The output
should be the thresholded wavelet coefficients (also in
the usual form). Remember that we only want to threshold the
detail coefficients, and not the approximation coefficients.
You should therefore leave the first entry of the input
coefficient list unchanged.
\end{problem}

\begin{problem}
Create a noisy version of the Lena image by adding Gaussian
white noise of mean 0 and standard deviation $\sigma = 20$ (i.e. \li{scale=20}).
Compute four levels of the wavelet coefficients using the Daubechies 4 Wavelet,
and input these into your
thresholding functions (with $\tau = 3\sigma$ for the hard threshold,
and $\tau = 3\sigma/2$ for the soft threshold). Reconstruct the
two denoised images, and then plot these together alongside the
noisy image. Your output should match Figure \ref{fig:denoise}.

What do you notice? How does lowering or raising the
threshold affect the reconstructed images? What happens if you use
a different Wavelet?
\end{problem}

\section*{Image Compression}
We now turn to the problem of image compression. Explicitly saving
the value of every pixel in an image can be very costly in both
storage and transmission, and numerous image compression techniques
have been developed over the years to deal with this problem.
Transform methods have long played an important role in these
techniques; the popular JPEG image compression standard is based on
the discrete cosine transform. Starting from the early 1990's, much
research has gone into compression methods using the discrete wavelet
transform, and to great success. The JPEG2000 compression standard
and the FBI Fingerprint Image database, along with other systems,
take the wavelet approach.

The general framework for compression is fairly straightforward. First,
the image to be compressed undergoes some form of preprocessing (this
can include subtracting out its mean, tiling the image, or perhaps
nothing at all). Next, the wavelet coefficients are computed using some
specially constructed wavelet (JPEG2000 uses the either the
Cohen-Daubechies-Feauveau 9/7 or 5/3 wavelet) and then \textit{quantized},
 a process that we will explain shortly. The quantized coefficients are
 then grouped in a particular way and passed through an entropy encoder
 (such as Huffman coding, run length coding, or arithmetic coding). This
 coding step comes from the realm of information theory, and we will not
 worry about it in this lab. What you have left is a compact stream of bits
 that can then be saved or transmitted much more efficiently than the
 original image. All of the above steps are invertible, allowing us to
 reconstruct the image from the bitstream.

 The step in this process that we will focus on is quantization. Put simply,
 quantization is a process whereby the coefficients are converted into
 integers. If the coefficients are floating-point numbers, then this
 process introduces some loss of precision, and we call this \textit{
 lossy compression}. In the situation where all the coefficients are already
 integers, it is possible to compress the image without any loss of precision,
 and this is called \textit{lossless compression}. Quantization can be
 performed in a variety of ways, and we will explore one particular method
 called a \emph{uniform null-zone quantizer}. Given a coefficient $x$, we assign
 to it an integer $q$ given by
\begin{equation*}
q =
 \begin{cases}
   \lceil x / \delta - t/2 \rceil, &  x \geq 0\\
   \lfloor x / \delta + t/2 \rfloor & x \leq 0
 \end{cases}
\end{equation*}
where $1 \leq t \leq 2$ and $\delta > 0$ are adjustable parameters.
The inverse process, called de-quantization, consists of recovering
the coefficient $y$ from the quantized value $q$ via the equation
\begin{equation*}
 y =
  \begin{cases}
   (q - 1/2 + t/2)\delta & q > 0\\
   (q + 1/2 - t/2)\delta & q < 0\\
   0,                    & q = 0
  \end{cases}
 \end{equation*}
What we are essentially doing is mapping all wavelet coefficients that
fall in the interval $[-\delta,\delta]$ to 0 and all wavelet coefficients
in the interval $[j\delta,(j+1)\delta]$ to $(2j+1)\delta/2$ for integers
$j \geq 1$ and $j \leq -2$. This greatly reduces the number of distinct
coefficient values (indeed, most of the coefficients are mapped to
zero), allowing us, at the cost of some precision, to store less
information. The larger we choose $\delta$, the more compression we
can achieve, albeit with a correspondingly larger loss in precision.
\begin{problem}
Write function \li{quantize} and \li{dequantize} based on the discussion above.
In both cases, the inputs should be a list of wavelet coefficients in
standard form, the $\delta$ parameter, and the $t$ parameter with default
value of 2. The functions should return the quantized list of wavelet
coefficients.

For the Lena image, calculate the complete set of wavelet coefficients and then
quantize and de-quantize the coefficients, and reconstruct the image.
Do this for a few different values of $\delta$ (with $t=2$), and observe how
the image is distorted. Try using different Wavelets. Keep in mind that there are
specially designed Wavelets used in image compression that are not available
in PyWavelets, so distortion will be greater than tolerable in real-life settings.
\end{problem}


\documentclass[12pt,a4paper,notitlepage]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsfonts}
\usepackage{amsmath}
\input magstyle.sty

% Title Page
\title{Calculation of Demagnetisation Field using domain truncation}
\author{Gabriel Balaban for the FinMag project at the University of Southampton}
\begin{document}
\maketitle
\abstract{}
In this document the mathematics of the python modules \texttt{prob\_testcases.py} and \texttt{solver\_nitsche.py} are explained.
The demagnetisation field is found using an approach that truncates an infinite domain and uses two scalar potentials.

\newpage
\section{The Demagnetisation Field inside a Magnetic Body}
The demagnetisation field $\dmag$ related to a magnetic body is generated by its magnetisation field $\magn$ ,
 and tends to oppose it.In this context the region in space occupied by the magnetic object of interest
(usually a ferromagnetic material) is denoted by  $\omM$.
The complement of this region $\omV$ is considered to be a vacuum, which extends to infinity and over which
the demagnetisation field
$\dmag $ is also defined.
According to basic electrodynamics the two fields are related by

\begin{equation}\label{dmagdef} \nabla \cdotp \dmag = - \nabla \cdotp \magn \  \mbox{in} \ \omM \end{equation}

\noindent The assumption is also made that there are no free currents and that the electric displacement field does not change over time so that
\begin{equation}\label{nocurl}  \nabla \times \dmag = 0\end{equation}

\noindent This means that we can introduce a magnetic potential function $\phi$, such that

\[ \dmag = - \nabla \phi \  \mbox{in} \ \ \mathbb{R}^3 \]

\noindent which means that

\[ - \nabla^2 \phi = -\nabla \cdotp \magn \ \ \mbox{in} \ \omM \]

\noindent so that our demagnetisation field can be obtained by solving a Poisson problem.

\section{Poisson Problem for the Scalar Potential}
The poisson problem for the scalar potential $\phi$ reads
\begin{equation}\label{phidef}
\left\{
\begin{array}{lr}
- \nabla^2 \phi = - \nabla \cdotp \magn  & \mbox{in } \ \omM \\
- \nabla^2 \phi = 0 & \mbox{in } \ \omV
\end{array}
\right.
\end{equation}
\noindent with corresponding boundary conditions

\begin{subequations}
\begin{align}
&\ \left[ \frac{\partial \phi}{\partial n} \right] = n \cdotp \magn  &\mbox{ on }  \domM \label{jumpbc} \\
&\lim_{\mvec{r} \in \omM \rightarrow \domM} \phi =  \lim_{\mvec{r} \in \omV \rightarrow \domM}
\phi & \mbox{on } \domM \label{contbc} \\
&\phi \rightarrow 0 \ & \mbox{as} \ |\rvec| \in \omV \rightarrow \infty \label{openbc}
\end{align}
\end{subequations}

\noindent Where $n$ denotes the unit outward normal. The prescribed jump in the normal derivative (denoted by $[ ]$) is a
consequence of the divergence theorem applied to the definition of $\dmag$, equation~(\ref{dmagdef}).
Using equation~(\ref{nocurl}) and Stokes' theorem we see that the tangential component of $\nabla \phi$ or $\dmag$ is
continuous over $\domM$. Which means that the scalar potential only differs by a constant as it crosses $\domM$.
This constant is set to 0, which means that $\phi$ is continuous over $\domM$. Finally we expect the demag field $\dmag$
to decay to 0 far away from the magnetic body, which results in the open boundary condition
$| \phi | \rightarrow 0  \ \mbox{as} \ |\rvec| \in \omV \rightarrow \infty$ for the potential $\phi$.

\section{Truncation of the Domain}
Since a computer cannot simulate an infinite domain there needs to be some way of dealing with the open boundary condition from
\ref{openbc}
\[\phi_2 \rightarrow 0 \ \mbox{as} \ \mvec{r} \in \omV \rightarrow \infty \]
Here the domain $\omV$ is truncated, using the rule of thumb that it should be about 5 times as large as $\omM$ in order to get
decent results.

\section{The Discontinuous Galerkin formulation for $\phi$}\label{dgsection}
The discontinuous Galerkin formulation is a finite element discretization scheme where the
solution is allowed to be discontinuous across element boundaries. The solution space for
$\phi$ is restricted to
\begin{equation}\label{dgdef}
 V = \{ v \in H^1 (T) \mbox{ for all } T \subset \Omega \mbox{ s.t. v is discontinous over } \ \partial T \}
\end{equation}

\noindent Where $\Omega$ denotes the mesh and $T$ a finite element domain in $\Omega$. More detail regarding this
definition is give in appendix \ref{appdgspace}. After integrating
equation~(\ref{phidef}) by parts we obtain the formulation.
\\
\\
Find $\phi \in V$ such that
\begin{equation}\label{dgform}
 \sum_T \int_T \nabla \phi \cdotp \nabla v \ dx - \int_{\partial T} \partial_n \phi \  v \ dS = \sum_T \int_T fv \ dx
\end{equation}
 \\
holds for all test functions $v \in V $. Here $f(\rvec) = - \nabla \cdotp (M)$ for $\rvec \in \omM$ and $f(\rvec) = 0$
for $\rvec \in \omV$.

\section{Nitsche's Method for $\phi$}
Continuing with the discontinuous Galerkin formulation we reintroduce the notation for the jump
of a function $A$ across a boundary with predefined '+' and '--' sides.
\[ [A] = A^+ - A^- \]
Similarly the average is defined to be
\[ <A> = \frac{1}{2} (A^+ + A^-) \]
\\
A simple calculation gives the following relation for the jump of the product of two functions.
\[ [AB] = [A]<B> + <A>[B] \]
which we can now use on the boundary term of equation~(\ref{dgform}). The sum over boundaries of each element
visits every facet twice, so we can instead replace it with a sum over each facet $\sum_S$, taking into account that
we can have different values on both sides of a facet.
\begin{eqnarray*}
- \sum_T \int_{\partial T} \partial_n \phi \  v \ dS &=&  - \sum_S \int_S [\nabla \phi v ] \cdotp n \ dS \\
&=&  - \sum_S \int_S ( [\nabla \phi] <v> + <\nabla \phi> [v]) \cdotp n \ dS
\end{eqnarray*}
Here $n$ has been fixed in one of the possible directions along a facet.

Nitsche's method applied to our problem \ref{dgform} consists of adding two additional terms that are equal to 0
for the true solution, but which are allowed to carry a tolerable value for the discrete solution.
The first one is
\[ <\partial_n v> [\phi] \]
which is added for symmetry and the next one is
\[\gamma h^{-1} [\phi][v] \]
which is added for stability. Here $h$ denotes the smallest diameter of an element domain in the mesh, and
$\gamma$ is a parameter that we choose. Experiments with the code have shown that a high gamma favours better
continuity at the cost of matching the normal derivative jump condition, and vice versa.

We can now formulate the Nitsche formulation for $\phi$. Since we are only interested in applying a jump condition
$ [\nabla \phi]  \cdotp n =  M \cdotp n $ on the magnetic core boundary $\domM$, we can sew together two continuous functions
together with Nitsche's method to get the result we need. Let $\phi = (\phi_0,\phi_1)$ where
$\supp(\phi_1) \subset \omM$ and $\supp(\phi_0) \subset \omV$. The Nitsche variational problem is to find  $\phi = (\phi_0,\phi_1)$ such that

\begin{align*}
&\int_{\omM} \nabla \phi_1 \cdotp \nabla v_1 \ dx + \int_{\omV} \nabla \phi_0 \cdotp \nabla v_0 \ dx \\
&- \int_{\domM} <\nabla \phi> [v]  + <\partial_n v> [\phi] - \gamma h^{-1} [\phi][v] \ dS \\
&= \int_{\domM} M  \cdotp n <v> \ dS - \int_{\omM} \nabla \cdotp M v_1 \ dx
\end{align*}

The global solution $\phi_{tot}$ is then obtained by adding together $\phi_1$ and $\phi_0$ and then halving the value
of the degrees of freedom of $\phi_{tot}$ on $\domM$.

\section{The Fredkin-Koehler approach}
The Fredkin-Koehler approach splits the magnetic potential $\phi$ into two parts $\phi_1$ and $\phi_2$,
\[ \phi = \phi_1 + \phi_2 \]
\noindent in order to get $\phi_2$ using the hybrid boundary/finite element method(FEM/BEM).

\noindent $\phi_1$ is defined to be the solution of the Neumann problem:

\[
\left\{
\begin{array}{lr}
- \nabla^2 \phi_1 = \nabla \cdotp \magn  & \mbox{in } \ \omM \\
 \frac{\partial \phi_1}{\partial n}  = -n \cdotp \magn & \mbox{on } \  \partial \omM \\
 \phi_1 = 0 & \mbox{on} \ \ \omV
\end{array}
\right.
\]

\noindent and $\phi_2$ is defined to be the solution of the Laplace problem
\[
\left\{
\begin{array}{lr}
- \nabla^2 \phi_2 = 0 & \mbox{in } \ \omM \cup \omV \\
\mbox{jump}(\phi_2) = \phi_1 & \mbox{on} \ \domM \\
\ \phi_2 \rightarrow 0 & \ \mbox{as} \ \mvec{r} \in \omV \rightarrow \infty
\end{array}
\right.
\]

\noindent Adding together the two potentials $\phi_1$ and $\phi_2$ should result in the solution to the problem described in
truncdemag1.

\section*{Appendix}
\appendix
\section{Explanation of the definition of the discontinuous finite element space $\ref{dgdef}$ }\label{appdgspace}
In section \ref{dgsection} describing the discontinuous FE space the following definition is made.
\[
 V = \{ v \in H^1 (T) \mbox{ for all } T \subset \Omega \mbox{ s.t. v is discontinous over } \ \partial T \}
\]
Here $H^1(T)$ is the sobolev space
\[ \left\{ v \in H^1(T) \mbox{ if } v \in L^2(T) \mbox{ and } \frac{\partial v}{\partial x_i} \in L^2(T) \ \forall
i \in \{1...,dim(\Omega)\}  \right\} \]
\\
where the partial derivatives are to be understood in the weak sense.
The possible discontinuities along element boundaries of a function $v \in V$ mean that a weak derivative cannot be
defined globally. However when we restrict $v$ to a finite element domain it becomes continuous, which
means we can define the global discontinuous function space $V$ by patching together local continuous function spaces.

\end{document}


\graphicspath{{Chapter4/Figs/simulations/}{Chapter4/Figs/scrna/}{Chapter4/Figs/scmet/}{Chapter4/Figs/scnmt/}}

\section{Model description}

In MOFA+ we introduce two key novelties, both in the model aspect and in the inference scheme. In the model side we introduce a principled approach for modelling multi-omic data set where the samples are structured into non-overlapping groups, where groups typically correspond to batches, donors or experimental conditions. In the inference side we implement a stochastic inference algorithm to improve scalability and enable inference with large single-cell datasets.

Formally, we generalise the model to a disjoint set of $M$ input views (i.e. groups of features) and $G$ input groups (i.e. groups of samples). The data is factorised according to the following model:
\begin{equation} \label{mofa_master_equation}
	\mathbf{Y}^{m}_{g} = \mathbf{Z}_{g} (\mathbf{W}^{m})^T + \bepsilon^{m}_{g}
\end{equation}
where $\bfZ_{g} \in \R^{N_{g} \times K}$ are a set of $G$ matrices that contain the factor values for the $g$-th group and $\bfW^{m} \in \R^{D_m \times K}$ are a set of $M$ matrices that define the feature weights for the $m$-th view. $\bepsilon^{m}_{g} \in \R^{D_m}$ captures the residuals, or the noise for each feature in each group. Notice that if $G=1$ then the model simplifies to the MOFA framework presented in Chapter 3.

It is important to get the intuition for the multi-group formulation right. In the factor analysis setting, the aim is not to capture differential changes in \textit{mean} levels between the groups but rather to exploit the covariation patterns of the features to identify which sources of variability (i.e. latent Factors) are consistently found across multiple groups and which ones are exclusively found within a single group. This is symmetric to the interpretation of the multi-view framework in MOFA v1: the absolute levels of the features are not compared across views, only the covariation patterns are of interest. To achieve this, the features are centered per view and also per group before fitting the model. \Cref{fig:mofa2_overview} summarises the MOFA+ pipeline.

As in MOFA v1, the linearity assumptions leads to an interpretable latent space that be visualised and employed for a range of downstream analyses, including clustering, inference of non-linear differentiation trajectories, denoising and feature selection, among others. The most important extension is the generalisation of the variance decomposition analysis, where a value of variance explained per view and group is obtained for every factor. For example, imagine that Factor 1 in \Cref{fig:mofa2_overview}b corresponds to cell cycle variation, the variance decomposition analysis indicates that cell cycle is a driver of cell-to-cell heterogeneity largely in views 2 and 3, but with only minor influence in view 1. Also, this effect is manifested in groups 1 and 2, but not in group 3. This simple visualisation provides a very intuitive approach to understand variability in complex experimental designs where observations are structured into multiple views and multiple groups of cells.

\begin{figure}[H]
	\centering
	\includegraphics[width=1.00\linewidth]{mofa2_overview}
	\caption[]{\textbf{Multi-Omics Factor Analysis v2 (MOFA+) provides an unsupervised framework for the integration of multi-group and multi-view (single-cell) data.}\\
	(a) Model overview: the input data consists of multiple datasets structured into M views and G groups. Views consist of non-overlapping sets of features that often represent different assays. Analogously, groups consist of non-overlapping sets of samples that often represent different conditions or experiments. Missing values are allowed in the input data. MOFA+ exploits the covariation between the features to learn a low-dimensional representation of the data ($\bfZ$) defined by $K$ latent factors that capture the global sources of variability. The weights ($\bfW$) provide a measure of feature importance. Model inference is performed using (GPU-accelerated) stochastic variational inference. \\
	(b) The trained MOFA+ model can be queried for a range of downstream analyses: variance decomposition, inspection of feature weights, visualisation of factors and other applications such as clustering, inference of non-linear differentiation trajectories, denoising and feature selection.
	}
	\label{fig:mofa2_overview}
\end{figure}

\subsection{Model priors and likelihood}

\subsubsection{Prior on the weights}

This remains the same as in MOFA v1. We adopt a two-level sparsity prior with an Automatic Relevance Determination per factor and view, and a (reparametrised \cite{Titsias2011}) feature-wise spike-and-slab prior:
\begin{equation}
	p(\hat{w}_{dk}^m,s_{dk}^m) = \Ndist{\hat{w}_{dk}^{m}}{0, 1/\alpha_{k}^{m}}  \text{Ber}(s_{dk}^{m} \,|\,\theta_{k}^{m})
\end{equation}
with the corresponding conjugate priors for $\theta$ and $\alpha$:
\begin{align}
	p(\theta_k^m) &= \Bdist{\theta_k^m}{a_0^\theta,b_0^\theta}\\
	p(\alpha_k^m) &= \Gdist{\alpha_k^m}{a_0^\alpha, b_0^\alpha}
\end{align}

As discussed in Chapter 3, the aim of the ARD prior is to encourage sparse associations between factors and views, such that the weight vector $\bfw_{:,k}^m$ is shrunk to zero if the factor $k$ does not explain any variation in view $m$. The aim of the spike-and-slab prior is to push individual weights to zero to yield a more interpretable solution.

\subsubsection{Prior on the factors}

In MOFA v1 we adopted an isotropic Gaussian prior which assumes an unstructured latent space \textit{a priori}:
\begin{equation}
	p(z_{nk}) = \Ndist{z_{nk}}{0,1}
\end{equation}
This is the assumption that we want to break. Following the same logic as for the weights, the integration of multiple groups of samples requires a flexible prior distribution that defines the existence of non-overlapping groups, such that the model encourages sparse linkages between factors and groups. To formalise this intuition we simply need to extrapolate the sparsity prior from the weights to the factors:
\begin{align}
	p(\hat{z}_{nk}^g,s_{nk}^g) &= \mathcal{N} (\hat{z}_{nk}^g \,|\, 0, 1/\alpha_k^g)\, \text{Ber}(s_{nk}^g \,|\,\theta_k^g) \\
	p(\theta_k^g) &= \Bdist{\theta_k^g}{a_0^\theta,b_0^\theta} \\
	p(\alpha_k^g) &= \Gdist{\alpha_k^g}{a_0^\alpha, b_0^\alpha},
\end{align}
where $g$ is the index of the sample groups.
% Notice that the spike-and-slab prior is introduced for completeness but is not necessarily required, and can be disabled by fixing $\E[\theta_k^g]=1$.

\subsubsection{Prior on the noise}

The variable $\bepsilon$ captures the residuals, or the noise, which is assumed to be normally distributed and heteroskedastic. In MOFA v2 we generalise the noise to have an estimate per feature and per group. This is important to capture the case where some features may be highly variable in one group but not variable in other groups.

\begin{align}
	p(\bepsilon^{m}_{g}) &= \Ndist{\bepsilon^{m}_{g}}{0,(\btau^{m}_{g})^{-1}\I} \\
	p(\tau^{m}_{g}) &= \prod_{d=1}^{D_m} \Gdist{\tau^{m}_{g}}{a_0^\tau, b_0^\tau}
\end{align}
% \begin{align}
% 	p(\epsilon^{m,g}_d) &= \Ndist{\epsilon^{m,g}_d}{0,1/\tau_d^{m,g}} \\
% 	p(\tau_{d}^{m,g}) &= \Gdist{\tau_{d}^{m,g}}{a_0^{\tau}, b_0^{\tau}}
% \end{align}

In addition, as in MOFA v1, non-Gaussian noise models can also be defined, but unless otherwise stated, we will always assume Gaussian residuals.

% \subsubsection{Likelihood}
% Altogether, this results in the following likelihood:
% \begin{equation}
% 	p(\bfY|\bfW,\bfZ,\bTau) = \prod_{m=1}^{M} \prod_{g=1}^{G} \Ndist{\mathbf{Y}^{m}_{g}}{\mathbf{Z}_{g} (\mathbf{W}^{m})^T,(\btau^{m}_{g})^{-1} \I}
% 	% p(\bfY|\bfW,\bfZ,\bTau) = \prod_{m=1}^{M} \prod_{g=1}^{G} \prod_{d=1}^{D_m} \prod_{n=1}^{N} \Ndist{y_{nd}^{mg}}{\bfz_{ng}^T\bfw_{d}^{mg},1/\tau_{d}^{mg}}
% 	% p(y_{nd}^m) = \Ndist{y_{nd}^m}{\bfz_{n,:}\bfw_{d,:}^{mT},1/\tau_d^m},
% \end{equation}

\subsubsection{Graphical model}

In summary, the updated model formulation introduces symmetric two-level sparsity priors in both the weights and the factors. The corresponding graphical model is shown below:
\begin{figure}[H]
	\centering
	\input{graphical_models/mofa2}
	\caption{\textbf{Graphical model for MOFA+.}\\
	The white circles represent hidden variables that are inferred by the model, whereas the grey circles represent the observed variables. There are a total of five plates, each one representing a dimension of the model: $M$ for the number of views, $G$ for the number of groups, $K$ for the number of factors, $D_m$ for the number of features in the $m$-th view and $N_g$ for the number of samples in the $g$-th group.
	}
	\label{fig:MOFA2}
\end{figure}

\subsubsection{Guidelines on the definition of views and groups} \label{section:mofa2_guidelines_views_groups}

\begin{itemize}
	\item \textbf{Views}: views typically correspond to different assays, but there is flexibility in their definition and the user can explore different definitions of views. For example, one could divide the RNA expression data into three views corresponding to mRNA, rRNA and miRNA. Similarly, one can quantify DNA methylation and chromatin accessibility data over different genomic context (enhancers, promoters, etc.).

	\item \textbf{Groups}: groups are generally motivated by the experimental design, but the user can also explore data-driven formulations. There is no \textit{right} or \textit{wrong} definition of groups, depending on the hypothesis that is sought to explore some definitions will be more useful than others.
\end{itemize}

\subsubsection{Model selection} \label{section:mofa2_model_selection}

As discussed in \Cref{section:mofa_robustness}, the inference procedure depends on the parameter initialisation. When using random initialisation, the Factors can vary between different model instances and a model selection step is advised. I realised that this was not a user-friendly solution and it requires a lot of computational resources when applying the model to large datasets. To simplify model training in MOFA+ we initialise the Factors using the principal components from the concatenated data set. In practice, we observe faster convergence times and better ELBO estimates when initialising with the PCA solution (\Cref{fig:mofa2_init}).

\begin{figure}[H]
	\centering
	\includegraphics[width=0.85\linewidth]{mofa2_init}
	\caption[]{
	\textbf{Comparison of PCA and Random initialisation in MOFA}.\\ Data was simulated from the generative model with the following dimensions: $M=2$ modalities, $G=2$ groups, $D=1000$ features, $N=1000$ samples and $K=10$ factors. The dashed lines mark the iteration at which the model converged.
	}
	\label{fig:mofa2_init}
\end{figure}

% COPIED
% \subsection{Solving the rotational invariance problem} \label{section:mofa2_rotational_invariance}

% Conventional Factor Analysis is invariant to rotations in the latent space \cite{Zhao2009}. To demonstrate this property, let us apply an arbitrary rotation to the weights and the factors, specified by the rotation matrix $\bfR \in \R^{K \times K}$:
% \begin{align*}
% 	\tilde{\bfZ} &= \bfZ \bfR^{-1} \\
% 	\tilde{\bfW} &= \bfR \bfW
% \end{align*}
% First, note that the model likelihood is unchanged by this rotation, irrespective of the prior distribution used.
% \begin{equation*}
% 		p(\bfY | \tilde{\bfZ} \tilde{\bfW}, \tau) = p(\bfY | \bfZ \bfR^{-1} \bfR \bfW, \tau) = p(\bfY | \bfZ \bfW, \tau)
% \end{equation*}
% However, the prior distributions of the factors and the weights are only invariant to rotations when using isotropic Normal priors:
% \begin{equation*}
% 	\ln p(\bfW) \propto \sum_{k=1}^{K} \sum_{d=1}^{D} w_{d,k}^2 = \mathrm{Tr}(\bfW^T \bfW) = \mathrm{Tr}(\bfW^T \bfR^{-1} \bfR \bfW) = \mathrm{Tr}(\tilde{\bfW^T} \tilde{\bfW})
% \end{equation*}
% where we have used the property $\bfR^{T} = \bfR^{-1}$ that applies to rotation matrices. The same derivation follows for the factors $\bfZ$.\\
% In practice, this property renders conventional Factor Analysis unidentifiable, hence limiting its interpretation and applicability. Sparsity assumptions, however, partially address the rotational invariance problem \cite{Hore2015}.

% It is important to remark that the factors are nonetheless invariant to permutations. This implies that under different initial conditions, the order of the factors is not necessarily the same in independent model fittings. To address this we manually sort factors \textit{a posteriori} based on total variance explained.

% \subsection{Stochastic variational inference algorithm}

% In \Cref{section:stochastic_variational_inference} I have explained how to derive a stochastic variational inference (SVI) algorithm for a general Bayesian model using an adapted version of the algorithm introduced in \cite{Hoffman2012}.\\
% To apply the SVI algorithm to MOFA the first step is to choose the \textit{local} and \textit{global} dimensions such that the \textit{local} dimension will be factorised in the ELBO and where the stochastic gradients apply.

% In single-cell studies we expect increasingly large datasets in the number of cells, but the number of features are expected to remain roughly constant. Thus, the natural dimension to define as \textit{local} is the axis of samples. In the case of MOFA+, the variables that are classified as \textit{local} are the Factors $\mathbf{Z_g} = \{ z_{nk}^g \}$, which due to the reparametrisation of the spike-and-slab prior consists on the element-wise product of two matrices: $\hat{\mathbf{Z_g}}$ and $\mathbf{S_g}$. All other hidden variables are global.

% This leads to the following SVI algorithm:
% \begin{algorithm}[h!]
%   \caption{Stochastic mean-field variational inference for MOFA+}
%   \begin{algorithmic}[1]
% 	\State Initialise randomly the parameters of the global variables \{$\bf\tau^{gm}$, $\bf\hat{W}^m$, $\bfS^\bfm$, $\bf\alpha^m, \bf\theta^m$, $\bf\alpha^g$, $\bf\theta^g$\}.
% 	\State Initialise the step size $\rho^{(t=0)}$
% 	\Repeat
% 	    \State \text{sample $\mathcal{B}$ a mini-batch of samples of size $S << N$}
% 		\For{\text{each local variational parameter $\phi_{nk}^g$ of nodes \{$\hat{z_{nk}^g}$, $s_{nk}^g$\} such that $n$ is in batch $\mathcal{B}$}} \\
% 			\State $\phi_{nk}^{(t+1)}$ is the updated parameter $\phi_{nk}$ following the classic VI update equation \\
%       	\EndFor
% 		\For{\text{each global variational parameter $\lambda$ of nodes \{$\bf\tau^{gm}$, $\bf\hat{\bfW}^m$, $\bfS^m$, $\bf\alpha^m, \bf\theta^m$, $\bf\alpha^g$, $\bf\theta^g$\}}}
% 		     \State
%         		\begin{align} \label{eq_elbo_factorised} \begin{split}
%             	\lambda^{(t + 1)} &= (1-\rho ^{(t)})\lambda^{(t)} +  \rho ^{(t)} \lambda_{\mathcal{B}}^{(t+1)}
%             \end{split} \end{align}
%             \State \text{where $\lambda_{\mathcal{B}}^{(t+1)}$ is the updated parameter $\lambda$ following the classic VI update equation,}
%             \State \text{but considering the selected batch $\mathcal{B}$ repeated $N/S$ times instead of the full dataset.}
%       	\EndFor
% 	\Until{ELBO convergence}
% 	\end{algorithmic}
% 	\label{MOFAstochasticascent}
% \end{algorithm}

% \subsection{Theoretical comparison with published methods}
% As discussed in Chapter 3, a variety of factor analysis models exist with the aim of perfoming multi-view and/or multi-group data integration. A comparison of multi-view method is shown in \Cref{GFAtable}. Here we specifically focus on the methods that have been applied to single-cell data:
% \begin{table}[h]
% 	\begin{tabular}{@{}llllll}
% 		\toprule
% 		{\textbf{Method}} & {\textbf{\parbox{2.0cm}{Scales to\\$>1e5$ cells?}}} & {\textbf{\parbox{2.0cm}{Multi-view}}} & {\textbf{\parbox{2.0cm}{Multi-group}}} & {\textbf{\parbox{2.0cm}{Missing values}}} & {\textbf{\parbox{2.0cm}{Likelihoods}}} \\ \toprule
% 		slalom \cite{Buettner2017} & No & No & No & No & ZI gaussian \\\midrule
% 		pCMF \cite{Durif2019} & Yes & No & No & No & ZI poisson \\\midrule
% 		ZIFA \cite{Pierson2015} & No & No & No & No & ZI gaussian \\\midrule
% 		scVI \cite{Lopez2018} & Yes & No & Yes & No & ZI negative binomial \\\midrule
% 		MSFA \cite{DeVito2019} & No & No & Yes & No & Gaussian \\\midrule
% 		ZINB-WaVE \cite{Risso2018} & No & No & No & No & ZI negative binomial \\\midrule
% 		AJIVE \cite{Feng2018} & No & Yes & No & No & NA \\\midrule
% 		DIABLO \cite{Singh2018} & Yes & Yes & No & * & NA \\\midrule
% 		scHPF \cite{Levitin2019} & Yes & No & No & Yes & Negative binomial \\\midrule
% 		MOFA \cite{Argelaguet2018} & No & Yes & No & Yes & Gaussian/Poisson/Bernoulli \\\midrule
% 		MOFA+ & Yes & Yes & Yes & Yes & Gaussian/Poisson/Bernoulli \\\midrule
% 	\end{tabular}
% 	\caption{Overview of Factor analysis methods for single-cell data.}
% 	% \label{XXX}
% \end{table}

\subsection{A note on the implementation}

The core of MOFA+ is implemented in Python, and the downstream analysis and visualisations are implemented in R. GPU acceleration is implemented using CuPy \cite{Okuta2017}, an open-source matrix library accelerated with NVIDIA CUDA. To facilitate adoption of the method, we deployed MOFA+ as open-source software\footnote{\url{https://github.com/bioFAM/MOFA2}} with multiple tutorials and a web-based analysis workbench\footnote{\url{http://www.ebi.ac.uk/shiny/mofa/}}.%, hopefully enabling a user-friendly exploration of complex single-cell datasets.