Variational autoencoders \citep[VAEs; ][]{Kingma2014,Rezende2014} are a popular class of deep latent-variable models. The VAE assumes that observations $\x$ are generated by first sampling a latent vector $\z$ from some tractable prior $\p(\z)$, and then sampling $\x$ from some tractable distribution $\p(\x \given \decoder_\genparam(\z))$.
For example, $\decoder_\genparam(z)$ could be a neural network with weights $\genparam$ and $\p(\x \given \decoder_\genparam(\z))$ might be a Gaussian with mean $\decoder_\genparam(\z)$.

VAEs, like other unsupervised latent-variable models \citep[e.g.; ][]{Tipping1999,Blei2003}, can uncover latent structure in datasets.
In particular, one might hope that high-level characteristics of the data are encoded more directly in the geometry of the latent space $\z$ than they are in the data space $\x$. For example, when modeling faces one might hope that one latent dimension corresponds to pose, another to hair length, another to gender, etc.

What kind of latent structure will the VAE actually discover?
\citet{Hoffman2016} observe that the ELBO encourages the model to make the statistics of the population of encoded $\z$ vectors resemble those of the prior, so that $\p(\z)\approx \E_\mathrm{population}[\p(\z \given \x)]$.
The prior $\p(\z)$ therefore plays an important role in shaping the geometry of the latent space.
For example, if we use the ``default'' prior $\p(\z)=\N(\z; 0, I)$, then we are asking the model to explain the data in terms of smoothly varying, completely independent factors \citep{Burgess2018}. These constraints may sometimes be reasonable---for example, geometric factors such as pose or lighting angle may be nearly independent and rotationally symmetric. But some natural factors exhibit dependence structure (for example, facial hair length and gender are strongly correlated), and others may have nonsmooth structure (for example, handwritten characters naturally cluster into discrete groups).

In this paper, we propose using a more opinionated prior on the VAE's latent vectors: the time-marginalized coalescent \citep[TMC; ][]{Boyles2012}.
The TMC is a powerful, interpretable Bayesian nonparametric hierarchical clustering model that can encode rich discrete and continuous structure.
Combining the TMC with the VAE combines the strengths of Bayesian nonparametrics (interpretable, discrete structure learning) and deep generative modeling (freedom from restrictive distributional assumptions).

% We can consider the models from an information-theoretic perspective \citep{alemi18brokenelbo}:
% a classical autoencoder will try to make $z$ encode as much information about $x$ as possible, subject to the constraints imposed by the training procedure and function class, while
% the evidence lower bound (ELBO) used to train VAEs adds a regularization term that penalizes the model for encoding too much information in $z$.
% But this says nothing about the geometry of the latent space, which is critical for downstream applications. For example, if we want to use the inferred $z$ vectors for a classification task we may want them to organize into well-separated clusters.

% If there are multiple factors of variation in the data, we claim that the model will choose to model those whose statistics

% For example, \citet{burgess2018understanding} observe that the common normal prior $\p(z) = \N(z; 0, I)$ encourages the model to learn \emph{disentangled} representations---this

% This implies that the prior $\p(z)$ plays an important role in determining what kind of latent structure the VAE discovers.

% Approximate maximum-likelihood training of VAEs can encourage the latent vector $z$ to have high mutual information with the observations $x$ \citep{alemi18brokenelbo}, while encouraging the model to make the statistics of the encoded $z$ vectors resemble those of the prior so that $\p(z)\approx\mathbb{E}_x[\p(z\mid x)]$ \citep{hoffman2016elbo}.

% If all goes well, the posterior $\p(z\mid x)$ will encode useful information about $x$.

% find an encoding scheme such that the aggregate posterior $\tilde \p(z)\triangleq \frac{1}{N}\sum_\n $

% encode information about $x$ in $z$ in a way that conforms to the (usually simple)

% %One advantage of VAEs over other powerful deep density estimators based on autoregressive models \citep[e.g.; ][]{salimans2017pixelcnn++,oord2016wavenet,papamakarios2017maf} is that the latent vectors $z$ may encode

% - Representation learning with VAEs, latent space encodes semantics

% - Often times people use generic marginals (N(0, I) beta VAE, (factors may not be independent hair length, beard length)

% - Alternatively use a powerful prior (MAF), but encode smoothness assumptions but aren't appropriate if data has natural latent discrete structure

Our contributions are:
\begin{itemize}
    \item We propose a deep Bayesian nonparametric model that can discover hierarchical cluster structure in complex, high-dimensional datasets.
    \item We develop a minibatch-friendly inference procedure for fitting TMCs based on an inducing-point approximation, which scales to arbitrarily large datasets.
    \item We show that our model's learned latent representations consistently outperform those learned by other variational (and classical) autoencoders when evaluated on downstream classification and retrieval tasks.
\end{itemize}

\section{Background}

\subsection{Bayesian priors for hierarchical clustering}
\label{sec:bnhc}

\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[frame, width=\textwidth]{img/loracs/tmc/tmc-small-cropped}
\end{subfigure}
~
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \reflectbox{\includegraphics[frame, width=\textwidth]{img/loracs/tmc/tmc-big-1-cropped}}
\end{subfigure}
\caption{Independent samples from a time-marginalized coalescent (TMC) prior and two-dimensional Gaussian random walk likelihood model (10 and 300 leaves respectively).
Contours in the plots correspond to posterior predictive density $r(\z_{\numdata + 1} \given \dataset, \tree)$.}
\label{fig:tmc-samples}
\end{figure*}

Hierarchical clustering is a flexible
tool in exploratory data analysis
as trees offer visual, interpretable
summaries of data. Typically,
algorithms for hierarchical clustering are
either agglomerative
(where data are recursively, greedily merged to form
a tree from the bottom-up)
or divisive (where data
are recursively partitioned, forming a tree from the
top-down). Bayesian nonparametric hierarchical clustering (BNHC)
additionally incorporates uncertainty over tree
structure by introducing
a prior distribution over trees $r(\tree)$ and
a likelihood model for data $r(\latentdataset \given \tree)$,
with the goal of sampling
the posterior distribution $r(\tree \given \latentdataset)$.\footnote{We use $r$ to denote probability distributions
relating to the TMC and distinguish from $p$ and $q$
distributions used later in the paper.}

In this paper, we focus on
rooted binary trees with $\numdata$ labeled leaves
adorned with branch lengths,
called \emph{phylogenies}.
Prior distributions over phylogenies
often take the form of a stochastic generative
process in which a tree is built
with random merges, as in the Kingman coalescent \citep{Kingman1982},
or random splits, as in the Dirichlet diffusion tree \citep{Neal2003}.
These nonparametric distributions have
helpful properties, such as exchangeability,
which enable efficient Bayesian inference.
In this paper, we focus on the time-marginalized
coalescent \citep[TMC; ][]{Boyles2012}, which decouples the
distribution over tree structure and branch length,
a property that helps simplify inference down the line.

\textbf{Time-marginalized coalescent (TMC):}
The time-marginalized coalescent defines a prior distribution over phylogenies.
A phylogeny $\tree = (V, E, T)$ is a directed rooted full binary tree, with vertex set $V$ and edges $E$, together with time labels $T: V \to [0, \, 1]$ where we denote $t_v = T(v)$.
The vertex set $V$ is partitioned into $\numdata$ leaf vertices $V_\text{leaf}$ and $\numdata-1$ internal vertices $V_\text{int}$, so that $V = V_\text{int} \cup V_\text{leaf}$, and we take $V_\text{leaf} = \{1, 2, \ldots, \numdata\}$ to simplify notation for identifying leaves with $\numdata$ data points.
The directed edges of the tree are encoded in the edge set $E \subset V_\text{int} \times V$, where we denote the root vertex as $v_\text{root}$ and for $v \in V \setminus \{v_\text{root}\}$ we denote the parent of $v$ as $\pi(v) = w$ where $(w, v) \in E$.

The TMC samples a random tree structure $(V, E)$ by a stochastic process in which the $\numdata$ leaves are recursively merged uniformly at random until only one vertex is left.
This process yields the probability mass function on valid $(V, E)$ pairs given by
\begin{equation}
    r(V, E) = \frac{(\numdata - 1)!}{\prod_{v \in V_\text{int}} c(v)} \prod_{i = 1}^{\numdata-1} {i+1 \choose 2}^{-1},
\end{equation}
where $c(v)$ denotes the number of internal vertices in the subtree rooted at $v$.
Given the tree structure, time labels are generated via the stick-breaking process
\begin{equation}
    t_v = \begin{cases} 0 & v = v_\text{root}, \\ 1 & v \in V_\text{leaf}, \\ t_{\pi(v)} - \beta_v (1 - t_{\pi(v)}) & v \in V_\text{int} \setminus \{v_\text{root}\}, \end{cases}
\end{equation}
where $\beta_v \iid \sim \mathrm{Beta}(a, b)$ for $v \in V$. These time labels encode a branch length $t_v - t_{\pi(v)}$ for each edge $e = (\pi(v), v) \in E$. We denote the overall density on phylogenies with $\numdata$ leaves as $\mathrm{TMC}_\numdata(\tree; a, b)$.

Finally, to connect the TMC prior to data in $\R^\d$, we define a likelihood model $r(\latentdataset \given \tree)$ on $\numdata$ data points, with $\z_\n$ corresponding to the leaf vertex $\n \in V_\text{leaf}$.
We use a Gaussian random walk (GRW), where for each vertex $v \in V$ a location $\z_v \given \z_{\pi(v)}$ is sampled according to a Gaussian distribution centered at its parent's location with variance equal to the branch length,
\begin{equation}
    \z_v \given \z_{\pi(v)} \sim \N(\z_{\pi(v)}, (t_v - t_{\pi(v)})I), \quad v \in V \setminus \{v_\text{root}\},
    \notag
\end{equation}
and we take $\z_{v_\text{root}} \sim \N(0, I)$.
As a result of this choice, we can exploit the Gaussian graphical model structure to efficiently marginalize out the internal locations $\z_v$ associated with internal vertices $v \in V_\text{int}$ and evaluate the resulting marginal density $r(\latentdataset \given \tree)$. For details
about this marginalization, please refer to \autoref{sec:algorithm-details}.
The final overall density is written as
\begin{equation}
    r(\latentdataset, \tree) = \mathrm{TMC}_\numdata(\tree; a, b)r(\latentdataset \given \tree).
\end{equation}
For further details and derivations related to the TMC,
please refer to \citet{Boyles2012}.

\textbf{TMC posterior predictive density:}
The TMC with $\numdata$ leaves and a GRW likelihood model can be a prior on a set of $\numdata$ hierarchically-structured data, i.e. data that correspond to nodes with small tree distance should have similar location values.
In addition, it also acts as a density from which we can sample new data.
The posterior predictive density $r(\z_{\numdata + 1} \given \latentdataset, \tree)$ is easy to sample thanks to the exchangeability of the TMC.

To sample a new data point $\z_{\numdata + 1}$, we select a branch (edge) and a time to attach a new leaf node.
The probability $r(e_{\numdata+1} \given V, E)$ of selecting branch $e_{\numdata+1}$ is proportional to the probability under the TMC prior of the tree with a new leaf attached to branch $e_{\numdata+1}$.
The density $r(t_{\numdata+1} \given e_{\numdata+1}, V, E)$ for a time label $t_{\numdata+1}$ is determined by the stick-breaking process (see \autoref{sec:algorithm-details} for details).
% given by a shifted-and-scaled $\mathrm{Beta}(a, b)$ distribution and inserting it into the stick-breaking process, creating a new time $t_{N+1}$ for the node attached to branch $b_{N+1}$.
Both of these probabilities are easy to calculate and sample due to the exchangeability of the TMC.

The new location $\z_{\numdata + 1}$ can be sampled from  $r(\z_{\numdata + 1} \given e_{\numdata + 1}, t_{\numdata + 1}, \tree)$, which is
the Gaussian distribution that comes out of the GRW likelihood model.
Pictured in \autoref{fig:tmc-samples} are samples from a TMC prior and GRW likelihood, where contours correspond to $r(\z_{\numdata + 1} \given \latentdataset, \tree)$.
In addition to modeling hierarchical structure, the TMC is a flexible nonparametric density estimator.

\textbf{TMC inference:}
The posterior distribution $r(\tree \given \latentdataset)$
is analytically intractable due to the normalization constant
$r(\latentdataset)$ involving a sum over all tree structures,
but it can be approximately sampled via Markov chain Monte-Carlo (MCMC) methods.
% In MCMC, we construct a proposal distribution
% $T(\tree' \given \tree)$
% a Markov chain with transition dynamics $T$
% has joint distribution $r(\tree, z_{1:N})$.
We utilize the Metropolis-Hastings
algorithm with a subtree-prune-and-regraft (SPR)
proposal distribution \citep{Neal2003}. An SPR proposal
picks a subtree uniformly at random from $\tree$
and detaches it.
It is then attached back on the tree
to a branch and time picked uniformly at random.
% The new tree $\tree'$ is accepted as the next
% state in the Markov chain
% with probability $\min\left(1, \frac{r(\tree', z_{1:N})T(\tree \given \tree')}{r(\tree, z_{1:N})T(\tree' \given \tree)}\right)$.
The Metropolis-Hastings acceptance probability is efficient to compute because the joint density $r(\tree, \latentdataset)$ can be evaluated using belief propagation to marginalize the latent values at internal nodes of $\tree$, and many of the messages can be cached.
See \autoref{sec:algorithm-details} for details.

\subsection{Variational autoencoder}
The variational autoencoder (VAE) is a generative model
for a dataset $\dataset$
wherein latent vectors $\latentdataset$ are sampled
from a prior distribution
and then individually passed into
a neural network observation model with parameters $\genparam$,
\begin{equation}
    \begin{split}
        \latentdataset \sim \p(\latentdataset),
        \qquad
        \x_\n \given \z_\n \sim \p_\genparam(\x_\n \given \z_\n),
    \end{split}
\end{equation}
We are interested in the posterior distribution
$\p(\z_\n \given \x_\n)$, which is not analytically tractable
but can be approximated with a variational distribution
$\q_\recparam(\z_\n \given \x_\n)$, typically a neural network
that outputs parameters of a Gaussian distribution.
The weights of the approximate posterior can be learned
by optimizing the evidence-lower bound (ELBO),
\begin{equation}
    \begin{split}
    \L[\q] &\triangleq \E_\q \left[\log \frac{\p_\genparam(\dataset, \latentdataset)}{\prod_\n \q_\recparam(\z_\n \given \x_\n)}\right] \\
          %&= \left(\sum_{n = 1}^N \E_\q\left[\log \p(\x_\n \given \z_\n)\right]\right) - \KL{q_\recparam(z_{1:N} \given x_{1:N})}{\p(z_{1:N})}.
    \end{split}
\end{equation}
The parameters of the model, $\genparam$ and $\recparam$,
are learned via stochastic gradient ascent on
the ELBO, using the reparametrization trick
for lower variance gradients \citep{Kingma2014, Rezende2014}.

\section{The TMC-VAE}
\label{sec:tmc-vae}

The choice of prior distribution in the VAE significantly affects the autoencoder and resulting latent space.
The default standard normal prior, which takes $\z_\n \iid\sim \N(0, I)$, acts as a regularizer on an otherwise unconstrained autoencoder, but can be restrictive and result in overpruning \citep{Burda2015}.
Extremely flexible, learnable distributions like masked autoregressive flow (MAF) priors \citep{Papamakarios2017} enable very rich latent spaces, but don't encode any interpretable bias for organizing the latent space (except perhaps smoothness).

In this paper, we explore the TMC prior for the VAE, which could potentially strike a sweet spot between restrictive and flexible priors.
We generate the latent values $\latentdataset$ of a VAE according to the TMC prior, then generate observations $\dataset$ using a neural network observation model,
\begin{gather}
        \tree \sim \mathrm{TMC}_\numdata(\tree; a, b), \\
        \latentdataset \given \tree \sim r(\latentdataset \given \tree),
        \qquad
        \x_\n \given \z_\n \sim \p_\genparam(\x_\n \given \z_\n).
\end{gather}
The TMC-VAE is a coherent
generative process that captures
discrete, interpretable structure in the latent space.
A phylogeny not only has an intuitive inductive bias,
but can be useful for
exploratory data analysis
and introspecting the latent space itself.

Consider doing inference in this model:
first assume variational distributions
$\q_\recparam(\z_\n \given \x_\n)$ (as in the VAE)
and $\q(\tree)$, which results in the
ELBO,
\begin{equation}
    \begin{split}
    \L[\q] &= \E_\q\left[\log \frac{\p(\tree, \latentdataset, \dataset)}{\q(\tree)\prod_\n \q(\z_\n \given \x_\n)}\right] \\
    \end{split}
\end{equation}
For fixed $\q_\recparam(\z_\n \given \x_\n)$, we can
sample the optimal $\q^*(\tree)$,
\begin{equation}
    \begin{split}
    \q^*(\tree) &\propto \exp{\E_\q\left[\log \p(\tree, z_{1:N}, x_{1:N})\right]} \\
              &\propto \exp{\E_\q\left[\log \p(\tree)\p(z_{1:N}|\tree)\right]} \\
              &\propto \exp{\log \p(\tree) + \E_\q\left[\log \p(\latentdata|\tree)\right]}
    \end{split}
\end{equation}
\begin{equation}
    \begin{split}
    \end{split}
\end{equation}
Because $\p(\latentdataset \given \tree)$ is jointly Gaussian (factorizing
according to tree structure) and $\q_\recparam(\z_\n \given \x_\n)$ is Gaussian,
expectations with respect to $\latentdataset$
can move into $\log \p(\tree, \latentdataset, \dataset)$. This
enables sampling the expected joint likelihood $\E_\q\left[\log \p(\tree, \latentdataset)\right]$
using SPR Metropolis-Hastings.
However, optimizing this ELBO is problematic.
$\p(\latentdataset \given \tree)$
does not factorize independently, so computing
unbiased gradient estimates from minibatches is impossible
and requires evaluating all the data.
Furthermore, the TMC is limiting
from a computational perspective.
Since a phylogeny has as many leaves
as points in the dataset,
belief propagation over internal nodes
of the tree slows down linearly as
the size of the dataset grows.
In addition,
% the search space of trees
% grows exponentially with the size of the dataset,
% rendering MCMC unusable with extremely
SPR proposals mix very slowly for large trees.
We found these limitations
make the model impractical for datasets
of more than 1000 examples.

In the next section, we address these computational
issues, while retaining
the interesting properties of the TMC-VAE.

\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.3\textwidth}
    \centering
    \includegraphics{tikz/loracs/ltmc.tikz}
    \caption{TMC-VAE graphical model}
\end{subfigure}
\begin{subfigure}[t]{0.34\textwidth}
    \centering
    \includegraphics{tikz/loracs/iltmc.tikz}
    \caption{\acronym-VAE graphical model}
\end{subfigure}
\begin{subfigure}[t]{0.34\textwidth}
    \centering
    \includegraphics{tikz/loracs/iltmc-v.tikz}
    \caption{\acronym-VAE variational factors}
\end{subfigure}
\caption{Graphical models and variational
approximations for TMC models described in the paper}
\label{fig:graphical-models}
\end{figure*}

\section{\acronym\;prior for VAEs}

In this section, we introduce
a novel approximation to the TMC
prior, which preserves many
desirable properties
like structure and interpretability,
while being computationally viable.
Our key idea is to use a set of
learned \emph{inducing points}
as the leaves of the tree in the latent
space, analogous to inducing-input
approximations for Gaussian processes \citep{Snelson2006}.
In this model, latent vectors $\latentdataset$
are not directly hierarchically clustered,
but are rather independent samples
from the induced posterior predictive density
of a TMC.
We call this the \acronymexplanation\;(\emph{\acronym}, pronounced ``\acronympronunciation'') prior.

To define the \acronym\;prior $\p(\tree, \dataset)$, we first define an auxiliary TMC distribution $r(\tree, \inducingpoints)$ with $M$ leaf locations $\inducingpoints$.
We treat $\inducingpoints$ as a set of learnable free parameters, and define the conditional $r(\tree \given \inducingpoints)$ as the \acronym\;prior on phylogenies $\tree$:
\begin{equation}
%   \tree \sim \p(\tree ; \inducingpoints) \triangleq r(\tree \given \inducingpoints).
  \p(\tree ; \inducingpoints) \triangleq r(\tree \given \inducingpoints).
\end{equation}
That is, we choose the prior on phylogenies $\tree$ to be the posterior distribution of a TMC with pseudo-observations $\inducingpoints$.
Next, we define the \acronym\;prior on locations $\z_\n\given\tree$ as a conditionally independent draw from the predictive distribution $r(s_{M+1} \given \tree, \inducingpoints)$, writing the sampled attachment branch and time as $e_\n$ and $t_\n$, respectively:
\begin{gather}
    % b_\n, t_\n \given \tree \sim \p(b_\n, t_\n \given \tree)
    \p(e_\n, t_\n \given \tree)
    \triangleq r(e_{M+1} = e_\n, t_{M+1} = t_\n \given \tree),
    \notag
    \\
    % \z_\n \given b_\n, t_\n \tree \sim \p(\z_\n \given b_\n, t_\n, \tree)
    \p(\z_\n \given e_\n, t_\n, \tree)
    \triangleq r(\inducingpoint_{M+1} = \z_\n \given e_\n, t_\n, \tree, \inducingpoints).
\end{gather}
To complete the model, we use an observation likelihood parameterized by a neural network, writing
\begin{equation}
    \x_\n \given \z_\n \sim \p_\genparam(\x_\n \given \z_\n).
\end{equation}
%
% We introduce a set of
% $M$ learnable inducing points $\inducingpoints$
% that are leaves of a phylogeny $\tree$,
% and adopt the following model:
%
% \todo{simplify this model description}
% \begin{equation}
%     \begin{split}
%         \tree &\sim \p(\tree ; \inducingpoints) \triangleq r(\tree \given \inducingpoints) \\
%         \z_\n \given \tree &\sim \p(\z_\n \given \tree; \inducingpoints) \triangleq r(s_{M + 1} = \z_\n \given \tree, \inducingpoints) \\
%         \x_\n \given \z_\n &\sim p_\genparam(\x_\n \given \z_\n)
%     \end{split}
% \end{equation}
% We choose our prior
% on phylogenies to be the posterior TMC
% with leaves $\inducingpoints$.
% The prior on latent vectors $\p(\z_\n \given \tree; \inducingpoints)$
% is the posterior predictive density for $\tree$ and $\inducingpoints$
% and the observation model is parametrized by a neural network,
% as in the VAE. Effectively, we have put a flexible,
% nonparametric density estimator as our VAE prior
% with the added benefit of learned discrete structure.
%
%By limiting the size of the tree to $M$ leaves,
%we've circumvented some computational issues
%issues that come with larger trees.
%This comes at the cost of the
%expressiveness and interpretability
%of this prior distribution, but
%this can be tuned by making $M$
%larger. However, some additional
%approximations will need to be made for tractable inference.
%
% For the purpose of inference,
% we work with a more explicit version
% of the model, where the branch and time of the
% posterior predictive distribution are
% are sampled explicitly.
% \todo{use this model for induced tmc}
% \begin{equation}
%     \begin{split}
%         \tree &\sim \p(\tree ; \inducingpoints) \\
%         b_\n, t_\n &\sim \p(b_\n, t_\n \given \tree) \\
%         \z_\n \given b_\n, t_\n, \tree &\sim \p(\z_\n \given b_\n, t_\n, \tree; \inducingpoints) \triangleq r(s_{M + 1} = \z_\n \given b_\n, t_\n, \tree) \\
%         \x_\n \given \z_\n &\sim p_\genparam(\x_\n \given \z_\n)
%     \end{split}
% \end{equation}
%
By using the learned inducing points $\inducingpoints$, we avoid the main difficulty of inference in the TMC-VAE of \autoref{sec:tmc-vae}, namely the need to do inference over all $\numdata$ points in the dataset.
Instead, dependence between datapoints is mediated by the set of inducing points $\inducingpoints$, which has a size independent of $\numdata$.
As a result, with the \acronym\;prior, minibatch-based learning becomes tractable even for very large datasets.
The quality of the approximation to the TMC-VAE can be tuned by adjusting the size of $M$.

However, this technique presents its own inference challenges.
Sampling the optimal variational factor $\q^*(\tree)$
is no longer an option as it was in the TMC-VAE:
\begin{equation}
\begin{split}
    \q^*(\tree; \inducingpoints)
    &\textstyle\propto \exp{\E_\q\left[\log \p(\tree, \latentdataset, \dataset)\right]} \\
    &\textstyle\propto \exp{\log \p(\tree ; \inducingpoints) + \sum_\n \E_\q\left[\p(\z_\n \given e_\n, t_\n, \tree)\right]} \\
    &\textstyle\propto \exp{\log \mathrm{TMC}_M(\tree; a, b)
    \\ &\textstyle\qquad\quad + \sum_{m = 1}^M \log r(\inducingpoint_m \given \inducingpoint_{1:m - 1}, \tree)
    \\ &\textstyle\qquad\quad + \sum_\n \E_\q\left[\log \p(\z_\n \given e_\n, t_\n, \tree)\right]}.
\end{split}
\end{equation}
This term has a sum over $\numdata$ expectations; therefore computing this likelihood
for the purpose of MCMC
would involve passing the entire dataset through a neural network.
Furthermore, the normalizer for this likelihood is intractable,
but necessary for computing gradients w.r.t $\inducingpoints$.
We therefore avoid using the optimal $\q^*(\tree; \inducingpoints)$
and set
$\q(\tree; \inducingpoints)$ to the prior.
This has the additional computational advantage of cancelling out
the $\E_\q[\log \p(\tree)]$ term in the ELBO, which also has an intractable normalizing constant.
If the inducing points are chosen so that they contain most of the information about the hierarchical organization of the dataset, then the approximation $\p(\tree \given \z)\approx r(\tree \given \inducingpoints)=\p(\tree)$ will be reasonable.
% Although this choice adds some approximation error, we still fit a distribution over tree structures by optimizing the inducing points $\inducingpoints$ which parameterize the prior.

We also fit the variational factors
$\q(e_\n)$,
$\q_{\xi}(t_\n \given e_\n, \z_\n; \inducingpoints)$,
and
$\q_\recparam(\z_\n \given \x_\n)$.
The factor for attachment times,
$\q_{\xi}(t_\n \given e_\n, \z_\n; \inducingpoints)$, is a
recognition network that outputs
a posterior over attachment times for a particular branch.
Since the $\q(\tree; \inducingpoints)$ and $\p(\tree; \inducingpoints)$ terms
cancel out, we obtain the following ELBO (some notation suppressed for simplicity):
\begin{equation}
    \begin{split}
    \L[\q] &\triangleq \E_\q \left[\log \frac{\prod_\n \p(e_\n, t_\n \given \tree) \p(\z_\n \given e_\n, t_\n, \tree)\p(\x_\n \given \z_\n)}{\prod_\n \q(e_\n)\q(t_\n \given e_\n, \z_\n)\q(\z_\n \given \x_\n)}\right].
    \end{split}
\end{equation}
This ELBO can be optimized by first computing
\begin{equation}
    \q^*(e_\n) = \exp{\E_\q\left[\log \p(e_\n \given t_\n, \z_\n, \tree; \inducingpoints)\right]}
\end{equation}
and computing gradients with respect to $\genparam$, $\inducingpoints$,
$\recparam$, and $\xi$ using a Monte-Carlo estimate of the ELBO
using samples from $\q(\tree; \inducingpoints)$, $q^*(e_\n)$, $\q_\recparam(\z_\n \given \x_\n)$, and $q_{\xi}(t_\n \given e_\n, \z_\n; \inducingpoints)$.
The factor $\q(\tree; \inducingpoints)$ can
be sampled using vanilla SPR Metropolis-Hastings.
The detailed inference procedure can be found in \autoref{sec:algorithm-details}.

\section{Related work}
As mentioned above, \acronym\;connects various ideas in the literature, including Bayesian nonparametrics \citep{Boyles2012}, inducing-point approximations \citep[e.g.; ][]{Snelson2006,Tomczak2017},
and amortized inference \citep{Kingma2014, Rezende2014}.

Also relevant is a recent thread of efforts to endow VAEs with the interpretability of graphical models \citep[e.g.; ][]{Johnson2016,Lin2018}.
In this vein, \citet{Goyal2017} propose using a different Bayesian nonparametric tree prior, the nested Chinese restaurant process (CRP) \citep{Blei2010}, in a VAE.
We chose to base \acronym\;on the TMC instead, as the posterior predictive distribution of an nCRP is a finite mixture, whereas the TMC's posterior predictive distribution has more complex continuous structure.
Another distinction is that \citet{Goyal2017} only consider learning from pretrained image features, whereas our approach is completely unsupervised.

\section{Results}
In this section, we
analyze properties of the
\acronym\;prior,
focusing on
qualitative aspects, like exploratory data analysis and interpretability,
and quantitative aspects, like few-shot classification and information retrieval.

\textbf{Experimental setup} We evaluated
the \acronym\;prior on three separate
datasets: dynamically binarized MNIST \citep{Lecun1998}, Omniglot \citep{Lake2015},
and CelebA \citep{Liu2015}.
For all three experiments,
we utilized convolutional/deconvolutional encoders/decoders
and a 40-dimensional
latent space (detailed architectures can be found
in \autoref{sec:implementation-details}).
We used 200, 1000, and 500 inducing points for MNIST, Omniglot, and CelebA respectively
with TMC parameters $a = b = 2$.
$\q_{\xi}(t_\n \given e_\n, \z_\n; \inducingpoints)$ was a
two-layer 500-wide neural network
with ReLU activations that output parameters of a
logistic-normal distribution over stick size
and all parameters were optimized with Adam \citep{Kingma2014adam}.
Other implementation details can be found in \autoref{sec:implementation-details}.

\subsection{Qualitative results}

A hierarchical clustering in the latent space
offers a unique opportunity for interpretability
and exploratory data analysis,
especially when the data are images.
Here are some methods for users to obtain
useful data summaries and explore a dataset.

\textbf{Visualizing inducing points}
We first inspect the learned inducing points $\inducingpoints$
by passing them through the decoder.
Visualized in \autoref{fig:mnist-inducing}
are the 200 learned inducing points for MNIST.
The inducing points are all unique
and are cleaner than pseudo-input reconstructions from VampPrior (shown in
\autoref{fig:mnist-vamp-inducing-outputs}).
Inducing points can help summarize a
dataset, as visualizations of the latent space
indicate they spread out and cover the data
(see \autoref{fig:mnist-tsne-tree}). Inducing points
are also visually unique and sensible
in Omniglot and CelebA (see
\autoref{fig:omniglot-inducing-points} and \ref{fig:celeba-inducing-points}).

\begin{figure}[t]
\centering
\includegraphics[width=0.8\textwidth]{img/loracs/mnist/mnist-inducing-points.png}
\caption{Learned inducing points for a \acronym(200) prior on MNIST.}
\label{fig:mnist-inducing}
\end{figure}

\textbf{Hierarchical clustering}
We can sample
$\q(\tree ; \inducingpoints)$ to obtain
phylogenies
over the inducing points,
and can visualize these clusterings
using the decoded inducing points;
subtrees from a sample in each dataset
are visualized in \autoref{fig:subtrees}.
In MNIST, we find large subtrees
correspond to the discrete classes
in the dataset. In Omniglot,
subtrees sometimes correspond to language groups
and letter shapes. In CelebA,
we find subtrees sometimes correspond
to pose or hair color and style.

We can further use the time at each internal node
to summarize the data at many levels of granularity.
Consider ``slicing'' the hierarchy
at a particular time $t$ by
taking every branch $(\pi(v), v) \in E$ with $t_{\pi(v)} \leq t < t_v$
and computing the corresponding expected Gaussian random walk value at time $t$.
At times closer to zero, we slice
fewer branches and are closer to the root
of the hierarchy, so the value at the slice
looks more like the mean of the data.
In \autoref{fig:celeba-evolution},
we visualize this process over a subset
of the inducing points of CelebA.
Visualizing the dataset in this way
reveals cluster structure at
different granularities
and offers an evolutionary interpretation
to the data, as leaves that coalesce
more ``recently'' are likely to be closer in
the latent space.

Although the hierarchical clustering is only
over inducing points, we can still visualize
where real data belong on the hierarchy
by computing $\q^*(e_\n)$ and
attaching the data to the tree.
By doing this for many points of data, and removing
the inducing points from the tree,
we obtain an induced hierarchical clustering.
%We visualize some example induced hierarchical clusterings
%in \autoref{fig:induced-clusters}.

\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{img/loracs/mnist/subtree1.png}
\caption{MNIST}
\end{subfigure}
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{img/loracs/omniglot/subtree1.png}
\caption{Omniglot}
\end{subfigure}
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{img/loracs/celeba/subtree1.png}
\caption{CelebA}
\end{subfigure}
\caption{An example learned subtree from a sample of $\q(\tree; \inducingpoints)$ for each dataset.
Leaves are visualized by passing inducing points throught the decoder.}
\label{fig:subtrees}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=0.8\textwidth]{img/loracs/celeba/celeba-evolution.png}
\caption{The evolution of a CelebA over a subset of inducing points. We create
this visualization by taking slices of the tree at particular times
and looking at the latent distribution
at each of the sliced branches.}
\label{fig:celeba-evolution}
\end{figure}

\textbf{Generating samples}
Having fit a generative model to our data,
we can visualize samples from the model.
Although we do not expect the samples
to have fidelity and sharpness comparable
to those from GANs or state-of-the-art
decoding networks \citep{Radford2015, Salimans2017},
sampling with the
\acronym\;prior can help us understand the latent space.
To draw a sample from a TMC's posterior predictive density,
we first sample a branch and time, assigning
the sample a place in the tree.
This provides each generated sample a \emph{context},
i.e., the branch and subtree it was generated from.
However, learning a \acronym\;prior allows us to
conditionally sample in a novel way. By restricting
samples to a subtree, we can generate samples
from the support of the posterior predictive density
limited to that subtree. This enables
conditional sampling at many levels of
the hierarchy. We visualize examples
of this in \autoref{fig:subtree-samples}
and \autoref{fig:celeba-samples}.

\begin{figure}[h]
\centering
\begin{subfigure}[t]{0.8\textwidth}
\centering
\includegraphics[width=\textwidth]{img/loracs/mnist/mnist-samples.png}
\caption{MNIST}
\end{subfigure}
\begin{subfigure}[t]{0.8\textwidth}
\centering
\includegraphics[width=\textwidth]{img/loracs/omniglot/omniglot-samples.png}
\caption{Omniglot}
\end{subfigure}
\caption{Conditional samples from subtrees.}
\label{fig:subtree-samples}
\end{figure}

\begin{figure}[h]
\centering
\begin{subfigure}[t]{0.8\textwidth}
\centering
\includegraphics[frame, width=\textwidth]{img/loracs/celeba/celeba-samples-1.png}
\end{subfigure}
\begin{subfigure}[t]{0.8\textwidth}
\centering
\includegraphics[frame, width=\textwidth]{img/loracs/celeba/celeba-samples-2.png}
\end{subfigure}
\caption{Samples from subtrees of CelebA.}
\label{fig:celeba-samples}
\end{figure}

\subsection{Quantitative results}

We ran experiments designed to
evaluate the usefulness of the \acronym's
learned latent space for downstream tasks. We
compare the \acronym\;prior
against a set of baseline priors
on three different tasks:
few-shot classification,
information retrieval,
and generative modeling.
Our datasets are
dynamically binarized MNIST and Omniglot
(split by instance) and
our baselines are representations
learned with the same encoder-decoder
architecture and latent dimensionality\footnote{Following the defaults
in the author's reference implementation, we evaluated
DVAE\# on statically binarized MNIST with smaller neural networks, but
with a higher-dimensional latent space.} but substituting
the following prior distributions over $z$:
\begin{itemize}
    \setlength\itemsep{0.2em}
    \item No prior
    \item Standard normal prior
    \item VampPrior \citep{Tomczak2017} - 500 pseudo-inputs for MNIST, 1000 for Omniglot
    \item DVAE$\sharp$ \citep{Vahdat2018dvaesharp} - latent vectors are 400-dimensional, formed from concatenating binary latents, encoder and decoder are two-layer feed-forward networks with ReLU nonlinearities
    \item Masked autoregressive flow \citep[MAF; ][]{Papamakarios2017} - two layer, 512 wide MADE
\end{itemize}
\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{img/loracs/mnist/cf-mnist.png}
    \caption{MNIST}
\end{subfigure}
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{img/loracs/omniglot/cf-omniglot.png}
    \caption{Omniglot}
\end{subfigure}
\caption{Few-shot classification results}
\vspace{-0.5cm}
\label{fig:fewshot}
\end{figure}

\textbf{Few-shot classification}
In this task,
we train a
classifier with
varying numbers of labels and measure
test accuracy. We pick
equal numbers of labels per class
to avoid imbalance and we use
a logistic regression classifier
trained to convergence to avoid
adding unnecessary degrees of freedom to the experiment.
We replicated the experiment across
20 randomly chosen label sets for MNIST
and 5 for Omniglot.
The test accuracy on
these datasets is visualized in
\autoref{fig:fewshot}. For MNIST, we
also manually labeled inducing points and
found that training a classifier on 200 and 500 inducing points
achieved significantly better test accuracy than
randomly chosen labeled points, hinting that
the \acronym\;prior has utility in an active learning setting.

The representations
learned with the \acronym\;
consistently achieve better accuracy,
though  in MNIST, \acronym\;prior
and MAF reach very similar test
accuracy at 100 labels per class.
The advantage of the \acronym\;prior is especially clear
in Omniglot
(\autoref{tab:semisupervised-mnist}
and \autoref{tab:semisupervised-omniglot} contain the exact numbers).
We believe our advantage in this task
comes from ability of the \acronym\;prior
to model discrete structure.
TSNE visualizations
in \autoref{fig:tsne-tmc-normal} and \autoref{fig:tsne}
indicate
clusters are more concentrated and separated
with the \acronym\;prior
than with other priors,
though TSNE visualizations should be
taken with a grain of salt.

\begin{figure}[H]
\centering
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{img/loracs/mnist/tsne/mnist2-tsne-normal.png}
    \caption{Normal prior}
\end{subfigure}
\begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{img/loracs/mnist/tsne/mnist2-tsne-tmc.png}
    \caption{\acronym(200) prior}
\end{subfigure}
\caption{TSNE visualizations of the latent space of the MNIST test set with different priors,
color-coded according to class. \acronym\;prior appears to learn a space with more
separated, concentrated clusters.}
\label{fig:tsne-tmc-normal}
\end{figure}

\textbf{Information retrieval}
We evaluated the meaningfulness of Euclidean distances
in the learned latent space
% Information retrieval can
% help inform whether distance measured
% in the latent space is informative.
% We evaluate this
by measuring precision-recall
when querying the test set.
We take each element of the test set
and sort all other members according to
their $L_2$ distance in the latent space.
From this ranking, we produce
precision-recall curves for each
of the query and
plot the average precision-recall
over the entire test set in \autoref{fig:prec-rec}.
We also report the area-under-the-curve (AUC)
measure for each of these curves in \autoref{tab:auc}.

AUC numbers for Omniglot are low across the board
because of the large number of classes
and low number of instances per class.
However,
in both datasets the \acronym\;prior consistently
achieves the highest AUC,
especially with MNIST.
The \acronym\;prior encourages
tree-distance to correspond to
squared Euclidean distance, as branch
lengths in the tree are
variances in a Gaussian likelihoods.
We thus suspect distances in a \acronym\;prior
latent space to be more informative
and better for information retrieval.

\textbf{Held-out log-likelihood}
% Held-out log-likelihood estimates
% the capacity of a generative model.
We estimate held-out log-likelihoods for the four VAEs we
trained with comparable architectures and different priors.
(We exclude DVAE$\sharp$ since its architecture is substantially
different, and the classical autoencoder since it lacks generative
semantics.)
We use 1000 importance-weighted samples \citep{Burda2015}
to estimate held-out log-likelihood,
and report the results in \autoref{tab:holl}.
We find that, although \acronym\;outperforms the other
priors on downstream tasks, it only achieves
middling likelihood numbers.
This result is consistent with the findings of \citet{Chang2009} that held-out log-likelihood is not necessarily correlated with interpretability or usefulness for downstream tasks.

\begin{table}
\centering
\caption{Averaged precision-recall AUC on MNIST/Omniglot test datasets}
\input{results/loracs/ir.tex}
\label{tab:auc}
\end{table}

\begin{table}
\centering
\caption{MNIST/Omniglot test log-likelihoods}
\begin{tabular}{r|cc}
\toprule
Prior & MNIST & Omniglot\\ \midrule
%No prior  & N/A & N/A\\
Normal    & -83.789 & -89.722\\
MAF       & \textbf{-80.121} & \textbf{-86.298}\\
Vamp & -83.0135 & -87.604\\
\acronym & -83.401 & -87.105\\
\bottomrule
\end{tabular}
\label{tab:holl}
\end{table}

\section{Discussion}
Learning discrete, hierarchical structure in a latent space
opens a new opportunity:
interactive deep unsupervised learning.
User-provided constraints have been used
in both flat and
hierarchical clustering \citep{Wagstaff2000,Awasthi2010}, so an interesting
follow up to this work would be incorporating
constraints into the \acronym\; prior, as
in \citet{Vikram2016}, which could potentially
enable user-guided representation learning.

\loracsack


\subsection{Three dimensional uniform dataset}

The three dimensional uniform dataset (that corresponds to the RGB color cube) is an interesting low dimensional case that requires a dimensionality reduction (from dimension 3 to 2). Since we used a uniform distribution this means the dataset is a dense three dimensional manifold that needs to be mapped to a two dimensional manifold which is known not to have an optimal solution. However, this difficulty can be partially alleviated using a loose topology in the RSOM. This is made possible by using a 2-neighbours induced topology as shown in figure \ref{fig:3D-uniform:results}A. This weak topology possesses several disconnected subgraphs that relax the constraints on the neighborhood of the BMU (see figure \ref{fig:topology-influence} in the supplementary section for the influence of neighborhood on the self-organization). This is clearly illustrated in figure \ref{fig:3D-uniform:results}B where the Voronoi cell of a neuron has been painted with the color of its codeword. We can observe an apparent structure of the RGB spectrum with some localized ruptures. To test for the completeness of the representation, we represented the position of six fundamental colors (C - white (1,1,1), D - black (0,0,0), E - yellow (1,1,0), F - red (1,0,0), G - green (0,1,0) and H - blue (0,0,1)) along with their associated distance maps after learning.

Furthermore, we performed the persistent homology to identify important topological features in the
input space and investigate how well the SOM and RSOM captured those features. Figures
\ref{fig:3D-uniform:analysis}A, B, and C show the persistent barcodes for the input space, SOM, and
RSOM, respectively. We can see how the RSOM (panel C) captures more $H1$- and $H2$-homological properties
(since there are more persistent line segments, orange and green lines). The SOM (panel B) seems to capture
some of those features as well but they do not persist as long as they in the case of RSOM. The persistence
diagrams of input, SOM and RSOM are shown in figures~\ref{fig:3D-uniform:analysis} D, E, and F, respectively.
These figures indicate that the RSOM has more persistent features (orange and green dots away from the diagonal
line) than the regular SOM.
The Bottleneck distance between the persistence diagrams of input space and those of SOM and RSOM reveals
that the SOM's persistence diagram is slightly closer to the input space's one for both the $H0$ (SOM: $0.00035$,
RSOM: $0.0007$), $H1$ (SOM: $0.006$, RSOM: $0.007$), and $H2$ (SOM: $0.0062$, RSOM: $0.0057$). Despite the
fact that the bottleneck distances show that
regular SOM's persistent diagram is closer to input space's one, the barcodes diagrams indicate that the RSOM
captures more persistent topological features suggesting that RSOM preserves in a better way the topology of the
input space. Furthermore, the RSOM seems to capture better the higher dimensional topological features since the
Bottleneck distances of $H2$-homological features are smaller for the RSOM than for the SOM.

% For the three-dimensional experiment we draw 50,000 three-dimensional points from a uniform distribution $\mathcal{U}([0, 1]\times [0, 1]\times [0, 1])$ (the cloud points form a cube in $\mathbb{R}^3$). The map is a two-dimensional manifold and thus the required task it to map the three-dimensional vectors  to the two-dimensional neural space. Once again we place the neurons on the two-dimensional neural space based on the sampling of a blue noise distribution and the induced topology is illustrated in Figure~\ref{fig:3D-uniform:results}A. The learning algorithm runs for $25000$ epochs over the three-dimensional vectors and after convergence we  obtain the map with the prototypes shown in Figure~\ref{fig:3D-uniform:results}B. Each color represents a different face of the three-dimensional cube (a cube has six faces so we obtain mainly six colors) and we observe that the SOM has mapped the three-dimensional cube on a two-dimensional neural space. The continuity and grouping of the colors indicates that the receptive fields of the neurons have been established properly. We illustrate this phenomenon in panels~\ref{fig:3D-uniform:results}C-H,  where the response of six different neurons (see the annotation in panel~\ref{fig:3D-uniform:results}B). The dark blue color indicates values close to zero and the yellow color represents values close to one.

\begin{figure}
  \includegraphics[width=\columnwidth]{experiment-3D-uniform.pdf}
  \vspace{2mm}
  \centering
  \includegraphics[width=.975\columnwidth]{figures/colormap.pdf}
  %
  \caption{%
  %
  {\bfseries \sffamily Three dimensional uniform dataset (results)}
  %
  Randomized SOM made of $4096$ neurons with a $3$-nearest neighbors induced topology. Model has been trained for $25,000$ epochs on three-dimensional points drawn from a uniform distribution on the unit cube. \textbf{A} Map topology in neural space. \textbf{B} Map codeword in neural space. Each neural voronoi cell is painted with the color of the codeword. \textbf{C to H} Normalized distance map for six samples, respectively (1,1,1), (0,0,0), (1,1,0), (1,0,0), (0,1,0) and (0,0,1) in RGB notations. Normalization has been performed for each sample in order to enhance contrast but this prevents comparison between maps.
  %
  }
  \label{fig:3D-uniform:results}
\end{figure}

\begin{figure}
     \centering
     \includegraphics[width=\textwidth]{figures/experiment-3D-uniform-analysis.pdf}
     \caption{ {\bfseries \sffamily Three dimensional uniform dataset (analysis)}
     Persistent Barcodes of \textbf{A} input space, \textbf{B} SOM, and \textbf{RSOM}.
  The blue, orange, and green line segments represent the $H0$-, $H1$-, and $H2$-homology, respectively.
  This means that blue color represents connected segments within the space and orange color reflects the holes
  within the space and the green one the voids. The longer the line segment the more important the
  corresponding topological feature. \textbf{D} illustrates the persistent diagram for the input space.
  \textbf{E} and \textbf{F} depict the persistent diagrams for SOM and RSOM, respectively. Again blue dots
  indicate $H0$-homology features, orange dots represent $H1$-homolocical features, and green the
  $H2$-homological features.}
 \label{fig:3D-uniform:analysis}
\end{figure}


\documentclass[11pt, a4paper]{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage[margin=0.6in]{geometry}
\usepackage{listings}
\usepackage{float}

\title{EE2703: Assignment 3} % Title
\author{Akilesh Kannan (EE18B122)} % Author name
\date{\today} % Date for the report

\begin{document}
    \maketitle % Insert the title, author and date

    \section{Abstract}
        In this assignment we aim to :
        \begin{itemize}
            \item Observe the error in fitting the \textit{Least Error Fit} function to a given set of data.
            \item Find the relation between the error observed and the noise in the data.
        \end{itemize}
    \section{Introduction}
        From linear algebra, we can condense any parameter estimation problem to a simple matrix equation of the form:
        \begin{equation}
            \left(\begin{matrix}
            F_1(t_1) & F_2(t_1) & ... & F_n(t_1)\\
            F_1(t_2) & F_2(t_2) & ... & F_n(t_2)\\
            ... & ... & ... & ...\\
            F_1(t_m) & F_2(t_m) & ... & F_n(t_m)\\
            \end{matrix}\right)
            \left(\begin{matrix}
            p_1\\
            p_2\\
            ...\\
            p_n\\
            \end{matrix}\right)
            =
            \left(\begin{matrix}
            a_1\\
            a_2\\
            ...\\
            a_m\\
            \end{matrix}\right)
            \label{eq0}
        \end{equation}
        where,
        \begin{equation}
            f(t;p_1, p_2,..,p_n) = \sum_{i=0}^{n}p_iF_i(t)
        \end{equation}
        is the function to be estimated, $F_i(t)$ are arbitrary functions of the variable $t$ and $p_i$ are constant parameters.\\

         Since we only have to ``fit'' the real-time data we have to a function, $F_i(t)$ are functions of our choice. Usually we make a few educated guesses for these functions, looking at the plots of these real-time data.\\

         Equation \eqref{eq0} can be written as:
        \begin{equation}
            F.\vec{p} = \vec{a_0}
            \label{eqwithoutnoise}
        \end{equation}

        But, in any real world situation, there will always be noise associated with the data. To account for that, we have to slightly modify \eqref{eqwithoutnoise} to:
        \begin{equation}
            F.\vec{p} = \vec{a_0}+\vec{n} = \vec{a}
            \label{truenoiseeqn}
        \end{equation}
        where $\vec{n}$ accounts for the noise in the data.\\

        However, the above equation cannot be satisfied exactly always (as number of equations is \textbf{\textit{N}}, but the number of measurements is \textbf{\textit{M}}).\\

        So, we make a few assumptions about the noise in the data and then try to ``best guess'' the solution, i.e., the error between the ideal solution, in the absence of noise and the one obtained has to be as minimum as possible.\\

        The assumptions we make about the noise are that it has zero mean and a standard deviation of $\sigma$.\\

        The error function $\epsilon$ is then given by:
        \begin{equation}
            \epsilon = F.\vec{p} - \vec{a}
        \end{equation}

        The norm of the error is:
        \begin{equation}
            \|\epsilon\|^2 = \sum_{i}\epsilon_i^2
        \end{equation}

        or in other words,\\
        \begin{equation}
            \|\epsilon\|^2 = \epsilon^T\epsilon = ((F.\vec{p}-\vec{a})^T(F.\vec{p}-\vec{a})) = \sum_{i}\epsilon_i^2
        \end{equation}

        On expanding the above equation and using the condition that the gradient at the minima is 0, we get:\\
        \begin{equation}
            \begin{aligned}
                2(F^TF)\vec{p_0} - 2F^T\vec{a} = 0\\
                \implies \vec{p_0} = (F^TF)^{-1}F^T\vec{a}
            \end{aligned}
        \end{equation}

        The above result is called the \textbf{\textit{Least Squares Estimate of $\vec{p_0}$ }}.\\

        However, the above result holds true only if the initial functions $F_i(t)$ are independent and that the noise is same for different measurements. This is because, we have to \textit{give lesser importance (\textbf{weight}) to those values which have more noise}, as they are more unreliable.

    \section{Procedure}
        The function to be fitted is:
        \begin{equation}
            f(t) = 1.05J_2(t)-0.105t
        \end{equation}
        where $J_2(t)$ is the \textit{Bessel Function of the first kind of Order 2}. The true data used for fitting is obtained using this equation.

        \subsection{Creating noisy data}
            To create the noisy data, we add random noise to $f(t)$. This random noise, denoted by $n(t)$, is given by the standard normal probability distribution:
            \begin{equation}
                P(n(t)|\sigma)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{n(t)^2}{2\sigma^2}} \label{eq1}
            \end{equation}
            The resulting noisy data will be of the form:
            \begin{equation}
                f(t) = 1.05J_2(t)-0.105t+n_{\sigma_i}(t)
            \end{equation}
            where, $n_{\sigma_i}(t)$ is the noisy data function with $\sigma = \sigma_{i}$ in \eqref{eq1}. Thus for 9 different values of sigma (in a log scale from 0.001 to 0.1), the noisy data is created and stored in the \texttt{\textbf{fitting.dat}} file.

        \subsection{Analyzing the noisy data}
            The data is read and plotted using PyPlot. The output result looks as follows:
            \begin{figure}[H]
                \centering
                \includegraphics[scale=0.5]{Fig 0.png}
                \caption{Noisy Data with True Data}
                \label{fig:noisyAndTrue}
            \end{figure}

            As we can see, the ``noisiness'' of the data increases with increasing value of $\sigma$. Another view of how the noise affects the data can be seen below:
            \begin{figure}[H]
                \centering
                \includegraphics[scale=0.5]{Fig 1.png}
                \caption{Noisy Data with Errorbar}
                \label{fig:noiseError}
            \end{figure}

            The blue lines (\textit{error bar}) indicate the standard deviation of the noisy data from the original data, at that value of $t$. It is plotted at every $5^\text{th}$ point to make the plot readable.

        \subsection{Finding the best approximation for the noisy data}
            From the data, we can conclude that the data can be fitted into a function of the form:
            \begin{equation}
                g(t, A, B) = AJ_2(t)+Bt
            \end{equation}
            where $A$ and $B$ are constants that we need to find.\\

            To find the coefficients $A$ and $B$, we first try to find the mean square error between the function and the data for a range of values of \textit{A} and \textit{B}, which is given by:
            \begin{equation}
                \epsilon_{ij} = \frac{1}{101}\sum_{k=0}^{101}(f(t_k) - g(t_k, A_i, B_j))^2
            \end{equation}
            where $\epsilon_{ij}$ is the error for $(A_i,B_j)$. The contour plot of the error is shown below:
            \begin{figure}[H]
                \centering
                \includegraphics[scale=0.5]{Fig 2.png}  % Mention the image name within the curly braces. Image should be in the same folder as the tex file.
                \caption{Contour Plot of $\epsilon_{ij}$}
                \label{fig:contourPlot}
            \end{figure}

            We can see the location of the minima to be approximately near the orignal function coefficients.\\

            Using the \texttt{lstsq} function in \texttt{scipy} package, we solve for:
            \begin{equation}
            M.p = D \label{eq5}
            \end{equation}
            where
            \begin{equation}
            M=\left[\begin{matrix}
            J_2(t_1)&t_1\\
            ...&...\\
            J_2(t_m)&t_m
            \end{matrix}\right]\text{, }p=\left[\begin{matrix}
            A_{fit}\\B_{fit}
            \end{matrix}\right]\ \text{and }D=\left[\begin{matrix}f(t_1)\\...\\f(t_m)\end{matrix}\right]
            \end{equation}\\

            Thus, we solve for $p$ and then find the mean square error of the values of $A_{fit}$ and $B_{fit}$ found using \texttt{lstsq} and the original values $(1.05,\ -0.105)$.

        \subsection{Finding out the variation of $\epsilon$ with $\sigma_n$}
            We solve \eqref{eq5} for different values of $\sigma_n$, by changing matrix D to different columns of \texttt{\textbf{fitting.dat}}. We find that the variation of the mean squared error of values $A_{fit}$ and $B_{fit}$ is as follows:
            \begin{figure}[H]
                \centering
                \includegraphics[scale=0.5]{Fig 3.png}  % Mention the image name within the curly braces. Image should be in the same folder as the tex file.
                \caption{Mean Squared Error vs Standard Deviation}
                \label{fig:errorSTD}
            \end{figure}

            This plot does not give that much useful information between $\sigma_n$ and $\epsilon$, but when we do the \texttt{loglog} plot as below:
            \begin{figure}[H]
                \centering
                \includegraphics[scale=0.5]{Fig 4.png}  % Mention the image name within the curly braces. Image should be in the same folder as the tex file.
                \caption{Error vs Standard Deviation \texttt{loglog} Plot}
                \label{fig:errorSTDloglog}
            \end{figure}

            We can see an approximately linear relation between $\sigma_n$ and $\epsilon$. This is the required result.

    \section{Conclusion}
        From the above procedure, we were able to determine that \textbf{the logarithm of the standard deviation of the noise} \textit{\textbf{linearly affects}} \textbf{the logarithm of the error} in the calcuation of the least error fit for a given data.
\end{document}


\documentclass{article}
\usepackage{mathrsfs}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{graphicx}
\DeclarePairedDelimiter{\ceil}{\lceil}{\rceil}

\begin{document}
\begin{center}
\textbf{\huge{Week 6}}
\end{center}

\section{Random variable source coding}

Let $\underbar{c}(x)$ be the codeword assigned to $x \in \mathcal{X}$.

$l(x)$ be the length of codeword assigned to $x$.

Here we are coding a single random variable and all codewords are binary strings. (Fixed-variable length source coding)
$$ \mathscr{C}= \{ \underbar{c}(x): x \in \mathcal{X}\}$$

$$ L_{\mathscr{C}}= \sum_{x \in \mathcal{X}}p(x)l(x)$$

Our goal is to design a code which has minimum $L_{\mathscr{C}}$, $L^{*}$
$$L^{*} = \text{min}_{\mathscr{C}} L_{\mathscr{C}}$$

\subsection{Kraft inequality}

Let $\mathscr{C}$ be any prefix-free (binary) code. Then,
$$ \sum_{x \in \mathcal{X}} 2^{-l(x)} \leq 1$$

Proof: We know that any prefix-free code can be represented using a binary tree which has `leaves' as the codeword. In the binary tree corresponding to the code, the depth of the tree would be the length of the largest codeword in the code. ($l_{\text{max}}=max_{x \in \mathcal{X}} l(x)$)\\

Suppose there is a codeword $\underbar{c}$ of length $l$ represented by a node at depth l($l \leq l_{\text{max}}$).\\

Then $\underbar{c}$ has $2^{l_{\text{max}}-l}$ successors at level $l_{\text{max}}$. Also, none of these are codewords as the code is prefix-free. Now,

\begin{align}
    \sum_{\underbar{x} \in \mathcal{X}} 2^{l_{\text{max}}-l(\underbar{x})} \leq 2^{l_{\text{max}}}
\end{align}
This is true if distinct codewords $\underbar{c1}$, $\underbar{c2}$ don't have common successors at $l_{\text{max}}$ level. Any successor of $\underbar{c1}$ at $l_{\text{max}}$ has $\underbar{c1}$ as a prefix. Similarly in the other case as well.

Suppose $l(\underbar{c1}) \leq l(\underbar{c2})$, so there can be a common successor $\underbar{v}$ at $l_{\text{max}}$.

Then, $\underbar{v}$ has first $l(\underbar{c1})$ places as $\underbar{c1}$ and first $l(\underbar{c2})$ places as $\underbar{c2}$.\\

$\Rightarrow$ $\underbar{c1}$ should be a prefix of $\underbar{c2}$ which isn't true as $\mathscr{C}$ is a prefix-free code. Hence, no pair of codewords in $\mathscr{C}$ have any common successors.

Hence $(1)$ is true. And dividing both sides by $2^{l_{\text{max}}}$ gives us the Kraft inequality.

\subsection{Lemma}
$$ L^{*} \geq H(X)$$
Any prefix-free code for $X$ has average length of at least $H(X)$.

Proof:
$$ L - H(X) =  \sum_{x \in \mathcal{X}}p(x)l(x) - \sum_{x \in \mathcal{X}} p(x) \log \frac{1}{p(x)}$$
We know,
 $$ D(p||q)= \sum_{x \in supp(P_X)}p(x)\log \frac{p(x)}{q(x)}$$
 Let
$$ q(x):= \frac{2^{-l(x)}}{\sum_{x \in \mathcal{X}} 2^{-l(x)}}   $$

\begin{align*}
    D(p_X || q_X) &= \sum_{x \in supp(P_X)}p(x)\log \frac{p(x)}{\frac{2^{-l(x)}}{\sum_{x' \in \mathcal{X}} 2^{-l(x')}}} \\
    &= - \sum p(x) \log \frac{1}{p(x)} + \sum p(x) \log \frac{1}{\frac{2^{-l(x)}}{\sum_{x' \in \mathcal{X}} 2^{-l(x')}}} \\
    &= -H(X) + \sum_{x \in supp(p_x)} p(x) \log 2^{l(x)} + \sum_{x \in supp(p_x)} p(x) \log \sum_{x'} 2^{-l(x')}  \\
    &= -H(X)+ L - \epsilon \qquad (\text{from Kraft's})\\
    & \leq - H(X)+L
\end{align*}
$$ \Rightarrow L_{\mathscr{C}} - H(X) \geq 0$$
 $$ L^{*} \geq H(X)$$

 Note: Equality happens only iff $p_x = q_x$ and $\epsilon$ is zero.
 %30/6
 \subsection{Lemma}
 Suppose we have a random variable $X \in \{ x_1, x_2, \cdots, x_k\}$ and positive integers such that $\sum_{i=1}^{k} 2^{-l_i} \leq 1$.

 Then there exists a prefix free code for $X$ with codeword lengths $l_1 , l_2 , \cdots, l_k$.\\

 Proof: We can construct a binary tree with leaves at depths $l_1, l_2, cdots, l_k$ such that none of these nodes are successors of each other, i.e. they are leaves of some binary tree (valid p-f code).

 Assume that $l_1 \leq l_2 \leq \cdots \leq l_k$, without loss in generality    , for any $i \leq k$,
 \begin{equation}
     \sum_{1}^{i-1} 2^{-l_i} <1
 \end{equation}

Imagine we take the full binary tree upto level $l_k$. At each step of the algorithm we intend to pick one available (undeleted) node from the above tree at level $l_i$ and delete all it's successors from the tree. Repeat this process for $i=1, \cdots , k$, we will then have a prefix-free tree.

We have to show that at each step $i=1, \cdots, k$, there is atleast one node left undeleted at depth $i$. We shall use observation $(2)$.

Clearly at step 1, there is a node at $l_1$, ($l_1 \geq 1$).

After $i-1$ steps, assume that we have picked nodes at level $l_1, \cdots, l_{i-1}$ and appropriately deleted. We want to show that there is a node at level $i$.

Total nodes at level $l_k$ which aren't in the tree after $(i-1)^{th}$ step
$$ = \sum_{j=1}^{i-1} 2^{l_k -l_j}$$

Hence, number of nodes remaining at level $l_k$
$$ = 2^{l_k}(1- \sum_{j=1}^{i-1}2^{-l_j})$$

$\Rightarrow$ Number of nodes remaining at $l_k$ in tree at after $(i-1)^{th}$ step $>1$.

$\Rightarrow$ At least one  survivor node should be present at level $l_i$ also. So we can pick a node for the $i^{th}$ step from level $l_i$ also. This completes the proof.

Remark: We construct the tree from smallest length to largest length codewords. Contrast this with optimal source coding we shall see Hoffman codes later.

\subsection{Example}
\includegraphics[width=\textwidth]{p-feg.png}

\subsection{}

Now, suppose that the source random variable $X \sim P_X$, we want to obtain a collection of integers $l_1, \cdots, l_k$ such that Kraft inequality is satisfied, then we know how to get the code for $X$.
$$ \sum_{i=1}^{k} 2^{-l} \leq 1$$
\begin{itemize}
    \item Suppose all codewords are of the same length, $$ k 2^{-l} \leq 1 \Rightarrow l \geq \log k$$

    Hence we can pick $l = \ceil{\log k}$ (ceil function). But there is no guarantee that this code is `good', it may not have small average length.

    \item We know average length $= \sum p_i l_i$.

    Then we will choose small $l_i$ for larger $p_i$. (while taking care that it satisfies Kraft inequality)
\end{itemize}

\section{Shannon-Fano code}

We fix,
\begin{equation}
    l_i = \ceil{ \log_2 \frac{1}{p_i}}
\end{equation}

 where $p_i$ is the probability of $X$ taking the $i^{th}$ value in $\mathcal{X}$.

Clearly $l_i \geq 1$.

\begin{align*}
    \sum_{i=1}^k 2^{-l_i} &= \sum_{i=1}^k 2^{-\ceil{\log_2 \frac{1}{p_i}}}  =\leq \sum_{i=1}^{k} 2^{- \log_2 \frac{1}{p_i}} \\
    &= \sum_{i=1}^k p_i = 1
\end{align*}

The lengths given by $(3)$ satisfy Kraft inequality. We can use the tree-pruning algorithm (see sec 1.3) to get a prefix-free code for $X$.

This code obtained is called as the Shannon-Fano code.

Now,

\begin{align*}
    L_{\text{Shannon-Fano}} &= \sum_{i=1}^{k} p_i \ceil{\log_2 \frac{1}{p_i}} \\
    &< \sum_{i=1}^{k} p_i \left( \log_2 \frac{1}{p_i} +1 \right) \\
    &< H(X)+1
\end{align*}

But S-F code is not always an optimal length prefix-free code.

\subsection{Example for Shannon-Fano code}
$$ X \in \mathcal{X}= \{ x_1,x_2,x_3,x_4 \}$$
\begin{equation*}
        P_X (x_i)=
        \begin{cases}
          1/4, & \text{if}\ i=1 \\
          1/2, & \text{if}\ i=2 \\
          1/9, & \text{if}\ i=3 \\
          5/36, & \text{if}\ i=4
        \end{cases}
    \end{equation*}

Lengths satisying the Kraft's inequality for the Shannon-Fano code
$$ l_i= \ceil{\log_2 \frac{1}{P_X (x_i)}}= \begin{cases}
  2, & \text{if}\ i=1 \\
  1, & \text{if}\ i=2 \\
  4, & \text{if}\ i=3 \\
  3, & \text{if}\ i=4
\end{cases} $$

$$ \bar{L}= \sum_{i=1}^{4} p_i l_i = \frac{57}{36}$$

Obtaining the Shannon-Fano code:
\begin{itemize}
    \item Arrange lengths in ascending order: $l_2 \leq l_1 \leq l_4 \leq l_3$. Hence we pick off a node at depth 2, then 1, 4 and at 3 and delete all the successors.

    \item Now we can choose the codewords from the binary tree, say
    $$ x_1 \to 01 \qquad x_2 \to 1 \qquad x_3 \to 0010 \qquad x_4 \to 000$$
    Hence, the prefix-free Shannon-Fano code is $\{ 01,1,0010,000\} $.
\end{itemize}

\section{Huffman coding}
We will show an optimal code construction that has smaller average length compared to the Shannon-Fano code.

\subsection{Lemma's}

Let us see some intuitive lemma's about an optimal code for a random variable $X$:

Assume that $X \in \mathcal{X}= \{ x_1, \cdots, x_k\}$.

$P_X(x_i)= p_i, i=1, \cdots , k$, without loss in generality, let $p_1 \geq \cdots \geq p_k$.
\begin{enumerate}
    \item Consider that $l_1, \cdots, l_k$ are the lengths of the codewords associated to the messages $x_i, i=1,\cdots, k$ respectively in any optimal code for X.

    Then,
    \begin{equation}
        l_1 \leq \cdots \leq l_k
    \end{equation}

    Proof: Suppose $\mathscr{C}$ is an optimal code in which $\exists $ some distinct $i,j \in \{1,\cdots, k \}$ such that $p_i > p_j$ but $l_i > l_j$. (Assumption of the contrary)

    We shall show that there is another code $\mathscr{C}'$ which has smaller average length $L_{\mathscr{C'}}$ than $L_{\mathscr{C}}$.

    Cosider the code $\mathscr{C}'$ in which the codewords for $x_i$ \& $x_j$ are swapped between those in $\mathscr{C}$.
    $$ \underbar{c}_{\mathscr{C'}}(x_j)=\underbar{c}_{\mathscr{C}}(x_i) \qquad \underbar{c}_{\mathscr{C'}}(x_i)=\underbar{c}_{\mathscr{C}}(x_j)$$

    Now in $\mathscr{C}$,

    $$l_{i, \mathscr{C}} :=  \text{length of codeword of } x_i \text{ in }\mathscr{C}= l_j$$
$$l_{i, \mathscr{C'}}= l_i$$

So, $l_{i, \mathscr{C'}}$ is having the same codewords as $\mathscr{C}$ and hence is also prefix-free.
$$ \bar{L}_\mathscr{C'}= \sum_{k= \{ 1, \cdots, k\} \setminus \{ i,j\}}  p_k l_k+ p_i l_i + p_j l_j$$
$$ \bar{L}_\mathscr{C}= \sum_{k= \{ 1, \cdots, k\}} p_k l_k$$
$$ \bar{L}_\mathscr{C'} - \bar{L}_\mathscr{C} = p_i(l_j -l_i)+ p_j(l_i -l_j) =(p_i-p_j)(l_j -l_i) $$

We have $p_i > p_j$, but $l_i > l_j$,
$$ \Rightarrow  \bar{L}_\mathscr{C'} - \bar{L}_\mathscr{C} < 0 \Rightarrow  \bar{L}_\mathscr{C'} < \bar{L}_\mathscr{C}$$
$\Rightarrow$ $\mathscr{C}$ is not optimal, thus this a contradiction. $(4)$ has to happen for an optimal code.

\item Consider the tree representation of a code which is optimal code. In such a tree, it should be true that either each node is a codeword or it has at least 2 successors which are codewords. i.e. There are no unused leaves in the tree of an optimal code.
%3/7

Proof: Let $\mathscr{C}$ be the optimal code. Let us consider the tree corresponding to this, suppose it has an unused leaf. Because the code is optimal, these unused leaves must be at maximum depth in the tree. Then, for at least one value $x_i$ of $\mathscr{C}$, we have the situation

\includegraphics[width=\textwidth]{huffman.png}

In either case, we can delete the last digit of the codeword for $x_i$ (without changing the other codewords) and still have a prefix-free code. But the new code has smaller $\bar{L}$ and thus the original code could not have been optimal.

\item There is an optimal prefix-free code for random variable $X$ such that the codewords associated to the two smallest probability symbols are siblings (share the same parent). i.e. two least likely codewords differ only in their last digit.

Proof: Let $\mathscr{C}$ be some optimal code for random variable. Suppose $\mathscr{C}$ already satifies the property specified above, then we are done.

Suppose $\mathscr{C}$ doesn't satify the property, we know:

$$ l_{k-1}\leq l_k \qquad as \qquad p_{k-1} \geq p_k$$

\includegraphics[width=\textwidth]{lemma3.png}

Note that $l_i = l_k$ and $i \neq k-1$.

$$ \Rightarrow l_i \leq l_{k-1} \leq l_k = l_i  \qquad \text{from Lemma-1}$$

$$ \Rightarrow l_i = l_{k-1}= l_k$$

Interchange the codewords for the $i^{th}$ and $(k-1)^{th}$ symbols and get a new code $\mathscr{C'}$.
$$ \Rightarrow \bar{L}_\mathscr{C} = \bar{L}_\mathscr{C'}$$

Now $\mathscr{C'}$ satisfies the property in Lemma 3.

\end{enumerate}

\subsection{Huffman coding}

Say we have $p_X$, assume that $p_1 \geq \cdots \geq p_k$. Unlike Shannon-Fano code, here we are building the tree in reverse. We pick $p_{k-1}$ and $p_k$ as they have the least 2 probabilities.

These are assigned to siblings in the tree and now the parent of them has probability $p_{k-1}+ p_k$.

Now we can form a new distribution using these $k-1$ probabilities (symbols) i.e. $p_1, \cdots, p_{k-2}, p_{k-1}+ p_k$. Now, if $\{ p_{k-2}, p_{k-1}+ p_k \}$ are the least 2 probabilities, a new node is assigned to $p_{k-2}$.
We repeat the process until all codewords are assigned.

\subsection{Example}

\includegraphics[width=\textwidth]{huffmaneg.png}

\end{document}


\section{Applications of spectral theory}

\begin{outcome}
  \begin{enumerate}
  \item Use diagonalization to find a high power of a matrix.
  \item Use diagonalization to solve dynamical systems.
  \end{enumerate}
\end{outcome}


\section{Conclusion}

Improving renewable energy forecasting is important for grid-planning
and unit commitment, especially as the share of variable renewable resources
increases, challenging grid stability. We first
demonstrated that our implementation of the \gls{esn} algorithm is consistent
with the literature. Then, we used it to predict
total demand, solar energy, and wind energy. Finally, we evaluated the
influence of several meteorological factors on prediction accuracy. Our results
show that researchers must carefully choose each additional training input to
avoid increasing the system complexity. Our results also indicate that properly
chosen training inputs can achieve comparable accuracy to more complex
algorithms. The conventional \gls{esn} used here did not demonstrate an
improvement over the state-of-the-art, nor was it accurate enough to improve
grid-scale energy economy. Future work will explore other applications of
\glspl{esn} and pursue improvements to the model algorithm.


\documentclass[11pt, letterpaper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{bm}

\newcommand{\e}{\epsilon}
\newcommand{\dl}{\delta}
\newcommand{\dij}{\delta_{ij}}
\newcommand{\mdij}{\delta^i_j}
\newcommand{\1}{\bm{1}}
\newcommand{\gd}{\dot \gamma}
\newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\uu}[1]{\underline{\underline{#1}}}
\newcommand{\vect}[1]{\underline{#1}} %\vect can be changed to make vectors appear bold or underlined or with arrow on top
\newcommand{\p}[1]{\grave{#1}} %\p is for primed variables. The accent on top can be changed to \hat \tilde etc.

\title{Lecture 7: Beyond Cartesian Tensors}
\begin{document}
\maketitle

In general the components of a vector maybe measured parallel to the axes or perpendicular to it. This distinction leads to two equivalent representations of a vector. If the components are measured parallel to the axes, the vector is said to be contravariant. Contravariant vectors are denoted by a superscript index as $x^i$. If the components are perpendicular to the axes, it's called a covariant vector. These are written with a subscript index $x_i$. The crucial difference between these two types of vectors (two representations of a vector) lies in the transformation rules for these vectors. These can be seen as follows.

\section{Transformation rules}
Assume that we want to transform a certain vector from an old coordinate system $(x_1, x_2, x_3)$ to a new coordinate system $(\p x_1, \p x_2, \p x_3)$. The new coordinates can be expressed as some functions of the old ones. Hence we have
\begin{align*}
\p x_1 &\equiv \p x_1(x_1, x_2, x_3)\\
\p x_2 &\equiv \p x_2(x_1, x_2, x_3)\\
\p x_3 &\equiv \p x_3(x_1, x_2, x_3)
\end{align*}

Then, using chain rule, we can write the components of vector $\vect{d\p x}$ as

\begin{align*}
d\p x^1 &= \pd{\p x^1}{x^1}dx^1+\pd{\p x^1}{x^2}dx^2+\pd{\p x^1}{x^3}dx^3\\
d\p x^2 &= \pd{\p x^2}{x^1}dx^1+\pd{\p x^2}{x^2}dx^2+\pd{\p x^2}{x^3}dx^3\\
d\p x^3 &= \pd{\p x^3}{x^1}dx^1+\pd{\p x^3}{x^2}dx^2+\pd{\p x^3}{x^3}dx^3
\end{align*}

or, as per Einstein's summation convention, we can condense these equations as
$$
d\p x^i=\pd{\p x^i}{x^j}dx^j
$$
Therefore, in general, for any contravariant vector we get
$$
\p a^i=\pd{\p x^i}{x^j}a^j
$$
which is the transformation rule for contravariant vectors (and hence the superscript indices). More precisely, we say that vectors which transform this way are called contravariant vectors. This can be generalized to contravariant tensors as well
$$
\p A^{ij}=\pd{\p x^i}{x^k}\pd{\p x^j}{x^l}A^{kl}
$$

The key idea is this: Let there be a vector denoted as $\vect{a}$ in the old coordinate system and $\vect{\p a}$ in the new coordinate system. Then, for contravariant tensors, going from old coordinate representation to the new one ($\vect{a} \rightarrow \vect{\p a}$), requires \textit{differentiating the new coordinates with respect to the old ones} ($\p x$ with respect to $x$).

There is another species of vectors which behave differently from the contravariant vectors. This difference is because the differentials of coordinates ($dx_1$ $dx_2$ $dx_3$) appear in the numerator in vectors like $\vect{dx}$, velocity vector $\vect{u}$ or acceleration vector $\vect{a}$, but they appear in the denominator for the gradient vector $\nabla \phi = (\pd{\phi}{x_1},\pd{\phi}{x_2},\pd{\phi}{x_3})$ for some scalar $\phi$. How does such a vector transform when we go from an old (unprimed) to a new (primed) coordinate system? Recall that
\begin{align*}
\p x_1 &\equiv \p x_1(x_1, x_2, x_3)\\
\p x_2 &\equiv \p x_2(x_1, x_2, x_3)\\
\p x_3 &\equiv \p x_3(x_1, x_2, x_3)
\end{align*}
Then, using chain rule,
\begin{align*}
\pd{\phi}{\p x^1} &= \pd{\phi}{x^1}\pd{x^1}{\p x^1}+\pd{\phi}{x^2}\pd{x^2}{\p x^1}+\pd{\phi}{x^3}\pd{x^3}{\p x^1}\\
\pd{\phi}{\p x^2} &= \pd{\phi}{x^1}\pd{x^1}{\p x^2}+\pd{\phi}{x^2}\pd{x^2}{\p x^2}+\pd{\phi}{x^3}\pd{x^3}{\p x^2}\\
\pd{\phi}{\p x^3} &= \pd{\phi}{x^1}\pd{x^1}{\p x^3}+\pd{\phi}{x^2}\pd{x^2}{\p x^3}+\pd{\phi}{x^3}\pd{x^3}{\p x^3}
\end{align*}
or, in index notation,
$$
\pd{\phi}{\p x^i} = \pd{x^j}{\p x^i}\pd{\phi}{x^j}
$$

Then, instead of $\pd{\phi}{x^j}$, if there was a general covariant vector $a$, we would get
$$
\p a_j = \pd{x^i}{\p x^j} a_i
$$
This can be generalized for a second order tensor
$$
\p a_{ij} = \pd{x^k}{\p x^i}\pd{x^l}{\p x^j} a_{kl}
$$
In essence, for a covariant vector $\vect{a}$, going from the old to the new coordinate system requires \textit{differentiation of the old coordinates with respect to the new ones} and therein lies the key difference. Finally, we can also note the transformation rule for mixed tensors
$$
\grave{a}^{i}_j = \pd{x^k}{\grave{x}^i}\pd{\grave{x}^j}{x^l} a^{k}_l
$$

\section{Index contraction and the metric tensor}
Knowing how the vectors transform under coordinate transformations, we can show an important result: \textit{Contraction of indices always occurs between a contravariant and a covariant index}. Two contravariant or two covariant indices cannot contract amongst each other.

To show this, let there be a scalar $S=a^ib_i$. Being a scalar, it will be invariant to coordinate transformations. Hence we must also have $S=\p a^i\p b_i$. Let's assume that $\p a^i\p b_i=\p S$ for some $\p S$. Then,
$$
\p S=\p a^i\p b_i=\pd{\p x^i}{x^k}a^k\pd{x^l}{\p x^i}b_l=\pd{x^l}{x^k}a^k b_l=\dl^l_k a^k b_l = a^kb_k = S
$$
Note that the key step in this process is
$$
\pd{\p x^i}{x^k}a^k\pd{x^l}{\p x^i}b_l=\pd{x^l}{x^k}a^k b_l
$$
This is only possible because one of the contracting vectors is contravariant and the other is covariant so that one of the primed coordinate can cancel the other. Hence, contractions can only occur between two different types of vectors. In general, this idea can be used to lower the indices of higher order tensors as we will soon see. Another thing to note is that $\mdij$ is, in fact, a mixed second order tensor- it has one contravariant and one covariant index.

This idea allows us to define the metric tensor. To begin with, consider the expression for a short length $ds$ in Cartesian coordinate system
$$
ds^2= d{x_1}^2+d{x_2}^2+d{x_3}^2
$$
For cylindrical coordinates $(x_1=r, x_2=\phi, x_3=z)$, the expression for $ds^2$ is given by
$$
ds^2= d{r}^2+r^2d{\phi}^2+d{z}^2
$$

Here $r$ is the \textit{metric factor}. It converts the coordinate differential $d\phi$ to a small distance $r d\phi$ in cylindrical coordinates. More generally, in an arbitrary coordinate system, we can write
$$
ds^2= h_1^2d{x_1}^2 + h_2^2d{x_2}^2 + h_3^2d{x_3}^2
$$
where $h_1,h_2,h_3$ are the metric factors corresponding to each of the coordinates. Yet, this is not the most general form for $ds^2$. This is the most general form for the metric \text{in an orthogonal coordinate system}. If the coordinate system is non-orthogonal, the form for the metric is given by a more general expression
$$
ds^2 = g_{ij}dx^idx^j
$$

For the Cartesian coordinates, $g_{ij} = \dij$. For cylindrical coordinates, $g_{ij}$ is such that $\{g_{11} = 1(=h_1^2),g_{22} = r(=h_2^2),g_{33} = 1(=h_3^2)\}$ and the rest of the terms are $0$. The fact that $g_{ij}$ is diagonal is an artifact of the orthogonality of the coordinate system. If the coordinate system is non-orthogonal, then the off-diagonal terms can also be non-zero. How does $g_{ij}$ transform upon change of coordinates? Let there be a metric $ds^2$ defined in the $\p x$ coordinate system as
$$
ds^2 = \p g_{ij} d\p x^i d \p x^j
$$
Now, in order to migrate to the old coordinates, we impose the transformation rules for contravariant vectors ($ds^2$, being a scalar, remains invariant to coordinate transformations)
\begin{align*}
ds^2 &= \p g_{ij}\pd{\p x^i}{x^k}dx^k \pd{\p x^j}{x^l}dx^l\\
\Rightarrow ds^2 &=\p g_{ij}\pd{\p x^i}{x^k}\pd{\p x^j}{x^l} dx^k dx^l
\end{align*}
Comparing this equation with the definition of metric in the old coordinate system $ ds^2 = g_{kl} dx^k dx^l $ we get the transformation rule for $g$ as
$$
g_{kl} = \pd{\p x^i}{x^k}\pd{\p x^j}{x^l} \p g_{ij}
$$
Similarly we can obtain the inverse transformation as
$$
\p g_{ij} = \pd{x^k}{\p x^i}\pd{x^l}{\p x^j}g_{kl}
$$
Therefore, allowing us to find the metric tensor for the new space from the knowledge of metric tensor for the old space. Specifically, if the old coordinate system is Cartesian, then $g_{kl} = \dl_{kl}$. Let's denote the Cartesian coordinates with $y_i$'s  and the new coordinate system with $x_i$'s. Then, the transformation becomes
\begin{align*}
g_{ij} &= \pd{y^k}{x^i}\pd{y^l}{x^j}\dl_{kl} \\
&=\pd{y^k}{x^i}\pd{y^k}{x^j}
\end{align*}
This will allow us to obtain the metric tensor when we go from Cartesian to a general coordinate system.

Let us also define an inverse contravariant metric tensor $g^{ij}$ such that
$$
g_{ik}g^{kj}=\mdij
$$

This will allow us to perform operations like lowering and raising of indices i.e. to transform a covariant vector into its contravariant representation and vice versa.
\begin{align*}
a_i &= g_{ij}a^j\\
a^i &= g^{ij}a_j
\end{align*}

With this formalism, we can consider a simple application.

\section{The Convection-Diffusion equation}

Let's say we have simple shear flow in a channel. Let's define two coordinate systems- the Cartesian coordinates which are fixed in space ($y_i$'s) and the convected coordinates which deform with the flow ($x_i$'s). Then, we have
\begin{align*}
u_1 &= \gd y_2\\
u_2 &= 0 \\
\Rightarrow y_1 &= \gd y_2 t + c_1 \\
y_2 &= c_2 \\
\Rightarrow c_1 &= y_1 - \gd t y_2 \\
c_2 &= y_2
\end{align*}
$c_1$ and $c_2$ are in fact the convected coordinates. At $t=0$, $c_1=y_1, c_2=y_2$. For later times, $c_1$ remains $y_1$ while $c_2$ convects with the flow. Therefore, our convected coordinates become
\begin{align*}
x_1 &= y_1 - \gd t y_2 \\
x_2 &= y_2
\end{align*}

Additionally, we can also find the new basis vectors ($\vect g_1,\vect g_2$) using transformation law for vectors
\begin{align*}
\p a_j &= \pd{x^i}{\p x^j} a_i\\
\Rightarrow g_i &= \pd{y^j}{x^i} \vect{1}_j\\
\Rightarrow g_1 &= \vect{1}_1\\
 g_2 &= \gd t\vect{1}_1+\vect{1}_2
\end{align*}

Using chain rule, we can express the derivatives in the new coordinates
\begin{align*}
\pd{}{y_1} &= \pd{}{x^1}\\
\pd{}{y_2} &= -\gd t \pd{}{x_1} + \pd{}{x_2}
\end{align*}

Now, the convection-diffusion equation equation, in Cartesian coordinates, is
$$
\pd{T}{t} + u_\alpha\pd{T}{y_\alpha} = \kappa \pd{^2T}{y_\alpha y_\alpha}
$$
Using the expressions for shear-flow velocity profile
$$
\pd{T}{t} + \gd y_2 \pd{T}{y_1} = \kappa \bigg(\pd{^2T}{y_1^2}+\pd{^2T}{y_2^2}\bigg)
$$
Expressing the derivatives in the new coordinates, we finally get the convection-diffusion equation in convected coordinates
\begin{align*}
\pd{T}{t} &= \kappa \bigg((1+(\gd t)^2)\pd{^2T}{x_1^2}+\pd{^2T}{x_2^2} - 2\gd t \pd{^2T}{x^1 \partial x^2} \bigg) \\
\end{align*}

What does this equation tell us? Notice that there is no convection term anymore. We only have the first time derivative proportional to the second order spatial derivatives which is a telltale sign of pure diffusion. Therefore, in the convected coordinates, the convection-diffusion equation appears as just the diffusion equation. But there is one important difference. The last term includes partial derivatives with respect to both $x_1$ and $x_2$. Presence of cross-derivatives is an artifact of non-orthogonality in the coordinate system. As time increases, the coefficient of this term increases in magnitude and hence the non-orthogonality in the coordinate system.

It also makes intuitive sense. Since, the coordinates move with the flow, an observer moving with the coordinates will not perceive any flow.

\subsection{Method II}

In the previous section we arrived at the result using just the coordinate transformation. Now we arrive at the same result using the isotropy of metric tensor. The heat equation is given by $\pd T t = \kappa \pd {q^i}{x^i}$. Using Fourier's law ($q^i=-\kappa g^{ij} \pd {T} {x^j}$), it becomes
\begin{align*}
\pd{T}{t} &= \kappa \pd{}{x^i}\big(g^{ij} \pd{T}{x^j}\big)
\end{align*}

Note that heat flux is a contravariant vector. This is because its contraction with a covariant vector, $\nabla \cdot q$ is a scalar. Since gradient vector $\pd{T}{x^j}$ is obviously a covariant vector, this implies that we need an isotropic second order contravariant tensor to carry out the transformation. Hence, $g^{ij}$ is a contravariant tensor. Now all we need to do is find $g^{ij}$.

Firstly, we find the elements of the metric tensor $g_{ij}$
\begin{align*}
g_{ij} &= \sum^{2}_{\alpha=1} \pd{y^k}{x_i}\pd{y^k}{x^j}\\
\Rightarrow g_{11} &= 1\\
g_{22} &= 1 + (\gd t)^2\\
g_{12} &= g_{21} = \gd t
\end{align*}
Using $g_{ik}g^{kj}=\mdij$, we can find the contravariant metric tensor $g^{ij}$. Its elements turn out to be
\begin{align*}
g^{11} &= 1 + (\gd t)^2\\
g^{22} &= 1\\
g^{12} &= g^{21} = -\gd t\\
\end{align*}

Substituting the values in the heat equation, we get exactly the same equation as before
\begin{align*}
\pd{T}{t} &= \kappa \bigg((1+(\gd t)^2)\pd{^2T}{x_1^2}+\pd{^2T}{x_2^2} - 2\gd t \pd{^2T}{x^1 \partial x^2} \bigg) \\
\end{align*}

However, there is a glitch in this process. In order to carry out the outer differentiation in the term $\kappa \pd{}{x^i}(g^{ij} \pd{T}{x^j})$, we need the covariant derivative. A covariant derivative is a form of differentiation that, besides considering the change in components of the vector, also accounts for change in the direction of unit vectors as we traverse a small distance in the space. It would not matter in the above example because the metric tensor $g^{ij}$ does not vary with space. Thus, at any given time, the direction of unit vectors is same at all points in the space. But for a general metric, we must use the covariant derivative to carry out the differentiation in the Fourier's law.

The above said differentiation requires us to differentiate a contravariant tensor and a covariant vector. Once we incorporate the laws for covariant derivative of the contravariant tensor and covariant derivative of the covariant vector, we can write down the heat-equation in a general, non-orthogonal coordinate system.

\end{document}