\documentclass{tufte-handout}
\title{Applied Algorithms\newline Mandatory Assignment \#$2$}

\usepackage[utf8]{inputenc}

\usepackage{graphicx} % for handling images

\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{microtype}

\usepackage{pgfplots} % layout and margin handling
\pgfplotsset{width=10cm,compat=1.13}

\usepackage{booktabs}
\usepackage{tcolorbox}

% Graphics
\usepackage{graphicx}
\usepackage{subfig}

% Code formatting
\usepackage{listings}
\lstset{
   language=java,
   extendedchars=true,
   basicstyle=\footnotesize\ttfamily,
   showstringspaces=false,
   showspaces=false,
   numbers=left,
   numberstyle=\footnotesize,
   numbersep=9pt,
   tabsize=2,
   breaklines=true,
   showtabs=false,
   frame=single,
   extendedchars=false,
   inputencoding=utf8,
   captionpos=b
}

\begin{document}

\thispagestyle{empty}

\maketitle
\includegraphics[width=\textwidth]{figs/logo_en.png}

\vspace{10mm}\noindent %10mm vertical space

\begin{center}
{\LARGE Distinct elements using hashing}
\end{center}

\vspace{5mm}\noindent %10mm vertical space

\begin{table}[!h]
\centering
\begin{tabular}{cc}
\multicolumn{2}{c}{Authors}                                   \\ \hline
\multicolumn{1}{c|}{\textbf{Hugo Brito}} & \textbf{René Haas} \\
\multicolumn{1}{c|}{\href{mailto:hubr@itu.dk}{hubr@itu.dk}}         & \href{mailto:renha@itu.dk}{renha@itu.dk}       \\ \hline
\end{tabular}
\end{table}

\vspace{5mm} %10mm vertical space

%\newpage

\section{Introduction}

In this report we show our findings with the HyperLogLog algorithm for estimating distinct elements.
All our experiments are reproducible and can be rerun by using the {\tt Makefile}. If the reader wishes to rerun our experiments simply {\tt cd} to the {\tt src} folder and type {\tt make}.

\section{\textbf{Exercise 1.}}
In this first exercise we are given a matrix $A\in\{0,1\}^{b\times k}$ where $b=k=32$. We are asked to implement the hash function
\begin{align*}
  h(x)_i = \sum_{j=1}^b A_{i,j}x_j \mod 2
\end{align*}
We implement the function in Java (see {\tt Ex1.java})
with the following code:

\begin{lstlisting}
public static int h(int x) {
    int res = 0;
    for (int i = 0; i < A.length; i++) {
        res += (Integer.bitCount(A[i] & x) % 2) << (31-i);
    }
    return res;
}
\end{lstlisting}
\noindent We start by calculating the bit-wise {\tt \&} (and) operation between $x$ and the rows of $A$. The result of this operation is a binary number which is equivalent to a vector consisting of the element-wise products of the components of $x$ and the $i^{th}$ row of $A$. We then count the sum with {\tt Integer.bitCount} and then take the modulus $2$ of that number. This gives us an integer which is either $0$ or $1$. We make this the integer the $j$ bit in the result by using the shift operator {\tt <<}. This implementation passes the tests on CodeJudge.

\section{\textbf{Exercise 2.}}

We are asked to implement the function $\rho(y) = \min\{i\ |\ y_{k-i}=1\}$. We do this the easy way in Java by simply using the {\tt Integer.numberOfLeadingZeros} function from the Java standard library and adding $1$ to the result. We add $1$ because we are interested in the first position that is not $0$. The implementation can be found on the {\tt Ex2.java} file and looks like the following:

\begin{lstlisting}
public static int rho(int x) {
    if (x == 0) { throw new InputMismatchException("Zero is undefined"); }
    return Integer.numberOfLeadingZeros(x);
}
\end{lstlisting}
\noindent
It is claimed that the distribution of hash values of $\rho$ satisfies $Pr(\rho(y)=i) = 2^{-i}$.
In Figure \ref{fig:ex2} we show a histogram over the values of $\rho(x)$ for $x \in \{1,\cdots,10^6\}$. We find that there is direct experimental support for this claim, although in reality $Pr(\rho(y)=i)=2^{-i}/2$. This is verified experimentally in Fig \ref{fig:ex2} and the factor of $1/2$ is also theoretically necessary since $\sum_{i=1}^{\infty} = 2$ and a probability distributions must be normalised.

\begin{figure}[h!]
  \includegraphics[width = \textwidth]{figs/ex2-hist}
  \caption{Histogram of $\rho(x)$ for $x \in \{1,\cdots,10^6\}$}
  \label{fig:ex2}
\end{figure}

\section{\textbf{Exercise 3.}}
% Exercise 3. Implement this algorithm with m = 1024 and k = 32. Test it on the input sequence 106; : : : ; 2  106 �� 1 with 1 million distinct items: How many distinct elements are reported by your implementation? (Add this number to your report.)

We have implemented the algorithm and this implementation for the values $m = 1\ 024$ and $k = 32$ can be found in {\tt Ex3.java} file. This file also passes all instances of CodeJudge.
\\\noindent
For the given $m$, $k$ and the given sequence of distinct integers $x \in \{10^6,\cdots,2\times 10^6 - 1\}$ the implementation of this algorithm estimates that there are $973\ 089,272$ distinct values. The error\footnote{Taken from \href{https://www.thoughtco.com/how-to-calculate-percent-error-609584}{thoughtco.com}} is then $|1\ 000\ 000 - 973\ 089,272| = 26\ 910,728$, which translates to $2,69\%$.

\section{\textbf{Exercise 4.}}

% As alluded to earlier, the space usage of the HyperLogLog counter influences its estimation error. In the last exercise, we want to experimentally find out how strong this influence is.

% Write an input generator that takes as input an integer n and a seed seed on standard input and outputs a list of n random 32-bit integers from a random integer generator using seed seed. Describe and run an experiment that gives a graphical representation of the connection of m and the estimation error. A recommended representation is a histogram plot over the distinct element count reported by the algorithm. Try at least 3 dierent values of m.

% we have the table and want to correlate the number of times for each m that an estimation falls inside the first sigma, the second sigma, or none. we have the columns that report true or false for that.

% we can make a graph out of that table

\noindent Lastly, we wanted to find the co-relation between the space usage of the HyperLogLog counter influences its estimation error.\\
\bigskip \noindent
The file {\tt Ex4.java} implements the input generator to be used by the HyperLogLog algorithm. We wanted to be certain that the algorithm would always estimate the number of distinct integers and compare that estimation to the real number of distinct integers. In order to achieve this machinery, the above-mentioned generator would add the generated integers to set. Below the code snippet that implements this description:

\begin{lstlisting}
do {
	int i = a + random.nextInt(-a + b + 1);
	if (i != 0) { // zero cannot be used because it has no leading zeros and cannot be hashed
		integers.add(i);
    // omitted for brevity
	}
} while (integers.size() < n);
\end{lstlisting}
\noindent
Variable description:
\begin{itemize}
    \item {\tt a}: The lower bound of the random generator
    \item {\tt b}: The upper bound of the random generator
    \item {\tt integers}: The set of distinct integers
    \item {\tt n}: The real number of distinct integers
\end{itemize}
\noindent
The file {\tt Experiment.java} implements experiment we designed to assess the relation between the $m$ and the estimation. It takes a set of distinct integers and $m$ as constructor parameters and outputs all the information needed to report on this relation, namely:
\begin{itemize}
    \item The estimation
    \item A {\tt boolean} value that would evaluate to {\tt true} should the estimation fall within one $\sigma$ range
    \item A {\tt boolean} value that would evaluate to {\tt true} should the estimation fall within $2\sigma$ range
    \item $\sigma$
    \item The lower and upper bounds of the $\sigma$ range
    \item The lower and upper bounds of $2\sigma$ range
\end{itemize}
\noindent
Finally, the file {\tt main.java} combines all the above and outputs the results to a file. In itself, it contains a {\tt long} generator that was initialised with the {\tt seed} $42$. The output of this {\tt long} generator is then used as {\tt seed} in the {\tt Ex4.java} file to generate sets of distinct integers. Once the set is created, $3$ {\tt Experiment} are instantiated with different values of $m\in\{16, 256, 1024\}$. In each iteration a new set is generated with an unused {\tt seed} (we garantee this by keeping a set {\tt usedSeeds}) this is passed to the $3$ instances. In order to achieve significant data, this process is repeated $100\ 000$ times.

\begin{figure}[!h]
  \includegraphics[width = \textwidth]{figs/ex416}
  \caption{HiperLogLog estimation distribution for $m = 16$ over $100\ 000$ distinct sets of $1\ 000\ 000$ distinct integers}
  \label{fig:16}
\end{figure}
\noindent In figures \ref{fig:16}, \ref{fig:256} and \ref{fig:1024} we see histograms that resulted from these experiments. In the figures the yellow vertical lines represent $n(1\pm\sigma)$ range and the red lines represent $n(1\pm2\sigma)$ range. We have also fitted a Gaussian distribution and plotted that on top of the histogram in orange.
\begin{figure}[h!]
  \includegraphics[width = \textwidth]{figs/ex4256}
  \caption{HiperLogLog estimation distribution for $m = 256$ over $100\ 000$ distinct sets of $1\ 000\ 000$ distinct integers}
  \label{fig:256}
\end{figure}
\begin{figure}[h!]
  \includegraphics[width = \textwidth]{figs/ex41024}
  \caption{HiperLogLog estimation distribution for $m = 1\ 024$ over $100\ 000$ distinct sets of $1\ 000\ 000$ distinct integers}
  \label{fig:1024}
\end{figure}
% \clearpage

\noindent We can see that the error distribution follows a normal Gaussian distribution. In Table \ref{fractions} we see the fraction of runs that fall inside $n(1\pm\sigma)$ and $n(1\pm2\sigma)$ for the three different values of $m$ respectively.

\begin{table}
  \centering
  \begin{tabular}{ l | c | c | c}
                     & $m = 16$ & $m = 256$ & $m = 1024$\\ \hline
  $n(1\pm\sigma)$    & $69,453\%$ & $68,596\%$ & $68,521\%$ \\ \hline
  $n(1\pm 2\sigma)$   & $95,042\%$ & $95,476\%$ & $95,523\%$ \\ \hline
  \end{tabular}
  \caption{Fractions of runs that fall inside $n(1\pm\sigma)$ and $n(1\pm2\sigma)$ for different $m$s}
  \label{fractions}
\end{table}

\section{\textbf{Conclusion}}

The HyperLogLog algorithm is an effective strategy to estimate the cardinality of a set of integers as it does not require to keep track of the integers, saving memory space.
\noindent We could also see that  allocating to space (buckets) by increasing the size of $m$ does not influence the distribution of the error, which always follows the Gaussian distribution. It is important to mention that being $\sigma$ a function of $m$, decreasing $m$ will also increase the error range, resulting in estimations that are further away from the real value in absolute terms. Se there is some gain in for instance increasing $m$ from $16$ to $256$, but this gain is less expressive from $256$ to $1024$.

\end{document}


\documentclass{article}
    % General document formatting
    \usepackage[margin=0.7in]{geometry}
    \usepackage[parfill]{parskip}
    \usepackage[utf8]{inputenc}
    \usepackage{mathrsfs}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{tikz}
    \usepackage{fancyhdr}
    \usepackage{multicol}

    \usetikzlibrary{positioning}

\pagestyle{fancy}
\fancyhf{}
\rhead{Edgar Jacob Rivera Rios - A01184125}

\renewcommand{\labelenumi}{\alph{enumi})}

\begin{document}
\section*{3.1.1}
A survey question for a sample of 150 individuals yielded 75 \textbf{Yes} responses, 55 \textbf{No} responses, and 20 \textbf{No Opinions}.
\begin{enumerate}
  \item What is the point estimate of the proportion in the population who respond \textbf{Yes}?
  \begin{equation*}
    \bar{p} = \frac{x}{n} = \frac{75}{150} = 0.5
  \end{equation*}
  \item What is the point estimate of the proportion in the population who respond \textbf{No}?
  \begin{equation*}
    \bar{p} = \frac{x}{n} = \frac{55}{150} = \frac{11}{30} = 0.36666666
  \end{equation*}
\end{enumerate}

\section*{3.1.2}
Many drugs used to treat cancer are expensive. BusinessWeek reported on the cost per treatment of Herceptin, a drug used to treat breast cancer(BusinessWeek, January 30, 2006).

Typical treatment costs (in dollars) for Herceptin are provided by a simple random sample of 10 patients.
\begin{center}
  \textbf{\$4,376 \$5,578 \$2,717 \$4,920 \$4,495 \$4,798 \$6,446 \$4,119 \$4,237 \$3,814}
\end{center}

\begin{enumerate}
  \item Develop a point estimate of the mean cost per treatment with Herceptin
  \begin{equation*}
    \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_{i} = \frac{45500}{10} = \$4,550.00
  \end{equation*}
  \item Develop a point estimate of the standard deviation of the cost per treatment with Herceptin.
  \begin{equation*}
    s = \sqrt{\frac{1}{n - 1} \sum_{i=1}^{n}(x_{i} - \bar{x})^{2}} = \sqrt{\frac{9,068,620.00}{9}}= \$1003.80498327337
  \end{equation*}
\end{enumerate}

\section*{3.1.3}
The American Association of Individual Investors (AAII) polls its subscribers on a weekly basis to determine the number who are bullish, bearish, or neutral on the short-term prospects for the stock market. Their findings for the week ending March 2, 2006, are consistent with the following sample results (AAII website, March 7, 2006).
\begin{table}[h!]
  \centering
  \begin{tabular}{c c c}
    Bullish 409 &Neutral 299 &Bearish 291\\
  \end{tabular}
\end{table}
Develop a point estimate of the following population parameters.
\begin{enumerate}
  \item The proportion of all AAII subscribers who are bullish on the stock market.
  \begin{equation*}
    \bar{p} = \frac{x}{n} = \frac{409}{999} = 0.409
  \end{equation*}
  \item The proportion of all AAII subscribers who are neutral on the stock market.
  \begin{equation*}
    \bar{p} = \frac{x}{n} = \frac{299}{999} = 0.299
  \end{equation*}
  \item The proportion of all AAII subscribers who are bearish on the stock market.
  \begin{equation*}
    \bar{p} = \frac{x}{n} = \frac{291}{999} = 0.291
  \end{equation*}
\end{enumerate}

\section*{3.1.4}
A simple random sample of 5 months of sales data provided the following information:
\begin{table}[h!]
  \centering
  \begin{tabular}{c c c c c c}
    Month: &1 &2 &3 &4 &5 \\
    Units Sold: &94 &100 &85 &94 &92\\
  \end{tabular}
\end{table}
\begin{enumerate}
  \item Develop a point estimate of the population mean number of units sold per month.
  \begin{equation*}
    \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_{i} = \frac{465}{5} = 93
  \end{equation*}
  \item Develop a point estimate of the population standard deviation.
  \begin{equation*}
    s = \sqrt{\frac{1}{n - 1} \sum_{i=1}^{n}(x_{i} - \bar{x})^{2}} = \sqrt{\frac{116}{4}}= 5.3851648071345
  \end{equation*}
\end{enumerate}
\end{document}


\section{Indexing}

\frame{\tableofcontents[currentsection]}

\begin{frame}
    \frametitle{Problem Statement}
    \begin{center}\ttfamily
        [1, 2, 3, 4, 5, 6][3] \\[4mm]
        $\downarrow$ \\[4mm]
        4
    \end{center}
\end{frame}

\begin{frame}
    \frametitle{Indexing Array}
    \begin{center}
        \begin{tikzpicture}
            \draw[thick] (0,0) grid (6,1);
            \foreach[evaluate={int(\x+1)} as \i] \x in {0,...,5} {
                \node[font=\ttfamily] at ($ (\x,0.5) + (0.5,0) $) {\i};
            }
            \draw[thick] (0,0) -- ++(0,-0.5);
            \node[anchor=north,font=\ttfamily\tiny] at (0,-0.5) {start};
            \draw[|-latex] (0,-.25) -- ++(3,0) node[midway,below,font=\tiny\ttfamily] {index * sizeof(T)};
        \end{tikzpicture}
    \end{center}
    \vskip4mm
    \structure{Algorithm}
    \begin{itemize}
        \item Memory location can be computed in a single step
        \item \texttt{location = start + index * sizeof(T)}
        \item Direct CPU support: only 1 instruction required
        \item Explains zero-indexing
        \item $O(1)$
    \end{itemize}
\end{frame}

\begin{frame}
    \frametitle{Indexing Linked List}
    \begin{center}
        \begin{tikzpicture}[link/.style={thick,-latex}]
            \path[use as bounding box] (0,0) rectangle (10,3);
            \coordinate (p1) at (0,0);
            \coordinate (p2) at (2,2);
            \coordinate (p4) at (4,1);
            \coordinate (p3) at (6,1);
            \coordinate (p5) at (8,2);
            \coordinate (p6) at (7,0);

            \foreach \i in {1,...,6} {
                \llnode[position={p\i},size=0.5cm,value=\i]
            }

            \draw[-latex] ($ (p1) + (0.75,0.25) $) to[bend right=30] (p2);
            \draw[-latex] ($ (p2) + (0.75,0.25) $) to[bend left=30] ($ (p3) + (0,0.5) $);
            \draw[-latex] ($ (p3) + (0.75,0.25) $) to[bend left=45] ($ (p4) + (1,0) $);
            \draw[-latex] ($ (p4) + (0.75,0.25) $) to[bend left=45] ($ (p5) + (0,0.25) $);
            \draw[-latex] ($ (p5) + (0.75,0.25) $) to[bend left=45] ($ (p6) + (1,0.5) $);
        \end{tikzpicture}
    \end{center}
    \vskip4mm
    \structure{Algorithm}
    \begin{itemize}
        \item Nodes are scattered unpredictably across memory
        \item Follow \texttt{Next} until \texttt{Next == null}
        \item Finding \texttt{n}th element takes \texttt{n} jumps
        \item $O(n)$
    \end{itemize}
\end{frame}


\subsection{The scale invariant limit in 2 dimensions}

In the spirit of Nussinov and Nussinov~\cite{nussinov2004bcs}, Nishida and Son~\cite{Nishida:2006eu} analyzed the two-dimensional unitary limit (or near it) by investigating the system in $2+\epsilon$ dimensions, using $\epsilon$ as a perturbative parameter (not to be confused with the lattice spacing).
They found the trivial solution in the scale-invariant limit; namely, the two-body system becomes non-interacting as the scattering length $\tilde a_{20} \goesto\infty$.
Our results also show this behavior, since, for fixed $L$, the intersection of a flat scattering amplitude given by $\tilde a_{20}$ goes to the non-interacting finite-volume energies as the scattering length grows large with either sign, indicating a trivial theory, as seen in \Figref{luescher2d}.

In exactly two dimensions, however, the logarithmic dependence of the scattering length requires an accompanying scale so as to make the argument of the logarithm dimensionless, a consequence of the fact that in 2 dimensions, the interaction parameters of the Schr\"odinger equation are dimensionless.
\emph{Dimensionful} observables occur via dimensional transmutation \cite{} which requires some fiducial external scale, which in any finite-volume calculation is naturally given by the size of the volume.
Thus the scale-invariant limit becomes sensitive to the order in which one takes either $\tilde a_{20}\goesto\infty$ and $L\goesto\infty$ limits.
Taking either limit first, indpendently of the other, will give the trivial scale-invariant limit found in \cite{Nishida:2006eu}.
However, there is a particular limit in which the scale-invariant system gives non-trivial solutions.
To see this, note that when $\tilde a_{20}=\frac{L}{2\pi}$ the finite-volume energies must zero the zeta function in the quanization condition~\eqref{2d luscher}, as in three dimensions when $a_{30}\to\infty$ and one dimension when $a_{10}\to 0$.

In those dimensions, such a situation is akin to the unitary limit, once the infinite volume limit has been taken.
This limit can be done independently of taking $a_{30}\to\infty$ (3-D) or $a_{10}\to0$ (1-D).
However, in two dimensions, because of the logarithmic dependence, demanding that the zeta function remains zeroed axis requires the infinite volume limit $L\to\infty$ taken together with $\tilde a_{20}$.
In particular, for the \emph{non-trivial} scale-invariant limit, one must take both $\tilde a_{20}\goesto \infty$ and $L\to\infty$ limits simultaneously \emph{such that} $L/\tilde a_{20}=2\pi$.
Such a procedure keeps the dispersion zeta function $S^{\dispersion}_2$ zeroed at each step of the extrapolation, resulting in a non-trivial, scale-invariant, two-body system in two dimensions.


We have implemented a suite of tools for analysis of error for all of the aforementioned Langevin Monte Carlo algorithms. This section contains a selection of results, as well as a description of error metrics available and any other design choices made. The \textsc{Python} code and a how-to
guide can be found at the following
\textsc{url}: \\

   \centerline{ \url{https://github.com/Tom271/LangevinMC}}

\subsection{Overview}\label{overviewprogram}
\begin{itemize}
\item In the program, we first defined 'Sliced Wasserstein Distance' for dimension 1 and other. \\
\item We then defined the class of the potential: Gaussian, Gaussian Gradient; Double Wells, Double Wells Gradient; Ginzburg Landau, Ginzburg Landau Gradient;     Rosenbrock etc.\\
\item We then defined the algorithms we will be using to analyse our samples. The algorithms we will be using are: ULA, MALA, RWM, LM, HOLA, HPH; Taming Algorithms are: tULA, tMALA, tHOLA, tLM etc.\\
\item To evaluate and compare which method is better, we will compare some properties obtained by each method. For example, we will compare 'first moment', 'second moment', 'trace', 'scatter', 'histogram', 'KL-divergence', 'total variation', 'Sliced Wasserstein' \\
\item The experiment will then be performed for each method and each property. We could increase the accuracy of our results by increasing the number of steps in each experiment. \\
\item To know the accuracy of a method, we will calculate the difference between the results obtained from our algorithm and the result obtained from the theory. \\ The smaller the difference is, the more accurate the approximated result is, therefore, the better the algorithm.\\
\end{itemize}
% \subsection{Results}
%
% omit this figure

% \begin{figure}[H]
% \centering
%   \begin{minipage}[b]{0.32\textwidth}
%   \centering
%     \includegraphics[width=\textwidth]{Figures/tula_tv.png}
%   \end{minipage} %
%   \begin{minipage}[b]{0.32\textwidth}
%   \centering
%     \includegraphics[width=\textwidth]{Figures/tulac_tv.png}
%   \end{minipage} %
%   \begin{minipage}[b]{0.32\textwidth}
%   \centering
%     \includegraphics[width=\textwidth]{Figures/tmala_tv.png}
%   \end{minipage}
%   \caption{Comparison of \texttt{tULA}, \texttt{tULAc} and \texttt{tMALA} for the total variation error evolving as a function of step size.}
% \end{figure}

\subsection{Sampling Approaches}

Throughout this project, we have encountered two main approaches for sampling. To obtain a set of samples, one may:

\begin{itemize}
    \item Run a single chain, discard a certain amount of the initial values (the burn-in period) and keep the rest, with possible thinning\footnote{To thin the chain, one only retains every $k$-th sample for some $k$.} to reduce autocorrelation. This is used for practical sampling.
    \item Run multiple chains and retain their final $N$-th value only. This is useful for verification/establishment of non-asymptotic bounds.
\end{itemize}

To accommodate for both possibilities, given positive integers $b, N_{\text{chains}}, N$ and a set $I \subset \mathbb \{1, 2, \dots, N - b\}$ of indices, our program aggregates samples with indices in $\{i+b\ |\ i \in I\} $  from $ N_{\text{chains}}$ independent chains.

\subsection{Estimating Error}

In the following, let $\pi$ be the pdf of true distribution from which we attempt to sample and $X = \{x_i\}_{i=1}^N$ denote the collection of samples collected by a sampling algorithm. The notation $x_i(d)$ will be used to refer to the $d$-th coordinate of $x_i$. We also denote by $\{m_d\}_{d=1}^D$ and $\{M_d\}_{d=1}^D$ the coordinate-wise minimums and maximums among $x_i$, that is:

$$
    m_d = \min \{x_i(d)\}_{i=1}^N \ \ \text{ and } \ \ M_d = \max \{x_i(d)\}_{i=1}^N
$$

For some of the error estimation methods, it will be necessary to construct multi-dimensional histograms. For this purpose, we define the following set of points (this is a minimal mesh that spans all of the samples):

\begin{align*}
    \#_X := \Big\{ \left(c_1^{(k_1)}, c_2^{(k_2)}, \dots, c_D^{(k_D)}\right)\ |\ &(k_1, k_2, \dots, k_D) \in \lbrace 0, 1, \dots, \text{bins - 1}\rbrace^D, \ c_d^{(k_d)} \\
    &= m_d + \left(k_d + \frac 1 2\right) \frac{M_d - m_d}{\text{bins}} \Big\}
\end{align*}

where $bins$ is a parameter describing the fineness of the mesh. The histogram of the collection of samples $X$ will be understood to be the following function $h: \#_X \rightarrow \mathbb R$:

$$
    h(c) = |\{x \in X\ |\ \arg \min_{c' \in \#_X} \norm{c' - x} = c \}|
$$

i.e. count of points $x$ such that the closes point of mesh to $x$ is $c$. This defines the discrete probability measure

$$H_X := \sum_{c \in \#_X} \frac {h(c)} N   \delta_c,$$

which we will understand to be the sampled approximation of the true distribution from which we are sampling. Finally, denote also by $Z$ the sum of the true distribution pdf values over the values of the mesh.

\[Z := \sum_{c \in \#_X} \pi(c)\]

\subsubsection{The Curse of Dimensionality}

In short, since the size of the mesh $|\#_X| = \text{bins}^D$ is exponential in dimension, it is impossible to construct histograms in high dimensions. Even within smaller dimensions, a fundamental trade-off arises; either one can construct a desirably fine mesh with many bins being empty (that is, $h(c) = 0$), or construct a coarse mesh with sufficiently many samples. The quality of the error estimation suffers in both cases. \\\\
For this reason, the implemented  histogram  error metrics are only feasible for smaller dimensions\footnote{Around $1 \leq D \leq 4$.}, whereas the other metrics should be considered only indicative with increasing dimension, rather than precise measures of error.

\subsubsection{Kernel Density Estimation}
As an alternative to histograms, and to smoothen (and improve) our approximation of the sampled distribution, we considered Kernel Density Estimation. KDE can be thought of as an extension of the idea of histogram which reduces the effect of bin placement.

\begin{defn}[KDE]
A kernel density estimator for a i.i.d. set of samples $X = \{x_i\}_{i=1}^N$ is
\[\hat{\pi}_{X} (x) = \frac 1 {Ns} \sum_{i=1}^N K\left( \frac{x - x_i}{s} \right)\]
where $s > 0$ is a smoothing parameter known as the bandwidth and $K$ is a kernel function. Based on this estimation, we define the following discrete probability measure:

\[ \text{KDE}_X := \frac 1 {\hat Z} \sum_{c \in \#_X} \hat{\pi}_X(c) \delta_c \qquad \text{with the normalizing constant } \hat Z = \sum_{c \in \#_X} \hat{\pi}_X(c).\]
\end{defn}

For our purposes, a Gaussian kernel was assumed.  This choice was fairly arbitrary, as time did not permit an in-depth review of the literature.  The main justification for the choice was that the kernel has unbounded support.  The bandwidth then determines the standard deviation of the Gaussian kernel.  For small bandwidths, the estimator will suffer from overfitting, and will result in a jagged estimate with high variance.  For high bandwidths, the estimator oversmooths, and hence results in a highly biased, underfit estimate \cite{kroese2013handbook}.  For dimension one, Silverman's rule or the Improved Sheather-Jones algorithm can be used for bandwidth selection.  The former assumes the true distribution is Gaussian, and is best suited for unimodal distributions.  Given large numbers of data points, the latter algorithm can deal with a wider range of multimodal distributions \cite{botev2010kernel}, \cite{wand1994kernel}.  These algorithms do not extend to higher dimensions, however, and so, like the choice of kernel, bandwidth also became relatively arbitrary.

Without theoretical justification for our choices, we did not use KDE for the results in this report.  We also note that, although they solve some of the issues histograms in dimensions higher than one, kernel density estimates are similarly affected by the curse of dimensionality, and are impractical in dimensions much greater than those recommended for histograms.

\noindent

\subsection{Implemented Metrics}

\subsubsection{Visualizations}
In special cases when the dimension is $1$ or $2$, we have implemented an option to visualize the samples either as a histogram plot (in 1D) or as a scatter or trace plot (in 2D or 3D).

\begin{figure}[H]
\centering
  \begin{minipage}[b]{0.3\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/histo_example.png}
  \end{minipage} %
  \begin{minipage}[b]{0.3\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/scatter_example.png}
  \end{minipage} %
  \begin{minipage}[b]{0.3\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/trace_example.png}
  \end{minipage}
   \caption{Example of a histogram plot, scatter plot and a trace plot.}
\end{figure}

\subsubsection{First/Second Moments}
The first and second moments are implemented in the standard way. By default, the result is computed \textit{in the first coordinate only\footnote{With option to use $\frac 1 N\sum_{i=1}^N \norm{x_i}$ and $\frac 1 N\sum_{i=1}^N \norm{x_i^2}$ instead.}}.

$$
    \text{first moment} = \frac 1 N\sum_{i=1}^N x_i(1) \qquad \text{ and } \qquad \text{second moment} = \frac 1 N\sum_{i=1}^N x_i(1)^2
$$

\subsubsection{Total Variation}
The total variation, being the first of implemented histogram measures, is calculated as

\[\text{total variation} = \norm{H_X - \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c}_{\text{TV}} = \frac 1 2\sum_{c \in \#_X} \left|\frac{h(c)}{N} - \frac{\pi(c)}{Z}\right| .\]

\subsubsection{KL Divergence}
The next histogram-based measure is KL divergence.

\[\text{KL divergence} = D_{\text{KL}} \left( \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c \,\bigg|\bigg| \, H_X) =  \sum_{c \in \#_X} \frac{\pi(c)}{Z} \log\left(\frac{\pi(c) N}{h(c) Z} \right)\right) .\]

\subsubsection{Sliced Wasserstein}

The Sliced Wasserstein distance, being defined via a multi-dimensional integral, cannot be computed exactly. Therefore, we resort to a simple Monte Carlo scheme where $L$ samples $\{\theta_i\}$ are drawn uniformly from the $(d-1)$-dimensional sphere $\mathbb S^{d-1}$.

$$
SW_p(P, Q) \approx \left( \frac 1 L \sum_{i=1}^L W_p^p\left(\mathcal{RI}_P(\cdot, \theta_i), \mathcal{RI}_Q(\cdot, \theta_i) \right) \right)^{\frac 1 p}
$$

The first, histogram based, implemented metric for Sliced Wasserstein distance further approximates the above Monte Carlo scheme for $SW_1$ as

\[\text{SW}_{histogram} = \begin{cases}
W_1 \left( H_X,\  \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c \right) & \text{if dimension } = 1 \\
\sum_{i = 1}^L W_1 \left( \frac 1 N \sum_{c \in \#_X} h(c) \delta_{\left<c, \theta_i \right>},\  \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_{\left<c, \theta_i \right>}  \right) & \text{ otherwise, }
\end{cases}\]

where $\left< \cdot \right>$ is the dot product, $\theta_i = \frac{\theta'_i}{\norm{\theta'_i}}$ is a point on $(d-1)$-dimensional sphere with $
\theta'_i \sim \mathcal N(0, I_D)$ and $W_1$ is computed explicitly.   No computationally efficient method was found for $SW_2$ \footnote{It is possible to calculate $SW_2$ using \cite{flamary2017pot}, which solves the optimal transport problem using linear programming techniques.  This is many orders of magnitude slower than $SW_1$}.   \\\\

The second, \textbf{heuristic-based}, approximation of the Sliced Wasserstein distance is calculated as follows:

\[\text{SW}_{no\ histogram} = \begin{cases}
W_1 \left( \frac 1 N \sum_{x \in X} \delta_x,\  \frac 1 {Z'} \sum_{x \in X} \pi(x) \delta_x \right) & \text{if dimension } = 1 \\
\sum_{i = 1}^L W_1 \left(  \frac 1 N \sum_{x \in X} \delta_{\left<x, \theta_i \right>},\ \frac 1 {Z'} \sum_{x \in X} \pi(x) \delta_{\left<x, \theta_i \right>}  \right) & \text{ otherwise. }
\end{cases}\]
where $Z' = \sum_{x \in X} \pi(x) $ is a normalizing constant. Intuitively, this measure of error describes whether the spatial distribution of samples is proportional to the true distribution, \textit{at the same points}.

\subsection{KDE-based metrics}

All three of the above histogram-based metrics are available in their KDE version as well.

\[\text{total variation}^{\text{KDE}} = \norm{\text{KDE}_X - \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c}_{\text{TV}} \]

\[\text{KL divergence}^{\text{KDE}} = D_{\text{KL}} \left(\text{KDE}_X \ \bigg|\bigg|\  \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c\right) \]

\[\text{SW}_{histogram}^{\text{KDE}} = \begin{cases}
W_1 \left( \text{KDE}_X,\  \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_c \right) & \text{if dimension } = 1 \\
\sum_{i = 1}^L W_1 \left( \frac 1 {\hat Z} \sum_{c \in \#_X} \hat{\pi}_X(c) \delta_{\left<c, \theta_i \right>},\  \frac 1 Z \sum_{c \in \#_X} \pi(c) \delta_{\left<c, \theta_i \right>}  \right) & \text{ otherwise. }
\end{cases}\]


    \documentclass{article}
    \usepackage{amsfonts}
    \usepackage{amsmath,multicol,eso-pic}
    \begin{document}
    \title{Example Worksheet 1}
 \date{\vspace{-5ex}}
 \maketitle

        \section{Linear equations}
        Solve the following equations for the specified variable.
        \begin{multicols}{1}
        \begin{enumerate}
        \item Solve for $E$ : $$- 15 E + 23 = E F + T$$
\item Solve for $q$ : $$E - 12 q = T q + 15$$
\item Solve for $v$ : $$f v - 15 = 18 v + y$$
\item Solve for $x$ : $$H x + 16 = - 5 x - 5$$
\item Solve for $n$ : $$c - 11 n = B n + 12$$
\item Solve for $b$ : $$11 b + g = A + T b$$
\item Solve for $X$ : $$- 18 X - 13 = - 5 X + a$$
\item Solve for $b$ : $$T b + h = b z - 2$$
\item Solve for $d$ : $$10 d - 3 = K d + j$$
\item Solve for $q$ : $$g + 4 q = 17 q + r$$
\item Solve for $d$ : $$d r + 21 = A + S d$$
\item Solve for $u$ : $$- 4 u - 17 = 24 u - 2$$
\item Solve for $Y$ : $$E - Y = - 7 Y - 17$$
\item Solve for $j$ : $$j t + 5 = - 13 j - 9$$
\item Solve for $W$ : $$- 26 W + c = C + W g$$
\item Solve for $r$ : $$- 19 r + t = H r - 2$$
\item Solve for $N$ : $$15 N + 4 = - 16 N - 14$$
\item Solve for $v$ : $$24 v + 5 = - 13 v - 6$$
\item Solve for $d$ : $$15 d - 4 = 23 d + 5$$
\item Solve for $x$ : $$N x + S = W x - 25$$
        \end{enumerate}
        \end{multicols}

        \section{Quadratic equations}
        Solve the following quadratic equations.
        \begin{multicols}{1}
        \begin{enumerate}
        \item $$- 8 y^{2} + 3 = y - 21$$
\item $$- 20 y^{2} - 20 y = 23$$
\item $$- 12 x^{2} = 14 x^{2} + 2 x + 18$$
\item $$y^{2} + 28 y + 132 = 0$$
\item $$x^{2} - 39 x + 368 = 0$$
\item $$x^{2} - 17 x + 42 = 0$$
\item $$x^{2} + 6 x - 520 = 0$$
\item $$y^{2} + 12 y - 364 = 0$$
\item $$y^{2} - 32 y + 231 = 0$$
\item $$17 y^{2} - 24 y = 0$$
\item $$y^{2} + 8 y + 12 = 0$$
\item $$y^{2} - 8 y - 308 = 0$$
\item $$15 x^{2} + 8 x - 21 = 9 x^{2} + 14$$
\item $$- 25 y^{2} - 8 y - 2 = 18 y^{2} - 12 y - 8$$
\item $$x^{2} - 18 x + 77 = 0$$
\item $$x^{2} + 6 x - 247 = 0$$
\item $$y^{2} - 27 y + 92 = 0$$
\item $$- 19 y^{2} = - 5 y^{2} + 2 y + 20$$
\item $$12 x^{2} - 4 x = - 19 x$$
\item $$x^{2} + 18 x - 208 = 0$$
        \end{enumerate}
        \end{multicols}

        \section{Differentiation}
        Compute each derivative
        \begin{multicols}{1}
        \begin{enumerate}
        \item $$\frac{d}{d x}\left(\frac{e^{x} + \log{\left (x \right )}}{\tan{\left (x \right )}}\right)$$
\item $$\frac{d}{d z}\left(\left(\sqrt{z} + \sin{\left (z \right )}\right) e^{- z}\right)$$
\item $$\frac{d}{d y}\left(\frac{- 5 y^{2} + y - 21}{\tan{\left (y \right )}}\right)$$
\item $$\frac{d}{d y}\left(\left(- 16 y^{3} - 19 y + \sin{\left (y \right )} + 23\right) e^{- y}\right)$$
\item $$\frac{d}{d z}\left(\frac{\sqrt{z} - 10 z^{3} + 15 z^{2} + z}{- 23 z^{2} + 11 z + 12}\right)$$
\item $$\frac{d}{d x}\left(- \frac{1}{11 x^{2}} \left(\log{\left (x \right )} + \cos{\left (x \right )}\right)\right)$$
\item $$\frac{d}{d y}\left(\frac{y + e^{y}}{17 y^{3} - 24 y - 10}\right)$$
\item $$\frac{d}{d z}\left(\left(z + \tan{\left (z \right )}\right) e^{- z}\right)$$
\item $$\frac{d}{d y}\left(\frac{\sin{\left (y \right )} + \tan{\left (y \right )}}{- 8 y + 5}\right)$$
\item $$\frac{d}{d y}\left(\frac{6 y^{3} + 36 y^{2}}{\sin{\left (y \right )}}\right)$$
        \end{enumerate}
        \end{multicols}

        \section{Compute the integral}
        Compute the integral of the polynomials.
        \begin{multicols}{1}
        \begin{enumerate}
        \item $$\int \left(-7\right)\, dz$$
\item $$\int 18 y\, dy$$
\item $$\int \left(- 13 z - 22\right)\, dz$$
\item $$\int 2\, dz$$
\item $$\int \left(- 17 y^{3} - 20 y^{2} - 26\right)\, dy$$
\item $$\int \left(25 y - 1\right)\, dy$$
\item $$\int \left(- 23 y\right)\, dy$$
\item $$\int \left(16 z^{3} - 12 z\right)\, dz$$
\item $$\int \left(8 z + 6\right)\, dz$$
\item $$\int 5\, dz$$
        \end{enumerate}
        \end{multicols}

        \section{Compute the integral}
        Compute the integral of the powers.
        \begin{multicols}{1}
        \begin{enumerate}
        \item $$\int \left(8 y^{\frac{5}{3}} - \frac{23 y^{\frac{4}{3}}}{15}\right)\, dy$$
\item $$\int \left(- \frac{7}{9 z^{\frac{3}{2}}}\right)\, dz$$
\item $$\int \left(\frac{\sqrt{z}}{6} + \frac{7}{\sqrt[4]{z}} - \frac{7}{2 z^{\frac{5}{4}}}\right)\, dz$$
\item $$\int \frac{13}{9 y^{\frac{3}{2}}}\, dy$$
\item $$\int \frac{7}{z^{\frac{5}{3}}}\, dz$$
\item $$\int \frac{51 z^{\frac{3}{5}}}{20}\, dz$$
\item $$\int \frac{5 y^{\frac{5}{2}}}{4}\, dy$$
\item $$\int \left(- \frac{5 z^{\frac{5}{4}}}{3}\right)\, dz$$
\item $$\int \left(- 6 z^{\frac{2}{5}} - \frac{7}{\sqrt[5]{z}}\right)\, dz$$
\item $$\int \left(- \frac{z^{\frac{5}{2}}}{4} + \frac{5 z^{\frac{3}{2}}}{4}\right)\, dz$$
        \end{enumerate}
        \end{multicols}

        \section{Compute the integral}
        Compute the integral of the powers.
        \begin{multicols}{1}
        \begin{enumerate}
        \item $$\int \frac{1}{z} \tan{\left (\log{\left (z \right )} \right )}\, dz$$
\item $$\int \left(- 24 z - 11\right) \sqrt{- 12 z^{2} - 11 z - 10}\, dz$$
\item $$\int \left(- \frac{14}{z} \log{\left (z \right )}\right)\, dz$$
\item $$\int \left(19 e^{3 y} - 2\right) e^{y}\, dy$$
\item $$\int \frac{1}{y} \tan{\left (\log{\left (y \right )} \right )}\, dy$$
\item $$\int \left(- 38 z + 22\right) \left(- 247 z^{2} + 286 z + 6 \left(- 19 z^{2} + 22 z - 18\right)^{3} - 234\right)\, dz$$
\item $$\int e^{2 y}\, dy$$
\item $$\int \sqrt{e^{y}} e^{y}\, dy$$
\item $$\int \frac{\sin{\left (\sqrt{y} \right )}}{2 \sqrt{y}}\, dy$$
\item $$\int 20 \sin^{2}{\left (z \right )} \cos{\left (z \right )}\, dz$$
        \end{enumerate}
        \end{multicols}

    \end{document}


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 2.3 Covariant derivative of $v^{a}{}_{b}$}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \nabla{#}::Derivative.
   \partial{#}::PartialDerivative.

   # template for covariant derivative of a vector

   derivU := \nabla_{a}{A?^{b}} -> \partial_{a}{A?^{b}} + \Gamma^{b}_{c a} A?^{c}.
   derivD := \nabla_{a}{A?_{b}} -> \partial_{a}{A?_{b}} - \Gamma^{c}_{b a} A?_{c}.

   vab := v^{a}_{b} -> A^{a} B_{b}.
   iab := A^{a} B_{b} -> v^{a}_{b}.

   pab := \partial_{a}{A^{b}} B_{c} -> \partial_{a}{A^{b} B_{c}} - A^{b} \partial_{a}{B_{c}}.

   # create an object

   Dvab := \nabla_{a}{v^{b}_{c}}.   # cdb (ex-0203.101,Dvab)

   # apply the rule, then simplify

   substitute     (Dvab,vab)        # cdb (ex-0203.102,Dvab)
   product_rule   (Dvab)            # cdb (ex-0203.103,Dvab)
   substitute     (Dvab,derivD)     # cdb (ex-0203.104,Dvab)
   substitute     (Dvab,derivU)     # cdb (ex-0203.105,Dvab)
   distribute     (Dvab)            # cdb (ex-0203.106,Dvab)
   substitute     (Dvab,pab)        # cdb (ex-0203.107,Dvab)
   canonicalise   (Dvab)            # cdb (ex-0203.108,Dvab)
   substitute     (Dvab,iab)        # cdb (ex-0203.109,Dvab)
   sort_product   (Dvab)            # cdb (ex-0203.110,Dvab)
\end{cadabra}

\begin{align}
   \cdb{ex-0203.101} &= \Cdb{ex-0203.102}\\
                     &= \Cdb{ex-0203.103}\\
                     &= \Cdb{ex-0203.104}\\
                     &= \Cdb{ex-0203.105}\\
                     &= \Cdb{ex-0203.106}\\
                     &= \Cdb{ex-0203.107}\\
                     &= \Cdb{ex-0203.108}\\
                     &= \Cdb{ex-0203.109}\\
                     &= \Cdb{ex-0203.110}
\end{align}

\end{document}


\section{Arith}\label{Arith}

The {\tt Arith} library deals with various arithmetical notions and
their properties.

\subsection*{Standard {\tt Arith} library}

The following files are automatically loaded by {\tt Require Arith}.

\begin{itemize}

\item {\tt Le.v} states and proves properties of the large order {\tt le}.

\item {\tt Lt.v} states and proves properties of the strict order {\tt
lt} (especially, the relationship with {\tt le}).

\item {\tt Plus.v} states and proves properties on the addition.

\item {\tt Gt.v} states and proves properties on the strict order {\tt gt}.

\item {\tt Minus.v} defines the difference on
{\tt nat} and proves properties of it. On {\tt nat}, {\tt (minus n p)} is
{\tt O} if {\tt n} $<$ {\tt p}.

\item {\tt Mult.v} states and proves properties on the multiplication.

\item {\tt Between.v} defines modalities on {\tt nat} and proves properties
of them.

\end{itemize}

\subsection*{Additional {\tt Arith} library}

\begin{itemize}

\item {\tt Compare.v}, {\tt Compare\_dec.v} and {\tt Peano\_dec.v} state
and prove various decidability results on {\tt nat}.

\item {\tt Wf\_nat.v} states and proves various induction and recursion
principles on {\tt nat}. Especially, recursion for objects measurable by
a natural number and recursion on {\tt nat * nat} are provided.

\item {\tt Min.v} defines the minimum of two natural numbers and proves
properties of it.

\item {\tt Eqnat.v} defines a specific equality on {\tt nat} and shows
the equivalence with Leibniz' equality.

\item {\tt Euclid.v} proves that the euclidean
division specification is realisable. Conversely, {\tt Div.v} exhibits
two different algorithms and semi-automatically reconstruct the proof of
their correctness. These files emphasize the extraction of program vs
reconstruction of proofs paradigm.

\end{itemize}


\chapter{Mutli-armed bandits}


%---------------------------Jacobian-----------------------------
\section{Jacobian\label{s:tet-jacobian}}

This metric is defined as follows:
\begin{displaymath}
q = \left( \vec L_2 \times \vec L_0 \right) \cdot \vec L_3
\end{displaymath}

\tetmetrictable{Jacobian}%
{$L^3$}%                  Dimension
{$[0,DBL\_MAX]$}%         Acceptable range
{$[0,DBL\_MAX]$}%         Normal range
{$[-DBL\_MAX,DBL\_MAX]$}% Full range
{$\frac{\sqrt{2}}{2}$}%   Equilateral tet
{\cite{knu:03}}%          Citation
{v\_tet\_jacobian}%                            Verdict function name


\documentclass[11pt,letterpaper]{article}

\usepackage[pdftex]{graphicx}
\usepackage{natbib}
\usepackage{fullpage}
\usepackage{lineno}
\usepackage{multirow}
\usepackage{wrapfig}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{sidecap}
\usepackage{hyperref}

\begin{document}

\setlength{\parindent}{0mm}
\setlength{\parskip}{0.4cm}

\bibliographystyle{apalike}

%\modulolinenumbers[5]
%\linenumbers

\title{ARCS scenario 1 data inversions}

\maketitle

\tableofcontents

\pagebreak

This document describes the mathematical formulation of the ARCS scenario 1 data inversion problem.

\section{Summary}

Measurements being made for ARCS scenario 1 crossings will result in production of maps of magnetic field fluctuations $\delta \mathbf{B}$, and flows $\mathbf{v}$ as described in the various instrument and data processing section.  These will be converted, as part of standard data processing, into parallel current density and electric field:
\begin{equation}
\mathbf{J} = \nabla \times \left( \frac{\delta \mathbf{B}}{\mu_0} \right); \qquad \mathbf{E} = -\mathbf{v} \times \mathbf{B}
\end{equation}
Because this is a swarm of satellites we will have \emph{datamaps} (2D images) of these parameters; additionally we note that the measurements may be directly used to also produce datamaps of Poynting flux:
\begin{equation}
\mathbf{S} = \mathbf{E} \times \frac{\delta \mathbf{B}}{\mu_0}
\end{equation}
In the case of scenario 1 data,  key unknown physical parameters are datamaps of ionospheric Pedersen and Hall conductances - without these we do not have a full picture of electrodynamics and energy flow in the auroral system and cannot, then, fully unlock the scientific potential of the large amount of scenario 1 data.

Scenario 1 current density and Poynting flux Datamaps are related to each other through electromagnetic conservation laws:  current continuity and the Poynting theorem.  Under assumptions of steady state conditions and equipotential field lines (appropriate at scales relevant to our science questions), these can be reduced to:
\begin{eqnarray}
J_\parallel &=& \Sigma_P \nabla \cdot \mathbf{E}_\perp + \nabla \Sigma_P \cdot \mathbf{E}_\perp - \nabla \Sigma_H \cdot \left( \mathbf{E}_\perp \times \hat{\mathbf{b}} \right) \\
S_{\parallel}  &=& - \Sigma_P E^2
\end{eqnarray}
with unknown Pedersen and Hall conductances.  These conductances may, in principle, be solved by simply inverting this system of equations.  In practice there are many different methods that could work to perform this inversion.  A first glance the system appears even-determined on account of the fact that we have two unknown fields (conductances) and two input datamaps (conserved variables current density and Poynting flux).  It is worth noting however, that the component of the Hall conductance gradient along the electric field direction is explicitly not part of this equation and so is in the null space of the problem.  Thus in addition to conservation laws additional prior information is needed.  For ARCS data processing this will come in two forms (both of which may not always be necessary depending on noise conditions):  (1) regularization that places constrains on solution norm or smoothness (i.e. Tikhonov regularization), and/or (2) inclusion of model information that further correlates Pedersen conductance (which is well-constrained by laws above) to Hall conductance (which requires further constraints).   This could come in the form of model-based inversions (e.g. using GEMINI-GLOW) or simply parameerizations based, e.g. on the Robinson formulas or updated version of these formulas.

\section{Physical constraints:  simplification of general conservation laws}

Conservation of charge in electromagnetic system is described by the current continuity equation:
\begin{equation}
  \frac{\partial \rho}{\partial t} + \nabla \cdot \mathbf{J} = 0
\end{equation}
In a steady state this reduces to:
\begin{equation}
\nabla \cdot \mathbf{J} = \frac{\partial J_\parallel}{\partial z} +  \nabla_\perp \cdot \mathbf{J}_\perp = 0,
\end{equation}
where the $z-$ direction represent altitude in a locally Cartesian coordinate system.  Integrating with respect to altitude:
\begin{equation}
\int \frac{\partial J_\parallel}{\partial z} dz +  \int \nabla_\perp \cdot \mathbf{J}_\perp dz = J_\parallel(\max(z)) - J_\parallel(\min(z)) + \nabla_\perp \cdot \left(  \Sigma \cdot \mathbf{E}_\perp \right) = 0
\end{equation}
Which can be expanded out and solved for the parallel current at the top of the domain, if the bottom current is assumed to be zero:
\begin{equation}
J_\parallel = - \nabla_\perp \cdot \left(  \Sigma \cdot \mathbf{E}_\perp \right) =  - \Sigma_P \nabla \cdot \mathbf{E}_\perp - \nabla \Sigma_P \cdot \mathbf{E}_\perp + \nabla \Sigma_H \cdot \left( \mathbf{E}_\perp \times \hat{\mathbf{b}} \right)
\end{equation}
The current continuity equation to be used in the ARCS analysis is then:
\begin{equation}
\boxed{
J_\parallel = -\Sigma_P \nabla \cdot \mathbf{E}_\perp - \nabla \Sigma_P \cdot \mathbf{E}_\perp + \nabla \Sigma_H \cdot \left( \mathbf{E}_\perp \times \hat{\mathbf{b}} \right)
} \label{eqn:continuity}
\end{equation}
Note that this equation effectively has two unknown fields $\Sigma_P,\Sigma_H$, but represents only one physical constraint; hence additional information is needed.  This is provided by conservation of electromagnetic energy, viz. the Poynting theorem:
\begin{equation}
\frac{\partial w}{\partial t} + \nabla \cdot \mathbf{S} = - \mathbf{J} \cdot \mathbf{E}
\end{equation}
Similar to the assumptions made to produce Equation \ref{eqn:continuity} we neglect time-dependent terms and proceed to integrate the equation along a geomagnetic field line:
\begin{equation}
S_{\parallel,top} - S_{\parallel,bottom} + \nabla_\perp \cdot \mathbf{\mathcal{S}}_\perp = - \Sigma_P E^2
\end{equation}
where $\mathbf{\mathcal{S}}_\perp$ is the column integrated perpendicular Poynting flux.  If we further assume that there is no Poynting flux through the bottom of the ionosphere or the lateral sides of our volume of interest (i.e. net incoming D.C. Poynting flux is dissipated) a simple relation between parallel Poynting flux and Pedersen conductance.
\begin{equation}
\boxed{
S_{\parallel}  = - \Sigma_P E^2
} \label{eqn:poynting}
\end{equation}

\section{Estimating conductances}

Several different procedures can be developed for converting the maps of electric field and Poynting flux into conductances. Two approach are discussed here.

Equation \ref{eqn:poynting} fully specifies the Pedersen conductance given quantities that are measurable by scenario 1 experiments, so the most obvious path would be to then provide the Pedersen conductance to Equation \ref{eqn:continuity}.  Superficially, the equation allows solution for the gradient of the Hall conductance and in principle one would need to compute a line integral of this quantity to solve for Hall conductance:
\begin{equation}
\Sigma_H(\mathbf{r}_2)-\Sigma_H(\mathbf{r}_1) = \int_{\mathbf{r}_1}^{\mathbf{r}_2} \nabla \Sigma_H \cdot d \mathbf{r}
\end{equation}
Moreover, one would also need the value of the Hall condutance at some reference point $\mathbf{r}_1$ to complete the solution for Hall conductance.  While it may be possible to choose a point with low density and assume zero Hall conductance at that reference point there is a more serious issue with this approach and with the set of physical constraints being used, more generally.  Equation \ref{eqn:continuity} only provides constraints on the derivative of the Hall conductance \emph{in the direction of the $\mathbf{E} \times \mathbf{B}$ drift}.  Thus, there is information about the Hall conductance (namely the variation in the direction of the electric field) that is completely unconstrained by current continuity.  As a result, the Hall conductance lies partly in the null space of the problem defined by Equations \ref{eqn:continuity} and \ref{eqn:poynting} and some additional assumptions/information/regularization will be required to solve the inverse problem.

Another approach to the inverse problem would be to view the conservation laws as constraints to be combined together with other prior information in the form of, e.g., smoothness constraints.  Here we rewrite the physical constraints in a matrix form to facilitate application of results from linear inverse theory.  Field quantities can be ``flattened'' into vectors using column major ordering and then operators can be represented through matrix operations.  The latter step can be understood as a decomposition of the derivative operations into finite difference matrices:
\begin{equation}
\underline{j} = - \underline{\underline{I}} ~ \underline{p} \left( \nabla \cdot \mathbf{E}_\perp \right) - \underline{\underline{L}}_x \underline{p} E_x - \underline{\underline{L}}_y \underline{p} E_y + \underline{\underline{L}}_{E \times B} \underline{h} E_\perp
\end{equation}
\begin{equation}
\underline{s} = - E^2 \underline{\underline{I}} ~ \underline{p}
\end{equation}
Concatenating the unknown conductances into a single vector we get:
\begin{equation}
\underline{x} \equiv \left[ \begin{array}{c} \underline{p} \\ \underline{h} \end{array} \right]
\end{equation}
The left-hand sides of each conservation law (i.e. measurements) are similarly stacked:
\begin{equation}
\underline{b} \equiv \left[ \begin{array}{c} \underline{j} \\ \underline{s} \end{array} \right]
\end{equation}
Finally the right-hand side operations may be expressed in block diagonal form:
\begin{equation}
\underline{\underline{A}} \equiv \left[ \begin{array}{cc} -\underline{\underline{I}}  \left( \nabla \cdot \mathbf{E}_\perp \right) -  \underline{\underline{L}}_x  E_x - \underline{\underline{L}}_y E_y  & ~ \underline{\underline{L}}_{E \times B} E_\perp \\ -E^2 \underline{\underline{I}} & \underline{\underline{0}} \end{array} \right]
\end{equation}
Yielding our full set of constrains as:
\begin{equation}
\underline{\underline{A}} ~ \underline{x} = \underline{b}
\end{equation}
As discussed previously this system will not be full-rank, but serves as a starting point for a suitable generalized inverse for this problem.  As a final note the full system has size $2 \cdot N \cdot M \times 2 \cdot N \cdot M$; where $N,M$ are the $x,y$ size of the data maps provided by instrument teams.

\section{Maximum likelihood estimators}

The maximum likelihood estimator, assuming Gaussian-distributed noise is (note we drop the underline notation here for brevity):
\begin{equation}
\hat{x}_{ML} = \left( A^T A  \right)^{-1} A^T b
\end{equation}
The matrix to be inverted here is singular for reasons noted previously; we adopt a Tikhonov regularization scheme to mitigate this:
\begin{equation}
\hat{x} = \left( A^T A  + \lambda I \right)^{-1} A^T b
\end{equation}
where $\lambda$ is a regularization parameter.  This approach regularizes the norm of the solution and coerces it to favor small norms.  One could also enforce any other conditions that can be expressed as a linear operation, yielding:
\begin{equation}
\hat{x} = \left( A^T A  + \lambda  \Gamma^T \Gamma \right)^{-1} A^T b
\end{equation}
where $\Gamma$ is an operator describing smoothness (e.g. Laplacian) or variation (gradient).  We find that the laplacian works well to keep the reconstructions as smooth as possible.

We can add in an offset term, i.e. solve a problem of the form:
\begin{equation}
\hat{x} = \min_x \left\{  || Ax -b ||^2 + || \Gamma x||^2 +  || x - x_0 ||^2 \right\}
\end{equation}
Which solves the least squares problem subject to constraints on smoothness and some expected value for the solution.  The solution is then given by:
\begin{equation}
\hat{x} = \left( A^T A  + \lambda  \Gamma^T \Gamma \right)^{-1} \left( A^T b - \Gamma^T \Gamma x_0\right)
\end{equation}
In the case of the problem of estimating the ionospheric conductances the Hall conductance could constrained to not vary too far from the Pedersen conductivity.

Lastly it may be advantageous to recast the current continuity in terms of the ratio of Hall to Pedersen conductance.  This retains the linearity of the problem only if the Pedersen conductance is known \emph{a priori}.
\begin{equation}
\nabla \left(  \frac{\Sigma_H}{\Sigma_P}  \right) \cdot \mathbf{E} \times \hat{\mathbf{b}} +  \left(  \frac{\Sigma_H}{\Sigma_P}  \right) \frac{\nabla \Sigma_P}{\Sigma_P}  \cdot \mathbf{E} \times \hat{\mathbf{b}} = \frac{J_\parallel}{\Sigma_P} + \nabla_\perp \cdot \mathbf{E}_\perp + \frac{\nabla \Sigma_P}{\Sigma_P}  \cdot  \mathbf{E}_\perp
\end{equation}
This problem can be expressed in matrix form, similar to the approaches described above, and also solve via regularized inverses.  Doing so in a linear fashion does require one to first solve for the Pedersen conductance using the Poynting theorem.  Such a formulation has the benefit that you can regularize deviations from a set conductance ratio.   If a ratio of 1 is used this is equivalent to assuming an average energy of 2.5 keV for the precipitating particles.

\section{Error Covariance}

\section{Connections to precipitating electrons}

Conductances are ultimately driven by electron precipitation, here encapsulated in terms of total energy flux $Q$ and average energy $E_{av}$ - these precipitation parameters are what is needed to ultimately drive the GEMINI simulations.

One of the simplest parameterizations of conductance is the Robinson formulas:
\begin{equation}
\Sigma_P = \frac{40 E_{av}}{16+E_{av}^2} \sqrt{Q}
\end{equation}
\begin{equation}
\frac{\Sigma_H}{\Sigma_P} = 0.45 E_{av}^{0.85}
\end{equation}
Using these to constrain conductances creates a physical correlations between them that does not exist using just the constraints from conservation laws.

\end{document}


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{December 8, 2014}
\maketitle
\section*{14b}
$m\mathbb{Z}\cdot n\mathbb{Z}=(mn)\mathbb{Z}$

$\subseteq$

let $x\in (n\mathbb{Z})(m\mathbb{Z})$ so $x=\sum\limits_{i=1}^k{a_ib_i}$ and $n|a_i$ and $m|b_i$ and so $nm|a_ib_i$ and so $nm|x$ and so $n\mathbb{Z}m\mathbb{Z}\subseteq nm\mathbb{Z}$

$\supseteq$

let $y\in mn\mathbb{Z}$ and so $y=mnz_0=(n\cdot 1)(m\cdot  z_0)\in n\mathbb{Z}m\mathbb{Z}$

\section*{20}
gaussian integers are ``complex'' integers

$\mathbb{Z}[i]/\langle p\rangle=\{[a+bi]|a,b\in\mathbb{Z}\}=\{[a]+[b][i]\}$

$[a]+[b]i=[a']+[b']i\to(a+bi)-(a'+bi)\in\langle p\rangle=p(c+di)\to a-a'=pc\text{ and }b-b'=pd\to a-a'\in \langle p\rangle\to [a]=[a']$ and similarly with b.

\section*{define}
$R$ comm ring and $I$ ideal where $I\ne R$ then $I$ prime ideal means that $ab\in I\to a\in I$or $b\in I$

\subsection*{example}
$R=\mathbb{Z}$, $p\in \mathbb{Z}$ then $p\mathbb{Z}$ is a prime ideal because if $ab\in I$ then wlog $p|ab\to p|b\to b\in I$.

\subsection*{example}
claim
$n\mathbb{Z}$ prime ideal then $n$ is prime

assume $n$ not prime and $n\ne 0$. $n=\alpha\beta$ where $1<\alpha<n, 1<\beta<n, \alpha,\beta\in \mathbb{Z}$. then $\alpha\beta\in n\mathbb{Z}$ but $\alpha\not\in n\mathbb{Z}$ and $\beta\not\in n\mathbb{Z}$. notice that $n\ne 0$ is key here.

\subsection*{example}
claim $\langle0\rangle$ is a prime ideal. $ab=0\to a=0$ or $b=0$ because $R$ is an integral domain

observation: $R$ commutative ring theen $R$is an integral domain iff $\langle0\rangle$is a prime ideal.

\subsection*{$\mathbb{Z}$}
$n\mathbb{Z}\subseteq m\mathbb{Z}\Leftrightarrow m|n$

take$n=p$ prime nmber. $p\mathbb{Z}\subseteq m\mathbb{Z}\Leftrightarrow m|p\Leftrightarrow m\pm 1$ or $m=\pm p$ and so $m\mathbb{Z}=\mathbb{Z}$ or $m\mathbb{Z}=p\mathbb{Z}$

\section*{definition}
if $J$ is an ideal of $R$ and $J\ne R$we say that $J$ is a maximal ideal of $R$ if for every ideal $I$ of $R$ $j\subseteq I\subseteq R\to I=J$ or $I=R$.

\subsection*{$J=p\mathbb{Z}$}
note that $\langle0\rangle$ is not a maximal ideal.

\section*{claim}
every maximal ideal is a prime ideal

\section*{proposition}
let $I$ be a proper ideal ofa commutative ring $R$ (proper means different from ring itself). then $I$ is maximum ideal iff $R/I$ is a field. also $I$ is a prime ideal iff $R/I$ is an integral domain. finally $I$ maximal implies $I$ is a prime ideal.

\subsection*{proof 1}
$R/I$ field iff $R/I$ has only the two trivial ideals (can you prove this?)

and this is true iff $I$ is maximal.

\subsection*{proof 2}
assume $I$ is a prime ideal. then $[x][y]\in R/I$ then $[xy]=[0]$  in $R/I$ and so $xy\in I$ means that $x\in I$ or $y\in I$ and so $[x]=0$ or $[y]=0$ so $R/I$ is an integral domain.

\subsection*{proof 3}
$R/I$ integral domain

then $[xy]=[0]$ in $R/I$ $[x][y]=[0]$ in $R/I$ so $[x]=0$ or$[y]=[0]$ then $x\in I$ or $y\in I$ hence $I$ is a prime ideal.

\section*{thrm}
if $R$ is a principle ideal domain and $P$ is a prime ideal different from zero. then $P$ is maximal.

\subsection*{proof}
$P\subseteq I\subseteq R$ write $P=aR$, $I=bR$. $P$ is non-zero and so $a\ne 0$ and $P\in I\to aR\in bR$. then $a\in bR$ we write $a=br, r\in R$ then $a=br\in P$ and so  $b\in P$ or $r\in P$ because $P$ is prime ideal. if $b\in P$ then $a|b$ and so $I\subseteq P$ but $P\subseteq I$ and so $I=P$. if $r\in P$ then $r\in aR$ and $r=as, s\in R$ and then $a=br=b(as)$ and so $a(1-bs)=0$ but $a\ne 0$ and because we are in an integral domain then $1-bs=0$. and so $1=bs\in I$ and $1\in I$ and so $I=R$.
\end{document}


\documentclass[10pt,a4paper]{article}
\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}
\author{Daniel Frederico Lins Leite}
\graphicspath{ {./} }
\newcommand{\qed}{\hfill\blacksquare}
\begin{document}

Dasgupta "Algorithms" book:
Exercises:
0.a
\begin{align*}
	f(n) = n - 100\\
	g(x) = n - 200
\end{align*}
\includegraphics[width=10cm]{0_a.png}
\paragraph{}
Is $f(n)=O(g(n))$?
\begin{align*}
	\frac{n-100}{n-200} = c\\
	n-100=c*(n-200)\\
	n-100=cn-200c\\
	n=cn-200c+100\\
	n-cn=-200c+100\\
	n(1-c)=-200c+100\\
	n=\frac{-200c+100}{1-c}\\
	n=\frac{100(-2c+1)}{1-c}\\
	n=100\frac{-2c+1}{1-c}\\
\end{align*}
\begin{align*}
	n*-1=100\frac{-2c+1}{1-c}*-1\\
	-n=100\frac{2c-1}{-1+c}\\
	-n=100\frac{2c-1}{c-1}\\
	-n=100\frac{2(c-1)}{c-1}\\
	-n=200\frac{c-1}{c-1}\\
	-n=200\\
	n=-200
\end{align*}
\paragraph{}
So
\begin{align*}
	\frac{f(n)}{g(n)}&<=-200*\frac{g(n)}{g(n)}\\
	f(n) &= O(g(n))\\
	\blacksquare
\end{align*}

\paragraph{}
0.2
Show that, if c is a positive real number, then $g(n) = 1 + c + c^2 + ... + c^n$ is:\\
(a) $\theta(1)$ if $c < 1$.\\
(b) $\theta(n)$ if $c = 1$.\\
(c) $\theta(cn)$ if c > 1.\\
The moral: in big-$\theta$ terms, the sum of a geometric series is simply the first term if the series is
strictly decreasing, the last term if the series is strictly increasing, or the number of terms if the
series is unchanging.
\paragraph{}
\begin{align*}
	f(n) &= 1 + c + c^2 + ... + c^n\\
	f(n) &= \sum_{i=0}^{n} {c^i} <= \sum_{i=0}^{\infty} {c^i}\\
	&<=\frac{1}{1-c}
\end{align*}
In the case of $c<1$ we have
\begin{align*}
	f(n) <= \frac{1}{1-c}\\
	f(n) <= z &&z > 0\\
	\frac{f(n)}{g(n)}<=k*\frac{g(n)}{g(n)}\\
	\frac{f(n)}{1}<=k*\frac{1}{1}&&\text{choose $k=z$}\\\\
	f(n) = O(1)\\
\end{align*}
In the case of $c=1$ we have
\begin{align*}
	f(n) &= 1 + 1^2 + ... + 1^n\\
	f(n) &= \sum_{i=0}^{n}{1} = n\\
	\frac{f(n)}{g(n)}&<=k*\frac{g(n)}{g(n)}\\
	\frac{n}{n}&<=k*\frac{n}{n}&&\text{choose $k=1$}\\
	f(n) &= O(n)
\end{align*}
In the case of $c>1$ we have
\begin{align*}
	f(n) &= 1 + c^2 + ... + c^n\\
	f(n) &= \sum_{i=0}^{n} {c^i} <= \sum_{i=0}^{\infty} {c^i}\\
	&=\sum_{i=0}^{\infty} {c^n}\\
	&=c^n*\sum_{i=0}^{\infty} {1}\\
	&=n*c^n\\
	f(n)&<=n*c^n\\
	\frac{f(n)}{g(n)}&<=k*\frac{g(n)}{g(n)}\\
	\frac{f(n)}{c^n}&<=k*\frac{c^n}{c^n} &&\text{choose $k=n$}\\
	f(n) &= O(c^n)
	\blacksquare
\end{align*}

0.3
The Fibonacci numbers F 0 ,F 1 ,F 2 ,..., are defined by the rule
F 0 = 0, F 1 = 1, F n = F n?1 + F n?2 .
In this problem we will confirm that this sequence grows exponentially fast and obtain some
bounds on its growth.
(a) Use induction to prove that F n ? 2 0.5n for n ? 6.
(b) Find a constant c < 1 such that F n ? 2 cn for all n ? 0. Show that your answer is correct.
(c) What is the largest c you can find for which F n = ?(2 cn )?

(a)
missing bases steps
\begin{align*}
F_n &>= 2^{n/2}\\
F_n &= F_{n-1}+F_{n-2}\\
&>= 2^{(n-1)/2} + 2^{(n-2)/2}\\
& &= (X+2^{(n-2)/2}) + 2^{(n-2)/2} &&\text{$X>0$}\\
& &= 2*2^{(n-2)/2}\\
&>= 2*2^{(n-2)/2}\\
& &=2^{\frac{n-2}{2}+1}\\
& &=2^{\frac{n-2}{2}+\frac{2}{2}}\\
& &=2^{\frac{n}{2}}\\
&>= 2^{n/2}\\
\qed
\end{align*}

(B)
\begin{align*}
	F_n &<= 2^{cn}\\
	F_{n-2} + F_{n-1} &<= 2^{cn}\\
	2^{c(n-2)} + 2^{c(n-1)} &<= 2^{cn}\\
	2^{cn-2c} + 2^{cn-c} &<= 2^{cn}\\
	\frac{2^{cn}}{2^{2c}} + \frac{2^{cn}}{2^c} &<= 2^{cn}\\
	\frac{(2^{c})^n}{(2^{c})^2} + \frac{(2^{c})^n}{2^c} &<= (2^{c})^n\\
	\frac{a^n}{a^2} + \frac{a^n}{a} &<= a^n\\
	\frac{aa^n}{aa^2} + \frac{a^2a^n}{a^2a} &<= a^n\\
	\frac{a^{n+1}}{a^3} + \frac{a^{n+2}}{a^3} &<= a^n\\
	\frac{a^{n+1}}{a^3} + \frac{a^{n+2}}{a^3} &<= a^n\\
	\frac{a^{n+1}+a^{n+2}}{a^n} &<= a^3\\
	\frac{aa^n+a^2a^n}{a^n} &<= a^3\\
	\frac{a^n(a+a^2)}{a^n} &<= a^3\\
	a+a^2 &<= a^3\\
	2^c+2^{2c}&<=2^{3c}\\
	\log_2{(2^c+2^{2c})}&<=\log_2{(2^{3c})}\\
\end{align*}
Fib
0
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
6765
\end{document}


\appendix

\section{Butcher Tableau for IMEX Schemes}
\label{app:butcherTables}

For easy reference, we include the Butcher tableau for the IMEX schemes considered in this paper, which can be written in the standard double Butcher tableau
\begin{equation}
  \begin{array}{c | c}
    \tilde{\vect{c}} & \tilde{A} \\
    \hline
    & \tilde{\vect{w}}^{T}
  \end{array}
  \qquad
  \begin{array}{c | c}
    \vect{c} & A \\
    \hline
    \alpha & \vect{w}^{T}
  \end{array}.
  \label{eq:butcher}
\end{equation}
The \emph{explicit tableau} (left; components adorned with a tilde) represents the explicit part of the IMEX scheme, and the \emph{implicit tableau} (right; unadorned components) represents the implicit part of the IMEX scheme.
For $s$ stages, $\tilde{A}=(\tilde{a}_{ij})$, $\tilde{a}_{ij}=0$ for $j\ge i$, and $A=(a_{ij})$, $a_{ij}=0$ for $j>i$, are $s\times s$ matrices, and $\tilde{\vect{w}}=(\tilde{w}_{1},\ldots,\tilde{w}_{s})^{T}$ and $\vect{w}=(w_{1},\ldots,w_{s})^{T}$.
The vectors $\tilde{\vect{c}}=(\tilde{c}_{1},\ldots,\tilde{c}_{s})^{T}$ and $\vect{c}=(c_{1},\ldots,c_{s})^{T}$, used for non autonomous systems, satisfy $\tilde{c}_{i}=\sum_{j=1}^{i-1}\tilde{a}_{ij}$ and $c_{i}=\sum_{j=1}^{i}a_{ij}$.
For the implicit tableau, we have included the scalar $\alpha$, used for the correction step in Eq.~\eqref{eq:imexCorrection}.

For the analysis of convex-invariant IMEX schemes, additional coefficients are defined \cite{hu_etal_2018} (cf. Eq.~\eqref{eq:imexStagesRewrite}).
First, let
\begin{equation}
  b_{ii} = \f{1}{a_{ii}}, \quad
  b_{ij} = -\f{1}{a_{ii}}\sum_{l=j}^{i-1}a_{il}b_{lj}, \quad
  \tilde{b}_{ij} = -\f{1}{a_{ii}}\Big(\tilde{a}_{ij}+\sum_{l=j+1}^{i-1}a_{il}\tilde{b}_{lj}\Big).
\end{equation}
Then, for IMEX schemes of Type~A \cite{dimarcoPareschi2013},
\begin{equation}
  \begin{aligned}
    c_{i0} &= 1-\sum_{j=1}^{i-1}\sum_{l=j}^{i-1}a_{il}b_{lj}, \quad &
    c_{ij} &= \sum_{l=j}^{i-1}a_{il}b_{lj}, \\
    \tilde{c}_{i0} &= 0, \quad &
    \tilde{c}_{ij} &= \tilde{a}_{ij} + \sum_{l=j+1}^{i-1}a_{il}\tilde{b}_{lj};
  \end{aligned}
  \label{eq:positivityCoefficientsA}
\end{equation}
for IMEX schemes of Type~ARS \cite{ascher_etal_1997},
\begin{equation}
  \begin{aligned}
    c_{i0} &= 1-\sum_{j=2}^{i-1}\sum_{l=j}^{i-1}a_{il}b_{lj}, \quad &
    c_{ij} &= \sum_{l=j}^{i-1}a_{il}b_{lj} \\
    \tilde{c}_{i0} &= \tilde{a}_{i1}+\sum_{j=2}^{i-1}a_{ij}\tilde{b}_{j1}, \quad &
    \tilde{c}_{ij} &= \tilde{a}_{ij}+\sum_{l=j+1}^{i-1}a_{il}\tilde{b}_{lj}.
  \end{aligned}
  \label{eq:positivityCoefficientsARS}
\end{equation}
Note that $c_{i1}=\tilde{c}_{i1}=0$ in Eq.~\eqref{eq:positivityCoefficientsARS} so that $\sum_{j=0}^{i-1}c_{ij}=1$.
Also note the difference between the matrix coefficients in Eqs.~\eqref{eq:positivityCoefficientsA} and \eqref{eq:positivityCoefficientsARS} and the vector components defined below Eq.~\eqref{eq:butcher}.

\paragraph{IMEX PA2}

A second-order accurate, convex-invariant IMEX scheme of type $A$ (the matrix $A$ is invertible) with four implicit solves was given in \cite{hu_etal_2018}.
We refer to this scheme as IMEX PA2.
For this scheme, the non-zero components of $\tilde{A}$ and $A$ are given by
\begin{align*}
  \tilde{a}_{21} &= 0.7369502715, \\
  \tilde{a}_{31} &= 0.3215281691, \quad \tilde{a}_{32} = 0.6784718309, \\
  a_{11} &= 0.6286351712, \\
  a_{21} &= 0.2431004655, \quad a_{22} = 0.1959392570, \\
  a_{31} &= 0.4803651051, \quad a_{32} = 0.0746432814, \quad a_{33} = 0.4449916135.
\end{align*}
The coefficient in the correction step is $\alpha = 0.2797373792$ and the CFL constant is $c_{\Sch} = 0.5247457524$.
This scheme is globally stiffly accurate (GSA), so that $\tilde{w}_{i}=\tilde{a}_{3i}$ and $w_{i}=a_{3i}$ for $i\le3$.

\paragraph{IMEX PA2+}

We have found another second-order accurate, convex-invariant IMEX scheme of type $A$ with four implicit solves, which we refer to as IMEX PA2+.
This scheme allows for a larger value of $c_{\Sch}$ than IMEX PA2 (i.e., a larger time step while maintaining admissible solutions).
The scheme was found by random sampling of the parameter space spanned by the IMEX coefficients and selecting the scheme with the largest $c_{\Sch}$.
For IMEX PA2+, $c_{\Sch} = 0.895041066934$.
The non-zero components of $\tilde{A}$ and $A$ are given by
\begin{align*}
  \tilde{a}_{21} &= 0.909090909090909, \\
  \tilde{a}_{31} &= 0.450000000000000, \quad \tilde{a}_{32} = 0.550000000000000, \\
  a_{11} &= 0.521932391842510, \\
  a_{21} &= 0.479820781424967, \quad a_{22} = 0.002234534340252, \\
  a_{31} &= 0.499900000000000, \quad a_{32} = 0.001100000000000, \quad a_{33} = 0.499000000000000.
\end{align*}
The coefficient in the correction step is $\alpha = 0.260444263529413$.
This scheme is also GSA; $\tilde{w}_{i}=\tilde{a}_{3i}$ and $w_{i}=a_{3i}$ for $i\le3$.

The rest of the IMEX schemes we consider here do not include the correction step in Eq.~\eqref{eq:imexCorrection}; i.e., $\alpha=0$.

\paragraph{IMEX PC2}

Another IMEX scheme was given in \cite{mcclarren_etal_2008} (referred to there as a semi-implicit predictor-corrector method).
This scheme has two implicit solves and can be written in the double Butcher tableau form, and we refer to this scheme as IMEX PC2.
The non-zero components of $\tilde{A}$ and $A$ are given by
\begin{align*}
  \tilde{a}_{21} &= 0.5, \quad \tilde{a}_{32} = 1, \\
  a_{22} &= 0.5, \quad a_{33} = 1.0,
\end{align*}
$\alpha=0$, and $\tilde{w}_{i}=\tilde{a}_{3i} = w_{i}=a_{3i}$ for $i\le3$.
IMEX PC2 is not convex-invariant, since $c_{\Sch} = 0$ (cf. discussion in Section~\ref{sec:imex}).

\paragraph{IMEX PD-ARS}

We have found a family of convex-invariant, diffusion accurate IMEX schemes of type ARS that are second-order accurate in the streaming limit, which we refer to as IMEX PD-ARS; see \ref{app:PD-ARS}.
For these schemes, $c_{\Sch}= 1 - 2\epsilon$ with $\epsilon \in [0, 1/2)$.
Here we give an example by setting $\epsilon=0.1$:
\begin{align*}
  \tilde{a}_{21} & = 1.0, \\
  \tilde{a}_{31} & = 0.5, \quad \tilde{a}_{32} = 0.5, \\
  a_{22} & = 1.0, \nonumber \\
  a_{32} & = 0.4 \,( = 0.5 - \epsilon\,), \quad a_{33} = 0.6 \,( = 0.5 + \epsilon\,).
\end{align*}
This scheme is GSA, $\alpha=0$, and requires two implicit solves per time step (same as IMEX PC2).

\paragraph{IMEX RKCB2}

We compare the performance of the convex-invariant IMEX schemes with two other (not convex-invariant) IMEX schemes.
The first one is the second-order accurate IMEX scheme given in \cite{cavaglieriBewley2015} with two implicit solves.
We refer to this scheme as IMEX RKCB2.
The non-zero components of $\tilde{A}$ and $A$ are given by
\begin{align*}
  \tilde{a}_{21} &= 2/5, \quad \tilde{a}_{32} = 1, \\
  a_{22} &= 2/5, \nonumber \\
  a_{32} &= 5/6, \quad a_{33} = 1/6,
\end{align*}
$\alpha=0$, and $w_{i} = a_{3i} = \tilde{w}_{i}$ (stiffly accurate \cite{pareschiRusso_2005}).

\paragraph{IMEX SSP2332}

Another scheme that we use for comparison is the second-order accurate IMEX scheme given in \cite{pareschiRusso_2005} with three implicit solves.
We refer to this scheme as IMEX SSP2332.
The non-zero components of $\tilde{A}$ and $A$ are given by
\begin{align*}
  \tilde{a}_{21} &= 1/2, \\
  \tilde{a}_{31} &= 1/2, \quad \tilde{a}_{32} = 1/2, \\
  a_{11} &= 1/4, \\
  a_{22} &= 1/4, \\
  a_{31} &= 1/3, \quad a_{32} = 1/3, \quad a_{33} = 1/3,
\end{align*}
$\alpha=0$, and $w_{i} = a_{3i} = \tilde{w}_{i}$ (stiffly accurate).

\paragraph{SSPRK2 and SSPRK3}

To compare the performance of the IMEX schemes in the streaming limit (no collisions), we also compute results with explicit strong stability-preserving Runge-Kutta methods \cite{gottlieb_etal_2001}.
(All elements of the implicit Butcher tableau are zero.)
The optimal second-order accurate, strong-stability-preserving Runge-Kutta scheme (SSPRK2) has the following non-zero components:
\begin{align}
  \tilde{a}_{21} &= 1, \nonumber \\
  \tilde{w}_{1}  &= 1/2, \quad \tilde{w}_{2} = 1/2. \nonumber
\end{align}
The optimal third-order accurate, strong-stability-preserving Runge-Kutta scheme (SSPRK3) has the following non-zero components:
\begin{align}
  \tilde{a}_{21} &= 1, \nonumber \\
  \tilde{a}_{31} &= 1/4, \quad \tilde{a}_{32} = 1/4, \nonumber \\
  \tilde{w}_{1} &= 1/6, \quad \tilde{w}_{2} = 1/6, \quad \tilde{w}_{3} =2/3. \nonumber
\end{align}

\section{Construction of IMEX Scheme PD-ARS}
\label{app:PD-ARS}

Here we construct a three-stage PD-IMEX scheme of Type~ARS, conforming to Definition~\ref{def:PD-IMEX}.
We refer to the resulting IMEX scheme as PD-ARS.
For a 3-stage scheme, the double Butcher tableau is
\begin{equation}
  \begin{array}{c | c c c}
  	         0           & 0                 & 0                   & 0  \\
  	\tilde{c}_{2} & \tilde{a}_{21} & 0                   & 0  \\
  	\tilde{c}_{3} & \tilde{a}_{31} & \tilde{a}_{32} & 0  \\ \hline
  	                   & \tilde{a}_{31} & \tilde{a}_{32} & 0
  \end{array}
  \qquad
  \begin{array}{c | c c c}
  	     0  & 0  & 0         & 0          \\
  	c_{2} & 0 & a_{22} & 0          \\
  	c_{3} & 0 & a_{32} & a_{33}  \\ \hline
  	         & 0 & a_{32} & a_{33}
  \end{array}
\end{equation}
The problem is then to find the coefficients $\{ \tilde{a}_{21}, \tilde{a}_{31}, \tilde{a}_{32}, a_{22}, a_{32}, a_{33} \}$ satisfying the constraints in Definition~\ref{def:PD-IMEX} while maximizing
\begin{equation}
  c_{\Sch} = \min \Big\{\, \dfrac{c_{20}}{\tilde{c}_{20}},\, \dfrac{c_{30}}{\tilde{c}_{30}},\, \dfrac{c_{32}}{\tilde{c}_{32}} \,\Big\}.
\end{equation}
By imposing the equality constraints (i.e., Eqs.~\eqref{eq:diffusionCondition}, \eqref{eq:orderConditionsEx}, and \eqref{eq:implicitConsistency}), the double Butcher tableau can be written in terms of two independent parameters ($x,y\in\bbR$) as
\begin{equation}
  \begin{array}{c | c c c}
  	     0       & 0            & 0 & 0 \\
  	\frac{1}{2x} & \frac{1}{2x} & 0 & 0 \\
  	     1       & 1-x          & x & 0 \\ \hline
  	             & 1-x          & x & 0
  \end{array}
  \qquad
  \begin{array}{c | c c c}
  	     0       & 0 & 0            & 0 \\
  	\frac{1}{2x} & 0 & \frac{1}{2x} & 0 \\
  	     1       & 0 & 1-y          & y \\ \hline
  	             & 0 & 1-y          & y
  \end{array}
\end{equation}
Computing the relevant coefficients in Eq.~\eqref{eq:positivityCoefficientsARS}, we find $c_{20}=1$, $c_{30}=1-2x(1-y)$, $c_{32}=2x(1-y)$, $\tilde{c}_{20}=\f{1}{2x}$, $\tilde{c}_{30}=(y-x)$, and $\tilde{c}_{32}=x$, so that
\begin{equation}
  c_{\Sch} = \min \Big\{\, 2x,\, \f{1-2x(1-y)}{y-x},\, 2(1-y) \,\Big\}.
\end{equation}
The convex-invariant property requires imposing the inequality constraints $a_{22},a_{33}>0$, $c_{20},c_{30},c_{32}\ge0$, and $\tilde{c}_{20},\tilde{c}_{30},\tilde{c}_{32}\ge0$, which imply that
\begin{equation}
  0 < x \le y
  \quad\text{and}\quad
  0 < y \le 1.
\end{equation}
We chose $x=\f{1}{2}$, so that the explicit part of the IMEX scheme is equivalent to the optimal second-order SSP-RK scheme in \cite{gottlieb_etal_2001} (SSPRK2 in \ref{app:butcherTables}).
Then, $y=\f{1}{2} + \epsilon$, where $\epsilon \in [0, \frac{1}{2})$, and $c_{\Sch} = 1 - 2\epsilon$ results in the PD-ARS IMEX scheme.
Setting $\epsilon = 0$ gives the optimal scheme with $c_{\Sch} = 1$.

\section{Nonexistence of Three-Stage PD-IMEX Scheme of Type~A}
\label{app:noTypeA}

Here we prove that a PD-IMEX scheme  of type~A (i.e., conforming to Definition~\ref{def:PD-IMEX}, but with Eq.~\eqref{eq:positivityConditionsTypeA} replacing Eq.~\eqref{eq:positivityConditionsTypeARS} in item 3) does not exist.
First, for a three-stage, GSA IMEX scheme of type~A the double Butcher tableau is
\begin{equation*}
  \begin{array}{c | c c c}
  	      0       & 0              & 0              & 0 \\
  	\tilde{c}_{2} & \tilde{a}_{21} & 0              & 0 \\
  	\tilde{c}_{3} & \tilde{a}_{31} & \tilde{a}_{32} & 0 \\ \hline
  	              & \tilde{a}_{31} & \tilde{a}_{32} & 0
  \end{array}
  \qquad
  \begin{array}{c | c c c}
    c_{1} & a_{11} & 0      & 0      \\
  	c_{2} & a_{21} & a_{22} & 0      \\
  	c_{3} & a_{31} & a_{32} & a_{33} \\ \hline
  	      & a_{31} & a_{32} & a_{33}
  \end{array}.
\end{equation*}
First we consider the equality constraints.
Consistency of the implicit coefficients and second-order accuracy in the streaming limit (Eqs.~\eqref{eq:implicitConsistency} and \eqref{eq:orderConditionsEx}, respectively) give
\begin{align}
  a_{31} + a_{32} + a_{33} = 1, \quad \tilde{a}_{31} + \tilde{a}_{32} = 1, \quad \text{and}\quad \tilde{a}_{32}\,\tilde{a}_{21} =  \f{1}{2}.
  \label{eq:1stOrderGeneral}
\end{align}
Accuracy in the diffusion limit (Eq.~\eqref{eq:diffusionCondition}) requires
\begin{align}
  \f{\tilde{a}_{21}}{a_{22}} = 1
  \quad\text{and}\quad
  -\f{a_{32}\,\tilde{a}_{21}}{a_{22}\,a_{33}} + \f{\tilde{a}_{31}+\tilde{a}_{32}}{a_{33}} = 1.
\label{eq:diffusionAccurate}
\end{align}
Eq.~\eqref{eq:diffusionAccurate} together with the second constraint in Eq~\eqref{eq:1stOrderGeneral} gives $a_{32}+a_{33}=1$, which together with the first constraint in Eq.~\eqref{eq:1stOrderGeneral} gives
\begin{align}
  a_{31} = 0.
  \label{eq:a31is0}
\end{align}
Next we consider the inequality constraints.
The convex-invariant property in Eq.~\eqref{eq:positivityConditionsTypeA} requires $a_{11}, a_{22}, a_{33}>0$, and
\begin{equation}
  c_{21} = \frac{a_{21}}{a_{11}} \geq 0, \quad
  c_{31} = \frac{a_{31}}{a_{11}} - \frac{a_{32}\,a_{21}}{a_{22}\,a_{11}} \geq 0, \quad\text{and}\quad
  c_{32} = \frac{a_{32}}{a_{22}} \geq 0.
  \label{eq:constraintsTypeA}
\end{equation}
As a consequence, $a_{21}, a_{32} \geq 0$.
However, since $a_{31} = 0$,
\begin{equation*}
  c_{31} = - \frac{a_{32}a_{21}}{a_{22}a_{11}} \le 0.
\end{equation*}
Thus the inequality constraints in Eq.~\eqref{eq:constraintsTypeA} hold only for $c_{31} = 0$, which gives $c_{\Sch} = \min \Big\{\,\f{c_{21}}{\tilde{c}_{21}}, \f{c_{31}}{\tilde{c}_{31}}, \f{c_{32}}{\tilde{c}_{32}}\Big\} = 0$.
Therefore, a three-stage PD-IMEX scheme (Definition~\ref{def:PD-IMEX}) of type~A does not exist.