\section{Introduction}

\subsection{Exercise 1}
It seems intuitive that the described self-play would converge to a maximin (or minimax, depending on order of play) policy, since the agent is
competing with itself (alternating between minimizing and maximizing its value function).

We can make this notion more precise by introducing some notation. Suppose that we are playing Tic-Tac-Toe
with ``X'' going first. Let $s$ be an arbitrary game state, $V(s)$ be the probability that ``O'' wins from $s$, $N(s)$ be the neighboring states of $s$,
and $A_X(s)$ and $A_O(s)$ be the next states chosen by the two respective player policies.
Then we have
\begin{align*}
        A_X(s) &= \argmax_{s' \in N(s)} 1 - V(s') = \argmin_{s' \in N(s)} V(s') \\
        A_O(s) &= \argmax_{s' \in N(s)} V(s') \\
        V(s) &\gets V(s) + \alpha [V(A_O(A_X(s))) - V(s)]
\end{align*}
from which we can see that if $V(s)$ converges, it converges to a maximin policy. This policy is not
necessarily the same as the one learned by playing against a random opponent for the reasons stated in the
text (a random opponent could make suboptimal moves).

\subsection{Exercise 2}
By mapping rotations and reflections of a given game state to a single state, we could significantly shrink
the size of our value function table along with the search space of moves. However, we can only do this if
the opponent also regards symmetric game positions as identical. If the opponent takes different actions
in symmetric positions, then treating symmetric positions as the same would lead to sub-optimal play on our
part.

\subsection{Exercise 3}
A greedy player could converge to a local optima, which may be worse than a nongreedy player who explores more
of the search space. To illustrate the kind of problems that could occur, consider the following simplified
scenario: the greedy player has just made its best move from position $s$, and the opponent must choose
between two moves leading to states $u$ and $v$ respectively. Suppose we only have one move from either of
the positions $u$ and $v$, and that we always lose from $u$ and always win from $v$. Furthermore, suppose
the opponent chooses the move that leads to $v$ with probability $p \gg V(s)$. Then, letting $s'$ be the final
state we reach, we can compute the change to $V(s)$ as
\begin{align*}
        \mathrm{E} [\alpha (V(s') - V(s))] = \alpha (\mathrm{E} [V(s')] - V(s)) = \alpha (p - V(s)) > 0
\end{align*}
so we will most likely keep making the same move from $s$. Thus, if there is another move from $s$ that
would actually lead to a certain win against the opponent's policy, we will not discover it.

\subsection{Exercise 4}
When we do not learn from exploratory moves, our expected updates are as described in the text:
\begin{align*}
        V(s) &\gets V(s) + \alpha [V(s') - V(s)]
\end{align*}
with $V(s)$ indicating the probability that we win from state $s$ assuming we make our best moves.
If we instead explore with likelihood $\epsilon$, then our expected updates look more like:
\begin{align*}
        V(s) &\gets V(s) + \alpha [(1 - \epsilon) V(s') + \epsilon \sum_{s'' \neq s'} P(s'') V(s'') - V(s)]
\end{align*}
where $P(s'')$ indicates the probability with which we end up in the non-greedy state $s''$ during exploration.
In this case, we can interpret the probability $V(s)$ to be the expected probability of winning assuming
we make our best move with probability $\epsilon$ and an exploratory move with probability $1 - \epsilon$.

If we assume that we want to keep making exploratory moves, then it makes more sense to learn the latter
set of probabilities, as they incorporate information from all possible moves from a given state. In terms of
winning, however, it seems like the former set of probabilities would be better, as they hone in on the
probabilities corresponding to the best possible outcomes.

\subsection{Exercise 5}
Some potential improvements that come to mind:
\begin{itemize}
        \item Track whether opponent plays symmetrically or not during learning, and then condense value
                function table afterwards if possible.
        \item If at any point during learning we play a sequence of moves that leads to a win, try to play
                the exact same sequence of moves again. Keep doing so for as long as it is possible to play
                the same sequence. This would help against a deterministic opponent - if we can win once, we
                can always win.
        \item We can prune the search space. We can disregard (set winning probability to 0)
                all states from which the opponent only has a single move, with that move being a winning move.
\end{itemize}

Perhaps a better approach to this problem would be to combine reinforcement learning with minimax optimal
play. In other words, if we reach a state where we can win no matter what the opponent does, then there is
no need to learn anything; we just use the minimax strategy from that state.


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}

\usepackage{hyperref}
\title{Exercise in aggregate signatures}
\author{Todor Milev}

\begin{document}
\maketitle
Aggregate signatures are one of the core technologies used by Fabcoin's Kanban system. Aggregate signatures have been proposed as early as 1995 in \cite{Horster1995} - an overview of the subject can be found, for example, in \cite{cryptoeprint:SimpleSchnorrMultisignatures}. Recently, aggregate signatures have been proposed as an optimization for Bitcoin's network. Besides Fabcoin, a number of crypto-currencies have put aggregate signatures in the core of their technology. In the present text we discuss one such scheme recently implemented by the Zilliqa crypto-currency and then propose an exercise on the topic.

The purpose of an aggregate signature is to have a message $m$ be signed by multiple entities. The signatories combine their public keys to a single aggregate one with the same size as a regular public key. Likewise, the signatories aggregate their signature to one of regular size. The entire process happens under the assumption that the entities do not trust one another and do not share any secrets. The resulting aggregate signature achieves the same goal as would be achieved by having each of the individual signatures for $m$ of every signatory, but is far shorter. To see why this optimization is significant consider the task of having $1000$ nodes sign a small message $m$ - say $90$ bytes. A standard ECDSA signature is $65$ bytes. This means that all $1000$ signatures would take $1000\cdot 65$ bytes - much more than a single aggregate signature.

Our exercise is expected to be completed ``by hand'' with the help of online cryptography tools or tools provided by FAB coin. Here, ``by hand'' means that individual cryptographic operations are expected to be carried via online tools/ready software with intermediate results recorded (copy+paste) in a resulting homework paper. The resulting homework paper is expected to have explanations and notation written by the student using proper academic language and references. The exercise can be expanded to a larger project by requesting an exposition of the theory behind aggregate signatures. The exercise can be further (significantly) expanded by requesting that the students carry out part (or all) of the computations using their own code (which will be evaluated), rather than using online tools.

\section{The exercise}
The Zilliqa aggregate signature scheme can be roughly described (omitting some technicalities) as follows. After each step, we give an exercise to be carried out by the student. The exercises roughly requests that the student simulate (on paper) the steps of the aggregate signature for $1$ aggregator and $3$ signer nodes.

The student is expected to properly cite all tools (online or otherwise) used to carry out the intermediate computations in the exercise.

\subsection{Preparation step}
In this step, the signers prove their identities to a special node called an aggregator.
\begin{itemize}
\item Select a special node called an aggregator.
\item Each signer: one-time send public key to aggregator.
\item Aggregator: send back challenge message to be signed.
\item Each signer: send signed challenge back.
\item Aggregator: verify the signers' signatures.
\end{itemize}

\subsubsection{Proposed exercise}
\begin{itemize}
\item The student prepares a challenge message and presents its encoding in a standard format (hex, base64, $\dots$).
\item The student generates $3$ public-secret key pairs corresponding to three signer nodes. Encodings  of the three pairs are presented.
\item The student generates $3$ signatures for the challenge message. Encoded results are presented.
\item[Optional] The student presents intermediate computations in the verification of the $3$ signatures.
\end{itemize}

\subsection{Signature aggregation}
In this step, the aggregator node composes an aggregate signature. The student should skip all steps related to networking and assume that all messages send between the nodes arrive without error.

\begin{itemize}
\item Each signer: choose random $\mathrm{nonce}_i$, compute $q_i = g^{\mathrm{nonce}_i}$. Here, we assume $g$ is the generator of the elliptic curve $y^2=x^3+7$ over $\mathbb Z/p\mathbb Z$ with $p = 2^{256}- 2^{32}-977$ - i.e., curve secp256k1, the standard curve used in Fabcoin and Bitcoin.
\item Each signer: send $q_i$ to aggregator.
\item Aggregator: compute $ \mathrm{Pub} =\prod_{i} \mathrm{pub}_i =\mathrm{pub}_1\cdot \dots \cdot \mathrm{pub}_n $.
\item Aggregator: compute $ Q = \prod_i q_i$.
\item Aggregator: compute  $\mathrm{challenge} = H(Q, \mathrm{Pub}, \mathrm{digest})$.
\item Aggregator: send $\mathrm{challenge},\mathrm{Pub}, \mathrm{digest}$ to signers.
\item Each signer: verify $\mathrm{challenge} = H(Q, \mathrm {Pub}, \mathrm{digest}) $.
\item Each signer: compute $\mathrm{solution}_i = {\mathrm{nonce}_i - \mathrm{challenge} \cdot \mathrm{secret}_i} $
\item Each signer: send $\mathrm{solution}_i$ to aggregator.
\item Aggregator: compute $\mathrm{solution} = \sum_i \mathrm{solution}_i $.
\item Aggregator: final signature: $(\mathrm{challenge}, \mathrm{solution})$.

\end{itemize}

\subsection{Proposed exercise}
\begin{itemize}
\item The student presents hex encodings of the generator $g$ of secp256k1, of $p = 2^{256}- 2^{32}-977$ and of the order (number of elements) of secp256k1.
\item The student selects $\mathrm{nonce}_1, \mathrm{nonce}_2,\mathrm{nonce}_3$. Encodings are presented.

\item The students computes the $q_i$'s. Encodings are presented.
\item The student computes $Q$. Encoding is presented.
\item The students computes $\mathrm{challenge}$ and presents its encoding.
\item The student computes each $\mathrm{solution}_i$ and presents encodings.
\item The student computes the final signature and presents its encoding.
\end{itemize}

\subsection{Aggregate signature verification}
[To be written].

\bibliographystyle{plain}
\bibliography{../bibliography}

\end{document}


\documentclass[11pt]{article}

\setlength{\oddsidemargin}{-0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.9 in}
\setlength{\textwidth}{7.0 in}
\setlength{\textheight}{9.0 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0.3 in}
\setlength{\parskip}{0.1 in}
\usepackage{epsf}
\usepackage{pseudocode}
\usepackage[shortlabels]{enumitem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{color}
\usepackage[normalem]{ulem}
\usepackage{graphicx}
\usepackage[export]{adjustbox}
\usepackage{wrapfig}
\pagenumbering{arabic}
\def\O{\mathop{\smash{O}}\nolimits}
\def\o{\mathop{\smash{o}}\nolimits}
\newcommand{\e}{{\rm e}}
\newcommand{\R}{{\bf R}}
\newcommand{\Z}{{\bf Z}}

\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{booktabs}

\usepackage{graphicx}
\usepackage{tikz}
\usetikzlibrary{arrows.meta}

%% display solutions or not
\newif\ifsol
\soltrue % comment out to hide solutions

\title{Section 11: Classification and Clustering}
\date{}
\author{CS 182 - Artificial Intelligence}
\begin{document}
\maketitle

\noindent Algorithms in machine learning fall under {\bf{supervised}} and {\bf{unsupervised}} approaches. In the former, we are given a set of inputs and labels/outputs, and would like to learn the relationship between them to guess label/output values for new data. In the latter, we are only given input values, and aim to find structure within them.

\section*{Supervised Learning}

Supervised learning can also be broken down into {\bf{parametric}} and {\bf{non-parametric}} approaches. Parametric algorithms, such as Naive Bayes or regression, rely on learning a set of parameters that define the classifier. Non-parametric algorithms, such as k-NN, rely directly on data rather than on a parameterized model.

\subsection*{Naive Bayes}

Naive Bayes can be thought of as a {\bf{probabilistic inference algorithm}} over a special case of a Bayes Net that is applied to classification. The model is made of an unknown variable $Y$ representing the possible classes, which causes a collection of emission variables $E_1$, $E_2$, $\dots$, $E_n$. We make a (``naive'') assumption that each of the emissions is conditionally independent of one another given $Y$. While this may not be true for most applications, it still results in a simple and useful algorithm. Finally, given the emissions, we can calculate classification probabilities by inference over the Bayes Net:
\[
P(Y | x_1, \ldots, x_n) \propto P(Y, x_1, \ldots, x_n) = P(Y) \prod_{i=1}^n P(x_i|Y)
\]
To build up this network from labeled data, we can count the frequencies of occurrence of observing $(x_i, Y)$ for each $x_i$ and normalize to construct conditional probabilities. Improvements on this approach involve {\bf{smoothing}}, which is discussed further down.

\subsection*{Regression}

When the output variable $y$ is continuous rather than discrete, we can perform regression. First, a {\bf{model}} $h_{\theta}$ is chosen to describe the input-output relationship, where the {\bf{model parameters}} $\theta$ are calculated by optimizing over a {\bf{loss function}} $L$ on the training data. For instance, if we assume that the underlying model is {\bf{linear}}, we choose $h_{\theta}(f^{(i)}) = \theta^T f^{(i)}$ for every $i$th data vector. Often, a quadratic loss function is chosen. This is both intuitive, but also theoretically supported -- a quadratic is the {\bf{maximum likelihood estimate}} of the data given the model parameters, under the assumption of i.i.d. Gaussian noise in the data. Overall, the optimization problem to solve {\bf{linear regression}} would be:
\[
\min_{\theta} \; \; \sum_{i = 1}^m (\theta^T f^{(i)} - y^{(i)})^2
\]
There are other possible loss functions, such as:
\begin{itemize}
    \item \bf{absolute ($L_1$) loss}: $L = |h_{\theta}(f) - y|$
    \item \bf{deadband loss}: $L = \max(0, |h_{\theta}(f) - y| - \epsilon)$
\end{itemize}

\noindent {\bf{Logistic regression}} is a variant of regression designed for binary classification (where the output variable does assume discrete values $0$ and $1$). Instead of using the linear function $h_{\theta}(f) = \theta^T f$, we apply a sigmoid:
\[
h_{\theta} = \frac{1}{1 + e^{-\theta^T f}}
\]
Similarly to linear regression, we choose a loss function, pose an optimization problem, and solve for the optimal parameters $\theta$.

\noindent While linear regression with a quadratic loss is {\bf{convex}} and has a closed-form solution, most of these optimization problems are non-convex and are solved with {\bf{stochastic gradient descent}}. Finally, after an optimization approach is applied and the model parameters $\theta$ are determined, we can guess the output of new data $f$ by calculating $h_{\theta}(f)$ (for the binary logistic case, rounding it to 0 or 1).

\subsection*{k-Nearest Neighbors}
This is a simple {\bf{non-parametric}} classification approach which assigns a label to a new point by choosing the most common class of the $k$ closest neighboring points. Variations on this algorithm might give a weighting scheme to the $k$ labels rather than choosing the most common value.

\noindent When designing an ML algorithm, the available data is usually broken up into the {\bf{training data}}, which is used to train the classifier, and the {\bf{test data}}, which is used to evaluate the classifier and can in no way be viewed or used beforehand. \noindent This division of datasets allows to deal with two issues:
\begin{enumerate}
    \item {\bf{Overfitting}} occurs when a classifier is trained for very high (near 100\%) training accuracy, but is not able to {\bf{generalize}} to the testing data, and has low test accuracy.
    \begin{enumerate}
        \item {\bf{Smoothing}} is used in discrete problems to cause more gradual updates to probabilities. For instance, one example of overfitting would be giving zero probabilities to all words not in the training set, which can be mitigated with {\bf{Laplace smoothing}}:
        \begin{align*}
        &\text{standard probability:} \; \; P_{ML}(x) = \frac{count(x)}{N} \\
        &\text{with smoothing:} \; \; P_{LAP, k}(x) = \frac{count(x) + k}{N + k|X|}
        \end{align*}
        \item {\bf{Regularization}} is used in continuous problems such as regression. For instance, an example of overfitting would be using a high-order polynomial that goes through all training data points. The addition of a regularization term to the loss function places an analytic penalty on problem parameters to encourage smaller values or sparsity:
        \begin{align*}
        &\text{standard loss function:} \; \; loss(f^{(i)}, y^{(i)}, \theta) = (\theta^T f^{(i)} - y^{(i)})^2 \\
        &\text{with $L_2$-regularization:} \; \; loss(f^{(i)}, y^{(i)}, \theta) = (\theta^T f^{(i)} - y^{(i)})^2 + \lambda ||\theta||_2^2 \\
        &\text{with $L_1$-regularization:} \; \; loss(f^{(i)}, y^{(i)}, \theta) = (\theta^T f^{(i)} - y^{(i)})^2 + \lambda ||\theta||_1
        \end{align*}
    \end{enumerate}
    \item Sometimes algorithms depend on a choice of {\bf{hyperparameters}}, which must be decided outside of the training process. There are a couple common approaches to tuning:
    \begin{enumerate}
        \item Part of the training data is taken to be the {\bf{held-out dataset}}. For each hyperparameter value, the classifier is trained with the rest of the training data and tested on the held-out data, and the one with the highest accuracy is chosen.
        \item If there are $m$ possible values for a hyperparameter, {\bf{cross-validation}} divides the training data into $m$ sections. For the $i$th value, the $i$th section is chosen to be the tuning test set and the others are chosen to be the tuning training sets. After evaluating all hyperparameter values, the one with the highest tuning test accuracy is chosen.
    \end{enumerate}
\end{enumerate}

\section*{Unsupervised Learning}

Unsupervised learning algorithms focus primarily on clustering, dimensionality reduction, anomaly detection, or other ways of inferring data structure without labels. The algorithms we have discussed all fall under clustering.

\noindent {\bf{K-means}} is a {\bf{centroid-based}} clustering algorithm. First, initialize $k$ cluster centroids. Then, iteratively:

\noindent Assign each point to nearest cluster centroid:
\[
a_i = \text{argmin}_k \; \text{dist}(x_i, c_k), \;\; i = 1, \dots, N
\]

\noindent Recompute the $k$th cluster centroid by taking the mean of the points assigned to cluster $k$:
\[
c_k = \frac{1}{|a_i \;:\; a_i = k|} \sum_{(i \;:\; a_i = k)} x_i, \;\; k = 1, \dots, K
\]

\noindent {\bf{Agglomerative clustering}} and {\bf{divisive clustering}} are both {\bf{connectivity-based}} clustering algorithms. The former iteratively constructs larger clusters from smaller clusters. The latter iteratively breaks down large clusters into smaller clusters.

\noindent {\bf{Choosing K:}} The outcome of all of these algorithms vary on the chosen number of clusters. Some common approaches for choosing $k$ are:
\begin{itemize}
    \item We can plot the average of the distances from each point to its cluster's centroid. Then, it is reasonable to choose the smallest value of $k$ for which this ``error'' flattens out.
    \item {\bf{Silhouette diagrams}} are plots of the measure
    \[
    m_i = \frac{(b_i - a_i)}{\max(a_i, b_i)}
    \]
    where $a_i$ is the mean distance from the point $x_i$ to all points in its cluster, and $b_i$ is the mean distance from $x_i$ to all points in the neighboring (``next best'') cluster.
\end{itemize}

%\begin{figure}[ht]
%\renewcommand{\labelenumii}{\arabic{enumii}.}
%\setlength{\parindent}{0pt}
%    \includegraphics[scale=0.4]{figs/carrepair}
%    \caption{Conditional implications example (left), example conditional probability tables (center), and example bayes net for car repair (right)}
%    \label{fig:bayes}
%\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage
\section*{Exercises:}
\begin{enumerate}

\item{{\bf{Silhouette Diagrams.}}}

Suppose we are trying to cluster an unlabeled dataset. We apply k-means with $k = 3$, and are evaluating the clusters with a with a silhouette diagram.
\begin{enumerate}
    \item Suppose that for a certain point $x_i$, $a_i << b_i$. What will be the approximate value of $m_i$, and what does this imply about the clustering of $x_i$? \\
    \ifsol
        {\color{blue} If $a_i$ is small relative to $b_i$, this is a good clustering of $x_i$, since it is (on average) much closer to points in its own cluster than the next best cluster. The measure $m_i$ will be close to 1.}
    \else
        \vspace{1cm}
    \fi
    \item Suppose that for a certain point $x_i$, $a_i \approx b_i$. What will be the approximate value of $m_i$, and what does this imply about the clustering of $x_i$? \\
    \ifsol
        {\color{blue} If $a_i$ is similar to $b_i$, this is not a very good clustering of $x_i$, since it is about the same distance away from points in its own cluster as those in the next best cluster. The measure $m_i$ will be close to 0.}
    \else
        \vspace{1cm}
    \fi
    \item Suppose that for a certain point $x_i$, $a_i >> b_i$. What will be the approximate value of $m_i$, and what does this imply about the clustering of $x_i$? \\
    \ifsol
        {\color{blue} If $a_i$ is much greater than $b_i$, this is an incorrect clustering of $x_i$, since it is much farther from points in its own cluster as those in the next best cluster (this situation should be impossible with k-means). The measure $m_i$ will be close to -1.}
    \else
        \vspace{1cm}
    \fi
\end{enumerate}

\item{{\bf{Regression.}}}

You are working for a hospital, and are given a dataset of patient information with hundreds of continuous features (height, weight, blood pressure, etc) along with whether the patient has diabetes. Your goal is to predict whether incoming patients will have diabetes.

\begin{enumerate}
\item Formalize the problem as an optimization problem using logistic regression. \\
\ifsol
    {\color{blue} Let $f^{(i)}$ be the vector of features for the $i$th patient, $y^{(i)}$ be the true outcome (diabetes or no diabetes) for each patient, and $\theta$ be the parameters we are learning. Then, if we use a sum-of-squared-error loss function, we are trying to solve the problem:
    \[
    \hat{\theta} = \text{argmin}_{\theta} \; \sum_{i} (\frac{1}{1 + e^{-\theta^T f^{(i)}}} - y^{(i)})^2
    \]

    }
\else
    \vspace{1cm}
\fi

\item Write down the formula your model would use to compute the probability that a patient has diabetes. \\
\ifsol
    {\color{blue} Once the parameters $\theta$ are learned by solving the problem above, a new patient's probability of having diabetes given a feature vector $f$ will be
    \[
    P(y | f) = \frac{1}{1 + e^{-\theta^T f}}
    \]
    .}
\else
    \vspace{1cm}
\fi

\item Suppose you are now trying to estimate blood pressure using linear regression on the 20 features you consider most important. If you calculate a regression model on a training set of 12 patients, what (approximately) will be your training set error? \\
\ifsol
    {\color{blue} In this problem, we have 20 ``unknowns'' (the parameters $\theta$ and only 12 ``equations'' (the training data). Therefore, unless the training data contains equations which exactly contradict each other, this is an under-determined system, and we can choose $\theta$ such that all the total error is 0. This is an example of overfitting.}
\else
    \vspace{1cm}
\fi
\end{enumerate}

\newpage

\item{\bf{Naive Bayes}}

In this problem, we will be using Naive Bayes to predict the probability that a given email is spam or not spam (ham). We assume a model in which an underlying unobserved state $y$ (spam/ham) creates emissions (words) $w_1, \ldots, w_n$. We use a bag-of-words approach to model text -- the ordering of words within a sentence is ignored. Then, $P(Y)$ is the prior probability across states $Y$, and  $p(w_i|Y)$ are the conditional probabilities of seeing a word $w_i$ given $Y$. For the problems, start by assuming a uniform distribution over $Y$, and use the following conditional probability table over words:

% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
\begin{table}[h!]
\centering
\begin{tabular}{@{}lll@{}}
\toprule
        & $p(w|ham)$ & $p(w|spam)$ \\ \midrule
CS182   & 0.05     & 0.01      \\
friend  & 0.02     & 0.04      \\
need    & 0.01     & 0.01      \\
on      & 0.02     & 0.02      \\
our     & 0.02     & 0.02      \\
project & 0.007    & 0.003     \\
start   & 0.01     & 0.03       \\
to      & 0.01     & 0.01      \\
you     & 0.01     & 0.015     \\
we      & 0.01     & 0.015     \\
working & 0.01     & 0.03      \\
...     & ...      & ...      \\ \bottomrule
\end{tabular}
\end{table}

\begin{enumerate}
\item
What is the probability of seeing the following spam email: ``we need you on our project friend''?\\
\ifsol
\textcolor{blue}{\[
p(email|spam) = 0.015*0.01*0.015*0.02*0.003*0.04 = 5.4 *10^{-12}
\]}
\else
    \vspace{1cm}
\fi

\item
What is the probability that the following text is spam: ``start CS182 project''?\\
\ifsol
\textcolor{blue}{\begin{align*}
p(email|ham) &= 0.01 * 0.05 * 0.007 = 0.0000035 \\
p(email|spam) &= 0.03 * 0.01 * 0.003 = 0.0000009\\
p(spam|email) &\propto 0.5 * 0.0000009\\
p(ham|email) &\propto 0.5 * 0.0000035%\frac{0.5 * 0.0000009}{0.0000009+0.0000035} = 0.102
\end{align*}
We can normalize this and get a probability for $p(spam|email) = 0.205$.
}
\else
    \vspace{1cm}
\fi

\item We now consider a fresh inbox for which we have no previous data. We want to build a new classifier from scratch. The inbox right now contains three emails:
\begin{enumerate}
\item [spam:] ``I am only prince''
\item [spam:] ``I will send money''
\item [ham:] ``I will send you the review for it''
\end{enumerate}
Determine the prior over Y (spam/ham), fill in the probability table below, and calculate the probabilities that the email ``I will send'' is spam/ham.\\
\ifsol
\begin{table}[h!]
\centering
\begin{tabular}{@{}lllll@{}}
\toprule
       & freq ham & $p(w|ham)$ & freq spam & $p(w|spam)$ \\ \midrule
am     & 0        & 0          & 1         & 1/8         \\
for    & 1        & 1/8        & 0         & 0           \\
I      & 1        & 1/8        & 2         & 2/8         \\
it     & 1        & 1/8        & 0         & 0           \\
money  & 0        & 0          & 1         & 1/8         \\
only   & 0        & 0          & 1         & 1/8         \\
prince & 0        & 0          & 1         & 1/8         \\
review & 1        & 1/8        & 0         & 0           \\
send   & 1        & 1/8        & 1         & 1/8         \\
the    & 1        & 1/8        & 0         & 0           \\
you    & 1        & 1/8        & 0         & 0           \\
will   & 1        & 1/8        & 1         & 1/8         \\ \bottomrule
\end{tabular}
\end{table}
\else
\begin{table}[h!]
\centering
\begin{tabular}{@{}lllll@{}}
\toprule
       & freq ham & $p(w|ham)$ & freq spam & $p(w|spam)$ \\ \midrule
am     &         &           &          &          \\
for    &         &         &          &            \\
I      &         &         &          &          \\
it     &         &         &          &            \\
money  &         &           &          &          \\
only   &         &           &          &          \\
prince &         &           &          &          \\
review &         &         &          &            \\
send   &         &         &          &          \\
the    &         &         &          &            \\
you    &         &         &          &            \\
will   &         &         &          &          \\
\bottomrule
\end{tabular}
\end{table}
\fi

\ifsol
\textcolor{blue}{We can determine the prior by counting the number of emails from each category:  $p(spam) = 2/3$ and $p(ham) = 1/3$. To construct the table, we count the word frequencies and normalize by the number of words. Using this table, we can take the same approach as before:}
\begin{align*}
p(email|ham) &= \left(1/8\right)^3 = 0.00195\\
p(email|spam) &= 2/8 * 1/8 * 1/8 = 0.0039\\
p(ham|email) &\propto \frac{1}{3} * 0.00195 = 0.00065\\%\frac{0.00195}{0.00195+0.0039} = 0.111\\
p(spam|email) &\propto \frac{2}{3} * 0.0039 = 0.0026\\%\frac{0.0039}{0.00195+0.0039} = 0.444
\end{align*}

\textcolor{blue}{Normalizing this yields 0.2 and 0.8 probabilities for the two classes respectively.}
\else
    \vspace{1cm}
    \newpage
\fi

\item For long texts, you will need to multiply many numbers smaller than 1. This can cause implementation problems, where all of the predicted probabilities are so small that they are numerically rounded to zero. What can you do to address this issue? Why does it work?\\
\ifsol
\textcolor{blue}{In order to avoid those issues, you can add up the log-probabilities. This works because $$P(Y) \prod_{i=1}^n P(w_i|Y) = e^{\log (P(Y) \prod_{i=1}^n P(w_i|Y))} = e^{\log P(Y) \sum_{i=1}^n \log P(w_i|Y)}$$}
\else
    \vspace{1cm}
\fi

\item Use your classifier to predict the emails ``Will you send the review'' and ``Please send the review''.\\
\ifsol
\textcolor{blue}{Those two are special cases. Since $p(review|spam)=0)$, we have a 0 probability for the whole term. The word ``please'' does not occur in the table and with the current approach, we can't compute the probabilities.} \newpage
\else
    \vspace{1cm}
\fi

\item Repeat task (c) with laplace smoothing and $k=1$.\\

\ifsol
\textcolor{blue}{
\begin{table}[h!]
\centering
\begin{tabular}{@{}lllll@{}}
\toprule
       & freq ham & $p(w|ham)$ & freq spam & $p(w|spam)$ \\ \midrule
am     & 1        & 1/20       & 2         & 2/20         \\
for    & 2        & 2/20       & 1         & 1/20           \\
I      & 2        & 2/20       & 3         & 3/20         \\
it     & 2        & 2/20       & 1         & 1/20           \\
money  & 1        & 1/20       & 2         & 2/20         \\
only   & 1        & 1/20       & 2         & 2/20         \\
prince & 1        & 1/20       & 2         & 2/20         \\
review & 2        & 2/20       & 1         & 1/20           \\
send   & 2        & 2/20       & 2         & 2/20         \\
the    & 2        & 2/20       & 1         & 1/20           \\
you    & 2        & 2/20       & 1         & 1/20           \\
will   & 2        & 2/20       & 2         & 2/20         \\
\bottomrule
\end{tabular}
\end{table}
\begin{align*}
p(email|ham) &= \left(1/10\right)^3 = 0.001\\
p(email|spam) &= 3/20 * 2/20 * 2/20 = 0.0015\\
p(ham|email) &\propto \frac{1}{3} * 0.001 = 0.000333...\\%\frac{0.00195}{0.00195+0.0039} = 0.111\\
p(spam|email) &\propto \frac{2}{3} * 0.0015 = 0.001\\%\frac{0.0039}{0.00195+0.0039} = 0.444
\end{align*}
Normalizing this yields 0.25 and 0.75 probabilities for the two classes respectively.
}
\else
\begin{table}[h!]
\centering
\begin{tabular}{@{}lllll@{}}
\toprule
       & freq ham & $p(w|ham)$ & freq spam & $p(w|spam)$ \\ \midrule
am     &         &           &          &          \\
for    &         &         &          &            \\
I      &         &         &          &          \\
it     &         &         &          &            \\
money  &         &           &          &          \\
only   &         &           &          &          \\
prince &         &           &          &          \\
review &         &         &          &            \\
send   &         &         &          &          \\
the    &         &         &          &            \\
you    &         &         &          &            \\
will   &         &         &          &          \\
\bottomrule
\end{tabular}
\end{table}
\fi

\item What are other possible features you can consider?\\
\ifsol
\textcolor{blue}{We can incorporate any features that are representative of spam emails in addition to word frequencies. Examples are
\begin{itemize}
\item ``Online Pharmacy''
\item Mentions large quantities of money
\item Sent by an unknown address with many numbers
\item Subject is all capitals
\item Email body has low ratio of text to images
\item ``One hundred percent guaranteed''
\item ``Prestigious Non-Accredited Universities''
\end{itemize}
Another approach is to also count occurrences of longer phrases (called n-grams).}
\else
\fi

\end{enumerate}

\end{enumerate}
\end{document}


\section{Aim and scope of the thesis}

The aim of this thesis is to develop a library for fast,
approximate \textit{maximum inner product search} (MIPS).

To this end we provide a set of tools that would help machine
learning developers and practitioners create faster, more scalable machine learning models.
Problems requiring MIPS naturally arise in many machine learning models, for example, in
classification problems with large output spaces, when linear models or deep neural networks are used.

Generally speaking, MIPS is the task involving searching through a massive collection of items for
an item most similar to a particular one, called query. The items may have many properties (features)
according to which items can be more related or differ from each other.
Every item is mathematically represented by a vector, with all vector's components being item's features.

Unfortunately, when one has to compare the query with hundreds of thousands of vectors,
a linear search time can become prohibitively expensive.
In other words comparing query and every database vector may not be feasible in practical applications.
To counter that, proposals were made in several articles to build an \textit{index} for speeding
up the searching time.
This index is essentially a specific data structure built by preprocessing database vectors (before
any of the queries are answered --- this is a one-time cost).
The most important feature of the index is short prediction (searching) time; training (index preparation)
can last much longer if needed.
It is not always necessary for the exact result to be returned --- it can often be approximate.
In this thesis we focus on such algorithms --- they trade some prediction precision for speed.

The library implemented in this thesis delivers three different MIPS algorithms,
recently introduced in the literature.
Two of these use $K$-means clusterization algorithm as part of their implementation: hierarchical $K$-means algorithm
selects candidate set by querying tree of $K$-means clusters, and quantization-based approach uses $K$-means as well,
but only after subdividing data vectors into smaller-dimensionality subvectors. The third algorithm we consider
is ALSH, or Asymmetric Locality Sensitive Hashing, which relies on specific hash functions preserving
vector locality.

Our library not only offers high performance, but is also easy-to-use.
It provides a set of wrappers around a range of machine learning libraries, that allow users to
speed up their models without ever touching any of the implemented MIPS algorithms. Consequently,
these wrappers can be used as drop-in replacements for a number of models in Python machine learning ecosystem.
We show some examples of how this library can be used in practice,
as well as detailed experiments measuring the speed and accuracy of our algorithms.

In particular we provide plugins for PyTorch (a deep learning library for Python), and scikit-learn
(a general-purpose machine learning library, also for Python), as well as a modified version of fastText
(a C++ library for efficient text classification).

All experimental results presented in this thesis were obtained using Amazon Elastic Compute Cloud (EC2).
EC2 is a part of Amazon.com's cloud-computing platform, Amazon Web Services (AWS). It allows users to rent virtual
computers on which they can run their own computer applications.

We have compared the implemented algorithms against the Faiss library \cite{faiss}, which is a library for efficient
similarity search and clustering of dense vectors, developed by Facebook AI Research. We refer the reader to
\cite{JDH17} for further details regarding this software.

The rest of this thesis is structured as follows.
The next chapter is dedicated to the theoretical description of the MIPS algorithms.
The third chapter describes implementation details of the library.
Chapter 4 presents experimental results.
The last chapter concludes the thesis.
In the appendix we describe how to run the library.

The list below shows in detail the contribution of each author of this thesis:
\begin{itemize}
    \item \textbf{Marcin Elantkowski} wrote wrappers exposing C++ indexes to Python, prepared datasets used in the
        experiments, adapted the code so it could be used from PyTorch and fastText,
            and ran final tests on Amazon Web Services
    \item \textbf{Adam Krasuski} refactored and optimized the C++ code of the core algorithms,
        implemented an interface to the Faiss library, analyzed and described the empirical results.
    \item \textbf{Agnieszka Lipska} implemented the first version of the quantization algorithm,
        implemented the Faiss interface, and adapted the code to use it from scikit-learn.
    \item \textbf{Franciszek Walkowiak} implemented the first versions of ALSH and
        hierarchical \mbox{$K$-means} and conducted preliminary tests on a personal computer.
\end{itemize}


\section{Full R code}

	This code will run all the indicated analysis and produce all plots.

	\lstinputlisting[style=Rsty]{../../code/hmc/d_sir_stan.r}

\section{Full Stan code}

	Stan model code to be used with the preceding R code.

	\lstinputlisting[style=Stansty]{../../code/hmc/d_sirode_euler.stan}


\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{fancyhdr, hyperref, amsmath, amsfonts, parskip, multicol, listings, xcolor}
\fancyhead[L]{\textbf{CMPUT 275 -- Tangible Computing} \hfill \textbf{Winter 2020}}

\hypersetup{
	colorlinks,
	citecolor=black,
	filecolor=black,
	linkcolor=blue,
	urlcolor=blue
}

% settings up listings environment
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}

\lstdefinestyle{mystyle}{
    backgroundcolor=\color{backcolour},
    commentstyle=\color{codegreen},
    keywordstyle=\color{magenta},
    numberstyle=\tiny\color{codegray},
    stringstyle=\color{codepurple},
    basicstyle=\ttfamily\footnotesize,
    breakatwhitespace=false,
    breaklines=true,
    captionpos=b,
    keepspaces=true,
    numbers=left,
    numbersep=5pt,
    showspaces=false,
    showstringspaces=false,
    showtabs=false,
    tabsize=2
}

\lstset{style=mystyle}

\title{\Large \textbf{Final Project: EEG Visualizer -- FFT Algorithms}}
\author{Eddie Guo}
\date{\today}

\begin{document}

\maketitle
\thispagestyle{fancy}

\section*{Discrete Fourier Transform}
The Fourier transform can be written in a few forms with forward and inverse transforms:

\begin{multicols}{2}
    \textbf{Hertz Frequency}
    \begin{align*}
        X(f) &= \int\limits_{-\infty}^{\infty} x(t) e^{-i 2\pi ft} dt\\
        x(t) &= \int\limits_{-\infty}^{\infty} X(f) e^{i 2\pi ft} df
    \end{align*}

    \textbf{Radian Frequency}
    \begin{align*}
        X(\omega) &= \frac{1}{\sqrt{2\pi}} \int\limits_{-\infty}^{\infty} x(t) e^{-i \omega t} dt\\
        x(t) &= \frac{1}{\sqrt{2\pi}} \int\limits_{-\infty}^{\infty} X(\omega) e^{i \omega t} d\omega
    \end{align*}
\end{multicols}

The discrete Fourier transform (DFT) is more useful to us as programmers. The na\"{i}ve method is an $O(n^2)$ computation which also has a forward and inverse form. A few notes on notation:

\begin{multicols}{2}
    \begin{itemize}
        \item $N$ -- the number of time samples.
        \item $n$ -- the current sample considered (0, 1, ..., $N$-1).
        \item $x_n$ -- the value of the signal at time $n$.
        \item $k$ -- the current frequency (0 Hz to N-1 Hz).
        \item $X_k$ -- the complex number representing amplitude and phase.
    \end{itemize}
\end{multicols}

\textbf{Inverse DFT} $$ x_n = \frac{1}{N} \sum_{n=0}^{N-1} X_k \cdot e^{i 2 \pi k n / N} $$
\textbf{Forward DFT} \vspace{-1em}

\begin{align*}
    X_k &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2 \pi k n / N}\\
    X_k &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2 \pi k n / N}\\
    \textbf{X} &= \textbf{x} \cdot M\\
    \intertext{Where the matrix $M$ is given by}
    M_{kn} &= \left[ e^{-i 2 \pi k n / N} \right]
\end{align*}

\newpage
\thispagestyle{fancy}
We can compute the DFT using matrix multiplication as shown below.
\begin{lstlisting}[language=Python, caption=Na\"{i}ve implementation of the DFT]
import numpy as np

def dft(x):
    """Computes the discrete Fourier transform of the 1-D array x"""
    x = np.asarray(x, dtype=float)
    n = np.arange(N)
    N = x.shape[0]
    k = n.reshape((N, 1))
    M = np.exp(-2j * np.pi * k * n / N)

    return np.dot(x, M)
\end{lstlisting}

\section*{Cooley-Tukey FFT Algorithm}
This algorithm exploits the symmetry in the DFT. Let us start by re-expressing the DFT in terms of $X_{N+k}$.\vspace{-1em}

\begin{align*}
    X_{N+k} &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2 \pi k n / N}\\
    &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2\pi n} \cdot e^{-i 2\pi k n / N}\\
    \intertext{We can now use the identity $e^{ 2\pi i} = 1 \Rightarrow e^{2\pi i n} = (e^{2\pi i})^n = 1^n = 1$.}
    &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2\pi k n / N}\\
    \intertext{However, this is just the original DFT. Thus, symmetry exists such that $X_{N+k} = X_k$. We can take this one step further $X_{k + iN} = X_k$, for any integer $i \in \mathbb{R}$.}
    X_k &= \sum_{n=0}^{N-1} x_n \cdot e^{-i 2\pi k n/N}\\
    &= \sum_{m=0}^{N/2 - 1} x_{2m} \cdot e^{-i2\pi k(2m)/(N/2)} + \sum_{m=0}^{N/2 - 1} x_{2m + 1} \cdot e^{-i2\pi k(2m + 1)/ (N/2)}\\
    &= \sum_{m=0}^{N/2 - 1} x_{2m} \cdot e^{-i2\pi k(2m)/(N/2)} + e^{-i 2\pi k/N} \sum_{m=0}^{N/2 - 1} x_{2m + 1} \cdot e^{-i2\pi k  m/ (N/2)}\\
\end{align*}

Here, the DFT is split into two terms: one for even-numbered values and one for odd-numbered values. This still gives $(N/2) * N$ computations giving the same time complexity of $O(n^2)$. However, we notice that $0 \leq k < N$ and $0 \leq n < M \equiv N/2$. From the symmetric properties shown, there is only need to perform half the computations for each partition of the original $N$-dimensional vector. Thus, the algorithm is converted from $O(n^2)$ to $O(M^2)$ where $M = N/2$.

It is clear here that we are cooking up a divide-and-conquer approach: partition the vector until the partitioning no longer provides any reasonable time benefits, say N $\leq 32$. We can recursively apply all computations such that $O(N^2)$ becomes $O(\frac{N}{2} log_2 N) = O(N log N)$.\footnote{Note: this algorithm works $\iff$ the input vector's size is a power of 2.}

\end{document}


\section{The \popheap algorithm}
\Label{sec:popheap}

The algorithm \popheap moves the first element of the heap, which holds
the heap's largest value, and places it at the the end of the underlying sequence.
Whereas in the \cxx Standard Library \cite[\S 28.7.7.2]{cxx-17-draft}
\popheap works on a range of random access iterators,
our version operates on an array of \valuetype.
We therefore use the following signature for \popheap

\begin{lstlisting}[style = acsl-block]

    void pop_heap(value_type* a, size_type n);
\end{lstlisting}

The \popheap algorithm expects that \inl{n} is greater or equal than~1
and that the array \inl{a[0..n-1]} forms a heap.
The algorithms then \emph{rearranges} the array \inl{a[0..n-1]} such that the
resulting array satisfies the following properties.

\begin{itemize}
\item \inl{a[n-1] = \\old(a[0])}, that is, the largest element
of the original heap is transferred to the end of the array.

\item the subarray \inl{a[0..n-2]} is a heap
\end{itemize}

In this sense the algorithm \emph{pops} the largest element from a heap.

\subsection{Formal specification of \popheap}

Based on the above semi-formal description we propose the
following function contract for \specref{popheap}.

\input{Listings/pop_heap.h.tex}

\subsection{Implementation of \popheap}
\Label{sec:pop-heap:impl}

In an abstract sense \popheap is quite similar to \pushheap.
In  \pushheap we started at the last array element and
climbed from there up the tree until we would find a node where to
insert the new value into the heap.
Every time we had reached the next parent node we
moved its value down to where we had just come from.

With \popheap its the other way round.
We start at the root of the tree and descend from there
by selecting an appropriate child.
Every time we lift the value of the selected child to the node where
just are.
We repeat this process until we find a node where we can insert
the last array element into the heap.
Once this is done, we can safely place the maximum element (that is the
the original root node) at the last element of the array.

\clearpage

The following two figures illustrate how \popheap affects an array,
which is shown again as a tree with blue and grey nodes, representing
heap and non-heap nodes, respectively.
Figure~\ref{fig:popheap-pre} is in fact the same figure as
Figure~\ref{fig:heap-tree}.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.70\linewidth]{Figures/pop_heap_pre.pdf}
\caption{\Label{fig:popheap-pre}Heap before the call of \popheap}
\end{figure}

\FloatBarrier

Figure~\ref{fig:popheap-post}, on the other hand, shows the heap after the call of \popheap
together with arrows that indicate how our implementation moves around elements
in the underlying array.
We can see that the first element of the original array,
where the maximum of the heap resides, is now the last element of the array.
Furthermore, the last array element is not part of the heap anymore.
The dashed nodes highlight which heap nodes have changed during the call to \popheap.
The arrows indicate the \emph{cyclic reordering} of array elements to achieve the
desired result.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.70\linewidth]{Figures/pop_heap_post.pdf}
\caption{\Label{fig:popheap-post}Heap after the call of \popheap}
\end{figure}

\FloatBarrier

As in the case of \implref{pushheap} we will subdivide the discussion of the
implementation of \implref{popheap} into a prologue, main act, and epilogue.

\input{Listings/pop_heap.c.tex}

\subsection{Prologue}

In the prologue we check whether the initial heap contains at least two elements,
initialize some variables, and also check whether
the last array element is by chance equal to the maximum element of the heap,
which resides at the index \inl{p == 0} of the array.
If this is not the case, then we set aside for future
reference the last array element in the variable \inl{v}.
Finally we copy the value \inl{a[p]} to its final destination at the end
of the array.
Note that this assignment only occurs if the respective values differ.
This allows us, as in the case of \implref{pushheap}, to formally describe the
effect of the assignment using the predicate \logicref{ArrayUpdate}.

Figure~\ref{fig:popheap-prologue} highlights the main effects of the prologue
at the hand of our exemplary heap.
Note that we have highlighted the root of the heap as the currently active node.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.65\linewidth]{Figures/pop_heap_prologue.pdf}
\caption{\Label{fig:popheap-prologue}Heap after the prologue of \popheap}
\end{figure}

\FloatBarrier

\subsection{Main act}

In the main act, we start at a child node \inl{c} of the prologue's index \inl{p}.
This means that compared to the pre-state of \popheap
at the beginning of the main act the array \inl{a[0..n-1]}
\begin{itemize}
\item contains the value \inl{v} one time less
\item contains the value \inl{a[p]} one time more
\item whereas all other values have not change their number of occurences.
\end{itemize}

Moreover, the maximum element of the original heap is now at the end of the array
and we can only guarantee that the first $n-1$ array elements got a heap.
These observations are necessary reason for our loop invariants.

To be more precise, when we talk in the context of \popheap
of a \emph{child node} we usually mean one of the possibly two children
where the maximum of the values resides.
We do this because copying that larger value to its parent node guarantees
that the resulting tree is still a heap.
We compute the maximum child of a node using the function \specref{heapchild}.

Now, as long as the index~\inl{c} is not yet the index of the last array element
of the heap and its value \inl{a[c]} is less than \inl{v},
we haven't found yet an index where we could insert \inl{v} without
violating the heap property.

\clearpage

In the loop body we proceed as follows.

\begin{itemize}
\item
If \inl{a[c]} is less than \inl{a[p]} we copy the former value on
the latter.
As mentioned above, using the index~\inl{c} of the maximum child
maintains  heap property of the array.
We use here the predicate \logicref{HeapCompatible} to express
that the insertion of the new value \inl{a[p]} maintains the heap
property of the array.

The value \inl{a[c]} now occurs one time more than in the pre-state whereas
the now overwritten value \inl{a[p]} occurs as often as in the pre-state of \popheap.
The value \inl{v} continues to occur one time less than in the pre-state.
%
We then proceed to the next iteration by setting \inl{p} to \inl{c}
and computing the next maximum child node.

As in the case of \implref{pushheap} the verification of
the correct number of occurences of the involved values
relies on the predicates \logicref{ArrayUpdate} and \logicref{MultisetUpdate}
and on lemma \logicref{MultisetParityCombined}.

\item
Otherwise, the array being a heap, we can conclude that
\inl{a[c]} equals \inl{a[p]} and we continue with the next iteration
after setting \inl{p} to \inl{c} and computing the corresponding
new maximum child node.
\end{itemize}

The following three figures depict how the main act of \popheap
modifies step by step our example heap.
In each step we highlight the currently active node \inl{c}.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.65\linewidth]{Figures/pop_heap_main_act1.pdf}
\caption{\Label{fig:popheap-main_act1}Heap after the first iteration of of \popheap}
\end{figure}

\FloatBarrier

\begin{figure}[hbt]
\centering
\includegraphics[width=0.65\linewidth]{Figures/pop_heap_main_act2.pdf}
\caption{\Label{fig:popheap-main_act2}Heap after the second iteration of \popheap}
\end{figure}

\FloatBarrier
\clearpage

Note that in the final step no value is actually copied as the involved nodes
hold the same value.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.65\linewidth]{Figures/pop_heap_main_act3.pdf}
\caption{\Label{fig:popheap-main_act3}Heap after the third iteration of \popheap}
\end{figure}

\FloatBarrier

We finally remark that in the main act the the last array element is never modified.
Thus, the root element of the original element is still safely stored there.

\subsection{Epilogue}

After leaving the loop, we know that value \inl{v} can be the inserted
in the array at the index \inl{p} without violating the heap property of
the first $n-1$ elements.
%
Moreover, compared to the pre-state of \popheap the array \inl{a[0..n-1]} still
\begin{itemize}
\item contains the value \inl{v} one time less
\item contains the value \inl{a[p]} one time more
\item whereas all other values have not change their number of occurences.
\end{itemize}

In other words, assigning the value \inl{v} to \inl{a[p]}
cancels this imbalance and establishes that \popheap
only reorders the array elements.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.65\linewidth]{Figures/pop_heap_epilogue.pdf}
\caption{\Label{fig:popheap-epilogue}Heap after the epilogue of \popheap}
\end{figure}

\FloatBarrier

In Figure~\ref{fig:popheap-epilogue} we have marked the value \inl{v}
as the currently active node despite not being an array element.

\clearpage


\chapter{Statistical method}
\label{ch:methods}
%%
In this chapter we will discuss the statistical method that we developed forbinary Population synthesis. Focusing on the difference between our method and traditional sampling methods in BS.

\section{Simulating populations of binaries}
In binary population synthesis, one can simulate the evolution of a synthetic population of binary systems.
For such simulations, bianries are randomly drawn from the distributions of initial binary parameters and evaluating it through binary evolution prescriptions. By doing so, they present a rapid
code that can compute the evolution of many stars within
a simulation.
Since it is now known a priori which initial conditions will produce an event of interest, the full initial parameter space needs to be explored. Traditional methods in BPS tackle this problem by randomly drawing binaries $\mathbf{x_i} \sim P(\mathbf{x_i})$ from their prior distributions which are based on observations, and evaluating them with the BPS model, $u$,  into their final state $\mathbf{x_f}$,

\begin{equation}
	\mathbf{x_f} = u(\mathbf{x_i}).
\end{equation}

Many parameters are used in BPS but most binaries can be described uniquely by a few important initial vari-
ables: the initial mass of the primary star (i.e., the most massive star) $M_{1,i}$, the mass ratio, $q_i = M_{1,i} / M_{2,i}$, between
the two stars and the initial separation, $a_i$, eccentricity $e_i$. Depending on the binary evolution we can also add the kick velocity received when the primary or secondary collapses to a compact object $\mathbf{v}_{\text{kick, SN}}$. Each initial binary sample can thus be represented by

\begin{equation}
	\mathbf{x_i} = (M_{1,i}, a_i, q_i, e_i, \mathbf{v}_{\text{kick, SN1}}, \mathbf{v}_{\text{kick, SN2}} )
\end{equation}

Often, BPS is used to study binaries that evolve to a certain subtype $\mathbf{X_t}$,

So in such a study the goal is to model the distribution:

\begin{align}
	\psi(\mathbf{x_f}  ) = \begin{cases} 1 \hspace{0.1cm} &\text{if } \mathbf{x_f} \in \mathbf{X_t} \\
	0  & \text{else}
	\end{cases}
\end{align}
that equals unity if $\mathbf{x_i}$ evaluated to the target binary system
$\mathbf{X_t}$ and zero if not. We will use this function throughout the
paper as it is the main objective to perform the inference
on.
Say something about this function.

However, in cases when the target population is a rare event in the simulation, e.g. when simulating Compact object binaries since most systems will disrupt during the supernova kick, and thus  $\psi(\mathbf{x_f}  )= 0 $ for most systems. Therefore simulating populations of such rare events becomes extremely computational expensive.

Therefore, instead, present the variance reduction method , adaptive importance sampling [1], that generates samples
from a distribution function which is automatically adapted to the scientific target by focusing on areas
of the parameter space found to produce events of interest.
 Instead of drawing random binaries from the prior distribution $P$ we draw them from a distribution that focuses around the target distribution of interest, thereby minimizing the computational costs spend on areas that don't produce events of interest, whilst maximizing the computational costs spend on binaries that become the rare event. \\

The method consists of three main steps that are also shown in Figure

\begin{enumerate}
	\item  The parameter space is explored to find a small population of events of interest.
	\item  The set of known events is used to build an instrumental distribution, to adaptively
guide future sampling of the parameter space.
	\item  Later simulations of population members are then drawn from this instrumental
distribution. The information from these simulations can in turn be used to
further improve the instrumental distribution.

\end{enumerate}
By doing so, the method minimizes the computational time spend on binary systems that don`t evolve to the event of interest, whilst maximizing the computational time spend on binaries of the target distribution.

\section{Initial binary parameters}

For the exploratory phase and comparison runs we choose $M_1, q, a, e, v_k $ similar to the distributions used in common binary population synthesis models (e.g. Belczynski, Pols)

\subsection{Initial mass}
The distribution of the initial primary mass M1 follows a
power law distribution also known as the initial mass func-
tion (IMF) (Kroupa 2001):

%
\begin{equation}
	p(M_{1,i}) = C_M M_{1,i}^{-\alpha} \vspace{0.2cm} \text{for} M_{1,i} \in [M_{1,i,\text{min}}, M_{1,i,\text{max}}]
\label{eq:priorIMF}
\end{equation}
%
where $C_M$ is the normalization constant given by
%
\begin{equation}
	C_M = \frac{\alpha + 1}{M_{1,i,\text{max}^{\alpha +1}} - M_{1,i,\text{min}}^{\alpha+1}}.
\end{equation}
We choose $\alpha = 2.35$ in agreement with (Salpeter or Kroupa), and $M_1 \in [5, 100] M_{\odot}$ to align with Vigna-Gomez (2017).

\subsection{initial mass ratio}
The mass ratio $q_i$ is suggested from observations to have
a flat distribution (Mazeh et al., 1992; Goldberg $\&$ Mazeh,
1994; Tout, 1991), given by
%
\begin{equation}
	p(q) = \frac{1}{q_{\text{max}} - q_{\text{min}}}, \vspace{2cm} \text{for } q \in [q_{\text{min}}, q_{\text{max}}],
\end{equation}
where $[q_{\text{min}}, q_{\text{max}}] = (0, 1]$ by definition of q. Nevertheless, it
is also suggested that there is some dependency of the mass
ratio on the period of the system (e.g. Moe $\&$ Di Stefano 2016). But this is beyond the scope of this thesis.

\subsection{initial separation}
%The separation $a_i$ is found to be uniform in log, also known
%as Opik law (Opik ̈1924) and consistent with findings by
%(Kobulnicky et al. 2014) and (Moe $\&$ Di Stefano 2015). The
%distribution function is given by
%
\begin{equation}
	p(a_i) = C_a \frac{1}{a} \vspace{2cm} \text{for } a \in [a_{\text{min}}, a_{\text{max}}]
\end{equation}
%
where $C_a$ is the normalization constant
%
\begin{equation}
C_a = \frac{1}{\log a_{\text{max}} - \log a_{\text{min}}}.
\end{equation}
%
We choose $[a_{\text{min}}, a_{\text{max}}]  = [0.1, 1000]$ AU consistent with (Vigna-Gomez 2018).

\subsection{eccentricity}
We assume that all of our binaries initially are orbiting in a circular orbit, $e = 0$ consistent with (Alejandro et al)

\subsection{Supernova}
We differentiate between three supernova scenarios: core col- lapse supernovae (CCSN), ultra-stripped supernova (USSN) and electron-capture supernova (ECSN). For the CCSN treatment, we use the rapid explosion
scenario, as presented in Fryer et al. (2012), to determine the compact object remnant mass according to the total and carbon-oxygen (CO) core mass of the progenitor, with
a maximum allowed NS mass of mNS,max $= 2.0$ M?. In this scenario, the collapse does not allow for accretion onto the proto-NS, and is able to reproduce the proposed mass gap between neutron stars and black holes (Ozel et al. 2010;
Farr et al. 2011). There is no consensus yet whether the mass gap is due to observational selection effects or if it is intrinsic to the explosion mechanism (Kreidberg et al. 2012; Wyrzykowski et al. 2016). Another explosion scenario comes from USSN (Tauris
et al. 2013, 2015). A star becomes stripped when it loses its hydrogen envelope during its evolution; if, during later stages, it manages to lose its helium envelope, it becomes ultra-stripped. In COMPAS, any star which engages in a stable case BB mass transfer episode with a NS as an accre- tor, is considered to be ultra-stripped.We define case BB as a mass transfer episode which involves a Helium donor star which has stopped burning helium in the core (naked helium star Hertzprung Gap, HeHG). Ultra-stripped stars are left with an ONeMg core with a thin carbon and helium layer (Tauris et al. 2013). The compact object remnant mass of an USSN is determined in the same way as for CCSN. A single star with 8  mZAMS $/$ M?  10 (binary stars spread the initial mass range) may collapse in an ECSN (Nomoto 1984). We assume the baryonic mass of the de- generate ONeMg core leading to an ECSN is 1.3

\section{Adaptive Importance Sampling algorithm}

\begin{table}[]
\centering
\caption{Table of variables used }
\label{my-label}
\begin{tabular}{l|l}
\hline
\textbf{variable}               & \textbf{description}           \\ \hline
\multicolumn{1}{|l|}{$u$} & \multicolumn{1}{l|}{BPS model} \\ \hline
\multicolumn{1}{|l|}{$\mathbf{x_i}$} & \multicolumn{1}{l|}{initial state of a binary system} \\ \hline
\multicolumn{1}{|l|}{$\mathbf{x_f}$} & \multicolumn{1}{l|}{final state of a binary system} \\ \hline
\multicolumn{1}{|l|}{$\mathbf{X_t}$} & \multicolumn{1}{l|}{target subpopulation of binaries of interest} \\ \hline
\multicolumn{1}{|l|}{$M_1$}  & \multicolumn{1}{l|}{mass of the primary star} \\ \hline
\multicolumn{1}{|l|}{$a$} & \multicolumn{1}{l|}{initial separation of the binary } \\ \hline
\multicolumn{1}{|l|}{$q$} & \multicolumn{1}{l|}{initial mass ratio $q = M_2 / M_1 $ of the binary} \\ \hline
\multicolumn{1}{|l|}{$e$} & \multicolumn{1}{l|}{initial eccentricity} \\ \hline
\multicolumn{1}{|l|}{$v$} & \multicolumn{1}{l|}{} \\ \hline
\multicolumn{1}{|l|}{$v_k$} & \multicolumn{1}{l|}{kick velocity amplitude} \\ \hline
\multicolumn{1}{|l|}{$\theta_k$} & \multicolumn{1}{l|}{} \\ \hline
\multicolumn{1}{|l|}{$\phi_k$} & \multicolumn{1}{l|}{} \\ \hline
\end{tabular}
\end{table}

\section{Adptive Importance Sampling}
\section{Fiducial method}
\subsection{Exploratory phase}
\subsection{Improve distribution}
\subsection{Run simulation (step 3) }

\section{Advanced Adaptive Importance Sampling}

\section{Other implementations}

\section{Population Synthesis}
For the population synthesis model we use COMPAS (Alejandro et al in prep and Stevenson et al. 2017)

based on (...)

\section{Fiducial Model}

Our fiducial model

\subsection{SN}
\subsection{mass transfer and stabillity}
\subsection{CE treatment}
\subsection{fall back}
\subsection{kicks}

%\section{\NoCaseChange{\acl{RXTE}}}
%While a number of X-ray missions have been conducted over the years, \ac{RXTE} remains a unique mission due to its extraordinary timing resolution \citep{bradt1993x}. Operating for more than 16 years from 1995 to 2012, \ac{RXTE} built up a significant archive of transient X-ray sources, providing a large repository of \ac{LMXB} observations \citep{heasarc}. Using the \ac{HEASARC} online services, provided by the NASA/Goddard Space Flight Center, data can freely be downloaded for the full length of the mission. With a wide range of data types available, the variety in data products originates in the different instruments carried on board \ac{RXTE}.\\
%
%\ac{RXTE} carried out observations with three observational instruments -- the \ac{ASM}, the \ac{HEXTE} and the \ac{PCA} \citep{bradt1993x}. The \ac{ASM} was designed to survey a large fraction of the sky every 1.5 hours, allowing the intensity and spectrum of more than 75 objects to be monitored. Upon rapid changes in either property, the \ac{HEXTE} and the \ac{PCA} could be pointed towards the target typically within a few hours. The energy ranges of the \ac{HEXTE} and the \ac{PCA} were designed to be complimentary, with the \ac{HEXTE} observing from 15-200 keV, and the \ac{PCA} from 2-60 keV \citep{jahoda1996orbit}. Both instruments had a $1^\circ$ field of view, and a minimum timing resolution of $8\mu$s for the \ac{HEXTE} and $1\mu$s for the \ac{PCA} \citep{rothschild1998flight, zhang1993laboratory}. \\
%
%Only observations conducted with the \ac{PCA} are used throughout this project, providing an extensive source of data. Comprised of five \acp{PCU}, the \ac{PCA} allowed energies to be determined to an energy resolution of less than 18\% at 6~keV \citep{pcainfo}. Each \ac{PCU} was filled with a mixture of Xenon and Methane gas, allowing charged particles to ionise the gas, resulting in an electrical pulse proportional to the energy carried by the incident particle \citep{zhang1993laboratory}. An additional layer on top of each \ac{PCU} contained propane gas in order to reduce the effect of background events. A gradual loss of propane occurred at the start of the mission, leaking through to the Xenon layers \citep{jahoda1996orbit}. Combined with other effects, this resulted a gradual change in gain for which gain epochs have to be defined to correct for these changes \citep{rxteenergychannel}. The loss of the PCU$0$ propane layer in 2000 required the adoption of an additional calibration epoch. An internal radioactive source allowed for continuous energy calibration \citep{zhang1993laboratory}.\\
%
%With each active \ac{PCU} providing data, information was sent to six \acp{EA} incorporated in the \ac{EDS} \citep{jahoda1996orbit}. This phase allows for data processing and compression before telemetry. Two of the \acp{EA} were programmed to run in standard configurations, with the others able to run in one of the seven different modes \citep{rxtepcaissues}. The sheer number of options, and parameters, needed to extract this data from the different modes requires a large set of extraction tools. With the previous paragraphs giving a brief overview of the hardware behind the observations, additional technical details on the \ac{PCA} can be found in \citet{zhang1993laboratory}. Shifting to the data extraction and analysis side of data reduction requires specialised software, and is explained in the following section.\\
%
%\section{Data Reduction}
%\label{sec:data_reduction}
%In order to conduct a systematic population study, a robust pipeline is needed to run through many scores of observations. While a number of tools are available to help in extracting data, these tools do not lend well to upscaling, often requiring a large number of input parameters for every data mode. To this end, the \chromos software pipeline was developed. The technical side of \chromos is described in appendix~\ref{ch:chromos}, in the form of a manual, with information on the underlying methods presented in this section. \ \\
%
%The \ac{HEASARC} online services provides several interfaces for searching and retrieving \ac{RXTE} data \citep{rxtearchive}. Using the web-based graphical user interface, a list of ObsIDs can be obtained for each target. ObsIds are used to classify observations, and are an identification code which changes when a new target is acquired, or when the \ac{EA}-modes change. ObsIDs follow the format of 'NNNNN-TT-VV-SSX', with NNNNN a five-digit proposal number, TT a two-digit target number, VV a two-digit viewing number, SS a two-digit number to identify different pointings and X a character to denote slewing, configuration changes etc \citep{rxtearchive}. An \sw{ftp}-server allows for downloads to be conducted via the command line. Extracting this data requires some knowledge of the available data modes per observation. Two \acp{EA} modes should be permanently available, \sw{standard1} with no energy information but a 0.125s timing resolution and \sw{standard2} data with 129 spectral channels and a 16s time resolution \citep{rxtestdproducts}. A whole range of other data modes can also be present depending on the observation. These modes fall into two categories - science array (also known as binned-mode data) and science event format. Both formats require different tools and input to extract the data. Science array format includes data modes such as \sw{binned} and \sw{standard2}, and science event format, data modes such as \sw{good\-xenon} and \sw{event} mode. Each data mode can provide diverging timing resolutions depending on the binning method, but files must have a timing resolution higher than $\sfrac{1}{128}\ $s for our subsequent analysis, with the exception of files for spectral analysis and background creation. \\
%
%To ensure data remains a reliable reflection of the actual target emission, it must be filtered using \acp{GTI} \citep{rxtecookbookevent}. Following the \ac{RXTE} cookbooks for science array and science event mode data, the ftool \sw{maketime} is used to set filter criteria \citep{maketime,rxtecookbookbinned,rxtecookbookevent}. To ensure the earth brightness does not contaminate observations, the pointing elevation is set to be above $10^\circ$. Additionally, the pointing offset is set to be less than $0.02^\circ$ and the number of active \acp{PCU} set to be greater than one. For all objects save for black holes and Sco~X-1, up to 10min since the \ac{SAA} is removed \marginpar{The \ac{SAA} refers to an area in which the Earth's magnetic field is reduced in strength, causing an increase in high energy particle count rates \ \ \ \ \citep{saa}.} and an electron count larger than 0.1 is removed to prevent electron contamination. The former sources show sufficiently high count rates to neglect this last criterion, and with current backgrounds able to account for the \ac{SAA} passage, the filtering on time since \ac{SAA} is not required. Using information from standard filter files, the times at which a change in number of \acp{PCU} occurs are noted, allowing for 32s around these transitions to be filtered during extraction. This prevents any surge, or change in electrical current, from contaminating the count rate. \\
%
%Background files are created from \sw{standard2} files together with standard filter files using the ftool \sw{pcabackest} \citep{pcabackest}. This estimated background spectrum also requires a provided model file. With sources showing a net count rate larger than 40~ct/s/PCU, the 'bright' background model can be used for all sources \citep{pcadigest}. Having created background files, the sole step left before extracting the main data files, is determining the correct energy channel ranges. This final part is potentially the most complicated part of \chromos, requiring a number of steps. On the basis of the observation date, an initial channel range can be selected using the energy-channel conversion table \citep{rxteenergychannel}. If files are in \sw{event} or \sw{binned} mode, then the header of these files will show the channel binning, which can vary from observation to observation. The final channels can be selected using this information, in which channel bins closest to the energy range 2-13~keV are selected while ensuring the lowest energy channels are omitted \citep[see][]{gleissner2004long}.\\
%
%Extracting data requires two ftools: \sw{saextrct} for \sw{event} and \sw{goodxenon} files and \sw{seextrct} for \sw{standard2} and \sw{binned} files \citep{saextrct,seextrct}. The input parameters can be determined using the files and information generated in the previous steps, with exception of the timing resolution. This is set to $\sfrac{1}{128}\ $s for all files save for \sw{standard2} files, which are extracted with their intrinsic resolution of 16s. A background file is extracted for each subsequent data file to ensure the same filter criteria are applied. \sw{standard2} files are extracted for PCU2 only, the sole PCU with an intact propane layer of the course of the mission, and the most stable one. While the low timing resolution of \sw{standard2} files prevents any high precision timing analysis from taking place, this data is well-suited to spectral analysis. \sw{standard2} files are therefore extracted as both spectra and light curves, rather than just as light curves as all other data modes are.\\
%
%\section{Timing Analysis}
%\label{sec:timing_analysis}
%Light curves must be background corrected, which necessitates the rebinning of background light curves to match the resolution of the original light curve files. This is done on basis of interpolation between each consecutive background data point. Subtracting these values from the light curve count rates produces a light curve suitable for further analysis. To prevent the flares from affecting the general variability trend throughout a single observation, X-ray bursts are identified and removed for all neutron star systems. Tests showed that two consecutive count rates above a \acf{RMS} per observation of 7$\sigma$ provided a strong indication for an X-ray burst. Numerous methods for automatically identifying the start and end of the bursts were tested, but in the end a choice was made to cut fixed time bins on other side. Approximately 3s was cut before each detection, and 625s afterwards. Further discussion on this method can be found in section~\ref{sec:dis_bursts}.\\
%
%In order to conduct timing analysis, a switch must be made to the frequency domain. Unless specified elsewhere, all following information is based on work presented in \citet{uttley2014x}. While a variety of techniques are available to conduct time-series analysis \citep[e.g.][]{maccarone2000time,legg2012direct}, discrete Fourier transforms are the most prevalent. These transforms can be calculated with
%\begin{align}
%X_n = \sum^{N-1}_{k=0}x_k\ e^{\sfrac{2\pi ink}{N}}
%\end{align}
%with $X_n$ the discrete Fourier transform, $N$ the number of time bins and $x$ the $k^\textrm{th}$ value of the light curve. $X_n$ is calculated in steps of $f_n=\sfrac{n}{N\Delta t}$ with $\Delta t$ the width of a time bin and with $n=0,1,2\ldots \sfrac{N}{2}$. The maximum frequency is thus 64~Hz due to the $\sfrac{1}{128}\ $s time resolution of the extracted light curves. In order to obtain a power spectrum, the discrete Fourier transform is multiplied with its complex conjugate
%\begin{align}
%|X_n|^2 = X_n X_n^*
%\end{align}
%The resulting power spectrum is subsequently normalised with
%\begin{align}
%P_n = \frac{2\Delta t}{\langle x \rangle ^2 N}|X_n|^2
%\end{align}
%in which $\langle x \rangle$ is the mean flux of the light curve. This leads to units in terms of fractional variance per Hz \citep{belloni1990variability}. With the exception of perhaps an \ac{AMSP}, few sources show fully periodic signals over all observations, but will usually show shifting power spectral components. A decision must therefore be made on the total time over which to Fourier transform. With power spectra errors dependent on the number of power spectra over which is averaged, a trade-off is established. Reducing the noise on the final power spectrum requires averaging over many segments, but in the process reduces the sensitivity to varying power spectral components. Following the procedure given in \citet{heil2015power}, a choice is made to take discrete Fourier transforms of each $m^{th}$ segment of 512s in an observation, and average these power spectra:
%\begin{align}
%\overline{P}_n = \frac{1}{M} \sum_{m=1}^M P_{n,m}
%\end{align}
%in which $M$ is the total number of segments in an observation. This number is limited not just by the total length of the light curve, but also by gaps due to PCU changes, X-ray bursts etc. The decision to bin across each observation is expanded on in section~\ref{sec:binning}, where alternative binning methods are also discussed. \\
%\clearpage
%With power spectra flattening towards higher frequencies, a correction can be applied by calculating the white noise present in the power spectrum with
%\begin{align}
%P_\textrm{noise} = 2\frac{\langle x \rangle + \langle b \rangle}{\langle x \rangle ^2} \label{eq:noise}
%\end{align}
%in which $\langle x \rangle$ is the mean count rate and $\langle b \rangle$ is the mean background count rate in $\overline{P}_n$. The resulting noise value can be subtracted from the power spectrum. The associated errors on the noise corrected power spectrum are then calculated by dividing each power by $\sqrt{M}\hspace{2pt}$. As the powers are $\chi^2_2$-distributed, errors on the power spectrum can be approximated as Gaussian provided a large number of samples are binned.\\
%
%Power spectral evolution can be parametrised using power colours, as described in section~\ref{sec:commontechniques}. These can be calculated using the power spectra obtained for each observation. First, the integrated power, or variance $V$, can be calculated using
%\begin{align}
%V = \Delta \nu \sum_{i=\nu_\textrm{low}}^{\nu_\textrm{high}} \overline{P}_i
%\end{align}
%with $\Delta \nu$ the Nyquist frequency and $\overline{P}_i$ the power in frequency $i$, running between the four equally spaced frequency bands of 0.0039-0.031~Hz, 0.031-0.25~Hz, 0.25-2.0~Hz and 2.0-16.0~Hz \citep{heil2015power}. This results in a variance value $V$ for each consecutive frequency band, respectively referred to as $V_A$, $V_B$, $V_C$ and $V_D$. Defining two power colours as
%\begin{align}
%	PC1 = \frac{V_C}{V_A} \hspace{20pt} \textrm{and} \hspace{20pt} PC2 = \frac{V_B}{V_D}
%\end{align}
%allows an observation to be placed in a \ac{PCC}~diagram, with $PC1$ on the horizontal axis and $PC2$ on the vertical axis. The associated error values can be calculated from the errors on the variance, determined with
%\begin{align}
%Err^2(V) = \frac{(\Delta \nu)^2}{M} \sum_{i=\nu_\textrm{low}}^{\nu_\textrm{high}} \overline{ P^2 }_i
%\end{align}
%as given by \citet{heil2012ubiquity}. Note that the summation is over the average squared power, rather than over the squared average power. Power colour errors can subsequently be calculated with simple error propagation
%\begin{align}
%Err(PC1) = PC1 \sqrt{\left(\frac{Err(V_C)}{V_C}\right) ^2 + \left(\frac{Err(V_A)}{V_A}\right) ^2}
%\end{align}
%and $Err(PC2)$ in analogous fashion.
%
%\section{Spectral Analysis}
%Spectral analysis is conducted with \sw{standard2} files, due to both their energy resolution, and their relative ease of use. The ftool \sw{pcarsp} is run for all files prior to any analysis, allowing response files to be created from energy spectra and filter files \citep{pcarsp}. These response files can be used by the X-ray spectral-fitting program \sw{xspec} \citep{arnaud1996astronomical} to ensure an accurate representation of count rates in each energy bin after subtracting background values. Energies outside the range 2-13~keV are ignored, before unfolding the energy spectrum around a flat powerlaw. In doing so, the energy spectrum is effectively being divided by the effective area of the instrument response, providing energy spectra independent of long term changes in instrument response \citep{heil2015inclination}. The energy spectral hardness is calculated for a variety of energy bands, by integrating the count rate over energy and dividing the resulting flux of the higher energy band by that of the lower energy band. These fluxes are additionally summed, to obtain the relative total intensity.\\
%
%%TODO Error calculation for integrating over energies? Does this paragraph need any formula's?
%
%\section{Targets}
%In order to compare the variability properties of both black hole and neutron star \acp{LMXB}, a selection of objects is required. An overview of the chosen objects can be found in Tab.~\ref{tab:objects} and includes information on various system parameters. The choice of objects was based primarily on the number of \ac{RXTE} observations available, but also on the system type. These were chosen to ensure a good spread across both atoll and Z sources, as well as various inclinations, accretion states etc. Sources have been divided by compact object, with neutron stars above the centre line, and black holes below. Most columns should be self-explanatory, however \spacedlowsmallcaps{\#Good} may require additional explanation. This refers to the number of ObsIDs which have power colour ratios with a fractional variance constrained at a 3$\sigma$-level in each frequency band. Further discussion on these values can be found in section~\ref{sec:selection_criteria}, in which various selection effects are examined. The object names used in this table and throughout this thesis were selected to reflect the most common name in use. Alternate source names can however be found in Tab.~\ref{tab:aka}, presenting source designations from the \sw{4U}, \sw{GX}, \sw{IGR}, \sw{INTEGRAL1}, \sw{SWIFT}, \sw{X} and \sw{XTE} catalogues \citep{SIMBAD}. The choice of black holes systems was made on basis on of \ac{PCC}~diagram coverage as presented in \citet{heil2015power}, providing a representative sample of possible black hole power colour tracks. \\
%
%
%\section{Robustness of \chromos}
%Initial tests of the pipeline were conducted by comparing power colours of Aql~X-1 with prior results presented in \citet{heil2015power}. Both the results can be seen in Fig.~\ref{fig:aqlx1}, with left showing power colours from \citet{heil2015power} and right showing results obtained with \chromos. Similar tracks are found for Aql~X-1 with both methods, save for the top left corner of the \ac{PCC}~diagram where observations are primarily in the soft state. A close inspection revealed differences in noise correction, where \citep{heil2015power} used spectral fitting to obtain the white noise, in contrast to the approach taken in this work. Additional binning effects also played a role, and are discussed further in section~\ref{sec:binning}. The power colours track of black holes obtained with \chromos showed only minor differences in comparison to those obtained in \citet{heil2015power}. Further tests were also conducted on parameters related to energy spectra, revealing similar hardness and intensity values in past spectral studies. Suggestions on scientific improvements to \chromos are given in the discussion, section~\ref{ch:discussion}, with coding recommendations in appendix~\ref{ch:chromos}.
%
%\begin{figure}%
%\myfloatalign%
%\makebox[\textwidth][r]{%
%\subfloat{\includegraphics[width=.42\largefigure,valign=t]{pc/aquila_lucy.png}}%
%\quad%
%\subfloat{\includegraphics[width=.45\largefigure,valign=t]{pc/individual/aquila_X1}}%
%}%
%\caption[Comparison of Aql~X-1 \acs{PCC}~diagrams]{\spacedlowsmallcaps{left} Power colour ratios obtained for Aql~X-1 by \citet{heil2015power}. To ensure clarity, the original figure has been adapted by removing power colours from black hole systems. \spacedlowsmallcaps{right} Power colour ratios obtained with \chromos for Aql~X-1. Only power colours with a fractional variance constrained at a $3\sigma$ level in each frequency band have been included.}\label{fig:aqlx1}
%\end{figure}
%
%\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
%\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
%\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
%
%%\setlength{LTcapwidth}{10.in}
%\begin{landscape}
%\begin{longtable}{@{\extracolsep{\fill}}>{\centering\arraybackslash}p{4cm}ccccccc@{}}
%\caption[Object properties]{Overview of \acp{LMXB} showing system properties and observation details, sorted by compact object type, and then by name. Alternative source names can be found in appendix~\ref{ch:sources}, in table~\ref{tab:aka}. Systems are divided into atolls (A), Z sources (Z), accreting pulsars (AP), accreting millisecond pulsars (AMP), and objects showing characteristics of both atoll and Z sources (AZ). A further division is made between persistent accretion-powered pulsars with burst oscillations (P), intermittent ones (I) and burst oscillation sources without detectable accretion-powered pulsations (B) \citep[see][for a review]{watts2012thermonuclear}. Intermittent sources have been assigned to the pulsar or burster group on basis of their timing properties \citep{marieke}. Spin frequencies have been given where possible, as are inclination angles. \spacedlowsmallcaps{\#Good} gives the total number of observations with a significant detected variance at a 3$\sigma$-level in all four power colour frequency bands. The total number of available ObsIDs in the \ac{RXTE} archive are also given per source. References are denoted with numbers, and are given at the bottom of this table.}\label{tab:objects}\\
%\multicolumn{8}{l}{} \\[-7pt]
%\toprule
%\tableheadline{Source}&\tableheadline{Type}&\tableheadline{Burster/Pulsar}&\tableheadline{Spin Freq. (Hz)}&\tableheadline{Inclination ($^\circ$)}&\tableheadline{\#Good}&\tableheadline{\#ObsID}&\tableheadline{References}\\
%\midrule
%\endfirsthead
%\toprule
%\tableheadline{Source}&\tableheadline{Type}&\tableheadline{Burster/Pulsar}&\tableheadline{Spin Freq. (Hz)}&\tableheadline{Inclination ($^\circ$)}&\tableheadline{\#Good}&\tableheadline{\#ObsID}&\tableheadline{References}\\
%\midrule
%\endhead
%4U 0614+09&A&B&414.7&86&85&502&1,2,2,3\\
%4U 1636-53&A&B&581.9&64&45&1556&4,5,2,6\\
%4U 1702-43&A&B&329&&28&210&7,5,5\\
%4U 1705-44&A&&&51&55&516&8,9\\
%4U 1728-34&A&B&363&50&55&405&7,5,5,10\\
%Aql X-1&A&I/B&550.3&72–79&145&596&4,5,2,11\\
%Cir X-1&AZ&&&90&52&811&12,13\\
%Cyg X-2&Z&&&62.5&62&567&8,14\\
%EXO 0748-676&A&B&552.5&75&93&746&15,5,16,17\\
%GX 17+2&Z&&&&4&206&8\\
%GX 340+0&Z&&&&5&97&8\\
%GX 349+2&Z&&&&3&142&8\\
%GX 5-1&Z&&&&5&167&8\\
%HETE J1900.1-2455&AMP&I/P&337.3&30&129&361&18,5,19,20\\
%IGR J00291+5934&AMP&&599&&45&479&21,22\\
%IGR J17480-2446&AP&P&11&&42&159&23,5,23\\
%IGR J17498-2921&AMP&P&401&&3&129&24,5,24\\
%KS 1731-260&A&B&524&&21&82&7,5,5\\
%SAX J1808.4-3658&AMP&P&401&55&25&1337&25,5,25,26\\
%SWIFT J1756.9-2508&AMP&B&182&&19&50&27\\
%Sco X-1&Z&&&30&48&598&8,28\\
%Sgr X-1&A&&&&12&109&8\\
%Sgr X-2&Z&&&&35&88&12\\
%V4634 Sgr&A&&&&119&1008&29\\
%XB 1254-690&A&&&&10&94&30\\
%XTE J0929-314&AMP&B&185&&7&46&31,31,22\\
%XTE J1701-462&AZ&&&60&94&872&12,28\\
%XTE J1751-305&AMP&B&435&&21&274&32\\
%XTE J1807-294&AMP&P&190&&4&112&33,2,22\\
%XTE J1814-338&AMP&P&314&&17&93&34,5,22\\
%\midrule
%\\[-20pt]
%GX 339-4&BH&&&&391&1401&35\\
%H1743-322&BH&&&&123&558&36\\
%XTE J1550-564&BH&&&&141&423&37\\
%\bottomrule
%\\[-7pt]
%\multicolumn{8}{L{23cm}}{
%\spacedlowsmallcaps{1}~\citet{mendez1997kilohertz} \spacedlowsmallcaps{2}~\citet{marieke} \spacedlowsmallcaps{3}~\citet{schulz2010dynamical} \spacedlowsmallcaps{4}~\citet{liu2001catalogue} \spacedlowsmallcaps{5}~\citet{watts2012thermonuclear} \spacedlowsmallcaps{6}~\citet{pandel2008relativistic} \spacedlowsmallcaps{7}~\citet{galloway2008thermonuclear} \spacedlowsmallcaps{8}~\citet{hasinger1989two} \spacedlowsmallcaps{9}~\citet{di2009relativistically} \spacedlowsmallcaps{10}~\citet{shaposhnikov2002bursting} \spacedlowsmallcaps{11}~\citet{galloway2016intermittent} \spacedlowsmallcaps{12}~\citet{fridriksson2015common} \spacedlowsmallcaps{13}~\citet{iaria2008chandra} \spacedlowsmallcaps{14}~\citet{orosz1999optical} \spacedlowsmallcaps{15}~\citet{homan2015geometric} \spacedlowsmallcaps{16}~\citet{galloway2010discovery} \spacedlowsmallcaps{17}~\citet{parmar1986discovery} \spacedlowsmallcaps{18}~\citet{watts2009discovery} \spacedlowsmallcaps{19}~\citet{morgan2005hete} \spacedlowsmallcaps{20}~\citet{papitto2013accretion} \spacedlowsmallcaps{21}~\citet{galloway2005discovery} \spacedlowsmallcaps{22}~\citet{gladstone2007analysing} \spacedlowsmallcaps{23}~\citet{papitto2011spin} \spacedlowsmallcaps{24}~\citet{papitto2011discovery} \spacedlowsmallcaps{25}~\citet{wijnands1998millisecond} \spacedlowsmallcaps{26}~\citet{cackett2009broad} \spacedlowsmallcaps{27}~\citet{krimm2007discovery} \spacedlowsmallcaps{28}~\citet{crampton1976spectroscopic} \spacedlowsmallcaps{29}~\citet{van2005relations} \spacedlowsmallcaps{30}~\citet{bhattacharyya2007timing} \spacedlowsmallcaps{31}~\citet{galloway2002discovery} \spacedlowsmallcaps{32}~\citet{markwardt2002discovery} \spacedlowsmallcaps{33}~\citet{markwardt2003discovery} \spacedlowsmallcaps{34}~\citet{markwardt2003xte} \spacedlowsmallcaps{35}~\citet{wijnands1999broadband} \spacedlowsmallcaps{36}~\citet{homan2005high} \spacedlowsmallcaps{37}~\citet{homan2001correlated}
%} \\
%\end{longtable}
%\end{landscape}
%%
%% \begin{table}
%%     \myfloatalign
%%   \begin{tabularx}{\textwidth}{Xccc} \toprule
%%     \tableheadline{Source}	& \tableheadline{Inclination}	& \tableheadline{Type (A/Z[Sco/Cyg])} & \tableheadline{Dipper} \\
%%     \midrule
%%     \TODO & \TODO & \TODO & \TODO\\
%%     \midrule
%%     \TODO & \TODO & \TODO & \TODO\\%\citeauthor{knuth:1976} \\
%%     \bottomrule
%%   \end{tabularx}
%%   \caption[\TODO]{\TODO}\label{tab:objects}
%% \end{table}


\subsubsection{Primary model}
\label{sec:methods_pipeline_primary_model}

The primary model consists of computing two moving average signals with
different time windows. The longer the time window, the more samples are
averaged and consequently the slower it varies with price variations. Having two
signals, one \emph{fast} and one \emph{slow} provides information of how short
and long term price movement vary one with respect to other. In particular, the
strategy takes advantage of the crossing points of both signals. When the fast
signal crosses above the slow signal, a buy event is generated. When the slow
signal crosses above the fast signal, a sell event is generated.

Note that from a daily sampled, positive and real signal as the bitcoin close
price in USD is, we obtain another two daily sampled, positive and real signals
(fast and slow). The latter two signals generate when they cross the events of
the primary model. These events are not equally spaced anymore, and the series
is categorical, its values are $\{1, -1\}$ which represent $\{buy, sell\}$
respectively. See figure \ref{fig:buy_and_sell_labels} to show these events over
the bitcoin price series.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{methods/images/buy_and_sell_labels.png}
    \caption{Buy and sell signals over the price series.}
    \label{fig:buy_and_sell_labels}
\end{figure}

The process of obtaining the signal with the bets is called \emph{labeling}
because it generates a series of \emph{labels} that determine a concrete action:
buy or sell the position. Next, \emph{metalabeling} process comes. It consists
in providing a probability to each \emph{label} which will be used to size the
bet. To assess whether the \emph{label} is correct or not, the triple barrier
method is used:

\begin{enumerate}
  \item Define a profit taking and stop loss rate for which a buy and sell
        signals will be considered valid. For a given price and time, a new
        greater value and lesser value are defined based on both rates before.
  \item Define a time constant $h$ (expressed as a positive and integer multiple
        of the sampling rate, i.e. number of days) that determines for each
        label, a new time stamp ahead.
  \item Determine whether the price signal hits the greater price or the lesser
        price before reaching the time stamp ahead of $h$ periods from the
        label's event. When:
  \begin{itemize}
    \item A $1$-valued label gets a cross with the greater price barrier, the
          metalabel is $1$ to indicate a positive label.
    \item A $-1$-valued label gets a cross with the lesser price barrier, the
          metalabel is $1$ to indicate a positive label.
    \item A $1$-valued label gets a cross with the lesser price barrier, the
          metalabel is $0$ to indicate a positive label.
    \item A $-1$-valued label gets a cross with the greater price barrier, the
          metalabel is $0$ to indicate a positive label.
    \item When both $1$ and $-1$ valued labels do not get a corresponding cross
          with any of the price barriers, the return sign between the price at
          $h$ sample periods ahead of label's time stamp and the label's price
          is used. If the sign of the return and the label match, the metalabel
          is $1$, otherwise it is $0$.
  \end{itemize}
\end{enumerate}

Labels and metalabels are fundamental series to build the secondary model. The
secondary model is a classifier that is trained with metalabels. The predicted
probability will help to size the bet on each label. In mathematical terms:
$Metalabels = f_{(Labels, ...)}$ where $f$ represents the secondary model.