%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Survival Analysis}
\label{chap:survival}

Survival analysis is a subfield of statistics which examines problems involving the timing of events
such as death, component failure, or exiting a line of therapy after an adverse effect, in a population of subjects under study.
Using the appropriate models, quantities such as the lifetime, or failure rate, of a subject
can be estimated, as well as their dependence on other independent variables.
A variety of common models are available,
each with their own assumptions, capabilities, and limitations,
which make them best suited for certain applications.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Nomenclature}
\label{survival:nomenclature}

\begin{symbollist}
  \item[Event] The event of interest in a study, \eg death, component failure\ldots
  \item[$t$] Time from the start of observation to an event, the conclusion of the study, or the withdrawal of a subject.
  \item[$T$] Time that an event occurred.
  \item[$x$] Independent variable(s) under consideration.
  \item[Censoring] Right\footnote{Left censored subjects enter the study after the event of interest has already occurred at an unknown $t < 0$.} censored subjects have no events during the observation period, either due to early withdrawal or the conclusion of the study. The true lifetime of these subjects is unavailable for analysis, \ie they have been censored.
  \item[$S\left(t\right)$] The survival function $S\left(t\right)$ is the probability that a subject survives longer than $t$, \ie $S\left(t\right) = P\left(T > t\right)$.
  \item[$F\left(t\right)$] The lifetime distribution function $F\left(t\right)$ is the complement of the survival function, \ie $F\left(t\right) = 1 - S\left(t\right) = P\left(T \leq t\right)$.
  \item[$f\left(t\right)$] The event density function $f\left(t\right)$ is the time derivative of the lifetime distribution function $F\left(t\right)$, $f\left(t\right) = \dv{F}{t} = -\dv{S}{t}$, if it exists.
  \item[$\lambda\left(t\right)$] The hazard function $\lambda\left(t\right)$ is the event rate at $t$ conditional on survival to time $t$, \ie $T > t$, see \cref{eq:survival:hazard_def}. Any $\lambda\left(t\right)$ can be a hazard function, provided it satisfies \cref{eq:survival:hazard_cond}.
  \item[$\Lambda\left(t\right)$] The cumulative hazard function $\Lambda\left(t\right)$ is the integral of $\lambda\left(t\right)$ with respect to time \cref{eq:survival:cum_hazard:def}. Also see the relations in \cref{eq:survival:cum_hazard:to_lambda,eq:survival:cum_hazard:to_S}.
  \item[HR] The hazard ratio (HR) compares the hazards of two subsets of subjects partitioned by $x$ at time $t$, $\text{HR} = \lambda\left(x = 1\right) / \lambda\left(x=0\right)$. Also known as the relative risk.
\end{symbollist}

\begin{subequations}\label{eq:survival:hazard_def}
\begin{align}
\lambda\left(t\right) \dd{t} &= P\left(T \leq t + \dd{t} \mid T > t\right),\,\text{as} \,\, \dd{t} \to 0 \label{eq:survival:hazard_def:a} \\
\lambda\left(t\right) &= \lim_{\dd{t} \to 0} \frac{P\left(T \leq t + \dd{t} \cap T > t\right)}{P\left(T > t\right)\,\dd{t}} \label{eq:survival:hazard_def:b} \\
&= \frac{1}{S\left(t\right)} \lim_{\dd{t} \to 0} \frac{P\left(t < T \leq t + \dd{t}\right)}{\dd{t}} \label{eq:survival:hazard_def:c} \\
&= \frac{1}{S\left(t\right)} \lim_{\dd{t} \to 0} \frac{F\left(t + \dd{t}\right) - F\left(t\right)}{\dd{t}} \label{eq:survival:hazard_def:d} \\
&= \frac{f\left(t\right)}{S\left(t\right)} = -\frac{1}{S} \dv{S}{t} \label{eq:survival:hazard_def:e}
\end{align}
\end{subequations}

\begin{subequations}\label{eq:survival:hazard_cond}
\begin{gather}
\forall t \geq 0, \, \lambda\left(t\right) \geq 0 \label{eq:survival:hazard_cond:a} \\
\int_{0}^{\infty} \lambda\left(t\right) \, \dd{t} = \infty \label{eq:survival:hazard_cond:b}
\end{gather}
\end{subequations}

\begin{subequations}\label{eq:survival:cum_hazard}
\begin{gather}
\Lambda\left(t\right) = \int_{0}^{t} \lambda\left(u\right) \, \dd{u} = - \ln\left(S\left(t\right)\right) \label{eq:survival:cum_hazard:def} \\
\lambda\left(t\right) = \dv{\Lambda}{t} = -\frac{1}{S} \dv{S}{t} \label{eq:survival:cum_hazard:to_lambda} \\
S\left(t\right) = \exp\left(-\Lambda\left(t\right)\right) \label{eq:survival:cum_hazard:to_S}
\end{gather}
\end{subequations}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Kaplan-Meier Model}
\label{survival:km}

The Kaplan-Meier model \cite{km} $\hat{S}_{\text{KM}}\left(t\right)$ is a non-parametric
estimate of $S\left(t\right)$ computed from empirical data.

\begin{equation}\label{eq:survival:km}
\hat{S}_{\text{KM}}\left(t\right) = \prod_{i:\,t_{i} < t} \left(1 - \frac{d_{i}}{n_{i}}\right)
\end{equation}

\noindent Here the product is over all times $t_{i}$ at which $d_{i}$ events occurred,
and $n_{i}$ is the number of subjects still under study at $t_{i}$,
\ie subjects who have not had an event or been censured.
Due to its construction, $\hat{S}_{\text{KM}}\left(t\right)$ remains steady
between $t_{i}$ but drops vertically at each data point.
Therefore the derivative does not exist and we can not estimate $f\left(t\right)$ or $\lambda\left(t\right)$.
See \cref{fig:stanford_km:km} for one example of a Kaplan-Meier curve.

We can apply the Kaplan-Meier estimator to different subsets of subjects partitioned by a categorical variable $x$
in order to understand the dependence of $S\left(t\right)$ on $x$.
Note that $x$ can not be a continuous variable, and must be binned to integer classes if that is the case.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Exponential Model}
\label{survival:exp}

The exponential model is a parametric estimate of $S\left(t\right)$
valid when the hazard is expected to be constant with respect to $t$.
In nuclear physics\footnote{$\dv{N}{t} = -\lambda N$, $N\left(t\right) = N_{0} e^{-\lambda t}$,
half-life $t_{1/2} = \ln\left(2\right) / \lambda = \tau \ln\left(2\right)$, where $\tau$ is the time constant.} $\lambda\left(t\right) = \lambda$
decays per unit time really is a constant.
However, in most situations $\lambda = c$ is unrealistic over longer time scales,
as illustrated in \cref{survival:additional:bathtub}.
The exponential model can accommodate both categorical and continuous $x$ independent variables.

The survival function for the exponential model is simply

\begin{equation}\label{eq:survival:exp}
\hat{S}_{\text{Exp}}\left(t\right) = e^{-\lambda t}
\end{equation}

\noindent where the hazard can be expanded in terms of $\vb{x}$, $\vb*{\beta}$ as in a regression analysis:

\begin{equation}\label{eq:survival:exp_lambda}
\begin{aligned}
\lambda\left(x\right) &= \exp\left(\beta_{0} + \sum_{j=1}^{n}\, \beta_{j} x_{j}\right) \\
\log\left(\lambda\right) &= \innerproduct{\vb{x}}{\vb*{\beta}}
\end{aligned}
\end{equation}

\noindent Here we are making the {\em proportional hazards assumption},
\ie the differences in $\lambda$ between subgroups in $x_{j}$
are proportional\footnote{Really $\log\left(\lambda\right) \propto \beta$.} to $\beta_{j}$
and constant over $t$.
To find the best estimate of $\hat{\vb*{\beta}}$,
$\lambda$ is used to derive a likelihood function which is
then optimized\footnote{The exact details of this optimization
are handled by common software libraries and are omitted here.
Similar approaches are also used to fit the later Weibull and Cox proportional-hazards models.}.

The hazard ratio for $x_{j}$ is:

\begin{equation}\label{eq:survival:exp_HR}
\begin{aligned}
\text{HR}_{\text{Exp}} &= e^{\ldots+\beta_{j} 1+\ldots} / e^{\ldots+\beta_{j} 0+\ldots} \\
&= e^{\beta_{j}}
\end{aligned}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Weibull Model}
\label{survival:weibull}

The Weibull model expands on the standard exponential model
by introducing a shape parameter $k$ to adjust the time dependence of the hazard:

\begin{equation}\label{eq:survival:weibull}
\begin{aligned}
\hat{S}_{\text{Weibull}}\left(t\right) &= e^{-\lambda_{\text{Weibull}} t} \\
\lambda_{\text{Weibull}} &= \exp\left(\beta_{0} t^{k} + \sum_{j=1}^{n}\, \beta_{j} x_{j}\right) \\
\text{HR}_{\text{Weibull}} &= e^{\beta_{j}}
\end{aligned}
\end{equation}

\noindent Here $\beta_{0}$ is the scale parameter of the hazard with respect to $t$,
while the other $\beta_{j}$ are scale parameters for the independent $x_{j}$.
For $k<0$ ($k>0$) the hazard monotonically decreases (increases) over time,
while $k=0$ returns the exponential model.
This allows phenomena such as burn in and compounding component failures to be modeled.
Numerous parameterizations and expansions of the Weibull model are available,
however they all share the property that
$\lambda = \exp\left(\beta_{0} f\left(t\right) + \sum_{j=1}^{n}\, \beta_{j} x_{j}\right)$
where $f\left(t\right)$ is a {\em known}, almost always monotonic, function of $t$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Cox Proportional-Hazards Model}
\label{survival:cox}

The Cox proportional-hazards model \cite{cox} is a
semiparametric\footnote{Semiparametric as $\lambda_{0}\left(t\right)$ is unspecified, but $\vb*{\beta}$ are still parameters of $\lambda$.} model
which further expands on the Weibull model by allowing
the time dependence of $\lambda$ to be an unknown function.
As the name suggests, the proportional hazards assumption is retained
allowing us to compute useful HR values
without even knowing $\lambda\left(t\right)$,
or $S\left(t\right)$\footnote{We can
plot Kaplan-Meier $S\left(t\right)$ curves of predictions from a Cox model,
after fitting the baseline hazard emperically on a reference subset,
but only at discrete values of $\vb{x}$.
See \cref{fig:cox:cox_age_baseline_survival} for one example.}.

In the Cox model we assume the hazard has the form of

\begin{equation}\label{eq:survival:cox_lambda}
\lambda_{\text{Cox}} = \lambda_{0}\left(t\right) \exp\left(\sum_{j=1}^{n}\, \beta_{j} x_{j}\right)
\end{equation}

\noindent where $\lambda_{0}\left(t\right)$ is the baseline hazard, an unspecified function of $t$.
When computing hazard ratios $\lambda_{0}\left(t\right)$ then drops out:

\begin{equation}\label{eq:survival:cox_HR}
\text{HR}_{\text{Cox}} = e^{\beta_{j}}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Method Comparison}
\label{survival:comp}

\begin{table}[H]
\centering
\begin{tabular}{l|l|l}
\multicolumn{1}{c|}{Kaplan-Meier} & \multicolumn{1}{c|}{Exponential / Weibull} & \multicolumn{1}{c}{Cox} \\
\cline{1-3}
\begin{tabular}[c]{p{0.3\textwidth}}
Pro
\begin{itemize}
  \item Simple to compute and understand
  \item Can estimate $S$
\end{itemize}
\\
Con
\begin{itemize}
  \item No functional form
  \item Can \textbf{not} estimate HR
  \item Only can handle a few categorical $x$
\end{itemize}
\end{tabular}

&

\begin{tabular}[c]{p{0.3\textwidth}}
Pro
\begin{itemize}
  \item Can estimate $S$ and HR
\end{itemize}
\\
Con
\begin{itemize}
  \item $\lambda\left(t\right) = c$ can be unrealistic
  \item Weibull: $\lambda = f\left(t\right)$ of a known form
\end{itemize}
\end{tabular}

&

\begin{tabular}[c]{p{0.3\textwidth}}
Pro
\begin{itemize}
  \item $\lambda$ can be an unknown function of $t$
  \item Can estimate HR
\end{itemize}
\\
Con
\begin{itemize}
  \item Can \textbf{not} estimate $S$
\end{itemize}
\end{tabular}

\\
\end{tabular}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Model Assumptions}
\label{survival:assumptions}

\begin{enumerate}[noitemsep]
  \item Any censoring is non-informative, \ie censoring is uncorrelated with the probabilities of different event outcomes.\label{item:survival:assumptions:censoring}
  \item Survival times $t$ are uncorrelated across subjects.\label{item:survival:assumptions:t_uncorr}
  \item $\vb{x}$ is constant over time, per subject.\label{item:survival:assumptions:X_constant}
  \item Hazards are proportional, \ie hazard ratios are constant over time.\label{item:survival:assumptions:prop_hazard}
  \item $\log\left(\lambda\right)$ is linear with respect to the continuous $x_{j}$ independent variables.\label{item:survival:assumptions:X_linearity}
\end{enumerate}

\Cref{item:survival:assumptions:censoring,item:survival:assumptions:t_uncorr,item:survival:assumptions:X_constant}
are assumptions for all survival models, while
\cref{item:survival:assumptions:prop_hazard,item:survival:assumptions:X_linearity}
apply to the exponential, Weibull and Cox models.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Tests and Corrections}
\label{survival:assumptions:tests_and_corrections}

\begin{itemize}[noitemsep]
  \item[\cref{item:survival:assumptions:censoring}.] Exploratory data analysis to detect differences in $\vb{x}$ distributions between censored and non-censored subjects. If violated, better input data is needed, redesign study.

  \item[\cref{item:survival:assumptions:t_uncorr}.] Exploratory data analysis, study design question. If violated, redesign study.

  \item[\cref{item:survival:assumptions:X_constant}.] Exploratory data analysis, study design question. If unavoidable, can try a more complex time dependent covariate model.

  \item[\cref{item:survival:assumptions:prop_hazard}.] Check with a Schoenfeld test or complementary log-log plot. If violated, stratify problematic $x_{j}$ variables into separate models, or use more complex models with time dependent $\beta$ coefficients.

  \item[\cref{item:survival:assumptions:X_linearity}.] Check the appropriate residual plots. If violated, try to linearize the offending continuous $x_{j}$ with a transformation or bin to a categorical variable.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Schoenfeld Test}
\label{survival:assumptions:schoenfeld}

Schoenfeld residuals are similar in concept to normal residuals in regression analyses,
but instead of looking at the difference between the true and estimated values of $y$,
$\hat{\vb*{\epsilon}} = \vb{y} - \mathbf{X} \hat{\vb*{\beta}}$,
we look at the difference in $\vb*{\beta}$ and $\hat{\vb*{\beta}}$ for a particular $y$ value.
In this way, Schoenfeld residuals represent the difference in the
coefficient $\beta_{j}$ fit from all data points, and a single point.
To validate the proportional hazards assumption
the Schoenfeld residuals \cite{schoenfeld} must not change over time.
An example Schoenfeld residual plot is provided in
\cref{fig:cox:schoenfeld_residuals}.
Besides graphically looking at the residuals,
we can use {\chiSqtest}s to compute {\pvalue}s
to test the null hypothesis, \ie the Schoenfeld residuals and time are independent.
In this case, the proportional hazards assumption is supported
if we find large, non-significant {\pvalue}s.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Complementary Log-Log Plot}
\label{survival:assumptions:cloglog}

For categorical variables we can look at the
complementary log-log plot
of $\log\left(-\log\left(S\left(t\right)\right)\right)$ versus $\log\left(t\right)$.
An example complementary log-log plot is provided in \cref{fig:stanford_cloglog}.
If the different categories have parallel curves
the proportional hazards assumption is
valid for the variable in question.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Other Residual Plots}
\label{survival:assumptions:linearity}

To test the linearity of $\log\left(\lambda\right)$ with respect to the continuous $x_{j}$
we may plot the Martingale residuals.
Martingale residuals range from $-\infty$ to $1$
and measure the difference between the observed and predicted survival for a subject.
Subjects who have an event ``to early'' will have residuals near one,
where as subjects who have an event ``to late'' will have residuals near $-\infty$.
The mean of the residuals should be $0$, see \cref{fig:cox:martingale_residuals:prediction} for an example.
We can also plot the Martingale residuals versus any particular $x_{j}$
to judge the linearity along that specific dimension, see \cref{fig:cox:martingale_residuals:age}.

The Martingale residuals can be symmetrized into deviance residuals,
which have a mean of $0$ and standard deviation of $1$,
for easier interpretation, see \cref{fig:cox:outliers:deviance}.

Lastly, we can recompute the model fit leaving out event $i$
and measure the resulting change in the $\beta_{j}$ coefficients,
$\hat{\Delta}_{ij} = \hat{\beta}_{j} - \hat{\beta}_{j}^{\left(i\right)}$, \ie ``dfbeta''.
Plotting $\hat{\Delta}_{ij}$ versus event number $i$
can then point to overly influential events, see \cref{fig:cox:outliers:dfbeta}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Additional Concepts}
\label{survival:additional}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Logrank Test}
\label{survival:additional:logrank}

The logrank test is a method for comparing the survival functions of two different groups.
Survival status versus group membership $2\times2$ tables are computed at each distinct time $t_{i}$,
before being combined into one $\chi_{\text{logrank}}^{2}$ test statistic
with the Cochran-Mantel-Haenszel test\footnote{Sometimes
the logrank test is itself referred to as the Mantel-Haenszel test.}.
$\chi_{\text{logrank}}^{2}$ approximately has the \chiSqdist with 1 degree of freedom,
allowing the production of {\pvalue}s to test the null hypothesis
that there is {\em no} difference in the group's survival functions.

The logrank test is most accurate when the proportional hazards assumption is satisfied,
but is still somewhat usable if the difference in hazard ratios has only one sign,
\ie the $S\left(t\right)$ curves do not cross as can be checked with a complementary log-log plot.
In \R the logrank test can be computed with the \texttt{survdiff} function,
with the parameter $\rho$ left at its default value of $0$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Odds, Log Odds, and the Odds Ratio in Relation to the Hazard Ratio}
\label{survival:additional:odds}

To begin, note that the odds of an event occurring are:

\begin{equation}\label{eq:survival:odds}
\text{Odds} = \frac{P\left(\text{Occurring}\right)}{P\left(\text{Not Occurring}\right)} = \frac{p}{1-p}
\end{equation}

\noindent For example, when rolling a fair dice, $\text{odds}\left(3\right) = (1/6) / (5/6) = 0.2$, or 1:5.
Odds may vary from $0$ to $\infty$, with $1$ representing fair odds.
As such, they are not symmetric and are somewhat unwieldy.
For this reason it is common to take the log of the odds,
\ie log odds, which varies symmetrically between $-\infty$ and $\infty$, with 0 being fair.

An odds ratio is simply the ratio of two odds.
If we take a biased coin with $\text{odds}\left(\text{heads}\right) = 2$, or 2:1 in favor of heads,
the odds ratio between the biased coin and a fair coin is $2/1 = 2$,
\ie the odds of getting heads with the biased coin are 2 times that of the fair coin.
If desired, we can similarly take the ratio of log odds.

In logistic regression models, see \cref{class:logistic},
it is possible to compute an odds ratio for the input feature $x_{j}$,
representing the change in odds due to a one unit increase in $x_{j}$.
This is very similar in concept to the HR,
which itself is a ratio of two probabilities from different sets of subjects separated by $x_{j}$.
However, as odds and probability are different quantities,
relative risks, including the HR, are not equivalent to odds ratios.
In some cases the relative risk and odds ratio may have similar values,
see Table 3 of \cite{pmid26623395},
but it is important to remember they are ultimately different concepts.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Expected Future Lifetime}
\label{survival::additional:efl}

The expected future lifetime $\expval{T-t_{0}}$ is the
expected additional time of survival for a subject having already survived to $t_{0}$.
We can derive $\expval{T-t_{0}}$ \cref{eq:survival:efl_der:efl}
from the probability of an event occurring between $t_{0}$ and $t_{0} + t$ \cref{eq:survival:efl_der:P}
using the prior work of \cref{eq:survival:hazard_def} and integration by parts:

\begin{subequations}\label{eq:survival:efl_der}
\begin{gather}
P\left(T \leq t_{0} + t \mid T > t_{0}\right)
= \frac{F\left(t_{0} + t\right) - F\left(t_{0}\right)}{S\left(t_{0}\right)} \label{eq:survival:efl_der:P} \\
\text{PDF}
= \dv{}{t} P\left(T \leq t_{0} + t \mid T > t_{0}\right)
= \frac{f\left(t_{0} + t\right)}{S\left(t_{0}\right)} \label{eq:survival:efl_der:pdf} \\
\expval{T-t_{0}}
= \int_{0}^{\infty} t \, \text{PDF} \, \dd{t}
= \frac{1}{S\left(t_{0}\right)} \int_{0}^{\infty} S\left(t\right) \, \dd{t} \label{eq:survival:efl_der:efl}
\end{gather}
\end{subequations}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Bathtub Curve}
\label{survival:additional:bathtub}

In reliability engineering the hazard function of many components
can be estimated as a three part ``bathtub'' curve.
Initially the failure, \ie event, rate decreases as defective components fail early,
before reaching a constant plateau where failures are essentially random events.
As components wear out and reach the end of their useful lifetimes, the failure rate increases again.
Combining these three effects produces a bathtub-shaped curve, as seen in \cref{fig:bathtub_curve}.
We can model these situations with the Cox proportional-hazards model,
or a Weibull model with an appropriately shaped $\lambda\left(t\right)$.

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/survival/bathtub_curve}
\vspace{0.2cm}
\caption{
Illustration of a bathtub curve hazard function, by \href{https://en.wikipedia.org/wiki/File:Bathtub_curve.svg}{Wyatts}.
}
\label{fig:bathtub_curve}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Example \R Code}
\label{survival:Rcode}

A simple example survival analysis in \R utilizing the built-in
\texttt{stanford2} Stanford heart transplant dataset is provided in this section.
The examples here have been adapted from Mike Marin's
\href{https://www.youtube.com/playlist?list=PLqzoL9-eJTNDdnKvep_YHIwk2AMqHhuJ0}{series of lectures},
and \href{http://www.sthda.com/english/wiki/cox-model-assumptions}{notes} available from the STHDA.
The full \R code can also be found in
\href{https://github.com/mepland/data_science_notes/blob/main/sections/appendixes/additional/example_survival.R}{\texttt{example\_survival.R}}.
An equivalent analysis has also been performed in \python using the
\href{https://lifelines.readthedocs.io/en/latest}{\texttt{lifelines}} package,
see
\href{https://github.com/mepland/data_science_notes/blob/main/sections/appendixes/additional/example_survival.ipynb}{\texttt{example\_survival.ipynb}}.

The status variable represents the event, death of a patient, as $\text{status}=1$,
and the censoring of a patient as $\text{status}=0$.
Three independent variables $x$ are available,
age in years,
age categories - under or over 40,
and the numeric T5 mismatch score between donor and recipient.

\begin{figure}[H]
\centering
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_km}
  \caption{Kaplan-Meier}
  \label{fig:stanford_km:km}
  \end{subfigure}
  ~
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_km_annotated}
  \caption{Annotated}
  \label{fig:stanford_km:annotated}
  \end{subfigure}
\caption{
Kaplan-Meier plot of the Stanford heart transplant dataset. Censored patients are displayed with marks.
The version on the right adds additional annotations, such as
subplots for patients at risk and censoring events,
the \pvalue from the logrank test, and a median reference line.
}
\label{fig:stanford_km}
\end{figure}

\subsubsection{Initialization}
\label{survival:Rcode:init}

Here we load the necessary packages and data,
and create the categorical age variable \texttt{agecat}.

\begin{lstlisting}[language=R]
> install.packages(c("survival", "survminer"))
> library("survival")
> library("survminer")

> df <- stanford2
> df$agecat <- cut(df$age, breaks=c(0,40, Inf), labels=c('Under 40', 'Over 40'), right=FALSE)
> df <- df[with(df, order(time)),]

> df[1:5,c('time', 'status', 'age', 'agecat', 't5')]
    time status age   agecat   t5
21   0.5      1  41  Over 40 0.87
133  1.0      1  21 Under 40 0.47
184  1.0      0  27 Under 40   NA
16   1.0      1  54  Over 40 0.47
183  2.0      0  39 Under 40   NA
> summary(df[c('time', 'age', 'agecat', 't5')])
      time              age             agecat          t5
 Min.   :   0.50   Min.   :12.00   Under 40: 65   Min.   :0.000
 1st Qu.:  64.75   1st Qu.:35.00   Over 40 :119   1st Qu.:0.690
 Median : 351.00   Median :44.00                  Median :1.040
 Mean   : 696.94   Mean   :41.09                  Mean   :1.117
 3rd Qu.:1160.75   3rd Qu.:49.00                  3rd Qu.:1.460
 Max.   :3695.00   Max.   :64.00                  Max.   :3.050
                                                  NA's   :27
\end{lstlisting}

\subsubsection{Kaplan-Meier Model}
\label{survival:Rcode:km}

The Kaplan-Meier model for this data partitioned by \texttt{agecat}
shows a median survival time of \num{1271} (\num{431}) for patients Under 40 (Over 40).
The logrank test, \texttt{survdiff}, returned a \pvalue of \num{0.07},
thus while the survival curves appear visually different we {\em can not} reject the null hypothesis,
the survival curves are the same, at the standard \pvalue of \num{0.05}.

\begin{lstlisting}[language=R]
> km.model <- survfit(Surv(time, status) ~ agecat, data=df, type='kaplan-meier')
> summary(km.model)$table
                records n.max n.start events   *rmean *se(rmean) median 0.95LCL 0.95UCL
agecat=Under 40      65    65      65     32 1520.783   220.1031   1271     731      NA
agecat=Over 40      119   119     119     81 1123.229   143.3516    431     202     897

> survdiff(Surv(time, status) ~ agecat, data=df)
Call:
survdiff(formula = Surv(time, status) ~ agecat, data = df)

                  N Observed Expected (O-E)^2/E (O-E)^2/V
agecat=Under 40  65       32     41.3      2.10      3.32
agecat=Over 40  119       81     71.7      1.21      3.32

 Chisq= 3.3  on 1 degrees of freedom, p= 0.07

> pdf('~/stanford_km.pdf')
> ggsurvplot(km.model, xlab='Time', ylab='S(t)', size = 1, linetype = 'strata', palette=c('#4e79a7', '#f28e2b'), conf.int = TRUE, legend = c(0.85, 0.85), legend.y = 1, legend.title = '', legend.labs = c('Under 40', 'Over 40'))
> dev.off()

> pdf('~/stanford_km_annotated.pdf')
> ggsurvplot(km.model, xlab='Time', ylab='S(t)', size = 1, linetype = 'strata', palette=c('#4e79a7', '#f28e2b'), conf.int = TRUE, legend = c(0.85, 0.85), legend.y = 1, legend.title = '', legend.labs = c('Under 40', 'Over 40'),
pval = TRUE, # Add survdiff p-value
risk.table = TRUE, # Absolute number at risk
risk.table.y.text.col = FALSE, risk.table.col = "strata",
ncensor.plot = TRUE, # plot censored patients vs time
surv.median.line = "h", # add horizontal median
)
> dev.off()
\end{lstlisting}

\subsubsection{Cox Proportional-Hazards Model}
\label{survival:Rcode:cox}

Two Cox proportional-hazards models are demonstrated here,
the first using age alone and the second adding the T5 score.
To compare apples-to-apples, subjects with null T5 values are excluded from both models.

In the age alone model, the coefficient for age is
$\beta_{\text{age}} = \num{0.02990}$, giving a HR of
$\exp\left(\beta_{\text{age}}\right) = \num{1.03035}$.
The \SI{95}{\percent} confidence interval for the HR is {1.008} to \num{1.054}.
The \pvalue for $\beta_{\text{age}}$ is \num{0.00846},
thus we can reject the null hypothesis that $\beta_{\text{age}} = 0$.
We are also provided with three {\pvalue}s for the model as a whole;
\num{0.006} for the likelihood-ratio test,
\num{0.008} for the Wald test,
and \num{0.008} for the logrank test.
These test the null hypothesis that all $\beta_{i} = 0$.
Here all three tests reject the null hypothesis,
as could be expected since the single $\beta_{\text{age}}$ was significant,
but would be more interesting for higher dimensional models.
The concordance statistic, as explained in \cref{ml_general:eval:concordance},
of \num{0.595} is fairly poor, but is better than random guessing.

In the extended mode the age HR is very similar, \num{1.03006},
while the T5 HR is \num{1.18579}.
We can interpret the second model's T5 HR as follows;
at a given instant in time, an individual of age $a$ is
\num{1.18579} times, \ie \SI{18.579}{\percent}, as likely to die as an individual of age $a+1$, when controlling for T5 score.
The inverse HR, $\exp\left(-\beta_{\text{age}}\right)$, where the reference group has been switched, is also provided for convenience.

We can compare the two nested models using a likelihood-ratio test.
The resulting large \pvalue of \num{0.3563} supports the null hypothesis,
thus we conclude there is not a statistically significant difference between the two models.
We can remove T5 from the model without a loss of predictive power in this case.

The baseline survival function for the age only model
is plotted in \cref{fig:cox:cox_age_baseline_survival}.
A more complete plot including a range of ages can be found in the
\href{https://github.com/mepland/data_science_notes/blob/main/sections/appendixes/additional/example_survival.ipynb}{\texttt{example\_survival.ipynb}} implementation.

\begin{lstlisting}[language=R]
> cox.model_age <- coxph(Surv(time, status) ~ age, data=df[!is.na(df$t5), ])
> summary(cox.model_age)
Call:
coxph(formula = Surv(time, status) ~ age, data = df[!is.na(df$t5),
    ])

  n= 157, number of events= 102

       coef exp(coef) se(coef)     z Pr(>|z|)
age 0.02990   1.03035  0.01136 2.633  0.00846 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    exp(coef) exp(-coef) lower .95 upper .95
age      1.03     0.9705     1.008     1.054

Concordance= 0.595  (se = 0.034 )
Likelihood ratio test= 7.62  on 1 df,   p=0.006
Wald test            = 6.93  on 1 df,   p=0.008
Score (logrank) test = 7.01  on 1 df,   p=0.008

> pdf('~/stanford_cox_age_baseline_survival.pdf')
> ggsurvplot(survfit(cox.model_age, data=df[!is.na(df$t5), ]), xlab='Time', ylab='S(t)', size = 1, linetype = 'strata', palette=c('#4e79a7'), conf.int = TRUE, legend = c(0.85, 0.85), legend.y = 1, legend.title = '', legend.labs = c('Baseline Survival'))
> dev.off()

> cox.model_age_t5 <- coxph(Surv(time, status) ~ age + t5, data=df[!is.na(df$t5), ])
> summary(cox.model_age_t5)
Call:
coxph(formula = Surv(time, status) ~ age + t5, data = df[!is.na(df$t5),
    ])

  n= 157, number of events= 102

       coef exp(coef) se(coef)     z Pr(>|z|)
age 0.02961   1.03006  0.01136 2.608  0.00911 **
t5  0.17041   1.18579  0.18326 0.930  0.35243
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    exp(coef) exp(-coef) lower .95 upper .95
age     1.030     0.9708     1.007     1.053
t5      1.186     0.8433     0.828     1.698

Concordance= 0.59  (se = 0.034 )
Likelihood ratio test= 8.47  on 2 df,   p=0.01
Wald test            = 7.81  on 2 df,   p=0.02
Score (logrank) test = 7.87  on 2 df,   p=0.02

> anova(cox.model_age, cox.model_age_t5, test='LRT')
Analysis of Deviance Table
 Cox model: response is  Surv(time, status)
 Model 1: ~ age
 Model 2: ~ age + t5
   loglik Chisq Df P(>|Chi|)
1 -447.29
2 -446.86 0.851  1    0.3563
\end{lstlisting}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/survival/stanford_cox_age_baseline_survival}
\vspace{0.2cm}
\caption{
Baseline survival function for
an univariate Cox model fit to the Stanford heart transplant dataset.
The documentation describing what reference value of age is utilized by \R here is unclear.
By comparing to the equivalent plot produced by \texttt{lifelines} in
\href{https://github.com/mepland/data_science_notes/blob/main/sections/appendixes/additional/example_survival.ipynb}{\texttt{example\_survival.ipynb}}
the age used appears to also be the mean.
}
\label{fig:cox:cox_age_baseline_survival}
\end{figure}

\subsubsection{Checking Assumptions}
\label{survival:Rcode:assumptions}

In this section we check the assumptions made by the Cox and Kaplan-Meier models.
The Schoenfeld test of the proportional hazards assumption
is implemented via the \texttt{cox.zph} function
for both example Cox models; age alone and age plus T5 score.
The large global {\pvalue}s, \num{0.38} and \num{0.25} respectively,
fail to reject the null hypothesis and show that the
proportional hazards assumption is valid here.
Additionally, we are provided with {\pvalue}s for each individual variable.

We can also check the proportional hazards assumption graphically
for categorical variables, see \cref{fig:stanford_cloglog}.
Note that the complementary log-log plot applies to the example Kaplan-Meier model,
while the rest of the plots apply to the age alone Cox model.
Please see the captions of
\cref{fig:cox:schoenfeld_residuals,fig:stanford_cloglog,fig:cox:martingale_residuals,fig:cox:outliers}
for further discussion of these diagnostic plots.

\begin{lstlisting}[language=R]
> cox.model_age.ph <- cox.zph(cox.model_age)
> cox.model_age.ph
       chisq df    p
age     0.76  1 0.38
GLOBAL  0.76  1 0.38

> cox.model_age_t5.ph <- cox.zph(cox.model_age_t5)
> cox.model_age_t5.ph
       chisq df    p
age     0.83  1 0.36
t5      2.06  1 0.15
GLOBAL  2.77  2 0.25

> pdf('~/stanford_cox_age_schoenfeld_residuals.pdf')
> ggcoxzph(cox.model_age.ph)
> dev.off()

> pdf('~/stanford_cloglog.pdf')
> plot(km.model, fun='cloglog', xlab='log(t)', ylab='log(-log(S(t)))', col=c('#4e79a7', '#f28e2b'))
> legend('bottomright', inset=.02, legend=c('Under 40', 'Over 40'), col=c('#4e79a7', '#f28e2b'), lty=1:2, box.lty=0)
> dev.off()

> pdf('~/stanford_cox_age_martingale_residuals.pdf')
> ggcoxdiagnostics(cox.model_age, type = "martingale", ox.scale='linear.predictions')
> dev.off()

> pdf('~/stanford_cox_age_martingale_residuals_age.pdf')
> ggcoxfunctional(Surv(time, status) ~ age, data = df)
> dev.off()

> pdf('~/stanford_cox_age_deviance_residuals.pdf')
> ggcoxdiagnostics(cox.model_age, type = "deviance", ox.scale='linear.predictions')
> dev.off()

> pdf('~/stanford_cox_age_dfbeta.pdf')
> ggcoxdiagnostics(cox.model_age, type = "dfbeta", ox.scale='observation.id')
> dev.off()

# export data for use in python with lifelines
write.csv(df[!is.na(df$t5), ],"~/stanford.csv", row.names = FALSE)
\end{lstlisting}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/survival/stanford_cox_age_schoenfeld_residuals}
\vspace{0.2cm}
\caption{
Schoenfeld residual plot for
an univariate Cox model fit to the Stanford heart transplant dataset.
The residuals do not appear dependent on time,
which is supported by the non-significant \pvalue,
and shows that proportional hazards assumption
has been satisfied for this model.
Note that in a multivariate model we
would have a Schoenfeld residual plot for
each covariate.
}
\label{fig:cox:schoenfeld_residuals}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/survival/stanford_cloglog}
\vspace{0.2cm}
\caption{
Complementary log-log plot for categorized patient age
in the Stanford heart transplant dataset.
These curves are not parallel,
the proportional hazards assumption is violated in this variable.
}
\label{fig:stanford_cloglog}
\end{figure}

\begin{figure}[H]
\centering
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_cox_age_martingale_residuals}
  \caption{vs Log Partial Hazard, \newline\ie Linear Predictions}
  \label{fig:cox:martingale_residuals:prediction}
  \end{subfigure}
  ~
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_cox_age_martingale_residuals_age}
  \caption{vs Age}
  \label{fig:cox:martingale_residuals:age}
  \end{subfigure}
\caption{
Plots of Martingale residuals for
an univariate Cox model fit to the Stanford heart transplant dataset.
On the left, the residuals are plotted versus the
log of the partial hazard, \ie log hazard excluding the baseline hazard.
In \R this is known as \texttt{linear.predictors}, while in \texttt{lifelines} it is
\href{https://lifelines.readthedocs.io/en/latest/fitters/regression/CoxPHFitter.html\#lifelines.fitters.coxph_fitter.SemiParametricPHFitter.predict_log_partial_hazard}{log partial hazard}.
As can be seen in the blue fit, there is a slight nonlinearity.
On the right, residuals of the null model are plotted against the age variable alone.
For ages $\lesssim 40$, age appears to have a non-linear relationship with $\log\left(\lambda\right)$.
}
\label{fig:cox:martingale_residuals}
\end{figure}

\begin{figure}[H]
\centering
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_cox_age_deviance_residuals}
  \caption{Deviance}
  \label{fig:cox:outliers:deviance}
  \end{subfigure}
  ~
  \begin{subfigure}[c]{0.48\textwidth}\centering
  \includegraphics[width=\textwidth]{figures/survival/stanford_cox_age_dfbeta}
  \caption{dfbeta}
  \label{fig:cox:outliers:dfbeta}
  \end{subfigure}
\caption{
Additional residual plots for
an univariate Cox model fit to the Stanford heart transplant dataset.
On the left, the deviance residuals are plotted versus the
log of the partial hazard, \ie log hazard excluding the baseline hazard.
In \R this is known as \texttt{linear.predictors}, while in \texttt{lifelines} it is
\href{https://lifelines.readthedocs.io/en/latest/fitters/regression/CoxPHFitter.html\#lifelines.fitters.coxph_fitter.SemiParametricPHFitter.predict_log_partial_hazard}{log partial hazard}.
As can be seen in the blue fit, there is a slight nonlinearity.
On the right, the $\hat{\Delta}_{ij}$, \ie ``dfbeta'', residuals for age
are plotted against event number.
No event is particularly influential as $\abs{\hat{\Delta}_{ij}} \leq \num{0.004}$
for $\hat{\beta}_{\text{age}} = \num{0.02990}$.
}
\label{fig:cox:outliers}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Random Survival Forest (RSF)}
\label{survival:RSF}
% TODO


\section{Secure Multi-party Computation (SMPC)}\label{s:smpc}
Secure multi-party computation (SMPC)\footnote{Both SMPC and MPC abbreviations are used for secure multi-party computation; in this thesis we use them interchangeably.} or Secure Function Evaluation (SFE) (in the two-party setting) is a field of cryptography aiming to create methods that enable distinct parties, to jointly compute a function over their private inputs.
Only the outcome of that function is made public and the parties don’t learn anything more than their own input except whatever can be learned from the output of the function.

Secure multi-party computation was introduced in 1982 by Andrew Yao.
Its first form was that of secure two-party computation (2PC) with the so-called Millionaire's Problem \cite{yao1982protocols}.
The problem states that there are two millionaires wishing to know who is richer.
However, they should not find out any additional information about each other’s wealth.

In the general case we have N parties $P_1, P_2, \dots, P_N$ with private inputs $x_1, x_2, \dots, x_N$ respectively.
The goal is to compute a function $f(x_1, x_2, \dots, x_N)$ and learn nothing more than what they would have if a separate trusted party had collected their inputs, computed function $f$ for them, and the return the result to all parties.

There are two important requirements on any protocol for secure computation, namely \textit{privacy} and \textit{correctness} \cite{lindell2009secure}.
The privacy requirement suggests that nothing but what is absolutely essential should be learned.
More specifically, all parties should learn nothing more but the computation output.
The correctness requirement states that every party should receive the correct computation output, that is an adversary should not be able to cause the result of the computation to be different than the outcome of the function the parties agreed to compute.

Plenty of tasks can be modeled using SMPC.
The tasks that can be modeled using SMPC include simple ones such as coin tossing, as well as far more complex such as electronic voting, electronic auctions, anonymous transactions, anonymous chatting and private information retrieval systems.

Take electronic voting as an example.
The privacy requirement ensures that no party can learn the individual votes of other parties, while still learning the election outcome.
The correctness requirement ensures that no party can affect the outcome with means other than casting their one individual vote.
Similarly, in the electronic auction example, the privacy requirement ensures that no party learns the bids of other parties while still learning the winning bid. The correctness requirement ensures that no party can bias the auction outcome and that in fact the winning party has places the highest bid.

\subsection{Millionaire's Problem}\label{ss:millionaire}
The first problem that was modeled using SMPC is the Millionaire's Problem introduced by Yao, and it is a basic building block for such secure computations.

Here the two parties $P_1$, $P_2$ are the two millionaires.
Their private inputs are each one’s wealth $x_1$ and $x_2$ respectively.
The function which they wish to jointly compute can be formulated as

\begin{equation}
  f(x_1, x_2)=\begin{cases}
    x_1, & \text{if $x_1 > x_2$}.\\
    x_2, & \text{otherwise}.
  \end{cases}
\end{equation}

A lot of solutions have been proposed to the Millionaire's problem starting with the one from Yao himself \cite{yao1982protocols} as well as multiple others including \cite{ioannidis2003efficient, lin2005efficient}.

\subsection{Oblivious Transfer}\label{ss:oblivious-transfer}
Another example for the two-party case and a basic building block of SMPC systems is Oblivious Transfer (OT) firstly introduced in 1981 by Michael O. Rabin \cite{rabin2005exchange}.
Its simple flavor is 1-2 oblivious transfer or ``1 out of 2 oblivious transfer".

In the 1-2 oblivious transfer case we have two parties, the sender $S$ and the receiver $R$.
The sender has two messages, $m_0$ and $m_1$, and the receiver has a bit $b$.
The receiver wishes to learn $m_b$, without learning anything about $m_{1-b}$ and without the sender learning $b$.

A solution for the above problem has been implemented using the protocol of Even, Goldreich, and Lempel in \cite{even1985randomized}.
Below we describe the otherwise generic protocol using RSA as en encryption scheme.

\begin{itemize}
  \item $S$ (who holds messages $m_0$ and $m_1$) generates an RSA key pair. Recall from section \ref{ss:rsa} that the public key is $(n,e)$ and the private key is $d$.
  \item $S$ randomly selects two values $x_0, x_1$.
  \item $S$ sends $x_0, x_1, N, e$ to $R$.
  \item $R$ picks $b \in \{0,1\}$
  \item $R$ randomly selects a value $k$ and computes $v = x_b + k^e \pmod{n}$.
  \item $R$ sends $v$ to $S$.
  \item $S$ computes $k_0 = (v-x_0)^d \pmod{n}$ and $k_1 = (v-x_1)^d \pmod{n}$. One of $k_0, k_1$ will be equal to $k$ randomly selected be $R$.
  \item $S$ sends to $m_0' = m_0 + k_0$ and $m_1' = m_1 + k_1$ to $R$.
  \item $R$ can compute exactly one of the messages, as $m_b = m_b' - k$.
\end{itemize}

\subsection{Garbled Circuits}\label{ss:garbled-circuits}
There are also generic constructions for building SMPC systems.
Generic protocols implement secure computation for any probabilistic polynomial time function.
One of the most commonly known such generic protocol is Garbled (or Encrypted) Circuits (GC).
The most simple version of SMPC using Garbled Circuits can be found in the two-party case.
The first protocol for such computations was introduced by Yao in \cite{yao1986generate}.

Assume that we have two parties, Alice and Bob, with private inputs $x$ and $y$, respectively.
They wish to compute function $f$ over their private inputs, and nothing more than $f(x,y)$ should be learned.

The GC protocol works by expressing $f$ as an combinatorial circuit.
The circuit contains logical gates that implement any function $g : \{0,1\} \times \{0,1\} \rightarrow \{0,1\}$.
These include for instance simple \texttt{AND, OR, XOR} and \texttt{NOT} gates.
This circuit takes as inputs the bitwise representation of $x$ and $y$ and has output the bitwise representation of the value $f(x,y)$.
The protocol is based on evaluating an encrypted version of this circuit.
A simple description of Yao's protocol can be seen below. A complete description as well as security proof of this protocol can be found in \cite{lindell2009proof}.

Let's assume without loss of generality that Alice is the so called \textit{garbled circuit generator}, or \textit{garbler}, and Bob the \textit{evaluator}. The circuit is known to both parties.

\subsubsection{Circuit Encoding}\label{sss:circuit-encoding}

\begin{itemize}
  \item Alice ``hardwires'' her input into the circuit. Thus the circuit now computes $f(x, \cdot)$.
  \item Alice assigns to each wire $i$ of the circuit two random string values $(W_i^0, W_i^1)$ which are called \textit{labels} that correspond to the Boolean values $0 /$ \texttt{false} and $1 /$ \texttt{true} respectively. These labels should be of adequate length (usually 128 bits) as they will be used as symmetric keys in an encryption scheme.
  \item For each gate $g$ in the circuit Alice prepares the truth table of $g$. In that truth table the values $0,1$ are replaced with the corresponding labels that were generated in the previous step. The output column is then encrypted \footnote{The notation $E_{K}(M)$ means an encryption of the value $M$ with key $K$.} using as keys the labels from the two input columns.
  For example the transformation of the truth table of an \texttt{AND} gate with input wires $A,B$ and output wire $C$ can be seen below.

  \begin{minipage}{0.2\textwidth}
    \begin{center}
      \begin{tabular}{ c c | c }
       $A$ & $B$ & $C$ \\
       \hline
       $0$ & $0$ & $0$ \\
       $0$ & $1$ & $0$ \\
       $1$ & $0$ & $0$ \\
       $1$ & $1$ & $1$ \\
      \end{tabular}
    \end{center}
  \end{minipage}
  $\xrightarrow[]{\text{becomes}}$
  \begin{minipage}{0.2\textwidth}
    \begin{center}
      \begin{tabular}{ c c | c }
       $A$ & $B$ & $C$ \\
       \hline
       $W_A^0$ & $W_B^0$ & $W_C^0$ \\
       $W_A^0$ & $W_B^1$ & $W_C^0$ \\
       $W_A^1$ & $W_B^0$ & $W_C^0$ \\
       $W_A^1$ & $W_B^1$ & $W_C^1$ \\
      \end{tabular}
    \end{center}
  \end{minipage}
  $\xrightarrow[]{\text{becomes}}$
  \begin{minipage}{0.2\textwidth}
    \begin{center}
      \begin{tabular}{ c c | c }
       $A$ & $B$ & $C$ \\
       \hline
       $W_A^0$ & $W_B^0$ & $E_{W_A^0}(E_{W_B^0}(W_C^0))$ \\
       $W_A^0$ & $W_B^1$ & $E_{W_A^0}(E_{W_B^1}(W_C^0))$ \\
       $W_A^1$ & $W_B^0$ & $E_{W_A^1}(E_{W_B^0}(W_C^0))$ \\
       $W_A^1$ & $W_B^1$ & $E_{W_A^1}(E_{W_B^1}(W_C^1))$ \\
      \end{tabular}
    \end{center}
  \end{minipage}

\item As a final step Alice generates a random permutation of the truth table's rows, and the result is a \textit{garbled} table.

\end{itemize}

\subsubsection{Data Transfer}\label{sss:data-transfer}
\begin{itemize}
  \item Alice sends the garbled truth table for every gate $g$ of the circuit to Bob.
  \item Alice also sends the randomly generated labels corresponding to her input.
  For example if Alice's input $a$ is represented by the bits $a_4a_3a_2a_1a_0 = 01101$ then she will send the labels $W_{a_4}^0$, $W_{a_3}^1$, $W_{a_2}^1$, $W_{a_1}^0$, $W_{a_0}^1$ to Bob.
  \item Bob also needs the labels corresponding to his input bits. For example if Bob's input $b$ is represented by the bits $b_4b_3b_2b_1b_0 = 10100$ then he will need the labels $W_{b_4}^1$, $W_{b_3}^0$, $W_{b_2}^1$, $W_{b_1}^0$, $W_{b_0}^0$
  To obtain these labels, Alice and Bob run an 1 out of 2 oblivious transfer protocol, for each bit (each input wire) of Bob's input $b$.
  Using this 1-2 OT protocol Bob learns only the labels corresponding to his input bits, and Alice learns nothing about Bob's input.
  If bob wants to obtain the label for input bit $b_4 = 1$, he will ask Alice between $W_{b_4}^0$ and $W_{b_4}^1$.
  Bob will learn only $W_{b_4}^1$ and not $W_{b_4}^0$, while Alice will not learn the value of $b_4$.
\end{itemize}

\subsubsection{Circuit Evaluation}\label{sss:circuit-evaluation}
\begin{itemize}
  \item Bob now has the garbled truth tables for each gate of the circuit as well as all input labels. Having one garbled value\myslash label per input wire and the truth table, Bob will try to decrypt every value in the output column of the table, but will succeed only once.
  The decryption will succeed only in the row for which Bob has the input labels so he can use them as keys in the decryption.
  The decryption result will be the output label of that gate (the garbled version of the gate's output value).
  This label will be used as an input for the next gate in the combinatorial circuit.
  \item Bob repeats the process for each gate $g$ of the circuit, until he reaches to the output gate(s) and acquires the output label(s).
\end{itemize}

\subsubsection{Output Revealing}\label{ss:output-revealing}
\begin{itemize}
  \item Alice knows The Boolean value corresponding to the output label(s), so Bob and Alice communicate in order for them to learn the computation result.
  \item Either Alice will share her information about the original values of the output label(s) with Bob, or Bob will share the output to Alice and she could respond accordingly so that one or both of them learn the output.
\end{itemize}

\subsubsection{Overhead}\label{ss:overhead}
This protocol inevitably inserts a notable overhead. An overview of the most substantial factors as described in \cite{lindell2009secure} can be found below.

\begin{itemize}
  \item Alice and Bob involve in a 1 out of 2 oblivious transfer protocol for each input wire related with Bob's input. These oblivious transfers can dominate the computation overhead -- at least for relatively small circuits -- since they require modular exponentiations.
  \item Alice sends to Bob a garbled truth table for every gate $g$ of the circuit. Thereby, this operation's cost is proportional to the circuit size.
  \item Bob decrypts a constant number of encrypted values for each gate $g$ of the circuit. This is also linear with respect to the size of the circuit.
\end{itemize}

Multiple attempts have been made to improve the performance of Yao's protocol. Many optimizations have been introduced like the ones found in \cite{beaver1990round, naor1999privacy, kolesnikov2008improved, bellare2013efficient, zahur2015two}.

Another generic protocol for Secure multi\hyp party computation can be constructed using \textit{Secret Sharing}, a technique which we will describe in section \ref{s:secret-sharing}.


\subsection{The reduce operator}
\label{reduce_operator}
Consider the case where we want to perform many different reductions (e.g. min,
max, sum, \ldots) over the same vector. The simple way to do this is
\begin{verbatim}
x = Q.sum(w); y = Q.min(w); z = Q.max(w)
\end{verbatim}
However, this necessitates several scans over the vector \(w\). It is more
efficient to evaluate the vector \(w\) a chunk at a time, perform all the
reductions on the chunk, store the partial results and then repeat over
successive chunks. The reduce operator takes a Vector as input and produces one
or more Scalars as output. It does so, by
(i) creating a Reducer for each of the desired Scalars,
(ii) repeatedly invokes {\tt next()} on them until no more data
(iii) invokes {\tt value()} on each Reducer and returns that.

We try to avoid memo-izing vectors when we don't have to because
of the cost of flushing to disk. In our experiments, for \(n > 2^{25}\),
memo-izing the input \(w\) and computing min/max/sum sequentially
is {\em six} times slower than using the reduce operator.


\documentclass[a4paper, 12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}

\usepackage[]{amsfonts}
\usepackage[]{graphicx}

\title{CS231A Course Notes 4: Stereo Systems and Structure from Motion}
\author{Kenji Hata and Silvio Savarese}
\date{}

\renewcommand\emph{\textbf}

\numberwithin{equation}{section}
\begin{document}

\maketitle

\section{Introduction}
In the previous notes, we covered how adding additional viewpoints of a scene can greatly enhance our knowledge of the said scene. We focused on the epipolar geometry setup in order to relate points of one image plane to points in the other without extracting any information about the 3D scene. In these lecture notes, we will discuss how to recover information about the 3D scene from multiple 2D images.

\section{Triangulation}
One of the most fundamental problems in multiple view geometry is the problem of \emph{triangulation}, the process of determining the location of a 3D point given its projections into two or more images.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/triangulation.png}
\caption{The setup of the triangulation problem when given two views.}
\label{fig:triangulation}
\end{figure}

In the triangulation problem with two views, we have two cameras with known camera intrinsic parameters $K$ and $K'$ respectively. We also know the relative orientations and offsets $R,T$ of these cameras with respect to each other. Suppose that we have a point $P$ in 3D, which can be found in the images of the two cameras at $p$ and $p'$ respectively. Although the location of $P$ is currently unknown, we can measure the exact locations of $p$ and $p'$ in the image. Because $K, K', R, T$ are known, we can compute the two lines of sight $\ell$ and $\ell'$, which are defined by the camera centers $O_1, O_2$ and the image locations $p, p'$. Therefore, $P$ can be computed as the intersection of $\ell$ and $\ell'$.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/real_triangulation.png}
\caption{The triangulation problem in real-world scenarios often involves minimizing the reprojection error.}
\label{fig:real_triangulation}
\end{figure}

Although this process appears both straightforward and mathematically sound, it does not work very well in practice. In the real world, because the observations $p$ and $p'$ are noisy and the camera calibration parameters are not precise, finding the intersection point of $\ell$ and $\ell'$ may be problematic. In most cases, it will not exist at all, as the two lines may never intersect.

\subsection{A linear method for triangulation}
In this section, we describe a simple linear triangulation method that solves the lack of an intersection point between rays. We are given two points in the images that correspond to each other $p = MP = (x,y,1)$ and $p'=M'P = (x', y', 1)$. By the definition of the cross product, $p\times(MP) = 0$. We can explicitly use the equalities generated by the cross product to form three constraints:
\begin{equation}
    \begin{split}
        x(M_3P) - (M_1P) = 0\\
        y(M_3P) - (M_2P) = 0\\
        x(M_2P) - y(M_1P) = 0
    \end{split}
\end{equation}
where $M_i$ is the $i$-th row of the matrix $M$. Similar constraints can be formulated for $p'$ and $M'$. Using the constraints from both images, we can formulate a linear equation of the form $AP = 0$ where
\begin{equation}
    A = \begin{bmatrix}
        xM_3 - M_1 \\
        y M_3 - M_2 \\
        x'M'_3 - M'_1 \\
        y'M'_3 - M'_2
    \end{bmatrix}
\end{equation}
This equation can be solved using SVD to find the best linear estimate of the point $P$. Another interesting aspect of this method is that it can actually handle triangulating from multiple views as well. To do so, one simply appends additional rows to $A$ corresponding to the added constraints by the new views.

This method, however is not suitable for projective reconstruction, as it is not projective-invariant. For example, suppose we replace the camera matrices $M, M'$ with ones affected by a projective transformation $MH^{-1}, M'H^{-1}$. The matrix of linear equations $A$ then becomes $AH^{-1}$. Therefore, a solution $P$ to the previous estimation of $AP=0$ will correspond to a solution $HP$ for the transformed problem $(AH^{-1})(HP) = 0$. Recall that SVD solves for the constraint that $\|P\| = 1$, which is not invariant under a projective transformation $H$. Therefore, this method, although simple, is often not the optimal solution to the triangulation problem.
-
\subsection{A nonlinear method for triangulation}
Instead, the triangulation problem for real-world scenarios is often mathematically characterized as solving a minimization problem:
\begin{equation}
    \min_{\hat{P}} \|M\hat{P}-p\|^2 + \|M'\hat{P}-p'\|^2
    \label{eq:reprojection_error}
\end{equation}
In the above equation, we seek to find a $\hat{P}$ in 3D that best approximates $P$ by finding the best least-squares estimate of the \emph{reprojection error} of $\hat{P}$ in both images. The reprojection error for a 3D point in an image is the distance between the projection of that point in the image  and the corresponding observed point in the image plane. In the case of our example in Figure~\ref{fig:real_triangulation}, since $M$ is the projective transformation from 3D space to image 1, the projected point of $\hat{P}$ in image 1 is $M\hat{P}$. The matching observation of $\hat{P}$ in image 1 is $p$. Thus, the reprojection error for point P in image 1 is the distance $\|M\hat{P} - p\|$. The overall reprojection error found in Equation~\ref{eq:reprojection_error} is the sum of the reprojection errors across all the points in the image. For cases with more than two images, we would simply add more distance terms to the objective function.
\begin{equation}
    \min_{\hat{P}} \sum_i \|M\hat{P_i}-p_i\|^2
    \label{eq:reprojection_error_multi_camera}
\end{equation}

In practice, there exists a variety of very sophisticated optimization techniques that result in good approximations to the problem. However, for the scope of the class, we will focus on only one of these techniques, which is the Gauss-Newton algorithm for nonlinear least squares. The general nonlinear least squares problem is to find an $x\in \mathbb{R}^n$ that minimizes
\begin{equation}
    \|r(x)\|^2 = \sum_{i=1}^m r_i(x)^2
\end{equation}
where $r$ is any residual function $r:\mathbb{R}^n\rightarrow \mathbb{R}^m$ such that $r(x) = f(x) - y$ for some function $f$, input $x$, and observation $y$. The nonlinear least squares problem reduces to the regular, linear least squares problem when the function $f$ is linear. However, recall that, in general, our camera matrices are not affine. Because the projection into the image plane often involves a division by the homogeneous coordinate, the projection into the image is generally nonlinear.

Notice that if we set $e_i$ to be a $2\times1$ vector $e_i = M\hat{P}_i - p_i$, then  we can reformulate our optimization problem to be:
\begin{equation}
    \min_{\hat{P}} \sum_i e_{i}(\hat{P})^2
\end{equation}
which can be perfectly represented as a nonlinear least squares problem.

In these notes, we will cover how we can use the popular Gauss-Newton algorithm to find an approximate solution to this nonlinear least squares problem. First, let us assume that we have a somewhat reasonable estimate of the 3D point $\hat{P}$, which we can compute by the previous linear method. The key insight of the Gauss-Newton algorithm is to update our estimate by correcting it towards an even better estimate that minimizes the reprojection error.  At each step we want to update our estimate $\hat{P}$ by some $\delta_P$: $\hat{P} = \hat{P} + \delta_P$.

But how do we choose the update parameter $\delta_P$? The key insight of the Gauss-Newton algorithm is to linearize the residual function near the current estimate $\hat{P}$. In the case of our problem, this means that the residual error $e$ of a point $P$ can be thought of as:
\begin{equation}
    e(\hat{P} + \delta_P) \approx e(\hat{P}) + \frac{\partial e}{\partial P}\delta_P
\end{equation}
Subsequently, the minimization problem transforms into
\begin{equation}
    \min_{\delta_P} \| \frac{\partial e}{\partial P}\delta_P - (-e(\hat{P})) \|^2
\end{equation}
When we formulate the residual like this, we can see that it takes the format of the standard linear least squares problem. For the triangulation problem with $N$ images, the linear least squares solution is
\begin{equation}
    \delta_P = -(J^TJ)^{-1} J^Te
\end{equation}
where
\begin{equation}
    e = \begin{bmatrix} e_1 \\ \vdots \\ e_N\end{bmatrix} = \begin{bmatrix}p_1 - M_1\hat{P} \\ \vdots \\ p_n - M_n \hat{P} \end{bmatrix}
\end{equation}
and
\begin{equation}
    J = \begin{bmatrix}
    \dfrac{\partial e_1}{\partial \hat{P}_1} & \dfrac{\partial e_1}{\partial \hat{P}_2}& \dfrac{\partial e_1}{\partial \hat{P}_3}\\
    \vdots & \vdots & \vdots\\
    \dfrac{\partial e_N}{\partial \hat{P}_1} & \dfrac{\partial e_N}{\partial \hat{P}_2} & \dfrac{\partial e_N}{\partial \hat{P}_3} \end{bmatrix}
\end{equation}

 Recall that the residual error vector of a particular image $e_i$ is a $2\times 1$ vector because there are two dimensions in the image plane. Consequently, in the simplest two camera case ($N=2$) of triangulation, this results in the residual vector $e$ being a $2N\times1 = 4\times1$ vector and the Jacobian $J$ being a $2N\times3 =4 \times 3$ matrix. Notice how this method handles multiple views seamlessly, as additional images are accounted for by adding the corresponding rows to the $e$ vector and $J$ matrix. After computing the update $\delta_P$, we can simply repeat the process for a fixed number of steps or until it numerically converges. One important property of the Gauss-Newton algorithm is that our assumption that the residual function is linear near our estimate gives us no guarantee of convergence. Thus, it is always useful in practice to put an upper bound on the number of updates made to the estimate.

\section{Affine structure from motion}
At the end of the previous section, we hinted how we can go beyond two views of a scene to gain information about the 3D scene. We will now explore the extension of the geometry of two cameras to multiple cameras. By combining observations of points from multiple views, we will be able to simultaneously determine both the 3D structure of the scene and the parameters of the camera in what is known as \emph{structure from motion}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/sfm_setup.png}
\caption{The setup of the general structure from motion problem.}
\label{fig:sfm_setup}
\end{figure}

Here, we formally introduce the structure from motion problem. Suppose we have $m$ cameras with camera transformations $M_i$ encoding both the intrinsic and extrinsic parameters for the cameras. Let $X_j$ be one of the $n$ 3D points in the scene. Each 3D point may be visible in multiple cameras at the location $x_{ij}$, which is the projection of $X_j$  to the image of the camera $i$ using the projective transformation $M_i$. The aim of structure from motion is to recover both the structure of the scene (the $n$ 3D points $X_j$) and the motion of the cameras (the $m$ projection matrices $M_i$) from all the observations $x_{ij}$.

\subsection{The affine structure from motion problem}
Before tackling the general structure from motion problem, we will first start with a simpler problem, which assumes the cameras are affine or weak perspective. Ultimately, the lack of the perspective scaling operation makes the mathematical derivation easier for this problem.

Previously, we derived the above equations for perspective and weak perspective cases. Remember that in the full perspective model, the camera matrix is defined as
\begin{equation}
M = \begin{bmatrix}
    A & b \\ v & 1
\end{bmatrix}
\end{equation}
where $v$ is some non-zero $1\times3$ vector. On the other hand, for the weak perspective model, $v=0$. We find that this property makes the homogeneous coordinate of $MX$ equal to $1$:
\begin{equation}
    x = MX = \begin{bmatrix}m_1 \\ m_2 \\ 0 ~~ 0 ~~ 0 ~~ 1\end{bmatrix} \begin{bmatrix}X_1\\X_2\\X_3 \\ 1\end{bmatrix} = \begin{bmatrix}m_1X \\ m_2 X \\ 1\end{bmatrix}
\end{equation}

Consequently, the nonlinearity of the projective transformation disappears as we move from homogeneous to Euclidean coordinates, and the weak perspective transformation acts as a mere magnifier. We can more compactly represent the projection as:
\begin{equation}
    \begin{bmatrix} m_1X \\ m_2X \end{bmatrix} = \begin{bmatrix}A & b \end{bmatrix} X = AX + b
\end{equation}
and represent any camera matrix in the format $M_\mathrm{affine} = \begin{bmatrix}A &b \end{bmatrix}$. Thus, we now use the affine camera model to express the relationship from a point $X_j$  in 3D and the corresponding observations in each affine camera (for instance, $x_{ij}$ in camera $i$).

Returning to the structure from motion problem, we need to estimate $m$ matrices $M_i$, and the $n$ world coordinate vectors $X_j$, for a total of $8m+3n$ unknowns, from $mn$ observations. Each observation creates 2 constraints per camera, so there are $2mn$ equations in $8m+3n$ unknowns. We can use this equation to know the lower bound on the number of corresponding observations in each of the images that we need to have. For example, if we have $m=2$ cameras, then we need to have at least $n=16$ points in 3D. However, once we do have enough corresponding points labeled in each image, how do we solve this problem?

\subsection{The Tomasi and Kanade factorization method}
In this part, we outline Tomasi and Kanade’s \emph{factorization method} for solving the affine structure from motion problem. This method consists of two major steps: the data centering step and the actual factorization step.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/factorization_centering.png}
\caption{When applying the centering step, we translate all of the image points such that their centroid (denoted as the lower left red cross) is located at the origin in the image plane. Similarly, we place the world coordinate system such that the origin is at the centroid of the 3D points (denoted as the upper right red cross).}
\label{fig:factorization_centering}
\end{figure}

Let’s begin with the data centering step. In this step, the main idea is center the data at the origin. To do so, for each image $i$, we redefine new coordinates $\hat{x}_{ij}$ for each image point $x_{ij}$ by subtracting out their centroid $\bar{x}_i$:
\begin{equation}
    \hat{x}_{ij} = x_{ij} - \bar{x}_{i} = x_{ij} - \frac{1}{n}\sum_{j=1}^{n}{x_{ij}}
    \label{eq:mean}
\end{equation}
Recall that the affine structure from motion problem allows us to define the relationship between image points $x_{ij}$, the camera matrix variables $A_i$ and $b_i$, and the 3D points $X_j$ as:
\begin{equation}
    x_{ij}= A_i X_j + b_i
    \label{eq:affine_relationship}
\end{equation}
After this centering step, we can combine definition of the centered image points $\hat{x}_{ij}$ in Equation~\ref{eq:mean} and the affine expression in Equation~\ref{eq:affine_relationship}:
\begin{equation}
\begin{split}
    \hat{x}_{ij} &= x_{ij} -  \frac{1}{n}\sum_{k=1}^{n}{x_{ik}}\\
    & = A_iX_j - \frac{1}{n}\sum_{k=1}^{n}{A_i X_k}\\
    & = A_i(X_j - \frac{1}{n}\sum_{k=1}^{n} X_k) \\
    & = A_i(X_j - \bar{X})\\
    & = A_i\hat{X}_j
\end{split}
\label{eq:centered_relationship}
\end{equation}

As we see from Equation~\ref{eq:centered_relationship}, if we translate the origin of the world reference system to the centroid $\bar{X}$, then the centered coordinates of the image points $\hat{x}_{ij}$ and centered coordinates of the 3D points $\hat{X}_{ij}$ are related only by a single $2\times 3$ matrix $A_i$. Ultimately, the centering step of the factorization method allows us to create a compact matrix product representation to relate the 3D structure with their observed points in multiple images.

However, notice that in the matrix product $\hat{x}_{ij} = A_i\hat{X_j}$, we only have access to the values on the left hand side of the equation. Thus, we must somehow factor out the motion matrices $A_i$ and structure $X_j$. Using all the observations for all the cameras, we can build a measurement matrix $D$, made up of $n$ observations in the $m$ cameras (remember that each $\hat{x}_{ij}$ entry is a 2x1 vector):
\begin{equation}
    D = \begin{bmatrix}
        \hat{x}_{11} & \hat{x}_{12} & \hdots & \hat{x}_{1n}\\
        \hat{x}_{2  1} & \hat{x}_{22} & \hdots & \hat{x}_{2n}\\
        & & \ddots & \\
        \hat{x}_{m1} & \hat{x}_{m2} & \hdots & \hat{x}_{mn}\\
    \end{bmatrix}
\end{equation}

Now recall that because of our affine assumption, $D$ can be expressed as the product of the $2m\times3$ motion matrix $M$ (which comprises the camera matrices $A_1, \ldots A_m$) and the $3\times n$ structure matrix $S$ (which comprises the 3D points $X_1, \ldots X_n$). An important fact that we will use is that $\mathrm{rank}(D) = 3$ since $D$ is the product of two matrices whose max dimension is 3.

To factorize $D$ into $M$ and $S$, we will use the singular value decomposition, $D = U \Sigma V^T$. Since we know the $\mathrm{rank}(D) = 3$, so there will only be 3 non-zero singular values $\sigma_1 , \sigma_2$, and $\sigma_3$ in $\Sigma$. Thus, we can further reduce the expression and obtain the following decomposition:
\begin{equation}
    \begin{split}
        D &= U \Sigma V^T\\
        &= \begin{bmatrix} u_1 & \hdots & u_n \end{bmatrix}
        \begin{bmatrix}
            \sigma_1 & 0 & 0 & 0 & \hdots & 0\\
            0 & \sigma_2 & 0 & 0 & \hdots & 0\\
            0 & 0 & \sigma_3 & 0 & \hdots & 0\\
            0 & 0 & 0 & 0 & \hdots & 0\\
             &  &  & & \ddots & \\
            0 & 0 & 0 & 0 & \hdots & 0\\
        \end{bmatrix}
        \begin{bmatrix} v_1^T \\ \vdots \\ v_n^T \end{bmatrix}\\
        & = \begin{bmatrix} u_1 & u_2 & u_3 \end{bmatrix}
        \begin{bmatrix}
            \sigma_1 & 0 & 0\\
            0 & \sigma_2 & 0\\
            0 & 0 & \sigma_3
        \end{bmatrix}
        \begin{bmatrix} v_1^T \\ v_2^T \\ v_3^T \end{bmatrix}\\
        & = U_3 \Sigma_3 V^T_3
    \end{split}
    \label{eq:svd_factorization}
\end{equation}

In this decomposition, $\Sigma_3$ is defined as the diagonal matrix formed by the non-zero singular values, while $U_3$ and $V^T_3$ are obtained by taking the corresponding three columns of $U$ and rows of $V^T$ respectively. Unfortunately, in practice, $\mathrm{rank}(D) > 3$ because of measurement noise and the affine camera approximation. However, recall that when $\mathrm{rank}(D) > 3$, $U_3 W_3 V_3^T$ is still the best possible rank-3 approximation of $MS$ in the sense of the Frobenius norm.

Upon close inspection, we see that the matrix product $\Sigma_3V_3^T$ forms a $3\times n$ matrix, which exactly the same size as the structure matrix $S$. Similarly, $U_3$ is a $2m\times 3$ matrix, which is the same size as the motion matrix $M$. While this way of associating the components of the SVD decomposition to $M$ and $S$ leads to a physically and geometrical plausible solution of the affine structure from motion problem, this choice is not a unique solution. For example, we could also set the motion matrix to $M = U_3\Sigma_3$ and the structure matrix to $S = V_3^T$, since in either cases the observation matrix $D$ is the same. So what factorization do we choose? In their paper, Tomasi and Kanade concluded that a robust choice of the factorization is $M = U_3\sqrt{\Sigma_3}$ and $S = \sqrt{\Sigma_3}V_3^T$.

\subsection{Ambiguity in reconstruction}
Nevertheless, we find inherent ambiguity in any choice of the factorization $D=MS$, as any arbitrary, invertible $3 \times 3$ matrix $A$ may be inserted into the decomposition:
\begin{equation}
    D = MAA^{-1}S = (MA)(A^{-1}S)
\end{equation}
This means that the camera matrices obtained from motion $M$ and the 3D points obtained from structure $S$ are determined up to a multiplication by a common matrix $A$. Therefore, our solution is underdetermined, and requires extra constraints to resolve this affine ambiguity. When a reconstruction has affine ambiguity, it means that parallelism is preserved, but the metric scale is unknown.

Another important class of ambiguities for reconstruction is the similarity ambiguity, which occurs when a reconstruction is correct up to a similarity transform (rotation, translation and  scaling). A reconstruction with only similarity ambiguity is known as a metric reconstruction. This ambiguity exists even when the camera are intrinsically calibrated.  The good news is that for calibrated cameras, the similarity ambiguity is the only ambiguity\footnote{See [Longuet-Higgins ’81] for more details.}.

The fact that there is no way to recover the absolute scale of a scene from images is fairly intuitive. An object's scale, absolute position and canonical orientation will always be unknown unless we make further assumptions (e.g, we know the height of the house in the figure) or incorporate more data. This is because some attributes may compensate for others. For instance, to get the same image, we can simply move the object backwards and scale it accordingly. One such example of removing similarity ambiguity occurred during the camera calibration procedure, where we made the assumption that we know the location of the calibration points with respect to the world reference system. This enabled us to know the size of the squares of the checkerboard to learn a metric scale of the 3D structure.

\section{Perspective structure from motion}
After studying the simplified affine structure from motion problem, let us now consider the general case for projective cameras $M_i$. In the general case with projective cameras, each camera matrix $M_i$ contains 11 degrees of freedom, as it is defined up to scale:
\begin{equation}
M_i=
\begin{bmatrix}
    a_{11}       & a_{12} & a_{13} &  b_{1} \\
    a_{21}       & a_{22} & a_{23} &  b_{2} \\
    a_{31}       & a_{32} & a_{33} &  1
\end{bmatrix}
\end{equation}

Moreover, similar to the affine case where the solution can be found up to an affine transformation, solutions for structure and motion can be determined up a projective transformation in the general case: we can always arbitrarily apply a $4\times 4$ projective transformation $H$ to the motion matrix, as long as we also transform the structure matrix by the inverse transformation $H^{-1}$. The resulting observations in the image plane will still be the same.

Similar to the affine case, we can set up the general structure from motion problem as estimating both the $m$ motion matrices $M_i$ and $n$ 3D points $X_j$ from $mn$ observations $x_{ij}$. Because cameras and points can only be recovered up to a $4\times 4$ projective transformation up to scale (15 parameters), we have $11m+3n-15$ unknowns in $2mn$ equations. From these facts, we can determine the number of views and observations that are required to solve for the unknowns.

\subsection{The algebraic approach}

\begin{figure}[h!]
\centering
\includegraphics[width=0.5\textwidth]{figures/algebraic_setup.png}
\caption{In the algebraic approach, we consider sequential, camera pairs to determine camera matrices $M_1$ and $M_2$ up to a perspective transformation. We then find a perspective transformation $H$ such that $M_1H = [I~~0]$ and $M_2H = [A~~B]$}.
\label{fig:algebraic_setup}
\end{figure}

We will now cover the \emph{algebraic approach}, which leverages the concept of fundamental matrix $F$ for solving the structure from motion problem for two cameras. As shown in Figure~\ref{fig:algebraic_setup}, the main idea of the algebraic approach is to compute two camera matrices $M_1$ and $M_2$, which can only be computed up to a perspective transformation $H$. Since each $M_i$ can only be computed up a perspective transformation $H$, we can always consider a $H$ such that the first camera projection matrix $M_1 H^{-1}$ is canonical. Of course, the same transformation must also be applied to the second camera which lead to the form shown:

\begin{equation}
    M_1 H^{-1} = [I ~~ 0] ~~~~~~~ M_2 H^{-1} = [A ~~ b]
    \label{eq:mh}
\end{equation}

In order to accomplish this task, we must first compute the fundamental matrix $F$ using the eight point algorithm covered in the previous course notes. We now will use $F$ to estimate the projective camera matrices $M_1$ and $M_2$. In order to do this estimation, we define $P$ to be the corresponding 3D point for the corresponding observations in the images $p$ and $p'$. Since we have applied $H^{-1}$ to both camera projection matrices, we must also apply $H$ to the structure, giving us $\widetilde{P} = HP$. Therefore, we can relate the pixel coordinates $p$ and $p'$ to the transformed structure as follows:

\begin{equation}
    \begin{split}
        p = M_1 P = M_1 H^{-1} H P = [I ~ | ~ 0] \widetilde{P}\\
        p' = M_2 P = M_2 H^{-1} H P = [A ~ | ~ b] \widetilde{P}
    \end{split}
    \label{eq:x}
\end{equation}
An interesting property between the two image correspondences $p$ and $p'$ occur by some creative substitutions:

\begin{equation}
    \begin{split}
        p'&= [A|b] \widetilde{P}\\
        &= A[I|0] \widetilde{P} +b \\
        &= Ap+b
    \end{split}
    \label{eq:xpx}
\end{equation}
Using Equation~\ref{eq:xpx}, we can write the cross product between $p'$ and $b$ as:

\begin{equation}
    p' \times b = (Ap+b) \times b = Ap \times b
    \label{eq:xptimesb}
\end{equation}
By the definition of cross product, $p' \times b$ is perpendicular to $p'$. Therefore, we can write:

\begin{equation}
    \begin{split}
        0 &= p'^T  ( p' \times b )  \\
        &= p'^T  (Ap \times b )\\
        &= p'^T \cdot (b \times Ap) \\
        & = p'^T[b]_\times Ap
    \end{split}
    \label{eq:xpdot1}
\end{equation}
Looking at this constraint, it should remind you of the general definition of the Fundamental matrix $p'^T Fp = 0$. If we set $F=[b]_\times A$, then extracting $A$ and $b$ simply breaks down to a decomposition problem.

Let us begin by determining $b$. Again, by the definition of cross product, we can simply write $Fb$ as
\begin{equation}
    Fb = [b]_\times A b = (b\times A)b = 0
    \label{eq:cross_b}
\end{equation}.
Since $F$ is singular, $b$ can be computed as a least square solution of $F b = 0$, with $\|b\|=1$, using SVD.

Once $b$ is known, we can now compute $A$. If we set $A=-[b]_\times F$, then we can verify that this definition satisfies $F = [b]_\times A$:
\begin{equation}
    \begin{split}
        [b_\times]A' &= -[b_\times][b_\times]F \\
        & =(bb^T - |b|^2I)F \\
        &= bb^TF + |b|^2F \\
        &= 0 + 1\cdot F \\
        &=F
    \end{split}
    \label{eq:bap}
\end{equation}
Consequently, we determine the two expressions for our camera matrices $M_1H^{-1}$ and $M_2H^{-1}$:
\begin{equation}
    \tilde{M}_1 = [I ~~ 0] ~~~~~~~~ \tilde{M}_2 = [- [b_\times]F ~~ b]
    \label{eq:mm}
\end{equation}

Before we conclude this section, we want give a geometrical interpretation for $b$. We know $b$ satisfies $Fb=0$. Remember the epipolar constraints we derived in the previous course notes, which found that the epipoles in an image are the points that map to zero when transformed by the Fundamental matrix (i.e. $Fe_2=0$ and $F^T e_1=0$). We can see, therefore, that $b$ is an epipole. This provides a new set of equations for the camera projection matrices (Eqs. \ref{eq:mm2}).

\begin{equation}
    \tilde{M}_1 = [I ~~ 0] ~~~~~~~~ \tilde{M}_2 = [- [e_\times]F ~~ e]
    \label{eq:mm2}
\end{equation}

\subsection{Determining motion from the Essential matrix}
One useful way of improving the reconstruction obtained by the algebraic approach is to use calibrated cameras. We can extract a more accurate, initial estimate of camera matrices by using the Essential matrix, which is a special case of the Fundamental matrix for normalized coordinates. Recall that, by using the Essential matrix $E$, we make an assumption that we have calibrated the camera and thus know the intrinsic camera matrix $K$. We can either compute the Essential matrix $E$ either from the normalized image coordinates directly or from its relationship with the Fundamental matrix $F$ and intrinsic matrix $K$:

\begin{equation}
    E = K^TFK
\end{equation}

Because the Essential matrix assumes that we have calibrated cameras, we should remember that it only has five degrees of freedom, as it only encodes the extrinsic parameters: the rotation $R$ and translation $t$ between the cameras. Luckily, this is exactly the information that we want to extract to create our motion matrix. First, recall that the Essential matrix $E$ can be represented as
\begin{equation}
    E = [t]_\times R
\end{equation}
As such, perhaps we can find a strategy to factor $E$ into its two components. First, we should notice that the cross product matrix $[t]_\times$ is skew-symmetric. We define two matrices that we will use in the decomposition:
\begin{equation}
     W = \begin{bmatrix} 0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1\end{bmatrix}, ~~~ Z = \begin{bmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 0\end{bmatrix}
\end{equation}
One important property we will use later is that $Z = \mathrm{diag}(1,1,0)W$ up to a sign. Similarly, we will also use the fact that $ZW = ZW^T = \mathrm{diag}(1,1,0)$ up to a sign.

As a result of eigenvalue decomposition, we can create a block decomposition of a general skew-symmetric matrix known up to scale. Thus, we can write $[t]_\times$ as
\begin{equation}
    [t]_\times  = UZU^T
\end{equation}
where $U$ is some orthogonal matrix. Therefore, we can rewrite the decomposition as:
\begin{equation}
    E = U\mathrm{diag}(1,1,0)  (WU^TR)
\end{equation}

Looking at this expression carefully, we see that it closely resembles the singular value decomposition $E = U\Sigma V^T$, where $\Sigma$ contains two equal singular values. If we know $E$ up to scale and we assume that it takes the form $E = U \mathrm{diag}(1,1,0)V^T$, then we arrive at the following factorizations of $E$:
\begin{equation}
    [t]_\times  = UZU^T, ~~~~ R= UWV^T ~ \mathrm{or} ~ UW^TV^T
\end{equation}
We can prove that the given factorizations are valid by inspection. We can also prove that there are no other factorizations. The form of $[t]_\times$ is determined by the fact that its left null space must be the same as the null space of $E$. Given unitary matrices $U$ and $V$, any rotation $R$ can be decomposed into $UXV^T$ where $X$ is another rotation matrix. After substituting these values in, we get $ZX = \mathrm{diag}(1,1,0)$ up to scale. Thus, $X$ must be equal to $W$ or $W^T$.

Note that this factorization of $E$ only guarantees that the matrices $UWV^T$ or $UW^TV^T$ is orthogonal. To ensure that $R$ is a valid rotations, we simply make sure that the determinant of $R$ is positive:
\begin{equation}
    R= (\det UWV^T)UWV^T ~ \mathrm{or} ~ (\det UW^TV^T)UW^TV^T
\end{equation}

Similar to how the rotation $R$ can take on two potential values, the translation vector $t$ can also take on several values. From the definition of cross product, we know that
\begin{equation}
    t \times t = [t]_\times t = UZU^T t = 0
\end{equation}
Knowing that $U$ is unitary, we can find that the $\|[t]_\times\|F = \sqrt{2}$. Therefore, our estimate of $t$ from this factorization will come from the above equation and the fact that $E$ is known up to scale. This means that
\begin{equation}
    t = \pm U\begin{bmatrix}0\\0\\1\end{bmatrix} = \pm u_3
\end{equation}
where $u_3$ is the third column of $U$. By inspection, we can also verify that we get the same results by reformatting $[t]_\times = UZU^T$ into the vector $t$ known up to a sign.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/four_rt.png}
\caption{There are four possible solutions for extracting the relative camera rotation $R$ and translation $t$ from the Essential matrix. However, only in (a) is the reconstructed point in front of both of the cameras. (Figure taken from Hartley and Zisserman textbook page 260)}
\label{fig:four_rt}
\end{figure}

As illustrated in Figure~\ref{fig:four_rt}, there are four potential $R,t$ pairings since there exists two options for both $R$ and $t$. Intuitively, the four pairings include all possible pairings of rotating a camera in a certain direction or rotating the camera in the opposite direction combined with the option of translating it in a certain direction or the opposite direction. Therefore, under ideal conditions, we would only need to triangulate one point to determine the correct $R,t$ pair. For the correct $R,t$ pair, the triangulated point $\hat{P}$ exists in front of both cameras, which means that it has a positive $z$-coordinate with respect to both camera reference systems. Due to measurement noise, we often do not rely on triangulating only one point, but will instead triangulate many points and determine the correct $R,t$ pair as the one that contains the most of these points in front of both cameras.

\section{An example structure from motion pipeline}
After finding the relative motion matrices $M_i$, we can use them to determine the world coordinates of the points $X_j$. In the case of the algebraic method, the estimate of such points will be correct up to the perspective transformation $H$. In extracting the camera matrices from the Essential matrix, the estimates can be known up to scale. In both cases, the 3D points can be computed from the estimated camera matrices via the triangulation methods described earlier.

The extension to the multi-view case can be done by chaining pairwise cameras. We can use the algebraic approach or the Essential matrix to obtain solutions for the camera matrices and the 3D points for any pair of cameras, provided that there are enough point correspondences. The reconstructed 3D points are associated to the point correspondences available between the camera pair. Those pairwise solutions may be combined together (optimized) in a approach called bundle adjustment as we will see next.

\subsection{Bundle adjustment}
There are major limitations related to the previous methods for solving the structure from motion problem that we have discussed so far. The factorization method assumes that all points are visible in every image. This is very unlikely to happen because of occlusions and failures to find correspondences when we either have many images or some of the images were taken far apart. Finally the algebraic approach produces pairwise solutions that can be combined into a camera chain, but does not solve for a coherent optimized reconstruction using all the cameras and 3D points.

To address these limitations, we introduce \emph{bundle adjustment}, which is a nonlinear method for solving the structure from motion problem. In the optimization, we aim to minimize the reprojection error, which is the pixel distance between the projection of a reconstructed point into the estimated cameras and its corresponding observations for all the cameras and for all the points. Previously, when discussing nonlinear optimization methods for triangulation, we focused primarily on the two camera case, in which we naturally assumed that each camera saw all the correspondences between the two. However, since bundle adjustment handles several cameras, it only calculates the reprojection error for only the observations that can be seen by each camera. Ultimately though, this optimization problem is very similar to the one we introduced when talking about nonlinear methods for triangulation.

Two common approaches for solving bundle adjustment's nonlinear optimization include the Gauss-Newton algorithm and the Levenberg-Marquardt algorithm. You can refer to the previous section about details on the Gauss-Newton algorithm and refer to the Hartley and Zisserman textbook for more details on the Levenberg-Marquardt algorithm.

In conclusion, bundle adjustment has some important advantages and limitations when compared to the other methods we have surveyed. It is particularly useful because it can handle a large number of views smoothly and also handle cases when particular points are not observable by every image. However, the main limitation is that it is a particularly large minimization problem, as the parameters grow with the number of views. Additionally, it requires a good initial condition since it relies on nonlinear optimization techniques. For this reason, bundle adjustment is often used as final step of most structure from motion implementations (i.e., after the factorization or algebraic approach), as a factorization or algebraic approach may provide a good initial solution for the optimization problem.
\end{document}


\subsection{Comparison of Mechanisms}
In the previous section, we studied several different algorithmic approaches that guarantee different optimality criteria. To better evaluate the algorithms against the optimality criteria we defined in Section \ref{sec:optimality}, we will now summarize and compare the properties of the algorithms. Afterwards, we will look at practical results that were obtained by performing experiments of matching mechanisms with one- and two-sided preferences by Diebold and Bichler \cite{DieboldBenchmark}.

\subsubsection{Theoretical Results}
Referring back to the list of desirable properties defined in Section \ref{criteria-application}, let us now recap and compare the aforementioned algorithms to evaluate which one could be applicable for the problem of matching students to seminars. Unfortunately, none of the algorithms guarantee all of the optimality criteria at the same time, which makes the choice of an algorithm non-trivial. Table \ref{tab:algorithm-comparison} gives an overview of the presented algorithms and their properties. Each of the algorithms mentioned in this section is listed, and for each optimality criteria, a yes/no encoding is used to make a statement about which properties an algorithm guarantees. It is important to note here that a "no" in a column does not strictly mean that the given optimality criteria cannot be fulfilled by the algorithm, but rather that the algorithm does not guarantee it. For instance, a matching computed with the greedy algorithm can be of maximum cardinality or be popular. Only the results for strategy-proofness are a strict yes or no, since fulfilling strategy-proofness does not depend on the instance of the problem, but only of the mechanism being used.

\begin{table}[h!]
    \begin{tabular}{l|llll}
    \hline
                        & RSD    & Max-PaCHA    & Assignment & Popular-CHA           \\ \hline
    Maximum Cardinality & no     & yes          & yes        & yes               \\
    Pareto-Optimal      & yes    & yes          & yes        & yes               \\
    Popular             & no     & no           & no         & yes               \\
    Rank Maximal        & no     & no           & yes        & no                \\
    Always Exists       & yes    & yes          & yes        & no                \\
    Strategy Proof      & yes    & no           & no         & yes               \\ \hline
    Time Complexity     & $\mathcal{O}(n)$   & $\mathcal{O}(\sqrt{n} * m)$ & $\approx\mathcal{O}(n^3)$    & $\mathcal{O}(\sqrt{C} * n_1 + m)$ \\ \hline
    \end{tabular}
    \caption{Comparison of Different Algorithmic Approaches}
    \label{tab:algorithm-comparison}
\end{table}

To summarize the results, we can see that all of the algorithms guarantee pareto-optimality, however only Popular-CHA guarantees popularity. At the same time, only RSD and Popular-CHA also guarantee strategy-proofness, which makes Popular-CHA particularly interesting for the student-seminar problem.

\paragraph{Strategy-proofness and Maximum Cardinality:}
One interesting observation is that fulfilling maximum cardinality comes at the cost of either not being strategy-proof, or not guaranteeing that a matching exists at all. Indeed, only RSD and Popular-CHA algorithm guarantee strategy-proofness. However, ensuring strategy-proofness and maximum cardinality at the same time comes at the cost of not always finding a matching (Popular CHA). If we look back at the Popular-CHA algorithm, we remember that a maximum cardinality matching $M'$ is computed on the reduced graph $G'$. We saw that a maximum popular matching does not exist, iff the matching is not agent-complete, meaning that one of the agents is matched to their last-resort house. While this mechanism ensures strategy-proofness, it is also not always possible to find such a maximum cardinality matching using the Popular-CHA algorithm. Therefore, it remains an open question whether or not a mechanism exists that both is strategy-proof and always produces maximum-cardinality matchings.

\paragraph{Max-PaCHA and the Assignment Problem:}
Another important thing to notice is the similarity of the properties between Max-PaCHA and assignment problem algorithm. Except for the fact that the assignment algorithm guarantees rank maximality, the two algorithms produce matchings with very similar characteristics, which then begs the question of why one should use the Max-PaCHA algorithm. But looking at the runtime complexity of the algorithms, we see that, while both algorithms run in polynomial time, the assignment problem takes longer to be solved.

\subsubsection{Practical Results in the Literature}\label{sec:practical-results-lit}
Diebold et al. have published results for extensive experiments on matching mechanisms with both one- and two-sided preferences \cite{DieboldBenchmark}. They used real course registration data from TUM for investigating properties, including size, rank and popularity, of matchings produced by several mechanisms. The mechanisms are the same as described in Section \ref{chapter:algorithms}, with one exception being that the \emph{ProB-CHAT} algorithm \cite{DieboldBenchmark} is used in place of the Hungarian algorithm for finding rank-maximal matchings.

The authors found that all algorithms' matchings achieve an average cardinality of at least 97.48\%. For one of the 9 instances, Popular-CHA failed to find a matching, but when it found one, the matchings' average ranks of 1.33 got close to the ones produced by ProB-CHAT of 1.26. Unsurprisingly, ProB-CHAT performed best on the rank metrics and also produced more popular matchings than all other algorithms except for Popular-CHA. However, its max runtime was the worst with 33.852s, compared to the second highest 2.458s of Popular-CHA on a dataset with 915 students and 51 courses.

Another surprising finding is that RSD performed better on rank metrics with an average rank of 1.41 than Max-PaCHA with an average rank of 1.51. Generally, the average ranks of all algorithms are somewhat close by being in a range of 1.26 (ProB-CHAT) to 1.51 (Max-PaCHA). Unfortunately, the authors do not disclose more detailed information about their datasets and structure of matchings.

While these results confirm some of the theoretical observations, it will be interesting to get more insights with differently structured datasets being used. Additionally, most of the data contains ties and we do not get any meaningful insights on the distribution of preferences.