The grey value of each pixel in the detector can be modelled as a random variable, due to the random behaviour of photons being produced, interacting with the test sample and the scintillator in the detector. By modelling using a random variable, the uncertainty can be quantified and considered when conducting inference about any detected defects.

The compound Poisson distribution is studied here because of the compound Poission-like behaviour from the detection of photons \citep{whiting2002signal, elbakri2003efficient, whiting2006properties}. It is defined by defining a latent variable $Y\sim\poisson(\lambda)$ with probability mass function (p.m.f.) $\prob(Y=y)=\euler^{-\lambda}\frac{\lambda^y}{y!}$ for $y=0,1,2,\ldots$, where $\lambda>0$ is the Poisson rate parameter. Let $U_i$ be some independent and identically distributed (i.i.d.) latent random variables with probability density function (p.d.f.) $p_U(u)$ for $i=1,2,3,\ldots$. Let $X$ be a compound Poisson random variable where
\begin{equation}
  X|Y = \sum_{i=1}^{Y}U_i \ .
  \label{eq:compoundPoisson_X|Y}
\end{equation}
The p.d.f.~of $X$ can be obtained by marginalising the joint p.d.f.
\begin{equation}
  p_X(x)=\sum_{y=0}^\infty p_{X|Y}(x|y)\prob(Y=y) \quad\text{for }x\geqslant 0
  \ .
\end{equation}
It should be noted that $X=0$ if and only if $Y=0$ and this happens with probability $\prob(Y=0)=\euler^{-\lambda}$. This implies that $X$ has probability mass at zero and probability density for positive numbers which results in the p.d.f.
\begin{equation}
  p_X(x) =
  \begin{cases}
    \delta(x) \euler^{-\lambda}  & \text{ for } x=0 \\
    \sum_{y=1}^\infty p_{X|Y}(x|y)\euler^{-\lambda}\frac{\lambda^y}{y!} \quad\text{for } & \text{ for } x>0
  \end{cases}
\end{equation}
where $\delta(x)$ is the Dirac delta function.

The compound Poisson distribution has applications in, for example, modelling rainfall \citep{revfeim1984initial} and insurance claims \citep{jorgensen1994fitting, smyth2002fitting}.

This chapter starts with a literature review on the compound Poisson distribution, how it is derived from the behaviour of photons, how its likelihood is evaluated and methods for fitting it onto data. A model was proposed for the grey values and the expectation-maximisation (EM) algorithm was implemented to fit the model onto data. It was found that for high photon rates, there were identifiability issues. The chapter is concluded on a discussion on why the EM algorithm failed.

\section{Literature Review}

\subsection{Compound Poisson in X-ray Detection}

In an x-ray tube, photons are emitted as a Poisson process \citep{whiting2006properties, cierniak2011x} and each photon has some random energy due to bremsstrahlung and characteristic radiation \citep{sun2012overview}. This is similar to the compound Poisson distribution. Let $Y$ be the number of photons emitted for some time exposure $\tau$, then $Y\sim\poisson(\lambda)$. Each photon is assumed to be i.i.d.~with random energy $U_i$ for $i=1,2,3,\ldots$ with p.d.f.~$p_U(u)$. The random variables discussed here cover all the latent variables in the compound Poisson.

Photons emitted from the x-ray tube undergo attenuation when propagating through the test sample. Assuming no beam hardening, some photons are either absorbed or scattered, making them undetectable. Scattered photons may be detected but it is very rare \citep{cantatore2011introduction}. The energy of the attenuated photons remain unaffected so attenuation decreases the parameter $\lambda$. The amount it decreases by depends on the attenuation coefficient of the material and the amount of material the x-ray attenuates. The density of the random variable $U_i$ remains unchanged because the energy of each photon remains the same after attenuation, assuming no beam hardening.

When the photons interact with the scintillator in the detector, they are converted into visible light. The visible light photons are then detected and converted into a digital signal or grey value. A quantum counter set the digital signal to be linear with the number of photons detected \citep{whiting2006properties}. Let $X$ be the grey value observed, then
\begin{equation}
X = bY + \epsilon
\end{equation}
where $\epsilon\sim\normal(a,\kappa)$, $b$ and $a$ are some constant and $\kappa$ is the variance of electronic noise. The mean and variance of the grey value are
\begin{equation}
\expectation\left[X\right] = b\lambda + a
\end{equation}
and
\begin{equation}
\variance\left[X\right] = b^2\lambda + \kappa
\end{equation}
respectively. By eliminating $\lambda$
\begin{equation}
\variance\left[X\right] = b\expectation\left[X\right]+\kappa-ab \ ,
\end{equation}
a linear relationship between the variance and expectation of the grey value \citep{ma2012varaince} is obtained.

In an energy integrating detector, the recorded grey value is linear to the energy detected \citep{whiting2006properties}. The grey value $X$ is
\begin{equation}
X|Y = \sum_{i=1}^Y U_i + \epsilon \ .
\end{equation}
This is the compound Poisson distribution with Normal noise added to it. The scale factor $b$ is not included as this can be absorbed into $U$. Using the result that $\expectation\left[X\right]=\expectation\expectation\left[X|Y\right]$ and $\variance\left[X\right] = \variance\expectation\left[X|Y\right] + \expectation\variance\left[X|Y\right]$, the mean and variance of the grey value are
\begin{equation}
\expectation\left[X\right] = \lambda\expectation\left[U\right]+a
\end{equation}
and
\begin{equation}
\variance\left[X\right] = \lambda \expectation\left[U^2\right]+\kappa
\end{equation}
respectively. Eliminating $\lambda$ obtains
\begin{equation}
\variance\left[X\right] = \dfrac{\expectation\left[U^2\right]}{\expectation\left[U\right]} \expectation\left[X\right] + \kappa - a\dfrac{\expectation\left[U^2\right]}{\expectation\left[U\right]} \ .
\end{equation}
By assuming no beam hardening, $\expectation\left[U\right]$ and $\expectation\left[U^2\right]$ remains constant, thus there is a linear relationship between the variance and expectation of the grey value \citep{yang2009evaluation}. There are other types of detection schemes \citep{whiting2006properties} but it shall not be considered here.

Experiments have been done to verify the compound Poisson nature of the detector. This was done by investigating the variance of radiographs of air \citep{hsieh2015compound} and a polyethene cylinder \citep{yang2009evaluation, yang2010noise} using different exposures. It was found there were two components in the noise, one was signal-dependent and comes from the compound Poisson distribution, the other was signal independent and maybe electronic noise. The electronic noise can be modelled using a Normal distribution \citep{xu2009electronic}.

\subsection{Moment Generating Function}

Returning to the compound Poisson distribution with no electronic noise $X|Y = \sum_{i=1}^{Y}U_i$, let the moment generating function (m.g.f.)~of $X$ be $M_X(\theta)=\expectation\left[\euler^{X\theta}\right]$. It can be shown that the m.g.f.~is
\begin{equation}
  M_X(\theta)=
  \exp\left[
    \lambda
    \left(
      M_U(\theta)-1
    \right)
  \right]
\end{equation}
\citep{gatto2010saddlepoint}. The derivation is shown in Appendix \ref{chapter:appendix_compoundPoissonMgf}. Moments of $X$ can be obtained from the m.g.f.~by differentiating it and setting $\theta$ to zero. In other words, $\expectation[X^r]=M_X^{(r)}(0)$. Then it can be shown that
\begin{align}
  \expectation\left[X\right]&=\lambda\expectation\left[U\right]
  \\
  \variance\left[X\right] &= \lambda\expectation\left[U^2\right]
  \\
  \expectation\left[(X-\expectation[X])^3\right] &=\lambda\expectation\left[U^3\right]
\end{align}
which agrees with the expectation and variance results in the previous section. The derivation is shown in Appendix \ref{chapter:appendix_compoundPoissonMgf}.

\subsection{Compound Poisson-Gamma Distribution}
\label{chapter:compoundPoisson_compoundPoissonGamma}
A special case of the compound Poisson distribution is when $U\sim\gammaDist\left(\alpha,\beta\right)$ where $\alpha>0$ is the gamma shape parameter and $\beta>0$ is the gamma rate parameter. This was used for example in \cite{xu2009electronic}. The distribution is known as the compound Poisson-gamma distribution and is denoted by $X\sim\CPoisson(\lambda,\alpha,\beta)$. The p.d.f.~is
\begin{equation}
  p_X(x) =
  \begin{cases}
    \delta(x) \euler^{-\lambda} & \text{ for } x=0 \\
    \displaystyle\sum_{y=1}^{\infty}\dfrac{\beta^{y\alpha}}{\Gamma(y\alpha)}x^{y\alpha-1}\euler^{-\beta x}\euler^{-\lambda}\frac{\lambda^y}{y!} & \text{ for } x>0
  \end{cases}
  \ .
  \label{eq:compoundPoisson_pdf}
\end{equation}
Recall that $X|Y=\sum_{i=1}^YU_i$ which involves a sum of gamma random variables. It can be shown that
\begin{equation}
  X|Y\sim\gammaDist\left(Y\alpha,\beta\right).
\end{equation}

The m.g.f.~of $U$ is $M_U(\theta) = \left(\dfrac{\beta}{\beta-\theta}\right)^\alpha$, then the m.g.f.~of $X$ is
\begin{equation}
  M_X(\theta)=\exp\left[\lambda\left(\left(\frac{\beta}{\beta-\theta}\right)^{\alpha}-1\right)\right]
\end{equation}
and moments can be obtained from it such as
\begin{equation}
  \expectation\left[X\right]=\frac{\alpha\lambda}{\beta}
\end{equation}
\begin{equation}
  \variance\left[X\right]=\frac{\alpha(\alpha+1)\lambda}{\beta^2}
  \label{eq:compoundPoisson_variance}
\end{equation}
and
\begin{equation}
  \expectation\left[(X-\expectation[X])^3\right] = \frac{\alpha(\alpha+1)(\alpha+2)\lambda}{\beta^3}
  \ .
\end{equation}

\subsection{Generalised Linear Model}

It can be shown that the compound Poisson-gamma distribution is in the exponential family for fixed $\alpha$ \citep{jorgensen1987exponential}. To show this, the compound Poisson-gamma distribution was parametrised using the following:
\begin{equation}
  p=\frac{2+\alpha}{1+\alpha}
  \ ,
\end{equation}
\begin{equation}
  \mu=\frac{\lambda\alpha}{\beta}
  \ ,
\end{equation}
\begin{equation}
  \phi = \frac{\alpha+1}{\beta^{2-p}(\lambda\alpha)^{p-1}}
  \ .
\end{equation}
The parameters $p$, $\mu$ and $\phi$ are called the index, mean and dispersion parameters respectively and take the values of $1<p<2$, $\mu>0$ and $\phi>0$. It can be shown that the p.m.f.~at zero is
\begin{equation}
  \prob(X=0) = \exp
  \left[
      -\frac{\mu^{2-p}}{\phi(2-p)}
  \right]
\end{equation}
and the p.d.f.~for $x>0$ is
\begin{equation}
  p_X(x) =
  \exp\left[
    \frac{1}{\phi}
    \left(
      x\frac{\mu^{1-p}}{1-p}-\frac{\mu^{2-p}}{2-p}
    \right)
  \right]
  \frac{1}{x}
  \sum_{y=1}^{\infty}W_y(x,p,\phi)
\end{equation}
where
\begin{equation}
  W_y = W_y(x,p,\phi)=\frac{x^{y\alpha}}{\phi^{y(1+\alpha)}(p-1)^{y\alpha}(2-p)^yy!\Gamma(y\alpha)}
  \ .
\end{equation}
The derivation is shown in Appendix \ref{chapter:appendix_tweedie}. This is in the form of a distribution in the dispersive exponential family \citep{nelder1972generalized, nelder1972generalized_2, mccullagh1984generalized} for fixed $p$.

Parameter estimation for known $p$ can be done via the generalised linear model framework and can be extended to include linear mixed models \citep{zhang2013likelihood}. Estimating $p$ is difficult and various methods were discussed \citep{zhang2013likelihood}. One way is to estimate $\mu$ and $\phi$ on a grid of $p$'s and then select the $p$ which maximises the likelihood \citep{dunn2005series}.

One special property of the compound Poisson-gamma distribution is that it is in the Tweedie dispersion exponential family \citep{jorgensen1987exponential}. It can be shown that it has a special variance-mean relationship
\begin{equation}
  \variance[X] = \phi \mu^p
\end{equation}
where $1<p<2$. This is derived in Appendix \ref{chapter:appendix_tweedie}. It should be noted that this relationship is for fixed $p$ and $\phi$. This is different from the linear variance-mean relationship found at the start of the chapter which was for fixed $\alpha$ and $\beta$ from assuming no beam hardening.

\subsection{Method of Moments}

The method of moments is a simpler method to estimate the parameters of a compound Poisson-gamma distribution. Suppose $\widehat{\mu}_1$ is an estimator of $\expectation[X]$ and $\widehat{\mu}_j$ is an estimator of $\expectation\left[\left(X-\expectation[X]\right)^j\right]$ for $j=2,3$, then the estimators
\begin{equation}
  \widehat{\lambda}=\frac{\widehat{\mu}_1^2\widehat{\mu}_2}{2\widehat{\mu}_2^2-\widehat{\mu}_1\widehat{\mu}_3}
\end{equation}
\begin{equation}
  \widehat{\alpha}=\frac{2\widehat{\mu}_2^2-\widehat{\mu}_1\widehat{\mu}_3}{\widehat{\mu}_1\widehat{\mu}_3-\widehat{\mu}_2^2}
\end{equation}
\begin{equation}
  \widehat{\beta}=\frac{\widehat{\mu}_1\widehat{\mu}_2}{\widehat{\mu}_1\widehat{\mu}_3-\widehat{\mu}_2^2}
\end{equation}
are method of moments estimators of $\lambda$, $\alpha$ and $\beta$ respectively \citep{withers2011compound}, this is shown in Appendix \ref{chapter:appendix_methodofmoments}. These estimators suffer because estimation is not done through the sufficient statistics and can be negative. This is a problem because the parameters do not take non-positive values.

\subsection{Normal Approximation}

The evaluation of the p.d.f.~of a compound Poisson-gamma distribution is useful so that the likelihood can be obtained. The likelihood can be used to find, for example, maximum likelihood estimators. A problem occurs when dealing with the infinite sum in the p.d.f.~because it cannot be simplified. There are several approximations or computational methods to evaluate the p.d.f.~such as Fourier inverting the characteristic function \citep{dunn2008evaluation}, using the saddlepoint approximation \citep{daniels1954saddlepoint} or cleverly sum over certain terms in the infinite sum \citep{dunn2005series}. Monte Carlo methods can be used to evaluate the p.d.f.~by simulating compound Poisson-gamma random variables.

The m.g.f.~provides a starting point to what limiting distributions the compound Poisson-gamma distribution converges to for large parameters. These limiting distributions can be used to approximate the p.d.f.~of the compound Poisson-gamma distribution.

It can be shown for large $\lambda$, the Normal approximation \citep{shevtsova2014on} is
\begin{equation}
  X\sim\normal\left(\frac{\lambda\alpha}{\beta},\frac{\lambda\alpha(\alpha+1)}{\beta^2}\right) \ .
\end{equation}
The solution is shown in Appendix \ref{chapter:appendix_normalApproximation}.

\subsection{Saddlepoint Approximation}

The saddlepoint approximation \citep{daniels1954saddlepoint} uses the approximate solution to inverting the Laplace transformation or the m.g.f. This has been used in XCT \citep{elbakri2003statistical, elbakri2003efficient}. Inverting the Fourier transformation of the characteristic function also gives the p.d.f.~using computational methods \citep{dunn2008evaluation}. The saddlepoint approximation will be studied here.

For a given m.g.f.~$M_X(\theta)$, the saddlepoint approximation \citep{daniels1954saddlepoint, butler2007saddlepoint} finds an approximate p.d.f.~$p_X(x)$. The saddlepoint approximation is given as
\begin{equation}
  p_X(x)\approx\left(2\pi K_X''(s)\right)^{-1/2}\exp\left[K_X(s)-sx\right]
  \label{eq:saddlePoint:generalSaddlePoint}
\end{equation}
where $K_X(\theta) = \ln\left(M_X(\theta)\right)$ and $s=s(x)$ is the solution to the saddle point equation $K_X'(s)=x$.

For the compound Poisson-gamma distribution, the saddle point approximation \citep{jensen1991saddlepoint} is given as
\begin{multline}
  p_X(x)\approx
  \frac{\left(\lambda\alpha\beta^\alpha\right)^{\frac{1}{2(\alpha+1)}}\euler^{-\lambda}}{\sqrt{2\pi(\alpha+1)}}x^{-\frac{\alpha+2}{2(\alpha+1)}}
  \euler^{-x\beta}
  \exp\left[x^{\frac{\alpha}{\alpha+1}}
    \frac{(\lambda\beta^\alpha)^{\frac{1}{\alpha+1}}(\alpha+1)}{\alpha^{\frac{\alpha}{\alpha+1}}}
  \right]
  \\
  \text{for }x>0
  \label{eq:saddle_point_approx}
\end{multline}
with the derivation shown in Appendix \ref{chapter:appendix_saddlepoint}. The approximation is not well defined for $x=0$.

The integral of the density approximation over the support may not equal to one and it can be numerically re-normalised if necessary. Thus, it may be more sensible to write the approximation up to a constant
\begin{equation}
  p_X(x)\propto x^{-\frac{\alpha+2}{2(\alpha+1)}}\euler^{-x\beta}
  \exp\left[
    x^{\frac{\alpha}{\alpha+1}}
    \frac{
      (\lambda\beta^\alpha)^{\frac{1}{\alpha+1}}(\alpha+1)
    }
    {
      \alpha^{\frac{\alpha}{\alpha+1}}
    }
  \right]
  \ .
\end{equation}

\subsection{Series Evaluation}

The infinite sum, $\sum_{y=1}^\infty W_y$, can be computationally summed cleverly to evaluate the p.d.f. This was done by summing only large terms in the sum and ignoring small terms \citep{dunn2005series}. \cite{dunn2005series} approximated the sum by truncation
\begin{equation}
  \sum_{y=1}^\infty W_y \approx \sum_{y=y_\text{l}}^{y_\text{u}}W_y
\end{equation}
where $y_\text{l}<y_{\text{max}}<y_\text{u}$ and $y_{\text{max}}$ is the value of $y$ which maximises $W_y$. \cite{dunn2005series} used Stirling's approximation to find that
\begin{equation}
  y_{\text{max}} \approx \frac{x^{2-p}}{\phi(2-p)}
\end{equation}
by treating $W_y$ as a continuous and differentiable function of $y$. The derivation is shown in Appendix \ref{chapter:appendix_compoundPoissonSeries}. Because values of $y$ are positive integers, it would be appropriate to round $y_\text{max}$ accordingly
\begin{equation}
  y_{\text{max}} = \text{max}\left[
    1,\text{round}\left(\frac{x^{2-p}}{\phi(2-p)}\right)
  \right]
  \ .
\end{equation}

The limits, $y_\text{l}$ and $y_\text{u}$, can be chosen such that $W_{y_\text{l}}$ and $W_{y_\text{u}}$ are less than $\epsilon W_{y_\text{max}}$ where $\epsilon$ is some small constant, for example, $\epsilon=\euler^{-37}$ will be better than machine precision in 64 bits \citep{dunn2005series}. To prevent overflow problems, it is advised to calculate each term in the summation in log scale \citep{dunn2005series} by using the equation
\begin{equation}
  \ln\left[
    \sum_{y=y_\text{l}}^{y_\text{u}}W_y
  \right]
  =
  \ln\left(
    W_{y_\text{max}}
  \right)
  +\ln\sum_{y=y_\text{l}}^{y_\text{u}}
  \exp\left[
    \ln\left(W_y\right)-\ln\left(W_{y_\text{max}}\right)
  \right]
  \ .
\end{equation}

\section{Simulation Studies on Density Evaluation}

Simulations of a compound Poisson-gamma random variable were conducted to compare how well these density evaluation methods performed. This was done by comparing the evaluated densities using the histogram of simulations and a Q-Q plot. The following compound Poisson-gamma distributions were used in the simulations to capture the variety in the compound Poisson-gamma family: $\CPoisson(1,1,1)$, $\CPoisson(1,100,1)$, $\CPoisson(10,1,1)$, $\CPoisson(100,100,1)$. For each of these distribution, 1\,000 random variables were simulated. Varying $\beta$ is not interesting as this only scales the random variable.

There were a few technical problems with the histogram because the compound Poisson has probability mass at zero and probability density for positive numbers. To correctly represent the empirical density of a compound Poisson random variable, a bar chart was used to show the frequency of zeros and a histogram to show the frequency density of positive numbers. However, the Normal approximation and the saddlepoint approximation does not have mass support at zero. Therefore, to compare these approximate densities to the empirical distribution fairly, a histogram containing both zero and positive samples was used when appropriate.

The evaluation of the p.d.f.~using the saddlepoint approximate required a bit of caution to avoid over/underflow problems. Suppose realisations of $X$ were simulated $\left\{x_1,x_2,x_3,\ldots,x_n\right\}$. The saddlepoint approximation was computed up to a constant using
\begin{equation}
  p_X(x) \propto
  \exp\left[
    -\frac{\alpha+2}{2(\alpha+1)}
    \ln(x)
    -x\beta
    +\left(
    \frac{x\beta}{\alpha}
    \right)^{\frac{\alpha}{\alpha+1}}\lambda^{\frac{1}{\alpha+1}}(\alpha+1) - k
  \right]
\end{equation}
for 10\,000 equally spaced points from and including the minimum non-zero simulated value to the maximum simulated value. $k$ is some constant which was chosen to be
\begin{equation}
  k =
  \max_{i\in\left\{1,2,3,\ldots,n\right\}}
  \left[
    -\frac{\alpha+2}{2(\alpha+1)}
    \ln(x_i)
    -x_i\beta
    +\left(
      \frac{x_i\beta}{\alpha}
    \right)^{\frac{\alpha}{\alpha+1}}\lambda^{\frac{1}{\alpha+1}}(\alpha+1)
  \right]
  \ .
\end{equation}
The density was then normalised by numerically integrating it using the trapezium rule using the 10\,000 evaluated points.

A Q-Q plot is a plot which compares the empirical quantiles with the theoretical quantiles. Let
\begin{equation}
  F_X(x) = \prob(X\leqslant x)
\end{equation}
and
\begin{equation}
  \widehat{F}_X(x) = \frac{1}{n}\cdot\text{max}
  \left[
    \left(\sum_{i=1}^n\mathbb{I}(x_i\leqslant x)\right)-0.5,0
  \right]
  \ ,
\end{equation}
then a Q-Q plot is a parametric plot which plots $\widehat{F}_X^{-1}(p)$ against $F_X^{-1}(p)$ for $p=\frac{0.5}{n},\frac{1.5}{n},\frac{2.5}{n},\ldots,\frac{n-0.5}{n}$. If $F_X(x)$ and $\widehat{F}_X(x)$ are similar, a Q-Q plot should be a straight line with gradient 1, intercepting the origin. For the exact method and the saddlepoint approximation, $F_X(x)$ was found numerically by evaluating the p.d.f.~at $10\,000$ equally spaced points from and including the minimum to the maximum of the simulated samples, and then summing the required trapeziums. For the saddlepoint approximation, it does not support zero, thus the numerical integration started at a non-zero value. $\widehat{F}_X^{-1}(p)$ was then calculated by interpolation.

When plotting the probability of obtaining a zero or the p.d.f.~for positive values, confidence intervals were plotted as well. Consider a bin in a histogram, the confidence intervals were obtained by assuming that the frequency in a bin $\sim\poisson(\text{p.d.f.~evaluated at the bin}\times 1\,000 \times \text{bin width})$. The 68\% confidence interval was chosen because this is a typical choice in physics \citep{cowan1998statistical}.

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\mainSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonlambda1alpha1beta1.eps}
        \caption{Left: Observed and expected frequency of a zero. Right: Histogram and p.d.f.~of non-zero values.}
    \end{subfigure}
    }
    \vspace{2em}
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonlambda1alpha1beta1.eps}
        \caption{Q-Q plot}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(1,1,1)$ random variables were simulated and its empirical density is compared to the p.d.f.~evaluated using the exact method. In a), the dotted red line shows the 68\% confidence interval of the expected frequency or frequency density.}
    \label{fig:compoundPoisson_histogram_1_1_1}
\end{figure}

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\mainSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonlambda1alpha100beta1.eps}
        \caption{Left: Observed and expected frequency of a zero. Right: Histogram and p.d.f.~of non-zero values.}
    \end{subfigure}
    }
    \vspace{2em}
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonlambda1alpha100beta1.eps}
        \caption{Q-Q plot}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(1,100,1)$ random variables were simulated and its empirical density is compared to the p.d.f.~evaluated using the exact method. In a), the dotted red line shows the 68\% confidence interval of the expected frequency or frequency density.}
    \label{fig:compoundPoisson_histogram_1_100_1}
\end{figure}

\begin{figure}[t]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonNormlambda1alpha1beta1.eps}
        \caption{Histogram - Normal approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonNormlambda1alpha1beta1.eps}
        \caption{Q-Q plot - Normal approx.}
    \end{subfigure}
    }
    \vspace{1em}
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonSaddlelambda1alpha1beta1.eps}
        \caption{Histogram - Saddlepoint approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonSaddlelambda1alpha1beta1.eps}
        \caption{Q-Q plot - Saddlepoint approx.}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(1,1,1)$ random variables were simulated and its empirical density is compared to the approximate p.d.f. On the left, the dotted red line shows the 68\% confidence interval of the expected frequency density.}
    \label{fig:compoundPoisson_histogram_approx_1_1_1}
\end{figure}

\begin{figure}[t]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonNormlambda1alpha100beta1.eps}
        \caption{Histogram - Normal approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonNormlambda1alpha100beta1.eps}
        \caption{Q-Q plot - Normal approx.}
    \end{subfigure}
    }
    \vspace{1em}
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonSaddlelambda1alpha100beta1.eps}
        \caption{Histogram - Saddlepoint approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonSaddlelambda1alpha100beta1.eps}
        \caption{Q-Q plot - Saddlepoint approx.}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(1,100,1)$ random variables were simulated and its empirical density is compared to the approximate p.d.f. On the left, the dotted red line shows the 68\% confidence interval of the expected frequency density.}
    \label{fig:compoundPoisson_histogram_approx_1_100_1}
\end{figure}

For low $\lambda$ (Figures \ref{fig:compoundPoisson_histogram_1_1_1} to \ref{fig:compoundPoisson_histogram_approx_1_100_1}) there was a chance of simulating zeros. The exact method has an advantage here because it can evaluate probability mass at zero. The Normal approximation failed to capture the probability mass at zero because the Normal distribution is symmetric and supports negative values. The saddlepoint approximation improves on the Normal approximation by capturing the skewness and kurtosis of the distribution \citep{bedrick1992empirical, butler2007saddlepoint}. As a result, it captured the probability mass at zero as shown by an increase in probability density towards zero.

For $\CPoisson(1,100,1)$, the compound Poisson-gamma p.d.f.~contained multiple peaks. Figure \ref{fig:compoundPoisson_histogram_approx_1_100_1} shows that the saddlepoint approximation was not flexible enough to capture them. In Figure \ref{fig:compoundPoisson_histogram_1_100_1}, the Q-Q plot for the exact method was quite sensitive at the tails of each peak, perhaps there were a few inaccuracies in the exact evaluation of the p.d.f.~or in the calculation of the numerical calculation of the c.d.f.~and its inversion.

As $\lambda$ increased (Figures \ref{fig:compoundPoisson_histogram_10_1_1} and \ref{fig:compoundPoisson_histogram_100_1_1}), all 3 density evaluation methods performed quite well. For $\CPoisson(10,1,1)$ in Figure \ref{fig:compoundPoisson_histogram_10_1_1}, the Normal approximation did not capture the skewness. In Figure \ref{fig:compoundPoisson_histogram_100_1_1}, $\lambda$ was high enough where the compound-Poisson distribution started to converge to a Normal distribution. All methods performed well in this realm.

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonlambda10alpha1beta1.eps}
        \caption{Histogram - Exact method}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonlambda10alpha1beta1.eps}
        \caption{Q-Q plot - Exact method}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonNormlambda10alpha1beta1.eps}
        \caption{Histogram - Normal approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonNormlambda10alpha1beta1.eps}
        \caption{Q-Q plot - Normal approx.}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonSaddlelambda10alpha1beta1.eps}
        \caption{Histogram - Saddlepoint approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonSaddlelambda10alpha1beta1.eps}
        \caption{Q-Q plot - Saddlepoint approx.}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(10,1,1)$ random variables were simulated and its empirical density is compared to the p.d.f. On the left, the dotted red line shows the 68\% confidence interval of the expected frequency density.}
    \label{fig:compoundPoisson_histogram_10_1_1}
\end{figure}

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonlambda100alpha100beta1.eps}
        \caption{Histogram - Exact method}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonlambda100alpha100beta1.eps}
        \caption{Q-Q plot - Exact method}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonNormlambda100alpha100beta1.eps}
        \caption{Histogram - Normal approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonNormlambda100alpha100beta1.eps}
        \caption{Q-Q plot - Normal approx.}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_CompoundPoissonSaddlelambda100alpha100beta1.eps}
        \caption{Histogram - Saddlepoint approx.}
    \end{subfigure}
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpHistogram_qq_CompoundPoissonSaddlelambda100alpha100beta1.eps}
        \caption{Q-Q plot - Saddlepoint approx.}
    \end{subfigure}
    }
    \caption{1\,000 $\CPoisson(100,100,1)$ random variables were simulated and its empirical density is compared to the p.d.f. On the left, the dotted red line shows the 68\% confidence interval of the expected frequency density.}
    \label{fig:compoundPoisson_histogram_100_1_1}
\end{figure}

\section{Proposed Model}
\label{chapter2:proposed_model}

The compound Poisson-gamma distribution can be used to model the grey value of each pixel in a projection. Suppose a projection has $N$ pixels and $m$ replicate projections were obtained. Let $X_{i,j}$ and $Y_{i,j}$ be the grey value and photon count, respectively, of the $i$th pixel in the $j$th replicate projection.

By assuming no beam hardening, the distribution of the photon energy does not change with attenuation. As a result, all pixels will detect photons with identical energy distributions, thus, $\alpha$ and $\beta$ are the same for all pixels. Attenuation does affect the photon count, the more material a photon has to attenuate, the lower the number of detectable photons. The amount of attenuation depends on the specific path from the source to a pixel in a detector, so $\lambda_i$ varies from pixel to pixel. Electronic Normal noise $\epsilon_{i,j}\sim\normal(a,\kappa)$ is added to model noise recorded by the x-ray detector when no x-rays are exposed. This was observed in the black images in Chapter \ref{chapter2}.

\begin{figure}
  \centering
  \includegraphics[width=0.6\textwidth]{../figures/compoundpoisson/graphicalModel.pdf}
  \caption{Graphical model of the grey value $X_{i,j}$ for each of the $N$ pixels in the $m$ replicate projections. $Y_{i,j}\sim\poisson(\lambda_i)$ is the photon count. The grey value has a compound Poisson gamma element $X_{i,j}|Y_{i,j}\sim\gammaDist(Y_{i,j}\alpha,\beta)$ and can be extended by adding electronic noise $\epsilon_{i,j}\sim\normal(a,\kappa)$.}
  \label{fig:compoundPoisson_graphicalModel}
\end{figure}

The graphical model in Figure \ref{fig:compoundPoisson_graphicalModel} illustrates how all of these variables are linked. $a$ and $\kappa$ are unknown parameters but can be estimated beforehand from replicate black images. Care must be taken to use the replicate black images for either shading correction or for estimating $a$ and $\kappa$ to avoid using the data twice. The energy of each photon $U\sim\gammaDist(\alpha,\beta)$ was omitted in the graphical model because the conditional distribution $X|Y\sim\gammaDist(Y\alpha,\beta)$ encapsulates each detected photon energy already.

The assumption of a gamma-distributed x-ray photon energy and Normal electronic noise was applied to XCT by \cite{xu2009electronic}. The assumptions were verified by investigating the variance of radiographs for different x-ray exposures. More statistically sound methods could use goodness of fit tests or Q-Q plots. The use of the gamma distribution will not be exactly correct because the sources of x-ray photon energy are characteristic and bremsstrahlung radiation which does not behave like a gamma random variable. However, characterisation of the density of the x-ray photon energy is difficult and usually involve more apparatuses and numerical methods \citep{fewell1977photon, stumbo2004direct}.

The model may be simplified by omitting the electronic noise and this can be done by setting $a=0$ and $\kappa=0$. Further simplification can be done by removing spatial variation by setting $\lambda=\lambda_1=\lambda_2=\ldots=\lambda_N$. Parameter estimation can be done using gradient methods because the likelihood of the compound Poisson can be evaluated \citep{dunn2005series}. However, the EM algorithm is faster and the model is well set up for it.

\section{EM Algorithm}

The EM algorithm \citep{dempster1977maximum} was proposed to estimate the parameters of a $X\sim\CPoisson(\lambda,\alpha,\beta)$ random variable given a sample of measurements of it $\left\{x_1,x_2,x_3,\ldots,x_n\right\}$. The use of the EM algorithm for the compound Poisson distribution in XCT have been studied in \cite{elbakri2003statistical, xie2008x, xu2009electronic}.

Let $\widehat{\lambda}$, $\widehat{\alpha}$ and $\widehat{\beta}$ be estimators of $\lambda$, $\alpha$ and $\beta$ respectively. The log-likelihood is defined to be
\begin{equation*}
  \ln L(\lambda,\alpha,\beta;X) = \sum_{i=1}^n
  \left[
    \mathbb{I}(x_i=0)\ln \prob(X=x_i)
    +\mathbb{I}(x_i>0)\ln p_X(x_i)
  \right]
\end{equation*}
and using the results in Appendix \ref{chapter:appendix_tweedie}
\begin{equation}
  \ln L(\lambda,\alpha,\beta;X) =
    \sum_{i=1}^n
    \left[
      -\mathbb{I}(x_i=0)\lambda
      \vphantom{\sum_i^i}
    \right.
    {}\\
    \left.
      +\mathbb{I}(x_i>0)\left(
      -\beta x_i-\lambda-\ln x +\ln\sum_{y=1}^\infty W_y
      \right)
    \right] \ .
\end{equation}
Maximum likelihood estimators are values of $\lambda$, $\alpha$ and $\beta$ which jointly maximise the log-likelihood.

The EM algorithm \citep{dempster1977maximum} treats $Y\sim\poisson(\lambda)$ as a latent variable. Let $\left\{Y_1,Y_2,Y_3,\ldots, Y_n\right\}$ be realisations of $Y$. Define the joint log-likelihood to be
\begin{multline*}
  \ln L(\lambda,\alpha,\beta;X,Y)
  =
  \sum_{i=1}^n
  \left[
    \mathbb{I}(x_i=0)
    \ln
    \prob(Y=0)
  \right.
  \\
  \left.
    +
    \mathbb{I}(x_i>0)
    \ln
    \left[
      p_{X|Y}(x_i|Y_i)\prob(Y=Y_i)
    \right]
  \right]
\end{multline*}
so that
\begin{multline}
  \ln L(\lambda,\alpha,\beta;X,Y)=
  \sum_{i=1}^n
  \left[
    -\mathbb{I}(x_i=0)
    \lambda
  \right.
  \\
  \left.+
    \mathbb{I}(x_i>0)
    \left(
      Y_i\alpha\ln\beta-\ln\Gamma(Y_i\alpha)+(Y_i\alpha-1)\ln x_i - \beta x_i
    \right.
  \right.
  \\
  \left.
    \left.
      - \lambda + Y_i \ln \lambda - \ln(Y_i!)
    \right)
  \right]
  \ .
  \label{eq:compoundPoisson_jointLogLikelihood}
\end{multline}
The estimators were found by optimising the joint log-likelihood. This was done iteratively by estimating the $Y$'s given the $X$'s and parameters (E step), followed by estimating the parameters given the $Y$'s and $X$'s (M step) until some convergence conditions were met.

It will be shown that using some approximations, the E step and M step can be implemented. However, simulations showed that these estimators struggle for high $\lambda$.

\subsection{E Step}

In the E step, the realisations of $Y$ are estimated using
\begin{equation}
  y_i =
  \expectation\left[
    Y|X=x_i
  \right]
\end{equation}
given the parameters $\lambda$, $\alpha$ and $\beta$. The conditional expectation is calculated using
\begin{equation}
  y_i =
  \begin{cases}
    0 & \text{ for } x=0 \\
    \dfrac{\sum_{y=1}^\infty y \prob(Y=y|X=x_i)}{\sum_{y=1}^\infty \prob(Y=y|X=x_i)} & \text{ for } x>0
  \end{cases}
\end{equation}
where
\begin{equation*}
  \prob(Y=y|X=x) = \frac{p_{X|Y}(x|y)\prob(Y=y)}{p_X(x)}
  \ .
\end{equation*}
Focussing on the $x>0$ case for now
\begin{equation*}
  \prob(Y=y|X=x) = \frac{1}{p_X(x)}\frac{\beta^{y\alpha}}{\Gamma(y\alpha)}x^{y\alpha-1}\euler^{-\beta x}\frac{\euler^{-\lambda}\lambda^y}{y!}
\end{equation*}
which is simplified to
\begin{equation}
  \prob(Y=y|X=x) = W_y \frac{\euler^{-\lambda-\beta x}}{x p_X(x)} \ .
\end{equation}
As a result, the conditional expectation is
\begin{equation}
  y_i = \frac{\sum_{y=1}^\infty y W_y}{\sum_{y=1}^\infty W_y}
  \ .
\end{equation}
As discussed before, the sum in the denominator can be evaluated using the method by \cite{dunn2005series}. A similar method for evaluating the numerator can be obtained by truncating the sum and summing over large terms.

Let
\begin{equation}
  W_y^{(r)} = y^r W_y \quad \text{for }r=1,2,3,\ldots
\end{equation}
so that
\begin{equation}
  y_i = \frac{\sum_{y=1}^\infty W^{(1)}_y}{\sum_{y=1}^\infty W_y} \ .
\end{equation}
Similarly,
\begin{equation}
  \zeta_i = \variance[Y|X=x_i] = \frac{\sum_{y=1}^\infty W^{(2)}_y}{\sum_{y=1}^\infty W_y} - \left(y_i\right)^2
  \ .
\end{equation}
The expectation terms, $y_i$ and $\zeta_i$, are evaluated here so that they can be used in the M step. The evaluation can be done by truncating the sum
\begin{equation}
  \sum_{y=1}^\infty W^{(r)}_y \approx \sum_{y=y_\text{l}}^{y_\text{u}} W^{(r)}_y
\end{equation}
where $y_\text{l}<y_\text{max}<y_\text{u}$ and $y_\text{max}$ is the value of $y$ which maximises $W_y^{(r)}$. This term can be expressed as
\begin{equation}
  \ln W_y^{(r)} =r\ln y+\ln W_y
\end{equation}
and taking the derivative with respect to $y$ obtains
\begin{equation}
  \frac{\partial }{\partial y} \ln W_y^{(r)} = \frac{r}{y } + \frac{\partial }{\partial y}\ln W_y
  \ .
\end{equation}
Keep in mind that $y=1,2,3,\ldots$ so for large $y$, an approximation can be made $r/y\approx 0$ so that
\begin{equation}
  \frac{\partial }{\partial y} \ln W_y^{(r)}\approx \frac{\partial }{\partial y} \ln W_y
  \ .
\end{equation}
Therefore, the maximum of $W_y^{(r)}$ is located at $y_\text{max}=\dfrac{x^{2-p}}{\phi(2-p)}$ for all $r=0,1,2,\ldots$. As a result, the same method for evaluating $W_y$ can be used to evaluate $W_y^{(r)}$. For a given $r=0,1,2,\ldots$, the limit of the sum $\sum_{y=y_\text{l}}^{y_\text{u}} W^{(r)}_y$ were chosen such that $W^{(r)}_{y_\text{l}}$ and $W^{(r)}_{y_\text{u}}$ are less than $\epsilon W_{y_\text{max}}^{(r)}$ where $\epsilon$ is some small constant. The limits will be different for different values of $r$.

\subsection{M Step}

In the M step, the conditional expected joint log-likelihood is maximised with respect to the parameters $\lambda$, $\alpha$ and $\beta$. The objective function is
\begin{align}
  T(\lambda,\alpha,\beta)&=
  \begin{multlined}[t]
    \sum_{i=1}^n
    \expectation\left[
      -\mathbb{I}(x_i=0)
      \lambda
    \right.
    \\
    \left.+
      \mathbb{I}(x_i>0)
      \left(
        Y_i\alpha\ln\beta-\ln\Gamma(Y_i\alpha)+(Y_i\alpha-1)\ln x_i - \beta x_i
      \right.
    \right.
    \\
    \left.
      \left.
        - \lambda + Y_i \ln \lambda - \ln(Y_i!)
      \right)
      |X_i=x_i
    \right]
  \end{multlined}
  \nonumber\\
  &=\begin{multlined}[t]
    -n\lambda
    \\
    +\sum_{i=1}^n
    \mathbb{I}(x_i>0)
    \left[
      \expectation[Y_i|X_i=x_i]\alpha\ln\beta-\expectation[\ln\Gamma(Y_i\alpha)|X_i=x_i]
    \right.
    {}\\
    \left.
      +\expectation[Y_i|X_i=x_i]\alpha\ln x_i - \beta x_i
      + \expectation[Y_i|X_i=x_i] \ln \lambda
    \right] + c
  \end{multlined}
\end{align}
where $c$ is some constant not dependent on $\lambda$, $\alpha$ or $\beta$.

The conditional expectation $y_i = \expectation[Y_i|X_i=x_i]$ and $\zeta_i = \variance[Y_i|X_i=x_i]$ were calculated beforehand in the E step. The quantity $\expectation[\ln\Gamma(Y_i\alpha)|X_i=x_i]$ can be calculated using the approximation
\begin{equation}
  \expectation[\ln\Gamma(Y_i\alpha)|X_i=x_i] \approx
  \ln\Gamma(\alpha y_i) + \frac{1}{2}\zeta_i\alpha^2\psi'(y_i\alpha)
\end{equation}
where $\psi(n)$ is the digamma function. The objective function is then
\begin{multline}
  T(\lambda,\alpha,\beta)\approx
  -n\lambda
  +\sum_{i=1}^n
  \mathbb{I}(x_i>0)
  \left[
    y_i\alpha\ln\beta-\ln\Gamma(\alpha y_i) - \frac{1}{2}\zeta_i\alpha^2\psi'(y_i\alpha)
  \right.
  \\
  \left.
    +y_i\alpha\ln x_i - \beta x_i
    + y_i \ln \lambda
    \vphantom{\frac{1}{1}}
  \right]
  + c
  \ .
\end{multline}

Taking the derivative with respect to $\lambda$
\begin{equation}
  \frac{\partial T}{\partial\lambda} = -n + \frac{\sum_{i=1}^ny_i}{\lambda}
\end{equation}
and setting it to zero
\begin{equation}
  \widehat{\lambda} = \frac{\sum_{i=1}^n y_i}{n}
\end{equation}
obtains a M step estimator for $\lambda$. Taking the second-order derivative with respect to $\lambda$
\begin{equation}
  \frac{\partial^2 T}{\partial \lambda^2} = -\frac{\sum_{i=1}^ny_i}{\lambda^2} < 0
\end{equation}
verifies that $\widehat{\lambda}$ maximises $T$. In addition,
\begin{equation}
  \frac{\partial^2 T}{\partial \alpha \partial \lambda } = 0
\end{equation}
and
\begin{equation}
  \frac{\partial^2 T}{\partial \beta \partial \lambda } = 0
  \ .
\end{equation}

Maximising $T$ with respect to $\alpha$ and $\beta$ can be done numerically using the Newton-Raphson method since derivatives up to the second-order can be obtained. For the first-order derivatives, these are
\begin{multline}
  \frac{\partial T}{\partial \alpha} =
  \sum_{i=1}^n\mathbb{I}(x_i>0)
  \left[
    y_i\ln\beta -\psi(\alpha y_i)y_i-\zeta_i\alpha\psi'(\alpha y_i)
    \vphantom{\frac{1}{1}}
  \right.
  \\
  \left.
    - \frac{1}{2}\zeta_i\alpha^2\psi''(\alpha y_i)y_i + y_i\ln x_i
  \right]
\end{multline}
and
\begin{equation}
  \frac{\partial T}{\partial \beta} = \sum_{i=1}^n\mathbb{I}(x_i>0)\left[
  \frac{\alpha y_i}{\beta}-x_i
  \right]
  \ .
\end{equation}
The second-order derivatives are
\begin{equation}
  \frac{\partial^2 T}{\partial \alpha \partial \beta} =
  \sum_{i=1}^n \mathbb{I}(x_i>0)\left[\frac{y_i}{\beta}\right]
  \ ,
  \label{eq:compoundPoisson:d2tdadb}
\end{equation}
\begin{equation}
  \frac{\partial^2 T}{\partial \beta^2} = \sum_{i=1}^n\mathbb{I}(x_i>0)\left[-\frac{\alpha y_i}{\beta^2}
  \right]
  \ ,
  \label{eq:compoundPoisson:d2td2b}
\end{equation}
and
\begin{multline*}
  \frac{\partial^2 T}{\partial \alpha^2} =
  \sum_{i=1}^n\mathbb{I}(x_i>0)
  \left[
    -y_i^2\psi'(\alpha y_i) - \zeta_i\psi'(\alpha y_i) - \zeta_i\alpha y_i\psi''(\alpha y_i)
    \vphantom{\frac{1}{2}}\right.\\\left.
    - \zeta_i\alpha\psi''(\alpha y_i)y_i
    -\frac{1}{2}\zeta_i\alpha^2\psi'''(\alpha y_i)y_i^2
  \right]
\end{multline*}
simplifying to
\begin{multline}
  \frac{\partial^2 T}{\partial \alpha^2} =
  \sum_{i=1}^n\mathbb{I}(x_i>0)
  \left[
    -(y_i^2+\zeta_i)\psi'(\alpha y_i) - 2\zeta_i\alpha y_i\psi''(\alpha y_i)
    \vphantom{\frac{1}{2}}
  \right.
  \\
  \left.
    -\frac{1}{2}\zeta_i\alpha^2y_i^2\psi'''(\alpha y_i)
  \right] \ .
  \label{eq:compoundPoisson:d2td2a}
\end{multline}

All the derivatives can be used in the Newton-Raphson iterative update to update the estimators $\widehat{\alpha}$ and $\widehat{\beta}$. The update is
\begin{equation}
  \begin{pmatrix}
    \widehat{\alpha} \\ \widehat{\beta}
  \end{pmatrix}
  \leftarrow
  \begin{pmatrix}
    \widehat{\alpha} \\ \widehat{\beta}
  \end{pmatrix}
  -
  \left[
    \nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T \left. T \right |_{\alpha = \widehat{\alpha}, \beta=\widehat{\beta}}
  \right]^{-1}
  \left[
    \nabla_{\alpha,\beta} \left. T \right |_{\alpha = \widehat{\alpha}, \beta=\widehat{\beta}}
  \right]
\end{equation}
where
\begin{equation}
  \nabla_{\alpha,\beta}=
  \begin{pmatrix}
    {\partial}/{\partial \alpha}
    \\
    {\partial}/{\partial \beta}
  \end{pmatrix}
  \ .
\end{equation}
Since increasing $T$ is sufficient for the EM algorithm \citep{dempster1977maximum}, one step of the Newton-Raphson iterative update was chosen in the M step to avoid implementing a convergence condition for the Newton-Raphson method.

\subsection{Simulations}

An experiment was conducted to assess the performance on the EM algorithm. For a given set of parameters, 1\,000 samples of a $\CPoisson(\lambda,\alpha,\beta)$ random variable were simulated. The EM algorithm was initialised with its parameters at the true values to investigate the convergence in that vicinity. The log-likelihood $\ln L(\lambda,\alpha,\beta;X)$ and the parameters were recorded at every EM step. The experiment was repeated 10 times using different simulated samples.

The results for $\CPoisson(1,1,1)$, $\CPoisson(1,100,1)$, $\CPoisson(10,1,1)$ and $\CPoisson(100,100,1)$ are shown in Figures \ref{fig:compoundPoisson_convergence_1}, \ref{fig:compoundPoisson_convergence_2}, \ref{fig:compoundPoisson_convergence_3} and \ref{fig:compoundPoisson_convergence_4} respectively. Good performance was observed in the $\lambda=1$ case with convergence of all three parameters within a step or two. The Cram\'er-Rao lower bound captured the spread of the estimates well.

For $\lambda=10$ and $\lambda=100$, the estimates of $\alpha$ and $\beta$ struggled to converge and increased/decreased without bounds without affecting the log-likelihood. It appeared that the EM algorithm failed for $\lambda>10$ looking at these particular examples.

\begin{figure}[p]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha1beta1_lnL.eps}
        \caption{Log-likelihood}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha1beta1_lambda.eps}
        \caption{$\lambda$}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha1beta1_alpha.eps}
        \caption{$\alpha$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha1beta1_beta.eps}
        \caption{$\beta$}
    \end{subfigure}
    }
    \caption{The EM algorithm was used to estimate the parameters of a $\CPoisson(1,1,1)$ random variable using 1\,000 simulated samples. The graphs show the log-likelihood and the estimated parameters at each EM step for 10 different simulations.}
    \label{fig:compoundPoisson_convergence_1}
\end{figure}

\begin{figure}[p]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha100beta1_lnL.eps}
        \caption{Log-likelihood}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha100beta1_lambda.eps}
        \caption{$\lambda$}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha100beta1_alpha.eps}
        \caption{$\alpha$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda1alpha100beta1_beta.eps}
        \caption{$\beta$}
    \end{subfigure}
    }
    \caption{The EM algorithm was used to estimate the parameters of a $\CPoisson(1,100,1)$ random variable using 1\,000 simulated samples. The graphs show the log-likelihood and the estimated parameters at each EM step for 10 different simulations.}
    \label{fig:compoundPoisson_convergence_2}
\end{figure}

\begin{figure}[p]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda10alpha1beta1_lnL.eps}
        \caption{Log-likelihood}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda10alpha1beta1_lambda.eps}
        \caption{$\lambda$}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda10alpha1beta1_alpha.eps}
        \caption{$\alpha$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda10alpha1beta1_beta.eps}
        \caption{$\beta$}
    \end{subfigure}
    }
    \caption{The EM algorithm was used to estimate the parameters of a $\CPoisson(10,1,1)$ random variable using 1\,000 simulated samples. The graphs show the log-likelihood and the estimated parameters at each EM step for 10 different simulations.}
    \label{fig:compoundPoisson_convergence_3}
\end{figure}

\begin{figure}[p]
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda100alpha100beta1_lnL.eps}
        \caption{Log-likelihood}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda100alpha100beta1_lambda.eps}
        \caption{$\lambda$}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda100alpha100beta1_alpha.eps}
        \caption{$\alpha$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/CpEmAlgorithm_CompoundPoissonlambda100alpha100beta1_beta.eps}
        \caption{$\beta$}
    \end{subfigure}
    }
    \caption{The EM algorithm was used to estimate the parameters of a $\CPoisson(100,100,1)$ random variable using 1\,000 simulated samples. The graphs show the log-likelihood and the estimated parameters at each EM step for 10 different simulations.}
    \label{fig:compoundPoisson_convergence_4}
\end{figure}

\afterpage{\clearpage}

\section{Failure Evaluation}

It should be convincing that the EM algorithm failed when the compound Poisson-gamma random variable starts behaving Normally. In particular, in Figure \ref{fig:compoundPoisson_convergence_4}, estimates of $\lambda$ were stable while estimates of $\alpha$ and $\beta$ struggled to converge. This could be because as the compound Poisson-gamma distribution approaches the Normal distribution, the parameters $(\lambda,\alpha,\beta)$ becomes degenerate because there is more than one way to represent a two-parameter random variable $\normal(\mu,\sigma^2)$ using three parameters $(\lambda,\alpha,\beta)$.

\subsection{Determinant of the Hessian}

One investigation is to look at the Newton-Raphson step in the M step, in particular, the Hessian matrix $\nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T T$ for high $\lambda$. The elements of the Hessian matrix can be found in Equations \eqref{eq:compoundPoisson:d2tdadb}, \eqref{eq:compoundPoisson:d2td2b} and \eqref{eq:compoundPoisson:d2td2a}. Recall that
\begin{multline*}
  \frac{\partial^2 T}{\partial \alpha^2} =
  \sum_{i=1}^n\mathbb{I}(x_i>0)\left[
    -(y_i^2+\zeta_i)\psi'(\alpha y_i) - 2\zeta_i\alpha y_i\psi''(\alpha y_i)
    \vphantom{\frac{1}{2}}
  \right.
  \\
  \left.
    -\frac{1}{2}\zeta_i\alpha^2y_i^2\psi'''(\alpha y_i)
  \right]
  \ .
\end{multline*}
For high $\alpha$ and $\lambda$, and hence high $y_i$'s, an approximation can be used for the polygamma functions $\psi^{(k)}(n)$. Using Stirling's approximation $\ln\Gamma(n)\approx\ln(n!)\approx n\ln n-n$, then
\begin{equation}
  \psi(n) = \frac{\partial\ln\Gamma(n)}{\partial n} \approx \ln n
  \ .
\end{equation}
Differentiating further
\begin{align}
  \psi'(n) &\approx 1/n \\
  \psi''(n) & \approx -1/n^2 \\
  \psi'''(n) & \approx 2/n^3
  \ .
\end{align}
$\dfrac{\partial^2T}{\partial\alpha^2}$ can be approximated
\begin{equation*}
  \frac{\partial^2 T}{\partial \alpha^2} \approx
  \sum_{i=1}^n\mathbb{I}(x_i>0)
  \left[
    -\frac{y_i^2+\zeta_i}{\alpha y_i} + 2\frac{\zeta_i\alpha y_i}{\alpha^2 y_i^2}
    -\frac{1}{2}\frac{2\zeta_i\alpha^2y_i^2}{\alpha^3 y_i^3}
  \right]
\end{equation*}
to get
\begin{equation}
  \frac{\partial^2 T}{\partial \alpha^2} \approx
  \sum_{i=1}^n\mathbb{I}(x_i>0)
  \left[
    -\frac{y_i}{\alpha}
  \right]
  \ .
\end{equation}

The Hessian matrix is, omitting the $\mathbb{I}(x_i>0)$ term,
\begin{equation}
  \nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T T \approx
  \sum_{i=1}^n
  \begin{pmatrix}
    -y_i/\alpha  & y_i/\beta \\
    y_i/\beta & -\alpha y_i/\beta^2
  \end{pmatrix}
  \ .
\end{equation}
The determinant of the Hessian matrix is
\begin{align*}
  \|\nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T T\|
  &\approx
  \left(\sum_{i=1}^n-\frac{\alpha y_i}{\beta^2}\right)
  \left(\sum_{i=1}^n-\frac{y_i}{\alpha}\right) - \left(\sum_{i=1}^n\frac{y_i}{\beta}\right)^2
  \\
  &\approx
  \left(\sum_{i=1}^n\sum_{j=1}^n\frac{y_iy_j}{\beta^2}\right)
   - \left(\sum_{i=1}^n\frac{y_i}{\beta}\right)\left(\sum_{j=1}^n\frac{y_j}{\beta}\right)
  \\
  &\approx
  \left(\sum_{i=1}^n\sum_{j=1}^n\frac{y_iy_j}{\beta^2}\right)
   - \left(\sum_{i=1}^n\sum_{j=1}^n\frac{y_iy_j}{\beta^2}\right)
\end{align*}
to obtain
\begin{equation}
  \|\nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T T\|
  \approx
  0
  \ .
\end{equation}

This results in a few things. Firstly, $\nabla_{\alpha,\beta}\nabla_{\alpha,\beta}\T T$ is singular thus its inverse cannot be evaluated which is needed for the Newton-Raphson method. Secondly, the sufficient conditions to classify a stationary point as a maximum are that the diagonal elements of the Hessian matrix are negative and the determinate is positive. For high $\lambda$ and $\alpha$, the second condition is not met.

\subsection{Constrained Objective}

The parameter $\beta$ can be constrained for a given mean to be
\begin{equation}
  \beta = \frac{\lambda\alpha}{\widehat{\mu}}
  \label{eq:compoundPoisson:beta_restrict}
\end{equation}
where
\begin{equation}
  \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i
\end{equation}
and it was investigated whenever a unique solution to $\dfrac{\partial T}{\partial \alpha} = 0$ can be found for high $\lambda$ and $\alpha$. The constrained objective function is
\begin{multline}
  T(\lambda,\alpha)
  =
  -n\lambda+
  \sum_{i=1}^n
  \mathbb{I}(x_i>0)
  \left[
  y_i\alpha
  \ln\left(\frac{\lambda\alpha}{\widehat{\mu}}\right)-\ln\Gamma(\alpha y_i) - \frac{1}{2}\zeta_i\alpha^2\psi'(y_i\alpha)
  \right.
  \\
  \left.
    +y_i\alpha\ln x_i - \frac{\lambda\alpha x_i}{\widehat{\mu}}
    + y_i \ln \lambda
    \vphantom{\frac{1}{1}}
  \right]
  + c
\end{multline}
which can be approximated to
\begin{align}
  T(\lambda,\alpha)&\approx
  \begin{multlined}[t]
    -n\lambda+
    \sum_{i=1}^n
    \mathbb{I}(x_i>0)
    \left[
      y_i\alpha\ln\left(\frac{\lambda\alpha}{\widehat{\mu}}\right)-\alpha y_i\ln(\alpha y_i) + \alpha y_i - \frac{1}{2}\frac{\zeta_i\alpha}{y_i}
    \right.
    \\
    \left.
      +y_i\alpha\ln x_i - \frac{\lambda\alpha x_i}{\widehat{\mu}}
      + y_i \ln \lambda
      \vphantom{\frac{1}{1}}
    \right]
    + c
  \end{multlined}
  \nonumber\\
  &\!\begin{multlined}[b]\approx
    -n\lambda+
    \sum_{i=1}^n
    \mathbb{I}(x_i>0)
    \left[
      y_i\alpha\ln\left(\frac{\lambda}{y_i\widehat{\mu}}\right) + \alpha y_i - \frac{1}{2}\frac{\zeta_i\alpha}{y_i}
    \right.
    {}\\
    \left.
      +y_i\alpha\ln x_i - \frac{\lambda\alpha x_i}{\widehat{\mu}}
      + y_i \ln \lambda
      \vphantom{\frac{1}{1}}
    \right]
    + c
    \ .
  \end{multlined}
\end{align}
Taking the derivative with respect to $\alpha$
\begin{equation}
  \frac{\partial T}{\partial \alpha} \approx
  \sum_{i=1}^n
  \mathbb{I}(x_i>0)
  \left[
    y_i\ln\left(\frac{\lambda}{y_i\widehat{\mu}}\right)
    + y_i - \frac{1}{2}\frac{\zeta_i}{y_i}
    +y_i\ln x_i - \frac{\lambda x_i}{\widehat{\mu}}
    \vphantom{\frac{1}{1}}
  \right]
\end{equation}
and this is not $\alpha$ dependent, thus there is no solution for $\alpha$.

\subsection{Log-Likelihood Plot}

\begin{figure}
  \centering
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpLogLikelihoodPlot_CompoundPoissonlambda1alpha1beta1.eps}
        \caption{$\CPoisson(1,1,1)$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpLogLikelihoodPlot_CompoundPoissonlambda1alpha100beta1.eps}
        \caption{$\CPoisson(1,100,1)$}
    \end{subfigure}
    }
    \centerline{
    \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpLogLikelihoodPlot_CompoundPoissonlambda10alpha1beta1.eps}
        \caption{$\CPoisson(10,1,1)$}
    \end{subfigure}
        \begin{subfigure}[b]{\subSize}
        \includegraphics[width=\textwidth]{../figures/compoundpoisson/cpLogLikelihoodPlot_CompoundPoissonlambda100alpha100beta1.eps}
        \caption{$\CPoisson(100,100,1)$}
    \end{subfigure}
    }
    \caption{Log-likelihood from a simulation of 100 compound Poisson-gamma random variables. $\lambda$ is fixed at the true value.}
    \label{fig:compoundPoisson_loglikelihoodplot}
\end{figure}

The log-likelihood, using a simulation of 100 compound Poisson-gamma random variables, are plotted for fixed $\lambda$ in Figure \ref{fig:compoundPoisson_loglikelihoodplot}. These plots are concerning because the likelihood appeared not convex and there is no unique maximum for all $\lambda$ investigated.

It was observed that there is a saddle point in the log-likelihood for the $\CPoisson(1,100,1)$ case. This is when the log-likelihood increased and decreased when varying $\alpha$ and $\beta$. This suggests the possibility of local maxima.

As a result, there is no unique maximum likelihood estimator and the EM algorithm would be very sensitive to the starting position, including for small $\lambda$.

\section{Conclusion}
The compound Poisson-gamma distribution was studied but it was found there were identifiability issues when fitting the model onto simulated data, in particular for high $\lambda$. It was found that the log-likelihood was quite flat because, by using numerous approximations, it was found that the determinant of the Hessian is about zero. This cast doubts if a unique maximum likelihood estimator exists. Tighter bounds for the Hessian is beyond the scope of this thesis.

The problem with the EM algorithm is that it is sensitive to the initial value and does not guarantee convergence to a global maximum \citep{moon1996expectation}. Another disadvantage is that the EM algorithm does not produce uncertainty quantification for the estimators. Methods such as bootstrapping \citep{efron1979bootstrap} and Louis' missing information principle \citep{louis1982finding} can be used to obtain such uncertainty quantification.

There are methods to deal with flat likelihoods by extending the likelihood to include penalising terms \citep{cole2014maximum}. Another approach is to use Bayesian methods \citep{bishop2006pattern} where a prior distribution is defined for the parameters $\alpha$, $\beta$, $\lambda_1,\lambda_2,\ldots,\lambda_N$, $a$ and $\kappa$ where $N$ is the number of pixels and $a$ and $\kappa$ are the mean and variance terms for the Normal noise, defined in Section \ref{chapter2:proposed_model}. The prior distribution can reflect prior knowledge of the model, for example, the statistical properties of the sample of black images can be embedded into the priors of $a$ and $\kappa$. Back of the envelope calculations can be used to show that $\lambda_i \sim 10^7\,\si{{photon}\,\pixel^{-1}}$ which can be used to aid in the creation of a prior distribution for $\lambda_i$. Poisson graphical models \citep{yang2013poisson} and Gaussian processes \citep{williams1996gaussian} can impose a spatial dependency on $Y_{i,j}$ and $\lambda_i$ respectively as well.

Bayesian inference is done by studying the posterior distribution where
\begin{equation*}
\mathrm{posterior} \propto \mathrm{likelhood} \times \mathrm{prior} \ .
\end{equation*}
By multiplying the likelihood with the prior, the posterior should hopefully be convex and enable better convergence. Studying the posterior distribution typically requires Markov chains using Monte Carlo methods \citep{brooks2011handbook} such as Metropolis-Hastings \citep{metropolis1953equation, hastings1970monte}, Gibbs sampling \citep{geman1984stochastic}, Hamiltonian Monte Carlo \citep{neal2011mcmc, hoffman2014no} and slice sampling \citep{neal2003slice, murray2010elliptical}.

To keep the frequentist theme throughout this thesis, the next chapter studies the linear relationship between the variance and mean of the grey value in a pixel. The uncertainty can be predicted by using such a relationship.


\chapter{Mass and Inertia}

\section{Empty Aircraft Moments of Inertia}

Aircraft is divided into structure groups which mass is estimated. This groups are assumed to be homogeneous rigid body with simple shape which allows to calculate its moment of inertia using an exact closed-form expression, given e.g. in \cite{HousnerHudson1980}.

Steiner’s theorem, given by the following expression, is used to express aircraft structure groups inertia tensor in Body Axis System. \cite{Taylor2005, ResnickHalliday2011}
\begin{equation}
  \label{eq-mass-steiners}
  {\boldsymbol I}_b
  =
  {\boldsymbol I}_0
  +
  m
  \left[
    \begin{matrix}
      y^2 + z^2 &       -xy &       -xz \\
            -yx & x^2 + z^2 &       -yz \\
            -zx &       -zy & x^2 + y^2 \\
    \end{matrix}
  \right]
\end{equation}

Sum of all aircraft structure groups inertia tensors gives empty aircraft inertia tensor:
\begin{equation}
  {\boldsymbol I}_b = \sum_{j} {\boldsymbol I}_{j,b}
\end{equation}

\section{Variable Masses}

All variable masses, crew, fuel, payload, etc., are considered to be point masses. Point mass inertia tensor can be calculated using formula (\ref{eq-mass-steiners}), where ${\boldsymbol I}_b = 0$. This tensors are then added to the empty aircraft inertia tensor giving total aircraft inertia tensor.

Aircraft total first moment of mass is given as follows:
\begin{equation}
  {\vec S}_b = \sum_{j} m_j {\vec r}_{CM,j,b}
\end{equation}

Position of aircraft center of mass including variable masses is then given by following formula:
\begin{equation}
  {\vec r}_{CM,b} = \frac{ {\vec S}_b }{ \sum_{j} m_j }
\end{equation}


\chapter{Linear and Logistic Regression}

\begin{ex}
  Recall that the least squares estimates are the values $\betahat_0$ and
  $\betahat_1$ that minimize
  $\textsc{rss}=\sum_{i=1}^n(Y_i-\betahat_1X_i-\betahat_0)^2$. This is a
  quadratic function with positive leading coefficients in both $\betahat_1$ and
  $\betahat_0$, and therefore will achieve its minimum at a critical point. To
  that end, we take partial derivatives with respect to $\betahat_1$ and
  $\betahat_1$, set them equal to zero and solve.
  \begin{align*}
    0 & =\frac{\pd{\textsc{RSS}}}{\pd \betahat_0}    \\
      & =2\sum_{i=1}^n(Y_i-\betahat_1X_i-\betahat_0) \\
      & =2n(\Ybar-\betahat_1\Xbar-\betahat_0),
  \end{align*}
  and therefore,
  \[
    \betahat_0 = \Ybar-\betahat_1\Xbar.
  \]

  Hence,
  \begin{align*}
    0 & =\frac{\pd{\textsc{RSS}}}{\pd \betahat_1}               \\
      & =-2\sum_{i=1}^nX_i(Y_i-\betahat_1X_i-\betahat_0)        \\
      & =-2\left(\sum_{i=1}^nX_iY_i-\betahat_1\sum_{i=1}^nX^2_i
    +n\betahat_1\Xbar^2-n\Xbar\Ybar\right),
  \end{align*}
  or,
  \[
    \betahat_1=\frac{\sum_{i=1}^nX_iY_i-n\Xbar\Ybar}{\sum_{i=1}^nX_i^2-n\Xbar^2}.
  \]

  Since
  \begin{align*}
    \sum_{i=1}^nX_iY_i-n\Xbar\Ybar
     & =\sum_{i=1}^n\left(X_iY_i-\Xbar\Ybar\right)                       \\
     & =\sum_{i=1}^n\left(X_iY_i-\Xbar\Ybar-\Xbar\Ybar+\Xbar\Ybar\right) \\
     & =\sum_{i=1}^n\left(X_iY_i-X_i\Ybar-\Xbar Y_i+\Xbar\Ybar\right)    \\
     & =\sum_{i=1}^n\left(X_i(Y_i-\Ybar)-\Xbar(Y_i-\Ybar)\right)         \\
     & =\sum_{i=1}^n(X_i-\Xbar)(Y_i-\Ybar),
  \end{align*}
  and
  \begin{align*}
    \sum_{i=1}^nX_i^2-n\Xbar^2
     & =\sum_{i=1}^n\left(X_i^2-\Xbar^2\right)           \\
     & =\sum_{i=1}^n\left(X_i^2-2\Xbar^2+\Xbar^2\right)  \\
     & =\sum_{i=1}^n\left(X_i^2-2X_i\Xbar+\Xbar^2\right) \\
     & =\sum_{i=1}^n\left(X_i-\Xbar\right)^2,
  \end{align*}
  it follows that
  \[
    \betahat_1
    =\frac{\sum_{i=1}^n(X_i-\Xbar)(Y_i-\Ybar)}{\sum_{i=1}^n\left(X_i-\Xbar\right)^2}.
  \]

  Note that
  \begin{align*}
    \widehat{\epsilon}_i
     & =Y_i-\Yhat_i                                                             \\
     & =\beta_1X_i+\beta_0+\epsilon_i
    -\betahat_1X_i-\betahat_0                                                   \\
     & =\beta_1X_i+\beta_0+\epsilon_i
    -\betahat_1X_i+\betahat_1\Xbar-\Ybar                                        \\
     & =(\epsilon_i+\beta_1\Xbar+\beta_0-\Ybar)-(\betahat_1-\beta_1)(X_i-\Xbar) \\
     & =(\epsilon_i-\epsbar)-(\betahat_1-\beta_1)(X_i-\Xbar),
  \end{align*}
  since
  \begin{align*}
    \beta_1\Xbar+\beta_0-\Ybar
     & =\beta_1\left(\sum_{i=1}^nX_i\right)+\beta_0-\sum_{i=1}^n Y_i                                                   \\
     & =\frac{\beta_1}{n}\left(\sum_{i=1}^n X_i\right)+\beta_0-\frac{1}{n}\sum_{i=1}^n (\beta_1X_i+\beta_0+\epsilon_i) \\
     & =-\frac{1}{n}\sum_{i=1}^n\epsilon_i.
  \end{align*}

  Therefore,
  \begin{align*}
    \var{\widehat{\epsilon}_i}
     & =\var{(\epsilon_i-\epsbar)-(\betahat_1-\beta_1)(X_i-\Xbar)}     \\
     & =\var{\epsilon_i-\epsbar}+\var{(\betahat_1-\beta_1)(X_i-\Xbar)}
    -2\cov{\epsilon_i-\epsbar, (\betahat_1-\beta_1)(X_i-\Xbar)}        \\
     & =\var{\epsilon_i-\epsbar}+(X_i-\Xbar)^2\var{\betahat_1-\beta_1}
    -2(X_i-\Xbar)\cov{\epsilon_i-\epsbar, \betahat_1-\beta_1},
  \end{align*}
  where
  \begin{align*}
    \var{\epsilon_i-\epsbar}
     & =\var{\epsilon_i}+\var{\epsbar}-2\cov{\epsilon_i,\epsbar}   \\
     & =\var{\epsilon_i}+\frac{1}{n^2}\var{\sum_{j=1}^n\epsilon_j}
    -\frac{2}{n}\cov{\epsilon_i,\sum_{j=1}^n\epsilon_j}            \\
     & =\sigma^2+\frac{n\sigma^2}{n^2}-\frac{2}{n}\sigma^2         \\
     & =\sigma^2\left(1-\frac{1}{n}\right),
  \end{align*}
  \[
    \var{\betahat_1-\beta_1}=\var{\betahat_1}=\frac{\sigma^2}{ns_X^2},
  \]
  \begin{align*}
    \betahat_1-\beta_1
     & =\frac{\sum_{j=1}^n(X_j-\Xbar)(Y_j-\Ybar)}{\sum_{j=1}^n(X_j-\Xbar)^2}-\beta_1                           \\
     & =\frac{\sum_{j=1}^n\left[(X_j-\Xbar)(Y_j-\Ybar)-\beta_1(X_j-\Xbar)^2\right]}{\sum_{j=1}^n(X_j-\Xbar)^2} \\
     & =\frac{\sum_{j=1}^n(X_j-\Xbar)(Y_j-\Ybar-\beta_1X_j+\beta_1\Xbar)}{\sum_{j=1}^n(X_j-\Xbar)^2}           \\
     & =\frac{\sum_{j=1}^n(X_j-\Xbar)(\epsilon_j-\epsbar)}{\sum_{j=1}^n(X_j-\Xbar)^2},
  \end{align*}
  \begin{align*}
    \cov{\epsilon_i-\epsbar, \betahat_1-\beta_1}
     & =\E{(\epsilon_i-\epsbar)(\betahat_1-\beta_1)}
    -\E{\epsilon_i-\epsbar}\E{\betahat_1-\beta_1}              \\
     & =\E{
      \frac{\sum_{j=1}^n(X_j-\Xbar)(\epsilon_i-\epsbar)(\epsilon_j-\epsbar)}{\sum_{j=1}^n(X_j-\Xbar)^2}
    }                                                          \\
     & =\frac{(X_i-\Xbar)
      \E{(\epsilon_i-\epsbar)^2}
      +\sum_{\substack{j=1                                     \\ j\neq i}}^n
      (X_i-\Xbar)\E{(\epsilon_i-\epsbar)(\epsilon_j-\epsbar)}
    }{\sum_{j=1}^n(X_j-\Xbar)^2}                               \\
     & =\frac{(X_i-\Xbar)\sigma^2}{\sum_{j=1}^n(X_j-\Xbar)^2},
  \end{align*}
  since
  \begin{align*}
    \E{(\epsilon_i-\epsbar)^2}
     & =\E{(\epsilon_i-\epsbar)^2}
    -\E{\epsilon_i-\epsbar}^2               \\
     & =\var{\epsilon_i-\epsbar}            \\
     & =\sigma^2\left(1-\frac{1}{n}\right),
  \end{align*}
  \begin{align*}
    \E{(\epsilon_i-\epsbar)(\epsilon_j-\epsbar)}
     & =\E{(\epsilon_i-\epsbar)(\epsilon_j-\epsbar)}
    -\E{\epsilon_i-\epsbar}\E{\epsilon_j-\epsbar}    \\
     & =\cov{\epsilon_i-\epsbar,\epsilon_j-\epsbar}  \\
     & =\cov{\epsilon_i,\epsilon_j}
    -\cov{\epsilon_i,\epsbar}
    -\cov{\epsbar,\epsilon_j}
    +\var{\epsbar}                                   \\
     & =-\frac{2\sigma^2}{n}+\frac{\sigma^2}{n}      \\
     & =-\frac{\sigma^2}{n}
  \end{align*}
  and
  \[
    \sum_{\substack{j=1                                       \\ j\neq i}}^n
    (X_j-\Xbar)\E{(\epsilon_i-\epsbar)(\epsilon_j-\epsbar)}
    =-\frac{\sigma^2}{n}\left(n\Xbar-X_i-(n-1)\Xbar\right)
    =\frac{\sigma^2}{n}\left(\Xbar-X_i\right).
  \]

  Therefore,
  \begin{align*}
    \var{\widehat{\epsilon}_i}
     & =\sigma^2\left(1-\frac{1}{n}\right)
    +(X_i-\Xbar)^2\frac{\sigma^2}{ns_X^2}
    -2(X_i-\Xbar)\frac{(X_i-\Xbar)\sigma^2}{ns_X^2} \\
     & =\sigma^2\left(1-\frac{1}{n}\right)
    -(X_i-\Xbar)^2\frac{\sigma^2}{ns_X^2},
  \end{align*}
  and
  \begin{align*}
    \sum_{i=1}^n\var{\widehat{\epsilon}_i}
     & =\sigma^2\sum_{i=1}^n\left(
    \left(1-\frac{1}{n}\right)
    -(X_i-\Xbar)^2\frac{1}{ns_X^2}
    \right)                                             \\
     & =n\sigma^2\left(\frac{n-1}{n}-\frac{1}{n}\right) \\
     & =(n-2)\sigma^2.
  \end{align*}
  Hence,
  \begin{align*}
    \E{\frac{1}{n-2}\sum_{i=1}^n\widehat{\epsilon}_i^2}
     & =\frac{1}{n-2}\sum_{i=1}^n\E{\widehat{\epsilon}_i^2}                                                     \\
     & =\frac{1}{n-2}\left[\sum_{i=1}^n\var{\widehat{\epsilon}_i}+\sum_{i=1}^n\E{\widehat{\epsilon}_i}^2\right] \\
     & =\sigma^2.
  \end{align*}
\end{ex}

\begin{ex}
  We begin by rewriting $\betahat_1$ in a more convenient form.
  \begin{align*}
    \betahat_1
     & =\frac{\sum_{i=1}^n(X_i-\Xbar)(Y_i-\Ybar)}{\sum_{i=1}^n(X_i-\Xbar)^2}               \\
     & =\frac{\sum_{i=1}^n(X_iY_i-\Xbar\Ybar)}{\sum_{i=1}^n(X_i-\Xbar)^2}                  \\
     & =\frac{\sum_{i=1}^n(X_i(\beta_1X_i+\beta_0+\epsilon_i)
      -\Xbar(\beta_1\Xbar+\beta_0+\epsbar))}{\sum_{i=1}^n(X_i-\Xbar)^2}                    \\
     & =\frac{\beta_1\sum_{i=1}^nX_i^2+n\beta_0\Xbar+\sum_{i=1}^n X_i\epsilon_i
      -n\beta_1\Xbar^2-n\beta_0\Xbar-n\epsbar\Xbar}{\sum_{i=1}^n(X_i-\Xbar)^2}             \\
     & =\beta_1+\frac{\sum_{i=1}^nX_i\epsilon_i-n\epsbar\Xbar}{\sum_{i=1}^n(X_i-\Xbar)^2}  \\
     & =\beta_1+\frac{\sum_{i=1}^n(X_i\epsilon_i-\epsbar\Xbar)}{\sum_{i=1}^n(X_i-\Xbar)^2} \\
     & =\beta_1+\frac{\sum_{i=1}^n(X_i-\Xbar)\epsilon_i}{\sum_{i=1}^n(X_i-\Xbar)^2}
  \end{align*}

  Therefore,
  \begin{align*}
    \E{\betahat_1}
     & =\E{\beta_1+\frac{\sum_{i=1}^n(X_i-\Xbar)\epsilon_i}{\sum_{i=1}^n(X_i-\Xbar)^2}} \\
     & =\beta_1+\frac{\sum_{i=1}^n(X_i-\Xbar)\E{\epsilon_i}}{\sum_{i=1}^n(X_i-\Xbar)^2} \\
     & =\beta_1,
  \end{align*}
  and
  \begin{align*}
    \E{\betahat_0}
     & =\E{\Ybar-\betahat_1\Xbar}                        \\
     & =\E{\beta_1\Xbar+\beta_0-\epsbar-\betahat_1\Xbar} \\
     & =\beta_0+\E{\beta_1-\betahat_1}\Xbar-\E{\epsbar}  \\
     & =\beta_0.
  \end{align*}

  Likewise,
  \begin{align*}
    \var{\betahat_1}
     & =\var{\beta_1+\frac{\sum_{i=1}^n(X_i-\Xbar)\epsilon_i}{\sum_{i=1}^n(X_i-\Xbar)^2}}          \\
     & =\frac{\sum_{i=1}^n(X_i-\Xbar)^2\var{\epsilon_i}}{\left(\sum_{i=1}^n(X_i-\Xbar)^2\right)^2} \\
     & =\frac{\sigma^2}{ns_X^2},
  \end{align*}
  \begin{align*}
    \cov{\epsbar, \betahat_1}
     & =\cov{\epsbar, \beta_1+\frac{\sum_{i=1}^n(X_i-\Xbar)\epsilon_i}{\sum_{i=1}^n(X_i-\Xbar)^2}} \\
     & =\frac{1}{s_X^2}\cov{\sum_{i=1}^n\epsilon_i, \sum_{i=1}^n(X_i-\Xbar)\epsilon_i}             \\
     & =\frac{1}{s_X^2}\sum_{i=1}^n(X_i-\Xbar)\sigma^2                                             \\
     & =0,
  \end{align*}
  \begin{align*}
    \var{\betahat_0}
     & =\var{\Ybar-\betahat_1\Xbar}                                           \\
     & =\var{\beta_1\Xbar+\beta_0+\epsbar-\betahat_1\Xbar}                    \\
     & =\var{\epsbar}+\Xbar^2\var{\betahat_1}-2\Xbar\cov{\epsbar, \betahat_1} \\
     & =\frac{\sigma^2}{n}+\Xbar^2\frac{\sigma^2}{ns_X^2}                     \\
     & =\frac{\sigma^2}{ns_X^2}\left(s_X^2+\Xbar^2\right)                     \\
     & =\frac{\sigma^2\left(\frac{1}{n}\sum_{i=1}^nX_i^2\right)}{ns_X^2},
  \end{align*}
  and
  \begin{align*}
    \cov{\betahat_0, \betahat_1}
     & =\cov{\Ybar-\betahat_1\Xbar, \betahat_1}         \\
     & =\cov{\beta_1\Xbar+\beta_0+\epsbar, \betahat_1}
    -\Xbar\cov{\betahat_1, \betahat_1}                  \\
     & =\cov{\epsbar, \betahat_1}-\Xbar\var{\betahat_1} \\
     & =-\Xbar\frac{\sigma^2}{ns_X^2}.
  \end{align*}
\end{ex}

\begin{ex}
  Assume that $Y_i=\beta X_i+\epsilon_i$, with $\cE{\epsilon_i}{X_i}=0$
  and $\cVar{\epsilon_i}{X_i}=\sigma^2$.

  The least squares estimate for $\beta$ is the value $\betahat$ that minimizes
  $\textsc{rss}=\sum_{i=1}^n(Y_i-\betahat X_i)^2$. This is a quadratic function
  in $\betahat$ with a positive leading coefficient and therefore will achieve
  its minimum at the critical point. We have
  \[
    0
    =\frac{\pd RSS}{\pd\betahat}
    =2\sum_{i=1}^nX_i(Y_i-\betahat X_i)
    =2\sum_{i=1}^n X_iY_i-2\betahat\sum_{i=1}^n X_i^2,
  \]
  which implies that
  \[
    \betahat
    =\frac{\sum_{i=1}^n X_iY_i}{\sum_{i=1}^n X_i^2}
    =\frac{\sum_{i=1}^n X_i(\beta X_i+\epsilon_i)}{\sum_{i=1}^n X_i^2}
    =\beta+\frac{\sum_{i=1}^n X_i\epsilon_i}{\sum_{i=1}^n X_i^2}.
  \]

  Note that then
  \[
    \E{\betahat}
    =\E{\beta+\frac{\sum_{i=1}^n X_i\epsilon_i}{\sum_{i=1}^n X_i^2}}
    =\beta+\frac{\sum_{i=1}^n X_i\E{\epsilon_i}}{\sum_{i=1}^n X_i^2}
    =\beta,
  \]
  and
  \[
    \var{\betahat}
    =\var{\beta+\frac{\sum_{i=1}^n X_i\epsilon_i}{\sum_{i=1}^n X_i^2}}
    =\frac{\sum_{i=1}^n X_i^2\var{\epsilon_i}}{\left(\sum_{i=1}^n X_i^2\right)^2}
    =\frac{\sigma^2}{\sum_{i=1}^nX_i^2}
  \]

  Therefore,
  \[
    \sehat(\betahat)=\frac{\sigmahat^2}{\sqrt{\sum_{i=1}^nX_i^2}}.
  \]

  Finally, note that
  \[
    \betahat
    =\beta+\frac{n^{-1}\sum_{i=1}^n X_i\epsilon_i}{n^{-1}\sum_{i=1}^n X_i^2},
  \]
  and that therefore, assuming the $X_i$ are \textsc{iid}, with finite mean
  $\mu$ and variance $\rho^2$, but not identically equal to zero, it follows by
  the weak law of large numbers that
  \[
    n^{-1}\sum_{i=1}^n X_i\epsilon_i\xrightarrow{P} \mu\cdot 0 =0,\quad
    n^{-1}\sum_{i=1}^n X_i^2\xrightarrow{P}\rho^2+\mu^2
  \]
  and that therefore, by Slutsky's theorem,
  \[
    \betahat\xrightarrow{P} \beta.
  \]
\end{ex}

\newcommand{\Rtr}{\widehat{R}_\text{tr}}

\begin{ex}
  We have
  \begin{align*}
    \E{\Rtr(S)}-R(S)
     & =\E{\sum_{i=1}^n(\Yhat_i-Y_i)^2}-\sum_{i=1}^n\E{\left(\Yhat_i-Y_i^*\right)}^2 \\
     & =\sum_{i=1}^n\left[\E{\Yhat_i^2}-2\E{Y_i\Yhat_i}+\E{Y_i^2}
    -\E{\Yhat_i^2}+2\E{Y_i^*\Yhat_i}-\E{Y_i^{*,2}}\right]                            \\
     & =\sum_{i=1}^n\left[-2\E{Y_i\Yhat_i}+\E{Y_i^2}
    +2\E{Y_i^*\Yhat_i}-\E{Y_i^{*,2}}\right]                                          \\
     & =-2\sum_{i=1}^n\left[\E{Y_i\Yhat_i}-\E{Y_i^*\Yhat_i}\right],
  \end{align*}
  since
  \begin{align*}
    \E{Y_i^{*,2}}
     & =\var{X_i\beta+\epsilon_i^*}+\left[\E{X_i\beta+\epsilon_i^*}\right]^2           \\
     & =\var{X_i\beta}+\var{\epsilon_i^*}+\left[\E{X_i\beta}+\E{\epsilon_i^*}\right]^2 \\
     & =\var{X_i\beta}+\var{\epsilon_i}+\left[\E{X_i\beta}+\E{\epsilon_i}\right]^2     \\
     & =\var{X_i\beta + \epsilon_i}+\left[\E{X_i\beta + \epsilon_i}\right]^2           \\
     & =\E{Y_i^2}.
  \end{align*}

  Finally, note that $Y_i^*$ and $\Yhat_i$ are independent, since the $Y_i^*$'s
  denote the values of different observations of $Y_i$ from the ones we used to
  compute the value of $\betahat$. Hence,
  \begin{align*}
    \E{\Rtr(S)}-R(S)
     & =-2\sum_{i=1}^n\left[\E{Y_i\Yhat_i}-\E{Y_i^*}\E{\Yhat_i}\right] \\
     & =-2\sum_{i=1}^n\left[\E{Y_i\Yhat_i}-\E{Y_i}\E{\Yhat_i}\right]   \\
     & =-2\sum_{i=1}^n\cov{\Yhat_i, Y_i}.
  \end{align*}
\end{ex}

% 5
\begin{ex}
  Note that $H_0:\beta_1=17\beta_0$ is equivalent to $H_0:\beta_1-17\beta_0=0$.
  We then have
  \begin{align*}
    \se(\beta_1-17\beta_0)
     & =\sqrt{\var{\beta_1}+\var{-17\beta_0}}                                                                    \\
     & =\sqrt{\frac{\sigma^2}{ns^2_X}+17^2\frac{\sigma^2\sum_{i=1}^nX_i^2}{n^2s^2_X}} & (\text{by Theorem 13.8}) \\
     & =\frac{\sigma}{ns_X}\sqrt{n+289\sum_{i=1}^nX_i^2}.
  \end{align*}
  The size $\alpha$ Wald test is then: reject $H_0$ when
  \[
    \left|\frac{\betahat_1-17\betahat_0}{\frac{\sigma}{ns_X}\sqrt{n+289\sum_{i=1}^nX_i^2}}\right|
    >z_{\alpha/2}.
  \]
\end{ex}

\begin{ex}
  \begin{enumerate}[(a)]
    \item~
          \inputminted{python}{../code/ex13_06a.py}
          \inputminted{text}{../output/ex13_06a.txt}

          \begin{figure}[H]
            \centering
            \includegraphics{../images/13-06a}
            \caption{Plot of the data with the fitted line from the linear model.}
          \end{figure}
    \item
          \inputminted{python}{../code/ex13_06b.py}
          \inputminted{text}{../output/ex13_06b.txt}

          \begin{figure}[H]
            \centering
            \includegraphics{../images/13-06b}
            \caption{Plot of the $\log(\text{MPG})$ response data with the
              fitted line from the linear model.}
          \end{figure}

          A linear model seems to fit the $\log(\text{MPG})$ response data
          better than then original $\text{MPG}$ response.
  \end{enumerate}
\end{ex}

\begin{ex}
  \begin{enumerate}[(a)]
    \item~
          \inputminted{python}{../code/ex13_07a.py}
          \inputminted{text}{../output/ex13_07a.txt}
    \item
          \inputminted{python}{../code/ex13_07b.py}
          \inputminted{text}{../output/ex13_07b.txt}
    \item
          \inputminted{python}{../code/ex13_07c.py}
          \inputminted{text}{../output/ex13_07c.txt}
    \item
          \inputminted{python}{../code/ex13_07d.py}
          \inputminted{text}{../output/ex13_07d.txt}
  \end{enumerate}
\end{ex}

\begin{ex}
  Let $Y=X\beta+\epsilon$. Suppose that $\epsilon_i\,|\,X_i\sim N(0,\sigma^2)$,
  and note that then by an identical argument to Section 13.2,
  \[
    \ell_S
    =-n\log{\sigma}
    -\frac{1}{2\sigma^2}\sum_{i=1}^n\left(Y_i-X_i\beta\right)^2.
  \]
  We then have
  \begin{align*}
    -2\sigma^2(\ell_S-|S|)-2n\sigma^2\log{\sigma}
     & =-2\sigma^2\left(
    -n\log{\sigma}-\frac{1}{2\sigma^2}\sum_{i=1}^n\left(Y_i-X_i\beta\right)^2-|S|
    \right)-2n\sigma^2\log{\sigma}                                                              \\
     & =\sum_{i=1}^n(Y_i-X_i\beta)^2+2|S|\sigma^2+2n\sigma^2\log{\sigma}-2n\sigma^2\log{\sigma} \\
     & =\Rtr+2|S|\sigma^2,
  \end{align*}
  i.e.\ Mallow's $C_p$ statistic is equal to a negative constant times AIC plus
  a constant. Therefore, maximizing AIC is equivalent to minimizing Mallow's
  $C_p$.
\end{ex}

\begin{ex}
  \begin{enumerate}[(a)]
    \item We have
          \begin{align*}
            \P{J_n=0}
             & =\P{AIC_0 > AIC_1}                                                           \\
             & =\P{\ell(0) > \ell(\thetahat)-1}                                             \\
             & =\P{-\frac{1}{2}\sum_{i=1}^nX_i^2 > -\frac{1}{2}\sum_{i=1}^n(X_i-\Xbar)^2-1} \\
             & =\P{\sum_{i=1}^nX_i^2-\sum_{i=1}^n(X_i-\Xbar)^2 < 2}                         \\
             & =\P{\sum_{i=1}^n\left[X_i^2-X_i^2+2X_i\Xbar-\Xbar^2 \right] < 2}             \\
             & =\P{n\Xbar^2 < 2}.
          \end{align*}
          If $\mathcal{M}_0$ is the true model, i.e.\ $\theta=0$, we have
          \[
            \P{J_n=0}
            =\P{|\sqrt{n}\Xbar| < \sqrt{2}}
            =\Phi(\sqrt{2})-\Phi(-\sqrt{2})
            \approx 0.8427.
          \]
          Otherwise,
          \begin{align*}
            \P{J_n=0}
             & =\P{n\Xbar^2 < 2}                                                                \\
             & =\P{-\sqrt{\frac{2}{n}} < \Xbar < \sqrt{\frac{2}{n}}}                            \\
             & =\P{-\sqrt{\frac{2}{n}}-\theta < \Xbar-\theta < \sqrt{\frac{2}{n}}-\theta}       \\
             & =\P{-\sqrt{2}-\sqrt{n}\theta < \sqrt{n}(\Xbar-\theta) < \sqrt{2}-\sqrt{n}\theta} \\
             & =\Phi(\sqrt{2}-\sqrt{n}\theta)-\Phi(-\sqrt{2}-\sqrt{n}\theta),
          \end{align*}
          and therefore
          \[
            \lim_{n\to\infty}\P{J_n=0}
            =\lim_{n\to\infty}\left[\Phi(\sqrt{2}-\sqrt{n}\theta)-\Phi(-\sqrt{2}-\sqrt{n}\theta)\right]
            =0.
          \]
    \item We proved earlier that assuming $\theta\neq 0$,
          $\widehat{f}_n\xrightarrow{P}\phi_{\thetahat}$. Note that
          \begin{align*}
            D(\phi_\theta, \phi_{\thetahat})
             & =\int \frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\theta)^2}{2}\right\}
            \log\left(\frac{\frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\theta)^2}{2}\right\}}{\frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\thetahat)^2}{2} \right\}}\right)\,\d{x} \\
             & =\int \frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\theta)^2}{2}\right\}
            \left(\frac{(x-\thetahat)^2}{2}-\frac{(x-\theta)^2}{2}\right)\,\d{x}                                                                                \\
             & =\frac{1}{2}\int \frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\theta)^2}{2}\right\}
            \left((\thetahat^2-\theta^2)-2x(\thetahat-\theta)\right)\,\d{x}                                                                                \\
             & =\frac{\thetahat^2-\theta^2}{2}-\theta(\thetahat-\theta)                                                                                \\\
             & =\frac{1}{2}(\thetahat-\theta)^2,
          \end{align*}
          which converges in probability to $0$ since $\thetahat-\theta$
          converges in probability to $0$.
    \item We have
          \begin{align*}
            \P{J_n=0}
             & =\P{BIC_0 > BIC_1}                                                                            \\
             & =\P{\ell(0) > \ell(\thetahat)-\frac{1}{2}\log{n}}                                             \\
             & =\P{-\frac{1}{2}\sum_{i=1}^nX_i^2 > -\frac{1}{2}\sum_{i=1}^n(X_i-\Xbar)^2-\frac{1}{2}\log{n}} \\
             & =\P{\sum_{i=1}^nX_i^2-\sum_{i=1}^n(X_i-\Xbar)^2 < \log{n}}                                    \\
             & =\P{\sum_{i=1}^n\left[X_i^2-X_i^2+2X_i\Xbar-\Xbar^2 \right] < \log{n}}                        \\
             & =\P{-\sqrt{\frac{\log{n}}{n}}<\Xbar < \sqrt{\frac{\log{n}}{n}}}                               \\
             & =\P{-\sqrt{\log{n}}-\sqrt{n}\theta <\sqrt{n}(\Xbar-\theta) < \sqrt{\log{n}}-\sqrt{n}\theta}.
          \end{align*}
          Assuming the true model is $\theta=0$,
          \begin{align*}
            \lim_{n\to\infty}\P{J_n=0}
             & =\lim_{n\to\infty}\P{-\sqrt{\log{n}}<\sqrt{n}\Xbar < \sqrt{\log{n}}}          \\
             & =\lim_{n\to\infty}\Phi(\sqrt{\log{n}})-\lim_{n\to\infty}\Phi(-\sqrt{\log{n}}) \\
             & =0,
          \end{align*}
          while if $\theta\neq 0$,
          \begin{align*}
            \lim_{n\to\infty}\P{J_n=0}
             & =\lim_{n\to\infty}\Phi(\sqrt{\log{n}}-\sqrt{n}\theta)
            -\lim_{n\to\infty}\Phi(-\sqrt{\log{n}}-\sqrt{n}\theta)
            =0.
          \end{align*}
  \end{enumerate}
\end{ex}

% 10
\begin{ex}
  \begin{enumerate}[(a)]
    \item Note that
          \[
            \P{\Yhat_*-2s < Y_* < \Yhat_*+2s}
            =\P{-2 < \frac{Y_*-\Yhat_*}{s} < 2},
          \]
          where $\var{Y_*-\Yhat_*}=\var{\epsilon}+\var{\Yhat_*}=\sigma^2+s^2$
          and $\E{Y_*-\Yhat_*}=0$. Therefore,
          \[
            \frac{Y_*-\Yhat_*}{s}\approx N\left(0, 1+\frac{\sigma^2}{s^2}\right),
          \]
          and thus
          \[
            \P{-2 < \frac{Y_*-\Yhat_*}{s} < 2}\neq 0.95.
          \]
    \item We have
          \begin{align*}
            \var{Y_*-\Yhat_*}
             & =\var{Y_*-\Yhat_*}                                                                     \\
             & =\var{\beta_0+\beta_1X_*+\epsilon-\betahat_0-\betahat_1X_*}                            \\
             & =\var{\betahat_0+\betahat_1X_*}+\var{\epsilon}                                         \\
             & =\var{\betahat_0}+X_*^2\var{\betahat_1}+2X_*\cov{\betahat_0,\betahat_1}+\var{\epsilon} \\
             & =\frac{\sigma^2}{n^2s_X^2}
            \sum_{i=1}^n\left(X_i^2-2X_*X_i+X_*^2
            \right)+\sigma^2                                                                          \\
             & =\left[\frac{\sum_{i=1}^n(X_i-X_*)^2}{n\sum_{i=1}^n(X_i-\Xbar)^2}+1\right]\sigma^2     \\
             & =\xi_n^2,
          \end{align*}
          and therefore,
          \begin{align*}
            \P{\Yhat_*-2\widehat{\xi}_n < Y_* < \Yhat_*+2\widehat{\xi}_n}
             & =\P{-2 < \frac{Y_*-\Yhat_*}{\widehat{\xi}_n} < 2} \\
             & \approx \P{-2 < N(0,1) < 2}                       \\
             & \approx 0.95.
          \end{align*}
  \end{enumerate}
\end{ex}

\begin{ex}~
  \inputminted{python}{../code/ex13_11.py}
  \inputminted{text}{../output/ex13_11.txt}
\end{ex}


\section{Results}

\begin{frame}{Results}

  Revisiting the model
  $$\begin{cases}
    \begin{array}{rl}
      dM\hspace{-.8em}&=[aMC - \frac{gM(t-\tau)}{1-C(t-\tau)} + \gamma M (1-M-C)]dt+\beta M(1-M)dW,\\
      dC\hspace{-.8em}&=[rC(1-M-C) - dC - aMC]dt.\\
    \end{array}
    \end{cases}$$\\
  We are interested in the effects of:
  \begin{itemize}
  \item $\theta$ (the initial condition)\\
  \item $\beta$\\
  \item $\tau$\\
  \item $g$\\
  \end{itemize}

\end{frame}

\begin{frame}{Theta}
\includegraphics[scale=.325]{nullclinesarc.png}

\end{frame}

\begin{frame}{The Simulation}
We sampled the following values:
\begin{itemize}
\item $\theta$ = $\frac{\pi}{42}$ to $\frac{20\pi}{42}$ by $\frac{\pi}{42}$\\
\item $\beta$ = 0 to 1 by 0.05\\
\item $\tau$ = 0.5 to 1 by 0.05\\
\item $g$ = 0.2 to 0.8 by 0.1\\
\end{itemize}
We performed Monte Carlo simulations.
\end{frame}

% %\begin{frame}\frametitle{At First Glance}
% %\includegraphics[scale=.325]{qqplot_lgcoral.png}
% %\end{frame}

\begin{frame}
\includegraphics[scale=.325]{scatter_lgcoral_gpt4_betapt4_theta10.png}
\end{frame}

\begin{frame}{Initial Conditions}
\includegraphics[scale=.325]{scatter_gpt4_betapt4.png}
\end{frame}

\begin{frame}{The Three Models}
Based on the plot, we will try the following three models
\begin{itemize}
\item Binomial Logit\\
Fits the model $logit(p)=a+b\theta$ where $logit(p)=log(\frac{p}{1-p})$\\
\item Poisson Log\\
Fits the model $log(n)=a+b\theta$\\
\item Ordinary Least Squares (OLS)\\
Fits the model $p=a+b\theta$\\
\end{itemize}
\end{frame}

\begin{frame}\frametitle{The Binomial Logit Model}
{\fontsize{8}{3} \color{RBlue} \verbatiminput{theta_logit_summary.txt}}

Conducting a goodness of fit test: \\
$P(\chi^{2}_{1}>10485-2171)<0.0001$\\

\end{frame}

\begin{frame}\frametitle{The Poisson Log Model}
{\fontsize{8}{3} \color{RBlue} \verbatiminput{theta_log_summary.txt}}

Conducting a goodness of fit test: \\
$P(\chi^{2}_{1}>3714.9-1970.4)<0.0001$\\

\end{frame}

%\begin{frame}\frametitle{Problematic Poisson}
%From the summary statistics, there is a problem with overdispersion.\\
%\\
%$\frac{G^2}{df}\approx1$\\
%$\frac{2006.7}{37}\approx54.2$\\
%\\
%Therefore, we fit a negative binomial model.\\
%\end{frame}

%\begin{frame}\frametitle{The Negative Binomial}

%\end{frame}

\begin{frame}\frametitle{Problematic Poisson}
There may be a problem with the model.
In the data, there is an overabundance of zeros.
Therefore, we fit a zero-inflated Poisson model (ZIP).
{\fontsize{8}{3} \color{RBlue} \verbatiminput{theta_zeros.txt}}
\end{frame}

\begin{frame}\frametitle{The ZIP Model}
{\fontsize{8}{3} \color{RBlue} \verbatiminput{theta_log_zip_summary.txt}}
\end{frame}

\begin{frame}\frametitle{The OLS Model}
{\fontsize{8}{3} \color{RBlue} \verbatiminput{theta_ols_summary.txt}}

Conducting a goodness of fit test: \\
$P(\chi^{2}_{1}>4.5824-1.4317)\approx0.07589$\\

\end{frame}

\begin{frame}\frametitle{Comparing the Models}
Our three models are \\
\begin{itemize}
\item Binomial Logit $p=\frac{e^{-2.48378+6.01631\theta}}{1+e^{-2.48378+6.01631\theta}}$\\
\item Poisson Log $n=e^{4.89295+0.90423\theta}$\\
\item OLS $p=0.22856+0.64310\theta$\\
\end{itemize}
How do we determine which model is the most appropriate?\\
\end{frame}

\begin{frame}{Binomial Logit, Poisson Log, or OLS}
\includegraphics[scale=.325]{theta_threemodels.png}
\end{frame}

\begin{frame}\frametitle{The Whole Picture}
\includegraphics[scale=.325]{allscatter11.png}
\end{frame}

\begin{frame}\frametitle{The Binomial Logit}
{\fontsize{8}{3} \color{RBlue} \verbatiminput{all_logit_summary.txt}}
\end{frame}

\begin{frame}\frametitle{Interpreting the Results}
Keeping theta and beta constant, an \textbf{increase} in g of:
\begin{itemize}
\item 0.01 has an odds ratio of coral survival of 3.21\\
\item 0.1 has an odds ratio of coral survival of 116413.1
\end{itemize}
\end{frame}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "Presentation2"
%%% End:


% !TEX root = hw3.tex

\section{Influence Maximization [25 points]}

In class we discussed the influence maximization problem and the greedy hill-climbing approach to solving it. In the algorithm, we add nodes to the current seed set one at a time. At step 0, we have an empty set $S_0$. At step $i > 0$, we pick the node which maximizes the marginal gain: $S_i = S_{i-1} \cup \left\{ \arg \max_u f(S_{i-1} \cup \{u\})-f(S_{i-1})\right\}$, where $f(S)$ denotes the number of nodes influenced by the initially active set $S$ (includes the set $S$ itself).

As we showed in  class the hill climbing algorithm cannot guarantee an optimal solution. In other words, there might exist a set $T$ with $|T| = i$ such that $f(S_i) < f(T)$. It is also known that the greedy algorithm is a $(1-(1-\frac{1}{k})^k)$ - approximation for the influence maximization problem (its sub-optimality is bounded). In other words, for every $k \geq 1$, the greedy hill-climbing algorithm outputs a set $S_k$ such that $f(S_k) \geq (1-(1-\frac{1}{k})^k) f(T)$ where T is the optimal set for influence maximization.

Parts 1 and 2 of this problem ask you to construct examples where the greedy hill climbing returns a sub-optimal solution. In part 2, you will explore examples close to the non-optimal bound for the greedy algorithm for $k=3$. Your answer should consist of: (1) For every node $u$ its influence set $X_u$ (you can describe the set or draw a directed graph where an edge from $A$ to $B$ indicates that node $A$ influences node $B$ with probability $1$),
(2) $S_i$, the set of nodes that a greedy hill climbing would choose after $i$ iterations, and (3) $T$, the optimal set of $i$ nodes.

For all the questions, you can assume: (1) The nodes in $S$ are influencing themselves, i.e., the count of total influence $f(S)$ includes the nodes in $S$. (2) The influence set $X_{u}$ contains all nodes that are influenced by node $u$, both directly and eventually. (3) When several nodes have the same level of marginal gain, we choose one of them at random.\\

\subsection{Non-Optimal Hill-Climbing [8 points]} For $k = 2$, construct an example where $f(S_k) < f(T)$. That is, hill-climbing will only find a non-optimal solution. (Hint: the last step of the greedy approach is optimal given the $k-1$ previous steps.)\\

\subsection{Bounded Non-Optimal Hill-Climbing [8 points]}

For $k= 3$, construct an example where  $f(S_k) \leq 0.8 f(T)$. That is, hill-climbing will only find a solution that is at most 80\% of the optimal solution.\\
( Note: 0.8 is very close to the true lower bound for $f(S_k)$ when $k=3$ which is approximately 0.70 )

\subsection{Optimality of Hill-Climbing [4 points]} Give a property of influence sets $X_u$ such that $f(S_i) = f(T)$.  In other words, what is a \emph{sufficient} property of influence sets of nodes such that greedy hill-climbing always outputs a set which achieves the maximum possible influence for a set of $i$ nodes? If your condition holds, the algorithm should produce an optimal output for any choice of $i$ between 1 and the number of nodes in the graph. The property does not need to be a necessary one. It must be a property of $X_u$. Properties such as ``the network has only $i$ nodes'' are not valid as correct answers.

There are several correct answers; we will accept all reasonable answers.\\

\subsection{More Hill-Climbing... [5 points]}

Assume that we stop hill-climbing after $k$ steps and $|S_k| = |T| = k$. Recall that in the class we proved a bound in the form of
\begin{equation}
f(T) \leq f(S_k) + \sum_{i=1}^k \delta_i,
\end{equation}
where $ \delta_1, .., \delta_k $ are the largest $k$ values of $f(S_k \cup \{u\}) - f(S_k)$ for any node $u$ in the graph. Construct a family of examples for which $f(S_k) + \sum_{i=1}^k \delta_i - f(T)$ can be arbitrarily large.

To be more specific, given any number $b$ you should exhibit a network (graph) such that $f(S_k) + \sum_{i=1}^k \delta_i - f(T)>b$.

Note: A {\em family} of examples is a set $F$ of examples such that for any number $b$, there exists a network (graph) $E(b) \in F$ (corresponding to $b$) such that $f(S_k) + \sum_{i=1}^k \delta_i - f(T)>b$.

\subsection*{What to submit}
\begin{enumerate}[{Page} 1:]
\setcounter{enumi}{12}
\item
\begin{itemize} \item Submit an example: $X_u$, $S$, $T$. \end{itemize}

\item
\begin{itemize} \item  Submit an example: $X_u$, $S$, $T$. \end{itemize}

\item
\begin{itemize} \item Submit a property and a brief explanation. \end{itemize}

\item
\begin{itemize} \item  Submit a family of examples and a brief explanation. \end{itemize}
\end{enumerate}


\section{Miscellaneous}%
\label{sec:miscellaneous}

\begin{thm}[Markov Inequality]
    If $X$ is a non-negative RV and $a > 0$ then $\prb(X \ge a) \le \frac{\E X}{a}$.
\end{thm}

\begin{thm}[Binomial $p_n$]
    If $ \frac{1}{n} = o_p(p_n)$ then
    \begin{equation*}
        \frac{S_n - n p_n}{\sqrt{n p_n q_n}} \convl N(0, 1).
    \end{equation*}
\end{thm}