\section{Monte Carlo Variance Reduction}
\label{sec:MCvar}

Monte Carlo methods for radiation transport are used in the nuclear
engineering community for a wide spectrum of application problems.
Monte Carlo methods aim to emulate
the transport of a particle from birth, through physical interaction, to death
by randomly sampling the probabilities of physics that the particle could
encounter, e.g.\ particle production, elastic and inelastic
scattering, absorption, and so forth.
This process of transporting a single particle
is repeated many times, to simulate the transport of many particles
throughout the problem. When the user achieves a
sufficient
number of samples--or particles--to reach the desired statistical precision for
the region of interest, the
simulation is complete. However, this naive approach to simulating each
particle--disregarding whether it is likely to
contribute to the tallied result--can be extraordinarily
computationally inefficient depending on the problem. A
code could waste time simulating millions of ``unusable'' particles and still not
reach the desired statistical precision for the tally. Variance
reduction techniques were developed to address this issue. In general, these
techniques bias the Monte Carlo transport to more effectively
contribute to a particular result, while not fundamentally changing the nature
of the problem being solved.

\subsection{Statistical Background}
\label{subsec:StatBkgnd}

Variance reduction techniques are rooted in statistics, so we begin our
discussion of variance reduction techniques with a brief primer on the
statistical background relevant to Monte Carlo radiation transport. Sections
\ref{subsubsec:PopStat} through \ref{subsubsec:FOM} are summarized from
\cite{lewis_computational_1984} and \cite{mcnp_manual_v1}.
Monte Carlo
methods transport many randomly sampled
particles, and when those particles reach a region of interest, they are scored
in a tally. The statistical precision of the tally
will reflect the total number of particles that were sampled in a chosen region
or at a chosen surface.
The reliability of the answer obtained in this region is then dependent
on the quantity and the history of the particles sampled.

\subsubsection{Population Statistics}
\label{subsubsec:PopStat}

In radiation transport, one desires to estimate some response in phase-space.
This response is the average behavior of the physical interactions in some
differential phase-space in energy, space, and time. If the probability density
function, $f(x)$, for the response is known exactly,
then the response in $dx$ can be calculated exactly by the true
mean, or
\begin{equation}
  \bar{x} = \int_{-\infty}^{\infty} xf(x) dx .
\end{equation}
Rarely is $f(x)$ known
exactly, so instead it is sampled.
Using $N$ randomly sampled particles, the estimate of the true mean value is given as
\begin{equation}
  \hat{x} = \frac{\sum_{i=1}^{N}{x_i}}{N} ,
\end{equation}
where $x_i$ is the i$^{th}$ event. $\hat{x}$ is the sample mean, or the
estimated value of $\bar{x}$
based on the $N$ number of samples that were used to calculate $\hat{x}$. As $N
\rightarrow \infty$, $\hat{x}$ will $\rightarrow \bar{x}$, which is given by the
Strong Law of Large Numbers \cite{mcnp_manual_v1}.
$\hat{x}$ in itself is a useful measure, but determining the spread of values
about $\hat{x}$ is also an important measure. This is called the variance. The
true variance of the distribution is
\begin{equation}
  \sigma^{2}\big( x \big) = \bar{x^2} - \bar{x}^2 ,
\end{equation}
and the standard deviation is the square root of the variance
\begin{equation}
  \sigma\big(x \big) = \big( \bar{x^2} - \bar{x}^2 \big)^{1/2}.
\end{equation}
The variance of the sampled distribution differs, as a finite number of samples
are used to calculate $\bar{x}$ and $\sigma$. The sample variance is defined by:
\begin{equation}
S^{ 2 }=\sum _{ i=1 }^{ N }{ \frac { (x_{ i }-\hat { x } )^{ 2 } }{ N-1 }  }
             \cong \widehat{x^2}-\hat{x}^2 ,
\label{eq:Var}
\end{equation}
where
\begin{equation}
  \widehat{x^2} = \frac{1}{N}\sum_{i=1}^{N} x_i^2 ,
\end{equation}
and the sample standard deviation is given by
\begin{equation}
  S = \big( \widehat{x^2}-\hat{x}^2 \big)^{(1/2)} .
\end{equation}

For \eqref{eq:Var} to hold true, the number of N samples must be large.
$S^2$ is the sample estimate of the true variance, $\sigma^2$.
The variance of the estimate of the mean value about $\bar{x}$ is:
\begin{equation}
S^{ 2 }_{ \hat { x }  }=\frac{S^2}{N}.
\label{eq:VarMean}
\end{equation}
From \eqref{eq:VarMean}, one can see the relationship between the sample standard
deviation and the standard error of $\hat{x}$ about $\bar{x}$ is
\begin{equation}
S_{ \hat { x }  }=\sqrt { \frac { S^{ 2 } }{ N }  } =\frac { S }{ \sqrt { N }}.
\label{eq:VarN}
\end{equation}
$S_{\hat{x}}$ is the standard error of the estimate of the sample mean.
The relative error normalizes the standard error by the estimate of the mean
\begin{equation}
R = \frac{S_{ \hat { x }  }}{\hat{x}} .
\label{eq:RelativeErr}
\end{equation}
As a result, $S$, $R$, and $N$ follow the relationship
\begin{equation}
S^2\:\propto\: R^2\:\propto\:\frac{1}{N} .
\label{eq:S to R}
\end{equation}

\subsubsection{The Central Limit Theorem}
\label{subsubsec:CLT}

Suppose $\hat{x}$ is calculated from several independent random particles
to estimate $\bar{x}$. At what point does one conclude that $\hat{x}$ sufficiently
reflects $\bar{x}$?
The central limit theorem (CLT) \cite{lewis_computational_1984, mcnp_manual_v1}
is a very powerful supplement to the quantities
described in Section \ref{subsubsec:PopStat}. The CLT states that for large N,
$\hat{x}$ will have a limiting distribution $f_N(\hat{x})$, and that distribution will be a
normal distribution
\begin{equation}
  f_N\big(\hat{x}\big) \approx \frac{1}{\sqrt{2\pi} \sigma(\hat{x})}\
           \exp\Bigg[ \frac{-\big( \hat{x}- \bar{x}\big)^2}{2\sigma^2(\hat{x})} \Bigg],\
           \quad N \rightarrow \infty.
  \label{eq:CLT1}
\end{equation}
The standard deviation of $\hat{x}$ can be related to the standard deviation of
the samples by
\begin{equation}
  \sigma(\hat{x}) = \frac{\sigma(x)}{\sqrt{N}}.
  \label{eq:hatx}
\end{equation}
Using the definition from Eq. \eqref{eq:hatx} in Eq. \eqref{eq:CLT1} results in
\begin{equation}
  f_N\big(\hat{x}\big) \approx \sqrt{\frac{N}{2*\pi}} \frac{1}{\sigma(x)}\
           \exp\Bigg[ \frac{-N\big( \hat{x}- \bar{x}\big)^2}{2\sigma^2(x)} \Bigg],\
           \quad N \rightarrow \infty .
  \label{eq:CLT2}
\end{equation}
This allows us to use known values for $\hat{x}$ and an approximation of
$\sigma(x)$--using $S$--to determine the probability density function of the
sample means
$f_N(\hat{x})$. Because $f_N(\hat{x})$ is normally distributed, we can find the
probability that $\hat{x}$ lies in $\bar{x} \pm \epsilon$ with
\begin{equation}
  P\big\{\bar{x} - \varepsilon < \hat{x} \leq \bar{x} + \varepsilon\big\} = \
   \int_{\bar{x}-\varepsilon}^{\bar{x}+\varepsilon}f_N\big( \hat{x} \big) d\hat{x}.
   \label{eq:probmean}
\end{equation}
Placing our definition for the distribution of $\hat{x}$, which is $f_N(\hat{x})$, into Eq.
\eqref{eq:probmean}, changing the limits of integration, and changing the
variables such that
\begin{equation*}
  t = \sqrt{N/2}\big[(\hat{x}-\bar{x})/\sigma(x)\big],
\end{equation*}
this becomes
\begin{equation}
  P\big\{\bar{x} - \varepsilon < \hat{x} \leq \bar{x} + \varepsilon\big\} = \
  \frac{2}{\sqrt{\pi}} \int_0^{(\sqrt{N/2})(\varepsilon/\sigma(x))} e^{-t^2} dt
  \:.
  \label{eq:bigP}
\end{equation}
Recalling the definition of the error function, Eq. \eqref{eq:bigP} becomes
\begin{equation}
  P\big\{\bar{x} - \varepsilon < \hat{x} \leq \bar{x} + \varepsilon\big\} = \
    \erf\Big[\sqrt{\frac{N}{2}} \frac{\varepsilon}{\sigma(x)}\Big].
\end{equation}
Then, using the calculated estimation for $\sigma(x)$ ($S$), and also recalling
that $S_{\hat{x}}= S/\sqrt{N}$ (Eq. \eqref{eq:VarN}), the error function reduces to
a function of $\varepsilon$ and $S_{\hat{x}}$, or:
\begin{equation}
    \erf\Big[\sqrt{\frac{N}{2}} \frac{\varepsilon}{\sigma(x)}\Big] = \
    \erf\Big[\sqrt{\frac{1}{2}} \frac{\varepsilon}{S_{\hat{x}}}\Big] .
\end{equation}
Should $\varepsilon$ be chosen to be a function of $S_{\hat{x}}$, the error
function reduces further and becomes merely an evaluation as multiples ($M$) of
$S_{\hat{x}}$ and $\sqrt{1/2}$. For the first few multiples of the standard
error, this is evaluated as
\begin{equation}
  P\big\{\bar{x} - M S_{\hat{x}} < \hat{x} \leq \bar{x} + M S_{\hat{x}} \big\} =
  \begin{cases}
    .683, & M = 1, \\
    .954, & M = 2, \\
    .997, & M = 3
  \end{cases}  .
\end{equation}

The central limit theorem tells us that the sample mean follows a normal
distribution, regardless of the distribution of the underlying sample, as the
number of samples approaches infinity. This
means that no matter what distribution is being sampled, the sampled mean will
have this expected behavior. As a result, given a calculated value for
$\hat{x}$ and $S$, the probability that $\hat{x}$ is near $\bar{x}$ is known
and calculable.
Further, the central limit theorem shows that this distribution is approached
very quickly as N increases, with most problems only requiring $N > 30$
\cite{lewis_computational_1984}. Note
that N is not the total number of samples, but the number of samples required to
calculate each mean.

However, for the central limit theorem to hold a number of
requirements must be satisfied. All of the quantities in Section
\ref{subsubsec:PopStat} have the underlying assumption that
each $x_i$ is assumed to be randomly sampled and
independent of other $x_i$. If some region of phase space is omitted
accidentally, these values will not be reflective of the true $f(x)$, and so
$\hat{x}$ will not approximate $\bar{x}$. Further, for $S$ to be a
good approximation of $\sigma(x)$, a large number of N samples must contribute
to the calculation of $\hat{x}$. The central limit theorem also assumes that
$f(x)$ is a probability density function that can be sampled and has a variance
that exists. As a result, one must be reasonably sure that all of these
requirements are satisfied if using Monte Carlo sampling methods.

\subsubsection{The Figure of Merit}
\label{subsubsec:FOM}

The equations in the preceding sections describe how to estimate the statistics
of a population given a finite number of samples. In radiation transport, a
user seeks to estimate some response, the relative error associated with that
response solution, and the time it takes to obtain those values. Equation
\eqref{eq:S to R} described the relationship between the sample variance, the
relative error, and the number of particles as
\begin{equation*}
S^2\:\propto\: R^2\:\propto\:\frac{1}{N} .
\end{equation*}
The relationship between the relative error, $R$, and the number of particles,
$N$, (recall that $R^2\:\propto\:\frac{1}{N}$)
will be some constant value (C):
\begin{equation*}
 C_1 = R^2N .
\end{equation*}
As a problem is simulated, the number of particles run, $N$, will increase
proportionally to the computational transport time, $T$. Therefore, the
relationship between $R$ and $T$ should also be a constant.
\begin{equation*}
  C_2 = R^2T
\end{equation*}
The figure of merit (FOM) shown in Eq. \eqref{eq:FOM}
is the most commonly reported metric using this relationship that is reported.
It is widely used in quantifying the effects of variance reduction methods.
Because it uses the inverse quantity of the relative error and time, a
``good'' result would be obtained from a low relative error in a short amount of
time, resulting in a FOM with a high numerical value.
\begin{equation}
FOM=\frac { 1 }{ R^{ 2 }T }
\label{eq:FOM}
\end{equation}
Further, a user may desire to determine how long a problem must be run to obtain
a desired relative error. In that case, Eq. \eqref{eq:FOM} can simply be
rearranged to
\begin{equation*}
  R = \frac{1}{(FOM*T)^{1/2}} .
\end{equation*}

The figure of merit is a very useful tool, but it is limited by statistical
precision in calculating $R$.
It is worth noting that early on in a transport simulation,
when too few particles have been simulated to
effectively capture $S$ or $\hat{x}$, the FOM will oscillate.
Eventually, the FOM will converge to a relatively constant value. This behavior can
also be used to determine whether one has sufficiently sampled the
region in which they are quantifying the response.

\subsection{Variance Reduction Methods for Monte Carlo Radiation Transport}
\label{subsec:MCVR}

Now that the different parameters that affect the variance in a problem have
been introduced, let us transition to different variance reduction techniques
that are available in Monte Carlo radiation transport packages.
Variance reduction
techniques in radiation transport methods fall
into four general categories: truncation methods, population control methods, modified
sampling methods, and partially-deterministic methods. Of importance for this
project are
population control methods and modified sampling methods, which are discussed in
a number
of the papers referenced herein. Truncation methods and partially-deterministic
methods do not contribute to and are not the focus of this work,
so will only be touched upon briefly.

A later section (\ref{sec:software}) of this dissertation will discuss the
choice of software packages used
for this project. In particular, our hybrid methods software package is designed
to accelerate the Monte Carlo radiation transport package MCNP
\cite{hendricks_mcnp_1985, brown_mcnp_2002, mcnp_manual_v1}. Variance reduction
methods are available in a number of other Monte Carlo radiation transport
packages, and are by no
means limited to a particular code. However, the implementation of methods
differs between software
and the specifics may differ slightly. The discussion
for the remainder of this subsection will be centered around the specifics of
the code used for this project.

\subsubsection*{Population Control Methods}

Population control methods adjust the particle population in the problem to
obtain better sampling in regions of interest by preferentially increasing or
decreasing the particle population.
The first two types of population control methods that will be discussed
are called splitting and rouletting.
Splitting is a method by which the particle population can be increased by
splitting a single higher-weight particle into several lower-weight particles.
Rouletting, conversely, reduces the particle population by stochastically
killing particles. Particles that survive a rouletting routine have their weight
adjusted higher, thereby conserving weight in the routine.
Both splitting and roulette maintain a
fair game by adjusting the particle weights as each routine is
performed; statistically, the sum of the child particle weights
is the same as the parent
weight as it entered the routine.

To use population control methods effectively as a variance reduction technique,
splitting is performed in high-importance regions to increase the particle
count, and thus the sampling, in important regions. Conversely, rouletting is
performed in
 low-importance
  regions to reduce the particle population in regions that are unimportant to
  the tally result.
Splitting and rouletting can be applied to include geometry, energy and time.

The weight window combines splitting and rouletting to keep particles within a
desired weight range. Figure \ref{fig:ww-mcnp}
illustrates the different processes a particle may go through when passing
through a
weight window.
%
\begin{figure}
  \centering
  \includegraphics[width=0.5\textwidth]{./chapters/lit_review/figures/ww-mcnp.png}
  \caption[Weight window illustration]{Cartoon illustration of a weight window,
    adapted from \cite{brown_mcnp_2002, mcnp_manual_v2}}
  \label{fig:ww-mcnp}
\end{figure}
%
The top  particle entering the weight window is a single, high-weight
particle. The weight of this particle is above the weight window bounds, so as
it enters the weight window it is split into multiple particles whose weight is
within the window bounds. The second particle entering the window is within the
weight window bounds, so it retains its weight and is not split or rouletted.
The last two particles entering the window have weights lower than the bound.
They undergo a rouletting routine and one particle is killed and the surviving
particle is increased in weight. As these particles leave the window, all of
them have weights within the range of the window. This will reduce the variance
of the particles contributing to a tally in that region.

While the use of weight windows in a problem helps to keep a more ideal
distribution of particle weights, the user is
faced with calculating a significant number of parameters to
determine weight windows for the entire problem. In the best case with an
experienced user, this may just take time. With an inexperienced user or a
complex problem this can
be insurmountable, and may be too difficult to do without some automated
assistance.

It should be noted that
while splitting and rouletting can be performed on a single variable--angle,
energy, space, or time--the weight windows generally used are either energy-space
dependent or space-time dependent. Further, the weight window will split or
roulette depending on the particle weight entering the window. Splitting and
rouletting on their own either increase or decrease the particle weight
proportional to the ratio of cell importances, or $I'/I$, no matter what
the entering particle weight is. As a
result, poorly chosen splitting or rouletting parameters can still
have significant tally variance, because particle weights may still span a wide
range.

\subsubsection*{Modified Sampling Methods}

Modified sampling methods adjust transport by sampling from a different probability
distribution function than the actual distribution for the problem. This is
possible if, as with population control methods, the particle weights are adjusted
 accordingly.
 The new probability distribution function should bias particles in regions of high
 importance to the problem tallies. In MCNP, a number of modified sampling
 methods exist.
 These include the exponential transform, implicit capture, forced collisions, source
 biasing, and neutron-induced photon production biasing.

The exponential transform modifies particle transport from the analog problem by
artificially modifying the macroscopic cross section, and thus the
distance-to-collision, to move particles in important directions. In directions
of higher importance, the cross section is reduced, and particles can flow more
freely. In directions of lower importance, the cross section is increased, and
particles more frequently interact, thereby increasing their probability of
directional change or absorption. The transformed cross section used by the
exponential transform is defined by
%
\begin{equation}
  \Sigma_t^* = \Sigma_t(1-p\mu) ,
  \label{eq:ExTrns}
\end{equation}
%
where $\Sigma_t^*$ is the transformed total cross section, $\Sigma_t$ is the
true total cross section, $p$ is the transform parameter, and $\mu$ is the
cosine of the angle between the preferred direction and the particle's transport
direction \cite{mcnp_manual_v1, mcnp_manual_v2, hendricks_mcnp_1985}.

Because the particle's transport is adjusted in the exponential transform, the
particle weight must be adjusted accordingly. This is given by
%
\begin{equation*}
  \begin{split}
  w^* &= \frac{\Sigma_t e^{-\Sigma_t s}}{\Sigma_t^* e^{-\Sigma_t^* s}} \\
      &= \frac{e^{-\rho \Sigma_t \mu s}}{1-p\mu}, \\
  \end{split}
  \label{eq:ExTrnsWt}
\end{equation*}
%
where $s$ is the phase space of particle residence. This weight adjustment
ensures that the particle weight is conserved throughout transport, even as
the cross section is altered. Because the cross section in the problem is both
energy and material dependent (depending on the geometry), the exponential
transform will be dependent on space and energy, and particles will be biased in
both. While a powerful method, the exponential transform is quite difficult to
use and if $p$ is ill-chosen this method can perform quite poorly. Further,
the user has to know quite a bit about the problem physics and material to choose an
optimal quantity for $p$.

Source biasing, rather than preferentially adjusting particles' directionality
by way of adjusting the cross sections, biases particles from their origin.
Source biasing has the option to bias particles in energy, direction, and space
(if the source is volumetric). This allows the user to choose importance
separately for each variable. First, the source variable (let us consider
energy for the moment) is defined as a series of bins or a function. Second, the bins are
assigned probabilities of occurrence according to their importance. An energy bin
with a high importance will be assigned a high probability of occurrence, and a
bin with low importance will be assigned a low probability of occurrence. As
particles are born in the bins with higher importances, they will have their
weights adjusted to the inverse of their probability of occurrence, or $w^* =
p/p^*$. Here $p$ refers to the probability density function for the source
particles; it bears no relation to the exponential transform factor.

Source biasing is a very simple method that can reduce the solution variance
significantly. However, if a user chooses bin sizes or a function that does not
properly reflect the particles importances in the problem, the source will be
poorly sampled. As a result, sampling may be very inefficient and the figure of
merit will decrease. In MCNP, if poor parameters are chosen for this method, the
user is given a warning.

\subsubsection*{Truncation Methods}

Truncation methods stop tracking particles in a region of phase-space that is of
low-importance to the tally. These methods can be used in space (a vacuum boundary
condition), energy (eliminate particles above or below a specified energy), or
time (stop
tracking after a given time). To effectively use these methods, the user must be
aware of
particles' importance to a tally result. If particles that are important to a
result are
 eliminated with a truncation method, the tally will lack the contribution from
 that particle's phase-space, and will be underestimated as a result. Further,
 as discussed in Section \ref{subsubsec:CLT}, the central limit theorem only
 holds assuming that the histories tracked are independent and drawn from
 identical distributions.
 Truncating particles of high importance removes the independence from the
 sampling and modifies the underlying PDF being sampled,
 so the estimate of the response will be wrong.

It is important in using any variance reduction technique to ensure that a
fair game
is being played. The user must ensure that the fundamental nature of the problem
is not
being changed by using a variance reduction technique, or the answer will not be
representative of the original problem. Automated variance reduction techniques aim to
eliminate this uncertainty for the user by estimating the importance of
particles in some
 way and then setting up variance reduction parameters automatically.
The remainder of this chapter review will focus on efforts to
automate population
control methods and modified sampling methods for variance reduction.

\subsection{Automated Variance Reduction Methods for Monte Carlo Radiation
Transport}
\label{subsec:AutomatedMCVR}

Section \ref{subsec:MCVR} described some methods that one may use to reduce the
variance in Monte Carlo radiation transport tallies. These methods, if used
correctly, can significantly increase the transport efficiency in a Monte Carlo
simulation. However, correct use of these methods often requires intelligent
selection of variance reduction (VR) parameters, which is a non-trivial task. Users
have found themselves often performing several trial runs before choosing final
quantities for the VR
parameters in their problems, which was computationally
inefficient and required significant knowledge of Monte Carlo and variance
reduction to execute well \cite{booth_automatic_1982}.

This has been addressed by using Monte Carlo to sample the problem in an initial
calculation to determine more favorable variance reduction parameters automatically.
Booth and Hendricks,
recognizing that choosing optimal weight window values for energy- and space-
dependent weight windows was difficult even for experienced users, proposed two
tools for Monte Carlo variance reduction in parallel. The first was a
Monte Carlo importance generator \cite{booth_automatic_1982} that could be used
to make informed decisions on cell importances throughout the problem. The
second method, a Monte Carlo generated weight window generator,
calculates the weight window values automatically for a given problem
\cite{hendricks_code-generated_1982}.
The importance generator estimates a cell's importance
by tracking the weights of the particles in the cell, or
\begin{equation}
  Importance  = \frac{\text{score of particles leaving the cell}}
                     {\text{weight leaving the cell}}.
\label{eq:BoothImp}
\end{equation}
The weight window generator calculates weight window values with
\begin{subequations}
\begin{equation}
  W_{i,low} = \frac{1}{kN}\big(\Sigma W_{i,in} + \Sigma W_{i,out} \big)
\end{equation}
\begin{equation}
  W_{i,high} =
  \begin{cases}
    k*W_{i,low} & \text{if } W_{i,low} \neq 0 \\
    \infty & \text{if } W_{i,low} = 0
  \end{cases}  \:,
  \label{eq:hendricksWW}
\end{equation}
\end{subequations}
where $W_{i,low}$ and $W_{i,high}$ are the weight window lower and upper weight
bounds respectively, $W_{i,in}$ and $W_{i,out}$ are the total weight entering
and leaving the cell, $N$ is the number of source particles, and $k$ is some weight
window width (a constant that Hendricks set to 5).

In his
paper, Booth notes that the weight window target value derived from the
importance generator was chosen so that the
track weight times the expected score in the tally region (for unit track
weight) was approximately constant. Booth's importance generator saw
improvements in the FOM between 1.5-8x when compared to the analog run for the
test problem presented.

Booth and Hendricks combined these two methods to
automate weight window generation based on phase-space importance
\cite{booth_deep_1982, booth_importance_1984}. They showed that the combination
of the importance estimator and the weight window generator was a successful way
to perform variance reduction in deep-penetration problems. However, their
method depended on several iterations of importance-determining runs to obtain a
satisfactory estimation of the importance. For a 300 cm slab problem, the FOM was
increased from 1.9 to 75, but took 10 subsequent runs to obtain the FOM of 75,
and these runs ranged from 1.2 min (for the analog problem) to 42 minutes (for
the 9th run \cite{booth_importance_1984}).

It should be noted that both the importance generator and the weight window
generator use a lower-fidelity
Monte Carlo run to gain an initial estimate for a cell's importance and generate
variance reduction parameters from them to bias a more computationally-intensive
and higher-fidelity
run. Naturally, the variance reduction parameters generated by using these
techniques are limited by the statistical precision in the regions used to
generate them. Hendricks also pointed out that the weight window generator
tended to populate all regions of phase space equally, which he conceeded was
not ideal for all problems \cite{hendricks_code-generated_1982}.
Furthermore, for deep-penetration particle transport, the
variance reduction parameters for low flux regions are exceedingly difficult to
generate, resulting in unfavorable VR parameters.

The MCNP \cite{mcnp_manual_v1, brown_mcnp_2002} weight window generator has been
extended beyond the initial space- and energy-implementation described in
Booth's paper. It now has the ability to automatically generate space- energy-
and angle-weight windows. The importance generator in MCNP also has been
extended to time-importance; the values of which can be used for splitting or
rouletting parameters \cite{brown_mcnp_2002}, and can be optimized on a grid
independent from the MCNP geometry \cite{evans_enhanced_1998}.

As with Booth and Hendricks'
original implementations, this updated weight window generator still relies on
adequate sampling to obtain sufficient weight window parameters. When additional
degrees of freedom, like angle-dependence, are added, the time
to converge on those parameters takes even longer. The weight window
generator also only allows for a single tally to be optimized at once, so
multiple tallies cannot be optimized simultaneously. Finally, the weight window
generator still requires user input and updating to seed the weight
window solution. The user must choose the meshing of the problem and have some
intuition as to how the problem should be subdivided. In the paper by Van Riper
et al, it was found that depending on user
experience, the weight window generator can have differences in the FOM from 2
to 10 times \cite{van_riper_generation_1995} for the problems that they
investigated.


\chapter{Physically Based Matrix Distribution}
\label{chapter:pbmd}

{\bf This chapter will be updated.}
We discuss how \plapack locally stores the data associated with the
different linear algebra objects.
Let us consider a column vector $ x $ distributed to $p$ nodes in a
$ r \times c $ mesh.
For simplicity, we will assume that the column vector is
aligned to the first entry of the template column vector, which is
partitioned using a distribution blocking size $ n_b $ and that
the template column vector is induced in row-major order.
Thus
\[ x = \left( \begin{array}{c}
x_0 \\ \hline
x_1 \\ \hline
\vdots \\ \hline
x_{N-1}
\end{array}
\right)
\]
where $ x_i $ is of length $ n_b $ (except perhaps the last sub-vector).
Node $ (i,j) $ will own the sub-vectors
\[
\left( \begin{array}{c}
x_{i * r + j} \\ \hline
x_{i * r + j + p} \\ \hline
x_{i * r + j + 2p} \\ \hline
\vdots
\end{array}
\right)
\]
This vector is locally stored in memory so that the order of
the elements is preserved (i.e., elements of $ x_{i * r+j+kp} $
will precede those of $ x_{i* r+j+(k+1)p} $).
The stride in memory between elements will be internally determined and
can be inquired by one of the subsequently described calls.

Next, let us consider a matrix $ A $ distributed to the same mesh.
For simplicity, we assume that $ A $ is aligned with the top-left
element of the template matrix induced by row and column template vectors
both induced in column-major order.
Then,
\[
A = \left( \begin{array}{c | c | c | c }
A_{0,0} & A_{0,1} & \cdots & A_{0,N-1} \\ \hline
A_{1,0} & A_{1,1} & \cdots & A_{1,N-1} \\ \hline
\vdots & \vdots &        & \vdots \\ \hline
A_{M-1,0} & A_{M-1,1} & \cdots & A_{M-1,N-1}
\end{array}
\right)
\]
and the following sub-matrix is assigned to node $ (i,j) $:
\[
\tilde{A}_{ij} = \left( \begin{array}{c | c | c || c | c | c || c }
A_{i,j * r} & \cdots & A_{i,(j+1)*r-1} &
A_{i,j * r+p} & \cdots & A_{i,(j+1)*r-1+p} & \cdots
\\ \hline \hline
A_{i+r,j * r} & \cdots & A_{i+r,(j+1)*r-1} &
A_{i+r,j * r+p} & \cdots & A_{i+r,(j+1)*r-1+p} & \cdots
\\ \hline \hline
A_{i+2r,j * r} & \cdots & A_{i+2r,(j+1)*r-1} &
A_{i+2r,j * r+p} & \cdots & A_{i+2r,(j+1)*r-1+p} & \cdots
\\ \hline \hline
\vdots &  & \vdots & \vdots &  & \vdots &
\end{array} \right)
\]
It is this matrix $ \tilde{A}_{ij} $ that is stored in a local
two-dimensional array, using FORTRAN style column-major
order storage, with a leading dimension determined internally.

Finally, let us discuss the storage of a projected
multivector.  Notice that if the column vector $ x $ discussed above
is spread within a column of nodes, or gathered
to the $ i $th node in each column,
the following sub-vectors will be collected on
node $ (i,j) $:
\[
\tilde{x}_{ij} = \left( \begin{array}{c}
x_{j * r} \\ \hline
\vdots \\ \hline
x_{(j+1)*r-1} \\ \hline \hline
x_{j * r+p} \\ \hline
\vdots \\ \hline
x_{(j+1)*r-1+p} \\ \hline \hline
\vdots
\end{array} \right)
\]
Notice that the sub-vectors from within a column of nodes
are interleaved so that the indices of sub-vectors are
strictly increasing (the order in the global vector is maintained).
It is this column vector that is then stored in a local linear array,
with a stride determined internally.


\chapter{\label{cha:Analytical-Solns}Analytical Solutions}

\section{\label{sec:TractionProblems}Traction Problems}

Computation of analytical solutions for elastostatic problems over
regular domains is a relatively straightforward procedure. These problems
are typically formulated in terms of a combination of displacement
and traction boundary conditions, and such problems provide a good
test of the code accuracy, as well as specifically testing the implementation
of traction boundary conditions. We present here two simple problems
for this purpose.

\subsection{Solutions Using Polynomial Stress Functions}

Our derivation follows the procedures outlined in Timoshenko and Goodier
\cite{Timoshenko:Goodier:1987}, and we restrict ourselves to two-dimensional
problems. Any problem in elastostatics must satisfy the equilibrium
equations
\begin{gather}
\frac{\partial\sigma_{xx}}{\partial x}+\frac{\partial\sigma_{xy}}{\partial y}+X=0\label{eq:traction:1}\\
\frac{\partial\sigma_{yy}}{\partial y}+\frac{\partial\sigma_{xy}}{\partial x}+Y=0,\nonumber
\end{gather}
where \textit{X} and \textit{Y} are the body force components in the
\textit{x} and \textit{y} directions, respectively, and the stress
components are given by $\sigma$. In the problems considered here,
we neglect body forces, so \textit{X} and \textit{Y} disappear from
the equilibrium equations. The solution must also satisfy the boundary
conditions, given as surface tractions over the surface of the body.
Finally, the solution must satisfy the conditions of compatibility,
which may be expressed as:
\begin{equation}
\left(\frac{\partial^{2}}{\partial x^{2}}+\frac{\partial^{2}}{\partial y^{2}}\right)\left(\sigma_{xx}+\sigma_{yy}\right)=0.\label{eq:traction:2}
\end{equation}
To compute the displacement field, it is also necessary to specify
displacement boundary conditions.

Equations \vref{eq:traction:1} may be satisfied by taking any function $\phi$
of \textit{x} and \textit{y}, and letting the stress components be
given by the following expressions:
\begin{equation}
\sigma_{xx}=\frac{\partial^{2}\phi}{\partial y^{2}},\;\sigma_{yy}=\frac{\partial^{2}\phi}{\partial x^{2}},\;\sigma_{xy}=-\frac{\partial^{2}\phi}{\partial x\partial y}.\label{eq:traction:3}
\end{equation}
The solution must also satisfy the compatibility equations. Substituting
Equations \vref{eq:traction:3} into Equation \vref{eq:traction:2}, we find that the
stress function $\phi$ must satisfy
\begin{equation}
\frac{\partial^{4}\phi}{\partial x^{4}}+2\frac{\partial^{4}\phi}{\partial x^{2}\partial y^{2}}+\frac{\partial^{4}\phi}{\partial y^{4}}=0.\label{eq:traction:4}
\end{equation}
A relatively easy way to solve a number of problems is to select expressions
for $\phi$ consisting of polynomial functions of \textit{x} and \textit{y}
of second degree or higher. Any polynomial of second or third degree
will satisfy Equation \vref{eq:traction:4}. For polynomials of higher degree,
they must be substituted into Equation \vref{eq:traction:4} to determine what
restrictions must be placed on the solution.

\subsection{\label{sub:Analytical-Constant-Traction}Constant Traction Applied
to a Rectangular Region}

For this problem, a constant normal traction, $\overrightarrow{N}$,
is applied along the positive \textit{x}-edge of the rectangular region
($x=x_{0}$), and roller boundary conditions are applied along the
left and bottom boundaries (\vref{fig:Const-tractions}). Since the
tractions are constant, we assume a second degree polynomial for the
stress function:
\begin{equation}
\phi_{2}=\frac{a_{2}}{2}x^{2}+b_{2}xy+\frac{c_{2}}{2}y^{2}.\label{eq:traction:5}
\end{equation}
This yields the following expressions for the stresses:
\begin{equation}
\sigma_{xx}=\frac{\partial^{2}\phi_{2}}{\partial y^{2}}=c_{2}=N,\;\sigma_{yy}=a_{2}=0,\;\sigma_{xy}=-b_{2}=0.\label{eq:traction:6}
\end{equation}
From Hooke's law, we have:
\begin{equation}
\epsilon_{xx}=\frac{\partial u}{\partial x}=\frac{\left(1-\nu^{2}\right)N}{E},\;\epsilon_{yy}=\frac{\partial v}{\partial y}=\frac{-\nu\left(1+\nu\right)N}{E},\;\epsilon_{xy}=\frac{1}{2}\left(\frac{\partial v}{\partial x}+\frac{\partial u}{\partial y}\right)=0.\label{eq:traction:7}
\end{equation}
The strain components are thus easily obtained from the stress components.

To obtain the displacements, we must integrate Equation \vref{eq:traction:7}
and make use of the displacement boundary conditions. Integrating
the first two of these, we obtain
\begin{equation}
u=\frac{\left(1-\nu^{2}\right)Nx}{E}+f\left(y\right),\; v=\frac{-\nu\left(1+\nu\right)Ny}{E}+f\left(x\right).\label{eq:traction:8}
\end{equation}
Substituting these into the third expression, we obtain
\begin{equation}
\frac{\partial f\left(x\right)}{\partial x}=-\frac{\partial f\left(y\right)}{\partial y},\label{eq:traction:9}
\end{equation}
which means that both $f\left(x\right)$ and $f\left(y\right)$ must
be constant. We solve for these constants using the displacement boundary
conditions along $x=x_{0}$ and $y=y_{0}$. Doing this, we obtain
the following expressions for the displacement components:
\begin{equation}
u=\frac{\left(1-\nu^{2}\right)N}{E}\left(x-x_{0}\right),\; v=\frac{\nu\left(1+\nu\right)N}{E}\left(y_{0}-y\right).\label{eq:traction:10}
\end{equation}

\noindent \begin{center}
\begin{figure}[H]

\caption{\label{fig:Const-tractions}Problem with constant traction boundary
conditions applied along right edge.}

\noindent \centering{}\includegraphics{analyticalsolns/figs/consttract}
\end{figure}

\par\end{center}


\documentclass{article}
\usepackage{graphicx}
\usepackage{pdfpages}

\begin{document}

\title{Extensions of L1 Trend Filtering: Seasonality, Outlier Modeling
and Other Effects with $l_1$ and $l_2$ Regularization}
\author{David Johnston}

\maketitle

\begin{abstract}
This paper extends and further elucidates ideas from Kim, Koh \& Boyd for L1 regularized
trend filtering and gives explicit formulas for reducing these problems down to
quadratic and linear programming problems that can be solved with various widely available
convex optimization libraries. We include seasonality effects with both
$l_1$ and $l_2$ regularization, outlier modeling, sparse level changes and
unequally spaced points. This paper also acts a documentation for the formulas used
in our open source python library \emph{myl1tf}
\footnote{https://github.com/dave31415/myl1tf}.
\end{abstract}

\section{The primary and dual quadratic programming problems}

We will start off at Section 5.2 of KKB where they state the quadratic programming problem for L1TF
and also state the dual problem.

\begin{eqnarray}
\mbox{minimize} & \frac{1}{2} ~ || y - x ||_2^2  + \lambda ||z||_1 \\
\mbox{subject to} & z = D x
\end{eqnarray}
The first is an $l_2$ norm and the second an $l_1$ norm. The Lagrangian, with a dual variable $\nu \in \mathbf{R}^{n-2}$, is
\[
L(x,z,\nu) =  || y - x ||_2^2  + \lambda ||z||_1 + \nu^T (D x -z)
\]

The dual function is
\[
\mbox{inf}_{x,z} ~ L(x,z,\nu) =
    \left\{
    \begin{array}{ll}
    - \frac{1}{2} ~ \nu^T D D^T \nu + y^T D^T \nu &  - \lambda \mathbf{1} < \nu < \lambda \mathbf{1} \\
  -\infty  & \mbox{otherwise.} \\
  \end{array}
  \right.
\]

since

\begin{equation}
\sup_\nu \inf_z \left(-\nu^T z + \lambda ||z||_1 \right) =
\left\{
    \begin{array}{ll}
   0 ~\mbox{if}~ |\nu| < \lambda \\
  -\infty  & \mbox{otherwise.} \\
  \end{array}
  \right. \label{eq:supinf}
\end{equation}
(see appendix for proof) and so the dual problem is

\begin{eqnarray}
\mbox{maximize} & -\frac{1}{2} ~ \nu^T D D^T \nu + y^T D^T \nu \\
\mbox{subject to} & - \lambda \mathbf{1} < \nu < \lambda \mathbf{1}
\end{eqnarray}

From the solution $\nu^*$ of the dual problem, we can compute the L1TF solution,
\[
x^* = y - D \nu^*
\]

\section{Seasonality}

As suggested by KKB we can adapt this to add a seasonal component.
\begin{eqnarray}
\mbox{minimize} & \frac{1}{2} ~ || y - x -s||_2^2  + \lambda ||z||_1 \\
\mbox{subject to} & z = D x \\
and & \sum_i^P s_i = 0\\
and & s_{i+P} = s_i
\end{eqnarray}

KKB does not go into detail on how to proceed with this and so we will begin here by deriving
the dual problem. We will do this by putting the constraints on $s$ directly into the
equation to be minimized.

To do this, we define $p$ to be the vector of independent variables defining the periodic components.
This vector has dimension $(P-1)$ as the $P$-th, dependent value is $-\sum_{i=1}^{P-1} p_i$ which enforces the constraint that
they sum to zero. This constraint is required if there is to exists a unique solution as otherwise one could add
any constant to $x$ and subtract it from $s$ without changing the model.

We can define $\tilde{p} \in \mathbf{R}^{P}$ as $\tilde{p} = \left( p,-\sum_{i=1}^{P-1} p_i \right) = T p$.
$T$ will be a $P \times (P-1)$ matrix with a $(P-1)$ identity matrix at the top and an extra row consisting of all -1.
The vector $s$ is now just a periodic re-cycling of $\tilde{p}$ which we can represent as s matrix $B$ which is formed by
row-wise stacking some number (ceil(N/P)) of $P \times P$ identity matrices and truncating rows to dimension $N$. So finally
we can write $s = B T p \equiv Q p$. By doing this we can rewrite the optimization problem as

\begin{eqnarray}
\mbox{minimize} & \frac{1}{2} ~ || y - x - Q p||_2^2  + \lambda ||z||_1 \\
\mbox{subject to} & z = D x
\end{eqnarray}
with the $p \in \mathbf{R}^{(P-1)}$ now unconstrained.

\subsection{Seasonality with $l_2$ regularization}

To improve stabilization and allow more control over $p$, we will add
a $l_2$ regularization constraint and write our problem.
\begin{eqnarray}
\mbox{minimize} & \frac{1}{2} ~ || y - x - Q p||_2^2  + \lambda ||z||_1 + \eta ~ \frac{1}{2} \tilde{p}^T \tilde{p}\\
\mbox{subject to} & z = D x
\end{eqnarray}

We will now proceed as before and derive the dual problem. To do this we first
write down the Lagrangian
\[
L(x,z,p,\nu,\eta,\lambda) =  R  + \lambda ||z||_1 + \nu^T (D x -z) + \eta ~ \frac{1}{2} p^T H ~p
\]
with  $H = T^T T$ and
\begin{eqnarray}
R & \equiv & || y - x - Q p||_2^2 \\
& = & \frac{1}{2} y^T y + \frac{1}{2} x^T x + \frac{1}{2} p^T Q^T Q p \\
&  & - y^T x - y^T Q p + x^T Q p
\end{eqnarray}

We can calculate $\mbox{inf}_{x,z,p} L(x,z,p,\nu,\eta,\lambda)$ by setting gradients w.r.t. $x$ and $p$ to zero. The
gradients of $R$ are

\begin{eqnarray}
\nabla_x R &  =  &  x^T - y^T - p^T Q^T\\
\nabla_p R &  =  &  \left( x^T - y^T - p^T Q^T \right) Q \\
& = & \left(\nabla_x R \right) Q \\
\end{eqnarray}
and the gradients of $L$ are
\begin{eqnarray}
\nabla_x L &  =  &  x^T - y^T - p^T Q^T + \nu^T D\\
\nabla_p L &  =  &  \left( x^T - y^T - p^T Q^T \right) Q + \nu p^T H \\
\end{eqnarray}

Setting $\nabla_x L$ =0 yields the equation.
\begin{equation}
y - x - Qp = D^T \nu \label{eq:residual}
\end{equation}
or
\[
x = y - Qp - D^T \nu
\]
Equation \ref{eq:residual} shows that $D^T \nu$ is the residual.
Setting $\nabla_p L$ =0 yields
\[
p = \eta^{-1} H^{-1} Q^T D^T \nu
\]
and we can use this last equation for $p$ to solve for $x$ and we can write these solutions as
\begin{eqnarray}
p^* & = &  \eta^{-1} H^{-1} Q^T D^T \nu \\
& and & \nonumber \\
x^* & = & y - D^T \nu - \eta^{-1} Q H^{-1} Q^T D^T \nu
\end{eqnarray}
and so once we have a solution for $\nu$ we can use these to obtain separate
solutions for $x$ and $p$.

The more subtle minimization w.r.t. $z$ will result in the same constraint in the dual problem as before. The reader
should ensure that they understand how the terms $\lambda ||z||_1 - \nu^T z$ result in the constraint
$- \lambda \mathbf{1} < \nu < \lambda \mathbf{1}$. One can show that outside of this range, the $inf_z$ of this term
is $-\infty$ (at either $z = \pm \infty$) and within is 0 (at $z=0$) (see appendix) and so any supremum for $\nu$
must lie within $|\nu| < \lambda$.

We then construct the dual problem by plugging these solutions into the Lagrangian and we arrive at
\begin{eqnarray}
\mbox{maximize} & -\frac{1}{2} ~ \nu^T A \nu + y^T D^T \nu \\
\mbox{subject to} & - \lambda \mathbf{1} < \nu < \lambda \mathbf{1}
\end{eqnarray}
with
\[
A = D D^T + \eta^{-1} D Q H^{-1} Q^T D^T
\]
We solve this quadratic programming problem as before for $\nu$ and then use the equations above
to calculate $x$ and $p$ from $\nu$. It is apparent that as $\eta \rightarrow \infty$ (seasonality is suppressed),
$p \rightarrow 0$ and we recover the same solution as before for $x$. We cannot use these equations directly with
the other limit $\eta = 0$, though we will address that in a later section. However there is no problem setting
$\eta$ to a neglibly small number and applying these formula.

\subsection{Seasonality using $l_1$ regularization}
We can also use an $l_1$ regularization term on the seasonality terms $p$ which will result in
sparse solutions for $p$. That is, with a well chosen regularization parameter,
no seasonality will be used when it isn't really required. This might be more useful than the $l_2$
regularization described above though the solution is a little more complicated.

The situation for $l_1$ regularization is similar to the above for the $x$ coordinate and leads to the
same Equation \ref{eq:residual} for the residual. Submitting this solution for $x$ in terms of $\nu$ and $p$
leads to an optimization problem for $\nu$ and $p$ as follows

\begin{eqnarray}
sup_{\nu} ~ inf_p & -\frac{1}{2} ~ \nu^T D D^T \nu + y^T D^T \nu -\nu^T D Q p + \eta ||T p||_1 \\
\mbox{subject to} & - \lambda \mathbf{1} < \nu < \lambda \mathbf{1}
\end{eqnarray}

We can write $||T p||_1 = ||p||_1 + |u^T p| $ where $u$ is the unity vector $u_i=1$. There are no constraints on
$p$ so this term $|u^T p|$ can be any non-negative number for some choice of $p$ and so
$inf_{p} ~  -\nu^T D Q p + \eta ||T p||_1 = inf_{p} ~  -\nu^T D Q p + \eta ||p||_1 $ and
\[
inf_{p} ~  -\nu^T D Q p + \eta ||p||_1 =
    \left\{
    \begin{array}{ll}
    0 &  - \eta \mathbf{1} <  Q^T D^T \nu < \eta \mathbf{1} \\
  -\infty  & \mbox{otherwise.} \\
  \end{array}
  \right.
\]

This implies that the Lagrangian for $\nu$ to be optimized is the same as the non-seasonal
version but with an additional constraint.

\begin{eqnarray}
\mbox{maximize} & - \frac{1}{2} ~ \nu^T D D^T \nu + y^T D^T \nu \\
\mbox{subject to} & - \lambda \mathbf{1} < \nu < \lambda \mathbf{1} \\
& - \eta \mathbf{1} < Q^T D^T \nu < \eta \mathbf{1} \\
\end{eqnarray}

Notice that the $l_2$ solution leads to a modification of the quadratic form whereas the
$l_1$ problem instead leads to an additional constraint. Both of these are standard quadratic
programming problems that can be solved with any convex optimization library (such as cvxopt for python).
A difference however is that the $l_2$ solution included a simple way of calculating $x$ and $p$ once
$\nu$ has been solved.

With the $l_1$ solution here, we have to solve a second optimization problem
in order to separate out the separate components. To do this, note that after $\nu$ has been solved for,
the first $\chi^2$ term is now constant
and so we need to optimize the sum of regularization terms by themselves

\begin{eqnarray}
\mbox{minimize} & ||D x||_1 + (\eta / \lambda) ||T p||_1 \\
\mbox{subject to} & y - x - Qp = D^T \nu \\
\end{eqnarray}

which we can write solely in terms of $p$ as

\begin{eqnarray}
\mbox{minimize over} ~p : & ||D (y - D^T \nu - Q p)||_1 + (\eta/\lambda) ||T p||_1 \\
\end{eqnarray}

The trick to solving this is to combine these into a larger vector space
\[
w = [D (y - D^T \nu), 0]^T
\]
and the $F$ matrix defined by two matrix blocks
\begin{equation}
F=\left[
\begin{array}{c}
D Q  \\
\hline
(\eta/\lambda) T \\
\end{array}\right]
\end{equation}

and write this as
\begin{eqnarray}
\mbox{minimize over}~ p : & ||w-F p||_1 \\
\end{eqnarray}

This is now in the form of a well known problem, the Least Absolute Deviation (LAD) problem and can be written as a
a linear program. The cvxopt library contains a program for solving this problem. Thus,
we can solve for $\nu$ by solving the quadratic programming problem and then solve this equation for $p$ and then
use Equation \ref{eq:residual} to solve for $x$.

\section{Modeling outliers and step functions for more robust fits}
KKB also suggest including outliers by adding terms $u$ to the model with a strong $l_1$ regularization. This will
result in a sparse solution for the $u$ values which can model things like spikes without throwing off the other terms.
These can be treated just like the $l_1$ regularized seasonality and will result in the same quadratic programming
problem as before with an additional constraint $-\delta < D^T \nu < \delta$ where delta is the $l_1$
regularization parameter for the $u$ vector.

As before, we end up with a solution for $\nu$ and must still solve the LAD problem to get separate solutions for
$x, p$ and $u$.
\begin{eqnarray}
\mbox{minimize over} ~p,u : & ||D (y - D^T \nu - Q p - u)||_1 + (\lambda / \eta) ||T p||_1 + \delta/\lambda ||u||_1 \\
\end{eqnarray}

We can employ the same trick as before and write this as a single LAD problem on a larger vector space.
\begin{eqnarray}
\mbox{minimize over}~ p,u : & ||w-F z||_1 \\
\end{eqnarray}
with
\begin{eqnarray}
z & = & [p , u]^T\\
w & = & [D (y - D^T \nu), 0, 0]^T \\
\end{eqnarray}
and $F$ specified by matrix blocks
\begin{equation}
F=\left[
\begin{array}{c|c}
D Q  & D   \\
\\
\hline
& \\
(\eta/\lambda) T & 0_{P \times n} \\
\\
\hline
& \\
0_{n \times P-1}, & (\delta/\lambda) I_n \\
\end{array}\right]
\end{equation}

\subsection{Step functions}

Rather than spikes ($\delta$-functions) we can also consider step functions (or Heaviside functions).
The solution is nearly identical to the above except that we replace matrix $D$ in the upper right
hand corner with $D H$ where $H$ is the matrix with 1 in it's upper triangle and diagonal and zero
in it's lower triangle. This is because step functions are simply cumulative sums of delta functions.

With both spikes $u$ and step function transitions $h$ (regularized by $\gamma$), we can write the
equations

\begin{eqnarray}
z & = & [p , u, h]^T\\
w & = & [D (y - D^T \nu), 0, 0, 0]^T \\
\end{eqnarray}
and $F$ specified by matrix blocks
\begin{equation}
F=\left[
\begin{array}{c|c|c}
D Q  & D  & D H \\
&\\
\hline
&\\
(\eta/\lambda) T & 0_{P \times n} & 0_{P \times n}\\
&\\
\hline
&\\
0_{n \times P-1} & (\delta/\lambda) I_n & 0_{n}\\
&\\
\hline
&\\
0_{n \times P-1} & 0_n & (\gamma/\lambda) I_n \\
\end{array}\right]
\end{equation}

By now, one should notice the pattern that adding any new $l_1$ regularization
terms simply requires a modification of the above equations which just requires
that one express the intentions in matrix form.

\section{L1 norms for everything}

Rather than using an $l_2$ on the residual we can use an $l_1$ term for this and so use
$l_1$ norms for everything. An $l_1$ norm on the residual is another way of giving less
weight to outliers. It also simplified the solution as there is no quadratic programming
problems to solve. We can use the same trick of writing the whole optimization as a single
LAD problem in a higher dimensional space.

Minimize the following for $x$, $p$ and $h$:
\begin{equation}
||y - x - Q p -H h||_1 + \lambda_1 ||D_1 x - g|| + \lambda_2 ||D_2 x||
+ \eta ||T p||_1 + \delta ||h||_1
\end{equation}

Here we are considering first and second derivatives: matrices $D_1$ and $D_2$ plus the seaosnality
term and level changes. In addition we allow for $g$ a non-zero preferred first derivative. As before
we can write this as a single LAD problem:
\[
\mbox{minimize over}~ z :  ||w-F z||_1 \\
\]
where

\begin{eqnarray}
z & = & [x, p , h]^T\\
w & = & [y, -\lambda_1 g, 0, 0, 0]^T \\
\end{eqnarray}
and $F$ specified by matrix blocks
\begin{equation}
F=\left[
\begin{array}{c|c|c}
I_n  & Q  & H \\
&\\
\hline
&\\
-\lambda_1 D_1 & 0_{m \times (P-1)} & 0_{m \times n}\\
&\\
\hline
&\\
-\lambda_2 D_2 & 0_{m \times (P-1)} & 0_{m \times n}\\
&\\
\hline
&\\
0_{P \times n} & -\eta T & 0_{P \times n} \\
&\\
\hline
&\\
0_{n} & 0_{n \times (P-1)} & -\delta I_n
\end{array}\right]
\end{equation}

\section{Implementation}
The Githib repository https://github.com/dave31415/myl1tf contains an implementation for this
L1TF modeling with seasonality. This was a fork of the repository
https://github.com/elsonidoq/py-l1tf by Pablo Zivic which implements the
simpler version without seasonality and other effects. Both versions are in python and
use the python cvxopt library to solve the convex optimization problems. Our
version contains some test programs as well. For example, the following command,
\begin{verbatim}
test_myl1tf.test_l1tf_on_mock_with_period(period=6, eta=1.0,alpha=0.5)
\end{verbatim}
creates a mock data-set, fits the model and displays the following plot.
\begin{figure}
\centering
\includegraphics[width=500pt]{example.png}
\end{figure}
(Note \emph{eta} = $\eta$ and \emph{alpha} = $\lambda$ as \emph{lambda} is a reserved word in python).

\section{Appendix}
\subsection{Calculating the derivative matrices for unequally spaced points}

The first and second derivatives of unequally spaced data points can be calculated, at those data points,
as follows. We can use Lagrange's interpolation formula to fit the unique quadratic function through any
three points $(x_1,y_1), (x_2,y_2), (x_3, y_3)$.

\begin{eqnarray}
P(x) & = & \sum_{i=1}^{3} y_i P_i(x) \\
P_i(x) & = & \prod_{j=1, j \ne i}^{3} \frac{\left( x-x_j \right)}{ \left( x_i - x_j \right)}
\end{eqnarray}

Written out in full this is
\begin{eqnarray}
P(x) & = & y_1 \left(\frac{x-x_2}{x_1-x_2}\right) \left( \frac{x-x_3}{x_1-x_3}\right) \nonumber \\
     & + & y_2 \left(\frac{x-x_1}{x_2-x_1}\right) \left( \frac{x-x_3}{x_2-x_3}\right) \nonumber \\
     & + & y_3 \left(\frac{x-x_1}{x_3-x_1}\right) \left( \frac{x-x_2}{x_3-x_2}\right) \nonumber
\end{eqnarray}
The first derivative is
\begin{eqnarray}
\frac{dP}{dx} & = & y_1 \frac{\left(2x-x_2-x_3 \right)}{\left(x_1-x_2\right)\left(x_1-x_3\right)} \nonumber \\
     & + & y_2 \frac{\left(2x-x_1-x_3 \right)}{\left(x_2-x_1\right)\left(x_2-x_3\right)} \nonumber \\
     & + & y_3 \frac{\left(2x-x_1-x_2 \right)}{\left(x_3-x_1\right)\left(x_3-x_2\right)} \nonumber
\end{eqnarray}
and second derivative is

\begin{eqnarray}
\frac{d^2P}{dx^2} & = & y_1 \frac{2}{\left(x_1-x_2\right)\left(x_1-x_3\right)} \nonumber \\
     & + & y_2 \frac{2}{\left(x_2-x_1\right)\left(x_2-x_3\right)} \nonumber \\
     & + & y_3 \frac{2}{\left(x_3-x_1\right)\left(x_3-x_2\right)} \nonumber
\end{eqnarray}

We are only interested in evaluating these at the middle point $x=x_2$. The second derivative is
indpendent of $x$ anyway but the first derivative evalued at $x=x_2$ is

\begin{eqnarray}
\frac{dP}{dx}(x=x_2) & = & y_1 \frac{\left(x_2-x_3 \right)}{\left(x_1-x_2\right)\left(x_1-x_3\right)} \nonumber \\
     & + & y_2 \frac{\left(2x_2-x_1-x_3 \right)}{\left(x_2-x_1\right)\left(x_2-x_3\right)} \nonumber \\
     & + & y_3 \frac{\left(x_2-x_1 \right)}{\left(x_3-x_1\right)\left(x_3-x_2\right)} \nonumber
\end{eqnarray}

If we assume that the $x_i$ are sorted to be increasing and there are no duplicates, this can be
written in matrix form, with first and second derivative matrices $F$ and $D$ being
block diagonal with block length 3 and specified by the last two equations.

For equally spaced points of unit separation, i.e. $x_2 = x_1 +1$ and $x_3 = x_1 + 2$ they reduced to the
usual formulas for finite difference derivatives,
$F = BlockDiag((-0.5,0,0.5))$ and $D = BlockDiag((1,-2,1))$. These results for unequally spaced
points should give results exactly equal to analytical calculations when evaluated on quadratic, linear
or constant functions. These two facts provides a useful test for any implementation.

\subsection{Proof of Equation \ref{eq:supinf}}

We wish to prove that
\begin{equation}
\sup_\nu \inf_z \left(-\nu^T z + \lambda ||z||_1 \right) =
\left\{
    \begin{array}{ll}
   0 ~\mbox{if}~ |\nu| < \lambda \\
  -\infty  & \mbox{otherwise.} \\
  \end{array}
  \right.
\end{equation}

Note that this equation is proven if we prove it for each component that is summed so
we can drop vector notation and treat them as scalars.

First define $f(\nu) = \inf_z \left(-\nu z + \lambda |z| \right)$. Notice the important fact
that $f(\nu) = f(-\nu)$ because if the $\inf$ is achieved at $z$ for some $\nu$ then the same
$\inf$ can be achieved at $-z$ for $-\nu$. Thus it is symmetric in $\nu$, $f(\nu) = f(|\nu|)$.
Note also that $f(0) = 0.$

Now let's consider the case $|\nu| \le \lambda$. Clearly $\nu z \le |\nu z|$ so
$f(\nu) = \inf_z \left(-\nu z + \lambda |z| \right) \ge  \inf_z \left(|z| (-|\nu| + \lambda) \right)
\ge 0$. So $f(\nu) = 0$.

Now consider the other case $|\nu| > \lambda$. Now for any positive $z$,
$f(\nu) < \left((-\nu + \lambda) |z| \right)$ which has $inf_z = -\infty$.

\section{References}
S.-J. Kim, K. Koh, S. Boyd, and D. Gorinevsky
SIAM Review, problems and techniques section, 51(2):339–360, May 2009.\\
http://stanford.edu/\~boyd/papers/pdf/l1\_trend\_filter.pdf

\end{document}