\documentclass[12pt]{article}
\usepackage{amssymb, amsmath,algorithmic,algorithm,natbib}

% \usepackage[dvipdfm,bookmarks=true,bookmarksnumbered=false,bookmarkstype=toc,hyperfootnotes=false, citebordercolor={1 1 1} ,linkbordercolor={1 1 1} ]{hyperref}

\def\d{{\mathrm{d}}}
\def\E{{\mathrm{E}}}
\def\Var{{\mathrm{Var}}}
\def\Cov{{\mathrm{Cov}}}
\def\max#1{{\underset{#1}{\mathrm{max}}}}
\def\min#1{{\underset{#1}{\mathrm{min}}}}
\def\({\left(}
\def\){\right)}
\def\R{{\mathbb{R}}}
\def\pdiff#1#2{\frac{\partial #1}{\partial #2}}
\def\xb{{\mathbf{x}}}
\def\fb{{\mathbf{f}}}
\def\nub{{\mathbf{\nu}}}
\def\gb{{\mathbf{g}}}
\def\gammab{{\mathbf{\gamma}}}
\def\pb{{\mathbf{p}}}
\def\db{{\mathbf{d}}}
\def\0{{\mathbf{0}}}
\def\1b{{\mathbf{1}}}
\def\deltab{{\boldsymbol{\delta}}}
\newcommand{\Gammab}{{\mathbf{\Gamma}}}
\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\setlength{\oddsidemargin}{0 in}
\setlength{\evensidemargin}{0 in}
\setlength{\textwidth}{6.5 in}
\setlength{\topmargin}{-.5 in}
\setlength{\textheight}{9 in}

\pagestyle{plain}
\begin{document}

\begin{center}
 {\Large A manual for the Modified Powell's Hybrid Method} {\large by Yoki Okawa }\\[0.5cm]
 {\large \today}\\[0.5cm]
\end{center}

\section{Introduction}

In this document, I explain the modified Powell's Hybrid method to solve the system of nonlinear
equations. The modified Powell's Hybrid method has several advantages: avaiability of public domain high quality code; and popularity in various packages. This method seems ``the method'' used in almost all packages I checked so far. It includes IMSL, NAG, Matlab,
and Octave. The algorithm is based on \cite{Powell1970, Powell1970a}. This
algorithm was implemented in MINPACK, a high-quality optimization package in public
domain\footnote{You can access MINPACK from http://www.netlib.org/minpack/ }. It is implemented in
Fortran 77 without the detail of the algorithm.

This document is intended to provide the code and explanation at the same time. There are many good
explanations of the idea behind this algorithm. For example, see \cite{NocedalWright2000}. Also,
there are high-quality free codes like MINPACK. But it is hard to connect two. It is easy to get confused with what is "psi" or "flag3" in the code. So I am trying to build a
bridge between algorithm and code by creating a one-to-one mapping of code and documentation.

\section{Basic Idea}
For the basic idea of trust region method and nonlinear optimization in general,
\cite{NocedalWright2000} provide readable yet detailed explanation.

We consider a following problem.
\[
\begin{pmatrix}
f_1(x_1,\dots,x_n)\\
f_2(x_1,\dots,x_n) \\
\vdots\\
f_n(x_1,\dots,x_n) \\
\end{pmatrix}
=
\begin{pmatrix}
0\\
0 \\
\vdots\\
0\\
\end{pmatrix}
\]
Or, equivalently,
\be \label{eq:prob}
\fb (\xb) = \0.
\ee

Instead of solving the equation directly, we consider the minimization of the residual function
\[
r(\xb) = \|\fb(\xb)\|^2 = \sum_{i=1}^n f_i(\xb)^2
\]

The nonlinear equations are solved by repeatedly solving the locally Quadratic approximation of
the minimization problem of $r(\xb)$ around the point $\xb_i$:

\be \label{eq:unconditionalQuadratic}
\min{\pb} \left[ r(\xb_i) + \tilde{\pb}^T J^T\fb(\xb_i)  +
        \frac{1}{2} \pb^T J^T(\xb_i) J(\xb_i)\pb\right]
\ee
where $J$ is a Jacobian matrix and
\[
J_{kj} =\pdiff{f_k(\xb)}{x_j}
\]
Quadratic approximation, which takes advantage of second derivative. Benefit of using second derivative is substantial. The first order convergence, only using first derivative will typically make the error half per iteration. This means $10^{3}$ improvement requires 10 iterations. In the second order convergence, the error becomes square of the original error. If original error was $10^{-3}$, $10^{3}$  improvement can be obtained by only one iteration.

The challenge for second order methods are calculating the second order derivative, or the Hessian. The nice point here is that we do not explicitly calculate the Hessian of $r(\xb)$ because its functional form of the sum of the square implies simple form of the Hessian, $ J^T(\xb_i)
J(\xb_i)$
The solution to the problem is
\[
\pb = - J^{-1}\fb(x_i),\qquad \xb _{i+1} = \xb_i + \pb_i
\]
This is the Newton-Raphson algorithm.

Newton-Raphson update can fail spectacularly when Jacobian changes fast. For illustration, let's
consider one dimensional problem $f(x) = x^3+1 = 0.$. It has a unique solution at $x = -1 $. Note that $f'(x) = 2 x^2$, which is a strictly concave function. Someone may expect nice convergence features.   However,
if we start from $x_0=0.01$ and apply the Newton-Raphson update, $x_1 $ is 3090. It is not approaching to the true solution at all.

Why the Newton Raphson failed? Since $f'(0.01) = 0.003$, the algorithm thinks $f(x)$ does not
respond sharply with the change $x$. And $f(0.01)$ is $ 1.0303$, which is not too close to zero. So
the algorithm thinks it has to take a large step to make $f(x)$ zero.

The problem is that the Newton-Raphson algorithm is taking locally Quadratic approximation too
seriously. The approximation is valid for the region such that Jacobian is not moving too fast. In
our example, $f'(-0.1) = 0.3$, which is more than hundred times larger than the case of $x=0.01$. So it
is not a good idea to move $x$ by three thousands. One solution is restricting the step size endogenously.  We modify \eqref{eq:unconditionalQuadratic} by the following manner:
\begin{align}
\min{\pb} &\left[ r(\xb_i) + \tilde{\pb}^T J^{T}\fb(\xb)  +
    \frac{1}{2} \pb^T J^T(\xb_i) J(\xb_i)\pb\right]
\label{eq:conditionalQuadratic} \\
&\mathrm{subject \ to}\qquad \|\pb\| \le \Delta
\end{align}
$\Delta$ is our parameter for the size of the region we trust the quadratic approximation. We
adjust $\Delta$ smartly (details below).

\section{Algorithm}
\subsection{Overview}
This is a basic component of the algorithm.
\begin{algorithm}[ht]
\caption{Main Algorithm (Subroutine Hybrid())}
\label{al:main}
\begin{algorithmic}[1]
\STATE Set Initial Value of $\Delta$ and $J$
\STATE Do QR Decomposition of $J$: $QR = J$
\WHILE{$\text{abs}(F(\xb))>tol$}
    \STATE Calculate $\pb$ given $Q,R$ and $\Delta$
    \IF{$F(\xb+\pb)<F(\xb)$} \label{ln:improvedF}
        \STATE $\xb\leftarrow \xb+\pb$
        \STATE Calculate $\fb(\xb+\pb)$
    \ENDIF
    \STATE Update $\Delta$ using $F(\xb+\pb),F(\xb),J\pb$
    \STATE Update $J$ using $f(\xb)$
    \STATE Update $Q,R$ using $J$
%   \IF{$\frac{\|\fb(\xb)\|-\|\fb(\xb+\pb)\|}{\|\fb(\xb)\|-\|\fb(\xb)+J\xb\|}<0.1$}
\ENDWHILE
\end{algorithmic}
\end{algorithm}
First, we decompose the Jacobian as $J = QR$.

The first step of the loop is finding the possible increment $\pb$. This is a compromise of Newton
direction and steepest descents of $F(\xb)=\sum_{k=1}^Nf_k(\xb)^2$ called dogleg method.

$\Delta $ is the size of the trust region.  We adjust $\Delta$ based on how well the current Jacobian is
fitting the data. We decrease $\Delta$ if the current improvement is not as good as we hoped because
it means our Jacobian is a poor proxy of the true nature of the objective function. We do not want
to move a lot based on a poor estimate.

Next, we update the Jacobian. We usually use Broyden's Rank 1 update for fast calculation. If the last
few steps was bad move, it might imply our Jacobian contains substantial errors. So we recalculate
Jacobian in that case. The last step of the iteration is an update of $Q, R$.  Because of their special
structure, it is relatively fast. There are many exceptional exits of the program. We explain it at
the end of the algorithm section.

\paragraph{Convergence}
This algorithm is designed so that if the Newton method fails to improve the residual function
$r(\xb)$, it chooses the Steepest Descent method with a gradually smaller step length. A small step
to the Steepest Descent direction always improves the objective function value. So the convergence
to some local optimal point is mathematically guaranteed. The mathematical theorem does not
guarantee convergence in a reasonable time and precision. But this algorithm pays
special attention to rounding errors and speed. So it would also likely to converge, at least much more than the Newton-Raphson.

\subsection{Update of $\pb$}
 \begin{algorithm}[ht]
\caption{Calculate  $\pb$, (subroutine dogleg)}
 \label{al:dogleg}
\begin{algorithmic}[1]
\STATE INPUT: $R,Q,\Delta,\fb(\xb),\Psi$
\STATE OUTPUT: $\pb$
\STATE Calculate Newton Correction by solving $R\boldsymbol{\nu} = - Q^{T} \fb (\xb)$
\IF{$\|\boldsymbol{\nu}\|\le \Delta$}
    \STATE  Accept Newton Correction : $\pb \leftarrow \boldsymbol{\nu}$
\ELSE
    \STATE /* Try Steepest descent Correction */
    \STATE Calculate steepest descent Correction: $\boldsymbol{g} = - (QR)^{T} \fb (\xb)$
    \STATE Calculate the predicted bottom at the direction of $\boldsymbol{g}$:
             $\mu = \| \boldsymbol{g} \|^2 / \| QR\boldsymbol{g} \|^2$
    \IF{$\mu \|\Psi^{-1} \gb\| \ge \Delta$}
        \STATE Move along Steepest descent correction $\pb = \Delta \gb/\|\gb\|$
    \ELSE
        \STATE /* Accept neither. Try linear combination */
        \STATE Find $\theta$ such that $\|(1-\theta)\mu  \gb + \theta\Psi \nub \| = \Delta$
        \STATE $\pb \leftarrow (1-\theta)\mu \gb + \theta \nub$
    \ENDIF
\ENDIF
\end{algorithmic}
\end{algorithm}

This is an update of $\pb$ by solving \eqref{eq:conditionalQuadratic}. We do not solve the problem
precisely because it takes time and gain is not that big.

First, we try the Newton Correction, which is an unconditional solution of \eqref{eq:conditionalQuadratic}. The solution of the
linear equation takes just $O(n^2)$ operation because $R$ is an upper triangular matrix. We can apply
successive substitutions. If it is not moving more than $\Delta$, that is the exact solution.

If not, we try the Steepest descent direction of $r(\xb)=\fb(\xb)^T\fb(\xb)$, which is $-J^T
\fb(\xb)$.
In a usual Steepest Descent method (or a more sophisticated algorithm like Conjugate
gradient method or Quasi-Newton method), we perform line search along the suggested direction to
find minimum on the line. But we don't go for that because the line search requires several
evaluations of the objective function, which can be costly.

Instead, we explores the special feature of $r(\cdot)$. Since $r(\cdot)$ is the sum of squares, we can derive the Hessian without
performing error-prone finite difference. Namely, the Hessian is $J^TJ$. With Hessian and
Jacobian, we can directly derive the minimum of a function that is a quadratic approximation of
the objective function at the point, which is $\xb + \mu \gb$. In this way, we can avoid costly
line search. If the minimum is outside of the permissible area, or $\mu \|\gb\| \ge \Delta$, we
accept up to $\Delta$.

If the minimum along $\gb$ is closer than $\Delta$, we move a bit to the $\nub$ too. We take a
linear combination of both $\boldsymbol{\nu}$ and $\gb$ so that $\|\pb\| = \Delta$. This can be
attained by setting
\[
\theta = \frac{\Delta^2 - \|\mu \gb \|^2}
{(\nub-\mu\gb,\mu\gb)+\sqrt{\{(\nub,\mu\gb)-\Delta^2\}^2+\{\|\nu\|^2-\Delta^2\} \{\Delta^2-\|\mu\gb\|^2 \}}}
\]

\paragraph{Normalization}
In this algorithm, it is important to normalize equations so that the problem would be well
behaved. Instead of considering the constraint in \eqref{eq:conditionalQuadratic} as a ball, we
can consider an eclipse constraint.
\[
 \|\Psi \pb\| < \Delta
\]
where  $\Psi$ is a diagonal normalization matrix. Let $\Psi \pb = \tilde{\pb}$. The problem
\eqref{eq:conditionalQuadratic} becomes
\begin{align}
\min{\tilde{\pb}} &\left[ r(\xb_i) +\tilde{\pb}^T\Psi^{-1} J^T\fb(\xb_i)  +
    \frac{1}{2} \tilde{\pb}^T \Psi^{-1}J^TJ\Psi^{-1}\tilde{\pb}\right]
\label{eq:conditionalQuadratic} \\
&\mathrm{subject \ to}\qquad \|\tilde{\pb}\| < \Delta
\end{align}
The solution of the problem above, $\tilde{\pb}$ is equal to the solution of unconstrained dogleg
problem with $\tilde{J} = J\Psi ^{-1}$. So the dogleg problem with normalization is follows:

\begin{algorithm}[ht]
\caption{Calculate  $\pb$ with normalization}
 \label{al:delta}
\begin{algorithmic}[1]
\STATE INPUT: $R,Q,\Delta,\fb(\xb),\Psi $
\STATE OUTPUT: $\pb$
\STATE $ \tilde{R} \leftarrow R \Psi^{-1}$
\STATE Apply algorithm \ref{al:dogleg}. Let its solution as $\tilde{\pb}$
\STATE $\pb = \Psi^{-1} \tilde{\pb} $
\end{algorithmic}
\end{algorithm}

\subsection{Revision of $\Delta$ and Recalculating $J$}
Algorithm \ref{al:delta2} is a update of the trust region size, $\Delta$.

\begin{algorithm}[ht]
\caption{Update $\Delta$ (subroutine UpdateDelta)}
\label{al:delta2}
\begin{algorithmic}[1]
\STATE Calculate predicted $f(\xb+ \delta)$: \ $\boldsymbol{\phi} = \fb(\xb)+ J\pb$
\STATE Calculate Predicted $r(\xb +\delta)$: \  $\Phi =\|\boldsymbol{\phi}\|  $
\IF{$\frac{r(\xb)- r(\xb+\pb)}{r(\xb)-\Phi} <0.1 $} \label{al:delta2:l1}
    \STATE  Change of objective function is too small. Reduce $\Delta$ by $\Delta
    \leftarrow DeltaSpeed\cdot \Delta$ \label{line1}
    \STATE GoodJacobian = 0
    \STATE BadJacobian = BadJacobian + 1
\ELSE
    \STATE BadJacobian = 0
    \STATE GoodJacobian = GoodJacobian +1
    \IF{GoodJacobian$ > 1$ OR $\frac{r(\xb)- r(\xb+\pb)}{r(\xb)-\Phi} >0.5 $}
        \STATE $\Delta = \max{}(\Delta, 2\|\pb\| )$
    \ELSIF{ abs$\big(\frac{r(\xb+\pb)-\Phi}{r(\xb)-\Phi}\big)< 0.1 $}
        \STATE $\Delta =2\|\pb\| $
    \ENDIF
\ENDIF
\IF {BadJacobian = 2}
    \STATE Recalculate Jacobian by forward differences.
    \STATE BadJacobian = 0
\ENDIF
\end{algorithmic}
\end{algorithm}

Line \ref{al:delta2:l1} checks if $J$ is good or not. If $J$ is a good prediction,
the predicted reduction of the objective function $r(\xb)-\Phi $ is almost equal to the
actual reduction. If the actual reduction is less than 10\% of the predicted reduction,
there is something wrong with $J$. It is either $\fb(\xb)$ is very nonlinear or
calculated $J$ contains too much error from the history. These errors generally become
smaller if the trust region $\Delta$  shrinks.

If the reduction is relatively satisfactory, we start to weakly increase $\Delta$. There is not
too much justification for this increase. But Powell found that the result is numerically quite
satisfactory. Note that usually $\|\pb\|$ is equal to $\Delta$.  We do not want to modify
$\Delta$ all the time because we might fall into the oscillation of increase $\rightarrow$
decrease $\rightarrow $ increase $\cdot \cdot \cdot$.  To avoid that, GoodJacobian is checking if
it is the first attempt to increase $\Delta$. If it is after the second attempt to increase $\Delta$ and
we attain a modest increase in the objective function, which is that actual reduction is more than 50\%
of predicted reduction, we double the trust region. If our Jacobian is a pretty good estimate
($r(\xb+\pb)-\Phi$ is small), we also expand the trust region.

BadJacobian is counting the number of consecutive failures of predicting changes in the objective
function. If the algorithm fails to predict the change twice consecutively, we suspect that
the current Jacobian contains a serious error which came from previous updates. The previous points may
be far from the current $\xb$. We recalculate it based on the forward difference from scratch, instead of
updating it.

\subsection{Update of J}
We update Jacobian using the Broyden's rank 1 update. For the detail, see Numerical Recipes
Ch. 9.7., Broyden's method.   :
\begin{align*}
  J \leftarrow J +\frac{1}{\|\pb\|^2} (\fb(\xb+\delta)-f(\xb) -J\pb )\pb^T
\end{align*}

\subsection{QR Decomposition}
QR decomposition is decomposing a matrix $A$ to multiple of $Q$, orthogonal matrix ($Q^{-1}=Q^T$),
and upper triangular matrix $R$. So $A=QR$. It is useful because solving $Ax=b$ takes $O(n^2)$
step once we know QR decomposition of A, which is faster than $O(n^3)$. Although QR decomposition
itself takes $O(n^3)$, once we calculate the decomposition, its update only takes $O(n^2)$. So it
is useful when we have to solve the linear equation repeatedly with slightly different
coefficients. It is the case when we calculate the Newton direction, which requires to solve
$J\delta = \fb(\xb)$ in every iteration. In this subsection, I assume that input is a square
matrix. The Fortran code is written in the way it accepts the rectangular matrix.

\subsubsection{Decomposition}
This is called at first and whenever $J$ is recalculated using finite difference.  The code uses
Householder transformation repeatedly. Let $a_j$ as an arbitrary vector and we want to find $P_j$
such that $P_j$ is orthogonal and $P_ja_j = (b_1, 0, \dots,0)^T$. We can set
\[
P_j = I - \frac{2}{\|u_j\|^2} u_ju_j^T, \qquad u_j = a_j - \|a_j\|e_1
\]
where $I$ is identity matrix and $e_1= (1,0,\dots,0)^T$ is a unit vector. We can apply this
repeatedly until all matrix becomes upper triangular. Many books about numerical algorithm covers
QR decomposition with Householder transformation.

\subsubsection{Update}
The update is called at the end of each iteration. Our goal is given
\[
A^{next} = A + uv^T, \qquad A = QR
\]
find $Q^{next}$ and $R^{next}$ such that $Q^{next}R^{next}=A^{next}$. For more information, see \cite{BjorckDahlquist2008}
section 8.4.

\paragraph{transform a bit}
Let $w=Q^Tu$. Using this,
\[
A^{next} = Q^{next}(R+wv^T)
\]
We are going to make $R+wv^T$ upper triangular matrix using Givens transformation.

\paragraph{Givens transformation of $w$}
We find a sequence of Givens transformation such that
\[
P_1\dots P_{n-1}w = \alpha e_1.
\]
Givens transformation is very similar to Jacobi transformation. It is a matrix defined by three
parameters $(k,l,\theta)$ such that $(i,j)$
element is
\[
P(k,l,\theta)_{i,j} = \begin{cases}
\cos \theta & i=j=k \\
\cos \theta & i=j = l\\
-\sin \theta & k = i , l = j \\
\sin \theta & k = j, l=i \\
1 & i = j \neq k, i = j \neq l\\
0 & otherwise
\end{cases}
\]
It is zero other than diagonal elements, $(k,l)$ element, and $(l,k)$ element. $P$ is orthogonal
matrix.  Consider
$P(n-1,n,\theta) w$.  $n$th element of $P(n-1,n,\theta) w$ is zero if
\[
\cos \theta = \frac{1}{\sqrt{t^2+1}}, \qquad
\sin \theta = t \cos \theta, \qquad t = \frac{w_n}{w_{n-1}}
\]
We can repeat this until all element other than the first element is zero.
Let
\[
P^w = P_1\dots P_{n-1}
\]

\paragraph{Transform upper Hessenberg matrix to upper triangular matrix}
Now,
\[
A^{next} = Q (P^w)^T(P^WR+P^wwv^T)
\]
Note that elements in $P^wwv^T$ are zero other than the first row and $P^WR$ is upper Hessenberg
matrix. So $P^WR+P^wwv^T$ is an upper Hessenberg matrix. We are going to eliminate (1,2) elements to
(n-1,n) elements using Givens transformation.  Let $H =P^WR+P^wwv^T$. To eliminate (1,2) element,
we need Givens transformation such that
\[
\cos \theta = \frac{1}{\sqrt{t^2+1}}, \qquad
\sin \theta = t \cos \theta, \qquad t = \frac{h_{21}}{w_{11}}
\]
We can repeat this until all lower triangular elements are zero. Let $P^H$ as the accumulation of
the transformation. We complete the algorithm by
\[
Q^{next} =  Q (P^w)^T(P^H)^T, \qquad R^{next} = P^H(P^WR+P^wwv^T)
\]

\subsection{Termination of the Algorithm}
There are several conditions for the termination of this algorithm.

\paragraph{Successful Convergence}
 We call it successful if
\[
\frac{1}{\sqrt{n}}\|\fb(\xb)\|=\sqrt{\frac{1}{n}\sum_{i=1}^m(f_i(\xb))^2} < ftol
\]
This means that the average squared residual is small. If this condition is satisfied, $|\fb_i(\xb)|$
is at most $n\cdot ftol1$

\paragraph{Trust region size shrinks to the tolerance level}
The program stops execution if $\Delta < xtol(\|\xb\|+xtol)$, the relative size of the trust
region is smaller than $xtol$. This happens if the Jacobian changes rapidly and wildly around the
point the program terminated. This is the most common type of unsuccessful termination. It can
happen for the following reasons:
\begin{itemize}
\item The problem you are going to solve has large second order derivatives. In this case,
  Jacobian is sensitive to where it is evaluated. The local quadratic approximation is inappropriate. This might be the case if you have highly discontinuous Jacobian, for example.
\item You set the $ftol$ too small. You might get this message even if the algorithm is in indeed
  the solution. If $ftol$ is too small, the algorithm can not attain it with the precision
  required.
\item You set the speed of shrinking $\Delta$ too small.
\item Your subroutine to be solved contains bugs. My personal experience suggests this explains
  most of this error message.
\end{itemize}

Here are possible solutions:
\begin{itemize}
\item Check the $\fb(\xb)$ and see if it is small enough or not.
\item Increase \textit{DeltaSpeed}
\item Increase \textit{ftol}
\item Decrease $JacobianStep$
\item Use different initial guesses
\item If you are using single precision real numbers, use double precision instead.
\end{itemize}

\paragraph{Too much function call}
If the number of function call exceeds MaxFunEval, the program stops the execution.

\paragraph{Algorithm reaches Local minima}
If the program terminates for the reason other than ``Successful Convergence'', we recalculate the
gradient of the objective function $\nabla r(\xb) = J^T\fb(\xb)$ by finite difference. If
$\|\nabla r(\xb)\| < gtol(\|\xb\|+gtol)$, we are at the locally optimal point, although average
residual is not close enough to zero in the tolerance required. This happens if the algorithm is
captured in the local minima. The solution is trying the different initial guess. This can also
happen if $ftol$ is too small to attain. You can check that by seeing the $\fb(\xb)$ at the
output.

\section{Usage of the Subroutine}
I explain inputs and outputs of this subroutine. Many arguments are optional.
\begin{description}
    \item[fun] Subroutine which returns the residual vector. It should take two arguments, and
      first argument should be $\xb$ and second argument should be the $\fb(\xb)$.
    \item[x0] Initial value of $\xb$
    \item[xout] Solution of the least square problem.
    \item[info] (optional) Variable for the information of the termination of the algorithm. For
      the detail of the conditions, see the Termination of the Algorithm.
    \begin{description}
        \item[info = 0] Improper input values
        \item[info = 1] Successful convergence.
        \item[info = 2] First order optimality satisfied
        \item[info = 3] Trust region size shrinks to the tolerance level
        \item[info = 4] Too much function call
    \end{description}
    \item[fvalout] (Optional) Output vector of residuals at $\xb = xout$.
    \item[Jacobianout]  (Optional) Output Jacobian Matrix at $xout$
    \item[xtol] (Optional) Tolerance of the x. If trust region size is smaller than
      $xtol(\|\xb\|+xtol)$, the program terminates. If this is not provided, it would be set to
      the default value of $10^{-8}$
    \item[ftol] (Optional) Tolerance of the residual. The program terminates if
      $\sqrt{\sum_{i=1}^m(f_i(\xb))^2/m} < ftol $. If this is not provided, it would be set to
      the default value is $10^{-8}$.
    \item[gtol] (Optional)  Tolerance of the gradient. If  $ \|\nabla r(\xb)\| <
      gtol(\|\xb\|+gtol)$ holds, we regard it first order optimality is satisfied. The default
      value is $10^{-8}$
    \item[JacobianStep] (Optional) Relative step size for calculating the Jacobian by Finite
      difference. Default value is $10^{-3}$.
    \item[display] (Optional) Option to specify the display preference. If display is 2, the
      algorithm shows the iteration. If display is 0, the algorithm shows nothing. Default value
      is 0.
    \item[maxFunctionCall] (Optional) Number of function to be called before
      terminating the algorithm. Default value is $100n$, where $n$ is number of equations.
    \item[noupdate] (Optional) Dummy variable for the Broyden's rank 1 update. If noupdate is 1,
      the algorithm recalculate Jacobian with finite difference in each iteration. This would
      almost always make the algorithm converged with fewer iteration. But this does not mean it
      would speed up in terms of CPU time because in general the calculation of the Jacobian is
      costly if the evaluation of the objective function is costly or $n$ is large. Turning off
      the update can speed up the total computation time when $n$ is very small, (say less than
      5). Default value is 0, so the Broyden's update is the default.
    \item[DeltaSpeed] (Optional) Controls the speed of $\Delta$ when it shrinks. In algorithm
      \ref{al:delta2}, line \ref{line1}, the trust region size $\Delta$ shrinks if the actual
      reduction do not match with predicted reduction. DeltaSpeed controls how fast it should
      shrink. It should always be always less than one. The smaller the value is, the faster the
      $\Delta$ shrinks. The default value is $1/4$.
\end{description}

\section{Other issues}

\subsection{Improving Performance}
Current implementation prefer the readability of the code to performance. If you are dealing with
large scale problems, you might want to change some parts of the code. In general, this code is
written for the case that number of parameters to be solved is not that large, (for example, less
than 50) and evaluation of the objective function is costly. If this is not your problem, you
might be able to get better performance for the following modifications.

\paragraph{QR factorization}
Other than the evaluation of the objective function, $QR$ decomposition would take most of the
time in this algorithm. It is called at first and called occasionally afterward if the current
approximation is suspected to contain substantial errors. If $n$ or $m$ is large (more than a few
hundred $n$ or more than 10,000 $m$) and the evaluation of the objective function is not
expensive, MINPACK's slow implementation of QR decomposition is likely to be a bottleneck. You
should consider using high-quality linear algebra package like LAPACK. Also, LAPACK routine would
improve the numerical stability of QR decomposition.

\paragraph{Better solution of Locally quadratic problem}
Also, faster speed can be attained by improving the dogleg method to the nearly exact solution
using Cholesky factorization. It takes more time than the dogleg method per iteration. But improvement
per iteration will be better. This appropriate when the evaluation of the function is costly and
the number of the variables are small.

\paragraph{Memory}
The memory should not be a concern for modern computers if $n$ is less than 1000.

If $n$ is larger than that, it might create a problem. This code stores a few $n$ by $n$
matrixes. To give you an idea for how serious this constraint is, here is the memory required to
store matrixes: To store a 1,000 by 1,000 matrix with 8 bytes (double) precision, it requires 8
Megabytes of memory. For 5,000 by 5,000 matrix, that is 190 Megabytes and for 10,000 by 10,000
matrix, that is 762 Megabytes. In case memory binds, it is not too difficult to decrease the
memory usage by 50\% or more with a little modification. For more memory savings, probably you
need a substantial rewrite of the code based on conjugate gradient method instead of Newton Based or
use sparse matrix.

\subsection{Comparison with various methods}
Here is the comparison of various methods with this method.

\paragraph{Compared to Newton-Raphson method with BFGS, DFP, or BHHH} Great advantage of
Newton-Raphson type method is its speed. But it is unstable, even if we update Jacobian/Hessian
smartly using BFGS, DFP, or BHHH. The Newton Raphson method just has local convergence. It
might fall into a cycle or diverge even if the function is smooth and globally concave unless the
initial guess is sufficiently close to the true value. So here is the origin of the agony of
initial guess. Also, Newton's method might take you to the saddle point. (see
\cite{BjorckDahlquist2008} section 11.3 for more discussion) On the other hand, the modified
Powell's Hybrid has global convergence property. It always converges if the first derivative is
continuous and bounded, no matter what the initial point is. This is a great improvement in terms
of convergence. For speed, the Hybrid method tries to use Newton-Raphson update whenever it looks
safe. The Hybrid method is as good as that in Newton-Raphson if the initial guess is good. If the
initial guess is not so good, only the Hybrid method has guaranteed converges.

\paragraph{Compared to Newton-Raphson with Line search}
(see \cite{BjorckDahlquist2008} section 11.3 for more discussion) To obtain global convergence,
there are two ways. One is the trust region method which the Hybrid method is relied on, and the
other is the line search method. They are different in what to do after we get the direction for
improvement. For line search, it searches the extremum along the direction. And trust-region
picks step size without function evaluation by local quadratic approximation.

\paragraph{Compared to Conjugate Gradient and its derivatives}
The Conjugate Gradient method stops relying on second order derivative entirely. Second order derivative is $n$  by $n$  matrix, which won't fit in the memory if $n$  is very large, such as the deep learning. The Powell's method cannot be applied for that case.

\paragraph{Compared to Ameba/Downhill Simplex } Their advantage is an ability to deal with
discontinuity. If the objective function contains severe discontinuities near the solution, you
should consider this type. But it is VERY slow. In my experience, the Hybrid method is not that fragile to the mild discontinuity in both objective function and its derivative because of its trust region feature, although there is no mathematics to support the claim. I suggest trying to the Hybrid method first because it is much
faster. Sophisticated termination criterion might be fooled by the discontinuity so you might want
to restart it once it converges.

\paragraph{Compared to Grid Search/Simulated annealing}
The global convergence property of the Hybrid method is that it would converge to local
extrema. Like any derivative-based method, it does not guarantees the global property of the point
obtained. If there are many local extrema and you need a global solution, probably there is no
choice other than Grid search or other globally convergent methods like the simulated annealing.

\paragraph{Compared to bisection/Brent's method} If the problem contains only one equation, we
have completely different algorithms. Bisection and its improvement, Brent's method, is regarded
as robust algorithms that can be applied to highly discontinuous functions. The problem is that it
requires two initial guesses which brackets the solution. It is not always easy to find them. I am
not sure if they are so robust or fast if we consider the step of finding the bracket.  The hybrid
algorithm is faster than these algorithms with only one initial guess. But it may be trapped at
the local extrema of the function. So it is your call to pick.

\paragraph{Fixed point approach} \cite{SuJudd2008} is fiercely criticizing this approach. It
makes some sense. But their solution, extensive usage of the canned package is not always a good idea. It is
not so hard to understand the algorithm, once we have nice documentation.

\section{Glossary}
Here, $\fb(\xb)$ is system of equations which we want to make it zero, and $F(\xb)$ is scalar
objective function which we want to minimize.
\begin{description}
    \item[Jacobian] Jacobian is a matrix of first order derivatives, which comes from a vector
    valued function. \[
    J_{ij} =\pdiff{f_i(\xb))}{x_j}\qquad J = \begin{pmatrix}
    \pdiff{f_1(\xb))}{x_1} & \cdots & \pdiff{f_1(\xb))}{x_n} \\
    \vdots & \ddots & \vdots \\
    \pdiff{f_n(\xb))}{x_1} & \cdot
 & \pdiff{f_n(\xb))}{x_n} \\
     \end{pmatrix}
    \]
    \item[Gradient] Gradient is a vector of first order derivatives, which comes from a scalar
      valued function. This is another name of first order conditions.  $\bigtriangledown F(\xb) =
      (\pdiff{F(\xb))}{x_1},\dots,\pdiff{F(\xb))}{x_N} )$
    \item[Hessian] Hessian is a matrix of second order derivatives, which comes from a scalar
      valued function.  $H = \bigtriangledown^2 F(\xb) =\bigtriangleup F(\xb)$. $
      H_{ij}=\frac{\partial^2 F(\xb)}{\partial x_i \partial x_j}$. Hessian is a symmetric and
      positive semidefinite.
    \item[Hessian and Jacobian] Since the optimization problem is the same as finding zero in the first
      order conditions, Hessian of the objective function is Jacobian of first order
      conditions. But Jacobian from the first order conditions behaves better than ordinary
      Jacobian, because Hessian has several special features. Many variants of the Newton-Raphson
      algorithm exploit the fact for an efficient algorithm.
    \item[Newton-Raphson algorithm] This algorithm is both for solving a system of nonlinear
      equations and optimization. If it is applied for solving a system of nonlinear equations, it uses the locally linear approximation of the equations. Linear approximation requires a slope,
      which is Jacobian. If it is applied for the optimization problem, it uses a locally linear
      approximation to the first order conditions. Jacobian of the first order conditions is
      Hessian. So it requires Hessian.
    \item[BFGS/DFP update] Smart way to calculate Hessian in the Newton Raphson algorithm for
      optimization. BFGS update is weakly better than DFP update in a sense that BFGS and DFP
      performs almost identically for most of the problems and there are some cases BFGS is
      clearly better than DFP update.
    \item[BHHH update] Another smart way to calculate Hessian in the Newton Raphson algorithm for
      maximum likelihood problems. Although this is popular for econometricians, plain-vanilla
      implementation of the BHHH update is usually inferior to BFGS/DFP update.
    \item[Gauss-Newton's algorithm] Yet another smart way to calculate Hessian applicable to
      nonlinear least square problem.
    \item[Steepest Descent algorithm/Cauchy algorithm] This algorithm is for solving a
      optimization problem. It simply picks a direction which is inverse of gradient. If the step size
      is sufficiently small, it always improves the value of the objective function. For the
      determination of step size, a line search is usually used. It is also called the Cauchy
      algorithm.
\end{description}

\bibliographystyle{aer}

\ifx\undefined\bysame
\newcommand{\bysame}{\leavevmode\hbox to\leftmargin{\hrulefill\,\,}}
\fi
\begin{thebibliography}{xx}

\harvarditem[Bjorck and Dahlquist]{Bjorck and
  Dahlquist}{2008}{BjorckDahlquist2008}
{\bf Bjorck, Ake and Germund Dahlquist}, {\it Numerical Methods in Scientific
  Computing Volume II} 2008.

\harvarditem[Nocedal and Wright]{Nocedal and Wright}{2000}{NocedalWright2000}
{\bf Nocedal, Jorge and Stephen Wright}, {\it Numerical Optimization},
  Springer, 2000.

\harvarditem[Powell]{Powell}{1970a}{Powell1970a}
{\bf Powell, Michael~J.D.}, ``A Fortran Subroutine For Solving Systems of
  Nonlinear Algebraic Equations,'' in Philip Rabnowitz, ed., {\it Nonlinear
  Methods for Nonlinear Algebraic Equations}, Gordon and Breach, Science
  Publishers Ltd., 1970, chapter~7.

\harvarditem[Powell]{Powell}{1970b}{Powell1970}
{\bf \bysame{}}, ``A Hybrid Method For Nonlinear Equations,'' in Philip
  Rabnowitz, ed., {\it Nonlinear Methods for Nonlinear Algebraic Equations},
  Gordon and Breach, Science Publishers Ltd., 1970, chapter~6.

\harvarditem[Su and Judd]{Su and Judd}{2008}{SuJudd2008}
{\bf Su, Che-Lin and Kenneth~L. Judd}, ``Constrainted Optimization Approaches
  to Estimation of Structural Models,'' January 2008, (1460).

\end{thebibliography}

\end{document}


% ----- CHAPTER 3: NOTATION, DEFINITIONS AND BACKGROUND ----- %

%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Notation}\label{subsec:notation}

For the rest of the body of this text we set the following notation:
\begin{itemize}
\item $E$ is an elliptic curve over $\QQ$ given by minimal Weierstrass equation
\begin{equation*}
y^2 + a_1 xy + a_3 y^2 = x^3 + a_2 x^2 + a_4 x + a_6,
\end{equation*}
 where $a_1, a_3 \in \set{0,1}$, $a_2 \in \set{-1,0,1}$ and $a_4,a_6 \in \ZZ$.
\item $D(E)$, $N(E)$ and $r_{al}(E)$  and $r_{an}(E)$ are the discriminant, conductor, algebraic rank and analytic rank of $E$ respectively. For ease of exposition, the dependence on $E$ will most often be indicated by a subscript $E$ instead, and when there is no ambiguity it may be dropped entirely. Also, since much of this body of work assumes the validity of the BSD conjecture, the algebraic and analytic rank of a curve will most often be assumed to be equal, in which case it will just be denoted $r_E$.
\item $p$ is a (rational) prime number and $q$ is a prime power.
\item $s$ is the generic complex variable.
\item $L(E,s)$ and $\Lambda(E,s)$ are the standard and completed $L$-functions attached to $E$ respectively. Again, for ease of exposition we will in general subsume the $E$ into a subscript and write $\Les$ and $\Lams$.
\item $C(E) = C_E$ is the leading nonzero coefficient of the Taylor series of $\Lambda_E(s)$ about $s=1$; \\
$C\pr(E) = C\pr_E$ is the leading nonzero coefficient of the Taylor series of $L_E(s)$ about $s=1$.
\item $\gamma$ will always be used to denote the imaginary parts of nontrivial zeros of an $L$-function.
\item $\beta(E) = \beta_E$ is the bite of $E$, defined as $\beta_E = \sum_{\gamma\ne 0} \gamma^{-2}$, where $\gamma$ ranges over the noncentral nontrivial zeros of $\Les$.
\item $\eta$ is the Euler-Mascheroni constant $= 0.5772156649\ldots$
\item $\Gamma(s)$ is the standard Gamma function on $\CC$, and the digamma function: $\digamma(s) = \frac{\Gamma\pr}{\Gamma}(s)$ is the logarithmic derivative of $\Gamma(s)$.
\end{itemize}

Furthermore, we define the following values associated to $E$ (in all cases the dependence on $E$ is understood):
\begin{itemize}
\item $b_2 = a_1^2 + 4a_2$
\item $b_4 = a_1 a_3 + 2a_4$
\item $b_6 = a_3^2 + 4a^6$
\item $b_8 = a_1^2 a_6 + 4 a_2 a_6 - a_1 a_3 a_4 + a_2 a_3^2 - a_4^2$
\item $c_4 = b_2^2 - 24 b_4$
\item $c_6 = -b_2^3 + 36 b_2 b_4  - 216 b_6$
\item $D = D(E) = -b_2^2 b_8 - 8 b_4^3 - 27 b_6^2 + 9 b_2 b_4 b_6$; this is the definition of the discriminant of $E$
\item $j = j(E) = \frac{c_4}{D}$ is the $j$-invariant of $E$
%\item $\omega = \frac{dx}{2y + a_1 x + a_3} = \frac{dy}{3x^2 + 2a_2 x + a_4 - a_1 y}$
\end{itemize}

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Definitions and Basic Results}

The rest of this chapter covers the basic definitions of and results needed for the rest of this work (namely, big-Oh notation, elliptic curves and $L$-functions). Feel free to skip this if you are familiar with them.

%%%%%%%%%%%%
\subsection{Big-Oh Notation}

Given that the running time of various algorithms will be discussed over the course of this work, we recall the definitions of big-Oh and soft-Oh notation, at least in the context of how they will be used here.

\begin{definition}
Let $x$ be a positive input, and let $g(x)$ be some positive-valued reference function on $x$.
\begin{itemize}
\item We say a function $f(x) = O(g(x))$ (read ``$f$ is big-Oh of $g$''), if
\begin{equation}
\limsup_{x \to \infty} \left| \frac{f(x)}{g(x)}\right| < \infty.
\end{equation}
That is, $f(x) = O(g(x))$ if the asymptotic growth/decay rate of $f$ is bounded by some multiple of that of $g$.
\item We say a function $f(x) = \softO(g(x))$ (read ``$f$ is soft-Oh of $g$''), if there is some $m>0$ such that
\begin{equation}
\limsup_{x \to \infty} \left| \frac{f(x)}{g(x)\left(\log g(x)\right)^m}\right| < \infty.
\end{equation}
That is, $f(x) = \softO(g(x))$ if the asymptotic growth/decay rate of $f$ scales like that of $g$, up to the inclusion of log factors.
\end{itemize}
\end{definition}
Note that $f(x) = \softO(g(x))$ implies that $f(x) = O(g(x)^{1+\epsilon})$ for any $\epsilon>0$, but not vice versa; there are complexity classes strictly between the two. In this thesis we will work exclusively with soft $O$ time complexities, so there is no need to elaborate on those classes here.\\

\begin{definition}
Let $A$ be an algorithm which takes input of size $k$, where for simplicity we may think of $k$ as a positive integer. Let $t_A(k)$ be the running time of $A$ thought of as a function of the input size $k$.
\begin{itemize}
\item $A$ is said to have {\it polynomial time complexity} if there is some $m>0$ such that the running time of $t_A(k) = O(k^{m})$, i.e. the asymptotic running time of the algorithm scales like some polynomial function of $k$.
\item If $t_A(k) = O(k^{\epsilon})$ for any $\epsilon>0$, then $A$ is said to have {\it sub-polynomial time complexity}. Note that if $t_A(k) = \softO(1)$, then $A$ has sub-polynomial time complexity.
\item If no $m>0$ exists such that $t_A(k) = O(k^m)$, then $A$ is said to have {\it super-polynomial time complexity}. If there is some $m>1$ such that $t_A(k) = O(m^k)$, then $A$ is said to have {\it exponential time complexity}.
\end{itemize}
\end{definition}
Again, there are complexity classes strictly between polynomial and exponential complexity, but we won't consider them in this thesis. The same terminology can be applied to the space requirements of an algorithm, wherein we would replace the word `time' with `space'. \\

Note that in theoretical computer science the $k$ is typically the number of bits needed to specify the input to the algorithm. However, in computational number theory the input itself is often a positive integer; many algorithms scale with some polynomial of the input magnitude as opposed to the number of bits defining the input. We therefore highlight the distinction between ``polynomial time in the number of bits of the input" and ``polynomial time in the magnitude of the input": the former is asymptotically much faster than the latter. When discussing time complexities we will always be clear to delineate what the measure of complexity $k$ is. \\

%%%%%%%%%%%%
\subsection{Elliptic curves}

\begin{definition}
An elliptic curve $E$ is a genus 1 smooth projective curve with a marked point $\cO$. $E$ is defined over a field $K$ if $E$ may be represented by the {\it Weierstrass equation} $y^2 + a_1 xy + a_3 y^2 = x^3 + a_2 x^2 + a_4 x + a_6$, where $a1,\ldots a_6 \in K$.
\end{definition}

For elliptic curves defined over $\QQ$, we may always find a model for $E$ such that $a_1, a_3 \in \set{0,1}$, $a_2 \in \set{-1,0,1}$ and $a_4,a_6 \in \ZZ$. Furthermore, there is the notion of {\it minimality} when it comes to models for elliptic curves. Without going into the definition thereof, unless stated otherwise we will assume that any given elliptic curve Weierstrass equation is specified by its global minimal model.

\begin{definition}
The set of $K$-rational points on $E$ is denoted $E(K)$. $E(K)$ comprises an abelian group, with the ``point at infinity" $\cO$ acting as the group identity element.
\end{definition}

It is often useful to view an elliptic curve $E$ as the vanishing locus of the polynomial
\begin{equation}\label{eqn:E_poly}
f(x,y) = y^2 + a_1 xy + a_3 y^2 - x^3 - a_2 x^2 - a_4 x - a_6.
\end{equation}
 That is $E(K) = \set{(x,y) \in K^2: \; f(x,y) = 0}$, along with the point at infinity $\cO$. \\

For a rational elliptic curve $E/\QQ$, we may consider the reduced curve $\tilde{E}/\Fp$ for any prime $p$. If $E/\QQ$ is given by the global minimal model $y^2 + a_1 xy + a_3 y^2 = x^3 + a_2 x^2 + a_4 x + a_6$, then the reduced curve is given by $y^2 + \overline{a_1} xy + \overline{a_3} y^2 = x^3 + \overline{a_2} x^2 + \overline{a_4} x + \overline{a_6}$, where $\overline{a_i}$ is $a_i$ reduced modulo $p$. For $p = 2$ or $3$ we may have to move to a different model for $E$ first to avoid the reduced curve being automatically singular.

\begin{definition}
A prime $p$ is called {\it good} if $\tilde{E}/\Fp$ is non-singular. The reduced curve is an elliptic curve over $\Fp$ (by definition) which we denote by $E/\Fp$; $E$ is said to have {\it good reduction at $p$}. Otherwise, $p$ is said to be {\it bad}, the reduced (singular) curve is denoted $\tilde{E}/\Fp$, and $E$ is said to have {\it bad reduction at $p$}.
\end{definition}

\begin{theorem}
For any $E/\QQ$, the set of bad primes is finite and non-empty.
\end{theorem}

Singular reduced curves may be thought of as finite-field analogues of singular cubics over the rationals, for example those given by $y^2 = x^3$ and $y^2 = x^3+x^2$ as seen below. Singular curves have a (unique) {\it singular point}, which is by definition where the partial derivatives $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ are both zero (here $f$ is as given by equation \ref{eqn:E_poly}).

\begin{figure}[!h]
    \centering
    \includegraphics[width=1.0\textwidth]{graphics/singular_cubics.png}
    \caption{An example of two singular cubics over the rationals. The singular point for both curves is at the origin; for the left curve the singular point is a {\it cusp}, and for the right curve it is a {\it node}.}
    \label{fig:singular_cubics}
\end{figure}

In the finite field setting the notion of partial derivatives still makes sense, so one may define singular points accordingly. Bad reduction at a prime may be classified into one of three types according to the nature of the tangent space at the singular point on $\tilde{E}/\Fp$.
\begin{definition}
Let $E$ have bad reduction at $p$; let $P$ be the singular point on $\tilde{E}/\Fp$, and let $T_P(E)$ be the tangent space at $P$.
\begin{itemize}
\item If the $T_P(E)$ is one-dimensional, then $P$ is a cusp, and $E$ is said to have {\it additive reduction} at $p$.
\item Otherwise $T_P(E)$ is two-dimensional, and $P$ is then a node; $E$ is then said to have {\it multiplicative reduction} at $p$. Furthermore, multiplicative reduction can be decomposed into two cases:
\begin{itemize}
\item If $T_P(E)$ is defined over $\Fp$, then $E$ is said to have {\it split multiplicative reduction} at $p$
\item Otherwise $T_P(E)$ is defined over a quadratic extension of $\Fp$, and $E$ is said to have {\it non-split multiplicative reduction} at $p$.
\end{itemize}
\end{itemize}
\end{definition}

Primes of bad reduction are packaged together into an invariant called the {\it conductor} of $E$:
\begin{definition}
The conductor of $E$, denoted by $N_E$, is a positive integer given by
\begin{equation}
N_E = \prod_{p} p^{f_p(E)},
\end{equation}
where $p$ ranges over all primes, and for $p \ne 2$ or $3$,
\begin{equation}
f_p(E) = \begin{cases} 0, & \text{$E$ has good reduction at $p$} \\ 1, & \text{$E$ has multiplicative reduction at $p$} \\ 2, & \text{$E$ has additive reduction at $p$.}\end{cases}
\end{equation}
For $p=2$ and $3$, the exponent $f_p(E)$ is still zero if $p$ is good; however the exponent may be as large as $8$ and $5$ respectively if $p$ is bad.
\end{definition}
The ``proper" definition of the conductor is Galois representation-theoretic and is defined in terms of the representation of the inertia group at $p$ on the torsion subgroup of $E$; for $p\ne 2$ or $3$ this reduces to the definition given above, but for $2$ and $3$ there may be nontrivial wild ramification which increases the exponent up to the stated amounts. A full technical definition of the conductor is given in \cite[pp. 379-396]{Sil-1994}. In any case (including $2$ and $3$), the exponent $f_p(E)$ may be computed efficiently by Tate's algorithm, as detailed in the previous section of the same book \cite[pp. 361-379]{Sil-1994}. \\

%%%%%%%%%%%%
\subsection{Elliptic curve $L$-functions}

We now move on to the definition of the $L$-function attached to an elliptic curve. For this we must define the numbers $a_p(E)$:
\begin{definition}\label{def:a_p} \mbox{}
\begin{itemize}
\item For good primes $p$ (i.e. when $p \nmid N_E$), let
\begin{equation}
a_p(E) = p+1-\#\set{\EFp},
\end{equation}
where $\#\set{\EFp}$ is the number of points on $E/\Fp$;
\item For bad primes (when $p \mid N_E$), let
\begin{equation}
a_p(E) := \begin{cases}
+1 & \text{if $E$ has split multiplicative reduction at $p$} \\
-1 & \text{if $E$ has non-split multiplicative reduction at $p$} \\
0 & \text{if $E$ has additive reduction at $p$.}
\end{cases}
\end{equation}
\end{itemize}
\end{definition}

Hasse's theorem states that the number of points on $E$ modulo $p$ can never be too far from $p+1$:
\begin{theorem}[Hasse, 1936]
For all elliptic curves $E/\QQ$ and all primes $p$,
\begin{equation}
|a_p(E)| \le 2\sqrt{p}.
\end{equation}
\end{theorem}

For ease of notation, when $E$ is fixed we will let $a_p := a_p(E)$, letting the dependence on $E$ be understood. \\

The Sato-Tate Conjecture, now a theorem thanks to Taylor, goes even further, giving an asymptotic distribution on the $a_p$:
\begin{theorem}[Taylor, 2006-]
For fixed $E/\QQ$, the set of normalized $a_p$ values $\set{\frac{a_p}{2\sqrt{p}}: p \text{ prime}}$ obey a semicircular distribution on the interval $[-1,1]$. That is, for $1\le a \le b \le 1$, the asymptotic proportion of primes for which $a \le \frac{a_p}{2\sqrt{p}} \le b$ is equal to the proportion of the area under the unit semicircle between $a$ and $b$.
\end{theorem}

\begin{definition} \mbox{}

The $L$-function attached to $E$ is a complex analytic function $L_E(s)$, defined initially on some right half-plane of the complex plane.
\begin{itemize}
\item The Euler product of the $L$-function attached to $E$ is given by
\begin{equation}
L(E,s) = \prod_{p} \frac{1}{1 - a_p p^{-s} + \epsilon(p)p^{1-2s}},
\end{equation}
where $\epsilon(p) = 0$ for bad $p$, and $1$ for good $p$.
\item The Dirichlet series for $L_E(s)$ is given by
\begin{equation}
L(E,s) = \sum_{n=1}^{\infty} a_n n^{-s}.
\end{equation}
where for composite $n$, $a_n$ is defined to be the integer coefficient of $n^{-s}$ obtained by multiplying out the Euler product for $L(E,s)$.
\end{itemize}
\end{definition}
Again, we will often write $\Les$ or just $L(s)$ to simplify notation. \\

\begin{corollary} \mbox{}
\begin{itemize}
\item Hasse's Theorem implies that the Euler product and Dirichlet series for $\Les$ converge absolutely for $\Re(s) > \frac{3}{2}$.
\item Sato-Tate implies that the Euler product and Dirichlet series for $\Les$ converge conditionally for $\Re(s) > \frac{1}{2}$.
\end{itemize}
\end{corollary}

In this work we more often use the completed $L$-function attached to $E$:
\begin{definition}
The {\it completed $L$-function} attached to $E$ is given by
\begin{equation}
\Lambda_E(s) = (N_E)^{\frac{s}{2}}(2\pi)^{-s}\Gamma(s)\Les,
\end{equation}
where $N_E$ is the conductor of $E$ and $\Gamma(s)$ the usual Gamma function on $\CC$.
\end{definition}

Thanks to the modularity theorem, we may in fact analytically continue $\Les$ and $\Lams$ to be entire functions defined on all of $\CC$.
\begin{theorem}[Breuille, Conrad, Diamond, Taylor, Wiles et al., 1995,1999,2001] \mbox{}\\
There exists an integral newform $f = \sum_n a_n q^n$ of of weight $k=2$ and level $N_E$ such that $\Les = L_f(s)$.
\end{theorem}

The modularity theorem above is essentially the converse of the theorem by Shimura in the 1960s: if $f$ is a weight $2$ newform of level $N_E$ with rational Fourier coefficients, then there exists some elliptic curve $E/\Q$ of conductor $N_E$ such that $L_f(s) = L_E(s)$. Hence any theorem about elliptic curve $L$-functions is thus really a theorem about $L$-functions of weight 2 newforms in disguise. \\

\begin{corollary} \mbox{}
\begin{itemize}
\item $\Lams$ extends to an entire function on $\CC$. Specifically, $\Lams$ obeys the {\it functional equation}
\begin{equation}
\Lams = w_E \Lambda_E(2-s),
\end{equation}
where $w_E \in \set{-1, 1}$ is the action of the Atkin-Lehner involution on the newform attached to $E$.
\item $\Les$ extends to an entire function on $\CC$ via the definition of $\Lams$ and the functional equation above.
\end{itemize}
\end{corollary}

We reproduce the analytic continuation for $\Lams$ explicitly below. Define the auxiliary function $\lambda_E(s)$ by
\begin{equation}\label{eqn:Lams_analytic_continuation}
\lambda_E(s) = \left(\frac{\sqrt{N_E}}{2\pi}\right)^{s} \sum_{n=1}^\infty a_n n^{-s}\Gamma \left(s,\frac{2\pi n}{\sqrt{N_E}}\right),
\end{equation}
where all the quantities are as defined previously, and $\Gamma(s,x)$ is the upper incomplete Gamma function on $\CC\cross \RR_{>0}$. The sum converges absolutely for any $s$, so $\lambda_E(s)$ is entire. Then
\begin{equation}
\Lambda_E(s) = \lambda_E(s) + w_E \lambda_E(2-s).
\end{equation}
Knapp goes through the proof of this formula in \cite[pp. 270-271]{Kna-1992}. \\

\begin{definition}
$E$ is said to have {\it even parity} if $w_E = 1$, and {\it odd parity} if $w_E = -1$.
\end{definition}

The functional equation for $\Lams$ shows that it is either symmetric or antisymmetric about the line $\Re(s) = 1$; moreover, since all the constituent parts for $\Lams$ are defined over the reals, $\Lams$ is also conjugate symmetric about the real axis. It follows that $\Lams$ is highly symmetric about the point $s=1$. This is formalized in the following statement:
\begin{proposition}
As a function of $s$, $\Lambda_E(1+s)$ is even if $E$ has even parity, and odd if $E$ has odd parity.
\end{proposition}
This follows immediately from the functional equation. \\

\begin{definition} For elliptic curve $L$-functions:
\begin{itemize}
\item The point $s=1$ is called the {\it central point} or the {\it critical point}.
\item The vertical line of symmetry $\Re(s)=1$ is called the {\it critical line}.
\item The vertical strip $0 \le \Re(s) \le 2$ is call the {\it critical strip}.
\end{itemize}
\end{definition}

There is an oft-quoted anecdote that the way to differentiate analytic number theorists from algebraic number theorists is that for elliptic curve $L$-functions the former normalize so that the critical line lies at $\Re(s) = \frac{1}{2}$ (as is the case with $\zeta(s)$), while the latter keep the critical line at $\Re(s)=1$. In this thesis we work mostly with $L_E(1+s)$ and $\Lambda_E(1+s)$ which shifts the critical line to the imaginary axis; a move which is bound to antagonize both parties equally! \\

A standard result with $L$-functions of Hecke eigenforms (of elliptic curve $L$-functions are a subset) is that ``all the interesting stuff happens inside the critical strip'':
\begin{proposition}
For any $E/\QQ$,
\begin{equation}
\Lambda_E(1+s) \ne 0 \;\;\mbox{when}\;\; |\Re(s)| > \frac{1}{2}.
\end{equation}
\end{proposition}
This can be proven by showing that the logarithmic derivative of $\Lambda_E(1+s)$ converges absolutely for $\Re(s) > \frac{1}{2}$; see the corollary to Proposition \ref{lem:ldLe_bound} for a proof. The statement can with a bit more work be strengthened to asserting that all zeros are {\it strictly} inside the critical strip). In fact, the Generalized Riemann Hypothesis asserts that
\begin{equation}
\Lambda_E(1+s) \ne 0  \;\;\mbox{when}\;\; \Re(s) \neq 0.
\end{equation}
From the functional equation we get that $\Les$ has simple zeros at the nonpositive integers; these are denoted the {\it trivial} zeros of $\Les$. Zeros inside the critical strip are called {\it nontrivial}. The Generalized Riemann Hypothesis (formally stated in Section \ref{sec:conjectures}) asserts that all nontrivial zeros of $\Les$ lie on the critical line $\Re(s)=1$.

If $\Les$ has a zero at the central point, it may or may not have multiplicity greater than 1.
\begin{definition}
Let $E$ be an elliptic curve over $\Q$ and let $\Les$ be its $L$-series. The {\it analytic rank} of $E$, denoted $r_{an}(E)$ or just $r_{an}$ is the order of vanishing of $L_E(s)$ at the central point $s=1$. That is, if the Taylor series of $\Les$ about $s=1$ is
\begin{equation}
L_E(1+s) = a_0 + a_1 s + a_2 s^2 + \cdots,
\end{equation}
then $a_n = 0$ for $0 \le n < r_{an}$ and $a_{r_{an}} \ne 0$.
\end{definition}
We will work a lot with the leading coefficient of the $L$-series at the central point, so it's worth giving it a name. To this end:
\begin{definition} \mbox{}
\begin{itemize}
\item Let $C\pr_E$ (or just $C\pr$ when $E$ is fixed) be the leading coefficient of $L_E(s)$ at the central point (the constant $a_{r_{an}}$ in the definition above).
\item Let $C_E$ (or just $C$ when $E$ is fixed) be the leading coefficient of $\Lams$ at the central point.
\end{itemize}
\end{definition}
Observe that $C\pr_E = \frac{2\pi}{\sqrt{N_E}}\cdot C_E$. We will most often work with the latter, hence the notation. \\

We may use Equation \ref{eqn:Lams_analytic_continuation} to produce formulae for the value of $\Lams$ and its higher derivatives at the central point:
\begin{proposition} \mbox{}
\begin{enumerate}
\item \begin{equation}
\Lambda_E(1) = \begin{cases} \frac{\sqrt{N_E}}{\pi} \sum_{n=1}^{\infty}\frac{a_n}{n} e^{-\frac{2\pi}{\sqrt{N_E}}\cdot n}, & w_E = 1 \\ 0, & w_E = -1 \end{cases}.
\end{equation}
\item When $m$ has the same parity as $E$, the $m$th derivative of $\Lams$ at the central point is given by
\begin{equation}\label{eqn:central_derivatives}
\Lambda_E^{(m)}(1) = 2 \sum_{n=1}^\infty a_n \int_{1}^{\infty} \left(\log \frac{t}{\sqrt{N_E}}\right)^m e^{-\frac{2\pi n}{\sqrt{N_E}}\cdot t} \; dt.
\end{equation}
When $m$ is opposite in parity to $E$, then $\Lambda_E^{(m)}(1) = 0$.
\end{enumerate}
\end{proposition}

\begin{proof}
Observe that the series in equation \ref{eqn:Lams_analytic_continuation} converges uniformly over the interval of integration; we may therefore swap the integral and summation signs. After a change of variables we get
\begin{equation*}
\lambda_E(1+s) = N_E^{\frac{1+s}{2}}  \int_{\frac{1}{\sqrt{N_E}}}^{\infty} x^s f_E(it) \; dt  = N_E^{\frac{1+s}{2}}  \sum_{n=1}^\infty a_n \int_{\frac{1}{\sqrt{N_E}}}^{\infty} t^s e^{-2\pi nt} \; dt,
\end{equation*}
where $f_E$ is the cusp form attached to $E$. Both $t^s e^{-2\pi nt}$ and its derivative w.r.t. $s$ are continuous over the integration interval for any $n$, so by the Leibniz integration rule we may differentiate under the integral sign and evaluate at $s=1$ to get
\begin{equation}\label{eqn:lambda_derivs}
\lambda_E^{(m)}(1) = \sqrt{N_E}\cdot \sum_{n=1}^\infty a_n \int_{\frac{1}{\sqrt{N_E}}}^{\infty} (\log t)^m e^{-2\pi n t} \; dt.
\end{equation}
Equation \ref{eqn:central_derivatives} follows by substituting $t \mapsto \sqrt{N_E} \cdot t$. For $m=0$ the integrals may be evaluated directly: $\int_{1}^{\infty} e^{-\frac{2\pi n t}{\sqrt{N_E}}} \; dt = \frac{\sqrt{N_E}}{2\pi n} e^{-\frac{2\pi n}{\sqrt{N_E}}}$.
\end{proof}

Equation \ref{eqn:central_derivatives} allows us to establish bounds on the coefficients of the Taylor expansion of $\Lams$ about the central point. For this we will need the following technical lemma:
\begin{lemma}\label{lem:central_deriv_int_bounds}
Let $N_E,n \in \ZZ_{>0}$, and suppose $m$ is a positive integer such that $m < \frac{1}{2}\log N_E$. Then
\begin{equation}
\left| \int_{\frac{1}{\sqrt{N_E}}}^{\infty} (\log t)^{m} e^{-2\pi n t} \; dt \right| < \frac{\left(\frac{1}{2} \log N_E\right)^{m}}{2\pi n}\left[ e^{-\frac{2\pi n}{\sqrt{N_E}}} + \frac{e^{-2\pi n\sqrt{N_E}}}{2\pi n \sqrt N_E} \right].
\end{equation}
\end{lemma}
\begin{proof}
We split the integral in two, dealing with the intervals $\frac{1}{\sqrt{N_E}}$ to $\sqrt{N_E}$ and $\sqrt{N_E}$ to $\infty$ separately. Now $(\log t)^{m}$ is at most $(\frac{1}{2}\log N_E)^m$ in magnitude on $[\frac{1}{\sqrt{N_E}},\sqrt{N_E}]$, so
\begin{equation*}
\left| \int_{\frac{1}{\sqrt{N_E}}}^{\sqrt{N_E}} (\log t)^{m} e^{-2\pi n t} \; dt \right| < \left(\frac{1}{2} \log N_E\right)^m \int_{\frac{1}{\sqrt{N_E}}}^{\sqrt{N_E}} e^{-2\pi n t} \; dt < \frac{\left(\frac{1}{2} \log N_E\right)^{m}}{2\pi n}\left(e^{-\frac{2\pi n}{\sqrt{N_E}}} - e^{-2\pi n\sqrt{N_E}}\right).
\end{equation*}
For the integral on $[\sqrt{N_E},\infty)$, we use integration by parts to get
\begin{equation*}
\int_{\sqrt{N_E}}^{\infty} \left(\log t \right)^{m} e^{-2\pi n t} \; dt = \frac{\left(\frac{1}{2} \log N_E\right)^{m}}{2\pi n}\cdot e^{-2\pi n\sqrt{N_E}} + \frac{m}{2\pi n} \int_{\sqrt{N_E}}^{\infty} \frac{\left(\log t \right)^{m-1}}{t} e^{-2\pi n t} \; dt.
\end{equation*}
If $m < \frac{1}{2}\log N_E$, then $\frac{\left(\log t \right)^{m-1}}{t}$ is decreasing for $t > \sqrt{N_E}$, so we have
\begin{equation*}
\frac{m}{2\pi n} \int_{\sqrt{N_E}}^{\infty} \frac{\left(\log t \right)^{m-1}}{t} e^{-2\pi n t} \; dt < \frac{m\left(\frac{1}{2} \log N_E\right)^{m-1}}{2\pi n\sqrt{N_E}} \int_{\sqrt{N_E}}^{\infty} e^{-2\pi n t} \; dt < \frac{\left(\frac{1}{2} \log N_E\right)^{m}}{(2\pi n)^2 \sqrt{N_E}} \cdot e^{-2\pi n \sqrt{N_E}}.
\end{equation*}
Add up all the values and you get the established result.
\end{proof}

With the above lemma in hand, we establish an upper bound on the magnitude of the $m$th Taylor coefficient of $\Lams$ at the central point.
\begin{proposition}\label{prop:central_deriv_bounds}
Let $E$ have conductor $N_E$ and completed $L$-function $\Lams$. Then so long as $m<\frac{1}{2}\log N_E$, the $m$th derivative of $\Lams$ at the central point is bounded explicitly in terms of $N_E$ and $m$ by
\begin{equation}
\left| \Lambda_E^{(m)}(1)\right| < \frac{(\frac{1}{2}\log N_E)^m}{2\pi^2}\left(N_E + \frac{1}{e^{2\pi\sqrt{N_E}}-1} \right).
\end{equation}
That is, for fixed $m$ the $m$th Taylor coefficient of $\Lams$ is $O\left( N_E(\frac{1}{2}\log N_E)^m\right)$; the second term inside the final parentheses is negligible for $N_E\gg1$.
\end{proposition}

\begin{proof}
From Lemma \ref{lem:central_deriv_int_bounds} and Equation \ref{eqn:lambda_derivs} we have that
\begin{equation*}
\left| \Lambda_E^{(m)}(1)\right| < 2 \sqrt{N_E} \sum_{n=1}^{\infty} |a_n| \cdot \left[\frac{\left(\frac{1}{2} \log N_E\right)^{m}}{2\pi n}\left( e^{-\frac{2\pi n}{\sqrt{N_E}}} + \frac{e^{-2\pi n\sqrt{N_E}}}{2\pi n \sqrt N_E} \right)\right].
\end{equation*}
Using the bound $|a_n(E)| \le n$ for any $E$, we get
\begin{equation*}
\left| \Lambda_E^{(m)}(1)\right| < \frac{ \sqrt{N_E}\left(\frac{1}{2} \log N_E\right)^{m}}{\pi} \sum_{n=1}^{\infty} e^{-\frac{2\pi n}{\sqrt{N_E}}} + \frac{\left(\frac{1}{2} \log N_E\right)^{m}}{2\pi^2} \sum_{n=1}^{\infty} \frac{e^{-2\pi n\sqrt{N_E}}}{n}.
\end{equation*}
Now
\begin{equation*}
\sum_{n=1}^{\infty} e^{-\frac{2\pi n}{\sqrt{N_E}}} = \frac{1}{e^{\frac{2\pi n}{\sqrt{N_E}}}-1}< \frac{\sqrt{N_E}}{2\pi},
\end{equation*}
while $\sum_{n=1}^{\infty} \frac{e^{-2\pi n\sqrt{N_E}}}{n} \le \sum_{n=1}^{\infty} e^{-2\pi n\sqrt{N_E}} = \frac{1}{e^{2\pi\sqrt{N_E}}-1}$.
\end{proof}

Note that for fixed $N_E$, if we allow $m \to \infty$, we actually have that the $m$th derivative can grow like $O\left(\frac{m!!}{(2\pi e)^{m/2}}\right)$, where $m!! = m(m-2)\cdots$ is the double factorial on $m$ i.e. faster than exponentially in $m$. However, this behavior only starts to show when $m\gg\log N_E$ -- hence our restriction on the magnitude of $m$. This will in practice never be an issue: we are primarily interested in the central derivatives in order to establish results about the analytic rank of $E$. Since maximum analytic rank grows more slowly than $\log N_E$ (c.f. Corollary \ref{cor:logderiv_rank_bound}), we will never need to consider $\Lambda_E^{(m)}(1)$ for $m> \frac{1}{2}\log N_E$. \\

Crucial to this thesis, $L_E(s)$ and its derivatives can be provably computed to a given precision in time that scales with the square root of the conductor of $E$:
\begin{proposition}\label{prop:L_E_time_complexity}
When $m<\frac{1}{2} \log N_E$, the $m$th derivative of $L_E(s)$ at the central point can be provably computed to $k$ bits precision in $\softO(k\cdot \sqrt{N_E})$ time, where $N_E$ is the conductor of $E$.
\end{proposition}
This is proven in full in the PhD thesis of Robert Bradshaw \cite{Bra-2010}. The basic argument is as follows:
\begin{enumerate}
\item Since the two differ by an exponential and a Gamma factor, computing $L_E^{(m)}(1)$ takes the same order of magnitude time as computing $\Lambda_E^{(m)}(1)$. This may be achieved, for example, by the formula given in Equation \ref{eqn:central_derivatives};
\item The integral $\int_{1}^{\infty} \left(\log \frac{t}{\sqrt{N_E}}\right)^m e^{-\frac{2\pi n}{\sqrt{N_E}}\cdot t} \; dx$ can be computed to $k$ bits precision in time that scales proportional to $k$, is independent of $n$ and subpolynomial in $N_E$;
\item The number of terms needed in the sum to achieve $k$ bits precision is $O\left( \log(N_E)^m\sqrt{N_E}\right)$;
\item Computing $a_n$ can be done in time polynomial in $\log n$;
\item Combining the above, computation time is dominated by evaluating $O( \log(N_E)^m\sqrt{N_E})$ integrals and $a_n$ values. That is, the sum can be evaluated to $k$ bits precision in time scaling with $k \sqrt{N_E}$ times some power of $\log N_E$.
\end{enumerate}
We will use the result of Proposition \ref{prop:L_E_time_complexity} directly in the proof of Theorem \ref{thm:main_theorem}. In fact, the $\softO(\sqrt{N_E})$ time needed to evaluate central derivatives of $\Les$ is the computational bottleneck in algorithm \ref{algo:compute_rank}; all other steps scale in time subpolynomial in $N_E$.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The Three Big Conjectures}\label{sec:conjectures}

The main results in this thesis are contingent on the Birch and Swinnerton-Dyer conjecture, The Generalized Riemann Hypothesis and the ABC conjecture. We reproduce the three conjectures in full below; citations for the papers in which they first appeared or are fully formulated are listed at the top of each conjecture. \\

The Birch and Swinnerton-Dyer conjecture (BSD) is needed to establish a way to compute and hence bound the magnitude of the leading coefficient of $\Les$ at the central point.
\begin{conjecture}[Birch, Swinnerton-Dyer]\label{conj:BSD} \cite{BSD-1965}
\mbox{}
\begin{enumerate}
\item $r_{an} = r$; that is, the analytic rank of $E$ is equal to its algebraic rank.
\item The leading coefficient at the central point in $L_E(s)$ is given by
\begin{equation}\label{eqn:BSD_formula}
C\pr_E = \left(\frac{\Omega_E\cdot\Reg_E\cdot\#\Sha(E/\Q)\cdot\prod_p c_p}{(\#E_{\text{Tor}}(\Q))^2}\right),\end{equation}
where
\begin{itemize}
\item $r$ is the algebraic rank of $E(\Q)$,
\item $\Omega_E$ is the real period of (an optimal model of) $E$,
\item $\Reg_E$ is the regulator of $E$,
\item $\#\Sha(E/\Q)$ is the order of the Shafarevich-Tate group attached to $E/\Q$,
\item $\prod_p c_p$ is the product of the Tamagawa numbers of $E$, and
\item $\#E_{\text{Tor}}(\Q)$ is the number of rational torsion points on $E$.
\end{itemize}
\end{enumerate}
\end{conjecture}

For an excellent description of the conjecture and a breakdown of the arithmetic invariants mentioned above, see Andrew Wiles' official description of the BSD Conjecture on the Clay Math website \cite{Wil-BSD}. \\

The Generalized Riemann Hypothesis (GRH), as the name suggests, generalizes the famous conjecture first posed by Bernhard Riemann in 1859 \cite{Rie-1859}. The (standard) Riemann Hypothesis asserts that all nontrivial zeros of the Riemann zeta function $\zeta(s)$ occur on the vertical line $\Re(s)=\frac{1}{2}$. It is hard to track down where the Generalized Riemann Hypothesis was first formulated in its full generality (Conrey gives a good exposition in \cite{Con-2003}), but it asserts that for a large class of suitably defined $L$-functions, the nontrivial nonreal zeros will occur on a single vertical line in the complex plane (where the exact lateral placement of said line depends on the class of $L$-function being considered). We use GRH as it applies to elliptic curve $L$-functions:
\begin{conjecture}[Generalized Riemann Hypothesis for Elliptic Curves, version 1] \cite{Rie-1859} \cite{Con-2003}
\label{conj:GRH1}
\mbox{}
Let $E$ be an elliptic curve over $\Q$, and let $\Les$ be its $L$-series. If $\rho$ is a nontrivial zero of $\Les$ with nonzero imaginary part, then $\Re(\rho) = 1$.
\end{conjecture}
That is, $\Les$ is never zero outside of the {\it critical line} $\Re(s)=1$ and the nonpositive integers. There are numerous equivalent formulations of GRH; we will most often use the following pertaining to the shifted completed $L$-function:
\begin{conjecture}[Generalized Riemann Hypothesis for Elliptic Curves, version 2] \cite{Rie-1859} \cite{Con-2003}
\label{conj:GRH2}
 \mbox{}
 Let $E$ be an elliptic curve over $\Q$, and let $\Lams$ be the completed $L$-function attached to $E$. Then
\begin{enumerate}
\item $\Lambda_E(1+s) = 0 \Longrightarrow \Re(s) = 0$.
\end{enumerate}
\end{conjecture}

Finally, we will need strong form of the ABC conjecture of Masser and Oesterl\'{e} in order to establish lower bounds on the regulator and real period of $E$.
\begin{conjecture}[Masser-Oesterl\'{e}] \cite{Mas-1985} \cite{Oes-1988}
\label{conj:ABC} \\
Let $(a,b,c)$ be a triple of coprime positive integers such that $a+b = c$, and let $\rad(abc) = \prod_{p | abc} p$ be the product of all primes dividing $a$, $b$ and $c$. Then for any $\epsilon > 0$ there is a constant $K_{\epsilon}$ such that
\begin{equation}
c < K_{\epsilon}\rad(abc)^{1+\epsilon}.
\end{equation}
\end{conjecture}

The ABC conjecture is famous for the large number of other results that it implies. Of these, we will need two that relate to elliptic curves. It is a relatively straightforward exercise to show that the conductor of an elliptic curve divides its minimal discriminant. Szpiro's conjecture, formulated in the 1980s, asserts that the latter cannot be too big in terms of the former:
\begin{conjecture}[Szpiro] \cite{szp-1987}
\label{conj:Szpiro} \\
Let $E$ be an elliptic curve over $\QQ$ with conductor $N_E$ and minimal discriminant $D_E$. Then for any $\epsilon > 0$ there is a constant $K_{\epsilon}$ such that
\begin{equation}
|D_E| < K_{\epsilon}\cdot (N_E)^{6+\epsilon}.
\end{equation}
\end{conjecture}

We will also invoke a equivalent version of the above conjecture:
\begin{conjecture}[Modified Szpiro]\label{conj:modified_szpiro}
Let $c_4$ and $c_6$ be the $c$-invariants of a minimal model of $E/\QQ$, as defined in Section \ref{subsec:notation}. Then for any $\epsilon>0$ there is a constant $K_{\epsilon}$ independent of $E$ such that
\begin{equation}
\max\set{|c_4|^3,|c_6|^2} \le K_{\epsilon}\cdot (N_E)^{6+\epsilon}.
\end{equation}
\end{conjecture}

Lang's conjecture posits that the height of a point on a rational curve cannot be too small in terms of the discriminant:
\begin{conjecture}[Lang] \cite[pp. 73-74]{Lang-1997}
\label{conj:Lang} \\
There is a positive constant $M_0$ such that for any elliptic curve $E/\QQ$ with minimal discriminant $D_E$, the N\'{e}ron-Tate canonical height of any nontorsion point $P\in E(\QQ)$ obeys
\begin{equation}
\hat{h}(P) \ge M_0\log |D_E|.
\end{equation}
\end{conjecture}

%\begin{conjecture}[Hall]\label{conj:Hall}
%For any $\epsilon > 0$ there is a constant $K_{\epsilon}$ such that for any $x,y \in \ZZ$ such that $x^3-y^2 \ne 0$, we have
%\begin{equation}
%|x^3-y^2| \ge K_{\epsilon}\cdot |x|^{\frac{1}{2}-\epsilon}.
%\end{equation}
%That is, if $x^3-y^2 \ne 0$, then it cannot be much smaller in magnitude than about $\sqrt{|x|}$.
%\end{conjecture}


\section{Analytic and Meromorphic Functions}
We have seen previously that there can be some multivalued functions in $\mathbb C$ carries important meanings.
Most important examples being $\log$ and $\sqrt[m]{\cdot}$.
What do we really mean by these multivalued functions?
When we evaluate them, we always have to choose a branch cut, but it does not capture the relationship between different branch cuts.
We shall try to find a way to characterise these functions in a proper way that realises their properties as they deserve.
\subsection{Analytic Functions and their Zeros}
\begin{definition}
    A domain is an open, connected subset $D\subset\mathbb C$.
\end{definition}
\begin{example}
    Open disks, annuli, punctured disks are domains.
\end{example}
\begin{definition}
    Let $D\subset\mathbb C$ be a domain.
    A function $f:D\to\mathbb C$ is holomorphic or analytic if either it is $\mathbb C$-differentiable everywhere on $D$ or it has a local Taylor series around every point in $D$.
\end{definition}
We know the two criteria above are equivalent from discussions back in IB Complex Analysis.
\begin{proposition}
    Let $f:D\to\mathbb C$ be an analytic functions on a domain.
    If $f(z_0)=0$, then either $f$ is identically zero or nowhere zero on a punctured disk centering at $z_0$.
\end{proposition}
\begin{proof}
    Obvious and already discussed in Complex Analysis, but let's do it again.
    Consider the Taylor series of $f$ in a neighbourhood $U$ of $z_0$.
    So for $z\in U$ we have
    $$f(z)=\sum_{n=0}^\infty a_n(z-z_0)^n$$
    If $f$ is not identically zero, then we can choose minimal $m$ such that $a_m\neq 0$, so $f(z)=(z-z_0)^mg(z)$ where
    $$g(z)=\sum_{n=0}^\infty a_{m+n}(z-z_0)^n\neq 0$$
    is nonzero at $z_0$, hence is nowhere zero in a punctured disk around $z_0$ in $U\subset D$ by continuity.
    $f$ is then nowhere zero in the same punctured disk.
\end{proof}
\begin{corollary}[Identity Principle]
    Let $f,g$ be analytic functions defined on a domain $D\subset\mathbb C$.
    If the subspace $\{z\in D|f(z)=g(z)\}$ is not discrete, then $f\equiv g$ on $D$.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
\subsection{Meromorphic Functions and Singularities}
\begin{definition}
    A function $f$ defined on a punctured disk around $z_0$ is said to have a isolated singularity at $z_0$.
\end{definition}
\begin{proposition}
    If an analysic function $f$ has an isolated singularity at $z_0$, then $f$ has a Laurent series
    $$f(z)=\sum_{n=-\infty}^\infty a_n(z-z_0)^n$$
    on a punctured disk around $z_0$.
\end{proposition}
\begin{proof}
    Complex Analysis.
\end{proof}
\begin{definition}
    If $a_n=0$ for $n<0$, then $z_0$ is said to be a removable singularity of $f$.\\
    If $a_n=0$ for $n<-m<0$ and $a_{-m}\neq 0$, then we say $z_0$ is a pole of order $m$.\\
    Otherwise, we say $z_0$ is an essential singularity.
\end{definition}
\begin{theorem}
    If $f$ is bounded near $z_0$, then $z_0$ has to be a removable singularity.
\end{theorem}
\begin{proof}
    Complex Analysis.
\end{proof}
\begin{theorem}[Casorati-Weierstrass]
    $z_0$ is an essential singularity iff $f(U)$ is dense in $\mathbb C$ for any punctured neighbourhood of $z_0$.
\end{theorem}
\begin{proof}
    Complex Analysis.
\end{proof}
\begin{definition}
    If a holomorphic function $f:D\setminus A\to\mathbb C$ for some domain $D$ and discrete $A$ has poles at the points of $A$, then $f$ is said to be meromorphic.
\end{definition}
\begin{example}
    The function $f(z)=1/(e^{1/z}-1)$ is meromorphic where one takes $D$ to be the open upper half-plane and $A=\{1/(2\pi in):n\in\mathbb N\}$.
    In particular, the poles are all simple (i.e. of order $1$).\\
    Note that the function can be extended to the whole of $\mathbb C$ except at the discrete set $\{1/(2\pi in):n\in\mathbb Z\setminus\{0\}\}\cup\{0\}$, which contains $0$ as an essential singularity and others as simple poles.
\end{example}
\subsection{Analytic Continuation}
\begin{definition}
    A function element $F=(f,U)$ on a domain $D$ consists of a subdomain $U\subset D$ and an analytic function $f:U\to\mathbb C$.
\end{definition}
\begin{lemma}[Direct Analytic Continuation]
    Let $(f,U),(g,V)$ are function elements such that $U\cap V\neq\varnothing$ and $f=g$ on $U\cap V$, then $f$ determines $g$.
\end{lemma}
\begin{proof}
    Identity Principle.
\end{proof}
We write $(f,U)\sim (g,V)$ for direct analytic continuation.
\begin{definition}
    Analytic continuation is a sequence of iteration of direct analytic continuations that get from one function element to another.
    We write $(f,U)\approx (g,V)$ in this case.
\end{definition}
\begin{remark}
    $\approx$ is an equivalence relation.
\end{remark}
\begin{definition}
    A $\approx$-equivalence class $\mathcal F$ of function elements on a domain $D$ is called a complete analytic function on $D$.
\end{definition}
\subsection{The Complex Logarithm}
Let $\mathbb C_\star=\mathbb C\setminus\{0\}$.
We want to invert the exponential function $\exp:\mathbb C\to\mathbb C_\star$, but $\exp$ is not injective on $\mathbb C$.
We used to see $\log$ as a multivalued function to deal with this problem, but in fact, we can see it as a complete analytic function on $\mathbb C_\star$.\\
Indeed, for $(\alpha,\beta)\subset\mathbb R$ with $|\alpha-\beta|<2\pi$, we can define
$$U_{(\alpha,\beta)}=\{re^{i\theta}|r>0,\alpha<\theta<\beta\},f_{(\alpha,\beta)}(z)=\log r+i\theta,r=|z|,\theta\in (\alpha,\beta)$$
Then $F_{(\alpha,\beta)}=(f_{(\alpha,\beta)},U_{(\alpha,\beta)})$ is a collection of function elements.
Let $I(n)=((n-1)\pi/2,(n+1)\pi/2)$ for $n\in\mathbb Z$.
\begin{proposition}
    $F_{I(n)}\sim F_{I(m)}$ iff $|m-n|\le 1$.
\end{proposition}
\begin{proof}
    Just do a case analysis based on $m-n\bmod 4$.
\end{proof}
\begin{corollary}
    $F_{I(m)}\approx F_{I(n)}$ for any $m,n\in\mathbb Z$.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
So this characterises a complete analytic function that is the complex logarithm.
\begin{remark}
    We know that $f_{I(0)}(1)=0$ but $f_{I(4)}(1)=2\pi i$, so analytic continuation in this way is not unique.
    But we have ``pasted'' them together to make it a complete analytic function.
\end{remark}
\begin{definition}
    Let $\gamma:[0,1]\to D$ be a path.
    We say $(f,U)\approx_\gamma (g,V)$ if there is some $0=t_0<t_1<\cdots<t_n=1$ and function elements $(f=f_0,U=U_0),(f_1,U_1),\ldots,(g=f_{n+1},V=U_{n+1})$ such that $(f_i,U_i)\sim (f_{i+1},U_{i+1})$ for $i=0,\ldots,n$ and $\gamma(t_i)\in U_i\cap U_{i+1}$.
    This is called analytic continuation along a path.
\end{definition}
Analytic continuation along a path has the desired uniqueness property.
This fact is known as the Classical Monodromy Theorem.


\section{Background on SMT-based $k$-induction}~\label{sec:background}
The focus of our  investigation has  been on applying model checking to  prove
invariant properties of our monitors.   We
employ a  technique known as $k$-induction~\cite{Sheeran00,EenS03} for verifying inductive
properties of infinite state systems.   $k$-induction  has the
advantage that it is well suited  to  \textsc{smt}
based bounded model checking. This section profiles the
basic concepts of the  $k$-induction proof technique needed in the
remainder of the paper. In practice, we use tools that implement  enhancements of the basic procedure such as path compression~\cite{dMRS03} that help the process scale, but are beyond the focus of the paper.

Consider  a state transition system  $(S,I,T),$
where $S$ is a set of states, $I \subseteq S$ is the set of initial
states and $T \subseteq S \times S $ is a transition relation over
$S.$ To show $P$ holds in the transition system one must show that (1)
the base case holds---that $P$ holds in all states reachable
from an initial state in $k$ steps, and (2) the induction step holds---that if $P$ holds in states $s_0,\ldots,s_{k-1}$ then it holds in
state $s_k.$ The $k$-induction principle is formally expressed in the
following two entailments:
\begin{eqnarray*}
I(s_0) \tland T(s_0,s_1) \tland \cdots \tland T(s_{k-1},s_k) &\models&
P(s_k) \\
P(s_0) \tland \cdots \tland P(s_{k-1}) \tland T(s_0,s_1) \tland \cdots \tland T(s_{k-1},s_k) &\models&
P(s_k)
\end{eqnarray*}
If one cannot show the property to be true, the
property is strengthened by either extending the formula or
progressively increasing the length of the reachable states
considered.

Property $P$ said to be a $k$-inductive property with respect to
$(S,I,T)$ if there exists some $k \in \mathbb{N}^{0<}$ such that $P$
satisfies the $k$-induction principle. As $k$ increases, weaker
invariants may be proved. If $P$ is a safety property that does not hold, then the first entailment will break for a finite $k$ and a counterexample will be provided. The trick is to find an invariant that is
tractable by the \textsc{smt} solver yet weak enough to satisfy the desired
property.