% Section 5: Numerical Results

\section{Numerical Results}

\subsection{Datasets}
In orde to compare the performance of different algorithms, we used three types of datasets to test algorithms discussed above, including artificial generated datasets and real images. In short, we have three different types of datasets:
\begin{itemize}
    \item Random Generated Dataset
    \item Ellipse and Caffarelli Dataset [5]
    \item DOTmark Dataset [6]
\end{itemize}

We assume that the cost of transporting a unit mass from $x \in X \subset \mathbb{R}^2$ to $y \in Y \subset \mathbb{R}^2$ is $c_p(x,y) = \|x-y\|^p$ for some $p \geq 1$. The minimum cost for transferring $\mu$ to $\nu$ is then given by

\begin{equation}
    C_p (\mu, \nu) = \min_{x \in \Pi(\mu,\nu)} \|x-y\|^p d\pi(x,y)
\end{equation}

For realistic considerations, we use $\mathbf{L}_2$ distance as our metrics.

Random generated samples consists of uniform samples on $[-1, 1]^2$ of size $n$, weights $\mu$ and $\nu$ are uniformly
sampled on $[-1,1]^2$ and normalized with $\sum_i \mu_i = 1$, $\sum_j \nu_j = 1$.

The ellipse example consists of two uniform samples (source and target data set) of size $n$ from the unit circle
with normal distributed noise added with zero mean and standard deviation $0.1$. The source
data sample is then scaled in the x-Axis by $1.3$ and in the y-Axis by $0.9$, while the target
data set is scaled in the x-Axis by $0.9$ and in the y-Axis by $1.1$.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\linewidth]{img/ellipse}
    \label{fig:ot}
    \caption{Illustration of Optimal Transport on Ellipse dataset with different coarsen level}
  \end{figure}

Caffarelli's example consists of two uniform samples on $[-1, 1]^2$ of size $n$. Any points outside the unit circle are then
discarded. Additionally, the target data sample is split along the x-Axis at $0$ and shifted by
$+2$ and $-2$ for points with positive and negative x-Axis values, respectively.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\linewidth]{img/ellipse_caffa}
    \label{fig:ot}
    \caption{Illustration of Optimal Transport on Ellipse dataset and Caffarelli dataset}
  \end{figure}

The DOT benchmark [6] consists of 10 classes of 10 different images, each of which is available at the 5 different
resolutions from $a \times a  = 32 \times 32$ to $512 \times 512$. In computation, each data point is $m = n = a^2$.  This allows
for a total of 45 computations of Wasserstein distances between two images for any one class at any fixed resolution. for convenience, we used the $32 \times 32$ and $64 \times 64$ smaples for experiments.

Table 1 gives an overview of how the classes were created.
Classes 1-7 are random simulations of scenarios based on various probability distributions.
Images at different resolutions are generated independently from each other but according to the same laws.
Classes 8-10 were obtained by ad-hoc choices of simple geometric shapes, classic test images and images of mitochondria acquired using STED
super-resolution microscopy. For geometric shapes and classic test images
the various resolutions available are coarsenings of a single image. For the microscopy
images different clippings of various sizes have been selected from larger images to obtain
the various resolutions.

\vspace{5ex}
\begin{table}[htbp]
	\caption{DOTmark datasent}
	\centering
	\begin{tabular}[width=1.0\linewidth]{lll}
		\toprule
		\quad  & Dataset & Description\\
    \midrule
            1  & WhiteNoise    & i.i.d. uniformly distributed values in [0, 1] at each pixel\\
            2  & GRFrough      & GRF with $\sigma = 1$, $\nu = 0.25$, $\gamma = 0.5$\\
            3  & GRFmoderate   & GRF with $\sigma = 1$, $\nu = 2.5$, $\gamma = 0.15$\\
            4  & GRFsmooth     & GRF with $\sigma = 1$, $\nu = 2.5$, $\gamma = 0.3$\\
            5  & LogGRF        & exp-function of a GRF with $\sigma = 1$, $\nu = 0.5$, $\gamma = 0.4$\\
            6  & CauchyDensity & Bivariate Cauchy density with random center and a varying scale ellipse\\
            7  & LogitGRF      & Logistic function of a GRF with $\sigma = 4$, $\nu = 4.5$, $\gamma = 0.1$\\
            8  & Shapes        & An ad-hoc choice of simple geometric shapes\\
            9  & ClassicImages & Standard grayscale test images used in image processing\\
            10 & Microscopy    & Clippings from STED microscopy images of mitochondria\\
        \bottomrule
	\end{tabular}
	\label{tab:tabledot}
\end{table}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\linewidth]{img/dot_1to6}
    \label{fig:ot}
    \caption{The images in the classes 1-6 at resolution 128 $\times$ 128}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\linewidth]{img/dot_7to10}
    \label{fig:ot}
    \caption{The images in the classes 7-10 at resolution 128 $\times$ 128}
\end{figure}

\clearpage
\subsection{Numercial Results and Intepretations}

We use three main metrics to evaluate all of these algorithms:

\begin{itemize}
    \item Running time
    \item Function value
    \item Constraints $\mu$ and $\nu$
\end{itemize}

To measure the satisfaction of constraints, $\mathbf{L}_1$ norm of $\mathbf{\mu} - \mathbf{\pi}_{i\cdot}$ and $\mathbf{\nu} - \mathbf{\pi}_{\cdot j}$ was introduced. On the consideration of time consuming, we conduct our experiments with size $m = n = 64, 128, 256, 512, 1024, 2048$.

Obviously, the running time is an important metrics to measure different algorithms under different test data. In addition, object value can also shows performance of different algorithms, though for entropy regularized methods such as sinkhorn algorithm the function value has been modified with regularization term and then can not be use for comparison.
However, the distance between the suboptimal tranport variable $\pi$ and the optimal transport variable $\tilde \pi$ does not reflect the merits of the algorithm, because it depends on the nature of the cost matrix $\mathbf{C}$. Consider the following example:

\begin{equation}
    \begin{array}{cl}
        {\min } & {\mathbf{t} \mathbf{r}\left(\left(\begin{array}{cc}
        {1+\epsilon} & {1 + \frac{\epsilon}{2}}\\
        {1} & {1} \\
        \end{array}\right)\left(\begin{array}{cc}
        {\pi_{11}} & {\pi_{12}} \\
        {\pi_{21}} & {\pi_{22}}
        \end{array}\right)\right.} \\
        {\text { s.t. }} & {\pi_{11}+\pi_{12}=1} \\
        & {\pi_{21}+\pi_{22}=1} \\
        & {\pi_{11}+\pi_{21}=1} \\
        & {\pi_{12}+\pi_{22}=1}
    \end{array}
\end{equation}

Then the difference between the object value of the optimal solution $\tilde \pi_{ij} = \left(\begin{array}{ll}{0} & {1} \\{1} & {0}\end{array}\right)$ and the suboptimal solution $\pi = \left(\begin{array}{ll}{1} & {0} \\{0} & {1}\end{array}\right)$ is less than $\epsilon$, but the distance after the two are transformed into a vector is significantly larger than $\epsilon$.

Tables below show our experiments results, more detailed data can be found in \textbf{/code/results} folder.

\begin{table}[htbp]
	\caption{Results of Raw Mosek solver and Raw Gurobi solver on Random Generated Dataset}
	\centering
    \begin{tabular}{|c|c|llllll|}
    \hline
    $m = n$             &          & Mosek primal & Mosek dual  & Mosek interior & Gurobi primal & Gurobi dual & Gurobi interior\\
    \hline
    \multirow{4}*{64}   &err $\mu$ & 1.73472e-17  & 1.56125e-17 & 6.99572e-11    & 1.56125e-17   & 1.90820e-17 & 1.56125e-17    \\
                        &err $\nu$ & 1.73472e-16  & 1.68268e-16 & 6.05657e-11    & 1.56125e-16   & 1.58727e-16 & 1.54390e-16    \\
                        &fval      & 2.07168e-02  & 2.07168e-02 & 2.07168e-02    & 2.07168e-02   & 2.07168e-02 & 2.07168e-02    \\
                        &time      & 3.12448e-02  & 3.47526e-02 & 3.42710e-02    & 4.78537e-02   & 4.80292e-02 & 7.72452e-02    \\
    \hline
    \multirow{4}*{128}  &err $\mu$ & 7.38910e-06  & 1.04083e-17 & 6.21169e-12    & 3.46945e-18   & 3.46945e-18 & 5.20417e-18    \\
                        &err $\nu$ & 7.38910e-06  & 1.14275e-16 & 6.21459e-12    & 1.18395e-16   & 1.18395e-16 & 1.17934e-16    \\
                        &fval      & 1.85776e-02  & 1.85775e-02 & 1.85775e-02    & 1.85775e-02   & 1.85775e-02 & 1.85775e-02    \\
                        &time      & 8.82461e-02  & 1.05152e-01 & 3.42710e-02    & 2.18717e-01   & 2.67161e-01 & 3.70859e-01    \\
    \hline
    \multirow{4}*{256}  &err $\mu$ & 1.75611e-05  & 3.62395e-17 & 8.84693e-11    & 1.04083e-17   & 1.12757e-17 & 1.12757e-17    \\
                        &err $\nu$ & 1.75611e-05  & 3.43123e-16 & 8.84696e-11    & 3.23526e-16   & 3.23526e-16 & 3.23960e-16    \\
                        &fval      & 9.37655e-03  & 9.37637e-03 & 9.37637e-03    & 9.37637e-03   & 9.37637e-03 & 9.37637e-03    \\
                        &time      & 3.19547e-01  & 4.38276e-01 & 5.48973e-01    & 1.01217e+00   & 1.12126e+00 & 1.07492e+00    \\
    \hline
    \multirow{4}*{512}  &err $\mu$ & 4.29996e-05  & 5.06695e-17 & 1.13676e-10    & 8.48388e-18   & 3.10109e-16 & 1.12757e-17    \\
                        &err $\nu$ & 4.29996e-05  & 3.38688e-16 & 1.13677e-10    & 8.48388e-18   & 1.25767e-17 & 3.23960e-16    \\
                        &fval      & 4.30472e-03  & 4.30454e-03 & 4.30454e-03    & 4.30454e-03   & 4.30454e-03 & 9.37637e-03    \\
                        &time      & 1.42172e+00  & 2.23522e+00 & 2.73145e+00    & 3.47143e+00   & 3.77973e+00 & 1.07492e+00    \\
    \hline
    \multirow{4}*{1024} &err $\mu$ & 1.06018e-04  & 1.47138e-16 & 1.01099e-13    & 1.09504e-17   & 3.64400e-16 & 1.20346e-17    \\
                        &err $\nu$ & 1.06018e-04  & 5.08930e-16 & 1.00871e-13    & 3.64753e-16   & 1.15196e-17 & 3.65837e-16    \\
                        &fval      & 1.94602e-03  & 1.94578e-03 & 1.94578e-03    & 1.94578e-03   & 1.94578e-03 & 1.94578e-03   \\
                        &time      & 6.45907e+00  & 1.37058e+01 & 1.33225e+01    & 1.73826e+01   & 1.31655e+01 & 1.73745e+01    \\
    \hline
    \multirow{4}*{2048} &err $\mu$ & 2.65157e-04  & 1.12464e-16 & 5.64365e-11    & 1.20889e-17   & 9.78113e-16 & 1.13841e-17    \\
                        &err $\nu$ & 2.65157e-04  & 1.08728e-15 & 5.36207e-11    & 9.76758e-16   & 9.18861e-18 & 9.75701e-16    \\
                        &fval      & 1.50347e-03  & 1.50297e-03 & 1.50297e-03    & 1.50297e-03   & 1.50297e-03 & 1.50297e-03   \\
                        &time      & 3.59507e+01  & 1.28866e+02 & 6.74333e+01    & 3.16742e+01   & 5.92715e+01 & 9.67955e+01    \\
    \hline
    \end{tabular}
    \label{tab:table1}
\end{table}

It seems that Gurobi can always reach the real optimal solution, and MOSEK sometimes fails. For MOSEK,
 the simplex method on primal problem is faster than the interior method and dual simplex method. For Gurobi, the simplex method is usually faster than the interior method. As one of the most famous algorithms in 20th century, simplex method was designed for solving linear programming problems. Poor performance of dual simplex may be caused by a large number of constraints. In summary, Gurobi has better performance that MOSEK.

\begin{table}[htbp]
	\caption{Results of ADMM, Sinkhorn and BlockCA algorithm on Random Generated Dataset}
	\centering
    \begin{tabular}{|c|c|lllll|}
    \hline
    $m = n$             &          & Gurobi primal & ADMM primal & ADMM dual   & Sinkhorn    & BlockCA    \\
    \hline
    \multirow{4}*{64}   &err $\mu$ & 1.56125e-17   & 1.40712e-05 & 6.50195e-01 & 1.00180e-16 & 5.41017e-16\\
                        &err $\nu$ & 1.56125e-16   & 1.41582e-05 & 6.27594e-01 & 1.90874e-16 & 3.94243e-16\\
                        &fval      & 2.07168e-02   & 2.07460e-02 & 8.77846e-04 & 2.73941e-02 & 2.73941e-02\\
                        &time      & 4.78537e-02   & 1.22792e+00 & 1.38214e+00 & 8.93044e-03 & 1.44658e-01\\
    \hline
    \multirow{4}*{128}  &err $\mu$ & 3.46945e-18   & 4.72795e-05 & 8.45667e-01 & 1.39456e-16 & 5.54624e-16\\
                        &err $\nu$ & 1.18395e-16   & 4.64422e-05 & 8.56649e-01 & 2.61211e-16 & 5.22084e-16\\
                        &fval      & 1.85775e-02   & 1.85941e-02 & 8.22017e-05 & 2.92237e-02 & 2.92237e-02\\
                        &time      & 2.67161e-01   & 4.53466e+00 & 3.89032e+00 & 6.39679e-02 & 4.98657e-01\\
    \hline
    \multirow{4}*{256}  &err $\mu$ & 1.04083e-17   & 4.96761e-05 & 9.04781e-01 & 3.11044e-16 & 6.12256e-16\\
                        &err $\nu$ & 3.23526e-16   & 5.02865e-05 & 9.00866e-01 & 3.62530e-16 & 5.78530e-16\\
                        &fval      & 9.37637e-03   & 9.38144e-03 & 1.20866e-05 & 2.88136e-02 & 2.88136e-02\\
                        &time      & 1.01217e+00   & 1.81699e+01 & 8.70793e+00 & 8.24754e-02 & 2.26562e+00\\
    \hline
    \multirow{4}*{512}  &err $\mu$ & 8.48388e-18   & 1.31110e-04 & 9.53515e-01 & 3.42523e-16 & 6.55963e-16\\
                        &err $\nu$ & 8.48388e-18   & 1.33937e-04 & 9.53115e-01 & 5.30280e-16 & 7.73707e-16\\
                        &fval      & 4.30454e-03   & 4.30575e-03 & 1.35802e-06 & 2.81078e-02 & 2.81078e-02\\
                        &time      & 3.47143e+00   & 7.43271e+01 & 6.13705e+01 & 2.81078e-01 & 1.08770e+01\\
    \hline
    \multirow{4}*{1024} &err $\mu$ & 1.09504e-17   & 2.73918e-04 & 9.72007e-01 & 3.86662e-16 & 6.89612e-16\\
                        &err $\nu$ & 3.64753e-16   & 2.74108e-04 & 9.72809e-01 & 7.61287e-16 & 1.02598e-15\\
                        &fval      & 1.94578e-03   & 1.94916e-03 & 2.51372e-07 & 2.79386e-03 & 2.79386e-02\\
                        &time      & 1.73826e+01   & 3.67489e+02 & 2.08239e+02 & 1.02975e+00 & 3.62734e+01\\
    \hline
    \multirow{4}*{2048} &err $\mu$ & 1.20889e-17   & 1.01361e-03 & 9.88752e-01 & 9.68066e-16 & 1.09343e-15\\
                        &err $\nu$ & 9.76758e-16   & 1.02573e-03 & 9.89591e-01 & 1.09554e-15 & 1.53059e-15\\
                        &fval      & 1.50297e-03   & 1.49998e-03 & 2.11058e-08 & 2.76252e-03 & 2.76252e-02\\
                        &time      & 3.16742e+01   & 5.71285e+03 & 3.14300e+03 & 8.98797e+00 & 4.12741e+02\\
    \hline
    \end{tabular}
    \label{tab:table1}
\end{table}

Generally speaking, The ADMM method does not have good performance. On the one hand, the error term is large, which is because the equality constraint is difficult to achieve. On the other hand, the required time is much longer than the general algorithm of Mosek and Gurobi. This is because the algorithm will stay at $\pi = \frac{1}{m n}$ for a long time, and it will not start the convergence step until some variables start to increase significantly. what's more, ADMM method is a first order method which means it has slow convergence rate.

As mentioned before, in practice, the Sinkhorn algorithm suffers from numerical overflow when the regularization parameter $\epsilon$ is small compared to the entries of the cost
matrix $\mathbf{C}$. This concern can be alleviated to some extent by carrying out computations
in the log domain. Here Block Coordinate Ascent algorithm was introduced. From the experiments we can see that entropy regularized methods applied with Sinkhorn algorithm and Block Coordinate Ascent algorithm convergence very fast with the slightest violation with all the constraints of $\mu$ and $\nu$ corresponding to err $\mu$ and err $\nu$ shown.

 \begin{table}[htbp]
	\caption{Results of Raw Mosek solver and Raw Gurobi solver on DOTmark Dataset}
	\centering
    \begin{tabular}{|c|c|llllll|}
    \hline
    Class                   &          & Mosek primal & Mosek dual  & Mosek interior & Gurobi primal & Gurobi dual & Gurobi interior\\
    \hline
    \multirow{4}*{WNoise}   &err $\mu$ & 1.04105e-04  & 1.43060e-16 & 2.60955e-11    & 0.00000e+00   & 0.00000e+00 & 0.00000e+00    \\
                            &err $\nu$ & 1.04106e-04  & 6.93859e-10 & 7.19976e-10    & 6.93859e-10   & 6.93859e-10 & 6.93859e-10    \\
                            &fval      & 6.49762e-04  & 6.49817e-04 & 6.49817e-04    & 6.49817e-04   & 6.49817e-04 & 6.49817e-04    \\
                            &time      & 6.37157e+00  & 1.00645e+01 & 1.09153e+01    & 1.96804e+01   & 1.80495e+01 & 2.56732e+01    \\
    \hline
    \multirow{4}*{GRFrough} &err $\mu$ & 9.86331e-05  & 2.14455e-16 & 3.88285e-14    & 0.00000e+00   & 0.00000e+00 & 0.00000e+00    \\
                            &err $\nu$ & 9.86322e-05  & 8.99345e-10 & 8.99386e-10    & 8.99345e-10   & 8.99345e-10 & 8.99345e-10    \\
                            &fval      & 1.24126e-03  & 1.24131e-03 & 1.24131e-03    & 1.24131e-03   & 1.24131e-03 & 1.24131e-03    \\
                            &time      & 6.22112e+00  & 1.60985e+01 & 1.20426e+01    & 1.70455e+01   & 1.44343e+01 & 2.98131e+01    \\
    \hline
    \multirow{4}*{GRFmid}   &err $\mu$ & 9.82285e-05  & 1.81604e-16 & 5.15640e-11    & 0.00000e+00   & 5.89424e-10 & 5.89424e-10    \\
                            &err $\nu$ & 9.82291e-05  & 5.89424e-10 & 7.33106e-10    & 5.89424e-10   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 1.29959e-02  & 1.29962e-02 & 1.29962e-02    & 1.29962e-02   & 1.29962e-02 & 1.29962e-02    \\
                            &time      & 7.08524e+00  & 3.11953e+01 & 1.08397e+01    & 1.62867e+01   & 1.74454e+01 & 2.98980e+01    \\
    \hline
    \multirow{4}*{GRFmid2}  &err $\mu$ & 9.78652e-05  & 1.96051e-16 & 2.27771e-12    & 0.00000e+00   & 4.81123e-10 & 4.81123e-10    \\
                            &err $\nu$ & 9.78657e-05  & 4.81123e-10 & 4.83410e-10    & 4.81123e-10   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 7.01347e-03  & 7.01329e-03 & 7.01329e-03    & 7.01329e-03   & 7.01329e-03 & 7.01329e-03    \\
                            &time      & 8.29048e+00  & 2.58863e+01 & 1.16190e+01    & 1.54850e+01   & 1.46085e+01 & 1.82575e+01    \\
    \hline
    \multirow{4}*{LogGRF}   &err $\mu$ & 9.79684e-05  & 1.74543e-16 & 1.41349e-13    & 0.00000e+00   & 0.00000e+00 & 0.00000e+00    \\
                            &err $\nu$ & 9.79684e-05  & 3.44473e-11 & 3.45826e-11    & 3.44471e-11   & 3.44471e-11 & 3.44471e-11    \\
                            &fval      & 1.14762e-02  & 1.14762e-02 & 1.14762e-02    & 3.44471e-11   & 1.14762e-02 & 1.14762e-02    \\
                            &time      & 6.61064e+00  & 2.25776e+01 & 1.25897e+01    & 1.46135e+01   & 1.81754e+01 & 2.23398e+01    \\
    \hline
    \multirow{4}*{Cauchy}   &err $\mu$ & 8.95380e-05  & 1.69054e-16 & 4.53139e-11    & 0.00000e+00   & 6.45741e-10 & 6.45741e-10    \\
                            &err $\nu$ & 8.95387e-05  & 6.45741e-10 & 5.36207e-11    & 6.45741e-10   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 2.70715e-02  & 2.70716e-02 & 2.70716e-02    & 2.70716e-02   & 2.70716e-02 & 2.70716e-02    \\
                            &time      & 6.27543e+00  & 3.43761e+01 & 1.25938e+01    & 1.41227e+01   & 1.66956e+01 & 4.97750e+01    \\
    \hline
    \multirow{4}*{LogitGRF} &err $\mu$ & 1.04014e-04  & 1.61844e-16 & 6.22315e-10    & 0.00000e+00   & 1.23453e-09 & 1.23453e-09    \\
                            &err $\nu$ & 1.04013e-04  & 1.23453e-09 & 1.83005e-09    & 1.23453e-09   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 1.90306e-02  & 1.90301e-02 & 1.90301e-02    & 1.90301e-02   & 1.90301e-02 & 1.90301e-02    \\
                            &time      & 6.59682e+00  & 2.55476e+01 & 1.09550e+01    & 1.51339e+01   & 1.87915e+01 & 3.24828e+01    \\
    \hline
    \multirow{4}*{Shape}    &err $\mu$ & 3.10808e-05  & 6.51926e-09 & 1.54226e-08    & 0.00000e+00   & 6.51926e-09 & 6.51926e-09    \\
                            &err $\nu$ & 3.10743e-05  & 4.77049e-17 & 1.07581e-08    & 6.51926e-09   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 6.00170e-03  & 6.00148e-03 & 6.00148e-03    & 6.00148e-03   & 6.00148e-03 & 6.00148e-03    \\
                            &time      & 2.45428e+00  & 4.44892e+00 & 2.97079e+00    & 1.17216e+01   & 1.16747e+01 & 1.31421e+01    \\
    \hline
    \multirow{4}*{Classic}  &err $\mu$ & 9.68013e-05  & 1.20292e-16 & 8.39560e-12    & 0.00000e+00   & 2.61934e-10 & 2.61934e-10    \\
                            &err $\nu$ & 9.68016e-05  & 2.61935e-10 & 2.70569e-10    & 2.61934e-10   & 0.00000e+00 & 0.00000e+00    \\
                            &fval      & 2.13840e-03  & 2.13855e-03 & 2.13855e-03    & 2.13855e-03   & 2.13855e-03 & 2.13855e-03    \\
                            &time      & 7.26325e+00  & 1.91323e+01 & 1.30328e+01    & 1.54186e+01   & 1.40239e+01 & 2.08600e+01    \\
    \hline
    \multirow{4}*{Micro}    &err $\mu$ & 9.11961e-05  & 1.09938e-16 & 1.40660e-10    & 0.00000e+00   & 0.00000e+00 & 0.00000e+00   \\
                            &err $\nu$ & 9.11973e-05  & 1.25146e-09 & 1.24353e-09    & 1.25146e-09   & 1.25146e-09 & 1.25146e-09    \\
                            &fval      & 5.60394e-02  & 5.60383e-02 & 5.60383e-02    & 5.60383e-02   & 5.60383e-02 & 5.60383e-02    \\
                            &time      & 5.27145e+00  & 2.11592e+01 & 9.32959e+00    & 1.44868e+01   & 1.58690e+01 & 1.78717e+02    \\
    \hline
    \end{tabular}
    \label{tab:table1}
\end{table}

\begin{table}[htbp]
	\caption{Results of ADMM, Sinkhorn and BlockCA algorithm on DOTmark Dataset}
	\centering
    \begin{tabular}{|c|c|lllll|}
    \hline
    Class                   &          & Gurobi primal & ADMM primal & ADMM dual  & Sinkhorn   & BlockCA    \\
    \hline
    \multirow{4}*{WNoise}   &err $\mu$ & 0.00000e+00   & 1.28341e-04  & 3.33216e-01 & 6.93859e-10 & 1.24606e-07 \\
                            &err $\nu$ & 6.93859e-10   & 1.16042e-04  & 3.33214e-01 & 7.56495e-16 & 1.27072e-07 \\
                            &fval      & 6.49817e-04   & 6.49518e-04  & 0.00000e+00 & 2.76750e-03 & 2.76750e-02 \\
                            &time      & 1.96804e+01   & 2.05072e+02  & 1.76776e+02 & 4.96325e-01 & 4.06536e+01 \\
    \hline
    \multirow{4}*{GRFrough} &err $\mu$ & 0.00000e+00   & 1.36891e-04  & 2.13545e-01 & 8.99345e-10 & 1.23457e-07 \\
                            &err $\nu$ & 8.99345e-10   & 1.35197e-04  & 2.13544e-01 & 7.54279e-16 & 1.22399e-07 \\
                            &fval      & 1.24131e-03   & 1.24030e-03  & 0.00000e+00 & 2.77295e-02 & 2.77295e-01 \\
                            &time      & 1.70455e+01   & 3.08484e+02  & 2.03046e+02 & 5.94349e-01 & 4.29790e+01 \\
    \hline
    \multirow{4}*{GRFmid}   &err $\mu$ & 0.00000e+00   & 1.51246e-04  & 2.28186e-01 & 4.81123e-10 & 1.20229e-07 \\
                            &err $\nu$ & 5.89424e-10   & 1.43893e-04  & 2.28171e-01 & 7.92382e-16 & 1.17900e-07 \\
                            &fval      & 1.29962e-02   & 1.29985e-02  & 0.00000e+00 & 2.67600e-02 & 2.70585e-02 \\
                            &time      & 1.62867e+01   & 3.04346e+02  & 2.04500e+02 & 6.11465e-01 & 3.94406e+01 \\
    \hline
    \multirow{4}*{GRFmid2}  &err $\mu$ & 0.00000e+00   & 1.32336e-04  & 1.90867e-01 & 5.89424e-10 & 1.25930e-07 \\
                            &err $\nu$ & 4.81123e-10   & 1.32346e-04  & 1.90878e-01 & 7.83878e-16 & 1.22960e-07 \\
                            &fval      & 7.01329e-03   & 7.01429e-03  & 0.00000e+00 & 2.70585e-02 & 2.67600e-01 \\
                            &time      & 1.54850e+01   & 2.56570e+02  & 1.77548e+02 & 5.40904e-01 & 3.97170e+01 \\
    \hline
    \multirow{4}*{LogGRF}   &err $\mu$ & 0.00000e+00   & 1.55836e-04  & 3.63980e-01 & 3.44471e-11 & 1.28669e-07 \\
                            &err $\nu$ & 3.44471e-11   & 1.46725e-04  & 3.63993e-01 & 8.62698e-16 & 1.28669e-07 \\
                            &fval      & 3.44471e-11   & 1.14813e-02  & 0.00000e+00 & 2.25891e-02 & 2.25891e-02\\
                            &time      & 1.46135e+01   & 2.46276e+02  & 1.92535e+02 & 6.65159e-01 & 6.66666e66 \\
    \hline
    \multirow{4}*{Cauchy}   &err $\mu$ & 0.00000e+00   & 2.44488e-04 & 4.14467e-01 & 6.45741e-10 & 1.21786e-07 \\
                            &err $\nu$ & 6.45741e-10   & 2.03140e-04 & 4.14312e-01 & 8.02487e-16 & 1.32475e-07 \\
                            &fval      & 2.70716e-02   & 2.70958e-02 & 0.00000e+00 & 2.07987e-02 & 2.07987e-01 \\
                            &time      & 1.41227e+01   & 3.26161e+02 & 2.28633e+02 & 4.73294e-01 & 3.97828e+01 \\
    \hline
    \multirow{4}*{LogitGRF} &err $\mu$ & 0.00000e+00   & 1.52596e-04  & 3.99687e-01  & 1.23453e-09 & 1.24918e-07 \\
                            &err $\nu$ & 1.23453e-09   & 1.43546e-04  & 3.99684e-01  & 8.23705e-16 & 1.26019e-07 \\
                            &fval      & 1.90301e-02   & 1.90381e-02  & 0.00000e+00  & 2.81800e-02 & 2.81800e-02 \\
                            &time      & 1.51339e+01   & 2.39396e+02  & 1.36987e+02  & 5.37684e-01 & 3.92088e+01 \\
    \hline
    \multirow{4}*{Shape}    &err $\mu$ & 0.00000e+00   & 1.85553e-04  & 3.60264e-01  & 6.51926e-09 & 4.34572e-08 \\
                            &err $\nu$ & 6.51926e-09   & 2.04018e-04  & 3.59738e-01  & 5.09141e-16 & 4.54084e-08 \\
                            &fval      & 6.00148e-03   & 6.00026e-03  & 0.00000e+00  & 1.45999e-02 & 1.45999e-02 \\
                            &time      & 1.17216e+01   & 2.02916e+02  & 1.35861e+02  & 5.54126e-01 & 3.13619e+01 \\
    \hline
    \multirow{4}*{Classic}  &err $\mu$ & 0.00000e+00   & 2.05559e-04 & 1.79544e-01 & 2.61935e-10 & 1.20315e-07 \\
                            &err $\nu$ & 2.61934e-10   & 2.05644e-04 & 1.79543e-01 & 7.72765e-16 & 1.17474e-07 \\
                            &fval      & 2.13855e-03   & 2.14597e-03 & 0.00000e+00 & 2.80976e-02 & 2.80976e-01 \\
                            &time      & 1.54186e+01   & 3.57276e+02 & 2.30821e+02 & 4.58951e-01 & 3.80998e+01 \\
    \hline
    \multirow{4}*{Micro}    &err $\mu$ & 0.00000e+00   & 1.49094e-04  & 5.35081e-01 & 1.25146e-09 & 1.29936e-07 \\
                            &err $\nu$ & 1.25146e-09   & 1.22823e-04  & 5.35094e-01 & 6.58111e-16 & 1.64514e-07 \\
                            &fval      & 5.60383e-02   & 5.60339e-02  & 0.00000e+00 & 2.80043e-01 & 2.80042e-01 \\
                            &time      & 1.44868e+01   & 2.01784e+02  & 1.38029e+02 & 5.59559e-01 & 3.95853e+01 \\
    \hline
    \end{tabular}
    \label{tab:table1}
\end{table}


\documentclass[]{report}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{graphicx}

\graphicspath{ {images/} }

\title{CSCI 567 HW \# 4}
\author{Mohmmad Suhail Ansari \\ USC ID: 8518586692\\e-mail: mohmmada@usc.edu}

\begin{document}

\maketitle

\paragraph{Sol. 1.1}
	Given,
	\[ L(y_i, \hat{y}_i) = (y_i - \hat{y}_i)^2\]
	then, its derivative w.r.t $g_i$ will be
	\[ \frac{\partial{L(y_i, \hat{y}_i)}}{\partial{g_i}} = -2(y_i - \hat{y}_i)\]

\paragraph{Sol. 1.2}
	Optimal $h^{*}$ can be found by calculating the gradient of
	\[ L = \sum_{i = 1}^{n} (-g_i - \gamma h(x_i)) \]
	 w.r.t $\gamma$ and equating it to 0, i.e.
	 \[ \frac{\partial{L}}{\partial{\gamma}} = \sum_{i = 1}^{n} - 2h(x_i)(-g_i - \gamma h(x_i)) = 0\]
	 \[ = \sum_{i = 1}^{n} h(x_i) g_i + \gamma h^2 (x_i) = 0\]
	 \[ \gamma^{*} = \frac{\sum_{i = 1}^{n}  2 h(x_i)(y_i - \hat{y}_i)}{\sum_{i = 1}^{n} h^2 (x_i)}\]

\paragraph{Sol. 1.3}
	Similar to 1.2 we again we select $\alpha^{*}$ which will minimize
	\[ \sum_{i = 1}^{n}  L(y_i, \hat{y_i} + \alpha h^{*}(x_i)) \]
	$\therefore$, we get
	\[ \frac{\partial{\sum_{i = 1}^{n}  L}}{\partial{\alpha}} = \sum_{i = 1}^{n} -2 h^{*}(x_i) (y_i - \hat{y_i} - \alpha h^{*}(x_i)) = 0 \]
	\[ = \sum_{i = 1}^{n}  -2h^{*}(x_i)(y_i - \hat{y}_i) + 2 \alpha h^{*2}(x)) = 0\]
	\[ \alpha^{*} = \frac{\sum_{i = 1}^{n} h^{*}(x_i)(y_i - \hat{y_i})}{\sum_{i = 1}^{n} h^{*2}(x_i)}\]

	Finally, the update rule can be given as
	\[ \hat{y_i} \leftarrow \hat{y_i} + \alpha^{*} h^{*}(x_i) \]
	\[ \hat{y_i} \leftarrow \hat{y_i} + \bigg[ \frac{\sum_{i = 1}^{n} h^{*}(x_i)(y_i - \hat{y_i})}{\sum_{i = 1}^{n} h^{*2}(x_i)}  \bigg] \bigg[ \frac{\sum_{
	i = 1}^{n}  2 h(x_i)(y_i - \hat{y}_i)}{\sum_{i = 1}^{n} h^2 (x_i)} \bigg] \]
	\[ \hat{y_i} \leftarrow \hat{y_i} + 2{\bigg[ \frac{\sum_{i = 1}^{n} h^{*}(x_i)(y_i - \hat{y_i})}{\sum_{i = 1}^{n} h^{*2}(x_i)} \bigg]}^2 \]

\paragraph{Sol. 2.1}
	Let our neural network consist of N input nodes, M linear activation hidden layers and K output nodes. Let $z_j^{L_i}$ be the $j^{th}$ node in the $L_{i}$ layer and $w_j^{L_{i}}$ be the weight, the we can write the output of the first hidden layer
	\[ z_j = \sum_{i=0}^{N} w_{ji}^{L_1} x_i \]
	Then for the second hidden layer, we have
	\[ z_j = \sum_{i=0}^{L_1} w_{ji}^{L_2} z_i \]
	and now, finally the output of the final output layer can be given as
	\[ y_j = \sigma (\sum_{i=0}^{L_{M-1}} w_{ji}^{L_M} z_i) \]
	\[ = \sigma (\sum_{i=0}^{L_{M-1}} w_{ji}^{L_M} \sum_{i=0}^{L_{M-2}} z_i) \]
	\[ = \sigma (\sum_{i=0}^{L_{M-1}} w_{ji}^{L_M} \sum_{i=0}^{L_{M-2}} w_{ji}^{L_M-1} \hdots \sum_{i=0}^{N} w_j^{L_1} x_i) \]

	Which can be written as
	\[ y_j = \sigma (WX) \]
	where
	\[ W = \sum_{i=0}^{L_{M-1}} w_{ji}^{L_M} \sum_{i=0}^{L_{M-2}} w_{ji}^{L_M-1} \hdots \sum_{i=0}^{N} w_{ji}^{L_1}\]
	and
	\[ X = [x_1 \quad x_2\quad x_3 \hdots x_N ]\]

	Which is similar to Logistic Regression.

\paragraph{Sol. 2.2}
	Given
	\[ L(y, \hat{y}) = \frac{1}{2}((y_1 - \hat{y_1})^2 + (y_2 - \hat{y_2})^2)  = \frac{1}{2} \sum_{j=1}^2 (y_j - \hat{y}_j)^2\]
	where
	\[ \hat{y}_j = \sum_{k=1}^{4} v_{jk} z_k\] and
	\[ z_k = tanh(\sum_{i=3}^{3} w_{ki} x_i) \]

	taking partial differential of $L$ w.r.t $v_{jk}$, we get
	\[ \frac{\partial{L}}{\partial{v_{jk}}} = -z_k (y_j - \hat{y}_j)\]
	we know that

	\[ \frac{\partial{tanh(x)}}{\partial{x}} = 1 - {tanh}^{2} (x) \]

	$\therefore$

	\[ \frac{\partial{tanh(\sum_{i=3}^{3} w_{ki} x_i)}}{\partial{w_{ki}}} = (1 - {tanh}^{2} (\sum_{i=3}^{3} w_{ki} x_i))x_i \]

	then, we have the partial differential of $L$ w.r.t $w_{ki}$ as
	\[ \frac{\partial{L}}{\partial{w_{ki}}} =-v_{jk} (y_j - \hat{y}_j)(1 - {tanh}^{2} (\sum_{i=3}^{3} w_{ki} x_i))x_i \]

	Therefore we get the update rules as

	\[ w_{ki} = w_{ki} + \eta v_{jk} (y_j - \hat{y}_j)(1 - {tanh}^{2} (\sum_{i=3}^{3} w_{ki} x_i))x_i\]
	and
	\[ v_{jk} = v_{jk} + \lambda z_k (y_j - \hat{y}_j)\]

\paragraph{Sol. 3.1}
	Linear Activations
	\begin{center}
		\begin{tabular}{l|c|c}
			\hline
			 Architecture                        & Accuracy   &   Time(seconds) \\
			\hline
			 {[$d_{in}$, $d_{out}$]}                      & 83.4198\%   &    3.0391190000 \\
			 {[$d_{in}$, 50, $d_{out}$]}                  & 83.5544\%   &    5.9478230000 \\
			 {[$d_{in}$, 50, 50, $d_{out}$]}              & 84.0503\%   &    8.1254930000 \\
			 {[$d_{in}$, 50, 50, 50, $d_{out}$]}          & 84.4424\%   &   10.2770940000 \\
			 {[$d_{in}$, 50, $d_{out}$]}                  & 83.9965\%   &    5.3710550000 \\
			 {[$d_{in}$, 500, $d_{out}$]}                 & 84.1925\%   &   22.6269160000 \\
			 {[$d_{in}$, 500, 300, $d_{out}$]}            & 84.7153\%   &   35.4776900000 \\
			 {[$d_{in}$, 800, 500, 300, $d_{out}$]}       & 85.1228\%   &   66.4187950000 \\
			 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]}  & 85.1882\%   &  103.6037080000 \\
			 %{[$d_{in}$, $d_{out}$]}                      & 55.7144\%   &         5.05344 \\
			 %{[$d_{in}$, 50, $d_{out}$]}                  & 30.3887\%   &         6.62551 \\
			 %{[$d_{in}$, 50, 50, $d_{out}$]}              & 64.0218\%   &         8.98298 \\
			 %{[$d_{in}$, 50, 50, 50, $d_{out}$]}          & 50.6439\%   &        10.83295 \\
			 %{[$d_{in}$, 50, $d_{out}$]}                  & 39.4034\%   &         6.70223 \\
			 %{[$d_{in}$, 500, $d_{out}$]}                 & 51.2782\%   &        35.09004 \\
			 %{[$d_{in}$, 500, 300, $d_{out}$]}            & 57.7173\%   &        56.51479 \\
			 %{[$d_{in}$, 800, 500, 300, $d_{out}$]}       & 72.3023\%   &       109.36959 \\
			 %{[$d_{in}$, 800, 800, 500, 300, $d_{out}$]}  & 72.3369\%   &       167.03330 \\
			\hline
		\end{tabular}
	\end{center}

	Both accuracy and training time for each architecture increases with increasing number of hidden layers and increasing number of nodes in each hidden layer.

\paragraph{Sol. 3.2}
	Sigmoid Activations
	\begin{center}
		\begin{tabular}{l|c|c}
			\hline
			 Architecture                        & Accuracy   &   Time(seconds) \\
			\hline
			 {[$d_{in}$, 50, $d_{out}$]}                 & 74.8280\%   &    9.4715190000 \\
 			 {[$d_{in}$, 500, $d_{out}$]}                & 76.7309\%   &   77.9355170000 \\
 			 {[$d_{in}$, 500, 300, $d_{out}$]}           & 72.2639\%   &  122.3402530000 \\
 			 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & 72.2639\%   &  241.0532810000 \\
 			 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & 72.2639\%   &  363.2464260000 \\
			 %{[$d_{in}$, 50, $d_{out}$]}                 & 72.3369\%   &         7.36070 \\
			 %{[$d_{in}$, 500, $d_{out}$]}                & 72.3484\%   &        75.58726 \\
			 %{[$d_{in}$, 500, 300, $d_{out}$]}           & 72.3408\%   &       120.31576 \\
			 %{[$d_{in}$, 800, 500, 300, $d_{out}$]}      & 72.3408\%   &       238.32516 \\
			 %{[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & 72.3408\%   &       362.21167 \\
			\hline
		\end{tabular}
	\end{center}

	The accuracy first increases when one hidden layer is used, but then decreases and remains stable for rest of the models with increasing number of hidden layers.

	I believe that the the lower accuracy (compared to linear activation function) for sigmoid activation function is the result of the sigmoid activation function's property of ``killing the gradient'', i.e. when a sigmoid neuron’s activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. Recall that during backpropagation, this (local) gradient will be multiplied to the gradient of this gate’s output for the whole objective.

\paragraph{Sol. 3.3}
	ReLu Activations
	\begin{center}
		\begin{tabular}{l|c|c}
			\hline
			 Architecture                        & Accuracy   &   Time(seconds) \\
			\hline
			 {[$d_{in}$, 50, $d_{out}$]}                 & 82.3165\%   &         5.90191 \\
			 {[$d_{in}$, 500, $d_{out}$]}                & 82.0282\%   &        35.21581 \\
			 {[$d_{in}$, 500, 300, $d_{out}$]}           & 81.8706\%   &        56.89574 \\
			 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & 81.0326\%   &       109.04267 \\
			 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & 80.1253\%   &       166.93421 \\
			\hline
		\end{tabular}
	\end{center}

	The accuracy seems to be decreasing with increasing hidden layers and number of nodes in hidden layers.

	The accuracy compared to linear activation is still lower, which I believe is because of ReLu neurons can ``die'', i.e. a large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any data point again. If this happens, then the gradient flowing through the unit will forever be zero from that point on.

	The training time for ReLu neurons is more than linear neurons but shorter than sigmoid neurons. Its shorter than sigmoid neurons because of ReLu's linear and non-saturating form which can be implemented by simply thresholding a matrix of activations at zero.

\paragraph{Sol. 3.4}
	L2 Regularization
	\begin{center}
		\begin{tabular}{l|c|c|c}
			\hline
			 Architecture                   &       $\lambda$ & Accuracy   &   Time(seconds) \\
			\hline
			{[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-7}$   & 80.5559\%   &  117.7750190000 \\
			{[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7} $  & 80.5059\%   &  117.2375600000 \\
			{[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-6}$  & 81.3901\%   &  117.9980010000 \\
			{[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-6} $  & 80.8019\%   &  118.1240910000 \\
			{[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-5}$  & 80.6135\%   &  118.6769920000 \\
			\hline
		\end{tabular}
	\end{center}

	Best $\lambda = 10^{-6}$

	The accuracy increases and decreases alternatively with increasing $\lambda$ (although the difference seems to be really small). We get this trend because we are trying to find the best $\lambda$ that generalizes our data the best and since the difference between the different $\lambda$ is small, so is the difference in their accuracies.

\paragraph{Sol. 3.5}
	Early Stopping and L2-regularization
	\begin{center}
	\begin{tabular}{l|c|c|c}
		\hline
		 Architecture                   &       $\lambda$ & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-7}$ & 79.8370\%   &  110.0074800000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7} $ & 78.7606\%   &  112.0828270000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-6}$ & 80.5251\%   &  111.0474780000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-6} $ & 79.7832\%   &  110.7920260000 \\
		\hline
		\end{tabular}
	\end{center}
	Best $\lambda = 10^{-6}$

	The trend is similar to the L2 regularization without early-stopping. Since, the $\lambda$ for both cases is the same and so is the trend in accuracy, I believe the early stopping did help, since it takes little to achieve similar results.

\paragraph{Sol. 3.5}
	SGD with weight decay
	\begin{center}
	\begin{tabular}{l|c|c|c|c}
	\hline
	 Architecture                   & Lambda   & Decay Rate   & Accuracy   &   Time(seconds) \\
	\hline
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $10^{-5}$      & 72.4791\%   &  424.3529670000 \\
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $5^{-5}$       & 73.2941\%   &  413.6669020000 \\
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $10^{-4}$      & 74.1322\%   &  429.5304860000 \\
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $3^{-4}$       & 72.5445\%   &  409.1038830000 \\
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $7^{-4}$       & 65.3097\%   &  391.0522660000 \\
	 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $10^{-3}$      & 72.3446\%   &  393.0722790000 \\
	\hline
	\end{tabular}
	\end{center}

	Best Decay Rate $ = 10^{-4}$

\paragraph{Sol. 3.6}
	Momentum
	\begin{center}
		\begin{tabular}{l|c|c|c|c}
		\hline
		 Architecture                   &   Decay Rate &     Momentum & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-4}$ & 0.99 & 84.6384\%   &  188.7084680000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-4}$ & 0.98 & 82.3704\%   &  188.8004400000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-4}$ & 0.95 & 78.8606\%   &  189.2771110000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-4}$ & 0.90 & 74.0822\%   &  189.9112340000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-4}$ & 0.85 & 75.6929\%   &  189.8105580000 \\
		\hline
	\end{tabular}
	\end{center}

	Best Momentum $= 0.99$

\paragraph{Sol. 3.7}
	Combination
	\begin{center}
		\begin{tabular}{l|c|c|c|c|c}
		\hline
		 Architecture                   &       Lambda &   Decay Rate &     Momentum & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]} & $10^{-6}$ & $10^{-4}$ & 0.99 & 85.6264\%   &  366.8762140000 \\
		\hline
		\end{tabular}
	\end{center}

	Yes, the accuracy for this combination is the best we have achieved so far.

\paragraph{Sol. 3.8}
	Grid search with Cross-Validation:
	\begin{center}
		\begin{tabular}{l|c|c|c|c|c}
		\hline
		 Architecture                        & Lambda   & Decay Rate   &     Momentum & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-7}$  & $10^{-5}$      & 0.99 & 84.6807\%   &   20.2832450000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-7}$  & $5^{-5}$       & 0.99 & 80.1753\%   &    3.3854140000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-7}$  & $10^{-4}$      & 0.99 & 84.3002\%   &   20.1343490000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-7}$   & $10^{-5}$      & 0.99 & 85.0459\%   &   20.6533200000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-7}$   & $5^{-5}$       & 0.99 & 84.2848\%   &   20.4618000000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-7}$   & $10^{-4}$      & 0.99 & 84.0541\%   &   20.3314530000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-6}$  & $10^{-5}$      & 0.99 & 84.2002\%   &   19.6094510000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-6}$  & $5^{-5}$       & 0.99 & 84.0234\%   &   19.9445860000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-6}$  & $10^{-4}$      & 0.99 & 83.3737\%   &   19.9657800000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-6}$   & $10^{-5}$      & 0.99 & 85.3765\%   &   19.4346490000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-6}$   & $5^{-5}$       & 0.99 & 84.1118\%   &   19.6701670000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $5^{-6}$   & $10^{-4}$      & 0.99 & 82.9009\%   &   19.8740320000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-5}$  & $10^{-5}$      & 0.99 & 85.2036\%   &   19.9823050000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-5}$  & $5^{-5}$       & 0.99 & 83.9888\%   &   20.1908340000 \\
		 {[$d_{in}$, 50, $d_{out}$]}    & $10^{-5}$  & $10^{-4}$      & 0.99 & 83.3237\%   &   20.2327580000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-7}$  & $10^{-5}$      & 0.99 & 85.1228\%   &  109.6329600000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-7}$  & $5^{-5}$       & 0.99 & 84.6846\%   &  110.6199690000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-7}$  & $10^{-4}$      & 0.99 & 84.5039\%   &  110.8651400000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-7}$   & $10^{-5}$      & 0.99 & 81.2363\%   &   12.2082780000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-7}$   & $5^{-5}$       & 0.99 & 84.7230\%   &  110.2654860000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-7}$   & $10^{-4}$      & 0.99 & 81.4862\%   &   12.9311620000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-6}$  & $10^{-5}$      & 0.99 & 85.0613\%   &  109.4635540000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-6}$  & $5^{-5}$       & 0.99 & 81.4900\%   &   10.8740570000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-6}$  & $10^{-4}$      & 0.99 & 84.4847\%   &  110.2498010000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-6}$   & $10^{-5}$      & 0.99 & 85.1267\%   &  109.9373850000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-6}$   & $5^{-5}$       & 0.99 & 84.7422\%   &  112.1203260000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $5^{-6}$   & $10^{-4}$      & 0.99 & 80.5405\%   &   12.0689210000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-5}$  & $10^{-5}$      & 0.99 & 85.0998\%   &  110.8005040000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-5}$  & $5^{-5}$       & 0.99 & 84.7615\%   &  109.8176930000 \\
		 {[$d_{in}$, 500, $d_{out}$]}   & $10^{-5}$  & $10^{-4}$      & 0.99 & 84.5270\%   &  110.5633620000 \\
		\end{tabular}

		\begin{tabular}{l|c|c|c|c|c}
		\hline
		 Architecture                        & Lambda   & Decay Rate   &     Momentum & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-7}$  & $10^{-5}$      & 0.99 & 85.6072\%   &  182.9771570000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-7}$  & $5^{-5}$       & 0.99 & 79.6717\%   &   19.3831510000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-7}$  & $10^{-4}$      & 0.99 & 78.8990\%   &   19.4750520000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-7}$   & $10^{-5}$      & 0.99 & 85.8071\%   &  182.3738550000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-7}$   & $5^{-5}$       & 0.99 & 79.4795\%   &   19.3143720000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-7}$   & $10^{-4}$      & 0.99 & 85.1920\%   &  182.6694480000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-6}$  & $10^{-5}$      & 0.99 & 86.4568\%   &  181.8394820000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-6}$  & $5^{-5}$       & 0.99 & 85.3958\%   &  182.5855900000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-6}$  & $10^{-4}$      & 0.99 & 85.2843\%   &  182.2573850000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-6}$   & $10^{-5}$      & 0.99 & 86.1608\%   &  183.5565350000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-6}$   & $5^{-5}$       & 0.99 & 85.5726\%   &  184.1752500000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $5^{-6}$   & $10^{-4}$      & 0.99 & 78.7529\%   &   18.0114930000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-5}$  & $10^{-5}$      & 0.99 & 86.1069\%   &  181.5487010000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-5}$  & $5^{-5}$       & 0.99 & 85.4880\%   &  181.8470030000 \\
		 {[$d_{in}$, 500, 300, $d_{out}$]}           & $10^{-5}$  & $10^{-4}$      & 0.99 & 85.0344\%   &  182.3495300000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-7}$  & $10^{-5}$      & 0.99 & 86.8796\%   &  370.1537460000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-7}$  & $5^{-5}$       & 0.99 & 86.3107\%   &  372.6882040000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-7}$  & $10^{-4}$      & 0.99 & 85.9647\%   &  375.3052920000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-7}$   & $10^{-5}$      & 0.99 & 77.2498\%   &   31.2794710000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-7}$   & $5^{-5}$       & 0.99 & 86.3453\%   &  367.1678460000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-7}$   & $10^{-4}$      & 0.99 & 85.9493\%   &  371.9213370000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-6}$  & $10^{-5}$      & 0.99 & 86.5260\%   &  369.9097790000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-6}$  & $5^{-5}$       & 0.99 & 86.3376\%   &  369.4920700000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-6}$  & $10^{-4}$      & 0.99 & 78.1340\%   &   35.0205110000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-6}$   & $10^{-5}$      & 0.99 & 86.9681\%   &  373.8522750000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-6}$   & $5^{-5}$       & 0.99 & 86.5490\%   &  381.4162190000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $5^{-6}$   & $10^{-4}$      & 0.99 & 85.8263\%   &  394.7567470000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-5}$  & $10^{-5}$      & 0.99 & 77.4036\%   &   32.6861520000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-5}$  & $5^{-5}$       & 0.99 & 86.3453\%   &  391.6142450000 \\
		 {[$d_{in}$, 800, 500, 300, $d_{out}$]}      & $10^{-5}$  & $10^{-4}$      & 0.99 & 76.4733\%   &   33.5267370000 \\
	  	\end{tabular}

		\begin{tabular}{l|c|c|c|c|c}
		\hline
		 Architecture                        & Lambda   & Decay Rate   &     Momentum & Accuracy   &   Time(seconds) \\
		\hline
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-7}$  & $10^{-5}$      & 0.99 & 87.5716\%   &  587.6533990000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-7}$  & $5^{-5}$       & 0.99 & 74.4282\%   &   48.7787180000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-7}$  & $10^{-4}$      & 0.99 & 73.1865\%   &   48.3710490000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $10^{-5}$      & 0.99 & 74.5627\%   &   48.9536650000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $5^{-5}$       & 0.99 & 72.6444\%   &   54.2774970000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-7}$   & $10^{-4}$      & 0.99 & 86.0800\%   &  583.3533200000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-6}$  & $10^{-5}$      & 0.99 & 87.3409\%   &  575.2738650000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-6}$  & $5^{-5}$       & 0.99 & 74.6896\%   &   48.5039770000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-6}$  & $10^{-4}$      & 0.99 & 74.9779\%   &   48.4011350000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-6}$   & $10^{-5}$      & 0.99 & 87.5293\%   &  580.0695650000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-6}$   & $5^{-5}$       & 0.99 & 73.9938\%   &   47.8524650000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $5^{-6}$   & $10^{-4}$      & 0.99 & 86.1761\%   &  575.5397880000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-5}$  & $10^{-5}$      & 0.99 & 87.7100\%   &  568.8467470000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-5}$  & $5^{-5}$       & 0.99 & 73.8708\%   &   47.4766680000 \\
		 {[$d_{in}$, 800, 800, 500, 300, $d_{out}$]} & $10^{-5}$  & $10^{-4}$      & 0.99 & 86.5952\%   &  569.0146650000 \\
		\hline
		\end{tabular}
	\end{center}

	The best configuration we could obtain is
	\[ Architecture = [d_{in}, \quad 800, \quad 800, \quad 500, \quad 300, \quad d_{out}] \]
	\[ \lambda = 10^{-05} \]
	\[ Decay = 10^{-05} \]
	\[ Momentum = 0.99 \]
	\[ Accuracy  = 87.71\% \]

\end{document}


\chapter{Description of Utilities in \numbers}
\label{c:numerics}

Each utility and the numerical algorithm used in the utility are
described in this chapter. Although most of the algorithms are
well-known and documented in several references, it is important to
understand the actual methods used in each of the utilities to avoid
misinterpretation of the output and to understand the limitations and
assumptions used to calculate the properties and values.

Another reason for describing the algorithms is that \numbers\ is
written as a basic shell program for reading \exo\ files.  If additional
utilities are developed for \numbers, it is likely that several of the
algorithms already implemented can be modified to perform the new
function.

This chapter is organized into several sections; each utility is
described in a separate section.  The descriptions include the function
of the utility and the algorithm used in the utility including its
assumptions and limitations.  The complete command syntax for each
utility is given in Section~\ref{sec:oper}; a summary of the command
syntax is given in Section~\ref{sec:opsum}.  Examples of the use of most
of the utilities are presented in Chapter~\ref{c:examples}.

\section{Mass Properties Utility}\label{sec:mass}

The mass properties utility calculates the volume, mass, mass moments of
inertia, and centroid location of the body.  These values are often
required in a finite element analysis to ensure that the finite element
idealization correctly approximates the actual body.  This utility also
lists the minimum, maximum, and average element volume for each material
block.  A summary of the equations used to calculate the mass properties
is given in this section.

The moments of inertia are originally calculated about the origin of the
coordinate system and then transferred to the centroid using the
parallel-axis theorem. The mass and the moments of inertia about the
origin of the coordinate system are given by the following integrals:
\begin{eqnarray}
 M      &=& \int_V \rho dV                         \label{V}\\
 I_x    &=& \int_V \rho\left(y^2 + z^2\right)\,dV \label{IX}\\
 I_y    &=& \int_V \rho\left(x^2 + z^2\right)\,dV \label{IY}\\
 I_z    &=& \int_V \rho\left(x^2 + y^2\right)\,dV \label{IZ}\\
 I_{xy} &=& \int_V \rho xy\,dV                    \label{IXY}\\
 I_{xz} &=& \int_V \rho xz\,dV                    \label{IXZ}\\
 I_{yz} &=& \int_V \rho yz\,dV                    \label{IYZ}
\end{eqnarray}

where $M$ is the mass, $\rho$ is the density, $I$ is the mass moment
of inertia, and the subscripts $x$, $y$, and $z$ denote the axes about
which the moment is measured.  The double subscripts indicate products
of inertia.  Note that the product of inertia with respect to any two
orthogonal axes is zero if either of the axes is an axis of symmetry.

\paragraph{Axisymmetric Two-dimensional Body:}
For an axisymmetric two-dimensional body, the integrals are written in
cylindrical coordinates using the radius $r$ and the angle $\theta$,
where $x = r\cos\theta$ and $z = r\sin\theta$.  The $y$-axis is assumed
to be the axis of revolution.  The infinitesimal volume $dV$ is equal to
$r\,dA\,d\theta$ where $dA$ is an infinitesimal area equal to $dr\,dy$.
Rewriting Equations \eref{V} through~\eref{IZ} in terms of
$r$ and $\theta$, and performing the theta integration from $-\pi$ to
$\pi$ gives:
\begin{eqnarray}
M   &=& 2\pi\rho\int_A r\,dA   \\
I_y &=& 2\pi\rho\int_A r^3\,dA   \\
I_x = I_z &=& 2\pi\rho\int_A r y^2\,dA + \pi\rho \int_A r^3\,dA\nonumber \\
    &=& 2\pi\rho\int_A r y^2\,dA + {I_y\over2}
\end{eqnarray}
The products of inertia $I_{xy}$, $I_{xz}$, and $I_{yz}$ are zero since
all three axes are symmetry axes.

\paragraph{Two-dimensional Planar Body:}
For a two-dimensional planar body, area moments of inertia are
calculated; the depth in the $z$-direction is ignored. The integrals
simplify as follows:
\begin{eqnarray}
 I_x    &=& \rho\int_A y^2\,dA                   \\
 I_y    &=& \rho\int_A x^2\,dA                   \\
 I_z    &=& \rho\int_A \left(x^2 + y^2\right)\,dA = I_x + I_y \\
 I_{xy} &=& \rho\int_A xy\,dA
\end{eqnarray}
The products of inertia $I_{xz}$ and $I_{yz}$ are both zero.

\paragraph*{Evaluation of Integrals:} The integrals are evaluated using
the isoparametric element formulation and Gaussian Quadrature. Details
of this process can be found in most books on finite element
methodology.  In \numbers, two options are available for integration of
the equations.  The two-dimensional equations can be integrated with
either 1- or 4-point quadrature, and the three-dimensional equations
with either 1- or 8-point quadrature.  The mass moments of inertia for
bodies with non-rectangular elements are calculated more accurately with
the higher-order integration; volumes and areas are integrated exactly
with either integration option.  The second-order quadrature rule is
also useful for calculating the section properties of a non-standard
shape.  For this purpose, the discretization of the body is only refined
enough to capture the essential details of the shape since a structural
analysis will not be performed.

\paragraph{Calculation of Centroid Location:} The location of the
centroid is calculated using the first mass moments about each of the
three coordinate axes.  The mass moments are summed in each of the three
coordinate directions over the range of the elements.  The centroid
location is then given by the quotient of the mass moment sums divided
by the total mass of the body.  For a two-dimensional axisymmetric body,
$x_c$ and $z_c$ (the $x$ and $z$ coordinates of the centroid) are zero;
for a two-dimensional planar body, $z_c$ is zero.

\section{Locate Utility}\label{sec:locate}

The locate utility outputs the numbers of all nodes or elements that are
within a specified distance from a user-defined point, line, or plane.
This information is often required for plotting results at a specified
location in the model, for example, the acceleration or velocity at the
centerline of a body during an impact analysis.  The equations used to
calculate the distances from a node or element centroid to the point,
line, or plane are defined in the following subsections.  All equations
are defined in terms of a three-dimensional body; for a two-dimensional
body the $z$ coordinates are deleted.

Note that the point location procedure is also a circle (2D) and sphere
(3D) location procedure and the line location procedure is also a
cylinder (3D) location procedure since the search interval distance from
the point and line can be specified.

In the location utility, the distance for an element is defined to be
the distance to the element geometric center. The location of the
element center is calculated by summing the four or eight nodal
coordinates in each coordinate direction and dividing by the number of
nodes.

\subsection{Point Location}\label{sec:plocate}
The point location utility outputs all nodes or elements that are a
specified distance from a user-specified point.  The distance $d_r$ from
the node or element to the point ($x_0$, $y_0$, $z_0$) is given by:
\begin{equation}
d_r = \sqrt{ (x_0-x_n)^2 + (y_0-y_n)^2 + (z_0-z_n)^2}
\end{equation}
where $x_n$, $y_n$, and $z_n$ are the coordinates of the node or
element.  The distance is calculated for each node or element and
compared to the user-specified distance and tolerance.  If the node or
element is within this range, its number and coordinates are output. The
angles $\theta$ and $\phi$ are also calculated.  The angle $\theta$ is
the angle between the $x$ axis and the projection of the line from the
point to the node onto the $x$--$z$ plane.  The angle $\phi$ is the
angle between the $y$ axis and the line from the point to the node.  In
two dimensions, $\theta$ is the angle between the line and the $x$ axis.
Figure~\ref{f:theta} illustrates the definitions of $\theta$ and $\phi$.

\begin{figure}
\caption{Illustration of the Angles $\theta$ and $\phi$
for the Locate Point Utility}\label{f:theta}
\end{figure}

\subsection{Line Location}\label{sec:llocate}
The line location utility outputs all nodes or elements that are within
a specified distance from a user-specified line.  The distance is
measured normal to the line.   The parametric representation of the line
from $P_1$($x_1$, $y_1$, $z_1$) to $P_2$($x_2$, $y_2$, $z_2$) is $P(t) =
P_1 + (P_2 - P_1)t$.  The components of this line are:
\begin{equation}
\begin{array}{ccccc}
x &=& x_1 + (x_2 - x_1) t &=& x_1 + at   \\
y &=& y_1 + (y_2 - y_1) t &=& y_1 + bt   \\
z &=& z_1 + (z_2 - z_1) t &=& z_1 + ct   \\
\end{array}
\end{equation}
The minimum distance $d_t$ from the line to the point ($x_n$, $y_n$, $z_n$)
is
\begin{equation}
d_t^2 = (x_1+at-x_n)^2 +(y_1+bt-y_n)^2 +(z_1+ct-z_n)^2
\end{equation}
where the parameter $t$ is
\begin{equation}
t = -\frac{a(x_1-x_n) + b(y_1-y_n) +
c(z_1-z_n)}{a^2+b^2+c^2}\label{parametric}
\end{equation}

\subsection{Plane Location}\label{sec:slocate}
The plane location utility outputs all nodes or elements that are within
a specified distance from a user-specified plane.  The distance is
measured normal to the plane. A unique plane can be defined by a
specified point and a normal vector to the plane.  Given the point
($x_0$, $y_0$, $z_0$) and the unit vector $\vec{n} = a\vec{i} + b\vec{j} +
c\vec{k}$, the equation of the plane is:
\begin{eqnarray}
0 &=& ax + by + cz + d      \label{plane} \\
d &=& -( ax_0 + by_0 + cz_0 )
\end{eqnarray}
The normal distance $d_n$ from the plane to a node or
element center is:
\begin{equation}
d_n = \frac{\left| a x_n + b y_n + c z_n - d\right|}%
                                    {\sqrt{a^2 + b^2 + c^2}}
\end{equation}
where the subscript $n$ refers to the coordinates of the node or
element.  The normal distance is calculated for each node or element and
compared to the user-specified distance and tolerance.  If the node or
element is within this range, its number, coordinates, normal distance,
and radial distance are output.  The radial distance is the same
distance calculated in the point location utility.

\subsection{Sort Algorithm}

Although sort is not a location option, it is used in the location
utility to order the output.  \numbers\ uses the {\em heapsort} routine
which has been recommended in Reference~\cite{Press:nr} as a good
routine for a variety of sorting applications.  It is an ``in-place''
sort requiring no auxiliary storage.  It is an $N\log_2N$ process, not
only on average, but also for worst-case order of input data.  A FORTRAN
listing of the sort subroutine is given in Appendix~\ref{a:sort}.  One
disadvantage of the sort routine is that it is not ``stable.''  This
means that sorting the data a second time on a different field will
destroy the order of the first sort.  A method for sorting on two or
more fields simultaneously is being investigated and will be implemented
in a later version of \numbers.

\subsection{Sort Fields}\label{sortfields}

The output from the location utility can be sorted on any of the
calculated quantities or ``fields.''  The table below lists the fields
that are defined for each of the location options; these are defined
following the table.

\begingroup\small
\tabcolsep=3pt
\begin{center}
%\begin{tabular}{|>{\sf}l|c|  *{8}{>{\sf}l}|  }\hline
\begin{tabular}{|l|c|  *{8}{l}|  }\hline
\rm Option  &2D/3D &\multicolumn{8}{|c|}{\rm Valid \param{sort\_fields}}\\
\hline\hline
POINT  & 2D &  X  &  Y  &     &   & DISTANCE &            & THETA &     \\
POINT  & 3D &  X  &  Y  &  Z  &   & DISTANCE &            & THETA & PHI \\
LINE   & 2D &  X  &  Y  &     & T & DISTANCE & PARAMETRIC &       &     \\
LINE   & 3D &  X  &  Y  &  Z  & T & DISTANCE & PARAMETRIC &       &     \\
PLANE  & 3D &  X  &  Y  &  Z  &   & DISTANCE &            & RADIUS&     \\
\hline
\end{tabular}
\end{center}
\endgroup

\def\descriptionlabel #1{\sf #1:\hfil}
\begin{description}
\item[X, Y, {\rm or} Z] The \cmd{x}, \cmd{y}, or \cmd{z} coordinate
of the located node or element center.
\item[T] The parametric distance from the initial point of the line to
the located node or element center as defined in
Equation~\eref{parametric}.  The initial point of the line is located at
\cmd{T} = 0; the final point is located at \cmd{T} = 1.
\item[PARAMETRIC] The same quantity as \cmd{T}.
\item[DISTANCE] The distance from the node or element center to the
location point, line, or plane.  See the respective sections above for
the definition of distance for each of the location options.
\item[RADIUS] The shortest distance from the plane definition point (see
Section~\ref{sec:slocate}) to the located node or element center.
\item[THETA] For three-dimensional bodies, \cmd{THETA} is the angle
between the $x$ axis and the projection of the line from the point to
the node or element center onto the $x$--$z$ plane (see
Section~\ref{sec:plocate}).  For two-dimensional bodies, \cmd{theta} is
the angle between the line from the point to the node or element center
and the $x$ axis.
\item[PHI] The angle between the $y$ axis and the line from the point to
the located node or element center.
\end{description}

\section{Cavity Volume Utility}\label{sec:cavity}
The cavity volume utility calculates the volume and change in volume of
a cavity or hole in a body.  The boundary of the cavity is defined by
side set flags in the \exo\ file.  Two separate calculations are
involved in this utility; the first is the calculation of the
cavity volume, and the second is the calculation of the change in
volume.

The cavity volume is calculated by forming triangles (2D) or
pentahedrons (3D) for each segment of the cavity boundary.  The apex of
the triangles or pentahedrons are at a user-specified point. The bases
of the triangles or pentahedrons are the segments of the cavity boundary
side set.  The segments are element faces; in two-dimensions, the faces
are lines; in three-dimensions, the faces are four-node quadrilaterals.
The faces are assumed to be planar.  The volume of each triangle or
pentahedron is calculated and the sum of the volumes is, for certain
geometries, the volume of the cavity.

Figure~\ref{cvol2d} is an example of this process for a two-dimensional
cavity with three boundary segments: 1--2, 2--3, and 3--4.
\begin{figure}
\vspace{3.5in}
\caption{Illustration of Cavity Volume Determination for
a Two-Dimensional Cavity}\label{cvol2d}
\end{figure}
The area is calculated by summing the areas of the three triangles
{\sf012}, {\sf023}, and {\sf034}, where the three numbers refer to the
points defining the triangle and point {\sf0} is the apex or center
point. A similar process is used for three-dimensional cavities except
that pentahedral volumes are calculated instead of triangular areas. The
volume $V$ of a pentahedron with the apex at the point $(x_c,y_c,z_c)$
and the base formed by a boundary segment is~\cite{flanagan}:
\begin{eqnarray}
V = &\frac{1}{12}&\{
 ((2y_c - y_3) z_{42} + y_2 (z_{c3} + z_{c4}) - y_4 (z_{c3} + z_{c2})) x_1 \nonumber \\
&+& \phantom{\{}((y_4 - 2y_c) z_{31} + y_3 (z_{c4} + z_{c1}) - y_1 (z_{c4} + z_{c3})) x_2 \nonumber \\
&+& \phantom{\{}((y_1 - 2y_c) z_{42} + y_4 (z_{c1} + z_{c2}) - y_2 (z_{c4} + z_{c1})) x_3 \nonumber \\
&+& \phantom{\{}((2y_c - y_2) z_{31} + y_1 (z_{c2} + z_{c3}) - y_3 (z_{c2} + z_{c1})) x_4 \nonumber \\
&+& \phantom{\{}(y_{24}   z_{31} + y_{31} z_{42})  2  x_c\}
\end{eqnarray}
where the numerical subscript refers to the sequence of the node of the
boundary segment, the $c$ subscript refers to the center location, and
the double subscript $z_{ij}$ is defined as $z_i-z_j$.

The planar area $V_P$ of a triangle with the apex at the point
$(x_c,y_c)$ and the base formed by a boundary segment is:
\begin{equation}
V_P = \sfrac{1}{2}[(y_1-y_c) (x_2-x_c) - (y_2-y_c) (x_1-x_c)]
\end{equation}
where the subscript refers to the first and second node of the boundary
segment, and $y_c$ is the approximate vertical geometric center of the
cavity. It is calculated by summing the $y$ coordinates and dividing by
the total number of nodes on the boundary.

If the body is axisymmetric, the volume $V_A$ is calculated as:
\begin{eqnarray}
x_c &=& \sfrac{1}{3}(x_1 + x_2) \nonumber \\
V_A &=& 2\pi x_c V_P
\end{eqnarray}
where $V_P$ is the area calculated for the plane strain cavity.

The above method correctly calculates the volume of cavities defined by
a closed boundary.  However, many cavities are bounded on one or more
sides by symmetry planes which are not included in the cavity boundary
definition.  In two dimensions, the correct volume will be calculated if
the triangle apex is on the axis of symmetry.  In three dimensions, the
correct volume will be calculated if the apex of the pentahedron is on
the symmetry plane, or if it is on the intersection of two or more
symmetry planes.  If the apex point is not specified  by the user, it is
set to the point $(0,0,0)$ for three-dimensional bodies, and the point
$(0,y_c)$ for two-dimensional bodies, where $y_c$ is the approximate
vertical geometric center of the cavity.

\paragraph*{Cavity Volume Change:}
The change in cavity volume is calculated by calculating the volume of
the hexahedron formed by the original element face and the displaced
element face.  This is repeated for each element face on the cavity
boundary and the total volume change is the sum of the element face
volume changes.  Note that this calculation is correct for all cavities
even if they are bounded by symmetry planes.

For a three-dimensional cavity, the volume change $\Delta V$ for each
element on the cavity boundary is given by the following sequence of
calculations~\cite{PRONTO3D}.
\begin{equation}
\Delta V = \sfrac{1}{12}\sum_{i=1}^8 x_i B_i
\end{equation}
where $x_i$ for $i\le 4$ is the $x$ coordinate of a node on the element
face and $x_i$ for $5 \le i\le 8$ is the displaced coordinate of node
$i-4$.  The $B_i$ values are given by:
\begin{eqnarray*}
B_{1} &=&   y_2 (z_{63}-z_{45}) + y_3 z_{24} + y_4 (z_{38}-z_{52})
          + y_5 (z_{86}-z_{24}) + y_6 z_{52} + y_8 z_{45} \\
B_{2} &=&   y_3 (z_{74}-z_{16}) + y_4 z_{31} + y_1 (z_{45}-z_{63})
          + y_6 (z_{57}-z_{31}) + y_7 z_{63} + y_5 z_{16} \\
B_{3} &=&   y_4 (z_{81}-z_{27}) + y_1 z_{42} + y_2 (z_{16}-z_{74})
          + y_7 (z_{68}-z_{42}) + y_8 z_{74} + y_6 z_{27} \\
B_{4} &=&   y_1 (z_{52}-z_{38}) + y_2 z_{13} + y_3 (z_{27}-z_{81})
          + y_8 (z_{75}-z_{13}) + y_5 z_{81} + y_7 z_{38} \\
B_{5} &=&   y_8 (z_{47}-z_{61}) + y_7 z_{86} + y_6 (z_{72}-z_{18})
          + y_1 (z_{24}-z_{86}) + y_4 z_{18} + y_2 z_{61} \\
B_{6} &=&   y_5 (z_{18}-z_{72}) + y_8 z_{57} + y_7 (z_{83}-z_{25})
          + y_2 (z_{31}-z_{57}) + y_1 z_{25} + y_3 z_{72} \\
B_{7} &=&   y_6 (z_{25}-z_{83}) + y_5 z_{68} + y_8 (z_{54}-z_{36})
          + y_3 (z_{42}-z_{68}) + y_2 z_{36} + y_4 z_{83} \\
B_{8} &=&   y_7 (z_{36}-z_{54}) + y_6 z_{75} + y_5 (z_{61}-z_{47})
          + y_4 (z_{13}-z_{75}) + y_3 z_{47} + y_1 z_{54}
\end{eqnarray*}
where $y_i$ and $z_i$ are defined in the same way as $x_i$, and $z_{ij}$
is equal to $z_i - z_j$.

For a two-dimensional, planar cavity, the volume change $\Delta V_P$ for
each element on the cavity boundary is equal to
\begin{equation}
\Delta V_P = \sfrac{1}{2}\left[x_{12} (\Delta y_2 + \Delta y_1) -
           y_{12} (\Delta x_2 + \Delta x_1) +
           \Delta x_1 \Delta y_2 -
           \Delta x_2 \Delta y_1\right]
\end{equation}
where $x$ and $y$ are the original coordinates of the face nodes and
$\Delta x$ and $\Delta y$ are the $x$ and $y$ displacements of the
nodes.

For an axisymmetric cavity, the volume change $\Delta V_A$ for each
element on the cavity boundary is equal to
\begin{equation}
\Delta V_A = 2\pi x_c \Delta V_P
\end{equation}
where $x_c$ is the $x$ coordinate of the geometric center which is equal
to
\begin{equation}
x_c = \frac{(\Delta x_2 + \Delta x_1)}{4} +
      \frac{(x_2 + x_1)}{2}
\end{equation}

\section{Overlap Utility}\label{sec:overlap}

The overlap utility is intended to assist the analyst in the generation
of a mesh with valid contact surfaces.  One of the most difficult tasks
in generating a valid contact surface occurs when the contact surface is
curved with different discretizations on the two surfaces.  In this
case, a small gap must be inserted between the surfaces to prevent
penetration. Figure~\ref{f:overlap} illustrates the problem that can
occur if the gap is not large enough or nonexistent. Note that the nodes
on the more refined side of the contact surface penetrate the elements
on the other side of the surface.  This is a somewhat contrived example
to illustrate the problem; in actual finite element meshes it is very
difficult to notice the overlap unless the mesh is examined element by
element. However, if an analysis is run with overlapping contact
surfaces, the problem manifests itself in a sudden increase in kinetic
energy or by excessive deformations and velocities in the overlapped
portion of the mesh since the code will separate the contact surface in
one timestep.  Currently, the utility only checks for penetration of the
master surface by the slave surface; however, other checks for valid
contact surfaces, such as continuity or other requirements imposed by
analysis codes, can be added if there is a need.

\begin{figure}
\vspace{3.1in}
\caption{Illustration of Contact Surface Overlap due to
Discretization Mismatch on Curved Contact Surfaces}
\label{f:overlap}
\end{figure}

Many times, the most efficient method to determine whether the contact
surfaces overlap is to run the analysis for a very short period of time,
and then examine the results to see if the kinetic energy has increased
suddenly, or if the mesh has been ``blown'' apart where the analysis
code detects an overlapping contact surface and separates it in one
timestep---both symptoms indicate that a slideline overlap exists in the
original mesh.

The overlap utility was written to provide an efficient means of
determining whether an overlap exists prior to running an analysis. The
algorithm is logically broken into three separate steps. In the first
step, a ``bounding box'' is defined for each element on the master
surface.  The bounding box contains the coordinate ranges in each of the
coordinate directions.   Secondly, for each element on the master
surface, each node on the slave surface is tested to determine if it is
within the bounding box.  This test is a simple comparison of the slave
node coordinates with the coordinates of the bounding box.  Finally, if
a node is within an element's bounding box, the more computationally
expensive calculation of determining whether the node penetrates the
element is performed.  To determine if the node penetrates the element,
four triangles (two-dimensional body) or six pentahedrons
(three-dimensional body) are formed with the slave node as the apex and
each element face as the base.  The volume of each triangle or
pentahedron is calculated; if all of the volumes are positive, the node
is inside the element; if a volume is equal to zero, the node is on
the face. If the node is inside the element, its node number and the
number and connectivity of the penetrated element are output.  At the
end of the calculation, the total number of nodes penetrating the
surface and the number of nodes on the surface are output.

The volume $V_P^i$ of the three-dimensional pentahedron formed by
slave node $s$ and the element face $i$ is equal
to~\cite{flanagan}:
\begin{eqnarray}
12V_P^i &=& \phantom{{}+{}}x_A[(2y_S - y_C)  z_{DB} + y_B  (z_{SC} + z_{SD}) -
     y_D  (z_{SC} + z_{SB})] \nonumber \\
&\ &  {}  + x_B[(y_D - 2y_S)  z_{CA} + y_C  (z_{SD} + z_{SA})-
     y_A  (z_{SD} + z_{SC})] \nonumber \\
&\ &  {}  + x_C[(y_A - 2y_S)  z_{DB} + y_D  (z_{SA} + z_{SB})-
     y_B  (z_{SD} + z_{SA})] \nonumber \\
&\ &  {}  + x_D[(2y_S - y_B)  z_{CA} + y_A  (z_{SB} + z_{SC})-
     y_C (z_{SB} + z_{SA})] \nonumber \\
&\ &  {}  + 2x_S[y_{BD} z_{CA} + y_{CA} z_{DB} ]
\end{eqnarray}
where the subscript $S$ refers to the slave node, and the subscripts
$A$, $B$, $C$, and $D$ refer to the nodes on an element face (assumed to
be planar) as given in Table~\ref{t:nodes}.

\begin{table}
\parbox[t]{3.0in}{
\begin{center}
\begin{tabular}{l|cccccc}
\multicolumn{7}{c}{Hexahedron} \\
\multicolumn{7}{c}{\ } \\
  & \multicolumn{6}{c}{Face} \\
  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
A & 1 & 6 & 6 & 5 & 4 & 1 \\
B & 2 & 7 & 5 & 1 & 3 & 5 \\
C & 3 & 3 & 8 & 4 & 7 & 6 \\
D & 4 & 2 & 7 & 8 & 8 & 2 \\
\end{tabular}
\end{center}}\hfil
\parbox[t]{3.0in}{
\begin{center}
\begin{tabular}{l|cccc}
\multicolumn{5}{c}{Quadrilateral} \\
\multicolumn{5}{c}{\ } \\
  & \multicolumn{4}{c}{Face} \\
  & 1 & 2 & 3 & 4 \\ \hline
A & 1 & 2 & 3 & 4 \\
B & 2 & 3 & 4 & 1 \\
\end{tabular}
\end{center}}
\caption{Numbering of Face Nodes on an Eight-Node Hexahedral Element and
a Four-Node Quadrilateral Element}\label{t:nodes}
\end{table}

The volume $V_T^i$ of the two-dimensional triangle formed by the
slave node $s$ and the element face $i$ is equal to:
\begin{equation}
2V_T^i = x_A (y_B - y_S) + x_B (y_S - y_A) + x_S (y_A - y_B)
\end{equation}
where the subscript $S$ refers to the slave node, and the subscripts
$A$ and $B$ refer to the nodes on an element face as given in
Table~\ref{t:nodes}.

\section{Time Step Estimation Utility}\label{sec:timestep}

Explicit integration is used in most large-deformation, nonlinear,
transient dynamics finite element codes, for example
\code{PRONTO}~\cite{PRONTO2D,PRONTO3D}. This utility provides an
estimate of the time step size which will be used in the explicit
integration in these codes.

The stable time step $\Delta t$ for the central difference operator
commonly used in transient dynamic analysis codes is given
by~\cite{Cook}
\begin{equation}
\Delta t \leq {2\over \omega_{\max}} \left(\sqrt{1+\epsilon^2} -
\epsilon\right)
\end{equation}
where $\omega_{\max}$ is the maximum frequency of the mesh, and
$\epsilon$ is the fraction of critical damping in the highest element
mode.  In a explicit integration finite element code, the linear bulk
viscosity term is an estimate of $\epsilon$~\cite{PRONTO2D}. The default
value of $\epsilon$ is 0.06 which is the default value of the linear
bulk viscosity used in \code{PRONTO2D} and \code{PRONTO3D}.

Flanagan and Belytschko~\cite{flanagan} have derived simple
formulas for bounding the maximum eigenvalues for the uniform strain
eight-node hexahedron and four-node quadrilateral which can be used to
provide conservative estimates of the maximum frequency.  The maximum
frequency estimate for a rectangular quadrilateral element is
\begin{equation}
\hat\omega_{\max}^2 = {4(\lambda + 2\mu)\over\rho}\left({1\over s_1^2} +
   {1\over s_2^2}\right)
\end{equation}
where $s_1$ and $s_2$ are the lengths of the sides of the rectangle,
$\lambda$ and $\mu$ are Lame's constants, $\rho$ is the density, and
$\hat\omega_{\max}$ is the predicted value for the maximum frequency.
Similarly, for a rectangular parallelepiped hexahedron,
\begin{equation}
\hat\omega_{\max}^2 = {4(\lambda + 2\mu)\over\rho}\left({1\over s_1^2} +
   {1\over s_2^2}+{1\over s_3^2}\right)
\end{equation}

Substituting the maximum frequency equation into the stable time step
equation gives the following estimate for the stable time step size:
\begin{equation}
\Delta\hat t\le \sqrt{\frac{\rho}{\lambda+2\mu}}
      \left(\sum_{i=1}^{n_D}\frac{1}{s_i^2}\right)^{-1/2}
      \left(\sqrt{1+\epsilon^2} - \epsilon\right)
\end{equation}
where $\Delta\hat t$ is the estimate of the stable time step, and $n_D$
is the number of dimensions.  The first quantity on the right-hand side
of the inequality is the inverse of the dilatational wave speed which is
input by the user.  The second quantity is calculated by \numbers\ with
the assumption that the element is rectangular.

The output from this utility includes the calculated time step, the
element with the minimum time step, and the number of time steps per
millisecond of analysis time for each element block.

The number of time steps per millisecond can be used to estimate the
computer time required to perform an analysis.  Most explicit transient
dynamics codes output the average CPU time required to perform the
calculations for one element for one time step. Although this quantity
varies for different constitutive models and the number of contact
surfaces (slidelines), the average value is usually relatively constant
and well known.  The CPU time per millisecond of analysis time can be
estimated using the formula
\begin{equation}
{CPU\over ms} = \left(10^{-3}\over \Delta\hat t\right)
                ({\sf Speed})({\sf NUMEL})
\end{equation}
where {\sf NUMEL} is the number of elements, {\sf Speed} is the CPU time
per element per timestep, and $(10^{-3}/\Delta\hat t)$ is the number of
time steps per millisecond.

\section{Gap Utility}\label{sec:gap}

The gap utility is used to determine the distance between nodes on two
surfaces.  One surface is called the ``master'' surface and the other
surface is the ``slave'' surface.  For each node on the master surface,
a normal vector is calculated as shown below.  A matching process is
then performed to determine the ``closest'' slave node to each master
node, where closeness can be defined either as closest to the master
node normal vector, or closest in absolute distance.  Figure~\ref{f:gap}
illustrates the two closeness measures.  In this figure, Node~$M$ is the
master node and nodes $S_1$ and~$S_2$ are two slave nodes.  If the
absolute distance is used, node~$S_1$ will be the matching node; if the
distance to the normal vector is used, node~$S_2$ will be the matching
node.

\begin{figure}
\caption{Illustration of Closeness Measures Used in the Gap Utility}
\label{f:gap}
\end{figure}

After all of the nodes on the master surface have been matched to a node
on the slave surface, the normal and tangential distances, measured in
the coordinate frame of the normal vector, are determined for the
undeformed geometry and for each of the selected timesteps in the
database.  This utility is normally used to calculate the change in
distance between two surfaces, for example, the closure of a drift in a
geomechanics problem or the slip (tangential movement) along a contact
surface.

The normal for a node is defined to be the average normal of the element
faces for all elements connected to the node.  For a three-dimensional
body, a technique developed by Newell~\cite{Rogers:pefcg} gives the
exact normal for planar faces and a ``best'' approximation for almost
planar faces. The coefficients $a$, $b$, and $c$ of the normal vector
for an element face $\vec{n} = a\vec{i} + b\vec{j} + c\vec{k}$ are given by:
\begin{eqnarray}
a &=& \sum_{i=1}^n (y_i - y_j)(z_i + z_j) \nonumber\\
b &=& \sum_{i=1}^n (z_i - z_j)(x_i + x_j) \label{3dnorm}\\
c &=& \sum_{i=1}^n (x_i - x_j)(y_i + y_j) \nonumber
\end{eqnarray}
where $n$ is the number of nodes per face, and $j = 1$ if $i=n$, or
$j=i+1$ if $i\ne n$.  The vector is then normalized and added to the
direction cosine matrix for each node on the face.  After all of the
element normals have been computed and added to the direction cosine
matrix, the average direction cosine unit vector for each node is
determined by normalizing each entry in the direction cosine matrix.

The procedure is similar for a two-dimensional body, except that the
normal vector for an element face $\vec{n} = a\vec{i} + b\vec{j}$ is given by:
\begin{eqnarray}
      r &=& \sqrt{(x_i - x_j)^2 + (y_i - y_j)^2} \nonumber \\
      a &=& (x_i - x_j) / r                      \label{2dnorm}\\
      b &=& (y_i - y_j) / r                      \nonumber
\end{eqnarray}
where $x_i$ and $x_j$ are the $x$ coordinates of the two face nodes, and
$y_i$ and $y_j$ are the $y$ coordinates of the two face nodes.

After the average nodal normals have been calculated, the node matching
process is performed.  For each node on the master surface, the {\em
closest} node on the slave surface is determined.  There are two
criteria for determining closeness: radial distance and tangential
distance.  The radial distance $d_r$ between two nodes is simply
\begin{equation}
d_r = \sqrt{(x_s - x_m)^2 + (y_s - y_m)^2 + (z_s - z_m)^2}\label{distr}
\end{equation}
where $x$, $y$, and $z$ are the nodal coordinates and the subscripts $m$
and $s$ refer to the master and slave nodes, respectively.

The tangential distance is the distance from the master node's normal
vector to the slave node.  The distance is given by~\cite{Rogers:pefcg}:
\begin{eqnarray}
d_t &=& \sqrt{d_r^2 - d_n^2}\label{distt}  \\
d_n &=& -\left(a(x_m - x_s) + b(y_m - y_s)+c(z_m-z_s)\right)\label{distn}
\end{eqnarray}
where $d_r$ is the radial distance given in Equation~\eref{distr}, $d_n$
is the distance to the slave node measured in the direction of the
normal vector, and $a$, $b$, and $c$ are the components of the unit
normal vector determined in Equations \eref{3dnorm} or~\eref{2dnorm}.

\section{Limits Utility}

The limits utility provides the basic function of determining the
minimum, maximum, and range of the coordinates for each of the material
blocks.  No special algorithms are used in this utility and there are no
inherent limitations or assumptions in its implementation. The utility
simply examines the coordinates of each element in each element block
and saves the minimum and maximum values.  After all of the elements
have been processed, the minimum and maximum values are subtracted to
determine the coordinate range of the data.


\newcommand{\RR}{\ensuremath{\mathbb{R}}}
The \texttt{scipy.optimize} subpackage provides functions for the numerical
solution of several classes of root finding and optimization problems.
Here we highlight recent additions through SciPy~1.0.

%\subsubsection{Root Finding}
%The general ``root finding'' problem is to find a root $\mathbf{x} \in \RR^m$ of $\mathbf{f}: \RR^m \rightarrow \RR^m$, that is, to solve
%\begin{equation}
%\mathbf{f}(\mathbf{x}) = \mathbf{0}
%\end{equation}
%for a solution $\mathbf{x}$.\footnote{Equivalently the problem is to simultaneously find the roots $x_i \in \RR$ of several scalar functions $f_i : \RR \rightarrow \RR$, that is, to solve $f_i(x_0, x_1, \dots, x_{m-1}) = 0$ for $x_i$, $i \in \{0, 1, \dots {m-1}\}$.} The function \texttt{scipy.optimize.root} provides a common interface to several algorithms for solving problems of this type. For the special case\footnote{that is, to solve a single scalar equation $f(x) = 0$ for a single scalar variable $x$} $m = 1$, any one of several specialized functions \texttt{brentq}, \texttt{brenth}, \texttt{ridder}, \texttt{bisect}, or \texttt{newton} may provide improved performance or accuracy. (Have there been any recent improvements? Do we want to summarize the methods as @antonior92 has done for $minimize$? Do we have to explain that these methods only provide \emph{one} solution, and that they are iterative based on a user-provided guess? Do we have to explain the notion of tolerance? Is this a good template for the beginning of the following subsections?)

\subsubsection*{Linear Optimization}

A new interior-point optimizer for continuous linear programming problems, \texttt{linprog} with \texttt{method='interior-point'}, was released with SciPy~1.0. Implementing the core algorithm of the commercial solver MOSEK \cite{andersen2000mosek}, it solves all of the 90+ NETLIB LP benchmark problems \cite{netlib} tested. Unlike some interior point methods, this homogeneous self-dual formulation provides certificates of infeasibility or unboundedness as appropriate.

A presolve routine \cite{andersen1995presolving} solves trivial problems and otherwise performs problem simplifications, such as bound tightening and removal of fixed variables, and one of several routines for eliminating redundant equality constraints is automatically chosen to reduce the chance of numerical difficulties caused by singular matrices. Although the main solver implementation is pure Python, end-to-end sparse matrix support and heavy use of SciPy's compiled linear system solvers---often for the same system with multiple right hand sides due to the predictor-corrector approach---provide speed sufficient for problems with tens of thousands of variables and constraints.

Compared to the previously implemented simplex method, the new interior-point method is faster for all but the smallest problems, and is suitable for solving medium- and large-sized problems on which the existing simplex implementation fails. However, the interior point method typically returns a solution near the center of an optimal face, yet basic solutions are often preferred for sensitivity analysis and for use in mixed integer programming algorithms. This motivates the need for a crossover routine or a new implementation of the simplex method for sparse problems in a future release, either of which would require an improved sparse linear system solver with efficient support for rank-one updates.

\subsubsection*{Nonlinear Optimization}
\paragraph{Local Minimization}
The \texttt{minimize} function provides a unified interface for finding local minima of nonlinear optimization problems. Four new methods for unconstrained optimization were added to \texttt{minimize} in recent versions of SciPy: \texttt{dogleg}, \texttt{trust-ncg}, \texttt{trust-exact}, and \texttt{trust-krylov}. All are trust-region methods that build a local model of the objective function based on first and second derivative information, approximate the best point within a local ``trust region'', and iterate until a local minimum of the original objective function is reached, but each has unique characteristics that make it appropriate for certain types of problems. For instance, \texttt{trust-exact} achieves fast convergence by solving the trust-region subproblem almost exactly, but it requires the second derivative Hessian matrix to be stored and factored every iteration, which may preclude the solution of large problems ($\geq 1000$ variables). On the other hand, \texttt{trust-ncg} and \texttt{trust-krylov} are well-suited to large-scale optimization problems because they do not need to store and factor the Hessian explicitly, instead using second derivative information in a faster, approximate way. A detailed comparison of the characteristics of all \texttt{minimize} methods is presented in Table~\ref{tab:minimize-si}; it illustrates the level of completeness that SciPy aims for when covering a numerical method or topic.

\paragraph{Global Minimization}
As \texttt{minimize} may return any local minimum, some problems require the use of a global optimization routine. The new \texttt{scipy.optimize.differential\textunderscore evolution} function \cite{Wormington1999,Storn1997} is a stochastic global optimizer that works by evolving a population of candidate solutions. In each iteration, trial candidates are generated by combination of candidates from the existing population. If the trial candidates represent an improvement, then the population is updated. Most recently, the SciPy benchmark suite gained a comprehensive set of 196 global optimization problems for tracking the performance of existing solvers over time and for evaluating whether the performance of new solvers merits their inclusion in the package.

\setlength{\tabcolsep}{3pt}
\begin{table}[H]
  \centering
  \caption{Optimization methods from \texttt{minimize}, which solves problems of the form $\min_x f(x)$, where $x \in \mathbb{R}^n$ and $f: \mathbb{R}^n \rightarrow \mathbb{R}$ .  The field \textit{version added} specifies the algorithm's first appearance in SciPy. Algorithms with \textit{version added} ``0.6*'' were added in version 0.6 or before.
    The field \textit{wrapper} indicates whether the implementation available in SciPy wraps a function written in a compiled language
    (e.g. C or FORTRAN). The fields \textit{\nth{1}} and \textit{\nth{2} derivatives}
    indicates whether first or second order derivatives are required. When \textit{\nth{2} derivatives} is flagged
    with $\sim$ the algorithm does not require second-order derivatives from
    the user; it computes an approximation internally and uses it to accelerate method convergence.
    \textit{Iterative Hessian factorization} denotes algorithms that factorize the Hessian in an iterative way,
    which does not require explicit matrix factorization or storage of the Hessian.
    \textit{Local convergence} gives a lower bound on the rate of convergence of the iterations sequence once the
    iterate is sufficiently close to the solution: linear (L), superlinear (S) and quadratic (Q). Convergence rates denoted S$^*$ indicate that the algorithm
    has a superlinear rate for the parameters used in SciPy, but can  achieve a quadratic convergence rate with other parameter choices.
    \textit{Global convergence} is marked for the algorithms with guarantees of convergence to a stationary
    point (i.e.\ a point $x^*$ for which $\nabla f(x^*) = 0$); this is \emph{not} a guarantee of convergence to a global minimum.
		\textit{Lines-search (LS) or trust-region (TR)} indicates which of the two globalization approaches is employed by the algorithm.
		The table also indicates which algorithms
    can deal with constraints on the variables. We distinguish among \textit{bound constraints} (i.e.\ $x^l \le x \le x^u$),
    \textit{equality constraints} (i.e.\ $c_{\text{eq}}(x) = 0$) and \textit{inequality constraints} (i.e.\ $c_{\text{ineq}}(x) \ge 0$).}
  \begin{tabular}{cccccccccccccc}
      & \rotatebox{80}{\texttt{Nelder-Mead}} & \rotatebox{80}{\texttt{Powell}} & \rotatebox{80}{\texttt{COBYLA}} & \rotatebox{80}{\texttt{CG}} & \rotatebox{80}{\texttt{BFGS}}&  \rotatebox{80}{\texttt{L-BFGS-B}} & \rotatebox{80}{\texttt{SLSQP}} & \rotatebox{80}{\texttt{TNC}} & \rotatebox{80}{\texttt{Newton-CG}} & \rotatebox{80}{\texttt{dogleg}} & \rotatebox{80}{\texttt{trust-ncg}} & \rotatebox{80}{\texttt{trust-exact}} & \rotatebox{80}{\texttt{trust-krylov}} \\
    \hline
    Version added &  0.6* &  0.6* &  0.6* &  0.6* &  0.6* &  0.6* &  0.9 &  0.6* &  0.6* & 0.13 & 0.13 & 0.19 & 1.0 \\
    \hline
    Wrapper & & & \cmark & & & \cmark & \cmark & \cmark & &  & & & \cmark \\
    \hline
    \nth{1} derivatives &  & & & \cmark  & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\
    \hline
    \nth{2} derivatives &  &  &  &  & $\sim$ & $\sim$ & $\sim$ & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\
    \hline
    \makecell{Iterative Hessian \\
    factorization} & & & &  & & & & \cmark & \cmark &  & \cmark &  & \cmark \\
    \hline
    Local convergence& & & & L & S &  L & S & S$^*$ & S$^*$ & Q & S$^*$ & Q & S$^*$  \\
    \hline
    Global convergence & & &  &   & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark  \\
    \hline
    \makecell{Line-search (LS) or\\ trust-region (TR)} & Neither  & LS &  TR & LS & LS & LS & LS & LS & LS & TR & TR & TR & TR \\
    \hline
    Bound constraints &&&\cmark&&&&\cmark&\cmark&\cmark&&&& \\
    \hline
    Equality constraints &&&&&&&\cmark&&&&& \\
    \hline
    Inequality constraint &&&\cmark&&&&\cmark&&&&& \\
    \hline
    References &
      \inlinecite{nelder_simplex_1965, wright_direct_1996} &
      \inlinecite{powell_efficient_1964} &
      \inlinecite{powell_direct_1994, powell_direct_1998, powell_view_2007} &
      \inlinecite{polak_note_1969, nocedal_numerical_2006} &
      \inlinecite{nocedal_numerical_2006} &
      \inlinecite{byrd_limited_1995, zhu_algorithm_1997} &
      \inlinecite{schittkowski_nonlinear_1982, schittkowski_nonlinear_1982-1, schittkowski_convergence_1983, kraft_software_1988} &
      \inlinecite{nash_newton-type_1984} &
      \inlinecite{nocedal_numerical_2006}  &
      \inlinecite{powell_new_1970, nocedal_numerical_2006} &
      \inlinecite{steihaug_conjugate_1983, nocedal_numerical_2006} &
      \inlinecite{conn_trust_2000, more_computing_1983} &
      \inlinecite{gould_solving_1999, doi:10.1080/10556788.2018.1449842} \\
    \hline
  \end{tabular}
  \label{tab:minimize-si}
\end{table}