   \section{Optimizers} \label{sec:Optimizers} The optimizer is another important entity in the
  RAVEN framework. It performs the driving of a specific ``goal function'' or ``objective function''
  over the model for value optimization. The Optimizer can be used almost anywhere a Sampler can be
  used, and is only distinguished from other AdaptiveSampler strategies for clarity.

\subsection{GradientDescent}
  The \xmlNode{GradientDescent} optimizer represents an a la carte option
  for performing gradient-based optimization with a variety of gradient
  estimation techniques, stepping strategies, and acceptance criteria. \hspace{12pt}
  Gradient descent optimization generally behaves as a ball rolling down a hill;
  the algorithm estimates the local gradient at a point, and attempts to move
  ``downhill'' in the opposite direction of the gradient (if minimizing; the
  opposite if maximizing). Once the lowest point along the iterative gradient search
  is discovered, the algorithm is considered converged. \hspace{12pt}
  Note that gradient descent algorithms are particularly prone to being trapped
  in local minima; for this reason, depending on the model, multiple trajectories
  may be needed to obtain the global solution.
\vspace{7pt} \\When used as part of a \xmlNode{MultiRun} step, this entity provides
        additional information through the \xmlNode{SolutionExport} DataObject. The
        following variables can be requested within the \xmlNode{SolutionExport}:
        \begin{itemize}
          \item \texttt{trajID}: integer identifier for different optimization starting locations and paths
             \item \texttt{iteration}: integer identifying which iteration (or step, or generation) a trajectory is on
             \item \texttt{accepted}: string acceptance status of the potential optimal point (algorithm dependent)
             \item \texttt{rejectReason}: discription of reject reason, 'noImprovement' means rejected the new optimization point for no improvement from last point, 'implicitConstraintsViolation' means rejected by implicit constraints violation, return None if the point is accepted
             \item \texttt{\{VAR\}}: any variable from the \xmlNode{TargetEvaluation} input or output; gives the value of that variable at the optimal candidate for this iteration.
             \item \texttt{stepSize}: the size of step taken in the normalized input space to arrive at each optimal point
             \item \texttt{conv\_\{CONV\}}: status of each given convergence criteria
             \item \texttt{CG\_task}: for ConjugateGradient, current task of line search. FD suggests continuing the search, and CONV indicates the line search converged and will pivot.

         \end{itemize}

  The \xmlNode{GradientDescent} node recognizes the following parameters:
    \begin{itemize}
      \item \xmlAttr{verbosity}: \xmlDesc{[silent, quiet, all, debug], optional},
        Desired verbosity of messages coming from this entity
      \item \xmlAttr{name}: \xmlDesc{string, required},
        User-defined name to designate this entity in the RAVEN input file.
  \end{itemize}

  The \xmlNode{GradientDescent} node recognizes the following subnodes:
  \begin{itemize}
    \item \xmlNode{objective}: \xmlDesc{string},
      Name of the response variable (or ``objective function'') that should be optimized
      (minimized or maximized).

    \item \xmlNode{variable}:
      defines the input space variables to be sampled through various means.
      The \xmlNode{variable} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{name}: \xmlDesc{string, optional},
            user-defined name of this Sampler. \nb As for the other objects,               this is
            the name that can be used to refer to this specific entity from other input blocks
          \item \xmlAttr{shape}: \xmlDesc{comma-separated integers, optional},
            determines the number of samples and shape of samples               to be taken.  For
            example, \xmlAttr{shape}=``2,3'' will provide a 2 by 3               matrix of values,
            while \xmlAttr{shape}=``10'' will produce a vector of 10 values.               Omitting
            this optional attribute will result in a single scalar value instead.               Each
            of the values in the matrix or vector will be the same as the single sampled value.
            \nb A model interface must be prepared to handle non-scalar inputs to use this option.
      \end{itemize}

      The \xmlNode{variable} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{distribution}: \xmlDesc{string},
          name of the distribution that is associated to this variable.               Its name needs
          to be contained in the \xmlNode{Distributions} block explained               in Section
          \ref{sec:distributions}. In addition, if NDDistribution is used,               the
          attribute \xmlAttr{dim} is required. \nb{Alternatively, this node must be omitted
          if the \xmlNode{function} node is supplied.}
          The \xmlNode{distribution} node recognizes the following parameters:
            \begin{itemize}
              \item \xmlAttr{dim}: \xmlDesc{integer, optional},
                for an NDDistribution, indicates the dimension within the NDDistribution that
                corresponds               to this variable.
          \end{itemize}

        \item \xmlNode{function}: \xmlDesc{string},
          name of the function that               defines the calculation of this variable from
          other distributed variables.  Its name               needs to be contained in the
          \xmlNode{Functions} block explained in Section               \ref{sec:functions}. This
          function must implement a method named ``evaluate''.               \nb{Each
          \xmlNode{variable} must contain only one \xmlNode{Function} or
          \xmlNode{Distribution}, but not both.}

        \item \xmlNode{initial}: \xmlDesc{comma-separated floats},
          indicates the initial values where independent trajectories for this optimization
          effort should begin. The number of entries should be the same for all variables, unless
          a variable is initialized with a sampler (see \xmlNode{samplerInit} below). Note these
          entries are ordered; that is, if the optimization variables are $x$ and $y$, and the
          initial               values for $x$ are \xmlString{1, 2, 3, 4} and initial values for $y$
          are \xmlString{5, 6, 7, 8},               then there will be four starting trajectories
          beginning at the locations (1, 5), (2, 6),               (3, 7), and (4, 8).
      \end{itemize}

    \item \xmlNode{TargetEvaluation}: \xmlDesc{string},
      name of the DataObject where the sampled outputs of the Model will be collected.
      This DataObject is the means by which the sampling entity obtains the results of requested
      samples, and so should require all the input and output variables needed for adaptive
      sampling.
      The \xmlNode{TargetEvaluation} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this entity (e.g. Samplers, Models, DataObjects)
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this entity; a subtype of the class (e.g. MonteCarlo, Code, PointSet)
      \end{itemize}

    \item \xmlNode{samplerInit}:
      collection of nodes that describe the initialization of the optimization algorithm.

      The \xmlNode{samplerInit} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{limit}: \xmlDesc{integer},
          limits the number of Model evaluations that may be performed as part of this optimization.
          For example, a limit of 100 means at most 100 total Model evaluations may be performed.

        \item \xmlNode{writeSteps}: \xmlDesc{[final, every]},
          delineates when the \xmlNode{SolutionExport} DataObject should be written to. In case
          of \xmlString{final}, only the final optimal solution for each trajectory will be written.
          In case of \xmlString{every}, the \xmlNode{SolutionExport} will be updated with each
          iteration               of the Optimizer.

        \item \xmlNode{initialSeed}: \xmlDesc{integer},
          seed for random number generation. Note that by default RAVEN uses an internal seed,
          so this seed must be changed to observe changed behavior. \default{RAVEN-determined}

        \item \xmlNode{type}: \xmlDesc{[min, max]},
          the type of optimization to perform. \xmlString{min} will search for the lowest
          \xmlNode{objective} value, while \xmlString{max} will search for the highest value.
      \end{itemize}

    \item \xmlNode{gradient}:
      a required node containing the information about which gradient approximation algorithm to
      use, and its settings if applicable. Exactly one of the gradient approximation algorithms
      below may be selected for this Optimizer.

      The \xmlNode{gradient} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{FiniteDifference}:
          if node is present, indicates that gradient approximation should be performed
          using Finite Difference approximation. Finite difference makes use of orthogonal
          perturbations         in each dimension of the input space to estimate the local gradient,
          requiring a total of $N$         perturbations, where $N$ is dimensionality of the input
          space. For example, if the input space         $\mathbf{i} = (x, y, z)$ for objective
          function $f(\mathbf{i})$, then FiniteDifference chooses         three perturbations
          $(\alpha, \beta, \gamma)$ and evaluates the following perturbation points:
          \begin{itemize}           \item $f(x+\alpha, y, z)$,           \item $f(x, y+\beta, z)$,
          \item $f(x, y, z+\gamma)$         \end{itemize}         and evaluates the gradient $\nabla
          f = (\nabla^{(x)} f, \nabla^{(y)} f, \nabla^{(z)} f)$ as         \begin{equation*}
          \nabla^{(x)}f \approx \frac{f(x+\alpha, y, z) - f(x, y, z)}{\alpha},
          \end{equation*}         and so on for $ \nabla^{(y)}f$ and $\nabla^{(z)}f$.

        \item \xmlNode{CentralDifference}:
          if node is present, indicates that gradient approximation should be performed
          using Central Difference approximation. Central difference makes use of pairs of
          orthogonal perturbations         in each dimension of the input space to estimate the
          local gradient, requiring a total of $2N$         perturbations, where $N$ is
          dimensionality of the input space. For example, if the input space         $\mathbf{i} =
          (x, y, z)$ for objective function $f(\mathbf{i})$, then CentralDifference chooses
          three perturbations $(\alpha, \beta, \gamma)$ and evaluates the following perturbation
          points:         \begin{itemize}           \item $f(x\pm\alpha, y, z)$,           \item
          $f(x, y\pm\beta, z)$,           \item $f(x, y, z\pm\gamma)$         \end{itemize}
          and evaluates the gradient $\nabla f = (\nabla^{(x)} f, \nabla^{(y)} f, \nabla^{(z)} f)$
          as         \begin{equation*}           \nabla^{(x)}f \approx \frac{f(x+\alpha, y, z) -
          f(x-\alpha, y, z)}{2\alpha},         \end{equation*}         and so on for $
          \nabla^{(y)}f$ and $\nabla^{(z)}f$.

        \item \xmlNode{SPSA}:
          if node is present, indicates that gradient approximation should be performed
          using the Simultaneous Perturbation Stochastic Approximation (SPSA).         SPSA makes
          use of a single perturbation as a zeroth-order gradient approximation,         requiring
          exactly $1$         perturbation regardless of the dimensionality of the input space. For
          example, if the input space         $\mathbf{i} = (x, y, z)$ for objective function
          $f(\mathbf{i})$, then SPSA chooses         a single perturbation point $(\epsilon^{(x)},
          \epsilon^{(y)}, \epsilon^{(z)})$ and evaluates         the following perturbation point:
          \begin{itemize}           \item $f(x+\epsilon^{(x)}, y+\epsilon^{(y)}, z+\epsilon^{(z)})$
          \end{itemize}         and evaluates the gradient $\nabla f = (\nabla^{(x)} f, \nabla^{(y)}
          f, \nabla^{(z)} f)$ as         \begin{equation*}           \nabla^{(x)}f \approx
          \frac{f(x+\epsilon^{(x)}, y+\epsilon^{(y)}, z+\epsilon^{(z)})) -               f(x, y,
          z)}{\epsilon^{(x)}},         \end{equation*}         and so on for $ \nabla^{(y)}f$ and
          $\nabla^{(z)}f$. This approximation is much less robust         than FiniteDifference or
          CentralDifference, but has the benefit of being dimension agnostic.
      \end{itemize}

    \item \xmlNode{stepSize}:
      a required node containing the information about which iterative stepping algorithm to
      use, and its settings if applicable. Exactly one of the stepping algorithms
      below may be selected for this Optimizer.

      The \xmlNode{stepSize} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{GradientHistory}:
          if this node is present, indicates that the iterative steps in the gradient
          descent algorithm should be determined by the sequential change in gradient. In
          particular, rather         than using the magnitude of the gradient to determine step
          size, the directional change of the         gradient versor determines whether to take
          larger or smaller steps. If the gradient in two successive         steps changes
          direction, the step size shrinks. If the gradient instead continues in the same
          direction, the step size grows. The rate of shrink and growth are controlled by the
          \xmlNode{shrinkFactor}         and \xmlNode{growthFactor}. Note these values have a large
          impact on the optimization path taken.         Large growth factors converge slowly but
          explore more of the input space; large shrink factors         converge quickly but might
          converge before arriving at a local minimum.

          The \xmlNode{GradientHistory} node recognizes the following subnodes:
          \begin{itemize}
            \item \xmlNode{growthFactor}: \xmlDesc{float},
              specifies the rate at which the step size should grow if the gradient continues in
              same direction through multiple iterative steps. For example, a growth factor of 2
              means               that if the gradient is identical twice, the step size is doubled.
              \default{1.25}

            \item \xmlNode{shrinkFactor}: \xmlDesc{float},
              specifies the rate at which the step size should shrink if the gradient changes
              direction through multiple iterative steps. For example, a shrink factor of 2 means
              that if the gradient completely flips direction, the step size is halved. Note that
              for               stochastic surfaces or low-order gradient approximations such as
              SPSA, a small value               for the shrink factor is recommended. If an
              optimization path appears to be converging               early, increasing the shrink
              factor might improve the search. \default{1.15}
          \end{itemize}

        \item \xmlNode{ConjugateGradient}:
          Base class for Step Manipulation algorithms in the GradientDescent Optimizer.
      \end{itemize}

    \item \xmlNode{acceptance}:
      a required node containing the information about the acceptability criterion for iterative
      optimization steps, i.e. when a potential new optimal point should be rejected and when
      it can be accepted. Exactly one of the acceptance criteria               below may be selected
      for this Optimizer.

      The \xmlNode{acceptance} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{Strict}:
          if this node is present, indicates that a Strict acceptance policy for         potential
          new optimal points should be enforced; that is, for a potential optimal point to
          become the new point from which to take another iterative optimizer step, the new response
          value         must be improved over the old response value. Otherwise, the potential opt
          point is rejected         and the search continues with the previously-discovered optimal
          point.
      \end{itemize}

    \item \xmlNode{convergence}:
      a node containing the desired convergence criteria for the optimization algorithm.
      Note that convergence is met when any one of the convergence criteria is met. If no
      convergence               criteria are given, then nominal convergence on gradient value is
      used.

      The \xmlNode{convergence} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{gradient}: \xmlDesc{float},
          provides the desired value for the local estimated of the gradient
          for convergence. \default{1e-6, if no criteria specified}

        \item \xmlNode{objective}: \xmlDesc{float},
          provides the maximum relative change in the objective function for convergence.

        \item \xmlNode{stepSize}: \xmlDesc{float},
          provides the maximum size in relative step size for convergence.

        \item \xmlNode{terminateFollowers}: \xmlDesc{[True, Yes, 1, False, No, 0, t, y, 1, f, n, 0]},
          indicates whether a trajectory should be terminated when it begins following the path
          of another trajectory.
          The \xmlNode{terminateFollowers} node recognizes the following parameters:
            \begin{itemize}
              \item \xmlAttr{proximity}: \xmlDesc{float, optional},
                provides the normalized distance at which a trajectory's head should be proximal to
                another trajectory's path before terminating the following trajectory.
          \end{itemize}

        \item \xmlNode{persistence}: \xmlDesc{integer},
          provides the number of consecutive times convergence should be reached before a trajectory
          is considered fully converged. This helps in preventing early false convergence.

        \item \xmlNode{constraintExplorationLimit}: \xmlDesc{integer},
          provides the number of consecutive times a functional constraint boundary can be explored
          for an acceptable sampling point before aborting search. Only apples if using a
          \xmlNode{Constraint}. \default{500}
      \end{itemize}

    \item \xmlNode{constant}: \xmlDesc{comma-separated strings, integers, and floats},
      allows variables that do not change value to be part of the input space.
      The \xmlNode{constant} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{name}: \xmlDesc{string, required},
            variable name for this constant, which will be provided to the Model.
          \item \xmlAttr{shape}: \xmlDesc{comma-separated integers, optional},
            determines the shape of samples of the constant value.               For example,
            \xmlAttr{shape}=``2,3'' will shape the values into a 2 by 3               matrix, while
            \xmlAttr{shape}=``10'' will shape into a vector of 10 values.               Unlike the
            \xmlNode{variable}, the constant requires each value be entered; the number
            of required values is equal to the product of the \xmlAttr{shape} values, e.g. 6 entries
            for shape ``2,3'').               \nb A model interface must be prepared to handle non-
            scalar inputs to use this option.
          \item \xmlAttr{source}: \xmlDesc{string, optional},
            the name of the DataObject containing the value to be used for this constant.
            Requires \xmlNode{ConstantSource} node with a \xmlNode{DataObject} identified for this
            Sampler/Optimizer.
          \item \xmlAttr{index}: \xmlDesc{integer, optional},
            the index of the realization in the \xmlNode{ConstantSource} \xmlNode{DataObject}
            containing the value for this constant. Requires \xmlNode{ConstantSource} node with
            a \xmlNode{DataObject} identified for this Sampler/Optimizer.
      \end{itemize}

    \item \xmlNode{ConstantSource}: \xmlDesc{string},
      identifies a \xmlNode{DataObject} to provide \xmlNode{constant} values to the input
      space of this entity while sampling. As an alternative to providing predefined values
      for constants, the \xmlNode{ConstantSource} provides a dynamic means of always providing
      the same value for a constant. This is often used as part of a larger multi-workflow
      calculation.
      The \xmlNode{ConstantSource} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, optional},
            The RAVEN class for this source. Options include \xmlString{DataObject}.
          \item \xmlAttr{type}: \xmlDesc{string, optional},
            The RAVEN type for this source. Options include any valid \xmlNode{DataObject} type,
            such as HistorySet or PointSet.
      \end{itemize}

    \item \xmlNode{Constraint}: \xmlDesc{string},
      name of \xmlNode{Function} which contains explicit constraints for the sampling of
      the input space of the Model. From a practical point of view, this XML node must contain
      the name of a function defined in the \xmlNode{Functions} block (see
      Section~\ref{sec:functions}).               This external function must contain a method
      called ``constrain'', which returns True for               inputs satisfying the explicit
      constraints and False otherwise.
      The \xmlNode{Constraint} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this source. Options include \xmlString{Functions}.
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this source. Options include \xmlNode{External}.
      \end{itemize}

    \item \xmlNode{ImplicitConstraint}: \xmlDesc{string},
      name of \xmlNode{Function} which contains implicit constraints of the Model. From a practical
      point of view, this XML node must contain the name of a function defined in the
      \xmlNode{Functions}               block (see Section~\ref{sec:functions}). This external
      function must contain a method called               ``implicitConstrain'', which returns True
      for outputs satisfying the implicit constraints and False otherwise.
      The \xmlNode{ImplicitConstraint} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this source. Options include \xmlString{Functions}.
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this source. Options include \xmlNode{External}.
      \end{itemize}

    \item \xmlNode{Sampler}: \xmlDesc{string},
      name of a Sampler that can be used to initialize the starting points for the trajectories
      of some of the variables. From a practical point of view, this XML node must contain the
      name of a Sampler defined in the \xmlNode{Samplers} block (see
      Section~\ref{subsec:onceThroughSamplers}).               The Sampler will be used to
      initialize the trajectories' initial points for some or all               of the variables.
      For example, if the Sampler selected samples only 2 of the 5 optimization
      variables, the \xmlNode{initial} XML node is required only for the remaining 3 variables.
      The \xmlNode{Sampler} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this entity (e.g. Samplers, Models, DataObjects)
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this entity; a subtype of the class (e.g. MonteCarlo, Code, PointSet)
      \end{itemize}

    \item \xmlNode{Restart}: \xmlDesc{string},
      name of a DataObject. Used to leverage existing data when sampling a model. For
      example, if a Model has               already been sampled, but some samples were not
      collected, the successful samples can               be stored and used instead of rerunning
      the model for those specific samples. This RAVEN               entity definition must be a
      DataObject with contents including the input and output spaces               of the Model
      being sampled.
      The \xmlNode{Restart} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, optional},
            The RAVEN class for this source. Options include \xmlString{DataObject}.
          \item \xmlAttr{type}: \xmlDesc{string, optional},
            The RAVEN type for this source. Options include any valid \xmlNode{DataObject} type,
            such as HistorySet or PointSet.
      \end{itemize}

    \item \xmlNode{restartTolerance}: \xmlDesc{float},
      specifies how strictly a matching point from a \xmlNode{Restart} DataObject must match
      the desired sample point in order to be used. If a potential restart point is within a
      relative Euclidean distance (as specified by the value in this node) of a desired sample
      point,               the restart point will be used instead of sampling the Model.
      \default{1e-15}

    \item \xmlNode{variablesTransformation}:
      Allows transformation of variables via translation matrices. This defines two spaces,
      a ``latent'' transformed space sampled by RAVEN and a ``manifest'' original space understood
      by the Model.
      The \xmlNode{variablesTransformation} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{distribution}: \xmlDesc{string, optional},
            the name for the distribution defined in the XML node \xmlNode{Distributions}.
            This attribute indicates the values of \xmlNode{manifestVariables} are drawn from
            \xmlAttr{distribution}.
      \end{itemize}

      The \xmlNode{variablesTransformation} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{latentVariables}: \xmlDesc{comma-separated strings},
          user-defined latent variables that are used for the variables transformation.
          All the variables listed under this node should be also mentioned in \xmlNode{variable}.

        \item \xmlNode{manifestVariables}: \xmlDesc{comma-separated strings},
          user-defined manifest variables that can be used by the \xmlNode{Model}.

        \item \xmlNode{manifestVariablesIndex}: \xmlDesc{comma-separated strings},
          user-defined manifest variables indices paired with \xmlNode{manifestVariables}.
          These indices indicate the position of manifest variables associated with multivariate
          normal               distribution defined in the XML node \xmlNode{Distributions}.
          The indices should be postive integer. If not provided, the code will use the positions
          of manifest variables listed in \xmlNode{manifestVariables} as the indices.

        \item \xmlNode{method}: \xmlDesc{string},
          the method that is used for the variables transformation. The currently available method
          is \xmlString{pca}.
      \end{itemize}
  \end{itemize}

\hspace{24pt}
Gradient Descent Example:
\begin{lstlisting}[style=XML]
<Optimizers>
  ...
  <GradientDescent name="opter">
    <objective>ans</objective>
    <variable name="x">
      <distribution>x_dist</distribution>
      <initial>-2</initial>
    </variable>
    <variable name="y">
      <distribution>y_dist</distribution>
      <initial>2</initial>
    </variable>
    <samplerInit>
      <limit>100</limit>
    </samplerInit>
    <gradient>
      <FiniteDifference/>
    </gradient>
    <stepSize>
      <GradientHistory/>
    </stepSize>
    <acceptance>
      <Strict/>
    </acceptance>
    <TargetEvaluation class="DataObjects" type="PointSet">optOut</TargetEvaluation>
  </GradientDescent>
  ...
</Optimizers>
\end{lstlisting}

\subsection{SimulatedAnnealing}
  The \xmlNode{SimulatedAnnealing} optimizer is a metaheuristic approach
  to perform a global search in large design spaces. The methodology rose
  from statistical physics and was inspitred by metallurgy where                             it was
  found that fast cooling might lead to smaller and defected crystals,
  and that reheating and slowly controling cooling will lead to better states.
  This allows climbing to avoid being stuck in local minima and hence facilitates
  finding the global minima for non-convex probloems.                             More information
  can be found in: Kirkpatrick, S.; Gelatt Jr, C. D.; Vecchi, M. P. (1983).
  ``Optimization by Simulated Annealing". Science. 220 (4598): 671–680.
\vspace{7pt} \\When used as part of a \xmlNode{MultiRun} step, this entity provides
        additional information through the \xmlNode{SolutionExport} DataObject. The
        following variables can be requested within the \xmlNode{SolutionExport}:
        \begin{itemize}
          \item \texttt{trajID}: integer identifier for different optimization starting locations and paths
             \item \texttt{iteration}: integer identifying which iteration (or step, or generation) a trajectory is on
             \item \texttt{accepted}: string acceptance status of the potential optimal point (algorithm dependent)
             \item \texttt{rejectReason}: discription of reject reason, 'noImprovement' means rejected the new optimization point for no improvement from last point, 'implicitConstraintsViolation' means rejected by implicit constraints violation, return None if the point is accepted
             \item \texttt{\{VAR\}}: any variable from the \xmlNode{TargetEvaluation} input or output; gives the value of that variable at the optimal candidate for this iteration.
             \item \texttt{conv\_\{CONV\}}: status of each given convergence criteria
             \item \texttt{amp\_\{VAR\}}: amplitued associated to each variable used to compute step size based on cooling method and the corresponding next neighbour
             \item \texttt{delta\_\{VAR\}}: step size associated to each variable
             \item \texttt{Temp}: temperature at current state
             \item \texttt{fraction}: current fraction of the max iteration limit

         \end{itemize}

  The \xmlNode{SimulatedAnnealing} node recognizes the following parameters:
    \begin{itemize}
      \item \xmlAttr{verbosity}: \xmlDesc{[silent, quiet, all, debug], optional},
        Desired verbosity of messages coming from this entity
      \item \xmlAttr{name}: \xmlDesc{string, required},
        User-defined name to designate this entity in the RAVEN input file.
  \end{itemize}

  The \xmlNode{SimulatedAnnealing} node recognizes the following subnodes:
  \begin{itemize}
    \item \xmlNode{objective}: \xmlDesc{string},
      Name of the response variable (or ``objective function'') that should be optimized
      (minimized or maximized).

    \item \xmlNode{variable}:
      defines the input space variables to be sampled through various means.
      The \xmlNode{variable} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{name}: \xmlDesc{string, optional},
            user-defined name of this Sampler. \nb As for the other objects,               this is
            the name that can be used to refer to this specific entity from other input blocks
          \item \xmlAttr{shape}: \xmlDesc{comma-separated integers, optional},
            determines the number of samples and shape of samples               to be taken.  For
            example, \xmlAttr{shape}=``2,3'' will provide a 2 by 3               matrix of values,
            while \xmlAttr{shape}=``10'' will produce a vector of 10 values.               Omitting
            this optional attribute will result in a single scalar value instead.               Each
            of the values in the matrix or vector will be the same as the single sampled value.
            \nb A model interface must be prepared to handle non-scalar inputs to use this option.
      \end{itemize}

      The \xmlNode{variable} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{distribution}: \xmlDesc{string},
          name of the distribution that is associated to this variable.               Its name needs
          to be contained in the \xmlNode{Distributions} block explained               in Section
          \ref{sec:distributions}. In addition, if NDDistribution is used,               the
          attribute \xmlAttr{dim} is required. \nb{Alternatively, this node must be omitted
          if the \xmlNode{function} node is supplied.}
          The \xmlNode{distribution} node recognizes the following parameters:
            \begin{itemize}
              \item \xmlAttr{dim}: \xmlDesc{integer, optional},
                for an NDDistribution, indicates the dimension within the NDDistribution that
                corresponds               to this variable.
          \end{itemize}

        \item \xmlNode{function}: \xmlDesc{string},
          name of the function that               defines the calculation of this variable from
          other distributed variables.  Its name               needs to be contained in the
          \xmlNode{Functions} block explained in Section               \ref{sec:functions}. This
          function must implement a method named ``evaluate''.               \nb{Each
          \xmlNode{variable} must contain only one \xmlNode{Function} or
          \xmlNode{Distribution}, but not both.}

        \item \xmlNode{initial}: \xmlDesc{comma-separated floats},
          indicates the initial values where independent trajectories for this optimization
          effort should begin. The number of entries should be the same for all variables, unless
          a variable is initialized with a sampler (see \xmlNode{samplerInit} below). Note these
          entries are ordered; that is, if the optimization variables are $x$ and $y$, and the
          initial               values for $x$ are \xmlString{1, 2, 3, 4} and initial values for $y$
          are \xmlString{5, 6, 7, 8},               then there will be four starting trajectories
          beginning at the locations (1, 5), (2, 6),               (3, 7), and (4, 8).
      \end{itemize}

    \item \xmlNode{TargetEvaluation}: \xmlDesc{string},
      name of the DataObject where the sampled outputs of the Model will be collected.
      This DataObject is the means by which the sampling entity obtains the results of requested
      samples, and so should require all the input and output variables needed for adaptive
      sampling.
      The \xmlNode{TargetEvaluation} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this entity (e.g. Samplers, Models, DataObjects)
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this entity; a subtype of the class (e.g. MonteCarlo, Code, PointSet)
      \end{itemize}

    \item \xmlNode{samplerInit}:
      collection of nodes that describe the initialization of the optimization algorithm.

      The \xmlNode{samplerInit} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{limit}: \xmlDesc{integer},
          limits the number of Model evaluations that may be performed as part of this optimization.
          For example, a limit of 100 means at most 100 total Model evaluations may be performed.

        \item \xmlNode{writeSteps}: \xmlDesc{[final, every]},
          delineates when the \xmlNode{SolutionExport} DataObject should be written to. In case
          of \xmlString{final}, only the final optimal solution for each trajectory will be written.
          In case of \xmlString{every}, the \xmlNode{SolutionExport} will be updated with each
          iteration               of the Optimizer.

        \item \xmlNode{initialSeed}: \xmlDesc{integer},
          seed for random number generation. Note that by default RAVEN uses an internal seed,
          so this seed must be changed to observe changed behavior. \default{RAVEN-determined}

        \item \xmlNode{type}: \xmlDesc{[min, max]},
          the type of optimization to perform. \xmlString{min} will search for the lowest
          \xmlNode{objective} value, while \xmlString{max} will search for the highest value.
      \end{itemize}

    \item \xmlNode{convergence}:
      a node containing the desired convergence criteria for the optimization algorithm.
      Note that convergence is met when any one of the convergence criteria is met. If no
      convergence               criteria are given, then the defaults are used.

      The \xmlNode{convergence} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{objective}: \xmlDesc{float},
          provides the desired value for the convergence criterion of the objective function
          ($\epsilon^{obj}$), i.e., convergence is reached when: $$ |newObjevtive - oldObjective|
          \le \epsilon^{obj}$$.                        \default{1e-6}, if no criteria specified

        \item \xmlNode{temperature}: \xmlDesc{float},
          provides the desired value for the convergence creiteron of the system temperature,
          ($\epsilon^{temp}$), i.e., convergence is reached when: $$T \le \epsilon^{temp}$$.
          \default{1e-10}, if no criteria specified

        \item \xmlNode{persistence}: \xmlDesc{integer},
          provides the number of consecutive times convergence should be reached before a trajectory
          is considered fully converged. This helps in preventing early false convergence.
      \end{itemize}

    \item \xmlNode{coolingSchedule}:
      The function governing the cooling process. Currently, user can select
      between,\xmlString{exponential},                  \xmlString{cauchy},
      \xmlString{boltzmann},or \xmlString{veryfast}.\\ \\In case of \xmlString{exponential} is
      provided, The cooling process will be governed by: $$ T^{k} = T^0 * \alpha^k$$
      In case of \xmlString{boltzmann} is provided, The cooling process will be governed by: $$
      T^{k} = \frac{T^0}{log(k + d)}$$                  In case of \xmlString{cauchy} is provided,
      The cooling process will be governed by: $$ T^{k} = \frac{T^0}{k + d}$$In case of
      \xmlString{veryfast} is provided, The cooling process will be governed by: $$ T^{k} =  T^0 *
      \exp(-ck^{1/D}),$$                  where $D$ is the dimentionality of the problem (i.e.,
      number of optimized variables), $k$ is the number of the current iteration
      $T^{0} = \max{(0.01,1-\frac{k}{\xmlNode{limit}})}$ is the initial temperature, and $T^{k}$ is
      the current temperature                  according to the specified cooling schedule.
      \default{exponential}.

      The \xmlNode{coolingSchedule} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{exponential}: \xmlDesc{string},
          exponential cooling schedule

          The \xmlNode{exponential} node recognizes the following subnodes:
          \begin{itemize}
            \item \xmlNode{alpha}: \xmlDesc{float},
              slowing down constant, should be between 0,1 and preferable very close to 1.
              \default{0.94}
          \end{itemize}

        \item \xmlNode{veryfast}: \xmlDesc{string},
          veryfast cooling schedule

          The \xmlNode{veryfast} node recognizes the following subnodes:
          \begin{itemize}
            \item \xmlNode{c}: \xmlDesc{float},
              decay constant, \default{1.0}
          \end{itemize}

        \item \xmlNode{cauchy}: \xmlDesc{string},
          cauchy cooling schedule

          The \xmlNode{cauchy} node recognizes the following subnodes:
          \begin{itemize}
            \item \xmlNode{d}: \xmlDesc{float},
              bias, \default{1.0}
          \end{itemize}

        \item \xmlNode{boltzmann}: \xmlDesc{string},
          boltzmann cooling schedule

          The \xmlNode{boltzmann} node recognizes the following subnodes:
          \begin{itemize}
            \item \xmlNode{d}: \xmlDesc{float},
              bias, \default{1.0}
          \end{itemize}
      \end{itemize}

    \item \xmlNode{constant}: \xmlDesc{comma-separated strings, integers, and floats},
      allows variables that do not change value to be part of the input space.
      The \xmlNode{constant} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{name}: \xmlDesc{string, required},
            variable name for this constant, which will be provided to the Model.
          \item \xmlAttr{shape}: \xmlDesc{comma-separated integers, optional},
            determines the shape of samples of the constant value.               For example,
            \xmlAttr{shape}=``2,3'' will shape the values into a 2 by 3               matrix, while
            \xmlAttr{shape}=``10'' will shape into a vector of 10 values.               Unlike the
            \xmlNode{variable}, the constant requires each value be entered; the number
            of required values is equal to the product of the \xmlAttr{shape} values, e.g. 6 entries
            for shape ``2,3'').               \nb A model interface must be prepared to handle non-
            scalar inputs to use this option.
          \item \xmlAttr{source}: \xmlDesc{string, optional},
            the name of the DataObject containing the value to be used for this constant.
            Requires \xmlNode{ConstantSource} node with a \xmlNode{DataObject} identified for this
            Sampler/Optimizer.
          \item \xmlAttr{index}: \xmlDesc{integer, optional},
            the index of the realization in the \xmlNode{ConstantSource} \xmlNode{DataObject}
            containing the value for this constant. Requires \xmlNode{ConstantSource} node with
            a \xmlNode{DataObject} identified for this Sampler/Optimizer.
      \end{itemize}

    \item \xmlNode{ConstantSource}: \xmlDesc{string},
      identifies a \xmlNode{DataObject} to provide \xmlNode{constant} values to the input
      space of this entity while sampling. As an alternative to providing predefined values
      for constants, the \xmlNode{ConstantSource} provides a dynamic means of always providing
      the same value for a constant. This is often used as part of a larger multi-workflow
      calculation.
      The \xmlNode{ConstantSource} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, optional},
            The RAVEN class for this source. Options include \xmlString{DataObject}.
          \item \xmlAttr{type}: \xmlDesc{string, optional},
            The RAVEN type for this source. Options include any valid \xmlNode{DataObject} type,
            such as HistorySet or PointSet.
      \end{itemize}

    \item \xmlNode{Constraint}: \xmlDesc{string},
      name of \xmlNode{Function} which contains explicit constraints for the sampling of
      the input space of the Model. From a practical point of view, this XML node must contain
      the name of a function defined in the \xmlNode{Functions} block (see
      Section~\ref{sec:functions}).               This external function must contain a method
      called ``constrain'', which returns True for               inputs satisfying the explicit
      constraints and False otherwise.
      The \xmlNode{Constraint} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this source. Options include \xmlString{Functions}.
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this source. Options include \xmlNode{External}.
      \end{itemize}

    \item \xmlNode{ImplicitConstraint}: \xmlDesc{string},
      name of \xmlNode{Function} which contains implicit constraints of the Model. From a practical
      point of view, this XML node must contain the name of a function defined in the
      \xmlNode{Functions}               block (see Section~\ref{sec:functions}). This external
      function must contain a method called               ``implicitConstrain'', which returns True
      for outputs satisfying the implicit constraints and False otherwise.
      The \xmlNode{ImplicitConstraint} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this source. Options include \xmlString{Functions}.
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this source. Options include \xmlNode{External}.
      \end{itemize}

    \item \xmlNode{Sampler}: \xmlDesc{string},
      name of a Sampler that can be used to initialize the starting points for the trajectories
      of some of the variables. From a practical point of view, this XML node must contain the
      name of a Sampler defined in the \xmlNode{Samplers} block (see
      Section~\ref{subsec:onceThroughSamplers}).               The Sampler will be used to
      initialize the trajectories' initial points for some or all               of the variables.
      For example, if the Sampler selected samples only 2 of the 5 optimization
      variables, the \xmlNode{initial} XML node is required only for the remaining 3 variables.
      The \xmlNode{Sampler} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, required},
            RAVEN class for this entity (e.g. Samplers, Models, DataObjects)
          \item \xmlAttr{type}: \xmlDesc{string, required},
            RAVEN type for this entity; a subtype of the class (e.g. MonteCarlo, Code, PointSet)
      \end{itemize}

    \item \xmlNode{Restart}: \xmlDesc{string},
      name of a DataObject. Used to leverage existing data when sampling a model. For
      example, if a Model has               already been sampled, but some samples were not
      collected, the successful samples can               be stored and used instead of rerunning
      the model for those specific samples. This RAVEN               entity definition must be a
      DataObject with contents including the input and output spaces               of the Model
      being sampled.
      The \xmlNode{Restart} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{class}: \xmlDesc{string, optional},
            The RAVEN class for this source. Options include \xmlString{DataObject}.
          \item \xmlAttr{type}: \xmlDesc{string, optional},
            The RAVEN type for this source. Options include any valid \xmlNode{DataObject} type,
            such as HistorySet or PointSet.
      \end{itemize}

    \item \xmlNode{restartTolerance}: \xmlDesc{float},
      specifies how strictly a matching point from a \xmlNode{Restart} DataObject must match
      the desired sample point in order to be used. If a potential restart point is within a
      relative Euclidean distance (as specified by the value in this node) of a desired sample
      point,               the restart point will be used instead of sampling the Model.
      \default{1e-15}

    \item \xmlNode{variablesTransformation}:
      Allows transformation of variables via translation matrices. This defines two spaces,
      a ``latent'' transformed space sampled by RAVEN and a ``manifest'' original space understood
      by the Model.
      The \xmlNode{variablesTransformation} node recognizes the following parameters:
        \begin{itemize}
          \item \xmlAttr{distribution}: \xmlDesc{string, optional},
            the name for the distribution defined in the XML node \xmlNode{Distributions}.
            This attribute indicates the values of \xmlNode{manifestVariables} are drawn from
            \xmlAttr{distribution}.
      \end{itemize}

      The \xmlNode{variablesTransformation} node recognizes the following subnodes:
      \begin{itemize}
        \item \xmlNode{latentVariables}: \xmlDesc{comma-separated strings},
          user-defined latent variables that are used for the variables transformation.
          All the variables listed under this node should be also mentioned in \xmlNode{variable}.

        \item \xmlNode{manifestVariables}: \xmlDesc{comma-separated strings},
          user-defined manifest variables that can be used by the \xmlNode{Model}.

        \item \xmlNode{manifestVariablesIndex}: \xmlDesc{comma-separated strings},
          user-defined manifest variables indices paired with \xmlNode{manifestVariables}.
          These indices indicate the position of manifest variables associated with multivariate
          normal               distribution defined in the XML node \xmlNode{Distributions}.
          The indices should be postive integer. If not provided, the code will use the positions
          of manifest variables listed in \xmlNode{manifestVariables} as the indices.

        \item \xmlNode{method}: \xmlDesc{string},
          the method that is used for the variables transformation. The currently available method
          is \xmlString{pca}.
      \end{itemize}
  \end{itemize}

\hspace{24pt}
Simulated Annealing Example:
\begin{lstlisting}[style=XML]
  <Optimizers>
    ...
    <SimulatedAnnealing name="simOpt">
      <samplerInit>
        <limit>2000</limit>
        <initialSeed>42</initialSeed>
        <writeSteps>every</writeSteps>
        <type>min</type>
      </samplerInit>
      <convergence>
        <objective>1e-6</objective>
        <temperature>1e-20</temperature>
        <persistence>1</persistence>
      </convergence>
      <coolingSchedule>
        <exponential>
          <alpha>0.94</alpha>
        </exponential>
      </coolingSchedule>
      <variable name="x">
        <distribution>beale_dist</distribution>
        <initial>-2.5</initial>
      </variable>
      <variable name="y">
        <distribution>beale_dist</distribution>
        <initial>3.5</initial>
      </variable>
      <objective>ans</objective>
      <TargetEvaluation class="DataObjects" type="PointSet">optOut</TargetEvaluation>
    </SimulatedAnnealing>
    ...
  </Optimizers>
\end{lstlisting}