\documentclass{article}
\usepackage{siunitx}
\usepackage{amsmath}
\title{Cardano's Formula for Cubic Equation}
\author{Ariyibi Joseph Iseoluwa}
\begin{document}
	\maketitle
	\begin{center}
		\textbf{ABSTRACT}
	\end{center}
Gerolamo Cardano was born in Pavia in 1501 as the illegitimate child of a jurist.He attended the University of Padua and become a physician in the town of Sacco, after being rejected by his home town of Milan. He became one of the most famous doctors in all of Europe, having treated the Pope. He was also an astrologerr and an avid gambler, to which he wrote the Book on Games of Chance, which was the first serious treatise on the mathematics of probability.\cite{ Ariyibi2021}
\section{Introdction To Cardano's Formula}
Cardano's formula for solution of cubic equations for an equation like: $x^3 + a_1x^2 + a_2x + a^3 = 0$
the parameters Q, R, S and T can be computed thus,\\
\raggedright
	\begin{align}
		\textbf{Q}= 3a_2 - \frac{a_1^2}{a} &                       \textbf{R}= \frac{9a_1a_2-27a_3-2a_1^3}{54} \\
		\textbf{S} =\sqrt[3]{R + \sqrt{-Q^3 + R^2}} &
		 \textbf{T} = \sqrt{R - \sqrt{Q^3 + R^2}} \\
	\end{align}
To give the roots
$x_1= S + T-\frac{1}{3}a_1$
$x_2= \frac{-(S+T)}{2} - \frac{a_1}{3} + i\frac{\sqrt{3}(s-T)}{2}$ \\
$x_2= \frac{-(S+T)}{2} - \frac{a_1}{3} - i\frac{\sqrt{3}(s-T)}{2}$ \\

Note : $x^3$ must not have a co-efficient
\subsection{Some Examples}
\begin{itemize}
	\item  $x^3 - 3x^2 + 4 = 0$
	\item $2x^3 + 6x^2 + 1 = 0$
\end{itemize}
\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
	\DeclareGraphicsExtensions{.png, .jpeg, .pgm}
\usepackage{caption}
% \usepackage{subcaption}
\usepackage{csvsimple}
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\title{PA03: Mandelbrot Image\\CS791v: Parallel Computing}
\author{Terence Henriod}
\date{\today}

\begin{document}

\clearpage
\maketitle
\thispagestyle{empty} % removes the page number from the title page

\begin{abstract}
The Mandelbrot set is a well known fractal among math enthusiasts. Computation of Mandelbrot sets/images is known to programmers to be an ``embarrassingly parallel" task. The task is somewhat compute heavy, and the only communication required (if any) is to collect the results of the computaion. In this assignment, we explore computation of Mandelbrot images on the GPU.
\end{abstract}

\newpage
\section{Introduction}
Computation of Mandelbrot images is ``embarassingly parallel" due to the fact that pixel values can be generated without communication, they can be computed independent of one another, and the only communication required through the whole process is to collect the computed pixel values in one location once the computation completes.

\section{Theory}
\subsection{Computing Pixel Values}
Computing a pixel value follows the flowing equation: $z_{k} = z_{k - 1}^{2} + c$ where $z$ is the pixel value, and $c$ is the coordinate of the pixel in the complex plane. This formula is computed iteratively until either a maximum number of iterations has been reached (of note is the number $1024$ for this exercise) or until $z$ begins to converge to zero or diverge to $\infty$ (which is know to happen if $z$ ever reaches $2$).

\subsection{Sequential Algorithm}
This algorithm is straightforward. The simple use of a for loop (or pair of nested for loops) to visit the storage location for each pixel's value is used, and at each location the pixel's value is located. The value of the pixel is dependent on its coordinates in the image, but these coordinates can be computed easily from the overall array index and the width of the image. That is, the coordinates $x, y$ can be computed from the main data array index $i$ and the image width $w$ by:
$$x = i / w$$
$$y = i \% w$$

\subsection{Parallel GPU Algorithm}
The GPU algorithm is also straightforward. Each GPU thread uses the same algorithm with one small difference: instead of incrementing the location of the next pixel to compute by one, each GPU thread strides across the data array in steps of the number of total threads.

I was curious about doing a version where a sort of ``work queue" would be implemented using an integer in the shared block memory indicating the next pixel to be computed's coordinate(s) and atomic operations to increment the shared value. The premise of this idea was that different pixels take different amounts of time to compute, so we could better balance the load on the threads using the work queue. Unfortunately, I ran out of time/motivation to experiment with this idea. It would indeed require more effort to implement, but I think that for very large images it might have its merits.

\newpage
\section{Results}
The performance results of the vector reduction implementation are listed here.

\subsection{Information on the GPU device used}
\csvautotabular{gpu_properties.csv}

\newpage
\subsection{Performance Graphs}
  \begin{figure}[h!]
    \centering
    \includegraphics[width=.9\linewidth]{runtime}
    \caption{A comparison of the runtimes for computing a 2000x2000 Mandelbrot image using 1024 iterations at most for each pixel.}
    \label{fig:runtime}
  \end{figure}

  \begin{figure}[h!]
    \centering
    \includegraphics[width=.9\linewidth]{throughput}
    \caption{A comparison of the throughput in integer pixel values produced per second.}
    \label{fig:throughput}
  \end{figure}

  \begin{figure}[h!]
    \centering
    \includegraphics[width=.9\linewidth]{compute_speedup}
    \caption{The speedup achieved. Note the peak in the surface at 64 threads for all block sizes.}
    \label{fig:compute_speedup}
  \end{figure}

  \begin{figure}[h!]
    \centering
    \includegraphics[width=.9\linewidth]{total_speedup}
    \caption{The speedup achieved when also considering the time to copy the computed data from the GPU to the Host.}
    \label{fig:total_speedup}
  \end{figure}

  \begin{figure}[h!]
    \centering
    \includegraphics[width=.7\linewidth]{testimage}
    \caption{A Mandelbrot image for verification.}
    \label{fig:total_speedup}
  \end{figure}

\newpage
\section{Discussion}
\subsection{Low Speedups}
The speedups were not as high as I had thought they would be. I believe that this is likely due to my use of a weak GPU - specifically the one that is in my laptop. I have little reason to believe that the lack of speedup was due to a mismatch of algorithms because the sequential and the parallel algorithms are so similar, literally the only difference is the increment factor in the for loop. It is not impressive, but I was able to write CUDA code and do the explorations for the assignment.

\subsection{The Right Number of Threads/Blocks (or The Right ``Stride")}
The highest speedup (for the 2000 x 2000 image) was observed using $255$ blocks $64$ and threads per block to produce a speedup of approximately $6.64$. Looking at the speedup surface graphs do seem to indicate that the number of threads per block was more important than the number of total blocks used. I believe that this is likely because Mandelbrot computation has a high computation to memory access ratio. It is likely that since the threads only need to make $1$ global memory access at the end of their computation, so it is better to keep the number of threads low so that they can spend their time performing the computations.

\subsection{Lack of Transfer Time}
Unlike with many other processes that are parallelized on the GPU, memory transfer time was almost negligible for this task. Only one transfer needs to occur since GPU threads ca figure everything about a pixel out based on which index they are processing

\section{Issues}
\subsection{Weak Hardware Setup}
The GPU used was a medium caliber laptop card. This resulted in limited performance, memory sizes, thread/block counts, etc. In addition, the Windows operating system was used, meaning that all GPU operations needed to occur within $2$ seconds. This limited My data collection from using anything less than 32 threads per block and be able to have consistent success (16 threads worked often, but not often enough). However, I believe enough data was gathered to be able to comprehensively address the topic.

\end{document}


\documentclass{article}

\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{alltt}
\usepackage[numbers]{natbib}
\usepackage[total={5.8in, 9in},top=1in,left=1.3in]{geometry}
\usepackage{fancyhdr}
\usepackage{hyperref}
\usepackage{enumerate}

\fancypagestyle{plain}{%
  \renewcommand{\headrulewidth}{0.001pt}%
  \fancyhf{}%
  \fancyfoot[C]{\footnotesize Page \thepage\ of \pageref{LastPage}}%
}

\lhead{\small Carl-Johan Thore}
\rhead{\small fminsdp - Optimization with matrix inequality constraints}
\pagestyle{fancyplain}

\newcommand{\bm}[1]{\mbox{\boldmath $#1$}}
\newcommand{\T}{\textsf{T}}
\newcommand{\svec}{\mbox{\textsf{svec}}}

\renewcommand{\headrulewidth}{0.002pt}

\hypersetup{bookmarksopen=true}

\title{FMINSDP -- a code for solving optimization problems with matrix inequality constraints \vskip 2mm
\footnotesize{\url{https://se.mathworks.com/matlabcentral/fileexchange/43643-fminsdp}} \normalsize}
\date{\today}
\author{Carl-Johan Thore}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%s

\begin{document}

\maketitle

\thispagestyle{empty}

\noindent This document is a theoretical and practical introduction to the Matlab-code \texttt{fminsdp}, designed to find local solutions to non-linear, non-convex optimization problems (NLPs) with both scalar constraints and (small-size) matrix inequality constraints.
\vskip 2mm
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\noindent \textbf{Notation.} A matrix $\bm{A} \in \mathbb{R}^{m \times m}$ is said to be \textit{positive semi-definite} if
$\bm{y}^{\T}\bm{A}\bm{y} \geq 0$ for all $\bm{y} \in \mathbb{R}^{m}$. It is convenient to introduce
the notation "$\bm{A} \succeq \bm{0}$" to indicate that $\bm{A}$ is positive semi-definite. A \textit{matrix inequality} is here
defined as an expression of the form
\begin{equation}\label{eq:matrix_ineq}
\bm{\mathcal{A}}(\bm{x}) \succeq \bm{0},
\end{equation}
where $\bm{\mathcal{A}}$ is a map from $\Omega \subset \mathbb{R}^{n}$ to the space $\mathbb{S}^{m}$ of symmetric, real-valued matrices of size $m \times m$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The optimization problem}

\texttt{fminsdp} attempts to find a local solution to non-linear, non-convex optimization problems of the form
\begin{equation}\label{eq:problem}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{n}}{\mbox{minimize}} \;\; f(\bm{x})  \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
			& \bm{A}_{eq}\bm{x} = \bm{b}_{eq} & \mbox{linear equality constraints}      \\
			& \bm{A}\bm{x} \leq \bm{b}	      & \mbox{linear inequality constraints} \\
			& \bm{c}_{eq}(\bm{x}) = \bm{0}	 	& \mbox{nonlinear equality constraints}			\\
		  & \bm{c}(\bm{x}) \leq \bm{0}			& \mbox{nonlinear inequality constraints}  \\
			& \bm{l} \leq \bm{x} \leq \bm{u}	& \mbox{box constraints}\\
			& \bm{\mathcal{A}}_{i}(\bm{x}) \succeq \bm{0}, \quad i = 1,...,q & \mbox{matrix inequality constraints},
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
\vskip 2mm
\noindent where $\bm{A}_{eq}$, $\bm{A}$, $\bm{b}_{eq}$, $\bm{b}$, $\bm{l}$ and $\bm{u}$ are constant matrices and vectors, respectively. The functions $f$, $\bm{c}$,  $\bm{c}_{eq}$ and $\bm{\mathcal{A}}_{i}:\Omega_{i} \rightarrow \mathbb{S}^{m_{i}}$, $i = 1,...,q$, can be non-linear and are (preferably) at least
twice continuously differentiable. Users familiar with \texttt{fmincon} from the Optimization Toolbox in Matlab \cite{fmincon:60} should recognize the
form of problem \eqref{eq:problem} --- the novelty here is the addition of $q$ matrix inequality constraints.

\texttt{fminsdp} offers three different methods to treat problem \eqref{eq:problem}: the ''cholesky-method'', the ''ldl-method'' \cite{Bogani:2009} and ''penlab'' \cite{Fiala:2013}. The first two methods works by reformulating the problem into a standard NLP which can be solved by any of the NLP-solvers interfaced by \texttt{fminsdp}. The third method relies on the external solver PENLAB\footnote{Downloadable from \url{http://web.mat.bham.ac.uk/kocvara/penlab/}.}, which treats the problem directly using an augmented Lagrangian-type algorithm \cite{Fiala:2013}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Theoretical background}

\subsection{The cholesky-method}

In the cholesky-method, \texttt{fminsdp} reformulates the matrix inequality constraints into scalar equality constraints using the fact that a matrix is positive semi-definite if and only if it admits a Cholesky decomposition. In other words,
\vskip 2mm
\noindent \textbf{Theorem 1.} A symmetric matrix $\bm{A} \succeq \bm{0}$ if and only if
\begin{equation}\nonumber
\bm{A} = \bm{L}\bm{L}^{\T}
\end{equation}
for some lower triangular matrix $\bm{L}$.
\vskip 2mm
\noindent By default in \texttt{fminsdp}, $\bm{L}$ is restricted to the space of lower triangular $m \times m$-matrices with non-negative diagonal elements, referred to as $\mathcal{L}^{m}$. Non-negativity of the diagonal elements is not required by the theory, but enforcement of this condition is often important for computational efficiency.

\texttt{fminsdp} makes use of the function $\svec : \mathbb{R}^{m\times m} \rightarrow \mathbb{R}^{p}$, where $p$ denotes the number of  non-zero elements in the Cholesky factor $\bm{L}$ of a given matrix; i.e., the number of elements in the sparsity pattern of $\bm{L}$, obtained by a \textit{symbolic} Cholesky factorization of the given matrix. $\svec$ takes, column-wise, the elements of the input matrix corresponding to potential non-zeros of $\bm{L}$ and stacks them on top of each other to form a vector of length $p$. If the lower triangular part of $\bm{L}\in\mathcal{L}^{m}$ is full we have $p=m(m+1)/2$ and
\begin{equation}\nonumber
\svec(\bm{A}) = (A_{11},A_{21},\ldots,A_{m1},A_{22},\ldots,A_{m2},\ldots,A_{mm})^{\T};
\end{equation}
i.e., the vector $\svec(\bm{A})$ contains the elements of the lower triangular part of $\bm{A}$.

Now, based on Theorem 1, problem \eqref{eq:problem} is reformulated into the following problem:
\begin{equation}\label{eq:problem2}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{n},\; \bm{\ell}_{1}\in\mathbb{R}^{p_{1}},\;\ldots ,\;\bm{\ell}_{q}\in\mathbb{R}^{p_{q}}}{\mbox{minimize}} \;\; f(\bm{x})  \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
			& \bm{A}_{eq}\bm{x} = \bm{b}_{eq} &\\
			& \bm{A}\bm{x} \leq \bm{b}	      &\\
			& \bm{c}_{eq}(\bm{x}) = \bm{0}	 	&\\
		  & \bm{c}(\bm{x}) \leq \bm{0}			&\\
			& \bm{l} \leq \bm{x} \leq \bm{u}	&\\
			& \tilde{\bm{l}} \leq (\bm{\ell}_{1},\ldots,\bm{\ell}_{q}) \leq \tilde{\bm{u}} & \\
			& \svec\left(\bm{\mathcal{A}}_{i}(\bm{x}) - \bm{L}_{i}(\bm{\ell}_{i})\bm{L}_{i}(\bm{\ell}_{i})^{\T}\right) = \bm{0}, & \quad i = 1,...,q \\
			& \mbox{\textsf{diag}}\{\bm{L}_{i}(\bm{\ell}_{i})\} \geq \bm{0}, & \quad i = 1,...,q,
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
where the variables $\bm{\ell}_{i}$ defining the non-zero elements of the Cholesky factors are referred to as \textit{auxiliary variables}, $\tilde{\bm{l}}$ are $\tilde{\bm{u}}$ are constant vectors, and $\mbox{\textsf{diag}}$ returns a vector containing the diagonal elements of a matrix. This problem is in a form amenable to direct treatment by an NLP-solver, and this is the problem to which \texttt{fminsdp} attempts to find a local solution.

\vskip 2mm
\noindent \textbf{Note.} Any local minimum of problem \eqref{eq:problem2} is also a local minimum for \eqref{eq:problem}, and vice versa (assuming that bounds on the auxiliary variables do not prevent this). However, problem \eqref{eq:problem2} may have additional stationary points not present in \eqref{eq:problem}. The author has not experienced any difficulties obviously attributed to this fact, but it cannot be ruled out that it might cause trouble on some problems.
\vskip 2mm

A key motivation behind \texttt{fminsdp} is to abstract away the exact treatment of the matrix inequality constraints so that the user only ''sees'' a problem of the form \eqref{eq:problem}; that is, the user should not have to deal with the Cholesky factors and the auxiliary variables. Thus, when using \texttt{fminsdp} one only works with the primary variables $\bm{x}$, and user-supplied derivatives are only with respect to $\bm{x}$. The examples in Section \ref{sec:tutorial} and the folder \texttt{examples} shows how this is done in practice.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The ldl-method}

In the ldl-method \cite{Vanderbei:2000,Bogani:2009}, \texttt{fminsdp} reformulates the matrix inequality constraints into scalar inequality constraints using the fact that a matrix is positive semi-definite if and only if the diagonal elements in an LDL-factorization of the matrix is non-negative. In other words,
\vskip 2mm
\noindent \textbf{Theorem 2.} A symmetric $m\times m$-matrix $\bm{A} \succeq \bm{0}$ if and only if
\begin{equation}
d_{i}(\bm{A}) \geq 0, \quad i=1,\ldots,m,
\end{equation}
where $d_{i}(\bm{A})$ are the diagonal elements in an LDL-factorization of $\bm{A}$.
\vskip 2mm
The original problem \eqref{eq:problem} is now replaced by
\begin{equation}\label{eq:problemldl}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{n}}{\mbox{minimize}} \;\; f(\bm{x})  \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
			& \bm{A}_{eq}\bm{x} = \bm{b}_{eq} &\\
			& \bm{A}\bm{x} \leq \bm{b}	      &\\
			& \bm{c}_{eq}(\bm{x}) = \bm{0}	 	&\\
		  & \bm{c}(\bm{x}) \leq \bm{0}			&\\
			& \bm{l} \leq \bm{x} \leq \bm{u}	&\\
			& d_{ij}(\bm{\mathcal{A}}_{i}(\bm{x})) \geq 0, \quad j=1,\ldots,m_{i},\; i=1,\ldots,q ,  &
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}

The functions $d_{ij} : \mathbb{S}^{m} \rightarrow \mathbb{R}$ are smooth, and concave, on the set of positive
definite matrices \cite{Vanderbei:2000}, so provided $\bm{\mathcal{A}}_{i}(\bm{x})$ are always positive definite,
\eqref{eq:problemldl} is a smooth NLP. In practise, even if $\bm{\mathcal{A}}_{i}(\bm{x})$ are all positive definite at the initial point (see section \ref{sec:infeasible}), this property can not be ensured throughout the optimization process without taking
special measures. When using \texttt{fmincon} the line search step-length can be reduced until, due to continuity, $\bm{\mathcal{A}}_{i}(\bm{x})$ is positive definite. When using the NLP-solver \texttt{gcmma} the subproblems are made more conservative to reach the same effect.

Compared to the cholesky-method, the ldl-method introduces no additional variables, but requires, e.g., shortening of the search step in line-search procedures, and derivatives of the matrix constraints can be more expensive to compute.

When using NLP-solver \texttt{gcmma} the computational cost can sometimes be reduced significantly by using an active-set approach \cite{Bogani:2009} where the gcmma subproblem only takes into accounts those constraints which satisfies
\begin{equation}\nonumber
d_{ij}(\bm{\mathcal{A}}_{i}(\bm{x})) \leq \eta,
\end{equation}
where $\eta$ is a positive constant. This approach can be efficient for problems with low-rank solutions, and is activated by setting the option \texttt{eta} (see Section \ref{sec:options}) to some suitable value.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Penlab}

The reader is referred to \cite{Fiala:2013} and references therein for a description of the algorithm implemented in PENLAB.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Infeasible initial points}
\label{sec:infeasible}

It is recommended that the user supplies an initial point which is feasible with respect to the constraints of problem \eqref{eq:problem}, in particular the matrix inequality constraints. If the latter is not possible, an option can be set (see section \ref{sec:options}) so that \texttt{fminsdp} attempts to solve the following problem in place of \eqref{eq:problem}:
\begin{equation}\label{eq:problem3}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{n},\; \mbox{$s$}\in\mathbb{R}}{\mbox{minimize}} \;\; f(\bm{x}) + cs \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
			& \bm{A}_{eq}\bm{x} = \bm{b}_{eq} &\\
			& \bm{A}\bm{x} \leq \bm{b}	      &\\
			& \bm{c}_{eq}(\bm{x}) = \bm{0}	 	&\\
		  & \bm{c}(\bm{x}) \leq \bm{0}			&\\
			& \bm{l} \leq \bm{x} \leq \bm{u}	&\\
			& \bm{\mathcal{A}}_{i}(\bm{x}) + s\bm{I}^{m_{i}} \succeq \bm{0}, & \quad i = 1,...,q \\
			& \underline{s} \leq s \leq \overline{s}. &
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
Here $s$ is an auxiliary variable, $\bm{I}^{m_{i}},\, i = 1,...,q$, are identity matrices of size $m_{i} \times m_{i}$, and
$\underline{s}$ and $\overline{s}$ are constants. The constant $c$ appearing in the objective should be set to some large positive number (finding a suitable value might require experimenting a bit) such that $s$ becomes close to zero at a solution.

Unless the user specifies something else, given $\bm{x}_{0}$ \texttt{fminsdp} will generate an initial value for the auxiliary variable according to
\begin{equation}\nonumber
s_{0} = \max_{i=1,\ldots,q}\Big(-\min \,\{\lambda_{1}(\mathcal{A}_{i}(\bm{x}_{0})),-10^{-12}\}\Big),
\end{equation}
where $\lambda_{1}(\cdot)$ returns the smallest eigenvalue of a matrix. This guarantees that $\bm{\mathcal{A}}_{i}(\bm{x}_{0}) + s_{0}\bm{I}^{m_{i}} \succ \bm{0}$ for all $i$.

\vskip 2mm
\noindent \textbf{Note.} The ldl-method requires a feasible initial point, so if you're unable to supply such a point you must set $c > 0$. The cholesky- and penlab-methods do not require a feasible initial point.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Problem size and some limitations}

Assuming the constraint matrices are of small size or very sparse, \texttt{fminsdp} might be able to solve large-scale problems within reasonable time. The ldl-method might also work for problems involving large constraint matrices. For the cholesky-method, one can expect a fairly large number of auxiliary variables and thus many non-zero elements in the constraint Jacobian (and Hessian of the Lagrangian), resulting in much memory and CPU time being devoted to solution of linear systems by the NLP solver used to solve \eqref{eq:problem2}. Penlab, finally, relies on exact second-derivative information which can make problems costly to solve.

Here are some additional things to note:
\begin{enumerate}
\item NLP-solvers ipopt, knitro, snopt, penlab, mma and gcmma must be downloaded and installed separately.
\item Penlab requires a user-supplied function for evaluating the Hessian of the Lagrangian.
\item The ldl-method can currently not make use of a user-supplied Hessian of the Lagrangian and has
      only been tested with NLP-solvers fmincon and gcmma.
\item Although it is possible to improve performance by implementing parts of the code as MEX-functions, \texttt{fminsdp} is a pure Matlab-code. The reason is to make installation, set up and maintenance as easy as possible.
The interested user could try out the Matlab-function \texttt{profile} to identify bottlenecks in the code and
implement them as MEX-functions instead.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Alternatives}

There are many important special cases of problem \eqref{eq:problem} for which specialized solvers
are available. Although this is possible, \texttt{fminsdp} is not intended to replace such solvers, which,
when applicable, can be a lot faster.

The following is an incomplete list of currently available solvers:
\begin{itemize}
\item Problems with linear matrix inequality (LMI) constraints:\\
\hskip2mm SeDuMi, SDPT3, SDPA, LMI Lab, PENSDP, BMISolver, PENBMI, PENNON
\item Problems with bilinear matrix inequality (BMI) constraints:.\\
\hskip2mm BMISolver, PENBMI, PENNON, PENLAB
\item General problems of type \eqref{eq:problem}:\\
\hskip2mm PENNON, PENLAB
\end{itemize}
Note that except for PENNON, and its open-source version PENLAB, there are additional constraints on the structure of the problems treated by the listed solvers.

The Matlab code YALMIP \cite{Yalmip:2004} provides a convenient unified interface to the solvers listed above (except BMISolver).

One could of think of other criteria, not without drawbacks of course, for positive semi-definiteness beside the ones used here that might be used to obtain a standard NLP formulation of \eqref{eq:problem}. Non-negativity of the smallest eigenvalue or of the leading principle minors of a matrix are both necessary and sufficient criteria. If one is satisfied with sufficiency, diagonal dominance could also be considered.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Using \texttt{fminsdp}}
\label{sec:using}

A user of \texttt{fminsdp} is expected to provide two functions; one for evaluating the objective function (i) and one for evaluating the non-linear constraints (ii):
\begin{enumerate}[i.]
\item \texttt{[fval,\textit{grad}] = objfun(x)}
\item \texttt{[cineq,ceq,\textit{cineqgrad},\textit{ceqgrad}] = nonlcon(x)}
\end{enumerate}
(Here and in the following it is assumed that the reader is familiar with the Matlab syntax.) The return arguments written in italics are optional and only needed if the user wishes to provide gradients for the objective and constraints. The matrix inequality constraints are defined in the non-linear constraints function (even linear matrix inequalities). The values of the constraint matrices, vectorized using \svec, are returned as the last elements in the output vector \texttt{ceq}; see Section \ref{sec:tutorial} for an example.

In addition to objective and non-linear constraints functions, the user may optionally provide a function for evaluating
the Hessian of the Lagrangian:
\begin{itemize}
\item \texttt{H = hessian(x,lambda)},
\end{itemize}
where \texttt{lambda} is a struct with two fields \texttt{ineqnonlin} and \texttt{eqnonlin} containing values of the Lagrange
multipliers associated with the non-linear inequality and equality constraints, respectively, and one field \texttt{sigma}
which only used when running with NLP-solver Ipopt.

The calling syntax of \texttt{fminsdp} is similar to that of \texttt{fmincon}:
\begin{verbatim}
>> [x,fval,exitflag,output,lambda,grad,hessian] = ...
										                        fminsdp(objfun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options)
\end{verbatim}
Unlike fmincon, at least 9 input arguments are required. If your problem has no linear constraints and no simple
bounds, the corresponding input arguments can be set to empty matrices. Note that \texttt{fminsdp} does not
support passing additional arguments to the objective or constraint functions through an 11-th input argument.
To pass additional arguments to the user-supplied functions one should use anonymous or nested functions.

\subsection{Options}
\label{sec:options}
A number of options are available when calling \texttt{fminsdp}. These are passed to the code in the form of a struct containing
parameter-value pairs. The most convenient way to specify options is to use the function \texttt{sdpoptionset}:
\begin{verbatim}
>> options = sdpoptionset('MatrixInequalities',true);
\end{verbatim}

In addition to the options accepted by the function \texttt{optimset} from the Matlab Optimization Toolbox, the
following options are available when calling \texttt{fminsdp}:
\vskip 2mm
\noindent \texttt{MatrixInequalities}  \hskip 1cm logical scalar\\
Indicates whether or not the problem has any matrix inequality constraints. If not, \texttt{fminsdp} will simply call the
NLP solver specified using options.NLPsolver.\\
\noindent  Default: true
\vskip 2mm
\noindent \texttt{Aind}  \hskip 3.5cm scalar or numeric array\\
Marks the beginning of each matrix constraint in the vector \texttt{ceq} returned from the non-linear constraints function. If
the option \texttt{sp\_pattern}, described below, is used it is only necessary to mark the beginning of the \texttt{first} matrix
constraint.\\
\noindent  Default: 1
\vskip 2mm
\noindent \texttt{method} \hskip 3.0cm \{'cholesky', 'ldl', 'penlab'\} \\
Select treatment of matrix inequality constraints. \\
Default: 'cholesky'
\vskip 2mm
\noindent \texttt{NLPsolver} \hskip 2.5cm  \{'fmincon', 'ipopt', 'snopt', 'knitro', 'mma', 'gcmma'\}               \\
Select NLP-solver (not applicable if \texttt{method}='penlab'). Interfaces are provided to fmincon, Ipopt \cite{Wachter:2006}, SNOPT \cite{Gill:2002}, KNITRO \cite{Byrd:2006}, MMA/GCMMA \cite{Svanberg:2007}, and PENLAB.
NOTE: If you run Ipopt older than 3.11.0, make sure to modify the file ipopt\_main.m appropriately. \\
Default: 'fmincon'
\vskip 2mm
\noindent \texttt{max\_cpu\_time}  \hskip 2.1cm positive scalar \\
Maximum CPU time. Applicable to NLP-solvers fmincon, ipopt, mma and gcmma.\\
Default: inf
\vskip 2mm
\noindent \texttt{sp\_pattern}  \hskip 2.3cm  (sparse) matrix or cell array of matrices\\
Sparsity patterns of the  matrix constraints. If this option is used, the user must provide one matrix for each matrix constraint.
\\
\noindent  Default: []
\vskip 2mm
\noindent \texttt{L0} \hskip 3.2cm  (sparse) matrix  or cell array of matrices \\
Initial guess for the Cholesky factors of the matrix constraints. If the user does not provide an initial
guess, \texttt{fminsdp} will generate one by attempting a Cholesky factorization of the constraint matrices
at the initial point. If this fails, \texttt{fminsdp} will add a multiple of the identity matrix to
the constraint matrices until all of them are positive definite and use the Cholesky factorizations of these
matrices as an initial guess. \\
Default: []
\vskip 2mm
\noindent \texttt{Ldiag\_low} \hskip 2cm  scalar or numeric array	\\
Lower bound(s) on the diagonal elements of the Cholesky factors. \\
Default: 0
\vskip 2mm
\noindent \texttt{L\_low} \hskip 2.75cm   scalar or array of doubles \\
Lower bound(s) on the off-diagonal element of the Cholesky factors \\
Default: -inf
\vskip 2mm
\noindent
\texttt{L\_upp} \hskip 2.75cm  scalar or array of doubles \\
Upper bound(s) on the elements of the Cholesky factors (including the diagonal elements).\\
Default: inf
\vskip 2mm
\noindent
\texttt{eta} \hskip 3.1cm  positive scalar  \\
Tolerance for determining the active set when using the ldl-method with NLP-solver gcmma.\\
Default: inf
\vskip 2mm
\noindent
\texttt{c} \hskip 3.47cm  non-negative scalar    \\
If $c>0$, \texttt{fminsdp} attempts to solve problem \eqref{eq:problem3} instead of
\eqref{eq:problem}. The user may in this case also let the objective function be empty; i.e., the first input argument to \texttt{fminsdp} can be set to '[]'. This is useful when one wants to check feasibility of one or more matrix inequalities. \\
Default: 0
\vskip 2mm
\noindent
\texttt{s\_low}   \hskip 2.8cm        scalar      \\
Lower bound on the auxiliary variable $s$ in problem \eqref{eq:problem3}.\\
Default: 0
\vskip 2mm
\noindent
\texttt{s\_upp}   \hskip 2.8cm        scalar      \\
Upper bound on the auxiliary variable $s$ in problem \eqref{eq:problem3}.\\
Default: inf
\vskip 2mm
\noindent
\texttt{HessianCheck} \hskip 1.4cm   \{'on', 'off'\} \\
Simple check of Hessian of the Lagrangian against finite differences at the initial point.
Assumes you have the code DERIVEST, which must be obtained separately, on your Matlab path.
This check can be very time consuming and should preferably be carried out on small instances of
a problem.\\
Default: 'off'
\vskip 2mm
\noindent
\texttt{HessMult} \hskip 2cm   \{function\_handle, 'on'\} \\
If using fmincon with \texttt{options.SubProblemAlgorithm = 'cg'}, you can work with Hessian
times vector products directly, thereby avoiding the formation of the full Hessian of
the Lagrangian. Set to a function handle or simply to 'on' if the Hessian of
the Lagrangian with respect to the primary variables is zero. Only applicable when using the cholesky-method. \\
Default: []
\vskip 2mm
\noindent
\texttt{ipopt} \hskip 2.7cm     struct   \\
Options to be passed on to NLP solver Ipopt. Please refer to the Ipopt documentation for a
list of available options. \\
Default: []
\vskip 2mm
\noindent
\texttt{eigs\_opts} \hskip 2.1cm       struct    \\
Options passed to the Matlab function \texttt{eigs} used for eigenvalue computations. \\
Default: struct('isreal',true,'issym',true)
\vskip 2mm
\noindent
\texttt{KnitroOptionsFile} \hskip 0.5cm  character array \\
Name of an options file to be read by NLP solver KNITRO. Please refer to the KNITRO
documentation for a list of available options.\\
Default: []
\vskip 2mm
\noindent
\texttt{SnoptOptionsFile} \hskip 0.6cm   character array \\
Name of an options file to be read by NLP solver SNOPT. Please refer to the SNOPT
documentation for a list of available options.\\
Default: []
\vskip 2mm
\noindent
\texttt{GradPattern} \hskip 1.5cm        numeric array   \\
Sparsity pattern for the gradient of the objective function. Only used by NLP solver SNOPT and
only effective if you also set \texttt{options.JacobPattern} (see the fmincon documentation for
details on the latter). \\
Default: []

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Output}

The available output arguments from \texttt{fminsdp} are the same as those of fmincon:
\begin{verbatim}
>> [x,fval,exitflag,output,lambda,grad,hessian] = fminsdp(...)
\end{verbatim}
The only difference is that the struct \texttt{output}, the fourth output argument, contains
some additional fields (some are only available when running the cholesky-method):
\vskip 3mm
\noindent
\texttt{A} \hskip 4.2cm        cell array of matrices  \\
Constraint matrices evaluated at the solution \texttt{x}.
\vskip 2mm
\noindent
\texttt{L} \hskip 4.2cm        cell array of matrices  \\
Cholesky factors of the constraint matrices evaluated at the solution \texttt{x}. These are computed
by assembling each $\bm{L}_{i}$ from the variable vector $\bm{\ell}_{i}$ and not by a Cholesky factorization
of the corresponding constraint matrix.
\vskip 2mm
\noindent
\texttt{L0} \hskip 4.0cm        cell array of matrices  \\
Cholesky factors of the constraint matrices evaluated at the initial point \texttt{x0}.
\vskip 2mm
\noindent \texttt{nxvars}		\hskip 3.2cm 		numeric scalar \\
Number of primary variables.
\vskip 2mm
\noindent \texttt{nLvars}		\hskip 3.2cm     numeric scalar \\
Number of auxiliary variables.
\vskip 2mm
\noindent \texttt{A\_size}   \hskip 3.3cm     numeric array \\
Size of the constraint matrices; i.e., $m_{i}$ in \eqref{eq:problem2}.
\vskip 2mm
\noindent \texttt{nMatrixConstraints}   \hskip 1cm     numeric scalar \\
Number of matrix constraints.
\vskip 2mm
\noindent \texttt{NLPsolver}   \hskip 2.7cm     string \\
Selected NLP solver.
\vskip 3mm
It the user has set \texttt{options.c} to some positive number in order to solve problem \eqref{eq:problem3},
then two additional fields are available:
\vskip 3mm
\noindent \texttt{s0}   \hskip 2.7cm     double scalar \\
Initial value for the auxiliary variable $s$.
\vskip 2mm
\noindent \texttt{s}   \hskip 2.9cm     double scalar \\
Value of $s$ at the solution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{A tutorial example}
\label{sec:tutorial}

To aid the user, a very simple tutorial example is provided here. For more details, please refer to the examples
found in the \texttt{examples}-folder.

Consider the following (non-sense) problem:
\begin{equation}\label{eq:tutorial}
\quad
	\begin{aligned}
    & \underset{\bm{x}\in\mathbb{R}}{\mbox{minimize}} \;\;  x^{2}  \\
	& \mbox{subject to} \;
	\left\{
		\begin{aligned}
			& x^{3} = 0 	\\
		    & x^{2} \leq 0			\\
			& \left(
			\begin{array}{cc}
					x   & x^2 \\
			        x^2 & 0
			      \end{array}
			\right) \succeq \bm{0}
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
To solve this problem using \texttt{fminsdp} we need to implement at least two functions:

\begin{enumerate}
\item The objective function:
\begin{verbatim}
function [fval,grad] = objfun(x)
fval = x^2;
if nargout>1
   grad = 2*x;
end
\end{verbatim}

\item The non-linear constraints function:
\begin{verbatim}
function [cineq,ceq,cineqgrad,ceqgrad] = nonlcon(x)
cineq = x^2;
ceq = [x^3; svec([x x^2; x^2 0]))];
if nargout>2
   cineqgrad = 2*x;
   ceqgrad = [3*x^2 svec([1 2*x; 2*x 0])'];
end
\end{verbatim}

\end{enumerate}

\noindent A function for evaluating the Hessian of the Lagrangian is optional, but recommended. One
such function is given here:
\begin{verbatim}
function H = hessian(x, lambda)
H = lambda.ineqnonlin*2 + lambda.eqnonlin(1)*6*x  + ...
    lambda.eqnonlin(2:end,1)'*svec([0 2; 2 0]));
\end{verbatim}

Using the functions specified above, a simple script to solve problem \eqref{eq:tutorial} can now be written:

\begin{verbatim}

% Mark the beginning of the matrix inequality constraints in the vector ceq
% returned from nonlcon
options.Aind = 2;

% Specify that analytical gradients should be used
options.GradObj = 'on';
options.GradConstr = 'on';

% Specify that the function "hessian" should be used for the
% Hessian of the Lagrangian
options.Hessian = 'user-supplied'
options.HessFcn = @(x,lambda) hessian(x,lambda);

% Specify initial point
x0 = 1;

% Call fminsdp
[x,fval] = fminsdp(objfun,x0,[],[],[],[],[],[],nonlcon,options);

\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Practicalities}
\label{sec:practicalities}

For a given problem there are usually a number of things that can be done to improve numerical performance. One can, for instance, try out various scalings or introduce new variables and constraints to reduce the degree of non-linearity. Here are some additional tips:

\vskip 5mm
\noindent\textbf{Exploit sparsity}
\vskip 2mm
\noindent In many cases, the functions $\bm{\mathcal{A}}_{i}$, $i = 1,...,q$, are sparse in the sense that the matrix $\bm{\mathcal{A}}_{i}(\bm{x})$
is sparse for every $\bm{x}$. This fact can be exploited to reduce the number of auxiliary variables and nonlinear constraints for the cholesky-method.

To take advantage of sparsity, the user should compute the sparsity pattern of each matrix constraint (or
an (pessimistic) estimate thereof) and supply this to \texttt{fminsdp} via the option \texttt{sp\_pattern}. For the
dummy problem in Section \ref{sec:tutorial} we simply add two lines to the driver script:
\begin{verbatim}
sp_A = [1 1; 1 0]					  		        % Sparsity pattern of the constraint matrix
options.sp_pattern = sp_A;

...

[x,fval] = fminsdp(@(x) objfun(x),x0,[],[],[],...
				   [],[],[],@(x) nonlcon(x),options);

\end{verbatim}

\vskip 2mm
\noindent\textbf{Add artificial upper and lower bounds}
\vskip 2mm
\noindent Even though some of the variables in your problem has no  ''natural'' bounds on them, it is usually wise
to add upper and lower bounds. These bounds should of course be loose enough to not exclude any solution of
interest, but even if they don't, setting them too tight might have a negative impact on the solution process.
Therefore, a bit of experimenting is recommended.

Bounds on the auxiliary variables in the cholesky-method can be specified by setting \texttt{options.L\_low} and \texttt{options.L\_upp}. Sometimes it is easy to derive
suitable bounds \textit{a priori} \cite{Thore:2015}.

\vskip 5mm
\noindent\textbf{Experiment with various solvers and algorithms}
\vskip 2mm
\noindent \texttt{fminsdp} can use fmincon, SNOPT, KNITRO, Ipopt, mma, gcmma and PENLAB to solve problems \eqref{eq:problem2}. In addition, fmincon
and KNITRO lets the user choose between three different algorithms (two interior-point and one sqp). It is unlikely that a single solver and/or algorithm is best suited for all types of problems so it is recommended that the user experiment with various
alternatives. The author's experience is that the interior-point method of fmincon (selected by setting \texttt{options.Algorithm='interior-point'} and \texttt{options.SubProblemAlgorithm = 'ldl-factorization'}) works well in
terms of the number of function evaluations, but that more efficient handling of the linear algebra makes the other solvers
better suited for larger problems.

\vskip 5mm
\noindent\textbf{Provide gradients and Hessian of the Lagrangian}
\vskip 2mm
\noindent Providing code that computes gradients and the Hessian of the Lagrangian is usually a good idea (if you use Ipopt you \textit{must} provide code for evaluating gradients; SNOPT does not make use of code for evaluating the Hessian of the Lagrangian). Due to the homogeneity of $\svec$ one has simply
\begin{equation}\nonumber
\frac{\partial \svec(\bm{\mathcal{A}})}{\partial x_{i}} = \svec\left(\frac{\partial \bm{\mathcal{A}}}{\partial x_{i}}\right),
\end{equation}
and for a function $L(\bm{x}) = \bm{\lambda}^{\T}\svec(\bm{\mathcal{A}}(\bm{x}))$,
\begin{equation}\nonumber
\frac{\partial^2 L}{\partial x_{i}\partial x_{j}} =  \bm{\lambda}^{T}\svec\left(\frac{\partial^2 \bm{\mathcal{A}}}{\partial x_{i}\partial x_{j}}\right).
\end{equation}

The user is strongly recommended to check the correctness of the derivatives by comparing against finite-difference
approximations. This can be done by setting \texttt{options.DerivativeCheck='on'} and \texttt{options.HessianCheck='on'}.
If necessary, the accuracy of the finite-difference approximations in \texttt{fmincon} can be increased by setting
\texttt{options.FinDiffType='central'}.

If you use NLP solvers SNOPT, Ipopt or KNITRO, you can also provide sparsity patterns for the constraint Jacobian, and Ipopt
and KNITRO can also exploit a sparsity pattern for the Hessian of the Lagrangian. Sparsity patterns are passed to the solvers
by setting \texttt{options.JacobPattern} and \texttt{options.HessPattern} to sparse matrices.

As an alternative to deriving and implementing derivatives one might consider using automatic differentiation, which provides
derivatives with accuracy to machine precision, and, at least in principle, does so completely automatically.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Examples}

This section comprise a brief description of three problems concerned with structural optimization of (plane) trusses. Implementations of the example problem can be found in the folder \texttt{examples}.\footnote{Note that there are often many different ways to formulate such problems \cite{Cristensen:2009,Bendsoe:1994}, and formulations other than those given here may certainly be better suited to one's needs. Here, however, we are specifically interested in solving problems with matrix inequality constraints.}

Let $n_{d} = 2$ be the number of spatial dimensions, $N$ the number of nodes in the truss, and $n_{fixed}$ the number of prescribed (to zero) displacement components. The number of displacement degrees of freedom is $n=n_{d}N-n_{fixed}$ and there are $m$ potential bars in the truss.
The $m$ bar volumes are collected in a vector $\bm{x}\geq\bm{0}$ which is used to parametrize the design of the truss. Assuming (infinitesimally) small deformations and a quasi-static situation, the nodal displacement vector $\bm{u}\in\mathbb{R}^{n}$ satisfies the equilibrium equation
\begin{equation}\label{eq:equlibrium}
\bm{K}(\bm{x})\bm{u} = \bm{f},
\end{equation}
where $\bm{K}(\bm{x})$ is known as the stiffness matrix and $\bm{f}\in\mathbb{R}^{n}$ contains forces applied to the nodes. The  stiffness matrix is given by
\begin{equation}\nonumber
\bm{K}(\bm{x}) = \sum_{i=1}^{m}x_{i}E_{i}\bm{b}_{i}\bm{b}_{i}^{\T},
\end{equation}
where $x_{i}$ is the volume of the $i$:th bar, $E_{i}$ the Young's modulus, and the vectors $\bm{b}_{i}$ depend on the geometry of the undeformed truss. Clearly, $\bm{K}(\bm{x})$ is positive semi-definite and symmetric. Assuming rigid body motions are prevented, by appropriate support conditions, and $\bm{x}>\bm{0}$, it is even positive definite.
\vskip2mm
\noindent\textbf{1.} The first problem in the \texttt{examples}-folder is to minimize the volume of a truss subject to the equilibrium condition \eqref{eq:equlibrium} and an upper bound $c$ on the so-called compliance $\bm{f}^{\T}\bm{u}$. This can be formulated as a problem involving a single LMI:
\begin{equation}\label{eq:prob1}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{m}}{\mbox{minimize}} \;\; \sum_{i=1}^{m}x_{i} \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
		  & \left(
\begin{array}{cc}
c &\bm{f}^{\T}         \\
\bm{f} & \bm{K}(\bm{x})
\end{array}
\right) \succeq \bm{0} \\
			& x_{i} \geq 0 , \quad i=1,\ldots,m.
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}

\vskip2mm
\noindent\textbf{2.}
A drawback of problem \eqref{eq:prob1} is that optimized structures may be unstable in the sense that they are prone to global buckling. One way to avoid many such designs is to impose an additional constraint \cite{Kocvara:2002} requiring that
\begin{equation}\label{eq:stability}
\bm{K}(\bm{x}) + \bm{G}(\bm{u}(\bm{x}),\bm{x}) \succeq \bm{0},
\end{equation}
where the geometrix stiffness matrix
\begin{equation}\nonumber
\bm{G}(\bm{u}(\bm{x}),\bm{x}) = \sum_{i=1}^{m}x_{i}E_{i}\bm{b}_{i}^{\T}\bm{u}(\bm{x})\bm{\gamma}_{i}\bm{\gamma}_{i}^{\T},
\end{equation}
in which $\bm{u}(\bm{x})$ denotes a solution to \eqref{eq:equlibrium} and $\bm{\gamma}_{i}$ depend on the geometry of the undeformed truss.

Adding \eqref{eq:stability} to \eqref{eq:prob1} leads to the following problem involving both a linear and a non-linear matrix
inequality\footnote{In practice it is perhaps better to replace the LMI in \eqref{eq:prob2} by the non-linear constraint $\bm{f}^{\T}\bm{u}(\bm{x})\leq c$ but we keep it to illustrate solution of a problem with more than one matrix inequality.}:
\begin{equation}\label{eq:prob2}
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{m}}{\mbox{minimize}} \;\; \sum_{i=1}^{m}x_{i} \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
		  & \left(
\begin{array}{cc}
c &\bm{f}^{\T}         \\
\bm{f} & \bm{K}(\bm{x})
\end{array}
\right) \succeq \bm{0} \\
			& \bm{K}(\bm{x}) + \bm{G}(\bm{u}(\bm{x}),\bm{x}) \succeq \bm{0} \\
			& x_{i} \geq \epsilon , \quad i=1,\ldots,m,
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
where $\epsilon$ is a small positive number introduced to avoid singularity of the stiffness matrix. \citet{Kocvara:2002} showed that a solution to this problem corresponds to a truss that will not exhibit global, linear buckling for loads
of the form $\tau\bm{f}$, $\tau\in[0,1)$.

\vskip2mm
\noindent\textbf{3.}
The third example problem is an alternative formulation of problem \eqref{eq:prob2} obtained by treating the displacements as explicit variables in the optimization problem:
\begin{equation}\nonumber
\quad
	\begin{aligned}
	&	\underset{\bm{x}\in\mathbb{R}^{m},\bm{u}\in\mathbb{R}^{n}}{\mbox{minimize}} \;\; \sum_{i=1}^{m}x_{i} \\
	&	\mbox{subject to} \;
	\left\{
		\begin{aligned}
		  & \bm{f}^{\T}\bm{u} \leq c \\
		  & \bm{K}(\bm{x})\bm{u} = \bm{f} \\
			& \bm{K}(\bm{x}) + \bm{G}(\bm{u},\bm{x}) \succeq \bm{0} \\
			& x_{i} \geq \epsilon , \quad i=1,\ldots,m.
		\end{aligned}
		\right.
	\end{aligned}
\end{equation}
In this formulation, the upper bound on the compliance is a linear constraint, the equilibrium equation a set of bilinear equality constraints,
and the stability constraint is a BMI.

\vskip 5mm
\noindent To see how these problems can be solved with \texttt{fminsdp}, and what optimized designs might look like, please check out the Matlab codes found in the \texttt{examples}-folder.

\newpage
\bibliographystyle{../../../spbasic}
\bibliography{references}

\end{document}


\section{Connections on Riemann manifolds}


\chapter{\texorpdfstring{Typed sets and category $\mathbf{Rel}$}%
{Typed sets and category Rel}}

\section{Relational structures}
\begin{defn}
\index{relational structure}A \emph{relational structure} is a pair
consisting of a set and a tuple of relations on this set.
\end{defn}
A poset $(\mathfrak{A},\sqsubseteq)$ can be considered as a relational
structure: $(\mathfrak{A},\llbracket\sqsubseteq\rrbracket).$

A set can $X$ be considered as a relational structure with zero relations:
$(X,\llbracket\rrbracket).$

This book is not about relational structures. So I will not introduce
more examples.

Think about relational structures as a common place for sets or posets,
as far as they are considered in this book.

We will denote $x\in(\mathfrak{A},R)$ iff $x\in\mathfrak{A}$ for
a relational structure $(\mathfrak{A},R)$.

\section{Typed elements and typed sets}

We sometimes want to differentiate between the same element of two
different sets. For example, we may want to consider different the
natural number~$3$ and the rational number~$3$. In order to describe
this in a formal way we consider elements of sets together with sets
themselves. For example, we can consider the pairs $(\mathbb{N},3)$
and $(\mathbb{Q},3)$.
\begin{defn}
\index{typed element}\index{element!typed}A \emph{typed element}
is a pair $(\mathfrak{A},a)$ where $\mathfrak{A}$ is a relational
structure and $a\in\mathfrak{A}$.

I denote $\type(\mathfrak{A},a)=\mathfrak{A}$ and $\GR(\mathfrak{A},a)=a$.
\end{defn}

\begin{defn}
I will denote typed element $(\mathfrak{A},a)$ as $@^{\mathfrak{A}} a$ or just
$@a$ when $\mathfrak{A}$ is clear from context.
\end{defn}

\begin{defn}
\index{typed set}\index{set!typed}A \emph{typed set} is a typed
element equal to $(\subsets U,A)$ where $U$ is a set and $A$ is
its subset.\end{defn}
\begin{rem}
\emph{Typed sets} is an awkward formalization of type theory sets
in ZFC ($U$ is meant to express the \emph{type} of the set). This
book could be better written using type theory instead of ZFC, but
I want my book to be understandable for everyone knowing ZFC. $(\subsets U,A)$
should be understood as a set $A$ of type $U$. For an example, consider
$(\subsets\mathbb{R},[0;10])$; it is the closed interval $[0;10]$
whose elements are considered as real numbers.\end{rem}
\begin{defn}
$\mathfrak{T}\mathfrak{A}=\setcond{(\mathfrak{A},a)}{a\in\mathfrak{A}}=\{\mathfrak{A}\}\times\mathfrak{A}$
for every relational structure~$\mathfrak{A}$.\end{defn}
\begin{rem}
$\mathfrak{T}\mathfrak{A}$ is the set of typed elements of~$\mathfrak{A}$.
\end{rem}

\begin{defn}
If $\mathfrak{A}$ is a poset, we introduce order on its typed elements
isomorphic to the order of the original poset: $(\mathfrak{A},a)\sqsubseteq(\mathfrak{A},b)\Leftrightarrow a\sqsubseteq b$.
\end{defn}

\begin{defn}
I denote $\GR(\mathfrak{A},a)=a$ for a typed element~$(\mathfrak{A},a)$.
\end{defn}

\begin{defn}
I will denote \emph{typed subsets} of a typed poset $(\subsets U,A)$
as $\subsets(\subsets U,A)=\setcond{(\subsets U,X)}{X\in\subsets A}=\{\subsets U\}\times\subsets A$.\end{defn}
\begin{obvious}
$\subsets(\subsets U,A)$ is also a set of typed sets.
\end{obvious}

\begin{defn}
I will denote $\mathscr{T}U=\mathfrak{T}\subsets U$.\end{defn}
\begin{rem}
This means that $\mathscr{T}U$ is the set of typed subsets of a set~$U$.\end{rem}
\begin{obvious}
$\mathscr{T}U=\setcond{(\subsets U,X)}{X\in\subsets U}=\{\subsets U\}\times\subsets U=\subsets(\subsets U,U)$.
\end{obvious}

\begin{obvious}
$\mathscr{T}U$ is a complete atomistic boolean lattice. Particularly:
\begin{enumerate}
\item $\bot^{\mathscr{T}U}=(\subsets U,\emptyset)$;
\item $\top^{\mathscr{T}U}=(\subsets U,U)$;
\item $(\subsets U,A)\sqcup(\subsets U,B)=(\subsets U,A\cup B)$;
\item $(\subsets U,A)\sqcap(\subsets U,B)=(\subsets U,A\cap B)$;
\item $\bigsqcup_{A\in S}(\subsets U,A)=(\subsets U,\bigcup_{A\in S}A)$;
\item $\bigsqcap_{A\in S}(\subsets U,A)=\left(\subsets U,\begin{cases}\bigcap_{A\in S}A&\text{ if }A\ne\emptyset\\U&\text{ if }A=\emptyset\end{cases}\right)$;
\item $\overline{(\subsets U,A)}=(\subsets U,U\setminus A)$;
\item atomic elements are $(\subsets U,\{x\})$ where $x\in U$.
\end{enumerate}
Typed sets are ``better'' than regular sets as (for example) for
a set~$U$ and a typed set~$X$ the following are defined by regular
order theory:\end{obvious}
\begin{itemize}
\item $\atoms X$;
\item $\overline{X}$;
\item $\bigsqcap^{\mathscr{T}U}\emptyset$.
\end{itemize}
For regular (``non-typed'') sets these are not defined (except of
$\atoms X$ which however needs a special definition instead of using
the standard order-theory definition of atoms).

Typed sets are convenient to be used together with filters on sets
(see below), because both typed sets and filters have a set~$\subsets U$
as their type.

Another advantage of typed sets is that their binary product (as defined
below) is a $\mathbf{Rel}$-morphism. This is especially convenient
because below defined products of filters are also morphisms of related
categories.

Well, typed sets are also quite awkward, but the proper way of doing
modern mathematics is \emph{type theory} not ZFC, what is however
outside of the topic of this book.

\section{\texorpdfstring{Category $\mathbf{Rel}$}{Category Rel}}

I remind that $\mathbf{Rel}$ is the category of (small) binary relations
between sets, and $\mathbf{Set}$ is its subcategory where only monovalued
entirely defined morphisms (functions) are considered.
\begin{defn}
Order on $\mathbf{Rel}(A,B)$ is defined by the formula $f\sqsubseteq g\Leftrightarrow\GR f\subseteq\GR g$.\end{defn}
\begin{obvious}
This order is isomorphic to the natural order of subsets of the set
$A\times B$.\end{obvious}
\begin{defn}
$X\rsuprel fY\Leftrightarrow\GR X\rsuprel{\GR f}\GR Y$ and $\rsupfun fX=(\Dst f,\rsupfun{\GR f}\GR X)$
for a $\mathbf{Rel}$-morphism~$f$ and typed sets $X\in\mathscr{T}\Src f$,
$Y\in\mathscr{T}\Dst f$.
\end{defn}

\begin{defn}
For category $\mathbf{Rel}$ there is defined reverse morphism: $(A,B,F)^{-1}=(B,A,F^{-1})$.\end{defn}
\begin{obvious}
$(f^{-1})^{-1}=f$ for every $\mathbf{Rel}$-morphism~$f$.
\end{obvious}

\begin{obvious}
$\rsuprel{f^{-1}}=\rsuprel f^{-1}$ for every $\mathbf{Rel}$-morphism~$f$.
\end{obvious}

\begin{obvious}
$(g\circ f)^{-1}=f^{-1}\circ g^{-1}$ for every composable $\mathbf{Rel}$-morphisms
$f$ and~$g$.\end{obvious}
\begin{prop}
$\rsupfun{g\circ f}=\rsupfun g\circ\rsupfun f$ for every composable
$\mathbf{Rel}$-morphisms $f$ and~$g$.\end{prop}
\begin{proof}
Exercise.\end{proof}
\begin{prop}
The above definitions of monovalued morphisms of $\mathbf{Rel}$ and
of injective morphisms of $\mathbf{Set}$ coincide with how mathematicians
usually define monovalued functions (that is morphisms of $\mathbf{Set}$)
and injective functions.\end{prop}
\begin{proof}
Let $f$ be a $\mathbf{Rel}$-morphism $A\rightarrow B$.

The following are equivalent:
\begin{itemize}
\item $f$ is a monovalued relation;
\item $\forall x\in A,y_{0},y_{1}\in B:(x\mathrel fy_{0}\land x\mathrel fy_{1}\Rightarrow y_{0}=y_{1})$;
\item $\forall x\in A,y_{0},y_{1}\in B:(y_{0}\ne y_{1}\Rightarrow\lnot(x\mathrel fy_{0})\lor\lnot(x\mathrel fy_{1}))$;
\item $\forall y_{0},y_{1}\in B\forall x\in A:(y_{0}\ne y_{1}\Rightarrow\lnot(x\mathrel fy_{0})\lor\lnot(x\mathrel fy_{1}))$;
\item $\forall y_{0},y_{1}\in B:(y_{0}\ne y_{1}\Rightarrow\forall x\in A:(\lnot(x\mathrel fy_{0})\lor\lnot(x\mathrel fy_{1})))$;
\item $\forall y_{0},y_{1}\in B:(\exists x\in A:(x\mathrel fy_{0}\land x\mathrel fy_{1})\Rightarrow y_{0}=y_{1})$;
\item $\forall y_{0},y_{1}\in B:y_{0}\mathrel{(f\circ f^{-1})}y_{1}\Rightarrow y_{0}=y_{1}$;
\item $f\circ f^{-1}\sqsubseteq1_{B}$.
\end{itemize}
Let now $f$ be a $\mathbf{Set}$-morphism $A\rightarrow B$.

The following are equivalent:
\begin{itemize}
\item $f$ is an injective function;
\item $\forall y\in B,a,b\in A:(a\mathrel fy\land b\mathrel fy\Rightarrow a=b)$;
\item $\forall y\in B,a,b\in A:(a\ne b\Rightarrow\lnot(a\mathrel fy)\lor\lnot(b\mathrel fy))$;
\item $\forall y\in B:(a\ne b\Rightarrow\forall a,b\in A:(\lnot(a\mathrel fy)\lor\lnot(b\mathrel fy)))$;
\item $\forall y\in B:(\exists a,b\in A:(a\mathrel fy\land b\mathrel fy)\Rightarrow a=b)$;
\item $f^{-1}\circ f\sqsubseteq1_{A}$.
\end{itemize}
\end{proof}
\begin{prop}
For a binary relation $f$ we have:
\begin{enumerate}
\item \label{br-j}$\rsupfun f\bigcup S=\bigcup\rsupfun{\rsupfun f}S$ for
a set of sets $S$;
\item \label{br-j1}$\bigcup S\rsuprel fY\Leftrightarrow\exists X\in S:X\rsuprel fY$
for a set of sets $S$;
\item \label{br-j2}$X\rsuprel f\bigcup T\Leftrightarrow\exists Y\in T:X\rsuprel fY$
for a set of sets $T$;
\item \label{br-j12}$\bigcup S\rsuprel f\bigcup T\Leftrightarrow\exists X\in S,Y\in T:X\rsuprel fY$
for sets of sets $S$ and $T$;
\item \label{br-at-r}$X\rsuprel fY\Leftrightarrow\exists\alpha\in X,\beta\in Y:\{\alpha\}\rsuprel f\{\beta\}$
for sets $X$ and $Y$;
\item \label{br-at-f}$\rsupfun fX=\bigcup\rsupfun{\rsupfun f}\atoms X$ for a
set $X$ (where $\atoms X=\setcond{\{x\}}{x\in X}$).
\end{enumerate}
\end{prop}
\begin{proof}
~
\begin{widedisorder}
\item [{\ref{br-j}}] ~

\begin{multline*}
y\in\rsupfun f\bigcup S\Leftrightarrow\exists x\in\bigcup S:x\mathrel fy\Leftrightarrow\exists P\in S,x\in P:x\mathrel fy\Leftrightarrow\\
\exists P\in S:y\in\rsupfun fP\Leftrightarrow\exists Q\in\rsupfun{\rsupfun f}S:y\in Q\Leftrightarrow y\in\bigcup\rsupfun{\rsupfun f}S.
\end{multline*}

\item [{\ref{br-j1}}]
\begin{multline*}
\bigcup S\rsuprel fY\Leftrightarrow\exists x\in\bigcup S,y\in Y:x\mathrel fy\Leftrightarrow\\
\exists X\in S,x\in X,y\in Y:x\mathrel fy\Leftrightarrow\exists X\in S:X\rsuprel fY.
\end{multline*}

\item [{\ref{br-j2}}] By symmetry.
\item [{\ref{br-j12}}] From two previous formulas.
\item [{\ref{br-at-r}}] $X\rsuprel fY\Leftrightarrow\exists\alpha\in X,\beta\in Y:\alpha\mathrel f\beta\Leftrightarrow\exists\alpha\in X,\beta\in Y:\{\alpha\}\rsuprel f\{\beta\}$.
\item [\ref{br-at-f}] Obvious.
\end{widedisorder}
\end{proof}
\begin{cor}
For a $\mathbf{Rel}$-morphism $f$ we have:
\begin{enumerate}
\item $\rsupfun f\bigsqcup S=\bigsqcup\rsupfun{\rsupfun f}S$ for $S\in\subsets\mathscr{T}\Src f$;
\item $\bigsqcup S\rsuprel fY\Leftrightarrow\exists X\in S:X\rsuprel fY$
for $S\in\subsets\mathscr{T}\Src f$;
\item $X\rsuprel f\bigsqcup T\Leftrightarrow\exists Y\in T:X\rsuprel fY$
for $T\in\subsets\mathscr{T}\Dst f$;
\item $\bigsqcup S\rsuprel f\bigsqcup T\Leftrightarrow\exists X\in S,Y\in T:X\rsuprel fY$
for $S\in\subsets\mathscr{T}\Src f$, $T\in\subsets\mathscr{T}\Dst f$;
\item $X\rsuprel fY\Leftrightarrow\exists x\in\atoms X,y\in\atoms Y:x\rsuprel fy$
for $X\in\mathscr{T}\Src f$, $Y\in\mathscr{T}\Dst f$;
\item $\rsupfun fX=\bigsqcup\rsupfun{\rsupfun f}\atoms X$ for $X\in\mathscr{T}\Src f$.
\end{enumerate}
\end{cor}

\begin{cor}
A $\mathbf{Rel}$-morphism $f$ can be restored knowing either $\rsupfun fx$
for atoms $x\in\mathscr{T}\Src f$ or $x\rsuprel fy$ for atoms $x\in\mathscr{T}\Src f$,
$y\in\mathscr{T}\Dst f$.\end{cor}
\begin{prop}
Let $A$, $B$ be sets, $R$ be a set of binary relations.
\begin{enumerate}
\item \textbf{\label{bsr-jf}$\rsupfun{\bigcup R}X=\bigcup_{f\in R}\rsupfun fX$}
for every set $X$;
\item \label{bsr-mf}$\rsupfun{\bigcap R}\{\alpha\}=\bigcap_{f\in R}\rsupfun f\{\alpha\}$
for every $\alpha$, if $R$ is nonempty;
\item \label{bsr-jr}$X\rsuprel{\bigcup R}Y\Leftrightarrow\exists f\in R:X\rsuprel fY$
for every sets $X$, $Y$;
\item \label{bsr-mr}$\alpha\mathrel{\left(\bigcap R\right)}\beta\Leftrightarrow\forall f\in R:\alpha\mathrel f\beta$
for every $\alpha$ and $\beta$, if $R$ is nonempty.
\end{enumerate}
\end{prop}
\begin{proof}
~
\begin{widedisorder}
\item [{\ref{bsr-jf}}] ~
\begin{multline*}
y\in\rsupfun{\bigcup R}X\Leftrightarrow\exists x\in X:x\mathrel{\left(\bigcup R\right)}y\Leftrightarrow\exists x\in X,f\in R:x\mathrel fy\Leftrightarrow\\
\exists f\in R:y\in\rsupfun fX\Leftrightarrow y\in\bigcup_{f\in R}\rsupfun fX.
\end{multline*}

\item [{\ref{bsr-mf}}] ~
\[
y\in\rsupfun{\bigcap R}\{\alpha\}\Leftrightarrow\forall f\in R:\alpha\mathrel fy\Leftrightarrow\forall f\in R:y\in\rsupfun f\{\alpha\}\Leftrightarrow y\in\bigcap_{f\in R}\rsupfun f\{\alpha\}.
\]

\item [{\ref{bsr-jr}}] ~
\begin{multline*}
X\rsuprel{\bigcup R}Y\Leftrightarrow\exists x\in X,y\in Y:x\mathrel{\left(\bigcup R\right)}y\Leftrightarrow\\
\exists x\in X,y\in Y,f\in R:x\mathrel fy\Leftrightarrow\exists f\in R:X\rsuprel fY.
\end{multline*}

\item [{\ref{bsr-mr}}] Obvious.
\end{widedisorder}
\end{proof}
\begin{cor}
Let $A$, $B$ be sets, $R\in\subsets\mathbf{Rel}(A,B)$.
\begin{enumerate}
\item \textbf{$\rsupfun{\bigsqcup R}X=\bigsqcup_{f\in R}\rsupfun fX$} for
$X\in\mathscr{T}A$;
\item $\rsupfun{\bigsqcap R}x=\bigsqcap_{f\in R}\rsupfun fx$ for atomic
$x\in\mathscr{T}A$;
\item $X\rsuprel{\bigsqcup R}Y\Leftrightarrow\exists f\in R:X\rsuprel fY$
for $X\in\mathscr{T}A$, $Y\in\mathscr{T}B$;
\item $x\rsuprel{\bigsqcap R}y\Leftrightarrow\forall f\in R:x\rsuprel fy$
for every atomic $x\in\mathscr{T}A$, $y\in\mathscr{T}B$.
\end{enumerate}
\end{cor}
\begin{prop}
$X\rsuprel{g\circ f}Z\Leftrightarrow\exists\beta:(X\rsuprel f\{\beta\}\land\{\beta\}\rsuprel gZ)$
for every binary relation~$f$ and sets $X$ and $Y$.\end{prop}
\begin{proof}
~
\begin{multline*}
X\rsuprel{g\circ f}Z\Leftrightarrow\exists x\in X,z\in Z:x\mathrel{(g\circ f)}z\Leftrightarrow\\
\exists x\in X,z\in Z,\beta:(x\mathrel f\beta\land\beta\mathrel gz)\Leftrightarrow\\
\exists\beta:(\exists x\in X:x\mathrel f\beta\land\exists y\in Y:\beta\mathrel gz)\Leftrightarrow\exists\beta:(X\rsuprel f\{\beta\}\land\{\beta\}\rsuprel gZ).
\end{multline*}
\end{proof}
\begin{cor}
$X\rsuprel{g\circ f}Z\Leftrightarrow\exists y\in\atoms^{\mathscr{T}B}:(X\rsuprel fy\land y\rsuprel gZ)$
for $f\in\mathbf{Rel}(A,B)$, $g\in\mathbf{Rel}(B,C)$ (for sets $A$,
$B$, $C$).\end{cor}
\begin{prop}
$f\circ\bigcup G=\bigcup_{g\in G}(f\circ g)$ and $\bigcup G\circ f=\bigcup_{g\in G}(g\circ f)$
for every binary relation~$f$ and set~$G$ of binary relations.\end{prop}
\begin{proof}
We will prove only $\bigcup G\circ f=\bigcup_{g\in G}(g\circ f)$
as the other formula follows from duality. Really

\begin{multline*}
(x,z)\in\bigcup G\circ f\Leftrightarrow\exists y:((x,y)\in f\land(y,z)\in\bigcup G)\Leftrightarrow\\
\exists y,g\in G:((x,y)\in f\land(y,z)\in g)\Leftrightarrow\exists g\in G:(x,z)\in g\circ f\Leftrightarrow(x,z)\in\bigcup_{g\in G}(g\circ f).
\end{multline*}
\end{proof}
\begin{cor}
Every $\mathbf{Rel}$-morphism is metacomplete and co-metacomplete.\end{cor}
\begin{prop}
\label{rel-mono}The following are equivalent for a $\mathbf{Rel}$-morphism~$f$:
\begin{enumerate}
\item \label{rel-mono-mono}$f$ is monovalued.
\item \label{rel-mmv}$f$ is metamonovalued.
\item \label{rel-wmmv}$f$ is weakly metamonovalued.
\item \label{rel-mono-atom}$\rsupfun fa$ is either atomic or least whenever
$a\in\atoms^{\mathscr{T}\Src f}$.
\item \label{rel-mono-bin}$\rsupfun{f^{-1}}(I\sqcap J)=\rsupfun{f^{-1}}I\sqcap\rsupfun{f^{-1}}J$
for every $I,J\in\mathscr{T}\Src f$.
\item \label{rel-mono-meet}$\rsupfun{f^{-1}}\bigsqcap S=\bigsqcap_{Y\in S}\rsupfun{f^{-1}}Y$
for every $S\in\subsets\mathscr{T}\Src f$.
\end{enumerate}
\end{prop}
\begin{proof}
~
\begin{description}
\item [{\ref{rel-mmv}$\Rightarrow$\ref{rel-wmmv}}] Obvious.
\item [{\ref{rel-mono-mono}$\Rightarrow$\ref{rel-mmv}}] Take $x\in\atoms^{\mathscr{T}\Src f}$;
then $fx\in\atoms^{\mathscr{T}\Dst f}\cup\{\bot^{\mathscr{T}\Dst f}\}$ and thus
\begin{multline*}
\rsupfun{\left(\bigsqcap G\right)\circ f}x=\rsupfun{\bigsqcap G}\rsupfun fx=\bigsqcap_{g\in G}\rsupfun g\rsupfun fx=\\
\bigsqcap_{g\in G}\rsupfun{g\circ f}x=\rsupfun{\bigsqcap_{g\in G}(g\circ f)}x;
\end{multline*}
so $\left(\bigsqcap G\right)\circ f=\bigsqcap_{g\in G}(g\circ f)$.
\item [{\ref{rel-wmmv}$\Rightarrow$\ref{rel-mono-mono}}] Take $g=\{(a,y)\}$
and $h=\{(b,y)\}$ for arbitrary $a\ne b$ and arbitrary~$y$. We
have $g\cap h=\emptyset$; thus $(g\circ f)\cap(h\circ f)=(g\cap h)\circ f=\bot$
and thus impossible $x\mathrel fa\land x\mathrel fb$ as otherwise
$(x,y)\in(g\circ f)\cap(h\circ f)$. Thus $f$ is monovalued.
\item [{\ref{rel-mono-atom}$\Rightarrow$\ref{rel-mono-meet}}] Let $a\in\atoms^{\mathscr{T}\Src f}$,
$\rsupfun fa=b$. Then because $b\in\atoms^{\mathscr{T}\Dst f}\cup\{\bot^{\mathscr{T}\Dst f}\}$
\begin{gather*}
\bigsqcap S\sqcap b\ne\bot\Leftrightarrow\forall Y\in S:Y\sqcap b\ne\bot;\\
a\rsuprel f\bigsqcap S\Leftrightarrow\forall Y\in S:a\rsuprel fY;\\
\bigsqcap S\rsuprel{f^{-1}}a\Leftrightarrow\forall Y\in S:Y\rsuprel{f^{-1}}a;\\
a\nasymp\rsupfun{f^{-1}}\bigsqcap S\Leftrightarrow\forall Y\in S:a\nasymp\rsupfun{f^{-1}}Y;\\
a\nasymp\rsupfun{f^{-1}}\bigsqcap S\Leftrightarrow a\nasymp\bigsqcap_{Y\in S}\rsupfun{f^{-1}}Y;\\
\rsupfun{f^{-1}}\bigsqcap S=\bigsqcap_{X\in S}\rsupfun{f^{-1}}X.
\end{gather*}

\item [{\ref{rel-mono-meet}$\Rightarrow$\ref{rel-mono-bin}}] Obvious.
\item [{\ref{rel-mono-bin}$\Rightarrow$\ref{rel-mono-mono}}]
$\rsupfun{f^{-1}}a\sqcap\rsupfun{f^{-1}}b=\rsupfun{f^{-1}}(a\sqcap b)=\rsupfun{f^{-1}}\bot=\bot$
for every two distinct atoms $a=\{\alpha\},b=\{\beta\}\in\mathscr{T}\Dst f$.
From this
\begin{multline*}
\alpha\mathrel{(f\circ f^{-1})}\beta\Leftrightarrow\exists y\in\Dst f:(\alpha\mathrel{f^{-1}}y\land y\mathrel{f}\beta)\Leftrightarrow\\
\exists y\in\Dst f:(y\in\rsupfun{f^{-1}}a\land y\in\rsupfun{f^{-1}}b)
\end{multline*}
 is impossible. Thus $f\circ f^{-1}\sqsubseteq1_{\Dst f}^{\mathbf{Rel}}$.
\item [{$\lnot$\ref{rel-mono-atom}$\Rightarrow\lnot$\ref{rel-mono-mono}}] Suppose
$\rsupfun fa\notin\atoms^{\mathscr{T}\Dst f}\cup\{\bot^{\mathscr{T}\Dst f}\}$
for some $a\in\atoms^{\mathscr{T}\Src f}$. Then there exist distinct
points $p$, $q$ such that $p,q\in\rsupfun fa$. Thus $p\mathrel{(f\circ f^{-1})}q$
and so $f\circ f^{-1}\nsqsubseteq1_{\Dst f}^{\mathbf{Rel}}$.
\end{description}
\end{proof}

\section{Product of typed sets}
\begin{defn}
Product of typed sets is defined by the formula
\[
(\subsets U,A)\times(\subsets W,B)=(U,W,A\times B).
\]
\end{defn}
\begin{prop}
Product of typed sets is a $\mathbf{Rel}$-morphism.\end{prop}
\begin{proof}
We need to prove $A\times B\subseteq U\times W$, but this is obvious.\end{proof}
\begin{obvious}
Atoms of $\mathbf{Rel}(A,B)$ are exactly products $a\times b$ where
$a$ and $b$ are atoms correspondingly of $\mathscr{T}A$ and $\mathscr{T}B$.
$\mathbf{Rel}(A,B)$ is an atomistic poset.\end{obvious}
\begin{prop}
$f\nasymp A\times B\Leftrightarrow A\rsuprel fB$ for every $\mathbf{Rel}$-morphism~$f$
and $A\in\mathscr{T}\Src f$, $B\in\mathscr{T}\Dst f$.\end{prop}
\begin{proof}
~
\begin{multline*}
A\rsuprel fB\Leftrightarrow\exists x\in\atoms A,y\in\atoms B:x\rsuprel fy\Leftrightarrow\\
\exists x\in\atoms^{\mathscr{T}\Src f},y\in\atoms^{\mathscr{T}\Dst f}:(x\times y\sqsubseteq f\land x\times y\sqsubseteq A\times B)\Leftrightarrow f\nasymp A\times B.
\end{multline*}
\end{proof}
\begin{defn}
\emph{Image} and \emph{domain} of a $\mathbf{Rel}$-morphism~$f$
are typed sets defined by the formulas
\[
\dom(U,W,f)=(\subsets U,\dom f)\quad\text{and}\quad\im(U,W,f)=(\subsets W,\im f).
\]
\end{defn}
\begin{obvious}
Image and domain of a $\mathbf{Rel}$-morphism are really typed sets.\end{obvious}
\begin{defn}
\emph{Restriction} of a $\mathbf{Rel}$-morphism to a typed set is
defined by the formula $(U,W,f)|_{(\subsets U,X)}=(U,W,f|_{X})$.\end{defn}
\begin{obvious}
Restriction of a $\mathbf{Rel}$-morphism is $\mathbf{Rel}$-morphism.
\end{obvious}

\begin{obvious}
$f|_{A}=f\sqcap(A\times\top^{\mathscr{T}\Dst f})$ for every $\mathbf{Rel}$-morphism~$f$
and $A\in\mathscr{T}\Src f$.
\end{obvious}

\begin{obvious}
$\rsupfun fX=\rsupfun f(X\sqcap\dom f)=\im(f|_{X})$ for every $\mathbf{Rel}$-morphism~$f$
and $X\in\mathscr{T}\Src f$.
\end{obvious}

\begin{obvious}
$f\sqsubseteq A\times B\Leftrightarrow\dom f\sqsubseteq A\land\im f\sqsubseteq B$
for every $\mathbf{Rel}$-morphism~$f$ and $A\in\mathscr{T}\Src f$,
\textbf{$B\in\mathscr{T}\Dst f$}.\end{obvious}
\begin{thm}
Let $A$, $B$ be sets. If $S\in\subsets(\mathscr{T}A\times\mathscr{T}B)$
then
\[
\bigsqcap_{(A,B)\in S}(A\times B)=\bigsqcap\dom S\times\bigsqcap\im S.
\]
\end{thm}
\begin{proof}
For every atomic $x\in\mathscr{T}A$, $y\in\mathscr{T}B$ we have
\begin{multline*}
x\times y\sqsubseteq\bigsqcap_{(A,B)\in S}(A\times B)\Leftrightarrow\forall(A,B)\in S:x\times y\sqsubseteq A\times B\Leftrightarrow\\
\forall(A,B)\in S:(x\sqsubseteq A\land y\sqsubseteq B)\Leftrightarrow\forall A\in\dom S:x\sqsubseteq A\land\forall B\in\im S:y\sqsubseteq B\Leftrightarrow\\
x\sqsubseteq\bigsqcap\dom S\land y\sqsubseteq\bigsqcap\im S\Leftrightarrow x\times y\sqsubseteq\bigsqcap\dom S\times\bigsqcap\im S.
\end{multline*}
\end{proof}
\begin{obvious}
If $U$, $W$ are sets and $A\in\mathscr{T}(U)$ then $A\times$ is
a complete homomorphism from the lattice $\mathscr{T}(W)$ to the
lattice $\mathbf{Rel}(U,W)$, if also $A\ne\bot$
then it is an order embedding.\end{obvious}