\section{Status and Problems} \label{sec:statusAndProblems}

\subsection{Status of the implementation} \label{sec:status}

At the time this technote is in writing, March 2019, only the \textbf{lsst\_distrib} repository is released and distributed.


\subsubsection{Packaging} \label{sec:statusPkgs}

The following technologies are already used in DM for packaging:

\begin{itemize}
\item {\bf Eups}: for the majority of DM packages. It is used also for all third party libraries that require customization, or that are not available in the conda channels. The lsst DM binary Eups repository is hosted at \url{https://eups.lsst.codes/}. 
\item {\bf Sonatype Nexus}: for java based software packages.
\item {\bf Conda}: for packages publicly available, that can be used without any customization
\item {\bf Pip/PiPy}: for python packages publicly available.
\end{itemize}


\subsubsection{Distribution} \label{sec:statusDistrib}

The main tool used for distribution to operations is \textbf{Docker}. 

The Science Pipelines distribution to the science comunity is mainly provided using the script \textit{newinstall.sh}. 
This scripts permits to retrieve a specific Science Pipelines build or release available in the Eups repository, and setup the environment for its build and execution.

In \url{pipelines.lsst.io} instructions are provided on how to deploy a Science Pipelines using \textit{newinstall.sh} or \textbf{Docker}.


\subsubsection{SW Products Identification} \label{sec:statusIdentification}

No software products as described in the product tree in \citeds{LDM-294} have official releases so far. 

Only the Science Pipelines distribution, identified by the \textit{lsst\_distrib} Git mata-package, has regular official releases.
All explicit and implicit dependencies in \textit{lsst\_distrib} which team is \textbf{Data Management} or \textbf{DM Externals} are considered part of the distribution.

The following Git teams are relevant for the identification of the Science Pipelines distribution:

\begin{itemize}
\item the \textbf{Data Management} team identifies all DM developed software included in the distribution.
\item the \textbf{DM Externals} team identifies third-party libraries required by \textit{lsst\_distrib}.
These are software packages developed outside DM, that are not available in public conda channels, or that are updated often and therefore can't be included in the conda environment definition (\ref{sec:statusEnvs}).
See draft \citeds{DMTN-110} for current problems and possible solutions on conda environemnts.
\end{itemize}

A third team, \textbf{DM Auxilliaries}, identifies auxiliary packages to be tagged when a release or build of the distribution is done, but they are not part of it, therefore not distributed with it.


\subsubsection{Environment} \label{sec:statusEnvs}

The conda environment used for building the Science Pipelines is defined in the Git repository \textit{scipipe\_conda\_env}.


\subsubsection{Other Tools} \label{sec:statusTools}

The build tools used to build Science Pipelines software are available in \textit{lsstsw} and \textit{lsst\_build} Git repositories.

The Jenkins scripts also are an important part of the tooling, facilitating a large number of actions, like, test a ticket branch, automate releases, or provide periodic build (weekly/daily).

The tools picture is completed by \textit{codekit}, which permits interaction with multiple Git repositories.
Just as an example, it is used to create the same tag on all the Git repositories composing a defined product automatically, instead of creating it manually.


\subsection{Open Problems} \label{sec:openProblems}

In this section, a list of problems derived from the current development approach is enumerated. 

While these problems are unresolved the only release process suitable for DM is to release the entire codebase each time.

Their resolution may lead to a different development approach, and therefore the proposed release procedure may need to be reviewed accordingly.


\subsubsection{SW Product Composition} \label{sec:problemId}

A Git repository may be included in more than one software product.

This makes impossible to apply in a consistent way the release procedure to an SW product that is not the Science Pipelines.
The reason is explained in the following example.

Product A is composed of 100 Git repositories.

Product B is composed of 80 Git repositories.

70 Git repositories are shared by both products.

When release 1.0 is done on product A, all 100 repositories will be tagged, and corresponding Eups packages are created. This will include the 70 shared repositories.

One week later, product B is ready for the release 1.0, and some of the 70 shared repositories have been updated.
For them, the 1.0 tag required for product B release, will not be the same as the 1.0 tag required for the product A release.
This implies that for some repositories, release 1.0 of product A, is different from release 1.0 of product B.

\textbf{Requirement}: there should be a 1 to 1 correspondence between software products and Git repository. 
If this is not possible, each Git repository shall be included in only one software product (one SW product to many Git repositories).
Note also that, each Git repository used for building, unit testing and packaging the software products shall not be included in the SW product itself, but versioned separately and be part of the environment definition. See section \ref{sec:swprod}.


\subsubsection{Code Fragmentation} \label{sec:problemCode}

The high number of repositories causes problems in that it :

\begin{itemize}
\item increases the build time: each Git repository needs to build each time. 
This imply that extra activities need to be done, such as clone, checkout, package creation and push to the \url{eups.lsst.code} repository.
All these activities are not time consuming on itself, and may not represent a problem is just one repository is added.
However, if the number of repositories included in a software product grows without control, the overall build time will become a problem.
\item increases the release time: all Git repositories need to be built and released each time.
As for the build time, adding one repository to the software product, will not represent a problem in terms of time.
However, if the number of repositories grows without control, the overall time required to do a release will become a problem, especially for those software products that require patches to be available very quick.
\item increases the failure probability: the tooling may be affected by network glitch or similar technical issues. 
This may lead to the failure of the build/release process in a non-deterministic way.
\end{itemize}

Moving 3rd party libraries to conda environment will mitigate these problem. 
However, this requires proper management of the conda environment. See draft \citeds{DMTN-110}.

\textbf{Requirement}: the number of Git repositories shall be kept low. DM-CCB shall approve each time a new repository is introduced since this has an impact on the maintainability of the system. 
All third party libraries, shall also not be part of the stack, if not requiring source code changes.


\subsubsection{Binaries Persistence} \label{sec:problemPersistence}

EUPS builds a Git repository and installs the binary packages locally. 
If a Git repository is not affected by any changes, once installed locally by EUPS, it will not be rebuilt.

Continuous integration tools instead are making available, in the remote repository at \url{https://eups.lsst.codes/}, the binary packages for macOS and Linux platforms.
However, the build tools in lsstsw are not able to resolve the binary packages from that remote repository.
Therefore, each time the Science Pipelines is built from scratch, all Git repositories are rebuilt.

A Git repository should be rebuilt, only in case its source code has changed, or one of its dependencies has changed (in case of semantic versioning, only for breaking changes that increase the major version).

\textbf{Requirement}: it shall be possible to resolve a package binaries from \url{https://eups.lsst.codes/} if available, instead of building it from the source code.

This problem is not blocking on the release process, but its resolution permits a better optimization of the builds.


