1.1   Switching Compilers¶

Switching to conda-forge dependencies means an end to devtoolset compilers and system dependencies for centos7 deployments, due to ABI breakage - centos7+devtoolset requires the pre-5.1 GCC ABI. So the options, for centos7 for example, are to either require a user to install system dependencies and a compiler themselves or get them from conda-forge, or through other non-standard channels. Retrieving those dependencies from conda-forge would be the most user-friendly way.

Matthew Becker merged conda compiler support into sconsUtils, so using conda compilers should work just fine, although they are not battle-tested. Testing with conda compilers on linux has occurred since July, and things tend to work just fine with that. Some issues with Mac compilers have occured in the past but are believe to be resolved now. Further testing will occur before this proposal is accepted.

Currently, if a user is using a newer system with the native compilers, or a newer compilers on an older system, there may still be issues around using those compilers even with the conda-forge third parties. In any case - conda compiler or newer compiler on top of conda - additional testing will still be required to ensure that builds work in all cases, including when compilers newer than those of conda are installed on the system outside conda.

2.1   End users consume releases with Stackvana¶

# conda already installed and conda-forge already configured
$ conda create -n lsst-20.1.0 stackvana=20.1.0
# lsst_distrib already activated
(lsst-20.1)$ eups list

Matthew Becker has created the `stackvana <https://github.com/beckermr/stackvana>`__ ecosystem, a collection of conda recipes, for building and distributing the stack. This ecosystem is oriented around a modified eups distrib install workflow that consumes stack releases, ideally within the conda-forge CI system. Users get an extremely simplified installation, as the stack is distributed similarly to other conda-forge package. The target for this use case, at least within the context of conda-forge, is really for releases only, which probably does not include weekly or daily builds, as these are not LSST releases, and conda-forge CI resources are relatively precious. It’s important to note that conda-forge states that it is not suitable as a target for rapid development. As is stated in their orgnazation guidelines:

Publishing a package to conda-forge signals it is suitable for users not involved with development. However, publishing does not always happen error-free. Multiple commits are acceptable when debugging issues with the release process itself.

As such, it would probably be prudent to discuss release cadence of the stack to meet the needs of the users consuming stackvana installs, or rely on another channel [1].

2.2   Developers start with a toolset environment¶

Installing a daily build:

# conda installed and conda-forge already configured, conda activated
# lsst_toolset_env is metapackage - eups, lsst_build, repos, scons
(base)$ conda create -n lsst-toolset lsst_toolset_env
(base)$ conda activate lsst-toolset
# If the previous command is the first activation, activate.d scripts
# could run `lsst-build config init`, which can setup repos/versiondb
(lsst-toolset)$ eups distrib install -t d_latest lsst_distrib
# A new conda environment was created as part of installing
# scipipe_conda_env.
(lsst-toolset)$ setup lsst_distrib
(scipipe-conda-env-1234abcde)$

Development:

(base)$ conda create -n lsst-toolset lsst_toolset_env
# lsst_toolset_env's activate.d scripts executed
(base)$ conda activate lsst-toolset
(lsst-toolset)$ cd ~/workspace/lsstsw
(lsst-toolset)$ rebuild afw
# internally - rebuild uses lsst-build to `prepare` and `build`
# scipipe_conda_env is prepped, config, installed, declared first
# Following that is the lsst-build build script for base, like so:
# > (lsst-toolset)$ eupspkg PRODUCT=base ... prep
# > # Note: scipipe_conda_env is setup via _build.tags
# > (lsst-toolset)$ setup --vro=_build.tags -r .
# > (scipipe-conda-env-1234abcde)$ eupspkg PRODUCT=base ... config
# ... the rest of lsst-build's build script for base is executed

This workflow is optimized around both eups and lsst-build workflows. This includes tasks such as the installation of weekly or daily builds via eups distrib install, and local development via lsstsw and rebuild.

Developers start with a toolset environment that packages eups, lsst-build, git, git-lfs, compilers, and scripts to configure data for lsst/repos and lsst/versiondb. The installed toolset environment also contains the eups database, with the installed eups managing conda environments with the help of environment stacking. The conda environment is an eups package based on Nate Lust’s scipipe_conda package and scipipe_conda_env repo. Nate’s code originally went one step further in relying on another miniconda eups package, so conda itself was installed with eups. It is felt that is not ideal as we still need a suitable python environment for eups itself, and for path length reasons explained in detail in [2]. One thing to note that running setup -r . on the directory will not activate a complete environment - the environment is always installed as part of the install phase of eupspkg.

Nate’s code had relied on manipulating environment variables in the table file as conda does, by setting CONDA_PREFIX, appending paths, etc… However, when conda is activated, conda is a shell function intended to be a wrapper over the actual conda CLI, modifying the environment as necessary when switching between conda environments or installing dependencies, as setup does as well, so this breaks or may have some funny side effects (bad $PS1) when not directly using conda activate or conda deactivate. Notably, if conda is not activated but the paths have been modified so that you can find the conda binary - running a successive conda activate will fail, notifying you with an error:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

To address this, we can add new actions for an eups table file:

• condaActivate(...)
• condaActivateStacked(...)

These actions will emit conda activate {env_name} and conda deactivate statements which get evaluated by running setup. The assumption here is that conda has been activated, which it should have been if you can use eups. The second action, condaActivateStacked is the one we actually need, but it is proposed to add both to eups, as the semantics are slightly different.

The lsst_toolset_env package will be derived from the stackvana-core build recipe, which packages up eups.

With scipipe_conda_env as an eups package, stackvana will possibly need an eups remap file for that package.

Relatedly, in both the stackvana case and the developer case, there is one important caveat to note by having base depend on scipipe_conda_env. If a product, such as qserv or sims, wishes to define its environment, but relies a dependency which relies on base, they will likely need to either remap scipipe_conda_env with their environment, or produce their version of scipipe_conda_env and rely on version restrictions in their table file on scipipe_conda_env. There may be some other solution that hasn’t been identified. This isn’t any different than the current situation with scipipe_conda_env.

#!/bin/bash
# re-init conda functions (conda/conda#7753)
CONDA_EXE_ROOT=$(dirname $(dirname $CONDA_EXE))
source $CONDA_EXE_ROOT/etc/profile.d/conda.sh
# The following will pick up the system git
echo "git in scipipe-env environment (conda run isolated environment)..."
conda run -n scipipe-env git --version
# The following will pick up git from the lsst-toolset environment
echo "git from scipipe-env environment (stacked environments)..."
conda activate --stack scipipe-env
git --version

$ git --version
git version 2.20.1 (Apple Git-117)
$ source /opt/conda/bin/activate lsst-toolset
(lsst-toolset)$ ./stacking.sh
Stacking example
git in scipipe-env environment (conda run isolated environment)...
git version 2.20.1 (Apple Git-117)
git from scipipe-env environment (stacked environments)...
git version 2.23.0

# write new config file. Stored at $CONDA_PREFIX/etc/lsst-build/
(lsst-toolset)$ lsst-build config init
# sync config - download lsst/repos/etc/repos.yaml, lsst/versiondb
(lsst-toolset)$ lsst-build config sync
# Switch repos. Equivalent to:
# cd $CONDA_PREFIX/share/lsst/repos; git checkout tickets/DM-98765
(lsst-toolset)$ lsst-build config write repos-ref tickets/DM-98765
(lsst-toolset)$ lsst-build config sync