Releasing on Conda

If you check the Astrorama GitHub Organization you will see there is a *-feedstock repository per component. Each one will have one master branch for the production release, and additional branches - as develop - if we want to distribute pre-releases for testing. Nothing much changes either way, just make sure to use the proper branch depending on what you want to build.

Builds are automated, so every time there is a push, Azure will pick the changes, build and upload to Anaconda.org the new package (unless you add [skip ci] to the commit message!).

Install tooling for conda

First, we need to make sure we have all the required utilities for making a release. We assume conda and git are at least installed. Let’s activate conda if it isn’t already:

1
2
# Replace bash with your shell of choice: e.g., shell.zsh
eval "$(${HOME}/miniconda3/bin/conda shell.bash hook)"

We consider good practice to have the tooling on its own environment. Let’s call it build. We need to install, at the very least, conda-smithy. It will also pull conda-build, which is a useful tool:

1
2
conda create -n build conda-smithy
conda activate build

*-feedstock content

Let’s pick Phosphoros as an example. First, make sure you have the feedstock checked out locally

1
2
git clone git@github.com:astrorama/phosphoros-feedstock.git
cd phosphoros-feedstock

Let’s do an ls -a1 an examine the content:

.azure-pipelines
.ci_support
.circleci
.git
.gitattributes
.github
.gitignore
.scripts
azure-pipelines.yml
build-locally.py
build_artifacts
conda-forge.yml
README.md
recipe

Most of these files are handled by conda-smithy and we do not need to worry about them. We care about conda-forge.yml and everything inside recipe. For the former, we refer to the official documentation.

For the later, we can also examine its content:

build.sh
conda_build_config.yaml
cross-linux.cmake
meta.yaml
phosphoros_64.png
trigger_compiler_id.patch

The main file is meta.yaml. It defines the package name, version, build number, origin for the sources, build and run-time dependencies, test command and general metadata.

build.sh contains the build instructions (i.e. invoke cmake with the right flags). conda_build_config.yaml lists the channels used to build (channel_sources) and the channel where to upload the result channel_targets. *.patch files are applied to the sources before configuration and build.

In the particular case of Phosphoros, we define an “app” that can be run from the Anaconda launcher. Its icon is phosphoros_64.png.

Releasing

Now everything is setup, we can try to do a release. First, let’s make sure everything is up to date:

1
2
git checkout develop && git pull
git checkout master && git pull

Let’s suppose we are about to release Phosphoros 1.4.0, and that the building of the develop branch is successful. We can modify directly the files under master, or we can do first a merge from develop so the dependencies are aligned:

1
git merge develop

Now, we need to modify meta.yaml, and update the head of the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{% set name = "Phosphoros" %}
{% set version = "1.4.0" %}       # Must align with a released tag
{% set build_number = 1 %}        # Start at 1 for a new release

package:
    name: {{ name|lower }}
    version: {{ version }}

source:
    git_rev: {{ version }}        # Make sure we are checking out the tag!
    git_url: https://github.com/astrorama/Phosphoros.git
    patches:
      - trigger_compiler_id.patch # Review if the patches are still needed

build:
    number: {{ build_number }}

Once the head is is right, cross-check the dependencies:

1
2
3
4
5
6
7
8
9
host:
    - python
    - boost-cpp
    - phosphoroscore ==1.4.0     # Need to depend on a released (and built) version!
    - qt >=5.12,<6
    - cfitsio >=3.470
    - CCfits >=2.5
    - lxml >=4.5
    - pybind11 >=2.6

1
2
3
4
5
build:
    number: {{ build_number }}
    run_exports:
      - {{ pin_subpackage('alexandria', max_pin='x.x.x') }}

Then, verify that conda_build_config.yaml tells conda to install the dependencies from the main tag only. Otherwise, we may build a release depending on a development built:

1
2
3
4
5
channel_sources:
  - astrorama,conda-forge,defaults

channel_targets:
  - astrorama main

Finally, we need to refresh all the files handled by conda-smithy:

1
conda smithy rerender

Generally this is enough to do a release. Depending on the changes, you may need to modify build.sh to pass appropriate flags to cmake. Once all is done, just git add, git commit, and git push. It is advisable to also create and push a tag (i.e. 1.4.0-1).

1
conda build -c astrorama -c conda-forge -m .ci_support/linux_64_python3.9.____cpython.yaml  recipe/

Rebuilding

From time to time, Conda Forge may upgrade the compiler version for osx (clang), for Linux (gcc) or both. Typically, this can break a release build since the (Element-based) dependencies were build with an older compiler than the just-released software.

The error will look like:

1
2
3
4
CMake Error at /usr/lib64/cmake/ElementsProject/ElementsProjectConfig.cmake:1567 (message):
  Incompatible values of SGS_SYSTEM:
    SourceXtractorPlusPlus -> x86_64-fc37-gcc122
    Alexandria 2.25.0 -> x86_64-fc37-gcc121

When this happens, everything will have to be re-built in order:

1. elements-feedstock
2. alexandria-feedstock
3. sourcextractor-feedstock / phosphoroscore-feedstock -> phosphoros-feedstock

For rebuilding, we can start directly on master, and not bother with the merge, nor the version bump. We just need to:

1. Bump the build number
2. Run conda smithy rerender

We commit and push the changes, create a new tag (i.e. 2.25.0-2) and push it too. Azure will pick the changes and rebuild everything. Once a component is done, we can follow with the dependent packages.