24. Building and releasing ASP¶
This chapter will describe how ASP can be built without and with using conda, how to build the documentation, and how to prepare a new ASP release. This is focused towards the developer. Users should read instead the installation guide in Section 2.
24.1. Building ASP without conda¶
This entails downloading all the ASP dependencies with conda first as pre-compiled binaries, then pulling the VisionWorkbench and Stereo Pipeline source code from GitHub, and building locally. This is suggested only for the very adventurous user.
The environments having the ASP dependencies are in the conda
directory of the Stereo Pipeline repository, as above. After
downloading those, one can run on Linux:
conda env create -n asp_deps -f asp_deps_3.3.0_linux_env.yaml
or on the Mac:
conda env create -n asp_deps -f asp_deps_3.3.0_osx_env.yaml
This will create an asp_deps
environment. Activate it with:
conda activate asp_deps
Some of the .la files created by conda point to other .la files that are not available. For that reason, those files should be edited to replace:
/path/to/libmylibrary.la
with:
-L/path/to -lmylibrary
This can be done with the following commands:
cd ~/miniconda3/envs/asp_deps/lib
mkdir -p backup
cp -fv *.la backup # back these up
perl -pi -e "s#(/[^\s]*?lib)/lib([^\s]+).la#-L\$1 -l\$2#g" *.la
The conda-provided compilers are used and should be installed in the environment, if not present already. The following command could be used:
conda install -c conda-forge compilers
Ensure that cmake>=3.15.5
and pbzip2
are installed. This can be done
as:
conda install -c conda-forge cmake pbzip2
For Linux only, install the chrpath
tool.
Set the compiler names. They may differ somewhat from what is in the block below, so this step may need some adjustments.
isMac=$(uname -s | grep Darwin)
if [ "$isMac" != "" ]; then
cc_comp=clang
cxx_comp=clang++
else
cc_comp=x86_64-conda_cos6-linux-gnu-gcc
cxx_comp=x86_64-conda_cos6-linux-gnu-g++
fi
Set up a work directory:
buildDir=$HOME/build_asp
mkdir -p $buildDir
Build VisionWorkbench and Stereo Pipeline:
cd $buildDir
envPath=$HOME/miniconda3/envs/asp_deps
$envPath/bin/git clone \
git@github.com:visionworkbench/visionworkbench.git
cd visionworkbench
# Uncomment below if desired to build a specific version
# git checkout 3.3.0
mkdir -p build
cd build
$envPath/bin/cmake .. \
-DASP_DEPS_DIR=$envPath \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DCMAKE_INSTALL_PREFIX=$buildDir/install \
-DCMAKE_C_COMPILER=${envPath}/bin/$cc_comp \
-DCMAKE_CXX_COMPILER=${envPath}/bin/$cxx_comp
make -j10 && make install
cd $buildDir
envPath=$HOME/miniconda3/envs/asp_deps
$envPath/bin/git clone \
git@github.com:NeoGeographyToolkit/StereoPipeline.git
cd StereoPipeline
# Uncomment below if desired to build a specific version
# git checkout 3.3.0
mkdir -p build
cd build
$envPath/bin/cmake .. \
-DASP_DEPS_DIR=$envPath \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DCMAKE_INSTALL_PREFIX=$buildDir/install \
-DVISIONWORKBENCH_INSTALL_DIR=$buildDir/install \
-DCMAKE_C_COMPILER=${envPath}/bin/$cc_comp \
-DCMAKE_CXX_COMPILER=${envPath}/bin/$cxx_comp
make -j10 && make install
24.2. Building ASP and its dependencies with conda¶
This page is meant for advanced users of ASP and maintainers who would like to use conda to rebuild ASP and all its dependencies. It is suggested to carefully read Section 2.3 before this page.
To simplify maintenance, ASP and its dependencies are built upon ISIS and its dependencies. Hence, in order to create a new conda ASP package, first one needs to create an environment having the latest released ISIS, then rebuild ASP’s other dependencies and ASP itself, while ensuring that the dependencies of each of these have their versions synced up with the ISIS dependency versions.
The rebuilt packages will be uploaded to the nasa-ames-stereo-pipeline
anaconda channel.
24.2.1. Setting up the ISIS environment¶
Search for the latest available ISIS conda package:
conda search -c usgs-astrogeology --override-channels isis
Here it was found that ISIS version 8.0.0 was the latest, which we will assume throughout the rest of this document. This needs to be adjusted for your circumstances.
Create a conda environment for this version of ISIS:
conda create -n isis8.0.0
conda activate isis8.0.0
Add these channels to conda:
conda config --env --add channels conda-forge
conda config --env --add channels usgs-astrogeology
Run:
conda config --show channels
and verify that usgs-astrogeology
and conda-forge
are in this
order and above all other channels, except perhaps the
nasa-ames-stereo-pipeline
channel.
Install the desired version of ISIS:
conda install isis==8.0.0
Search and install the latest version of the usgscsm
package,
for example, as:
conda search -c conda-forge --override-channels usgscsm
conda install -c conda-forge usgscsm==1.7.0
If that package is too old, consider rebuilding it, following the recipe at:
See Section 24.2.3 for how to fetch and build this.
Save the current environment as follows:
conda env export > isis8.0.0.yaml
24.2.2. Fetching the build tools¶
We will create a new tools
environment to have all the tools we
will need. These could be appended to the earlier environment, but it
is less likely to to have issues with dependency conflicts if these
are kept separate.
conda create -n tools
conda activate tools
conda install -c conda-forge anaconda-client conda-build \
conda-verify
For local builds it is necessary to also fetch the compilers (for C, C++, and Fortran), which can be done with:
conda install -c conda-forge compilers
24.2.3. Packages to build¶
Many additional package need to be built, using conda build
. These packages
can be downloaded with git clone
from:
https://github.com/NeoGeographyToolkit/geoid-feedstock.git https://github.com/NeoGeographyToolkit/htdp-feedstock.git https://github.com/NeoGeographyToolkit/laszip-feedstock.git https://github.com/NeoGeographyToolkit/fgr-feedstock.git https://github.com/NeoGeographyToolkit/libnabo-feedstock.git https://github.com/NeoGeographyToolkit/libpointmatcher-feedstock.git https://github.com/NeoGeographyToolkit/gdal-feedstock.git https://github.com/NeoGeographyToolkit/liblas-feedstock.git https://github.com/NeoGeographyToolkit/s2p-feedstock.git https://github.com/NeoGeographyToolkit/libelas-feedstock.git https://github.com/NeoGeographyToolkit/multiview-feedstock https://github.com/NeoGeographyToolkit/visionworkbench-feedstock.git https://github.com/NeoGeographyToolkit/stereopipeline-feedstock.git
Also, per the earlier note, consider rebuilding usgscsm
if
there there are updates in its GitHub repository which are not yet
released on conda-forge.
24.2.4. Synchronize the versions with the existing environment¶
For each of the above feedstocks, check the recipe/meta.yaml
file
and ensure all dependencies are in sync with what is in the file
isis8.0.0.yaml
generated earlier. This can be done automatically
with a provided script in the ASP repository:
python StereoPipeline/conda/update_versions.py isis8.0.0.yaml \
gdal-feedstock
and the same for the other packages.
It is very important to note that this script is not fool-proof, and the changes it makes should be very carefully examined.
It is suggested to examine the changed meta.yaml
, and if in doubt,
leave the values as they were before modified by this script.
In the visionworkbench
and stereopipeline
recipes update the
git_tag
value to reflect the desired commit from the Git
history.
When making an ASP release, one can tag the commit based on
which the release happens in the VisionWorkbench and StereoPipeline
repositories, and then that tag can be used in the git_tag
field.
See Section 24.4 for more details.
Later on, after the packages are built and tested, ensure that all the changes to the feedstock repositories are checked in.
24.2.5. Build the conda packages¶
When building a package that depends on other packages in the
nasa-ames-stereo-pipeline
channel, edit its meta.yaml
file and specify
the appropriate version for those dependencies.
It is very important to also ensure there is a new version for this package at
the top of meta.yaml
.
Each of the packages above can be built as follows:
conda build -c nasa-ames-stereo-pipeline -c usgs-astrogeology \
-c conda-forge gdal-feedstock
(Consider using the options --no-verify --no-test
with this tool
if it fails with with unrelated errors at the packaging stage, as
it happened on OSX on occasion. This is a risky option and should
be a measure of last resort.)
Upload it to the nasa-ames-stereo-pipeline
channel by
first logging in, via the command:
anaconda login
and specifying the channel as the user name, and then running a command along the lines:
anaconda upload \
$HOME/miniconda3/envs/asp_deps/conda-bld/linux-64/mypackage.tar.bz2
(Use above the path echoed on the screen by the conda build
command.)
Use the --force
option if desired to overwrite any existing
package with the same name and version. Be careful not to overwrite
a package that is meant to be used with a prior version of ASP.
After a package is uploaded, it can be installed in the existing
isis8.0.0
environment as:
conda install -c nasa-ames-stereo-pipeline \
-c usgs-astrogeology \
-c conda-forge \
gdal==3.5_isis7
To list all packages in that channel, do:
conda search -c nasa-ames-stereo-pipeline --override-channels
To delete a package from this channel, run:
anaconda remove nasa-ames-stereo-pipeline/mypackage/myversion
24.2.6. Order of building the packages¶
It is suggested to build the above packages in the order listed earlier, as some of them depend on others.
Note that libpointmatcher
depends on libnabo
, while liblas
depends on laszip
and gdal
, theia
depends on
imagemagick
, and visionworkbench
depends on gdal
. The
stereopipeline
package depends on all of these so it should be
built the last.
24.2.7. Additional ASP dependencies¶
VisionWorkbench and StereoPipeline have a few more conda dependencies
that need to be fetched from conda-forge
.
If desired to create an environment in which to build ASP or to update
the one in Section 24.1, the dependencies can be looked
up in the meta.yaml
files for these conda packages, after fetching
them according to Section 24.2.3.
24.3. Building the documentation¶
The ASP documentation is encoded in ReStructured Text and is built with the Sphinx-Doc system (https://www.sphinx-doc.org) with sphinxcontrib-bibtex (https://sphinxcontrib-bibtex.readthedocs.io). These packages can be installed and activated as follows:
conda create -n sphinx -c conda-forge python=3.6 \
sphinx=3.5.4 sphinxcontrib-bibtex=2.1.4
conda activate sphinx
Note that we used a separate conda environment to minimize the chance of conflict with other dependencies. Also, Sphinx version 4 seems to have trouble compiling our documentation, hence a lower version is used here.
In order to build the PDF (but not the HTML) document, a full LaTeX distribution is also necessary, such as TeX Live.
The docs
directory contains the root of the documentation. Running
make html
and make latexpdf
there will create the HTML and PDF
versions of the documentation in the _build subdirectory. In
particular, the PDF document will be at:
./_build/latex/asp_book.pdf
24.4. Releasing a new version of ASP using conda¶
This and subsequent sections are mostly reading for ASP developers. This content is included with the user documentation as this way it is easier to refer to relevant sections in the user guide.
24.4.1. Update the version number¶
Inside both the ASP and VisionWorkbench code, edit src/CMakeLists.txt
and
set the new version, which should be the same for both packages, and in the
format x.y.z
. If the value there is x.y.z-alpha
, which is used to tag a
pre-release, remove the -alpha
part. Increment one of these digits,
depending on whether this is a major, minor, or bugfix release. See
https://semver.org for guidance.
24.4.2. Update the documentation¶
Search all documentation for the old version number for ASP and ISIS (such as
8.0.0) and replace it with the new version numbers. This includes files in the
base directory, not just in docs
.
Update NEWS.rst. Add the release date on top, along the lines of prior releases (see further down in that file). This file must have a detailed log of all changes, especially those that result in changed behavior or options, and it should be incrementally updated as changes are made during development.
24.4.3. Commit and tag¶
Commit all changes. Tag the release in the VisionWorkbench and ASP repos. Example:
git tag 3.3.0
git push origin 3.3.0 # commit to your branch
git push god 3.3.0 # commit to main branch
(Here it is assumed that ‘origin’ points to your own fork and ‘god’ points to the parent repository.)
If more commits were made and it is desired to apply this tag to a different commit, first remove the exiting tag with:
git tag -d 3.3.0
git push origin :refs/tags/3.3.0
git push god :refs/tags/3.3.0
24.4.4. Build ASP with conda¶
See Section 24.2.
24.4.5. Test ASP¶
Fetch the stereo-pipeline
conda package, per Section 2.3. Save it,
for example, in a conda environment named asp
.
Use the ASP test framework to test it. Set the correct paths in the configuration file used for testing, as described there.
24.4.6. Save a record of the conda packages¶
It is suggested to save a complete record of all packages that went into this conda release, as sometimes conda may have issues solving for the dependencies or it may return a non-unique solution.
The conda environment having the given ASP release can be exported as:
conda activate asp
conda env export > asp_3.3.0_linux_env.yaml
This was for Linux, and it works analogously on OSX. How to recreate ASP from this file is described in Section 2.3.
A file can also be made that lacks the entries for ASP and visionworkbench, so
keeping only the dependencies. It can be saved with a name like
asp_3.3.0_linux_deps.yaml
(also edit it and change the name of the
environment).
It is suggested to commit these in to the ASP repository, in the conda
subfolder. These files can be checked in after the release is already tagged,
built, and tested.
An example for how to use this file to create the environment having the ASP dependencies in Section 24.1.
24.5. Building self-contained binaries¶
In addition to creating a conda package, it is also convenient and ship an archive having all ASP tools and needed libraries (this includes the ISIS libraries but not the ISIS tools).
Such a build is created for each release and also daily. These are posted on the GitHub release page (Section 2.1).
This work is a continuation of the process described in Section 24.4.
24.5.1. Use BinaryBuilder¶
ASP uses a custom build system. It can be downloaded with git
from:
Create a conda environment that has the dependencies for building ASP, as
described in Section 24.1. Assume it is called asp_deps
.
Install the C, C++, and Fortran compilers, cmake>=3.15.5
, pbzip2
,
and for Linux also the chrpath
tool, as outlined on that page.
Go to the directory BinaryBuilder
, and run:
/path/to/python3 \
./build.py \
--cc <path to C comipler> \
--cxx <path to C++ compiler> \
--gfortran <path to Fortran compiler> \
--asp-deps-dir $HOME/miniconda3/envs/asp_deps \
--build-root build_asp \
--skip-tests \
visionworkbench stereopipeline
This will fetch and build the latest VisionWorkbench and Stereo Pipeline in
build_asp/build
, and will install them in build_asp/install
.
24.5.2. Create a conda environment having Python and numpy¶
ISIS expects a full Python distribution to be shipped. To avoid shipping
the entire asp_deps
environment, we create a separate environment
having only Python, numpy, with versions as expected by current ISIS.
Run, for example:
conda create -n python_isis8 python=x.y.x numpy=a.b.c
Note that different versions of these may be needed for Linux and OSX.
The conda list
command within the asp_deps
environment
can be used to look up the desired versions.
24.5.3. Prepare the documentation¶
Follow the instructions in Section 24.3. Copy asp_book.pdf
to the BinaryBuilder/dist-add/
directory.
24.5.4. Package the build¶
Run in BinaryBuilder
the command:
/path/to/python3 \
./make-dist.py build_asp/install \
--asp-deps-dir $HOME/miniconda3/envs/asp_deps \
--python-env $HOME/miniconda3/envs/python_isis8
The same command can be used to package the asp
conda environment created
earlier. Then, one should use instead of build_asp/install
the directory
$HOME/miniconda3/envs/asp
. The dependencies will still come from
$HOME/miniconda3/envs/asp_deps
.
Building and packaging should be done separately for Linux and OSX.
24.6. Creating a GitHub release¶
Create a release from the GitHub web interface. Use the tag for the current release. Upload the binaries (for Linux and OSX, Section 24.5) and pdf documentation (asp_book.pdf, Section 24.3). Add to the release notes a link to the appropriate NEWS section of the documentation. Only after all this save the release.
24.6.1. Zenodo link for the release¶
Saving the release will trigger creating the Zenodo permanent link. That link cannot be changed, and the text it copies from the release notes cannot be changed either.
It is very important to create the release just once for the given tag. Otherwise, the Zenodo link will be messed up. It is fine to later overwrite the binaries for this release, if only minor changes happened. Just do not delete and recreate the release.
Do not just rename the latest automatically uploaded daily build, as that will create an incorrect Zenodo link.
Wait a few minutes for Zenodo to mint the link for this release, then visit the
Zenodo page at https://zenodo.org/badge/latestdoi/714891 (which will always
point to the ‘latest’ DOI) and find there this release’s URL. Put it at the
top of README.rst, in the appropriate target
field. Increment the version in
the image
field right above that.
Add this link also to the NEWS.rst page, under the section name for the current release.
Push this update to GitHub. The new commit will be after the tag for the release, but that is not an issue. It is best to not change the tag after the release and Zenodo link got created.
24.6.2. Updating the release from the command line¶
The GitHub tool gh
can be used to push files to a release.
Here’s an example usage:
cd BinaryBuilder
for file in asp_tarballs/StereoPipeline-3.3.0-Linux.tar.bz2 \
asp_tarballs/StereoPipeline-3.3.0-OSX.tar.bz2 \
dist-add/asp_book.pdf; do
gh release upload 3.3.0 $file \
-R git@github.com:NeoGeographyToolkit/StereoPipeline.git
done
As before, do not delete and recreate the release, but it is fine to delete and re-upload the binaries and documentation.
24.6.3. Announce the release¶
Send an announcement of the new release to the mailing list, at:
https://groups.google.com/forum/#!forum/ames-stereo-pipeline-support
and the old stereo-pipeline@lists.nasa.gov, with a link to the NEWS section for the current release from the documentation.
24.6.4. Post-release work¶
Update the version number in src/CMakeLists.txt
in boh the VisionWorkbench
and ASP repositories.
If version 3.3.0 just got released, we expect that the next feature release will be 3.4.0, if a major release, or 3.3.1 if a minor release. So, the version tag should be updated to 3.3.1-alpha in anticipation (see https://semver.org for guidance).
24.7. Nightly regression¶
The script auto_build/launch_master.sh
in BinaryBuilder
is used to
build and test ASP nightly. If these succeed, the produced daily build is
automatically uploaded to the GitHub release page.
This script needs to be read carefully and some variables adjusted.