16.70. stereo_dist

The stereo_dist program performs distributed stereo processing. Nearly all steps are run on small tiles in parallel across a set of computing nodes. This also includes the preprocessing and filtering steps, which are serial in parallel_stereo. DEM creation per tile and mosaicking of produced DEMs are also distributed.

This program requires that input images be mapprojected, or the --mapproject option be passed in to mapproject them automatically. GNU Parallel manages the jobs. It is expected that all nodes can connect to each other using ssh without password and that they share the same storage space.

16.70.1. Example with mapprojected images

stereo_dist                       \
  left_map.tif right_map.tif      \
  left.json right.json            \
  run/run                         \
  --dem input_dem.tif             \
  --tile-size 2048                \
  --tile-padding 256              \
  --nodes-list machines.txt       \
  --processes 5                   \
  --threads-multiprocess 4        \
  --threads-singleprocess 16      \
  --point2dem-options             \
    '--tr 2.0
    --orthoimage
    --errorimage'

The input images must be mapprojected (Section 6.1.7). The mapprojection DEM should be set with --dem. All output files (Section 19) will start with the provided prefix (run/run above).

The result of this program will be the stereo DEM (and optionally ortho and error images). The workflow in Section 16.70.3 has more details.

This program can be run on PBS and SLURM systems in a manner analogous to parallel_stereo. The --nodes-list option is then required. See Section 8.19 for details.

The option --point2dem-options must include --tr and the grid size (typically in meters) to ensure a consistent grid size across all tiles that are later merged.

16.70.2. Example with raw images

With the --mapproject option, raw (not mapprojected) images can be specified as inputs. The program will mapproject both images onto the DEM before running stereo.

stereo_dist                 \
  --mapproject              \
  left.cub right.cub        \
  run/run                   \
  --nodes-list machines.txt \
  --dem input_dem.tif       \
  --tile-size 2048          \
  --tile-padding 256        \
  --point2dem-options       \
    '--tr 2.0'

Here we omitted the cameras as those are contained in the cub files.

The projection is auto-determined based on the DEM and the left image (Section 16.41.1).

The mapprojection grid size (GSD) is set to the minimum of those estimated from the input images.

The mapprojection region is the intersection of the regions for the two images.

Any of these can be explicitly set with --t_srs, --tr, and --t_projwin (Section 16.70.4).

The mapprojected images are saved as <output prefix>-left_map.tif and <output prefix>-right_map.tif. Then, the same processing is done as with pre-existing mapprojected images. These are reused with --entry-point 1 or higher.

16.70.3. Workflow

The processing is as follows:

  • The images are mapprojected (with the --mapproject option only).

  • Statistics is computed for the input mapprojected images. This is a serial process but is quite fast.

  • The list of tiles is created and saved with a name based on the output prefix and ending in -distTileList.txt. A shapefile with the tile boundaries is also created for visualization.

  • All tiles are run in parallel with the stereo_tile program. That program creates a subdirectory for the given tile and provides the global statistics via a symbolic link. It then does preprocessing, correlation, refinement, filtering, triangulation, and DEM creation. These operations are described in Section 16.51.7.

  • The per-tile DEMs are mosaicked into a single output DEM with dem_mosaic. This is done in parallel as well, for subsets of DEMs, whose results are then mosaicked together.

  • If --orthoimage is in --point2dem-options, the per-tile ortho images (DRG files, Section 16.56.2.2) are mosaicked the same way. The L.tif file needed by --orthoimage is autocompleted for each tile. Similarly, if --errorimage is passed, the per-tile intersection error images are mosaicked.

  • Unlike parallel_stereo, the blend step for disparities is skipped. Each tile is processed fully independently, and blending only happens between DEMs during the final mosaic.

The --entry-point and --stop-point options can be invoked to run only a portion of these steps. See Section 16.70.4 for the step numbers.

Any options that are not specific to this program are passed directly to stereo_tile and the stereo executables (Section 6.1.5).

16.70.4. Command-line options

--dem <string>

Input DEM. Required. Used for mapprojection (with --mapproject) and for stereo triangulation.

--mapproject

Mapproject the input images onto the DEM before running stereo. The projection, grid size, and bounding box are auto-determined to be consistent for both images. See Section 16.70.2.

--tile-size <integer (default: 2048)>

Size of each tile (in pixels) for distributed processing. This is before the padding is added.

--tile-padding <integer (default: 256)>

Padding around each tile (in pixels) to avoid boundary artifacts.

--point2dem-options <string>

Options to pass to point2dem. Must include --tr to set the grid size for consistent tiling. Can pass --orthoimage with no argument, and the L.tif file for each tile will be autocompleted. Can also pass --errorimage. See Section 16.70.3.

--nodes-list <filename>

A file containing the list of computing nodes, one per line. If not provided, run on the local machine.

--processes <integer>

The number of processes to use per node.

--threads-multiprocess <integer>

The number of threads to use per process when running multiple processes.

--threads-singleprocess <integer>

The number of threads to use when running a single process.

-e, --entry-point <integer (default: 0)>

Stereo pipeline entry point. Values: 0=pprc, 1=corr, 2=blend (skipped, kept for compatibility with parallel_stereo steps), 3=rfne, 4=fltr, 5=tri, 6=cleanup, 7=dem, 8=mosaic.

--stop-point <integer (default: 9)>

Stereo pipeline stop point (stop before this step).

--t_srs <string>

Output projection for mapprojection. A PROJ string, EPSG code, or path to a WKT file. Auto-determined from the left image if not set. Only used with --mapproject.

--tr <float>

Output grid size (ground sample distance) for mapprojection, in units of the projection. Auto-determined as the finer of the two images if not set. Only used with --mapproject.

--t_projwin <xmin ymin xmax ymax>

Bounding box for mapprojection in projected coordinates. Auto-determined as the intersection of the two image footprints if not set. Only used with --mapproject. If the images are already mapprojected, use instead the --proj-win option which will be passed to the stereo executables (Section 17).

--parallel-options <string (default: “–sshdelay 0.2”)>

Options to pass directly to GNU Parallel.

--verbose

Display the commands being executed.

--dry-run

Do not launch the jobs, only print the commands that should be run.

-v, --version

Display the version of software.

-h, --help

Display this help message.