9.3. MSL navcam example¶
This is an example of using the ASP tools to process images taken by the Mars Science Laboratory (MSL) rover Curiosity. See Section 9 for other examples.
This approach uses only the images to create a self-consistent solution, without placing it in the right location on the Mars surface. Section Section 8.12.6 discusses using the known camera poses for MSL.
9.3.1. Illustration¶
9.3.2. Sensor information¶
Curiosity has two navcam sensors (left and right) mounted on a stereo rig. Each records images at a resolution of 1024 x 1024 pixels. The field of view is 45 degrees.
9.3.3. Challenges¶
The navcam images are used to plan the path of the rover. They are not acquired specifically for mapping.
While there is good overlap and perspective difference between images that are taken at the same time with the stereo rig, these assumptions may not hold for images produced at different times. Moreover, after the rover changes position, there is usually a large perspective difference and little overlap with earlier images.
A very useful reference on processing MSL images is [CLMouelicM+20]. It uses the commercial Agisoft Photoscan software.
To help with matching the images, this paper uses the global position and orientation of each image and projects these onto the ground. Such data is not fully present in the .LBL files in PDS, as those contain only local coordinates, and would necessitate queering the SPICE database. It also incorporates lower-resolution “TRAV” images to tie the data together.
In the current example only a small set of data from a single day is used.
9.3.4. Data preparation¶
The images are fetched from PDS. For example, to get the data for day (SOL) 597 on Mars, use the command:
dir=data/msl/MSLNAV_0XXX/DATA/SOL00597
wget -r -nH --cut-dirs=4 --no-parent \
--reject="index.html*" \
https://pds-imaging.jpl.nasa.gov/$dir \
--include $dir
This will create the directory SOL00597
containing .IMG data files
and .LBL metadata. Using the ISIS pds2isis
program (see
Section 2.1.1 for installation), these can be converted to
.cub files as:
pds2isis from = SOL00597/image.LBL to = SOL00597/image.cub
A .cub file obtained with the left navcam sensor will have a name like:
SOL00597/NLB_451023553EDR_F0310946NCAM00492M1.cub
while for the right sensor the prefix will be instead NRB
. The
full-resolution images have _F
as part of their name, as above.
We will convert the .cub files to the PNG format so that they can be
understood by image-processing programs. The rig_calibrator
convention will be used, of storing each sensor’s data in its own
subdirectory (Section 16.53.2). We will name the
left and right navcam sensors lnav
and rnav
. Then, the conversion
commands are along the lines of:
mkdir -p SOL00597/lnav
isis2std from = SOL00597/left_image.cub \
to = SOL00597/lnav/left_image.png
9.3.5. Image selection¶
A subset of 22 images was selected for SOL 597 (half for each of the left and right navcam sensors). Images were chosen based on visual inspection. A fully automatic approach may be challenging (Section 9.3.3).
This dataset is available for download.
9.3.6. Setting up the initial rig¶
Given the earlier sensor information, the focal length can be found using the formula:
where \(w\) is sensor width in pixels and \(\theta\) is the field of view. The focal length is then about 1236.0773 pixels. We will start by assuming that the optical center is at the image center, and no distortion. Hence, the initial rig configuration (Section 16.53.5) will look like:
ref_sensor_name: lnav
sensor_name: lnav
focal_length: 1236.0773
optical_center: 512 512
distortion_coeffs:
distortion_type: no_distortion
image_size: 1024 1024
distorted_crop_size: 1024 1024
undistorted_image_size: 1024 1024
ref_to_sensor_transform: 1 0 0 0 1 0 0 0 1 0 0 0
depth_to_image_transform: 1 0 0 0 1 0 0 0 1 0 0 0
ref_to_sensor_timestamp_offset: 0
with an additional identical block for the rnav
sensor (without
ref_sensor_name
).
9.3.7. SfM map creation¶
Given the data and rig configuration, the image names in .png format
were put in a list, with one entry per line. The theia_sfm
program (Section 16.64) was run to find initial camera poses:
theia_sfm \
--rig_config rig_config.txt \
--image_list list.txt \
--out_dir theia_rig
Next, rig_calibrator
(Section 16.53) is used, to
enforce the rig constraint between the left and right navcam sensors
and refine the intrinsics:
params="focal_length,optical_center"
float="lnav:${params} rnav:${params}"
rig_calibrator \
--rig_config rig_config.txt \
--nvm theia_rig/cameras.nvm \
--camera_poses_to_float "lnav rnav" \
--intrinsics_to_float "$float" \
--num_iterations 30 \
--calibrator_num_passes 2 \
--num_overlaps 5 \
--robust_threshold 3 \
--out_dir rig_out
Here, --robust_threshold
was increased from the default value of
0.5 to focus more on larger errors. To optimize the distortion,
one can adjust the rig configuration by setting initial distortion
values and type:
distortion_coeffs: 1e-10 1e-10 1e-10 1e-10 1e-10
distortion_type: radtan
and then defining the list of parameters to optimize as:
params="focal_length,optical_center,distortion"
For this example, plausible solutions were obtained with and without using distortion modeling, but likely for creation of pixel-level registered textured meshes handling distortion is important.
The produced pairwise matches in rig_out/cameras.nvm
can be
inspected with stereo_gui
(Section 16.62.8.4).
9.3.8. Mesh creation¶
Here, a point cloud is created from every stereo pair consisting of a left sensor image and corresponding right image, and those are fused into a mesh. Some parameters are set up first.
Stereo options (Section 17):
stereo_opts="
--stereo-algorithm asp_mgm
--alignment-method affineepipolar
--ip-per-image 10000
--min-triangulation-angle 0.1
--global-alignment-threshold 5
--session nadirpinhole
--no-datum
--corr-seed-mode 1
--corr-tile-size 5000
--max-disp-spread 300
--ip-inlier-factor 0.4
--nodata-value 0"
Point cloud filter options (Section 16.48):
maxDistanceFromCamera=100.0
pc_filter_opts="
--max-camera-ray-to-surface-normal-angle 85
--max-valid-triangulation-error 10.0
--max-distance-from-camera $maxDistanceFromCamera
--blending-dist 50 --blending-power 1"
Mesh generation options (Section 16.66):
mesh_gen_opts="
--min_ray_length 0.1
--max_ray_length $maxDistanceFromCamera
--voxel_size 0.05"
Set up the pairs to run stereo on:
outDir=stereo
mkdir -p ${outDir}
grep lnav list.txt > ${outDir}/left.txt
grep rnav list.txt > ${outDir}/right.txt
The optimized rig, in rig_out/rig_config.txt
, and optimized
cameras, in rig_out/cameras.txt
, are passed to multi_stereo
(Section 16.38):
multi_stereo \
--rig_config rig_out/rig_config.txt \
--camera_poses rig_out/cameras.txt \
--undistorted_crop_win '1100 1100' \
--rig_sensor "lnav rnav" \
--first_step stereo \
--last_step mesh_gen \
--stereo_options "$stereo_opts" \
--pc_filter_options "$pc_filter_opts" \
--mesh_gen_options "$mesh_gen_opts" \
--left ${outDir}/left.txt \
--right ${outDir}/right.txt \
--out_dir ${outDir}
This created:
${outDir}/lnav_rnav/fused_mesh.ply
See the produced mesh in Section 9.3.1.
9.3.9. Notes¶
No ground registration was done, so neither the scale nor the pose of the produced mesh is accurate. The mesh is, however, self-consistent.
The voxel size for binning and meshing the point cloud was chosen manually. An automated approach for choosing a representative voxel size is to be implemented.
The
multi_stereo
tool does not use the interest points found during SfM map construction. That would likely result in a good speedup. It also does not run the stereo pairs in parallel.