Hierarchical Object-Aware Dual-Level Contrastive Learning for Domain Generalized Stereo Matching

Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:

Q1: The method requires semantic labels to achieve SOTA performance: it is true that the proposal is effective even without the object prior, however, it is necessary to achieve SOTA performance: not all datasets output this information. Authors could have better studied the usage of a SOTA segmentation framework (e.g., (R3)) to replace ground-truth semantic labels.

A1: Thanks for expressing your concerns. We would like to highlight that, in contrast to other methods that utilize additional information (e.g., using pretrained features on large-scale labeled datasets [3]) to improve the generalization ability of stereo matching networks, our method relies solely on synthetic data sources. In synthetic stereo datasets, object labels can be generated along with disparity maps without much extra effort, and they are often available [4, 5]. As using pseudo labels generated by pre-trained models trained on realistic data does not strictly follow the synthetic-to-real setting, in the future we plan to directly derive pseudo object masks from disparity for finding semantically and structurally driven matching.

Additionally, our HODC is orthogonal with other proposed SOTA domain generalization approaches (e.g., hierarchical visual transformation [1], adversarial shortcut perturbations [2]). The proposed HODC can incorporate these methods for data augmentation, which could further improve its generalization performance.

Q2: Furthermore, experiments could be extended using other recent stereo networks robust against domain shift (e.g., RAFT-Stereo (R4)).

A2: Thank you for your advice. In our experiments, we included IGEV [6], a more recent deep network architecture that serves as a strong baseline, as it incorporates RAFT-Stereo with Geometry Encoding Volume (GEV) that shows SOTA in-domain as well as generalization performance. Experiments in Table 1 show that the generalization performance of HODC-IGEV is also substantially improved compared to the baseline, demonstrating the compatibility of HODC with recent stereo networks to enhance their robustness to domain shift.

Q3: It is true that the framework does not require changes in the stereo network architecture, however, it still needs access to the model to train it with the proposed losses -- i.e., the stereo network is seen as a "white box" model.

A3: Thank you for providing insights regarding the extensibility of HODC with particular constraints. We wish to highlight that our HODC operates on the intermediate stage (i.e., after feature extraction) in a stereo matching pipeline, and hence it does not need to access the particular structure of the model. Furthermore, as feature extraction is a very common component in deep-learning-based stereo matching networks, it is rather straightforward to integrate HODC into existing models, thus ensuring its universality.

Q4: Row 220: I suggest the authors add a little example of global and local scale to further reduce ambiguities (e.g., k=2, N_h = 2, N_w = 4 -> global scale is (2,4) and local scale is (4,8)).

A4: Thank you for the suggestion to improve our paper. We will include the suggested example in the final manuscript to reduce ambiguities.

Q5: Tab. 1: techniques are evaluated using all pixels or only non-occluded pixels?

A5: Following the implementations of recent SOTA methods (e.g., ITSA and HVT), we evaluate all pixels in the KITTI (and DrivingStereo) dataset and non-occluded pixels in Middlebury and ETH3D datasets.

Q6: There is a little typo: it is ITSA, not ISTA.

A6: Thank you for pointing this out. We will fix the typos in our final manuscript.

References

[1] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.

[2] ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks. CVPR 2022.

[3] GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature. CVPR 2022.

[4] Virtual Worlds as Proxy for Multi-Object Tracking Analysis. CVPR 2016.

[5] A naturalistic open source movie for optical flow evaluation. ECCV 2012.

[6] Iterative Geometry Encoding Volume for Stereo Matching. CVPR 2023.

Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:

Q1: the smooth L1 loss should be mathematically defined or a reference should at least be provided.

A1: Thank you for pointing this out. We will add a reference to the smooth L1 loss in the final manuscript.

Q2: Reporting baseline results without the proposed additional losses in Table 3 would make the table easier to interpret.

A2: Thanks for your advice. We will include the baseline results in Table 3 in the final version of the paper.

Q3: How do the different methods evaluated perform on Sceneflow test set? Reporting impact of the proposed approach on in-domain performance might be insightful.

A3: Thank you for your suggestions. Following your suggestion, we have evaluated the in-domain performance of the models with and without our dual-level contrastive loss using the SceneFlow test set following our experiment settings. The results are listed in the table below.

Interestingly, the results revealed that our HODC can achieve high cross-domain performance as well as in-domain performance by establishing semantic and structural correspondence, with the same or even better threshold error rate compared to the baselines. Only a small decline is observed in the average end-point error metric.

Q4: Ablation results in Table 3 using the proposed losses without object index map are rather good compared to the baseline methods without contrastive learning. This suggests that training for multi-scale correspondences may be of greater importance than using instance segmentation cues, which would downgrade the importance of semantics. Reporting results when using small base superpixel size (e.g. M=32), with contrastive losses but without object index priors could bring insights regarding this.

A4: Thanks for your advice regarding the role of instance segmentation cues. Instance segmentation cues allow us to generate regions with unambiguous semantic meanings, enabling us to establish semantically and structural correspondence accurately. Referring to Table 3, omitting the object prior will lead to a decline in the generalization performance of our HODC. We would like to highlight that $M$ denotes the segmentation scale, and a larger $M$ derives smaller base 'superpixel' sizes with finer representations. We restrict $M$ to be within $128$, as larger values will significantly increase computation cost and memory usage when calculating all pair correspondences.

References

[1] Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective. CVPR 2022.

[2] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.

Thanks for your positive feedback and helpful suggestions. We would like to address your concerns as follows:

Q1: The qualitative comparison in the article is insufficient. Considering that the generalization capability of stereo networks on public datasets is already quite good, the improvement of the method becomes more important from a visualization perspective. Especially since the article only uses PSMNet for qualitative comparison in Figure 1.

A1: Thanks for your advice. Due to space limitations, we included qualitative comparison results for PSMNet, GwcNet and IGEV on KITTI-2012, KITTI-2015, Middlebury, ETH3D in Figures E, F, and G, and results on the additional DrivingStereo [1] dataset in Figure A in the appendix of our paper.

Q2: The paper lacks visualization analysis, making it difficult to discern how the proposed strategy impacts the learned feature representations.

A2: Thank you for expressing your concerns. Our proposed HODC aligns the regional features from the left image to the right under different scales, aiming at pulling the corresponding representations closer while pushing non-corresponding representations further apart. To validate the feasibility of this strategy in learning features with semantic and structural awareness, we visualized the representation similarity between a selected region within the left image and all pixels in the right image. As shown in Figure 4, the proposed HODC can accurately identify matched regions with limited ambiguities. More visualization results for feature representations on KITTI, Middlebury and ETH3D datasets are available in Figures B, C and D in the appendix of our paper.

Q3: I would like to see this paper adds more qualitative visualizations, as well as evaluating the domain generalization performance on more challenging unseen domains such as the Booster and Spring datasets.

A3: Thanks for your suggestions. Following your advice, we evaluated the generalization performance of HODC on the challenging realistic Booster [2] dataset with quarter resolution and provided comparisons with other domain generalization methods. The results on the Booster training set are reported in the table below.

The results on the Booster dataset indicate that HODC achieves satisfying results on challenging unseen domains, with significant improvements in generalization ability compared to baseline models. Following your advice, qualitative results on the Booster dataset are also provided in Figure I in the attached file in the global response.

We would like to highlight that to further evaluate the generalization performance on challenging realistic scenarios, apart from using the widely used datasets, we have also provided quantitative and qualitative results on the DrivingStereo dataset, which contains diverse and challenging driving scenes. These results can be found in Appendix D of our paper.

References

[1] Drivingstereo: A large-scale dataset for stereo matching in autonomous driving scenarios. CVPR 2019.

[2] Open challenges in deep stereo: the booster dataset. CVPR 2022.

[3] Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective. CVPR 2022.

[4] Domain Generalized Stereo Matching via Hierarchical Visual Transformation. CVPR 2023.

Thanks for your positive feedback and helpful suggestions. We would like to make the following response to your questions:

Q1: As mentioned, it is not a new direction to explore the semantic and structural information in the stereo matching task (lines 43-45). Although the previous works took a different path when using this information, it is worth comparing the performance between those approaches and the method from this work.

A1: Thanks for your suggestions. In our paper, we have highlighted that unlike our work, earlier works have explored semantic information by introducing subnetworks for semantic segmentation [1] or edge detection [2] within the stereo matching pipeline, adopting a direct multi-task methodology. Additionally, these prior works primarily focused on in-distribution scenarios instead of cross-domain ones.

Following your recommendation for a more comprehensive analysis, we have conducted an experiment to compare the generalization performance of our HODC with the prior works [1, 2] that incorporate semantic information in stereo matching. As the source code for EdgeStereo [2] is unavailable, we replicated the experimental settings described in [2], utilizing only the Flyingthings3D training set for training, and then compared the generalization performance.

The results presented in the table below demonstrate that networks trained with our HODC significantly outperform these previous approaches and exhibit superior generalization capability.

References

[1] SegStereo: Exploiting Semantic Information for Disparity Estimation. ECCV 2018.

[2] EdgeStereo: An Effective Multi-Task Learning Network for Stereo Matching and Edge Detection. IJCV 2020.