Robust Stereo Matching by Risk Minimization

Q1: The current iterative optimization based methods such as RAFT-Stereo directly predict continuous prediction and show strong robustness. It seems the method in this paper is not suitable for these iterative optimization methods, which I consider a major weakness restricting its application.

A1: Thank you for highlighting this. This comment of yours aligns well with the advice provided by R2. Following this, we performed experiments IGEV, an iterative optimization method. We have added Table 13 in the revised draft showing the benefit of our iterative optimization approach with a well-known existing architecture in stereo matching.

Q2: How important is the Laplacian kernel for interpolation in Eq.(1)? I wonder if using the simple bilinear interpolation is enough for capturing continuous cost values.

A2: The importance of the Laplacian kernel can be observed from the Table 14 added in the revised draft. Using other popular kernel such as Gaussian kernel degrades the performance and over-smooths the solution, whereas, the Laplacian kernel preserves high-frequency details providing better results. We have added this ablation study on the choice of kernel in the appendix—refer Table 14.

Q3: Considering the paper builds its own network architecture, I wonder how the method can improve existing cost aggregation based networks such as PCWNet or GWCNet.

A3: In the revised draft, we have a table — Table 8, showing the benefit of our proposed optimization when applied to existing cost aggregation based networks such as PCWNet.

Q1: The authors claim that it is better than other methods in cross-domain generalization. However, they just compare with non-domain generalization stereo matching methods. Thus, the claim is lack of the justification. It is recommended to put more analysis on the cross-domain evaluation in terms of the proposed risk minimization as done by [A] in Table 1.

A1: Following your comments, we have conducted the suggested experiment showing the benefit of our method on cross-domain generalization with a recent work ITSA (Chuah et al., CVPR 2022). Statistical results related to this are provided in Table 15 in the revised draft . Despite we discussed [A] in the related work, the official source code for [A] is not public. That said, we will be happy to have a comparison with [A] once the code is available.

Q2: In the ablation study, the authors compare the performance of L1 risk minimization in two worst methods in Middlebury evaluation. It makes the claim to improve the performance on different networks is not justified enough.

A2: Thank you for highlighting this. This comment of yours aligns well with the advice provided by R2. Following this, we performed experiments IGEV, an iterative optimization method. We have added Table 13 in the revised draft showing the benefit of our iterative optimization approach with a well-known existing architecture in stereo matching.

Q3: Thus, it is unclear whether the performance in the experimental results is because of the proposed new architecture or the L1 risk minimization. More detailed analysis and ablation study are required.

A3: In this paper, we primarily aim to propose a method to capture the continuous disparity via stereo matching networks. Furthermore, we put forward the benefit of L1 formulation to risk minimization loss function and how it can be incorporated in a differentiable way for neural network optimization via the use of implicit function theorem. Following your comments, we have more experimental results and ablation study on different architectural backbone showing the benefit of our proposed L1 risk optimization. Kindly refer Table 8, Table 13 and Table 15 for statistical results in our revised draft.

Q4: Does the proposed risk minimization really improve the cross-domain generalization?

A4: Yes, the experimental results show that the proposed risk minimization improved the cross-domain generalization. We have addressed this concern following your comments in Table 15 in our revised draft.

Q5: Is there any reason to choose ACVNet and PCWNet for comparison in Table 8? Is it possible to perform experiment where the L1-risk is also used in the training stage. More comparisons with other networks are needed.

A5: The reason we choose ACVNet and PCWNet is because those are recent feature aggregation networks that seems to work really well for stereo matching problems. Yes, it is possible to do risk optimization at train time. Yet for simplicity we used pretrained feature networks. That said this is an encouraging direction for future extension.

We have added comparison with other network architecture such as IGEV. Kindly refer to table 13 in the revised draft.

Q6: If the new architecture is not used, would L1 risk minimization method improves the current state-of-the-art performance?

A6: We have tested our L1 risk minimization on recent off-the-shelf network designs such as IGEV, PCWNet and added its results in table 13 and table 8, respectively in the revised draft. We observed a clear improvement.

Q7: Is there any limitation of the proposed L1 risk minimization method?

A7: Despite its benefit, L1 risk minimization is slightly more time consuming than the popular loss function such as L2 loss. To make such a continuous optimization time efficient is an attractive avenue for research in deep learning. Kindly refer to Table 8 in our revised draft for inference time analysis.

Q1: There are parts of the paper that wording can be improved. Proofreading by a native speaker will further strengthen the paper. Examples: a) impressive deep-learning architecture -> advancements in deep-learning. Something being impressive doesn't necessarily mean that it works b) Typical stereo matching algorithm will sample ... and compute a discrete distribution p: Typical stereo matching algorithms will compute a cost that merely can be described as a PMF. c) To take a value from -> To choose a value as the final prediction? d) where the Sign() is the sign function -> where the sign is the signum function? e) In implementation, we clip -> we clip…

A1: We appreciate you pointing out the paper's minor typos and writing mistakes. Thank you so much for this. We have improved the highlighted typos and minor mistakes in the revised draft.

Q2: The symbols in the math equation can easily cause confusion: a) The symbol p is used to denote the probability mass function and the probability density function. It is common to use P for mass and F for density. Another option would be to use subscripts if the authors want to keep using the notation p(x;p) for the probability density function. b) in equation (2) y is in the left and right hand of the equation. If y is the result of the argmin shouldn't the right-hand y be defined as a variable in a range of values? similarly eq(4)

A2: Thank you for the great advice on the notation and its possible confusion. (a) We added superscript $m$ to $\mathbf{p}$, i.e. $\mathbf{p}^m$ to the notation of probability mass function to avoid the confusion. (b) We have refined the draft accordingly.

Q3: The binary search algorithm can be moved to the appendix.

A3: We have moved the binary search algorithm in the appendix–refer page 16 Algorithm 1.

Q4: The related work is very dense.

A4: We agree that related work is dense. But this is mainly because stereo matching is a very classical problem with more than 25 years of literature. In our paper’s related work, we have tried to keep the essence of classical as well as modern approaches. We have slightly detailed on the related work in our revised draft and added a few comparison with our approach.

Q5: Figure 3: The captions are not aligned with the different stages of the architecture. You can condense the schematics and allow more space for risk minimization since the other parts of the proposed architecture are described in prior art

A5: Excellent advice for improving our draft. For now, we are keeping the Figure 3 illustration the same but we are open to changing it after the decision. We have changed the Figure 3 caption following your advice.

Q6: Table 4: Are these results after training and testing all these methods on Middlebury? Please specify in the captions of all tables if the reported results are after self-evaluation or the official results on the dataset evaluation page.

A6: For Table 4, the methods are trained on SceneFlow and evaluated on the training set of Middlebury. For Table 2 and Table 3, the results are obtained from the KITTI official website. Following your comments, we revised the captions.

Q7: Submit also on the test set of Middlebury.

A7: We are in the process of submitting the results on the middlebury public benchmark leaderboard and soon it will be online.

Q8: Since the main proposal of this paper is risk minimization, It would strengthen the paper if they apply their proposed module to more architectures, like IGEV or Raft-stereo which has shown strong cross-domain generalization.

A8: Indeed that is good advice to strengthen the current paper. We have added Table 13 in the revised draft that shows the results of our formulation on IGEV network architecture. Furthermore, we have shown results on two other networks namely, ACVNet and PCWNet —refer Table 8 in the revised draft.

Q1: While theoretically motivated, the actual network architecture and components like cost volumes seem fairly standard. More implementation details could help highlight if other architectural modifications were needed to fully take advantage of risk minimization.

A1: We agree that a more insightful architectural design modification could benefit the proposed risk loss function more. However, in this paper, we primarily aim to propose a method to capture the continuous disparity via stereo matching networks. To this end, we put forward the benefit of L1 formulation to risk minimization loss function and how it can be incorporated in a differentiable way for neural network optimization via the use of implicit function theorem. Following your comments, we have added more implementation details in A.2.

Q2: The binary search algorithm for L1 optimization, while efficient, involves heuristics like the tolerance hyperparameter. More analysis could be provided on algorithm convergence and optimality guarantees. For real-time applications, the running time and complexity should be analyzed more thoroughly. Comparisons to optimizations like pruning could better highlight efficiency.

A2: Thank you for highlighting this. The argument on the tolerance hyperparameter is indeed important and generally true yet the suitable target function introduced in the paper is convex and its optimal solution can be guaranteed -refer Eq.(4) in the revised draft. Details such as O(log N) time complexity with N symbolizing the number of disparities is added in the revised draft–refer [1] for details. Following your suggestion, we performed an experimental study in relation to the tolerance hyperparameter. Table 11 in the revised draft shows the performance of our method with different tolerance hyperparameters across different disparity levels. In relation to pruning, it will be insightful to have a rigorous analysis of the proposed optimization and we are still to conduct more intricate and extremely detailed statistical analysis for a decisive agreement on our formulation's efficiency to sparse networks.

[1] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms, Third Edition (3rd. ed.). The MIT Press.

Q3: How does the risk minimization formulation compare to other robust loss functions like smooth L1 that try to reduce outlier influence? Is the improvement mainly from switching to L1 or from the risk view? Have the authors experimented with other error functions like Huber loss within the risk framework? This could help determine the benefits of specifically L1 vs. just a robust loss.

A3: Thank you for this excellent suggestion. We have added Table 12 in the revised draft—refer A.4.2. that provides an ablation study showing the benefit of the proposed risk minimization with L1 in relation to the Huber loss, i.e. a combination of L1 and L2 norm depending on the thresholding value $\beta$. The table in the revised draft clearly shows the benefit of using risk minimization loss under L1.

Q4: Can the method apply to other vision tasks with multi-modal outputs like optical flow or depth estimation? Does it mainly benefit stereo matching or generalize broadly?

A4: This comment is very insightful. Given that depth, optical flow and disparity are correlated concepts in stereo vision, we believe our proposed loss function could provide a fruitful direction to investigate all those related problems and may also benefit similar problems in computer vision. But in this work, we primarily concern ourselves with risk minimization idea to the stereo matching problem with application to robotics and control field and showed its possible applicability accordingly.