Evidential Learning-based Certainty Estimation for Robust Dense Feature Matching

• W1. Why EDL is still effective for binary classification task

As mentioned by the reviewer in the comments, EDL's main advantage is to detect out-of-distribution (OOD) samples or mining pseudo-unknown objects. This advantage comes from the fact that by modeling second-order probabilities and uncertainty, EDL provides improved uncertainty estimation. As shown in the seminal work [Sensoy et al., NeurIPS2018], EDL will produce larger uncertainty for OOD samples, while the standard approach (training the network by minimizing cross-entropy loss) tends to produce over-confident (low-uncertainty) predictions. Assuming a binary classification task and an OOD sample with ground truth label [1, 0], the standard approach may produce an over-confident prediction [0.01, 0.99], while EDL may produce a high-uncertainty prediction [0.4, 0.6]. Now assume that the system needs to continue the downstream estimation pipeline with the over-confident or high-uncertainty prediction, with which prediction shall the system perform better? Intuitively, an over-confident prediction is more devastating as the system cannot recover from it, while a high-uncertainty prediction still assigns some confidence to the correct class and the system may still be able to perform well with it. This is exactly what happens when we apply EDL for the certainty estimation task in dense matching. In Fig.6 of the paper, we provide visualization of the certainty map estimated by RoMa and our method. RoMa tends to predict very low certainty value (e.g., below 0.05) for matchable regions on corrupted images (i.e., over-confidently classify the region as unmatchable), while our model produce a certainty value around 0.4 for matchable regions -- it is not perfect, but good enough to facilitate the following balanced sampling step to sample a diverse set of matches from matchable regions. We have modified the Introduction section and Section 4.5 to illustrate this point better.

• W2. The overall contribution is limited, lacking enough in-depth discussion

We added Fig.7 in the revised paper to cast more insight into the behavior of EDL in dense feature matching. We visualize the evidence map estimated by our model, and reveal that under corrupted cases, the model cannot produce high evidence for both classes in the matchable region. This cause high-uncertainty estimation for the matchable region. Compared to the over-confident prediction generated by RoMa, the EDL's prediction enables more effective sampling of reliable matches and thus achieves better performance.

• W3. The introduction of EDL in Section 3.2 is insufficient

We have revised Section 3.2 to add more details in introducing the method.

• W4. Missing discussion of visual localization (InLoc and AachenDay-Night) and homography estimation (HPatches)

We added additional benchmarking datasets in A.3 of the revised paper. Our method obtains 1.7% increase in mAA@10px than RoMa on WxBS, and 0.5% increase in AUC$@3$px on HPatches. For visual localization (InLoc and AachenDay-Night), RoMa does not release their evaluation code. We need more time to reproduce their results and evaluate our method, and thus do not report the results in current version of the paper.

• Q1. Difference or relationship of the proposed method compared to outlier filtering post-processing method like RANSAC

The outliers in RANSAC refer to points that do not fit into the current estimation of the geometry model (e.g., homography, essential matrix). The certainty estimation in our task does not involve a geometry model. Instead, the matches sampled based on the estimated certainty will be used in downstream geometry estimation tasks, where RANSAC will be applied.

Thanks for the response. The rebuttal addressed most of my concerns, so I raised my score to 5. The remaining concern is that the novelty of this work is somehow incremental, i.e., incorporating EDL to dense feature matching intuitively. Moreover, the overall improvement of this work is mainly in corrupted images, which is not significant enough for the foundational feature-matching task.

• W1. Only a single model is tested

We verify our method on another dense matcher, DKM [Edstedt et al., 2023] in A.4 of the revised paper. Our method increases the clean AUC$@5 by 1.6% on MegaDepth-1500, and increases the mean corruption AUC@5 by 1.3% on MegaDepth-1500-C. These results demonstrate the generalizability of our method beyond RoMa.

• W2. Explanation of EDL is difficult to understand

We have revised Section 3.2 to make it more clear.

• W3. EDL's built-in uncertainty measure is not used and apply EDL on the regression-by-classification in the coarse matching step

In our current approach, EDL's built-in uncertainty measure is not used, and instead its expected value over the first class is used to indicate matching reliability. Actually in the seminal work [Sensoy et al., NeurIPS2018], EDL's built-in uncertainty measure is also not used: for fair comparison with other methods, the authors choose to use the entropy computed from the expected class probabilities and demonstrate improved uncertainty measurement for out-of-distribution detection. This suggests that the expected class probability of EDL itself is informative and can be directly used for following tasks.

It is possible to apply EDL for the regression-by-classification in the coarse matching step, but there are two drawbacks for this approach. First, the uncertainty at coarse scale does not necessarily represent the uncertainty at the finest scale. In the current method, only the matches and certainty score at the finest scale are used for balanced sampling and downstream tasks. As shown in the ablation study on coarse vs. fine scale EDL in the revised paper (Table 5), it is the EDL at fine scale that contributes most to the improvement, and coarse-scale EDL is not effective. Second, while RoMa use a regression-by-classification formulation in the coarse matching step, other dense matchers, e.g., DKM [Edstedt et al., 2023] and DGC-Net [Melekhov1 et al., 2019], use L1 or L2 regression for coarse matching. Nevertheless, all these dense matchers use the same classification formulation for certainty estimation. Our current formulation of EDL for certainty estimation thus avoids model-specific design and can be easily applied to other dense matchers.

• Q1. How much does the threshold used for balanced sampling matter for the robustness?

We observe that inappropriate choice of the threshold can adversely affect both RoMa and our method. For example, increasing the threshold from 0.05 to 0.1 degrades the performance of our method by 2.6%, and RoMa by 1.7% on MegaDepth-1500 corrupted by Gaussian noise at severity level 5. A sampling technique without such hardcoded threshold would be an interesting future work.

• Q2. Is the new certainty estimation more or less as heavy as the old in terms of inference and training speed?

Yes. Your reading is correct. We report the training and inference time of RoMa and our method in Table 6 of the revised paper. For training, our method takes 1.4% more GPU hours than RoMa. For inference, our method actually incur marginally lower cost than RoMa due to the elimination of sigmoid computation for obtaining final probability.

• Q3. Is the increased performance due to only better certainties? Is the performance the same if certainty scores from the new model are combined with matches from the original RoMa?

We added in A.5 of the revised paper the results of combining certainty score from our model with the warp from the original RoMa model. We observe that using our certainty score, the performance of RoMa is increased from 25.3% to 36.5%, close to our results of 37.9%. In Section 4.5 of the paper, we report the average endpoint error (AEPE), which is defined as the average Euclidean distance between the estimated and ground truth warp. The AEPE for our method and RoMa on MegaDepth-1500-C is 6.12 and 6.07, respectively. The comparable AEPE values suggest that the warp prediction branch of our method performs similarly to RoMa. These two studies combined suggest that it is mainly the better certainty estimation that brings the improvement in performance.

• Q4. Is there a way to make use of the built-in uncertainty in the Dirichlet distribution?

One way to utilize the built-in uncertainty may be to use it as a weighting on top of the predicted reliability score, so that matches with high reliability score and low uncertainty score are more likely to be sampled in the following balanced sampling step. However, our preliminary study shows that this weighting strategy does not produce better results. How the built-in uncertainty in EDL can be used to improve feature matching performance is an interesting and relevant problem and we would like to explore it in the future.

• W1. Ablation study on coarse-scale and fine-scale EDL

We added an ablation study on applying EDL to different scales in Table 5 of the revised paper. We observe that applying EDL on the coarse scale alone is not effective, as the final prediction comes from the finest scale (which is still learnt by the BCE loss). Applying EDL on fine scales improve the performance on corrupted samples significantly. The best performance is achieved when EDL is applied on both coarse and fine scales.

• W2. Experiments on IMC2022 and WxBS

We added additional benchmarking results in A.3 of the revised paper. Compared to RoMa, our method obtains 1.7% increase in mAA@10px on WxBS, and 0.5% increase in AUC@3px on HPatches. For IMC2022, RoMa does not release their evaluation code. We need more time to reproduce their results and evaluate our method, and thus do not report the results in current version of the paper.

• W3. Computational cost of our method

We report the training and inference time of RoMa and our method in Table 6 of the revision . For training, our method takes 1.4% more GPU hours than RoMa (128.4 vs. 126.6 GPU hours). For inference, our method actually incurs marginally lower cost (327 vs. 329 ms). This is probably due to the fact that EDL uses simple operations like addition and division to obtain the final probability, eliminating the more complicated sigmoid computation in RoMa.

Thanks for the response. The authors presented results from the ablation study, experiments on the mentioned datasets as well as details on training and inference times. I think my concerns have been addressed and I will retain my positive rating.

• W1. Implementation details are missing

We added the required details in A.1 of the revised paper.

• W2. Needs stronger argument for why corruption and adversarial robustness is important for dense feature matching

Feature matching models are typically trained with 3D supervision, i.e., ground truth matches are established by using 3D information including camera pose and depth. 3D datasets are more expensive to collect and are usually smaller in size compared to 2D datasets. For example, the MegaDepth dataset used in our paper contains 260k images, while the scale of modern 2D image datasets is usually in millions (ImageNet-1k:1.3M, ImageNet-21k: 14M, JFT: 303M). The relatively small amount of training data may limit the model robustness to testing data that differ from the training distribution. Previous benchmarking datasets like IMC2022 and WxBS focus on evaluating images captured in different conditions, e.g., viewpoints, timings (day vs. night, years apart) and illuminations. However, these benchmarks do not consider those corruptions that are likely to occur in real world, e.g., various kinds of imaging noise, blurring caused by camera motion, reduced visibility caused by adverse weather. Adversarial attacks serve as a worst-case analysis of model robustness. When deploying the feature matching model in safety-critical applications, it is essential to understand the lower bound of correctness. Adversarial robustness is thus investigated in our paper. We have modified the Introduction section to better highlight the relevance of the proposed study.

• W3. Outdated evaluations for robustness

We added experiments on 3D Common Corruptions (3DCC) [Kar et al., 2022] and CosPGD [Agnihotri et al., 2024] in A.2 of the revised paper. For 3DCC, we evaluate on three corruption types (low light noise, ISO noise and color quantization). Our method demonstrates significant and consistent advantage over RoMa, achieving up to 9.4% increase in AUC@5 on MegaDepth-1500-3DCC. We also observe that for corruption types that require depth information for generation, e.g., motion blur and fog 3D, the generation code of 3DCC cannot work for the MegaDepth dataset, resulting in unrealistic corrupted images (some examples are provided in Fig.9). We need more time to look into the 3DCC code to adjust the parameters and thus do not provide the results for other corruption types in 3DCC. For CosPGD, our method achieves up to 2.8% gain over RoMa.

• W4. Using 2D Common Corruptions on other known datasets is not always a novel contribution

We have modified our contribution as ``We propose to evaluate the robustness of feature matching methods under common image corruptions and adversarial attacks, which has not been studied in previous work".

• W5. Almost redundant presentation of results in Table 1 and Figure 3

In Table 1, we not only provide the mean AUC for each corruption type, but also the clean AUC and the mean AUC for all corruption types, which cannot be achieved by drawing dashed lines in each subfigure of Figure 3. Also, we believe it is informative to provide a table summarizing the numerical results for each dataset. Therefore, we choose to keep Table 1.

• Q1. Evaluation details for adversarial attacks

The attacks are L-infinity norm bounded. The valid image space is [-2.1, 2.6] (the images are first scaled to [0,1] and then mean-std normalized by using ImageNet mean and std values). Attacks with epsilon=1 approximately modify pixel value magnitude by 20%, which is noticeable yet still semantically meaningful to human eyes. In practice, adversarial attacks are desired to be imperceptible so it is unlikely to use such large epsilon values. Here we experiment with [0, 1] to reveal the trend of degradation under different perturbation budgets. We have modified the description in Section 4.3 to provide the evaluation details to avoid confusion.

• Q2. Novelty of using evidential learning for pixel-matching task and difference form [Wang et al., 2022]

The different distributions mentioned in the comment (Normal Inverse-Gamma (NIG) distribution in [Wang et al., 2022] vs. Dirichlet distribution in our work) stem from the different formulations of evidential learning: [Wang et al., 2022] use the deep evidential regression formulation [Amini et al., NeurIPS2020], while ours use the classic classification formulation [Sensoy et al., NeurIPS2018]. The nature of task (stereo matching in [Wang et al., 2022] and certainty estimation in our work) necessitate different choice of formulations. The formulation in [Wang et al., 2022] cannot be used for our task -- it assumes a Gaussian distribution of predicted variable, which does not hold for a classification task. Therefore, despite both works dealing with pixel-matching task, it is formulated in very different ways. Our formulation of using evidential learning for certainty estimation has not been explored in previous work, which constitutes a novel contribution.

Dear reviewers,

As today is the last day you can post a message to us, may we check if our rebuttal has addressed your concerns? Is there any clarification needed? We appreciate you taking the time to read our revision and response. Please do not hesitate to let us know if you have any further concerns. We will try our best to address them.