Consensus Learning with Deep Sets for Essential Matrix Estimation

We thank the reviewer for these comments.

Q1. UNet:

Our network obtains as input an unordered set of point matches of apriori unknown cardinality. Such input is naturally suitable for DeepSets and other permutation equivariant architectures. Standard convolutional and the UNet architecture cannot handle such input, since it requires input on a grid. Graph convolutional architectures can be applied (and indeed are applied in previous methods, including CLNet, NCMNet, and also U-Match, which uses a graph UNet architecture). These methods, however, are more complex and, as our experiments show, are slower (see the attached Table R.1) and yield inferior accuracies (see Tables 1-2 in the paper).

Q2. Pruning amount:

For our ablation, we followed previous approaches, including CLNet, NCMNet, and BCLNet, which pruned half of the input matches. MGNet further performed an ablation study with different pruning ratios and showed that pruning reduces the accuracy of the predictions. We should emphasize that our confidence scores (Eq. 2 in the paper) perform “soft pruning,” which appears to be more effective than “hard pruning.”

Ablating the classification head:

We evaluate the performance of our model without the classification head. Due to the prevalence of outliers, performance deteriorates to chance levels (0.014% and 0.025% mAP5 score for in-scene and cross-scene, respectively). We further demonstrate the effectiveness of our classification head by providing its precision, recall, and F1 scores (Table R.3).

Q3. Hyperparameters ablation studies:

We include in Table R.5 an ablation of the number of NAC blocks (which affects the number of DeepSet layers) and the encoder dimension. These ablations further justify our choices.

Q4. Optimal Triangulation Method:

We will add these details to the paper.

Q5. Y-axis units in Fig. 3:

The Y-axis represents the number of image pairs.

Q6. ScanNet evaluation:

Indeed, we used SUN3D to follow the evaluation protocol of previous methods. For the rebuttal, we include in Table R.2 the results of cross-dataset generalization on ScanNet. We compared our results to NCMNet predictions. Our method achieves superior results on this highly challenging dataset. Since previous methods have not published their model trained on SUN3D, we trained NCMNet from scratch using the official code release.

Q7. Resources usage:

We address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference.

We thank the reviewer for these comments.

Q1.Set transformer ablation:

We tested the SetTransformer in a quick synthetic experiment. We trained both the SetTransformer and our DeepSet model on noise-free data. While our model achieves a highly accurate pose estimation of 86.52% mAP5 (due to the lack of noise), the SetTransformer model achieves much less accurate results of 66.79% mAP5. We believe the SetTransformer architecture suffers from the lack of global features and a narrow bottleneck in the ISAB block. We leave improvement of the SetTransformer architecture to future work.

Q2. Number of DeepSet layers:

We provide further ablations in Table R.5, showing that the removal of one NAC block (two blocks instead of three) reduces accuracy by nearly 7%.

Thank you also for bringing up the work of Zhang et. al [3]. Despite some architectural differences, our implementation respects the principle of keeping a “clean path” for the gradient descent (e.g., by utilizing skip connections in a similar design as in [3]), enabling the usage of multiple DeepSet layers.

We thank the reviewer for these comments.

W1.Denoising evaluation:

We include a summary box plot (Figure R.1) of the noise distribution before and after our key-point denoising, measured with respect to the ground truth essential matrix. The median of the mean reprojection error (over each pair) is reduced by 0.202 pixels and even more (0.246 pixels) for image pairs with pose error (maximum between the translation and rotation errors) lower than five degrees.

W2. Noise prediction:

In principle, our denoising module is trained with correspondences cleaned with the ground truth essential matrix, and so it is expected to predict noise-free key-point locations independent of the predicted essential matrix. Moreover, as our ablation indicates (Table 4 in the paper), this denoising process improves the accuracy of the essential matrix regression, indicating that it yields a better fit to true key-point positions rather than to the predicted model.

W3. Performance with different outlier ratios:

We address this question in Table R.4. As expected, accuracy drops as the fraction of outliers increases or as the number of inliers decreases. Somewhat counter-intuitively, we obtain somewhat low accuracy with small fractions of outliers when SIFT features are used. This occurs because our test data in this segment is small (only 30 pairs).

Q1. Meaning of “noise-free”:

We apologize for this confusion. Existing methods use the essential matrix estimated in inference to correct the positions of key-points. Our network, in contrast, learns to denoise these positions independently from the estimated essential matrix, as we further explained in W2 above. Of course, the noise-free points used in training are obtained by applying geometry (triangulation). We will rephrase this statement

Q2. Model size and Inference time:

We address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference. We further compare our runtime to RANSAC-based methods. With 100K iterations, RANSAC is significantly slower than our method. We note that RANSAC is implemented in CPU. Kornia[1] introduced a GPU RANSAC implementation for fundamental matrix estimation. We tested their resource usage on the same data to demonstrate the potential of GPU implementations for RANSAC. Using a batch size of 10000 samples, their model runtime and maximum GPU memory usage were 40.94ms and 414.66MB, respectively, which are 4 times higher than our model.

[1] Riba, Edgar, et al. "Kornia: an Open Source Differentiable Computer Vision Library for PyTorch.” Winter Conference on Applications of Computer Vision. 2020.

We thank the reviewer for these comments.

W1. Network capacity and complexity:

We address this question in Table R.1. While our model uses more parameters than NCMNet and BCLNet, which use graph attention architectures, it is 4-6 times faster than these methods and consumes less GPU memory at inference.

W2. Pruning and classification evaluation:

We discuss the effect of correspondence pruning in our paper (ablation is mentioned in Section 4.4, lines 246-255). Similar to MGNet, we see that pruning has a slight negative effect (~-0.8%) on accuracy. We further attach Table R.3 to show classification accuracy. It can be seen that our model achieves higher F1 scores compared to previous methods, possibly explaining our overall improved regression accuracy.