ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses

Thank you for your review and valuable comments.

1.1 What is the computation manner of the Transformer before the hypothesis estimation step? In line 237, the authors state that "then we perform transformer five times at M_1". Does the transformer contain some cross-attention processes? Or does it only involve the self-attention?

No, we only use this strategy for the fine level, because this approach cannot cope with the need to iterate the transformer multiple times, and it only manages to improve the efficiency without compromising the accuracy if it is only iterated once. This drawback stems from the fact that it only needs to optimize the feature that is used to estimate the refined bias, which is only one out of every 16 features on the source image on the $M_3$ feature map according to our method, and the features on the target image are not optimized in this uni-directional attention operation. Therefore, the scope of this uni-directional attention can only be used in refinement stage, and it is not suitable for the scenarios that require multiple iterations such as coarse level.

1.2 The computation process of the hypothesis estimation step is confusing.

We are trying here to use $a$ and $i$ to represent the units on the target images and source images respectively, and then $a_i$ to describe the unit on the target image that corresponds to unit $i$ on the source image. Since you find this confusing, we will add an $s$ or $t$ annotation in the upper right corner, for example: $f_k^{3s}$ and $f_k^{3t}$

1.3 The computation architecture of the segmentation step is unclear. The authors should provide the computation details on how to predict the classification confidence from the intermediate features. The statement in lines 177-180 is too brief to understand.

Segmentation refers to the classification of each unit, where we determine which homography hypothesis should be adopted for unit j on $M_2$ through classification. The way to obtain the classification result is by comparing the classification score matrix $C_j$ of unit $j$ for different hypotheses $H_i$, where the largest one is the result of our classification operation. This classification uses the concept of multi-label classification, a method widely applied in detection problems. Therefore, we refer to DETR and use focal loss to optimize segmentation here. We can describe the process of obtaining the classification score matrix $C_j$ in the form of a formula: $C_{ji} = (T(f_i) + P(i), f_j)$, where $C_{ji}$ refers to the matching score of unit j for hypothesis i. $T$ refers to the function that converts the feature dimension of i (256 dimensions) to the feature dimension of j (128 dimensions); here, we use a 2D CNN to perform $T$. $P$ refers to positional embedding, which directly represents the relative position of the unit corresponding to the hypotheses i in the local 3*3 units. And ( * , *) indicates the inner product. We will add this part to the supplementary materials later.

1.4 Why is the classification label in the segmentation step termed as the "pseudo" ground truth (line 181)? To my understanding, the classification label of every unit should be obtained from the ground truth camera pose and depth. It should be the "normal ground truth" rather than the "pseudo ground truth" if the above understanding is correct. The authors should clarify this detail.

Here we use the term 'pseudo ground truth' merely because the annotations are computed in real-time. However, considering that the term 'pseudo-labels' is generally used for labels obtained from the predictions of neural networks, this usage is indeed incorrect and your are right. We will correct this term to 'computed groundtruth'.

2.1 The statement "while we only need to feed 300" (line 50-51) was not clarified in the subsequent text. Is it the unit number on the 1/32 resolution for a 640x480 input image?

300 indeed refers to the number of units obtained at 1/32 resolution on a $640 \times 480$ resolution image. We will change this to: 'Previous methods feed $80 \times 60$ tokens to the transformer with 1/8 resolution, while we only need to feed $20 \times 15$ with 1/32 resolution.' This will be easier to understand.

2.2 In the segmentation step, what is the size of output segmentation results for a local input window whose size is 3x3? Is it 4x4? Or 12x12?

The output of the segmentation part is an H/8×W/8 array, with values ranging from 0 to 8. This number determines which local hypothesis should be adopted to calculate the input for the refinement step.

2.3 The statement in lines 304-305 should be further discussed. The authors just state that ETO is better on the more difficult YFCC100M without discussing the probable reason.

All of the methods we tested achieved lower metrics on the YFCC dataset. From this perspective, we consider the YFCC dataset to be more challenging.

2.4 The title "Hypotheses Estimation" (line 137) should be "Hypothesis Estimation".

Thank you for pointing out the typos. We will correct the typos later.

3.1 Some real intermediate results after the hypothesis estimation/segmentation steps should be visualized to show how these steps provide appropriate predictions. The virtual intermediate "results" in Figure 3 are helpful to understand, but the actual results are still necessary.

We provide the visualization results in the pdf of "Author Rebuttal".

Thank you for your review and valuable comments.

1. Though the time usage is decrease, the accuracy is also decrease from Tab.1&2&3.

Yes, you are right. But we claim that the huge improvement in runtime is valuable in realtime applications such as robotics or SLAM.

2. The paper lacks significant citations in feature matching methods, such as Efficient LoFTR, RoMa. Some of these works also focus on improving efficiency in feature matching and should be referenced to provide a comprehensive background. Including these methods in the experiments would offer a more thorough comparison of the proposed approach's performance.

We add a couple of experiments on Megadepth dataset in Author Rebuttal materials. Here, to be mentioned that our paper is contemporaneous with the Efficient LoFTR. The accuracy of Efficient LoFTR is a bit better than LoFTR, while the runtime of Efficient LoFTR is 56.9 ms, much faster than LoFTR, which is 93.2 ms, however, it is much slower than our 21 ms . And while PATS gets extremely good results, it is almost 100 times slower than our method.

3. Though the authors acknowledge that some other methods (e.g., [14, 38]) are better in certain aspects (line356-357), it would be beneficial to include a comparative analysis with these methods. This would provide a clearer understanding of the strengths and weaknesses of the proposed approach.

The same as Q.2

The experiments show that when efficiency and accuracy are all taken into account, ETO has advantages over the state-of-the-art methods. Note that tiny-roma's demo on github uses a very strong set of ransac parameters (ransacReprojThreshold=0.2, method=cv2.USAC_MAGSAC, confidence=0.999999, maxIters=10000) , so it's not surprising that the results decrease when using settings consistent with ours.

Thank you for your review and valuable comments.

1. The symbols used in the method part are too complicated, making it hard to read.

Thank you for pointing this out and I'm so sorry for it.

2. The symbol $\mathscr{H}$ used in figure 4 is undefined in the text.

Thanks for pointing this out, the $\mathscr{H}$ here refers to the output of segmentation, which is the set of homography transformations that each unit on $M_2$ is ultimately assigned to, we will add this explanation.

3. The framework adopts two-stage training, possibly making it hard to train and limiting the accuracy.

Yes, an end-to-end pipeline may improve our method. But we don't have enough computing resources to complete it. We will try this idea later in the next work.

4. Do you have any plan to reorganize the symbols?

Maybe I can directly add $s$ and $t$ to the features, from $f_k^3$ to $f_k^{3s}$ and $f_k^{3t}$ , it will be more clear.

5. If the framework is trained end-to-end, will it yield acceptable performance?

Then the batch size will be very small and make the process of training very slow, maybe an end-to-end pipeline will improve our method, but I don't have enough resources to complete it.

Thank you for your review and valuable comments.

1. The pre-compiled transformer model may have some impact on inference speed, can additional tests be conducted on the efficiency of a normal transformer module?

Without the pre-compiled transformer model, the runtime of ETO will be delayed from 21.0 ms to 32.8 ms for the images of 640*480. While the runtime is almost the same for LightGlue.

2. Lack of the latest state-of-the-art comparison.

To be mentioned that our paper is contemporaneous with the Efficient LoFTR. The accuracy of Efficient LoFTR is a bit better than LoFTR, while the runtime of Efficient LoFTR is 56.9 ms, much faster than LoFTR, which is 93.2 ms, however, it is much slower than our 21 ms . And while PATS gets extremely good results, it is almost 100 times slower than our method. More detailed results on Megadepth dataset can be found in the supplementary table inside Author Rebuttal.

3. In Section 3, it is 2-stage training adopted, but in Implementation Details, the training phase contains 3 stages. Why there is a joint training stage at first? Moreover, the joint training is usually conducted for the final finetuning, why it is the first stage in your strategy?

Our purples is to handle multi-resolution images better, which is of great significance for real-world applications, although it is not demonstrated in standard experimental data in this paper.This step is placed in the first stage rather than the second, this is because the different resolutions of the images mean that in the coarse matching stage one has to deal with data inputs of variable sizes, which has a relatively large impact on the results. Correspondingly, the fine matching stage can use inputs of fixed size from the coarse stage and is therefore inherently more robust to resolution changes.

4. For Basic Refinement w/ Segmentation in Section 4.4, why it is the same training hours instead of the same training samples? Early stopping may lead to performance drop.

We try to demonstrate our superiority in training efficiency. While using the same training samples, the accuracy of this method will be 52.3/67.1/77.9 (auc@5/auc@10/auc@20), and the runtime is still 32.8 ms. While our accuracy is 51.7/66.6/77.4 (auc@5/auc@10/auc@20), and the runtime is 21.0 ms.

5. In the introduction, the authors claim that the multiple self- and cross-attention in the fine-level stage are redundant, is there any numerical results? What if you stack several uni-directional cross-attention in your method?

The numerical results are in given in Q.4, only a simple version of cross attention can get similar accuracy as complete transformer with self-attention and cross attention, while much faster.

Thanks for the response.

Although most of my quetions are solved. There are some questions ignored by the authors.

1. In Q3, I wonder if stacking some more the proposed uni-directional cross-attention would be helpful and makes the method more powerful.

2. The authors compared with PATS instead of RoMa as I mentioned in W2. Additionally, there are also some other reviewers mentioned RoMa, I don know why the author try to avoid comparing with RoMa. In my experience, RoMa runs about ~180ms/Frame on MegaDepth with RTX3090, and there is a tiny version for RoMa. Acoording to the paper and official repo, the accuracy for RoMa (62.6/76.7/86.3) and Tiny-RoMa (56.4/69.5/79.5) are much better than the proposed method (although it is faster).

The author's evasiveness heightens my concern. I do not know if stacking the proposed uni-directional cross-attention is useless. And I do not know why ignoring RoMa, which has been open-sourced for a long time. However, it is still an interesting work, I will maintain the original rating (Weak Accept).

The experiments show that when efficiency and accuracy are all taken into account, ETO has advantages over the state-of-the-art methods. Note that tiny-roma's demo on github uses a very strong set of ransac parameters (ransacReprojThreshold=0.2, method=cv2.USAC_MAGSAC, confidence=0.999999, maxIters=10000) , so it's not surprising that the results decrease when using settings consistent with ours.

And ETO with 2 uni-directional attention only make a littile difference on performance which is not important,.We consider that it is because the uni-directional attention only manages to improve the efficiency without compromising the accuracy if it is only iterated once. This drawback stems from the fact that it only needs to optimize the feature that is used to estimate the refined bias, which is only one out of every 16 features on the source image on the $M_3$ feature map according to our method, and the features on the target image are not optimized in this uni-directional attention operation. Therefore, more uni-directional attention can not make great improvements for our method.

Thanks for the response, which partly addressed my question.

According to the response, stacking the uni-directional attention achieves improvement on auc@5 but get degreation on auc@20 (both are small), and it seems that you did not re-train a new model (due to the rebuttal schedule) but simply reuse the attention parameters.

When it comes to the comparison with RoMa and Tiny-RoMa, thanks for your insight of the experiment settings. As the metric is the accuracy of camera pose ,which only need several matches (eg. 8 matches) to regress, it is acceptable to filter out less-reliable matches when computing camera poses. Maybe you can also test your method with a strict setting (just a suggestion, not required).

Nevertheless, this work is interesting and provides a effective and efficient mothod for 2-view matching.

Thank you for your review and valuable comments.

1. Some descriptions in Sec 3.4 need further clarification.

Yes. Due to space limitations, we have included a more intuitive description and illustrations of this step in Sec.2 in the supplementary materials. Specifically, since the corresponding point $P_j^s$ on the source image is self-chosen, following LoFTR and for computational convenience, we select a point $P_j^s$ at the center of each specific feature $\hat{f}_j^3$ on the $M_3$ feature map. For the target image, after calculating the initial $P_j^t$ coordinates through the accepted $H$ hypotheses, we query the 77 features around $P_j^t$ with $\hat{f}_j^3$. We then perform cross attention between this single feature from the source image and the 49 features from the target image. If we were to use traditional methods to perform a complete transformer, we would need a 4949 transformer to achieve such a large receptive field, but here we only need a 1*49 transformer.

2. How is $\Delta P_j^t$ in Line 188 obtained or is it predicted?

After refining $\hat{f}_j^3$ using the above method, we follow traditional methods (such as LoFTR) to correlate this feature with the 49 features on the target image, we obtain a local heatmap representing the matching probability for each feature. By computing the expectation over this probability distribution, we can obtain the final offset for matches.

3. In Figure 3, is there an MLP after cross-attention block? If yes, what is the input/output. Also, the presence of “Module 3” is confusing, should it contain the cross attention?

Here, Module 3 refers to the entire refinement part, including the cross-attention part. The MLP shown in the figure follows traditional methods(like PATS, ASpanFormer or LoFTR) in the module of transformer, passing the feature through an MLP module before computing the expectation. Its function is to further enhance the $\hat{f}_j^3$ mentioned in Section 3.4.

4. In Table 4, it seems the usage of bidirectional cross-attention performs worse than that of uni-directional cross-attention at 5. Is that reasonable, please give some analysis.

This is because we are using the same training time rather than the same number of training samples as the comparison criterion, which makes a small but not significant difference to the effectiveness of the model. If we had used the same training samples, auc@5 would have achieved 52.3, which is better ours though while still much slower.

5. From Table 2, the performance of lightGlue is much worse than ASpanFormer, LoFTR. Interestingly, ASpanFormer is better than LoFTR with a small margin in the Megadepth dataset, similar to the results reported in LightGlue. The performance gap between LightGlue and LoFTR is much larger than that in LightGlue, could you please explain? Moreover, why LightGlue is not reported in the dataset of Scannet?

It is important to note that the LightGlue employs two RANSAC methods in their paper: Lo-RANSAC and OpenCV's RANSAC. We use OpenCV's RANSAC exclusively. In the experiments of the LightGlue, it is evident that under OpenCV's RANSAC, LightGlue's performance significantly decreases. Additionally, to better demonstrate the impact of sub-pixel precision, we use a RANSAC threshold of 0.25 in outdoor scenarios. Under this threshold, LoFTR, ASpanFormer, and our method ETO achieve smaller errors within this precision range, further widening the gap with LightGlue. The reason LightGlue was not tested on ScanNet is that its model was not trained on indoor scenes, which would be unfair.

6. Some of the network architectures are not depicted in figures, making it difficult to know the overall diagram of the method.

Due to space limitations, we have placed the illustrations of the refinement steps in the supplementary materials.

7. The two-stage training process may not be easily trained.

Yes, an end-to-end pipeline may improve our method. But we don't have enough computing resources to complete it.

8. It would be better to present some failure case studies.

There exists a shortcoming of our approach, which is that in the case of very complex planar assemblages, we provide too few planar proficiencies, and it is also more difficult to distinguish. It can be found in the Figure.1 of our Author Rebuttal.