Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization

Thank you for taking the time to review our work. We apologize for the lack of detailed explanations regarding the proposed method. Below are some additional clarifications:

Weakness 1.

Weakness 1.1. No other direct supervisions

Increasing the similarity of local features between two inputs is sufficient and equivalent to estimating homography. This is because homography estimation assumes a linear transformation between pairs of static scenes. Therefore, smoothness is automatically satisfied, and global motion is equivalent to local motion and homography.

Weakness 1.2. How is AltO applied to iterative frameworks such as RAFT, IHN, or RHWF?

We used each prediction from the registration network to individually calculate the GBT loss. Then, all GBT losses at each time step were combined by weighted summation, similar to the supervised learning situation. The weights used are also borrowed from the supervised learning case. The only difference from the supervised case is that AltO is used instead of the ground truth. If we had not effectively utilized the full capability of the registration networks, we would not have been able to achieve such performance.

In Figure R1 of the attached rebuttal PDF, there is an illustration of the explained details, so please refer to it.

Weakness 1.3. How can RAFT be used for homography estimation?

A motion flow map from an optical flow network, such as RAFT, can be regarded as HW-correspondences (H: height, W: width). If HW is 4 or larger, the Direct Linear Transform (DLT) algorithm can fit a single homography. This homography best approximates the flow map in terms of least squared error.

Generally, in homography estimation, which has 8 degrees of freedom, most algorithms predict only 4 correspondences at the corners of images. They then convert these correspondences to a homography using DLT. Therefore, there is no reason HW-correspondences cannot be used for estimating homography.

If you are concerned about secondary aspects of optical flow, such as smoothness or the correction between global camera motion and local motion, please refer to our response in Weakness 1.1. As we explained there, homography estimation is estimating a linear transformation between pairs of static images. Therefore, these issues do not arise.

Weakness 1.4. What is the effectiveness of Modality-Agnostic Representation Learning (MARL)?

It is to train the encoder that maps input images to the same feature space, not to train registration network directly. This goal allows our geometry loss to function properly regardless of modality difference. To achieve this goal, the loss term of MARL should be designed to enhance global similarity, rather than the local similarity of corresponding points, as is done with the GBT loss. This is why the global average pooling (GAP) layer should be included at the end of the projector.

Table R2 shows the training results on Google Map dataset when GAP is not applied and spatial information is preserved to compute the similarity of local correspondences. It can be observed that proper training is not achieved in this case because local similarity is already met by the GBT loss, without considering global similarity.

Table R2: Ablation results for including and excluding GAP.

Weakness 2. Effectiveness of the Proposed Method. (Why is alternating necessary?)

To prevent unintended collaborations and a collapse into trivial solutions, we introduced an alternating training strategy. Training the encoder and registration network together end-to-end without alternating can cause the encoder to output a constant value and the registration network to always output the identity matrix as the homography. Alternating training isolates these modules. This ensures the encoder maps input images to the same feature space, and the registration network estimates the true homography between the two images.

Table R1 in the Auther Rebuttal (above) shows the experimental results when alternating training is not applied, demonstrating that proper training does not occur.

Weakness 3. & Limitation. Insufficient Experimental Validation

You pointed out that the difficulty of the DeepNIR dataset is reduced due to the presence of simulated pairs. So we conducted experiments on the 'RGB-NIR' [1] dataset, which consists purely of RGB and NIR pairs without any synthetic images. We selected 'IHN' as the registration network, as it represents our primary model. The experimental results, as shown in Table R3 below, indicate that all methods performed better on this dataset compared to DeepNIR. This suggests that the DeepNIR dataset is harder. We believe this is because the DeepNIR dataset requires the model to learn two distributions (real NIR and synthetic NIR images), whereas the RGB-NIR dataset only requires learning a single distribution (real NIR).

Additionally, we conducted experiments on the 'OS Dataset' [2], which is a VIS-SAR type dataset, as per your suggestion. The results are also presented in Table R3.

In conclusion, our method outperformed other unsupervised methods on both of the additional datasets.

Table R3: MACE evaluation on two datasets.

[1] Matthew Brown et al., Multi-spectral sift for scene category recognition. CVPR 2011.

[2] Xiang, Yuming, et al., Automatic registration of optical and sar images via improved phase congruency model, IEEE Journal of Earth Observations and Remote Sensing, 2020

Dear Reviewer EsVS

Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review:

• Provided detailed answers to all the questions you raised.
• Conducted a new experiment to demonstrate the effectiveness of MARL and the global average pooling (GAP) layer.
• Conducted a new experiment to demonstrate the effectiveness of alternating training.
• Conducted new experiments on additional datasets to demonstrate the generalization capability of our AltO framework.

If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work.

Thank you once again for your time and consideration.

Best regards, Authors

Thanks to the authors for the rebuttal. I have read the rebuttal and other reviewers' comments. However, I still have the following concerns:

(1) In Weakness 1, my concern is mainly about why the GBT loss can enable the homography network to converge well without direct supervision of global/local motion. In my view, the GBT loss can be seen as the 'cross-modal image intensity similarity' mentioned in SCPNet, which may lead to non-convergence. Is there any theoretical basis or proof to support this?

(2) Figure R2 further raises my concern about the experimental results. The visualized feature maps are not similar at all. They barely retain the structural information of the original images and contain many artifacts. I am not convinced that such feature maps can effectively supervise the registration network. What's more, in Table 2 in the paper, the author claims that simply using MSE on such feature maps (the mean of squared error of the intensity of the feature maps in Figure R2) as the Geometry loss can produce a relatively accurate result, which also reduces the credibility of the experiments.

(3) Insufficient references and comparison experiments. As mentioned by reviewer sFim, this paper did not discuss many references. Moreover, comparison experiments with other methods should be conducted on all datasets to demonstrate the effectiveness of AltO, instead of only on Google Maps.

(4) The homography estimation accuracy on the OS Dataset is unsatisfactory with the MACE of 14.12, which may not be regarded as a converged training. The effectiveness of the proposed method is insufficient on such cross-modal dataset.

For the reasons mentioned above, I am still inclined to reject.

(1) We apologize for the misunderstanding. It seems we may have overcomplicated your question. To clarify, let’s first look at UDHN [1], the pioneering unsupervised method in a same-modality scenario, which might address most of your concerns. UDHN predicts four corresponding points, then converts them into a homography using Direct Linear Transformation (DLT). The source image is then warped using the predicted homography, and a reconstruction loss or intensity-based similarity is calculated with the target image. This entire process, including DLT and warping, is differentiable and converges very well in a same-modality scenario. A similar approach is taken by biHomE [2], and both UDHN and biHomE are incorporated into our proposed method.

Our method targets multimodal datasets. Therefore, we internally convert them to a uni-modal condition to apply an approach similar to UDHN or biHomE. To facilitate this multimodal-to-uni-modal conversion, we introduced MARL and alternating training. Thus, the GBT loss is calculated under uni-modal conditions thanks to the encoder's mapping.

In the aforementioned processes, although methods like RANSAC are not used, the approaches work well. It seems you might be considering feature-based frameworks like SIFT, SURF, ORB or methods like LIFT [3] and Super Point [4], which replace parts of this process with deep learning. However, our method falls within the category of end-to-end homography estimation methods, like DHN [5], UDHN, and biHomE. These methods are composed entirely of differentiable processes for end-to-end learning.

Rest assured, we are fully aware of the fundamentals of feature-based frameworks and understand practical know-how, such as:

• If the corresponding points are clustered in one region or many lie on a straight line, the error in homography estimation increases significantly.
• Conversely, the homography estimation becomes more accurate when the corresponding points are widely distributed across the entire image.
• If only a few corresponding points are known, even with methods like RANSAC, the homography estimation error can increase due to outliers.

Finally, regarding Figure R2, please consider that this feature map is simply an average across 128 channels. Since our GBT is based on the Pearson correlation coefficient, the strength level (bias) difference between the two feature maps is not important; only the similarity in trends between the two feature maps reduces the loss. Additionally, both MSE and GBT, it can allow a few strongly activated regions (not just single points, but multiple points) to accurately infer the overall homography. This can be empirically observed by looking at the regression values computed by the registration network. Again, unlike explicit methods like RANSAC, the registration network implicitly calculates these values through learning.

[1] Ty Nguyen et al., Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics Autom. Lett., 2018.

[2] Daniel Koguciuk et al., Perceptual loss for robust unsupervised homography estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021.

[3] YI, Kwang Moo, et al. Lift: Learned invariant feature transform. In: Computer Vision–ECCV 2016

[4] DETONE, Daniel et al., Superpoint: Self-supervised interest point detection and description. In: CVPRW 2018

[5] Daniel DeTone et al., Deep image homography estimation. CoRR, abs/1606.03798, 2016.

(2) We now understand that your main concern is whether the encoder's output can maintain geometric consistency. We agree that the lack of an explicit mechanism could be a potential weakness, as you mentioned. However, as shown in Table R2, removing the global average pooling layer is not an option. We think that further improvements may require additional processing directly on the encoder’s output to address this issue.

Additionally, regarding the discussion on MSE, please refer to the last paragraph of (1).

(3) We agree that the OS dataset is more multimodal than the Google Map dataset, so further research will need to focus on this dataset.

Thank you for your detailed and ongoing discussion.

We regret that our explanation did not fully satisfy you, but we appreciate your engagement in the discussion until the end.

(1) As mentioned, UDHN and biHomE assume a uni-modal scenario. Our main contribution lies in expanding this to a multimodal application by introducing MARL and alternating training. Regarding the iterative framework, we retained the original structure of RAFT, IHN, and RHWF, simply replacing the ground truth with AltO. This is why we did not emphasize it in the paper, but we demonstrated that it can be applied without significant issues through the main experiments.

(2) Evaluations of the visualized feature maps are subjective, so we believe further discussion on this might not be productive.

(3) In our previous comment, we mentioned that "it can allow a few strongly activated regions (not just single points, but multiple points) to accurately infer the overall homography." In other words, even with only a few distinctive regions in a solid image, the registration network can capture these and perform homography estimation relatively accurately.

Thank you for taking the time to review our work in detail. Below are our responses to your comments and concerns.

Weakness 1.

Weakness 1.1. Why is alternating necessary?

To prevent unintended collaborations and a collapse into trivial solutions, we introduced an alternating training strategy. Training the encoder and registration network together end-to-end without alternating can cause the encoder to output a constant value and the registration network to always output the identity matrix as the homography. Alternating training isolates these modules. This ensures the encoder maps input images to the same feature space, and the registration network estimates the true homography between the two images.

Table R1 in Auther Rebuttal (above) shows the experimental results when alternating training is not applied, demonstrating that proper training does not occur.

Weakness 1.2. What is the superiority of the proposed Geometry Barlow Twins (GBT)?

GBT is robust in both cases: whether the i.i.d. assumption of dataset holds or not. The method you suggested, using NHW as a batch, matches GBT's performance only when the i.i.d. assumption is satisfied. In the Google Map dataset, where the i.i.d. assumption is met, both loss functions perform similarly (Table R2). However, in the case shown in Figure R3 of the rebuttal PDF, the i.i.d. assumption is not met, and GBT performs better.

The key point is whether the distribution over NHW dimensions is similar to that over HW dimensions. If the i.i.d. assumption is not met, the distribution over N distorts NHW, leading to a distribution dissimilar compared to HW dimensions.

Table R2: Comparison of two types of Geo. loss on the Google Map dataset.

Weakness 1.3. & Weakness 2. The proposed approaches should be compared with some recent unsupervised approaches and hand-crafted approaches.

Sorry for the insufficient number of baselines. We have added more baselines to Table R3 below. Some of the entries in Table R3 cite SCPNet [1], a very recent paper accepted at ECCV 2024. Nonetheless, our method demonstrates even superior performance compared to SCPNet.

Additionally, the paper [2] you mentioned is designed to address the 6-DOF (degree of freedom) problem, so it cannot be applied to our dataset, which has 8-DOF. Furthermore, the code for that paper is not fully available. Moreover, we also attempted to run the code for paper [3], but encountered errors with the sample images, so we could not include it in our results.

Table R3: MACE evaluation results on the Google Map dataset. * indicates values reported from SCPNet [1].

[1] Runmin Zhang et al., SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning, (ECCV 2024 Accepted.)

[2] Yuanxin Ye et al., A Multiscale Framework with Unsupervised Learning for Remote Sensing Image Registration, IEEE Transactions on Geoscience and Remote Sensing, 2022.

[3] ZHANG et al., Histogram of the orientation of the weighted phase descriptor for multi-modal remote sensing image matching. ISPRS Journal of Photogrammetry and Remote Sensing, 2023.

Weakness 3. The discussion of the motivation is not sufficient.

The problem we aim to solve is the unsupervised learning of 8-DOF homography estimation for multimodal image pairs. These conditions are very common in many fields, such as industry. For example, matching a real photo to a CAD image. At the time of our research, we found very few papers that directly addressed this specific problem. This scarcity of relevant work highlighted the difficulty and value of addressing this challenge, which motivated us to pursue it.

The papers you suggested, [2] and [4], do not directly apply: [2] focuses on a 6-DOF problem, and [4] employs a supervised learning approach.

[4] A Novel Coarse-to-Fine Deep Learning Registration Framework for Multi-Modal Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 2023.

Weakness 4. This paper misses some references to hand-crafted and unsupervised approaches.

We apologize for missing them. We will include and refer them during the revision.

Question 1. Is there some literature that mentions the relationship between 3D reconstruction and homography estimation?

Below are several documents regarding your question. We will revise the citation section related to 3D reconstruction in our paper.

• Zhang, Zhongfei, and Allen R. Hanson. "3D reconstruction based on homography mapping." Proc. ARPA96 (1996): 1007-1012.
• Mei, Christopher, et al. "Efficient homography-based tracking and 3-D reconstruction for single-viewpoint sensors." IEEE Transactions on Robotics 24.6 (2008): 1352-1364.
• https://github.com/ziliHarvey/Homographies-for-Plane-Detection-and-3D-Reconstruction
• Yong-In, Yoon, and Ohk Hyung-Soo. "3D reconstruction using the planar homograpy." The Journal of Korean Institute of Communications and Information Sciences 31.4C (2006): 381-390.
• Zhang, Beiwei, and Y. F. Li. "An efficient method for dynamic calibration and 3D reconstruction using homographic transformation." Sensors and Actuators A: Physical 119.2 (2005): 349-357.
• Dubrofsky, Elan. "Homography estimation." Diplomová práce. Vancouver: Univerzita Britské Kolumbie 5 (2009).

Dear Reviewer sFim

Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review:

• Conducted a new experiment to demonstrate the effectiveness of alternating training and MARL.
• Conducted a new experiment and presented a case to compare GBT with the new method you suggested.
• Added more baselines for performance evaluation by conducting a new experiment and incorporating reported values.
• Provided detailed answers to all the questions you raised.

If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work.

Thank you once again for your time and consideration.

Best regards, Authors

Thank you very much for raising the rating of our paper. We appreciate your recognition of the additional experiments and discussions we provided, and we are glad that they addressed your main concerns.

We would like to provide further clarification:

1. Intuitive Explanation for End-to-End Training without Alternating:

If the encoder and registration network collapse as we mentioned, then all reconstruction-based losses, including L1-loss or our GBT loss, would become zero, the minimum possible value for such losses. This occurs because the encoder's outputs are constant, resulting in the difference or similarity always converging to a trivial value, regardless of the homography. Consequently, the registration network would not make any effort to find the true homography, outputs only identity matrix.

2. Difference Between Our Approach and Literature [2]:

As you mentioned, extending from 6-DOF to 8-DOF is indeed straightforward, but it is a more challenging task, and some performance degradation is to be expected. The CFOG, used in [2], seems to be a key element. The multi-scale framework is aimed at more precise estimation, but it does not address different modalities. Additionally, since the code is not fully available at the moment, quickly reproducing and modifying it is difficult. However, we fully agree that the extended version of [2] would certainly be a valuable baseline to consider.

We hope these clarifications address your questions, and we are more than happy to discuss further if needed. Thank you once again for your valuable insights and support.

Thank you for your detailed review and for taking the time to provide your feedback. Below is our rebuttal.

Weakness 1. Why not use an edge-based approach as a baseline?

Edge-based approaches have limitations when used as baselines with different modalities. When dealing with two images of different modalities, one image often has rich edges while the other does not. Even within the closed modality, such as day and night photos, edge differences can arise due to shadows. Thus, converting both images to edge maps and comparing them might not achieve high performance. Although paper [1] utilizes edges in its learning process, it is limited to using image pairs of exactly the same modality.

Below Table R2 shows the results of applying UDHN [2], a simple unsupervised method that assumes the same modality, to edge maps converted from the Google Map dataset. The results indicate poor performance.

Table R2: Evaluation result of applying edge maps to UDHN

[1] Feng, Xiaomei & Jia et al., Edge-Aware Correlation Learning for Unsupervised Progressive Homography Estimation, in IEEE Transactions on Circuits and Systems for Video Technology, 2023.

[2] Ty Nguyen et al., Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics Autom. Lett., 2018.

Weakness 2. Demonstrate the effectiveness of each modules, GL and MARL.

The GL module enhances geometry alignment by increasing the similarity between local correspondences. It is inspired by biHomE [3], with the only differences being that biHomE uses an ImageNet-pretrained encoder which is frozen, and applies triplet loss as the loss term. The effectiveness of this approach has already been demonstrated in the paper [3] using the S-COCO dataset, which has the same modality.

The role of the MARL module is to train the encoder to map input images to the same feature space. This properly trained encoder allows our geometry loss to function correctly regardless of modality differences. Table R1 in the Author Rebuttal (above) shows the results of training on the Google Map dataset without the MARL module or with the MARL module but no alternating. The result indicates that proper learning did not occur.

In summary, for successful training, the GL module, MARL module, and alternating training are all essential components.

[3] Daniel Koguciuk et al., Perceptual loss for robust unsupervised homography estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021.

Question 1.

Question 1.1. What is the size of the embedding?

AltO serves as a learning framework that does not impose restrictions on the internal structure of each component. However, the encoder and projector used as examples in our paper are based on ResNet-34. The encoder uses the first two stages of ResNet-34, resulting in an embedding size of 128. Nevertheless, these can be flexibly replaced if needed.

Question 1.2. What is the training time?

Table R3 below shows the training time measurements on Google Map dataset. Although the training time increases, the inference time at run-time is the same as that of supervised learning since the AltO module is not needed.

Table R3: Training times for each registration network. (Nvidia RTX 8000)

Question 1.3. Are the Barlow Twins trained from scratch?

There are no pretrained components. All layers are trained from scratch.

Question 2. Are strong geometric features (i.e., lines) necessary for the proposed method?

It is evident that lines, such as edges, are helpful. However, as visualized in Figure R2 of the rebuttal PDF, the output of the encoder shows strong features in the form of blobs rather than edges. This indicates that not only lines but also blobs play a significant role in the proposed method.

Limitation. Small image size and strong overlapped pairs

The dataset settings we used (such as size and overlap) are based on the pioneering paper DHN [4] on end-to-end homography estimation. These settings have become the standard and are used by many studies, including [1-7]. For fair comparison, we followed these settings. However, AltO, as a learning framework, has no size or overlap restrictions. Constraints, if any, come from the registration network (e.g., IHN), encoder, or projector.

Table R4 shows experiments where the Google Map dataset size and displacement were doubled, called 'Google Map x2.' We used DHN as the registration network, which is flexible with size constraints. The results show that DHN with AltO can reduce MACE even without any hyperparameter tuning.

Table R4: MACE evaluation results on Google Map x2.

[4] Daniel DeTone et al., Deep image homography estimation. CoRR, abs/1606.03798, 2016.

[5] Yiming Zhao et al., Deep lucas-kanade homography for multimodal image alignment. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021.

[6] Si-Yuan Cao et al., Iterative deep homography estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022.

[7] Si-Yuan Cao et al., Recurrent homography estimation using homography-guided image warping and focus transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023.

Dear Reviewer PrYW

Thank you once again for participating in the review process and for providing such thoughtful feedback. We wanted to kindly follow up regarding the rebuttal we submitted. We understand that this is a busy time, and we greatly appreciate your efforts. To summarize our rebuttal to your review:

• Conducted a new experiment to demonstrate the performance of the edge-based method in multimodal case.
• Conducted a new experiment to show the effectiveness of alternating training and MARL.
• Provided detailed answers to all the questions you raised.
• Conducted a new experiment to demonstrate that our method, AltO, is also effective for larger image set.

If there are any further questions or clarifications needed from our side, we would be more than happy to provide them. We look forward to any feedback you might have and are eager to engage in any further discussion to improve our work.

Thank you once again for your time and consideration.

Best regards, Authors

We would like to thank you again for highly appreciating the strengths of our work.

If you have any further questions or concerns, please feel free to ask. We would be more than happy to provide a thorough response to any additional inquiries.

Best regards, Authors