GFNet: Homography Estimation via Grid Flow Regression

Related Work:

1. Several deep homography methods have been proposed, for example, [1-3]. [1] estimates homography across resolution and modality; [2] generates a supervised homography dataset that simulates the real-world distribution; [3] directly represents the homography via an 8-rank flow field, eliminating the need for post-solving from flow fields.

Method Part:

1. The technical contribution (GRID FLOW REGRESSION) includes initially estimating the flow, then solving for homography, and learning the flow motion between fixed grids. The former seems to has been applied in previous methods [2]. I would like to discuss with the authors the advantages compared to it; the latter, in my view, can be more regarded as a useful engineering trick rather than a major technique contribution.

2. Leveraging priors in foundation models (DINO or Stable Diffusion) to produce features and estimate geometric transformations has also been seen in previous works, such as [4]. These works demonstrate that features from foundation models are very helpful for multi-modality tasks, such as semantic matching.

3. The proposed dataset generating method may not meet the realism criteria (the realism of frame content and inter-frame motion, please refer to [3] for more details), which is crucial for ensuring performance and generalizability. For example, the proposed method cannot simulate parallax changes or human walking.

Experiments:

1. I recommend that the authors conduct experiments, at least zero-shot inference, on recent deep homography datasets [2], which represent general real-world scenes, including parallax changes, dynamic foregrounds, and adverse conditions.

[1] CrossHomo: Cross-Modality and Cross-Resolution Homography Estimation. TPAMI 2024

[2] Supervised Homography Learning with Realistic Dataset Generation. ICCV 2023

[3] Unsupervised Global and Local Homography Estimation with Motion Basis Learning. TPAMI 2023

[4] Emergent Correspondence from Image Diffusion. NIPS 2023

However, I still have some concerns as follows and tend to raise my rating if the authors can properly address them.

• The proposed grid flow representation is similar to the classical 'mesh flow' (MeshFlow: Minimum Latency Online Video Stabilization) used in the video stabilization task. Their differences and limitations are expected to be clarified in the context of the homography estimation.
• The pixel-wise flow can flexibly adapt to different input resolutions, but it might be redundant to describe the limited DoF homography matrix, which typically only has 8 DoF. Do we really need such a dense flow in a homography estimation network? More discussions should be presented when applying the flow and grid representations.
• This work introduces an iterative flow regression approach to prevent suboptimal multi-scale flow optimization in challenging scenes. However, this iterative or progressive regression method has also been explored in previous flow estimation and image warping works. For example, "Raft: Recurrent all-pairs field transforms for optical flow", "MOWA: Multiple-in-One Image Warping Model", "Semi-supervised coupled thin-plate spline model for rotation correction and beyond", etc. The authors are suggested to highlight their unique contributions and provide some discussions compared with the above works.
• Some recent homography estimation works are also missing in this work. For example, "DMHomo: Learning Homography with Diffusion Models", "Supervised Homography Learning with Realistic Dataset Generation", "Depth-aware multi-grid deep homography estimation with contextual correlation", etc.
• Is the sota performance of this work mainly derived from the powerful DINOv2 backbone? Did the author try other backbone networks such as the classical ResNet?
• When performing the ablation study, an important experiment would be using the grid or 4-pt representation (widely used in previous homography estimation works) to compare the proposed grid flow representation in different resolution settings, including their specific estimation accuracy, complexity, and efficiency.

1. While the paper mentions that iterative grid flow regression reduces the computation load of global correlation at each scale, the improvement appears limited. The time complexity remains in the same order as pixel-based regression. To clarify the extent of the efficiency gain, it would be useful to see a detailed breakdown of computation time across different components or empirical runtime comparisons on various hardware configurations.

2. I couldn’t find any clear ablation study examining the impact of using grid flow regression on both accuracy and computational load. A comparison between the proposed grid-based approach and a pixel-based version, with other components held constant, could provide valuable insights.

3. While the grid flow regression aims to reduce the computational load, the addition of DINOv2 seems to offset this benefit, potentially increasing the overall computation. To better understand the computational trade-offs, it would be helpful to provide a detailed analysis, including a breakdown of computational costs for each component and how they balance out in the overall architecture.

4. To better clarify the novel contributions of GFNet beyond the use of grid flow and global correlation computation, it would be useful to provide a more detailed comparison between GFNet's iterative structure and that of MCNet.

5. The experimental results are not convincing enough, as many compared methods report their accuracies of MACE (in pixels), such as RHWF, MCNet, and PRISE, but this paper doesn't compare them. The results in Table 2 in this paper aren't consistent with the reported ones in the paper of RHWF, MCNet, and PRISE, as their accuracy is very high on MSCOCO and GoogleMap. Including MACE comparisons and addressing the discrepancies with the results reported in the RHWF, MCNet, and PRISE papers could enhance the clarity and robustness of the evaluation.