Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence

The dimension of concatenation in Eq.3.

We apologize for any confusion caused by the presentation of our methodology and appreciate the opportunity to clarify. To enhance understanding, we will include the dimensions of all tensors in our method section.

Addressing the reviewer's query regarding the concatenation process, it is performed as follows: We start with a cost volume of shape h×w×h×w. To this, we concatenate a feature map of shape h×w×c, treating one of the h×wh×w dimensions as the spatial dimension. This results in a combined shape of h×w×(hw+c), effectively merging the cost volume and feature map information.

Regarding Equation 5, the use of C′ is an intentional workaround to avoid the conventional computation of the attention map, typically represented as QKT in standard attention mechanisms. Instead of computing the attention map traditionally, we utilize the cost volume to transform it into a matching distribution. This approach effectively functions as a cross-attention map. The effectiveness of this method is validated in Table 4, where we demonstrate its practical utility and performance. We hope this explanation clears up any ambiguities. If there are further questions or additional clarifications needed, we are more than willing to provide further details.

Include more recent works in Table 2, and GMFlow is trained on different dataset. Verify effectiveness of the proposed method in optical flow.

We apologize for any oversight in our initial presentation of recent methods in our comparative analysis. While Table 1 comprehensively includes recent methods, we acknowledge that Table 2 appeared somewhat outdated. To address this, we have now included an evaluation of DKM, a method that has been tested on HPatches, with its results available in its public GitHub repository. We have compared these results with those of our method:

As illustrated, our method demonstrates highly competitive performance, even when juxtaposed with newer methods like DKM.

Regarding the comparison with GMFlow, we recognize and apologize for the previously unfair comparison. Unlike other methods in our comparison that were trained on MegaDepth, GMFlow was not, which might have skewed the results. To address this and to further evaluate the generalizability of our method as suggested by the reviewer, we conducted additional evaluations in the context of optical flow. For these evaluations, we utilized the KITTI and SINTEL datasets. Aligning with the practices of GLU-Net, PDC-Net, and GoCOR, our method was not trained on the Chair and Things datasets but rather on MegaDepth. The results of these additional evaluations are presented below:

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness.

We thank the reviewer for the constructive comments! Our responses can be found below:

Unsure whether the term "aggregation" is suitable for operations performed over cost volumes.

In the stereo matching literature, "cost aggregation" plays a pivotal role. This process involves combining primary matching costs, typically through a filtering process, to generate a comprehensive disparity map, as noted in source [A]. The scope of "aggregation" in this context is broad and versatile. It can encompass various techniques ranging from the application of simple Gaussian kernels to more complex methods like 3D convolutions, as highlighted in source [B]. Additionally, the term also covers techniques such as semi-global or local attentions, as well as other forms of attention mechanisms, indicated in source [C].

These diverse aggregation techniques serve to illustrate the multifaceted nature of cost volume learning in stereo matching. The term "aggregation" is not merely confined to the domain of attention-based feature combination but extends to a variety of methods, each contributing to the accurate and efficient processing of stereo imagery. This wide-ranging application of the term confirms its appropriateness and relevance in the field, demonstrating that cost volume learning indeed benefits from and incorporates a spectrum of aggregation strategies. However, we can change the term to "refinement" if it better fits to describe our approach. We highly appreciate the reviewer's comment in this regard.

[A] Cross-Scale Cost Aggregation for Stereo Matching [B] Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation [C] GA-Net: Guided Aggregation Net for End-to-end Stereo Matching

Evidences to claim that feature matching is challenged by repetitive patterns and background clutters and cost aggregation enjoys robustness to repetitive patterns and background clutter.

There is a well-established understanding in computer vision that feature matching can be particularly challenging in scenes with repetitive patterns and cluttered backgrounds. This difficulty arises from the reliance of feature matching techniques on identifying and aligning distinct keypoints or features between images. In environments where textures are repetitive or backgrounds are complex, these features often become ambiguous, leading to mismatches and inaccuracies. This limitation is widely recognized in the literature and has led to the development of enhanced feature detection algorithms, such as SIFT, and more recent methods like LOFTR, which aim to refine local features to extract high-quality matches in indistinctive or low-texture regions.

To support our claim, we will reference these matching methods in our paper. They exemplify the ongoing efforts in the field to address the challenges posed by such complex environments. However, we acknowledge that the assertion regarding the robustness of cost aggregation to these challenges might be contentious. Nevertheless, while our paper does not provide specific empirical evidence to support this claim (although we provide visualizations that can explain such robustness in Fig. 4 to some extent), it is nonetheless informed by a substantial body of research in the semantic matching literature. Numerous studies in this area have demonstrated the effectiveness of cost aggregation in various contexts.

Nevertheless, in light of the comment, we agree that it is prudent to moderate our claim regarding the robustness of cost aggregation to environments with repetitive patterns and cluttered backgrounds. We recognize that this aspect could be seen as a hypothesis rather than a conclusively proven fact. We appreciate the constructive comment and will accordingly adjust the tone of our claim to reflect a more cautious stance on this matter.

Thanks for the reply!

1. We wish to emphasize that although we said that we will tone down the sentence, this does not mean that the substantial body of study in the semantic matching literature has been adopting cost aggregation for no benefits. It has been recognized by several works that cost aggregation enjoys strong generalization power [A, B], many works in semantic matching adopts cost aggregation to account for the background clutters, repetitive patterns and intra-class variations and finally, we also clearly showed adopting cost aggregation boosts the performance. Moreover, our contribution should not be diminished from the toned down claim, but rather be enhanced since we managed to show the improved performance in semantic matching. This is an evidence that our method better considered such challenges.

2. We agree that for solid comparisons, we should add DKM if we include pose estimation and visual localization. However, we wish to highlight that these sparse matching tasks are beyond our paper's scope and we already evaluated our method on 5 dense correspondence benchmarks, and also yielded competitive performance on additional benchmarks we used for this rebuttal. We also discussed a limitation that the typical dense matching method faces for sparse matching tasks (while DKM is also a dense method and it attains SOTA, it's their main contribution that they propose a method that overcomes this drawback. Our contribution lies more on thorough and comprehensive investigation and focus on dense tasks rather than achieving SOTA on every matching related tasks). We also outperforms DKM in HPatches, which we believe this indicates some trade-offs. Similar to any other works, we believe that it would not be fair to penalize our work for not including as many as 10 datasets (if we include sparse matching tasks) from different tasks.

For the small improvements in Optical flow datasets, we wish to stress that not only it should not diminish our contribution because we don't outperform them by significantly large margin, but also it should be noted that it is quite common for most (even the very recent papers) optical flow methods to also report similar extent of improvements every year (We respectfully refer the reviewer to paperswithcode. The changes in EPEs are mostly decimal points). The focus should be on how we obtain better results than PDCNet+, as we adopt similar training strategy and exceeding their performance should sufficiently demonstrate the effectiveness of the unified aggregation approach we introduce.

We hope that the merits of our approach is recognized, as we have shown that our approach achieved highly strong performance in 5 different benchmarks (with large margin.). We sincerely hope that the contributions of our work is not ignored for not being able to achieve SOTA in tasks beyond the paper's scope, since MegaDepth, ScanNet and YFCC100M are used for evaluating sparse matching tasks in outdoor and indoor pose estimation, respectively, while our focus is on dense matching.

[A] A simple and efficient approach for adaptive stereo matching. [B] Graftnet: Towards domain generalized stereo matching with a broad-spectrum and task-oriented feature.

We thank the reviewer for the thorough reviews. Our response can be found below :

Whether Table 1 and 2 competitors are trained on the same datasets:

In Table 1, to ensure a fair and consistent comparison, all methods were trained and evaluated on the same datasets. Specifically, for evaluations on SPair-71k, all methods were trained on SPair-71k, and for evaluations on PF-PASCAL and PF-WILLOW, they were trained on PF-PASCAL. This uniform approach guarantees that the results in Table 1 are directly comparable, as each method was subjected to the same training and testing conditions.

For Table 2, we acknowledge that there was some variation in the training strategies employed. All methods, with the exception of GMFlow, were trained either on MegaDepth or a combination of DPED, ADE20K, and Cityscapes (DPED-ADE-Cityscape). Notably, except for DMP, which is trained on DPED-ADE-Cityscape, all other methods were trained on MegaDepth. While this introduces some variation, the majority of methods being trained on MegaDepth still allows for a reasonably fair comparison. However, we recognize and apologize for the potential unfairness in the comparison with GMFlow, which was not trained on MegaDepth.

To address this and to further evaluate the generalizability of our method as suggested by the reviewer, we conducted additional evaluations in the context of optical flow. For these evaluations, we utilized the KITTI and SINTEL datasets. Aligning with the practices of GLU-Net, PDC-Net, and GoCOR, our method was not trained on the Chair and Things datasets but rather on MegaDepth. The results of these additional evaluations are presented below:

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness.

We hope this additional information and the results of our extended evaluations address the concerns raised and provide a more complete understanding of the capabilities of our method.

Optical flow transferability.

We provide quantitative results above.

Pose estimation evaluation

Following reviewer's suggestion, we perform additional evaluation on pose estimation. Specifically, we use YFCC100M and ScanNet. The results are shown below:

The upper table is YFCC and the below is ScanNet.As shown, although it surpasses PDCNet and others, it is apparent that our method is not state-of-the-art. Nevertheless, we wish to highlight that our formulation, which aims at dense matching, should be separated from sparse matching (feature matching) works. A pervasive challenge in dense matching methods is their requirement to extract all matches between views. This challenge often results in many dense matching networks facing difficulties in sparse or semi-sparse methods for geometry estimation, which is shared by many other works. Our framework is also not immune to this limitation. A specific area where our current architecture falls short is in its approach to handling confidence scores. While our cost volumes do encode a confidence score for each tentative match, a recent study like Lightglue has shown the benefits of explicitly estimating these confidence scores directly from the network, particularly for sparse matching tasks. This explicit estimation technique enhances performance in challenging scenarios, such as those involving occlusions or uncertain regions. Through empirical experiments, we've found that our model struggles more in these situations, largely attributed to the absence of a dedicated confidence estimation module. To provide a complete understanding of our framework's capabilities and limitations, we will introduce a new section in our paper dedicated to discussing this limitation.

Moreover, we wish to stress that the true essence and value of our work lie in the comprehensive and thorough investigations we have conducted into the relationships and synergies between feature aggregation and cost aggregation, as appreciated by the reviewer HY22, especially in the context of dense matching tasks. Our research goes beyond merely presenting a new technical design for state-of-the-art performance. Instead, it offers a deeper understanding of the underlying principles and interactions within these aggregation methodologies. This in-depth exploration, we believe, provides a more substantial and lasting contribution to the field.

Whether the proposed method works on minimal overlapping image pairs (MegaDepth Evaluation)

We agree that if the cost score matrix is noisy, it may provide wrong guidance for feature updating. However, as shown in qualitative results on semantic matching datasets in the supplementary material, our method performs quite well given extreme scale differences, and as there are numerous image pairs with extreme geometric scenarios as well as additional challenges like background clutters and intra-class variations, we may assume that our method is quite competitive. Nevertheless, following the reviewer's suggestion, we also try evaluating our method on the MegaDepth dataset and the results are summarized below:

While ours achieves competitive performance, we agree that ours may struggle when image pairs with large geometric deformations are given. We will be sure to add the limitations and future directions of our method in the manuscript.

Failure cases

We will visualize failure cases for both geometric and semantic matching and include the limitation discussed above as well. We appreciate the reviewer's comment.

We thank the reviewer for the reviews. Our response is shown below :

An analysis of the local discriminative ability of features.

One of the goals of our proposed aggregation is also to synthesize and incorporate broader contextual information. This often involves balancing local feature details with more global, semantic information. While the process enhances the overall semantic understanding of the image, it can sometimes appear to diminish the acuity of local features. This is not necessarily a flaw but rather a trade-off, aiming for a more holistic understanding of the image data. As demonstrated in Fig. 4 of the supplementary material, top row, where image pairs have extreme geometric deformation and repetitive patterns, our framework can warp all pixels to the corresponding locations unlike others. This shows our framework does not necessarily experience such damage.

Resolution of cost volumes at each level

We apologize for any confusion caused by our initial explanations and appreciate the opportunity to clarify. In our architecture, as depicted in the upper branch of Figure 5, the cost volumes at each level are indeed fixed to the coarsest resolution. This specification applies specifically to the region labeled as "Cost Volume," where Conv4D operations are performed over the output cost volumes resulting from both self- and cross-attention mechanisms.

Additionally, within the aggregation modules illustrated in Figure 4, the dimensions of the cost volumes are maintained to be consistent with those of the feature maps. This design choice ensures that the cost volumes and feature maps are compatible for effective integration within these modules.

The rationale behind fixing the cost volume dimensions to the coarsest resolution in certain parts of the network, particularly for the residual connections, is driven by memory efficiency considerations. Since cost volumes inherently require more memory to store compared to feature maps, reducing their dimensions in specific regions of the network helps in managing the overall memory usage more effectively. This approach strikes a balance between maintaining the integrity and effectiveness of the cost volume information while optimizing the memory demands of the network.

We hope this clarification addresses the concerns and provides a clearer understanding of our architectural choices and their underlying motivations. If there are further questions or aspects that require additional explanation, we are more than willing to provide the necessary information.

Qualitative comparisons of Table 4.

In response to the suggestion of including qualitative comparisons, we recognize the importance of such visualizations in complementing our quantitative results, as presented in Table 4. However, the qualitative results as the performance improves would primarily manifest as minor adjustments in the accuracy of warped keypoints or pixels. These changes, while significant in a quantitative sense, may not be visibly distinguishable. However, as an alternative, we present PCA visualization as in Fig.9 and Fig.10 in the supplementary material, where we provide more examples of how our method works. These visualizations are designed to provide an intuitive understanding of how our model achieves its performance gains. They offer a clear illustration of the internal workings and improvements brought about by our framework. For example, if we look at (I-IV), they semantically match with the change from raw features or cost volume to those that are aggregated, which are visualized in Fig.9 and Fig.10 in the supplementary material, while the rest (V-VI) do not provide much intuition but rather are similar to the already presented visualizations with being higher resolution (meaning more fine details).

Nevertheless, to provide additional information for better understanding, we revised the supplementary material and included Fig.11, where we show PCA visualization comparisons of the each variant introduced in the component ablation table. The caption is colored red to draw attention to the change. As the comparison shows, while there exist apparent differences between (I) and (IV), we acknowledge that the differences are not visually apparent as we add the components from (II) to (IV). This is because the resolutions of features are defined in the coarsest level, which makes it little bit hard for us to interpret the visualization. Nevertheless, we believe that the differences present in the visualizations contributed to the improvements and that we provide more intuitive visualizations in Fig. 9 and Fig. 10.

It would be good to include the number of learnable parameters for Table 3 and Table 4.

We appreciate the reviewer's suggestion. A small issue is that we intentionally adjusted all variants' number of learnable parameters to be similar for the experiments in Table 3 and 4 as mentioned in section 5.5, but we agree that it would be more informative to include the number of learnable parameters for each module, i.e., integrative self-attention and cross-attention, in our architecture. The table below shows the comparison.

Note that we also include similar ablation study in Table 2 of the supplementary material, which shows the comparison by varying depth of the transformer layers. We will include this in the supplementary material, as a separate table.

We highly appreciate for the reviewer's constructive comments. Our response is shown below :

What would be the key component to benefit the tasks? And would it also benefit optical flow?

The key technical design to benefit the task is the capturing of synergy between features, cost volumes, and their aggregations. While our approach to achieve this is indeed by incorporating self-attention and cross-attention mechanisms for effective aggregation, it's important to note that these are part of a suite of carefully considered technical designs. These designs are meticulously crafted to meet three key objectives: minimizing computational resources (achieved through Conv4D operations and maintaining a fixed resolution of the cost volume), maximizing performance (enhanced by residual connections among aggregated costs for training stability), and fostering synergy between feature aggregation and cost aggregation (via our innovative attention designs).

However, we contend that these technical aspects, while significant, constitute a smaller part of our overall contribution. The true essence and value of our work lie in the comprehensive and thorough investigations we have conducted into the relationships and synergies between feature aggregation and cost aggregation, especially in the context of dense matching tasks. Our research goes beyond merely presenting a new technical design for state-of-the-art performance. Instead, it offers a deeper understanding of the underlying principles and interactions within these aggregation methodologies, as appreciated by the reviewer HY22. This in-depth exploration, we believe, provides a more substantial and lasting contribution to the field.

This general approach, as the reviewer kindly suggested to show, is not constrained to geometric and semantic and it can also be evaluated on other tasks, e.g., optical flow, to further demonstrate the generalizability of our framework. The results are shown below:

From the results, while ours outperforms others in KITTI-2015, it is observed that ours achieves lower AEPE than existing flow methods. However, it should be noted that when compared to PDC-Net and PDC-Net+, ours consistently performs better, which validates the effectiveness of our approach in optical flow task as well. We acknowledge that newer, more sophisticated technical designs may emerge in the future, but we wish to emphasize that while numerous architectural designs, e.g., convex upsampling, uncertainty module or GOCor-like correlation optimization scheme, could potentially be included and enhance the synergy between feature aggregation and cost aggregation, our framework works as a stepping stone that stands out for its simplicity and effectiveness.

We highly appreciate the reviewer's constructive comment, and we will integrate the results into the final manuscript.

Limitations

A pervasive challenge in dense matching methods is their requirement to extract all matches between views. This challenge often results in many dense matching networks facing difficulties in sparse or semi-sparse methods for geometry estimation. Our framework is not immune to this limitation. A specific area where our current architecture falls short is in its approach to handling confidence scores. While our cost volumes do encode a confidence score for each tentative match, a recent study like Lightglue has shown the benefits of explicitly estimating these confidence scores directly from the network, particularly for sparse matching tasks. This explicit estimation technique enhances performance in challenging scenarios, such as those involving occlusions or uncertain regions. Through empirical experiments, we've found that our model can struggle in these situations, largely due to the absence of a dedicated confidence estimation module. To provide a complete understanding of our framework's capabilities and limitations, we will introduce a new section into our paper dedicated to discussing this limitation.

Thanks for the reply and we are glad to find that most of the concerns are resolved!

Before answering your question, we first would like to ask if the reviewer wants to see PCA visualizations of feature maps (H x W x 3) for geometric matching or the flow map (H x W x 2) itself.

For the question, we can easily answer this, since it's because of the way visualize them by PCA, and the distinctive sparse blobs are the 3 principal components that represent the most distinctive ones. Other components represent the other regions well, but one thing is that they just are not very intuitive. We have a few PCA visualizations for geometric matching datasets ready, and if this is what the reviewer wants to see, we can quickly revise the supplementary material. Note that these were also not very intuitive, visualizing some horizontal or vertical lines (we hypothesize that they indicate flows).

Please let us know and we will get back to you as soon as possible!