Referencing Where to Focus: Improving Visual Grounding with Referential Query

Q1:It is better to provide the experiment results without using auxiliary loss, which can further observe the influence of auxiliary loss on the referential query.

We conduct the ablation study with auxiliary loss on RefCOCOg, and the results demonstrate the effectiveness of auxiliary loss. By employing auxiliary loss, the reference query can capture the target-related visual contexts more effectively.

Thank you, we will add this experiment to the ablation studies.

Q2: Is it feasible to directly utilize the referential query without subsequent decoding operations? I am interested in the accuracy of the referential query.

The referential query is designed to provide prior information to the decoder. Since the channel dimension in the QA module is lower, the reference query may not accurately predict the coordinates of the targets. We conduct the experiment on RefCOCOg, and the results are illustrated below.

Thank you, we will add this explanation to the method.

Q3: Typo: On line 72, "we propose a query adaption module, RefFormer...". It appears that there might be an error in this sentence.

Thank you. We will rectify "RefFormer" to "QA".

Q1: This idea seems straightforward but lacks some innovation. The method mentioned in the paper has been applied in other fields, such as R2-tuning.

Thank you for your question. We analyze the differences between our approach and R2-tuning [a] from two aspects: motivation and implementation:

[Motivation]: [a] designs an image-to-video transfer framework that applies CLIP to video temporal grounding tasks without fine-tuning. In contrast, our work focuses on improving learnable queries in DETR-like structures for visual grounding tasks. Specifically, [a] designs a parameter-tuning strategy to model spatial-temporal information from coarse to fine using CLIP, which can be served as an adapter. Instead, our work aims to generate prior information for the decoder to improve the learnable query by integrating the QA module into the CLIP layers.

[Implementation]: In our work, we focus on improving learnable queries rather than merely tuning gradient flows. The comparison is shown as follows:

1. The feature interaction in CLIP layers. It primarily encompasses three key differences: 1) the generated features (frame-level spatial-temporal features vs. referential queries), 2) vision-language interaction (unidirectional vs. bidirectional), and 3) gradient flow tunning (visual side vs. visual and language sides). Specifically as follows:

   • [a] introduces the $R^2$ blocks to interact the patch-level representations with language representations for adaptively pool spatial features, and then combine the pooled features with the [CLS] token to extract the frame-level spatial-temporal features. In our work, we incorporate the learnable referential queries to interact with the language and visual features for capturing target-related visual contexts.

   • The interaction in [a] is unidirectional, focusing solely on extracting query (language)-modulated spatial features. Conversely, the interaction in our work is bidirectional, not only highlighting language-related context within visual features but also integrating context-aware visual features into language features.

   • The gradient flows tuning in [a] that forward in CLIP are solely within the vision encoder and are specific to the [CLS] token, while in our method, they are adjusted in both the vision and language encoders and apply to patch features.

2. Decoding. [a] apply the traditional convolution layers to spatial-temporal features to predict the temporal boundary, while our work adopts the DETR-like structure and employs reference queries to provide the prior information for the decoder.

3. Multi-level features. We introduce language-guided multi-level fusion to aggregate the image features from the different layers rather than applying 1D convolutions to generate the temporal feature pyramid in [a] for subsequent decoding.

To further validate the effectiveness of our approach, we integrate the method from [a] into our framework. The results, presented below, demonstrate that our method achieves superior performance.

Additionally, we would like to highlight that [a] is a concurrent work that has been accepted by ECCV. We will reference it and discuss it in the related work.

[a] Liu Y, He J, Li W, et al. $R^ 2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding[J]. arXiv preprint arXiv:2404.00801, 2024.

Q2: Could you provide the number of trainable parameters and the inference speed?

We provide the number of trainable parameters and the inference speed below and compare them with other open-source methods. When compared to Transvg and RefTR, which also use the DETR-like structure, our approach demonstrates superior performance in both trainable parameters and inference speed.

Thank you, we will add this experiment to the experiments.

Thank you for your feedback. Our approach fundamentally differs from [a]. As previously mentioned, our study concentrates on improving learnable queries within DETR-like architectures, as opposed to solely concentrating on parameter adjustments as in [a]. Furthermore, experiments have demonstrated significant improvements resulting from our approach.

We would like to emphasize once again our motivation:

1. We focus on improving the learning process of the learnable queries, different from the previous work that emphasizes the design of sophisticated multi-modal decoders.
2. We propose a query adaption module that not only adaptively captures the target-related context, providing valuable referential knowledge for the decoder, but can also serve as the adapter.
3. By strategically inserting the QA module into different layers of CLIP, the query adaptively learns target-related information from multi-level image feature maps, and iteratively refines the acquired information layer by layer.

Additionally, we conduct extensive experiments to validate the effectiveness of our method and provide the visualization results to demonstrate the reasonability of our proposed referential query.

We sincerely hope you will reconsider our paper and appreciate your valuable suggestions for improving this work. If there are any other questions or areas you'd like to discuss, we welcome further conversation.

Dear Reviewer DDuF,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion. Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Regards, Authors

Q1: Line 72 should correct "RefFormer" to "QA".

Thank you. We will correct the typos in our paper.

Q2: A more detailed description is needed for the text side in the QA module, such as how the interaction with visual features is initiated.

In the CAMF block, we take the language features as the query, and interact them with the image features using cross-attention. By doing so, we can incorporate rich visual context into language features to better indicate the target object. Subsequently, in the TR block, we employ self-attention to enhance the context-aware language features produced above to refine the expression condition.

Thank you, we will add this description in Sec 4.1.

Q3: Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?

As outlined in line 189, we feed the referential query into $\phi_q(\cdot)$ to adjust its significance. When the referential query is inaccurate (i.e., its significance approaches zero), the query in the decoder degenerates to the vanilla query.

Thank you, we will add this explanation in line 189.

Q4: Specify the dimension along which the concatenation operation in Section 4.1 is performed.

Thank you for your suggestion. We will further improve the description details in our paper. In Eq.7,9,11,12, the concatenation is along with the patch dimension.

Q5: Correct "[r_t, F^i_t]" to "[r_t; F^i_t]" in Equation 9.

Thank you. We will correct the typos in our paper.

Q6: Provide experimental details regarding the channel dimension of the QA module in the down-projection process.

Thank you for your suggestion. In our experiments, we set the channel dimension of the down-projection to 128. Furthermore, we provide an ablation study on the channel dimension of the down-projection below. Increasing the dimension improves performance, but to strike a balance between performance and computational cost, we set the channel dimension to 128.

Thank you, we will add this experiment to the ablation studies.

Dear Reviewer RwV9,

I am glad for the recognition of our work. Please feel free to raise any further questions, and we are more than happy to continue the discussion with you. Thanks again for your great efforts and constructive advice in reviewing this paper!

Regards, Authors

Q1: How does the paper select the number of inserted QA modules? How does the paper select the indices of inserted layers? Why do more layers or lower layers hurt the performance?

We provide more experiments on layer selection on RefCOCOg below. We categorize [1-4], [5-8], and [9-12] as low-level, mid-level, and high-level layers, respectively. As shown in the table below, we observe that introducing more layers can improve performance. However, with continued layer addition, the performance gains become less significant. Therefore, to strike a balance between performance and computational cost, we opt for [4,6,8,10,12]. Compared to IDs 7, 9, and 10, line 8 leads to a slight performance decline. This could be attributed to the fact that shallower layers focus on the local details and convey less semantic information, which may introduce noise. Similarly, in object detection, RPN-based models [a,b] do not use C1 feature maps for the same reason. Additionally, we explore other combinations of layers, but the performance changes show low sensitivity.

Thank you, we will improve Table 5 and the corresponding experimental analysis.

[a] Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149.

[b] He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.

Q2: In condition aggregation and multi-modal fusion (CAMF) module, it seems that the query interacts more with the visual representations (i.e., the upper part in Figure 3), while the alternative that mainly integrates query with textual representations seems feasible (i.e., place the query to the lower part in Figure 3). The motivation and advantage of design are missing.

We would like to clarify that reference queries aim to capture the target-related visual context. Initially, they interact with expressions to aggregate text conditions (in CAMF). Subsequently, they interact with visual features based on text conditions to capture and refine visual contexts about the target object (in TR). If we place them below, we will not be able to capture the target-related visual features based on the conditional information. Additionally, we provide the performance comparison on RefCOCOg as below. The results further demonstrate the effectiveness of the method we designed.

Thank you, we will add this experiment to the ablation studies.

Q3: The backbone and training data information (especially the version of Reformer) is not provided in Tables 2 and 3.

For a fair comparison, we utilize the ViT-Base 32 and train on a single dataset (Flickr30K Entities and ReferItGame, respectively) to compare with other methods. Consistent with prior approaches, we perform ablation studies (Table 3-6) using ViT-Base 32 and train on a single dataset (RefCOCOg).

Thank you, we will further improve the experimental details in our paper.

Q4: Does QA support different model architectures?

We apply our method to single-modal encoders, i.e., Swin-base + Bert, and conduct experiments on RefCOCOg. The results, as presented below, demonstrate that our method is also compatible with single-modal encoders.

Thank you, we will add this experiment to the experiments.