Generation and Comprehension Hand-in-Hand: Vision-guided Expression Diffusion for Boosting Referring Expression Generation and Comprehension

We appreciate your insightful comments and have addressed them as follows.

[Q1] Some recent representative works, such as CCL [Wang et al. AAAI'24], are not compared. A comparison between our method and CCL, presented in the table below, demonstrates the superiority of our method. While CCL generates expressions based solely on predicted objects, our method incorporates a holistic perspective via the proposed VTC (vision-text condition) module, considering both the target object and the surrounding image context. This enables our method to generate more accurate and unambiguous expressions, as illustrated by the example in Figure 6 of the supplemental materials. The comparison with CCL has been included in Table 1 of the revised paper.

[Q2] Failure cases are lacking; diffusion data generation is usually unstable, and the authors need to analyze this point. We would like to clarify that failure cases are provided in the original paper. Some inaccurate expressions generated by our method are shown in the third row of Figure 3. An analysis of these failures and a feasible solution for augmentation are given in Lines 508-515.

We acknowledge the potential instability of diffusion-based generation in the original paper. To address this, we propose a strategy for selecting more stable expressions for augmentation, in Lines 469-476 of the original paper, and evaluate it in Table 6. By using this strategy, the selected augmented data consistently and significantly improve six state-of-the-art REC methods across multiple datasets, as shown in Table 2.

[Q3] Statistics on model parameters, training time, and inference time are required. The established model comprises 1.2 billion parameters. Training on the RefCOCO dataset (120,624 image-expression pairs) with classifier-free guidance takes approximately 86 hours on 4 NVIDIA V100 GPUs. The inference time on a single NVIDIA V100 GPU is 8.74 seconds per image with DDPM and 0.93 seconds per image with DDIM. These details have been included in Lines 357-360 of the revised paper.

[Q4] Why does performance on certain metrics improve after losing CFG in Table 1?. In CFG, the guidance weight significantly influences the diversity of expressions generated by our method. A higher guidance scale enhances the accuracy of the generated expressions but reduces their diversity, vice versa. In this paper, we set the guidance weight to 0.2 to enhance the diversity of expressions generated by our method, accepting a slight decrease in accuracy as a trade-off.

[Q5] What is the number of samples in each augmented dataset? This needs to be reported. Thank you for the comment. We have described how to select augmented data and how to determine the number of augmented data in Section 3.6 of the original paper. During augmentation, all training samples from the REC dataset were input into our method to generate expressions. For example, the RefCOCO training set contains 120,624 samples, resulting in the generation of an equal number of image-expression pairs. Among these, 30% of the pairs with the highest scores (i.e., 36,187 pairs) were selected for augmentation. Since the size of the training sets varies across REC datasets, the number of generated samples also varies accordingly.

We have made every effort to address your concerns and sincerely hope our responses meet your expectations. We would greatly appreciate your feedback on our replies to help us further enhance the manuscript.

We appreciate your insightful comments and have addressed them as follows.

[Q1] No obvious disadvantages were seen. Including a section on potential limitations would provide a more balanced perspective. Thank you for appreciating this paper. Two limitations of our method are pointed out in the paper. The first limitation is that our method occasionally generates inaccurate expressions. Some inaccurate examples are shown in the third row of Figure 3 of the paper. An analysis of these failures and a feasible solution for augmentation are given in Lines 508-515. Another limitation, shown in Table 5, is that augmenting too many expressions generated by our method results in performance drops. The corresponding discussion is given in Lines 461-468.

We appreciate your insightful comments and have addressed them as follows.

[Q1] The generated expressions in the third row of Figure 3 are inaccurate. While the current method enhances diversity, it still includes some erroneous responses. How do you ensure the quality of the generated responses? The third row of Figure 3 does indeed display some inaccurate expressions generated by our method. We acknowledge that our method may occasionally produce incorrect responses, as illustrated in the third row of Figure 3. However, in most cases, it successfully generates diverse and accurate expressions, as demonstrated in the top two rows of the same figure.

As detailed in Lines 303-315 of the main paper, we present a strategy to estimate the quality of generated expressions. Specifically, we leverage Meteor scores to assess images and their corresponding generated expressions, selecting those with high scores as augmented data. As shown in Table 2, the selected augmented data by our method consistently and significantly improves six state-of-the-art REC methods across five datasets.

[Q2] The ViT backbone of CLIP is considered as a unified vision encoder in MLLM. Could this architecture produce different patterns and further improve performance? We recognize that the ViT backbone of CLIP is a powerful vision encoder for multimodal LLMs due to its ability to effectively align visual features with text features. As reported in Table 1 of the main paper, the competing method MiniGPT-v2, which utilizes ViT, achieves promising results. Thus, we believe that incorporating ViT or other CLIP vision encoders can be beneficial, and we will evaluate our method with different CLIP vision encoders in the paper.

[Q3] The definition of CFG is missing in Table 1. We appreciate you pointing out this oversight. CFG is an acronym for classifier-free guidance. We have included the definitation in Lines 325-326 of the revised paper.

[Q4] This paper lacks an in-depth analysis of the reasons behind the observed patterns in the results. The first two rows of Figure 3 illustrate the effectiveness of our method in generating accurate and diverse expressions, where the noun phrases in the original expressions are usually replaced with synonyms while the sentences are sometimes restructured. Among the observed patterns, the faithful image-text relationships can be attributed to the proposed VTC (vision-text condition) module, which ensures precise alignment between visual and textual tokens. Meanwhile, the transformer decoder and diffusion model contribute to high diversity through synonym replacement and sentence restructuring. The analysis has been included in Lines 481-506 of the revised paper.

[Q5] The writing is somewhat verbose. For instance, in Subsection 3.6, the second sentence ... We appreciate your feedback regarding the verbosity of the writing. We have improved the sentences in Subsection 3.6 of the revised paper and will work to improve the overall conciseness of the paper.

[Q6] Some equations could be improved; for example, Equations 9 and 10 differ by only one symbol. Thank you for pointing out this issue. In the revised paper, Eq. 9 is kept, while Eq. 10 is replaced by simply stating its difference from Eq. 9 in Line 294.

[Q7] There are a few typos, such as a missing period on line 82 and an incorrect number on line 360. Thank you. We have corrected the typos in the revised paper and will proofread the entire paper.

We appreciate your insightful comments and have addressed them as follows.

[Q1] The proposed method is composed of a straightforward combination of existing methods. We acknowledge that some existing techniques have been adapted and used in our proposed method. However, to the best of our knowledge, this work is the first to apply conditional text diffusion models to the REG task. Moreover, it seamlessly integrates the complementary tasks of REG and REC. Additionally, we have developed effective components specifically for REG. For example, the proposed Vision-Text Condition (VTC) module aligns our diffusion-based approach with the REG task, significantly improving its performance. The token selection strategy within the VTC module is designed to mitigate the negative impact of abundant and irrelevant image tokens, a common challenge in REG.

[Q2] The author forgot to explain what CFG stands for. We appreciate you pointing out this oversight. CFG is an acronym for classifier-free guidance. We have included the definitation in Lines 325-326 of the revised paper.

[Q3] It is not clear how the combination of a transformer-based decoder and diffusion model outperforms only a transformer-based decoder. We appreciate this comment and conducted an experiment, where the diffusion model is removed from our method VIE-DM. In this simplified variant, the embedded image and target object are directly fed into the token selection strategy, and the outputs are passed to the transformer decoder. As shown in the table below, performance decreases significantly without the diffusion model. This underscores the importance of our proposed vision-guided diffusion model for generating high-quality expressions suitable for REC dataset augmentation. In the revised paper, this experiment has been included in Table 9 and Lines 801-809 of supplementary materials.

[Q4] It is misleading to make such a claim of diversity based on Table 1. ... The argument regarding Table 3 is more convincing. Good catch. In the revised paper, we have removed the claim of diversity based on Table 1 and now claim the advantages of high diversity based on the results reported in Table 3.

[Q5] Table 5 shows that augmenting the REC data using VIE-DM only leads to a limited improvement. We would like to clarify that what Table 5 presents is the performance of an existing REC method QRNet working with different amounts of augmented data generated by VIE-DM, instead of with or without augmented data. The effect of augmenting the REC data using VIE-DM is reported in Table 2, where our method VIE-DM consistently and substantially improves six powerful REC methods across five benchmark datasets.

[Q6] Meanwhile, it is not clear how much accuracy is improved when the data set is augmented using methods other than VIE-DM. Table 8 in the appendix of the original paper presents a comparison of our VIE-DM method with four existing REG methods in terms of their ability to augment REC datasets. The results demonstrate that VIE-DM more effectively enhances the REC datasets, enabling existing REC methods to achieve better performance.

Thank you very much for raising the rating of our paper!

We agree that while REG-based expression augmentation can enhance REC performance, the improvement tends to plateau once the diversity of the augmented image-expression pairs reaches a sufficient level (e.g., an increase of 30% as shown in Table 5). We have clarified this point further in the revised manuscript.