Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

Thank you for your thorough review and for acknowledging the effectiveness of our experimental results. Below, I will provide further clarification regarding the weaknesses you highlighted.

The definition in DEEP PROMPTING is confused. What is the definition of $E_{i}$

We apologize for any confusion caused by the lack of clarity in our initial description. Thank you for pointing it out. Deep prompting refers to the use of prompts in the deeper layers of the modality encoder. Take GLIP as an example: both the vision and language encoders consist of 12 transformer layers. The image is first divided into multiple patches, which are then embedded into $E_{0}^{v}$. The text is tokenized and embedded to produce $E_{0}^{l}$. These embeddings are then input into the modality encoder. For the i-th transformer layer, the input is $E_{i}$, and the output is $E_{i+1}$. In deep prompting, prompts are concatenated with $E_{i}$ in the first $D$ transformer layers, serving as the new input to each transformer layer. We have added detailed explanations in Section 3.1 of the updated version of the paper.

What mechanism do you use to the D latent features of generated prompt? It is not clear

Conditioned on an image with shape [3, 224, 224] and its corresponding caption, the prompt generator randomly samples Gaussian noise with shape [4, 28, 28]. This Gaussian noise is then denoised through multiple steps to obtain latent prompts with shape [4, 28, 28], which correspond to $\widetilde{z}_i$ in the Figure 3. The latent prompts are in dense form. Considering that the prompt generator and GLIP may operate in different data distributions, we first use the VAE decoder in Stage 2 to reconstruct the latent prompts, generating prompts (generated prompts in Figure 3) of the same size as the image. These generated prompts form a saliency map, which informs the foundation model about which parts of the image require more attention. Subsequently, we design a vision adapter and a language adapter to map the generated prompts, i.e., the saliency map, into different modality encoders, as shown in Figure 2. In other words, we convert the [224, 224] sized generated prompt into prompts with shapes [4, 96] and [4, 768]. These prompts are then concatenated with $E_i$ and global prompts, and fed into the transformer layer. We have added detailed explanations in Section 4.2 and Section 4.3 of the updated version of the paper.

In subsection 3.1, this study is linked to referring expression comprehension. Nonetheless, there is a lack of information on how the proposed method generates a position to accurately locate the target object described in the expression.

In the GLIP model, visual and textual features are first extracted from the image and text using the vision and language encoders, respectively. The visual features of the image are then input into the Region Proposal Network (RPN). RPN generates multiple anchor boxes by sliding a window over the image feature map, where each anchor box predicts a foreground/background classification and bounding box regression values to adjust its coordinates. Next, RPN applies Non-Maximum Suppression (NMS) to remove redundant and low-confidence proposals, ultimately outputting a set of high-quality candidate regions. We have added detailed explanations in Section 3.1 of the updated version of the paper.

The advantage of using the generated prompt is not clear. Why can we replace the generated prompts with learnable prompts

From a broader perspective, generated prompts have significant advantages over directly learnable prompts. Generated prompts can be designed based on prior knowledge, making them more controllable and interpretable. For example, segmentation information, edge features, or other task-specific information can be used as prompts, thereby clearly defining the semantic meaning they carry. In this paper, we use masks as prompts and employ a diffusion model to generate task-guided prompt information. In contrast, directly learnable prompts are optimized end-to-end, resembling a "black box" process, where it is unclear what knowledge has been learned. This lack of interpretability is unacceptable for scenarios where prompts must have explicit meanings. Similar to how large language models require precise prompts to produce desired outputs, prompts themselves should carry clear and interpretable semantics.

From a practical application perspective, this approach allows us to design more customized prompts. As indicated by the result in Table 1, our proposed method achieves relatively strong performance.

Dear Reviewer A4tj,

Thank you once again for dedicating your time and effort to reviewing our work. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns. With the rebuttal phase nearing its conclusion, we look forward to your feedback and are available to provide any further clarification you might need.

  As the deadline of discussion period is today, please respond to the author reply to see if your concerns are well-addressed at your earliest convenience. Thank you very much.

Thank you for recognizing the innovation of our method and acknowledging the effectiveness of our experiments. Regarding the weaknesses you mentioned, I will provide a detailed response below.

The authors do not report on other generalization benchmarks such as the ones reported by prompt learning techniques like CoOP and Maple, which the authors compare with. These are representative tasks in generalization to novel classes, new target datasets and unseen domain shifts. Why were these benchmarks not considered, since methods reporting on these benchmarks are considered in the evaluation?

Thank you for pointing out the lack of generalization benchmarks in our experiments. In our method, we focused on addressing complex downstream tasks but overlooked comparisons on more commonly used benchmarks. To address this, we have supplemented experiments on cross-dataset generalization and cross-domain generalization, and included comparisons with the methods highlighted by the reviewer. The results have been added to the updated version in Section 5.4 Ablation Study under Cross-Domain Generalization and Cross-Dataset Generalization. The specific results are shown in the table below. The experimental results show that, while our method does not achieve state-of-the-art performance, it still demonstrates considerable generalization ability.

Comparison with SOTA Methods on Cross-Domain Evaluation

Comparison with SOTA Methods on Cross-Dataset Evaluation Part 1

Comparison with SOTA Methods on Cross-Dataset Evaluation Part 1

It is worth noting that PromptAlign [3] represents a test-time adaptation approach, which operates in a different application scenario compared to the other methods. Therefore, we did not include it in direct comparisons but discussed it in the related work section.

The authors are missing comparisons with multiple recent works related to prompt learning for generalization such as [1,2,3,4,5,6]. Why were these methods not considered during evaluation and comparison?

Thank you for providing the suggested methods. In the generalization ablation experiments, we have supplemented comparisons with these methods. Regarding the comparison on the RES task, we found that these methods cannot be effectively transferred, and the reasons are as follows:

The GLIP model uses Swin as the vision encoder and BERT as the language encoder. Its weights have been fine-tuned for downstream tasks, which makes the GLIP encoder's output incompatible with computing logits in the same way as CLIP. Key components in these methods, such as the distillation loss in PromptKD, $L_{\text{SCL-logits}}$ in PromptSRC, $L_{\text{entropy}}$ in PromptAlign, $L_{\text{ce}}$ in CoPrompt, and the key-value caching strategies in CPL and CoCoLe, cannot be seamlessly applied to the GLIP model. For GLIP or other complex downstream task models, designing different prompt strategies is more appropriate.

Considering the lack of comparisons with the latest methods, we have supplemented our experiments with a recently designed prompt tuning method VFPT (NeurIPS 2024)[1] and several adapter tuning methods, Tip-Adapter (ECCV 2022)[2], Meta-adapter (NeurIPS 2023)[3], and MMA (CVPR 2024)[4]. The results demonstrate that our proposed method still shows certain performance advantages. We have added these experiments and conclusions to Table 1 and Section 5.2 in the updated version.

[1] Zeng R, Han C, Wang Q, et al. Visual Fourier Prompt Tuning[J]. arXiv preprint arXiv:2411.01327, 2024.

[2] Zhang R, Zhang W, Fang R, et al. Tip-adapter: Training-free adaption of clip for few-shot classification[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 493-510.

[3] Song L, Xue R, Wang H, et al. Meta-adapter: An online few-shot learner for vision-language model[J]. Advances in Neural Information Processing Systems, 2023, 36: 55361-55374.

[4] Yang L, Zhang R Y, Wang Y, et al. MMA: Multi-Modal Adapter for Vision-Language Models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23826-23837.

Dear Reviewer LEnk,

Thank you once again for dedicating your time and effort to reviewing our work. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns. With the rebuttal phase nearing its conclusion, we look forward to your feedback and are available to provide any further clarification you might need.

Thank you for your response. Regarding your concerns about the generalization ability of our method, I would like to provide further clarification.

• First, our method is less suited for image classification tasks. This is because Diff-Prompt relies on paired images and captions to generate prompts. However, in image classification tasks, each class name is encoded, and a prompt is generated for every image paired with each class name, resulting in a quadratic increase in computational cost. For example, in ImageNet, which has 1000 classes, a single image would need to generate prompts with 1000 textual features. To address this in our experiments, we averaged the 1000 textual features, which inevitably diluted the representation of the text.

• Second, our method demonstrates good generalization ability, though it has not yet reached the state-of-the-art (SOTA) level. This is because our method is not specifically designed for generalization; instead, it focuses on solving complex, multimodal, fine-grained downstream tasks.

Currently, methods with strong generalization capabilities, such as CoPrompt and CoCoLe, achieve this by designing regularization terms. CPL and CoCoLe, for example, enhance generalization through the use of caches. Therefore, achieving strong generalization often requires specific tricks. In contrast, our method is centered around prompt design, with the primary purpose of guiding the model to focus on specific parts of the image based on textual input. This approach provides significant advantages for visual understanding tasks.

Our prompts can thus be considered specifically designed for complex visual understanding tasks, which cover a wide range of applications. From the results, Diff-Prompt demonstrates superior accuracy across various datasets compared to other prompt-based methods.

• Furthermore, Diff-Prompt can be seamlessly integrated with regularization or cache-based techniques to achieve stronger generalization capabilities. Notably, these regularization terms and cache-based methods typically operate on the outputs of the encoder, whereas Diff-Prompt provides guidance during the encoding process itself. This ensures that our method does not affect the encoder’s outputs, maintaining their integrity while enhancing the performance.

• Finally, these methods are evaluated using a 16-shot setting and trained for a small number of epochs (set to 5 in our experiments) on ImageNet. This indicates that the improvement in accuracy from generalization experiments is limited by the model's fitting ability on the ImageNet dataset. Complex network structures typically require more extensive training. To further investigate the model's generalization capability, it is more reasonable to evaluate it using zero-shot methods. As shown in Appendix D, we explored the zero-shot capability of models trained on the RefCoCo dataset and conducted zero-shot experiments on 10 unseen object detection tasks. While the GLIP model itself has strong generalization capabilities, the experimental results demonstrate that Diff-Prompt enhances the generalization ability of GLIP-T(A) to some extent.

  As the deadline of discussion period is today, please respond to the author reply to see if your further concerns are well-addressed at your earliest convenience. Thank you very much.

The introduction of diffusion slows inference speed by twofold, which is a significant cost. Could this be optimized?

The optimization can be considered from the following aspects:

• Optimizing the Diffusion Model: The optimization direction for the diffusion model aligns with the general advancements in diffusion models. This includes improving sampling strategies and distillation strategies. Recent studies[1] have demonstrated that diffusion models can generate results in a single step. By leveraging such approaches, we could significantly accelerate the inference speed of the diffusion model.
• Alternative Prompt Generators: From the perspective of the prompt generator, we could explore alternative models that generate results in a single step, such as GANs or VAEs. While the generated quality may be slightly inferior, it would still suffice for prompt generation. Additionally, traditional machine learning methods could be employed to extract high-frequency or edge information from images, which can serve as the base prompts.

In Tables 1 and 2, the improvement in the UB metric compared to MaPLe is minimal, despite MaPLe being a much simpler method. Why?

The UB metric reflects the potential of the model to identify the correct target across all predictions, and as such, the room for further improvement in UB is limited for both MaPLe and Diff-Prompt. This is because UB largely represents the upper limit of the foundation model's capabilities. Prompt learning methods are designed to maximize the foundation model's inherent performance but cannot surpass its intrinsic upper limit. Compared to MaPLe, Diff-Prompt achieves an improvement of nearly 3% in both R@1 and R@5, which demonstrates the superiority of our approach.

The effectiveness of using diffusion in this paper warrants deeper analysis. Since the core contribution of the method itself isn’t very substantial, the primary value lies in using diffusion to generate prompts. Exploring the underlying mechanism further could provide more profound insights.

Firstly, most existing works design automatically learned prompts, leaving it unclear what the model ultimately learns. We believe that our primary contribution is the design of a visual prompt generation method, which differs from traditional automated learning strategies by being more controllable and meaningful—a direction that has been largely unexplored in previous works.

Now, let us explain why we chose to use diffusion models for prompt generation:

1. Rich Information Representation: Extensive cutting-edge research has demonstrated that diffusion models can generate rich and detailed information. This means that, for models of the same size, diffusion models can generate more information. Moreover, while producing the same amount of information, diffusion models maintain a minimal model size, with the trade-off being computational complexity. Given our focus on semantic richness, we opts for diffusion models.
2. Key Benefits of Diffusion-Generated Prompts: Diffusion models play a critical role in multimodal information fusion. For the vision encoder, the generated prompts indicate which visual embeddings to focus on. For the language encoder, the generated prompts capture semantic and contextual information, effectively guiding the model's attention.
3. Alignment of Diffusion and Encoder Processes: In our experiments, we align the denoising process of the diffusion model with the encoding process of the encoder. The results demonstrate the effectiveness of this approach. The underlying principle is that the diffusion denoising process can be viewed as a progressive information fusion process, while the encoder encodes sparse data into dense representations. This hierarchical alignment enables the encoder to better perceive the modality interaction process, thereby enhancing overall model performance.

[1] Luo W, Huang Z, Geng Z, et al. One-step diffusion distillation through score implicit matching[J]. arXiv preprint arXiv:2410.16794, 2024.

Thank you for your insightful feedback on our paper. We sincerely appreciate your recognition of the motivation behind our approach, as well as your acknowledgment of our experimental results and the value of our work for further exploration. We are eager to address your concerns and questions and look forward to providing clarifications where needed.

Generalizability needs further verification: Why was testing only done on GLIP? How would it perform with other models? Why was the evaluation limited to the referring expression comprehension task? What about tasks like RES and PNG [1] for segmentation? Would it also be effective for other more complex tasks? If it only works with this model and task, then the contribution is quite limited, and the paper's title should be adjusted to specify that it is focused on the REC task.

Thank you for pointing out the limitations of our method in terms of generalization, particularly noting that it was only evaluated on the GLIP model for the REC downstream task. In response, we have conducted additional experiments to address this concern.

First, we sincerely apologize for our lack of familiarity with the PNG task, which made it challenging to identify a suitable backbone and adapt our method within a short timeframe. As an alternative, we selected two other downstream tasks—Referring Expression Segmentation (RES) and Visual Question Answering (VQA)—to further validate our approach. Regarding the PNG task, we have included it in our future work to explore the generalization capabilities of our method more comprehensively. As part of future work, we first need to train a foundation model for the PNG task. This model uses an encoder to extract features from both images and text, which are then passed to a downstream head to generate results for the PNG task. Additionally, the encoder needs to adopt a transformer architecture to allow for the dynamic addition of any number of prompts. This has also been explicitly acknowledged in the limitations section of the revised version of our paper.

For the RES task, we use the CLIPSeg model, and for the VQA task, we employ the BLIP model. Detailed explanations of these experiments are provided in the "Generalization Across Downstream Tasks and Backbones" section of the ablation study in the updated version. The results, presented in the table below, show that our method improves performance for both models. Notably, the selected models are already extensively trained on their respective datasets (e.g., CLIPSeg was thoroughly trained on the PhaseCut dataset). Despite this, our method is still able to achieve further accuracy improvements. This effectiveness stems from our method's ability to guide the model to focus on relevant parts of the image based on captions, making it highly applicable to various visual understanding tasks.

This part has been added to Section 5.4 Generalization Across Downstream Tasks and Backbones in the updated version.

Result for Referring Expression Segmentation

Result for Visual Question Answering

To more comprehensively evaluate the generalization capability of our method, we also conduct experiments on two more commonly used benchmarks: cross-domain generalization and cross-dataset generalization, both of which involve image classification tasks. These experiments are detailed in the Cross-Domain Generalization and Cross-Dataset Generalization sections of the Section 5.4 ablation study in the updated version. The results demonstrate that our method maintains strong generalization capabilities across these benchmarks as well.

Dear Reviewer cYSP,

Thank you once again for dedicating your time and effort to reviewing our work. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns. With the rebuttal phase nearing its conclusion, we look forward to your feedback and are available to provide any further clarification you might need.

We sincerely appreciate your positive feedback and recognition of the effectiveness of our method and experiments. We would like to address the concerns you have raised as follows:

The vision and language adaptors adopt the same architecture. How to reasonably exploit the advantages of vision and language cues to ensure consistent fine-tuning of the pre-trained model?

The adapters adopt the same architecture, which first performs a downsampling operation to extract dense prompt information and then upsamples it to map into the corresponding semantic space of each modality. The outputs of the adapters for different modalities are passed to their respective modality encoders, where they interact with the modality-specific embeddings through attention mechanisms. As a result, the knowledge learned by the adapters for different modalities is distinct. For example, the vision encoder encodes information from images, and the prompts generated by the vision adapter tend to focus on spatial information in the images. In contrast, the language encoder encodes textual information, and the prompts generated by the language adapter are more inclined to capture semantic or contextual cues.

What potential strategies could be employed to address the high computational complexity and image input size limitations of the Diff-Prompt model?

Diff-Prompt has high computational complexity. I believe this issue can be addressed from the following perspectives:

• Optimizing the Diffusion Model Design: The optimization methods for the diffusion model can align with mainstream strategies. This includes employing distillation techniques to compress the model size, designing more efficient sampling strategies, and exploring other diffusion acceleration methods. For example, [1] generates results in a single step using a distillation strategy, from which we can draw inspiration to reduce computational complexity.
• Enhancing the Prompt Generator: From the perspective of the prompt generator, future work could explore alternative generative models. In Appendix E (Ablation Study for Prompt Precision), we concluded that the generated prompts do not need to be exceptionally precise to provide effective guidance. This suggests that we could trade off some performance for higher efficiency. For instance, we could utilize more mature models such as VAE or GAN. Additionally, traditional machine learning methods could be considered, such as using the Canny operator to extract image edges as base prompts or leveraging other techniques to extract textual and visual information. However, traditional machine learning methods have limitations, such as difficulty in achieving cross-modal interaction, which requires further research.

The input image is used as a condition for the network and Diff-Prompt generates latent prompts that are decoded back to the original image size using the Mask-VAE decoder. The simplest solution is to constrain the model's input and output sizes and resize the output results as needed. However, this approach limits the model's generalization capability. A more optimal solution is as follows:

(1) Improve Mask-VAE Compatibility Across Resolutions: Our Mask-VAE encoder and decoder are composed of convolutional layers and activation functions. Therefore, as long as the input image's width and height are multiples of 8 (a limitation of the model due to the need for upsampling and downsampling), the Mask-VAE can encode and decode the image effectively. To enhance its performance across varying resolutions, the Mask-VAE needs to be trained on images of multiple resolutions. This ensures robust performance across different resolutions, which serves as the foundation of Diff-Prompt.

(2) Improving Prompt Generation: Several enhancements can be considered for the prompt generator in Diff-Prompt: replacing the linear layers in the DiT model with convolutional layers, similar to Mask-VAE, to enable dynamic input and output lengths; designing an image condition and output alignment module to further constrain the output based on the input conditions; or dynamically adjusting the size of the initial noise based on the input image, allowing the model to better adapt to varying input resolutions.

[1] Luo W, Huang Z, Geng Z, et al. One-step diffusion distillation through score implicit matching[J]. arXiv preprint arXiv:2410.16794, 2024.

Dear Reviewer 3hUM,

Thank you once again for dedicating your time and effort to reviewing our work. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns. With the rebuttal phase nearing its conclusion, we look forward to your feedback and are available to provide any further clarification you might need.

  As the deadline of discussion period is today, please respond to the author reply to see if your concerns are well-addressed at your earliest convenience. Thank you very much.

The authors mentioned that Diff-Prompt may suffer from poor generalization. Based on this, have they proposed any improvement plans to mitigate this issue and enable reproduction by other researchers?

Our method's generalization capability is not poor. In the original Appendix D, we stated: "However, compared to GLIP-L, which has a larger parameter size and more training data, Diff-Prompt and GLIP-L still have a significant gap." However, Diff-Prompt use GLIP-T(A) as foundation model. Our findings indicate that the zero-shot performance of our method is not inferior to GLIP-T(A), though there is a noticeable gap compared to GLIP-L. Throughout the paper, we have consistently used GLIP-T(A) as the foundation model, so a fair comparison would be with GLIP-T(A). However, we also included a comparison with GLIP-L to serve as an upper bound for the zero-shot capability of the model. The significant performance gap between the original GLIP-T(A) and GLIP-L can be attributed to several factors. Compared to GLIP-T(A), GLIP-L has a much larger model size (GLIP-L’s model weights are 6.9GB, whereas GLIP-T(A)’s are only 2.43GB) and is trained on more extensive datasets, which endows it with stronger generalization capabilities.

Furthermore, to more comprehensively evaluate the generalization ability of our proposed method, we conducted comparisons using more general benchmarks, as presented in the updated version's Section 5.4 ablation study on cross-domain generalization and cross-dataset generalization.

Notably, methods like PromptSRC, CoPrompt, CPL, and CoCoLe are specifically designed for generalization tasks, utilizing regularization terms or caches to achieve generalization capability. In contrast, methods such as CoOp, MaPLe, and Diff-Prompt focus on the design of prompts. Our proposed method demonstrates competitive performance compared to state-of-the-art methods, showcasing that our approach also possesses a certain degree of generalization capability. Parts of the results are shown in the table below. For detailed results, please refer to the updated version.

To better improve the generalization ability of the model, a general-purpose prompt generator is essential, as the generated prompts determine the foundation for fine-tuning. In future work, we could consider training a universal prompt generator or using traditional machine learning methods to extract features from images to generate prompts.

As for reproducibility, we have already provided an anonymous repository for the code. We will release the model weights and continue to update the code in the future.

How does the strategy of Diff-Prompt differ from independently training a conditional mask generator and then performing information interaction with a multimodal encoder?

• Rich Expressive Capability: Our model leverages the rich expressive capability of diffusion models, which enables it to generate fine-grained and detailed representations. Furthermore, by incorporating multimodal information during the design process, it ensures a comprehensive and effective approach, making it both innovative and forward-looking.
• Unique Modality Alignment: We align the denoising process of the diffusion model with the encoding process of the encoder, effectively simulating a multimodal information fusion process. This alignment provides a distinctive and efficient approach to integrating multimodal data.
• Lightweight and Efficient: While ensuring the quality of the generated prompts, we have minimized the model's parameters. The final model has a FP32 storage size of only 309MB, and in practical applications, using FP16 computation can further reduce memory overhead. On the contrary, the smallest ViT-B SAM model is 358MB and cannot incorporate language modality information.

In summary, we have developed a customized lightweight prompt generator that not only effectively considers multimodal information but also generates rich prompts while occupying minimal memory, making it efficient and practical.

During prompt generation, did the authors consider how to maintain a balance between visual and textual information to avoid information imbalance caused by modality bias?

The prompts generated by the prompt generator incorporate information from both modalities, focusing on the location of the object described in the text within the image. The absence or insufficiency of information from either modality would make it impossible to approximate the object's position. A prompt is meaningful only if it can effectively indicate the object's approximate location; otherwise, it may mislead the pre-trained model and result in decreased accuracy. Therefore, the closer the reconstructed prompt is to the mask, the better it reflects the degree of modality fusion, indicating sufficient integration of information from both modalities. The effectiveness of our method further validates that our generated prompts are indeed useful, demonstrating the successful fusion of information from different modalities.

The authors claim to generate rich prompts, but the masks generated during the diffusion process do not exhibit finely controllable generation.

I will explain the controllability of our method from the following two perspectives:

• Controllability of the Diffusion Model Generation Process. In the design of the prompt generator, the image and text are concatenated after passing through the embedding layer and used as a condition to guide the generation of the diffusion model, enabling control over the generated results. This approach also ensures that the generated prompts are highly correlated with both modalities.
• Average Recall for IoU Intervals. For the RefCOCO dataset, where each caption corresponds to a bounding box and a segmentation mask, we conducted the following experiment: we performed an in-depth analysis of the results on the RefCOCO validation dataset by reconstructing the prompts generated by the prompt generator. We then calculated the Intersection over Union (IoU) between the reconstructed masks and the ground-truth masks in the labels. Based on the IoU values, we divided all results into 10 bins, such as IoU > 0 and ≤ 0.1, IoU > 0.1 and ≤ 0.2, and so on, and calculated the accuracy for each bin. The results are shown in the table below. We conduct a more detailed analysis in Appendix E Ablation Study for Prompt Precision of the updated version.

From the results, we can observe that when IoU is below 0.4, the accuracy significantly improves as IoU increases. However, when IoU exceeds 0.4, the improvement rate slows down, and when IoU exceeds 0.8, the accuracy stabilizes. This indicates that even a coarse prompt, without requiring a highly precise mask, can provide effective guidance. This finding aligns with the essence of prompts, which is to serve as guidance rather than the final result.

This approach may not necessarily adapt to highly fine-grained tasks, which is evidently inconsistent with the problem the authors intend to solve.

The above statements, along with the results in Table 1, demonstrate that the prompts proposed by our method do not need to be highly precise to effectively guide fine-grained downstream tasks. To further illustrate this point, we added two additional downstream tasks: Referring Expression Segmentation and Visual Question Answering, as shown in Table 4 and Table 5 of the revised version. The experimental results confirm the effectiveness of our method.

Result for Referring Expression Segmentation

Result for Visual Question Answering

Compared with previous work, this paper seems focused on training masks through diffusion models to generate masks with fine edges for boundary control, but the contribution beyond this aspect is insufficient.

Thank you for your feedback. I would like to further elaborate on the contributions of our method:

(1) Generalization to various visual unstanding tasks: Diff-Prompt can be applied to a wide range of text-based visual understanding tasks, such as referring expression, referring expression segmentation, visual question answering, image captioning, visual reasoning, and more. The concept behind our method can also be extended to other modalities, such as visual-audio classification, as the prompts we design are based on adding attention from other modalities to the visual modality, offering a wide range of application scenarios.

(2) Better Interpretability: Prompt learning initially emerged in the natural language processing domain, where natural language prompts (e.g., CoT) were used to guide models. While randomly learned prompts can improve performance to some extent, what the model actually learns remains unclear. This lack of interpretability hinders deeper research, and in practical applications, interpretable prompts are preferable. Similarly, in the computer vision, most existing visual prompt learning methods rely on end-to-end random optimization, which is a black-box process. Our goal is to generate specific prompts, using these as the foundation to guide pre-trained models. Therefore, our contribution is not to generate masks but to use masks as a prompt foundation to make the approach more interpretable.

(3) Multimodal Alignment: Current interaction methods rely on network designs to enable communication between modalities. This is simple but effective. In our work, we conduct another exploration, that is to align the denoising process in diffusion models with the encoding process of the pre-trained model. Experimental results show that this alignment is effective and leads to improved performance.

(3) Complex Downstream Tasks: Unlike prior prompt learning research, which mainly focuses on simpler tasks like image classification, our work applies prompt learning to more complex multimodal tasks, such as referring expression comprehension, which requires precise object localization. We also analyze multiple metrics to provide a more comprehensive evaluation of our method’s performance.

(4) Boundary Control: Our work does not aim to generate fine edges for boundary control but rather to produce a saliency map. The distinction is that boundary control requires high precision, while a saliency map simply informs the model about which regions of the image require more attention. Since the saliency map functions as a prompt rather than a precise segmentation result, pixel-level accuracy is unnecessary. Our ablation studies show that even a coarse saliency map can effectively guide the model, demonstrating the robustness of our approach. Additionally, when we refer to "fine-grained" tasks, we mean downstream tasks like referring expression comprehension, where the final output demands precise four-pixel bounding box coordinates.

(5) Outstanding performance: Our experiments demonstrate that generating prompts using customized methods also delivers excellent performance compared to randomly learned prompt learning methods.

Thank you for your detailed review and for recognizing the experimental contributions and innovative aspects of our work. We have carefully reviewed the weaknesses and questions you raised and have provided responses to each of them below.

Diff-Prompt is compared with two different types of efficient parameter tuning methods, but the number of baselines for the two methods is seriously imbalanced. It would be preferable to include more comparative experiments with adapter methods.

Thank you for pointing this out. Indeed, we initially lacked comparisons with Adapter Tuning methods. To address this, we have added three adapter tuning methods: Tip-Adapter (ECCV 2022)[1], Meta-Adapter (NeurIPS 2023)[2], and MMA (CVPR 2024)[3]. Tip-Adapter employs a training-free adapter tuning approach, which adjusts data distribution solely by updating key-value pairs without the need for training. Meta-Adapter introduces meta-learning into adapter tuning, while MMA designs a multimodal adapter that fully considers interactions between modalities. Tip-adapter and Meta-adapter are methods frequently used for comparison in adapter tuning, while MMA is a very recent and highly effective approach.

The results are shown in the table below. We have also updated the original manuscript and merged the two previous comparison tables into a single table, which can be found as Table 1 in the revised version.

From the results, it can be observed that prompt tuning methods generally achieve greater improvements in accuracy compared to adapter tuning methods. This is because adapter tuning typically requires adjusting the learned network to fit the entire data distribution, which may compromise the generalization ability of the original backbone network, thereby making training more challenging. As a result, the accuracy improvement is relatively limited.

[1] Zhang R, Zhang W, Fang R, et al. Tip-adapter: Training-free adaption of clip for few-shot classification[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 493-510.

[2] Song L, Xue R, Wang H, et al. Meta-adapter: An online few-shot learner for vision-language model[J]. Advances in Neural Information Processing Systems, 2023, 36: 55361-55374.

[3] Yang L, Zhang R Y, Wang Y, et al. MMA: Multi-Modal Adapter for Vision-Language Models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23826-23837.

Dear Reviewer PtHg,

Thank you once again for dedicating your time and effort to reviewing our work. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns. With the rebuttal phase nearing its conclusion, we look forward to your feedback and are available to provide any further clarification you might need.