Prompt Diffusion Robustifies Any-Modality Prompt Learning

Q3: The approach has limited empirical exploration outside the image-text domain, raising questions about its generalizability to other modalities.

Thank you for this suggestion. To explore the generalizability of our method beyond the image-text domain, we applied our approach to video understanding tasks, specifically using the setup from Ju et al. ("Prompting visual-language models for efficient video understanding," ECCV 2022). We conducted experiments on closed-set action recognition datasets, including HMDB-51, UCF-101, Kinetics-400 (K-400), and Kinetics-700 (K-700). The results, in terms of Top-1 accuracy, are shown below:

Our method consistently improves performance across all datasets. This demonstrates that Prompt Diffusion can effectively adapt to video tasks, leveraging its ability to generate instance-specific prompts that capture temporal and contextual information unique to video data.

However, we recognize that applying our method to other modalities, such as audio or multi-modal tasks, may introduce new challenges. For instance, the nature of sequential and hierarchical dependencies in audio signals may require further adaptations to the diffusion process, such as incorporating domain-specific priors or preconditioning steps for better feature alignment.

We will include these experimental results and a discussion of potential challenges and adaptations in Appendix K of the revised paper to address this suggestion comprehensively.

Q4: The high resource demands of diffusion models, including substantial GPU and training time requirements, make them impractical for parameter-efficient methods such as prompt learning.

While diffusion models are often associated with high resource demands, we have designed our method to minimize these costs and maintain practicality for parameter-efficient prompt learning.

Training efficiency: As discussed earlier, the additional training time introduced by our method is modest, ranging from 1.1x to 1.3x compared to baseline methods. For instance, training times for CoCoOp and VPT increase from 17 hours to 20 hours and from 25 hours to 28 hours, respectively, when incorporating Prompt Diffusion. This small increase is justified by the better performance improvements demonstrated across various metrics, as highlighted in our comparative analysis.

Inference efficiency: During inference, we employ a fast ODE-based sampling strategy, which reduces the number of timesteps required for the diffusion process. As shown in Figure 4, our method achieves substantial performance gains (Harmonic Mean) with minimal increases in inference time. For example, with five function evaluations (NFE), we achieve near-optimal performance while the inference time remains competitive with the baseline CoCoOp method. This balance ensures that our method is both computationally efficient and effective.

These results demonstrate that our approach effectively balances performance gains with computational costs, making it accessible and practical for real-world applications. We have included these clarifications in the revised manuscript to address concerns about resource demands comprehensively.

We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions.

Q1: The paper does not fully articulate the specific limitations of the SOTA prompt methods in adapting to distributional shifts in data.

Thank you for your insightful feedback. We acknowledge the importance of quantifying the limitations of existing SOTA prompt methods under different distributional shifts. To address this, we have conducted a comprehensive evaluation of our method in combination with Xiao et al.'s "Any-Shift Prompting for Generalization over Distributions" (CVPR 2024) across various types of distributional shifts, including covariate, label, concept, conditional, and multiple shifts. Below, we provide a detailed comparison:

Covariate Shift Comparison

Label Shift Comparison

Concept and Conditional Shift Comparison

Multiple Shifts Comparison

From these results, it is evident that our method improves performance across all types of shifts compared to Xiao et al.'s method alone. This comprehensive comparison not only highlights the limitations of existing SOTA methods in adapting to distributional shifts but also underscores the critical importance of our contribution. Specifically, Prompt Diffusion enhances instance-level adaptability by refining prompts during inference, thereby addressing the instability and inefficiency caused by fixed prompts under shifting distributions.

We have included these experimental results and analyses in Appendix J of the revised paper for further reference.

Q2: Although the diffusion model is proposed to generate sample-speciìc, customized prompts, the paper does not clearly explain why diffusion was chosen over other, potentially simpler methods.

To address why diffusion was chosen over other simpler statistical approaches, we conducted comprehensive comparisons with alternative methods, as shown in Table 4 of our paper. This table evaluates different probabilistic approaches such as GANs, VAEs, and Normalizing Flows. The results clearly demonstrate that our diffusion-based method achieves the best overall performance across Base, New, and Harmonic Mean (H), highlighting its superior ability to generate instance-level, customized prompts and adapt effectively to diverse data samples.

In addition, we directly compare our approach with Bayesian Prompt Learning by Derakhshani et al. (2023) in handling challenging datasets, as shown in the following table:

Our method consistently outperforms Bayesian Prompt Learning across all datasets, achieving higher average performance. The key advantage of our diffusion model lies in its iterative generative framework, which allows for richer representations and more precise refinements compared to the statistical modeling employed by Bayesian approaches.

These results validate the unique strengths of our diffusion-based approach, both in terms of practical effectiveness and its ability to handle complex, instance-level prompt generation.

Dear Reviewer xQzt:

Thank you for your valuable feedback. We have conducted all the additional experiments and incorporated clarifications in the revised manuscript, including addressing distributional shifts, justifying the choice of diffusion models, extending evaluations to video tasks, and analyzing computational efficiency.

As the rebuttal phase is coming to a close, we kindly ask if these updates have resolved your concerns. Please let us know if you have any additional questions or feedback—we would be happy to address them promptly.

Best regards,

Authors

Q3: The length of prompts in prompt learning methods can affect the ìnal performance. Does the proposed method also encounter similar situations?

Thank you for sharing the insight. The length of prompts indeed plays a role in the final performance of prompt learning methods. To address this, we conducted experiments with our prompt diffusion method using different prompt lengths across various baseline methods.It is worth noting that the prompt lengths used in our experiments align with the default prompt lengths adopted by the respective baseline methods: L = 4 for VPT and CoCoOp, and L = 9 for MaPLe. The results are summarized below:

VPT + Prompt Diffusion

CoCoOp + Prompt Diffusion

MaPLe + Prompt Diffusion

These results demonstrate that the optimal prompt length varies across different baselines, with moderate lengths generally leading to better performance. For our method, we use the respective default prompt lengths for fair comparisons: L = 4 for VPT and CoCoOp, and L = 9 for MaPLe. We have included these results and their analysis in Appendix E for further details.

Q4: There are also some works in the field of prompt learning that address the limitations of fixed prompts by generating instance-level prompts (e.g. [1]).

Thank you for pointing us to the work by Liu et al.

We agree that related works addressing the generation of instance-level prompts are highly relevant to our study. For example, Liu et al. propose a method to generate instance-specific prompts using a Bayesian approach. Their work shares similarities with our approach in addressing the limitations of fixed prompts, as both aim to adapt prompts at the instance level. However, our method differs in its use of a diffusion process for progressively refining prompts, allowing for a more generative and flexible approach across diverse modalities and tasks.

We will supplement the related work section with a discussion of Liu et al.'s work and highlight the distinctions and complementary aspects between their method and ours to make the paper more comprehensive.

Q5: I am curious about the setting of the two loss weights $\beta$ in Equation (8). Can further experimental analysis be provided?

Thank you for your question regarding the loss of weight $\beta$ in Equation (8). To analyze the effect of $\beta$ on model performance, we conducted experiments with different values of $\beta$ across VPT, CoCoOp, and MaPLe. The results are shown below:

VPT + Prompt Diffusion

CoCoOp + Prompt Diffusion

MaPLe + Prompt Diffusion

From the results, we observe that the best performance across all metrics (Base, New, and Harmonic Mean) is achieved when $\beta$= 0.01 This suggests that a small weight for the diffusion loss term provides the optimal balance between the cross-entropy loss and the reconstruction objective. Setting $\beta$ too high $\beta$ = 1 places excessive emphasis on the diffusion term, which slightly degrades performance. Conversely, setting $\beta$= 0 removes the benefits of the diffusion process entirely, resulting in a significant drop in performance.

We have included these results and analysis in Appendix I for further clarification.

We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions.

Q1: During training, does it introduce a significantly larger computational load compared to conventional prompt learning methods? Can a comparative analysis be provided to address this concern?

To address the reviewer’s concern about computational load, we conducted a comparative analysis of training time across baseline methods and our proposed approach. While our method introduces a slightly higher computational load due to the per-sample prompt overfitting step and the diffusion process, the increase is modest. Specifically, the per-sample prompt overfitting step requires only three iterations to generate the overfitted prompts, ensuring efficiency without compromising performance. Below, we provide the training times in hours for each method:

The per-sample prompt overfitting step, coupled with our diffusion model, plays a critical role in enhancing the model's adaptability to diverse samples. Despite the slightly longer training time (approximately 1.1x to 1.3x that of baseline methods), the better improvement in Base, New, and Harmonic Mean (H) demonstrates a favorable trade-off between computational load and performance.

We have clarified this in Appendix E, where additional details about computational cost and training efficiency are provided.

Q2: While the proposed method is plug-and-play and the pipeline figure demonstrations are based on CoCoOp, it would be benefical to include sections addressing visual prompt tuning and multi-modal prompt tuning.

While the proposed method uses the meta-net \pi in CoCoOp, it is fully adaptable to prompt learning methods like VPT and MaPLe, which do not rely on \pi. For these methods, during training, we similarly perform per-sample prompt overfitting to generate the corresponding overfitted prompts or tokens. These overfitted prompts or tokens are then reconstructed using the diffusion process. Finally, the reconstructed prompts or tokens are embedded back into the original models, such as VPT and MaPLe, for prediction. This flexibility demonstrates that our method is not tied to any specific architecture and can seamlessly integrate with various prompt learning paradigms, including visual prompt tuning and multi-modal prompt tuning. We have clarified this in the revised paper and expanded the discussion to explicitly address these methods (See Appendix H).

Thanks for your responses. Most of my concerns have been addressed and I will increase my score to 6. However, I still have a question regarding the reconstruction objective when the proposed method is applied to multi-model prompts: Is it only the textual branch that will be reconstructed, or will there be an additional reconstruction loss for the visual branch prompt? I encourage the authors to provide a more detailed explanation.

Thank you for recognizing the strengths of our method, including its ability to generate customized prompts, its plug-and-play nature, and its effectiveness in extracting unique domain details for improved prediction and generalization. We are especially grateful for your support and for recommending acceptance of our work.

Q1: The introduction of a diffusion model increases the complexity of the system, and therefore whether the training time is likely to be relatively long.

The reviewer is correct that our method requires additional training time due to the per-sample prompt overfitting step. However, this process converges within just three iterations, keeping the overall training time manageable. Specifically, our training time is approximately 1.1x, 1.2x, 1.3x, and 1.3x that of VPT, CoCoOp, MaPLe, and CoPrompt, respectively. Importantly, our method performs better than these baselines, achieving a favorable trade-off between training time and performance. We have highlighted this trade-off more clearly in the revised paper.

Q2: Is the author's approach a two-stage process, starting with a prompt study followed by a prompt proliferation?

Our prompt diffusion model is an end-to-end framework. During training, for each sample, we first perform per-sample prompt overfitting to generate an overfitted prompt, which is then used as input for the diffusion process to reconstruct the prompt. This ensures seamless integration of both stages within the model. We have included this clarification in the revised paper.

Q3: Diffusion models incorporate randomness in the generation process, which may lead to uncontrollable íuctuations in the generated prompts and thus affect the robustness of the model. How to cope with the randomness of the generated prompts and avoid the instability of prediction caused by it?

Our method indeed accounts for the inherent randomness in diffusion models by incorporating robust denoising mechanisms and a fast ODE-based sampling strategy. These ensure that even with variations in the noise vectors, the diffusion process remains stable and reliable. Empirical results across diverse datasets consistently demonstrate that our approach achieves stable performance without noticeable degradation caused by noise sensitivity (See Figure 6). Specifically, our plugin effectively extracts unique domain details from the test image without conflating them with class labels, regardless of the initial random noise. This robustness is key to maintaining the stability of predictions and the overall performance of the model.

Q4: Whether the authors' approach can be migrated to the VPT-deep prompt learning paradigm?

To demonstrate the versatility of our method, we have included additional experiments with VPT-deep in the revised paper. As shown in the table below, incorporating our prompt diffusion improves the performance of VPT-deep across all metrics, including Base, New, and Harmonic Mean (H):

These results confirm that our approach is not limited to VPT-shallow but can also effectively enhance the VPT-deep prompt learning paradigm. We have added these results in the revised paper (See Appendix G).