VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models

We appreciate the reviewer's valuable feedback and recognition of VaMP's clear presentation, well-defined mathematical formulations, sound reasoning, and superior performance across multiple benchmarks. We carefully address the reviewer's comments below.

W1: I feel that uncertainty-based modeling may increase computational overhead. Could the authors compare the models and computational complexity of deterministic and uncertainty-based prompting strategies?

R: We thank the reviewer for raising this important practical consideration. As mentioned in Section S3 of our supplementary material, we have provided a quantitative analysis of computational efficiency. Here, we further provide a more detailed quantitative analysis of the trade-off between inference cost and accuracy under different numbers of Monte Carlo samples S. As shown in Table A7, even with a modest number of samples (e.g., S=5 or S=10), VaMP achieves clear gains in harmonic mean accuracy over the strong MMRL baseline, with only marginal increases in inference time (e.g., 5.3ms/image → 6.1ms/image). Specifically, at S=10, VaMP improves HM from 81.20 to 82.37 with a latency increase of only 0.8ms per image—demonstrating an efficient balance between generalization and overhead. Notably, the performance saturates quickly with increasing S, suggesting that our method remains highly efficient even with limited sampling. This confirms that VaMP can be deployed with low overhead while still leveraging uncertainty modeling for improved generalization.

Table A7: Comparison of inference cost and parameter count.

W2: The current benchmarks and settings are relatively simple. Can the proposed method be extended to dense prediction or video-based benchmarks?

R: Thank you for this valuable suggestion. We evaluated VaMP's generalizability beyond classification by conducting experiments on open-vocabulary segmentation and action recognition tasks. We integrated VaMP into state-of-the-art frameworks of two tasks: CAT-Seg [a] for segmentation and FROSTER [b] for action recognition, utilizing CLIP ViT-B/16 as the backbone encoder. Both baselines are extensions built upon CLIP, allowing our method to be seamlessly integrated into these approaches.

For open-vocabulary segmentation, we followed CAT-Seg's protocol, training on COCO-Stuff (118K images, 171 categories) and evaluating across multiple benchmarks including ADE20K, PASCAL-Context, and PASCAL VOC with varying category scales (59-847 classes). The results are shown in Table A8:

Table A8: Performance comparison of different methods on ViT-B/16 backbone for open-vocabulary segmentation.

For open-vocabulary action recognition, we adopted FROSTER's base-to-novel evaluation protocol on Kinetics-400 and UCF-101, where models trained exclusively on base classes are required to generalize to unseen novel action categories. The results are shown in Table A9:

Table A9: Performance comparison under base-to-novel setting on two widely-used video benchmarks, i.e., Kinetics-400 and UCF-101. B: Base classes, N: Novel classes, HM: Harmonic mean.

As shown, VaMP achieves consistent improvements across different tasks, demonstrating the scalability of our approach and its strong generalization capabilities.

[a] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation. CVPR 2024

[b] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition. ICLR 2024.

W3: There are many strategies for uncertainty modeling, such as Monte Carlo sampling and evidence learning. Why did the authors choose to model the Gaussian mean and variance using MLPs?

R: We adopt MLPs to model the posterior mean and variance due to their simplicity, efficiency, and compatibility with variational inference. This design choice follows the well-established principles of Variational Autoencoders (VAEs)[c], which have been widely adopted in the machine learning community for modeling uncertainty. Similar to VAEs, our approach allows lightweight, layer-wise modeling of uncertainty and integrates naturally with the reparameterization trick for stable end-to-end training. Compared to alternatives like evidence learning, our approach does not require additional supervision or calibration loss, and directly optimizes downstream performance. Empirically, Table 4(b) in our main manuscript shows consistent improvements over deterministic prompts, validating our choice. Exploring richer uncertainty modeling strategies remains a promising direction for future work.

[c] Auto-Encoding Variational Bayes. ICLR 2014.

W4: According to Figure 1, the prompts only operate at the encoding stage. Why not consider integrating them at the decoding stage as well?

R: We would like to clarify that, our method builds on CLIP-like architectures following previous methods [79, 80, 24, 71, 11, 66, 15], which employ a dual encoder architecture—a vision encoder and a text encoder—without a decoding stage. As such, there is no separate decoder module where prompts could be inserted. All computations are encoder-based, and predictions are made via similarity in the joint embedding space. Our prompt adaptation targets the transformer layers within the text encoder, which directly influence the representation used for image-text matching. For specific models or applications when the decoders are needed, our prompt design can be equally applicable.

Thank you for the reply. We would like to clarify several important points regarding the originality and contributions of our work.

Regarding Q1

We hold a sincere belief in the importance of evaluating research novelties and contributions within their rightful context and problem scope. We would like to clarify that, both [1] and VaMP build upon foundational techniques from established variational inference literature [a, b]. Indeed, [1] also clearly pointed out that their variational inference modeling was inspired by these works (P4, Sec 3.2).

Below, we will clarify the key differences between VaMP and [1]:

[1] applies variational inference techniques only to the input textual prompts by modeling them as latent variables drawn from a standard Gaussian distribution, enabling diverse prompt generation during inference for improved robustness.

However, this approach has several limitations. First, by restricting variational modeling to input-level prompts with global latent variables, [1] cannot capture hierarchical feature interactions throughout the network nor fine-grained semantic variations between individual tokens, significantly limiting representational capacity. Additionally, [1]'s text-only prompt tuning disregards valuable information from visual modalities that could better align cross-modal representations. Finally, the standard Gaussian prior assumption is shared across all classes, preventing the model from capturing inter-class variations and resulting in less discriminative prompt distributions.

In contrast, VaMP addresses these weaknesses through a comprehensive variational framework that operates across modalities and network depths. By implementing token-wise variational modeling across multiple intermediate network layers, VaMP simultaneously captures fine-grained semantic relationships between individual tokens and enables prompts to influence representations at multiple abstraction levels throughout the network hierarchy. As evidenced in Table 1, our model performs quite well on fine-grained datasets, with consistent improvements across base classes, novel classes, and harmonic mean metrics compared to SOTA approaches. Particularly notable gains are observed on StanfordCars (Base +2.48, Novel +5.07, H +3.85), Food101 (Base +2.20, Novel +1.66, H +1.93), FGVCAircraft (Base +0.47, Novel +4.10, H +2.61), and DTD (Base +0.47, Novel +2.20, H +1.68). These results further demonstrate the importance of both token-wise and layer-wise variational modeling for capturing subtle discriminative features in fine-grained visual classification tasks. Secondly, VaMP's multi-modal design incorporates both visual and textual signals when inferring posterior distributions, creating more aligned cross-modal representations. Finally, by employing class-aware priors instead of standard Gaussian distributions, VaMP generates more discriminative prompts that better capture category-specific features and decision boundaries.

Therefore, we believe that it is not a fair judgment to say that “the insight … is rather trivial”. Our idea and design principles are fundamentally different from [1].

[a] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR 2014

[b] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta-learning probabilistic inference for prediction. ICLR 2019.

Regarding Q2

• Regarding token-level prompt modeling, existing approaches[24, 15] treat tokens as deterministic optimization targets that cannot capture uncertainty or adaptively modulate based on cross-modal alignment confidence. However, our method treats each token as a latent variable under a probabilistic variational framework, rather than a deterministic optimization target. This uncertainty-aware token modulation allows us to capture ambiguity in the cross-modal alignment and selectively adapt informative tokens. This probabilistic formulation is novel and provides clear performance and generalization benefits over deterministic tuning strategies, as shown in Table 4.

• Regarding prior-guided strategies, existing variational prompt tuning methods[1] universally adopt standard Gaussian distributions as priors, while we are the first to introduce class-aware prototypes that integrate both instance features and class prototypes as priors to capture task- and class-level structure in variational prompt tuning. This class-aware prior design enables the model to leverage semantic relationships between categories and provides more informed initialization for prompt tokens, leading to better convergence and stronger cross-modal alignment compared to uninformative standard Gaussian priors.

We sincerely appreciate the reviewer's comments. We will further clarify these distinctions in our revision. Any further comments and suggestions are welcome.

We appreciate the reviewer's valuable feedback and recognition of VaMP's technical soundness. We carefully address the reviewer's comments below.

W1/Q1: Unclear implementation: How exactly is $z_{i-1}$ calculated? What input is used to compute it? The process is vague and could hinder reproducibility.

R: The term $z_{i-1}$ represents the learnable tokens in the visual branch, which we have briefly introduced in L118. Its exact computation depends on the multi-modal prompt learning method. In MaPLe, these visual tokens are generated from the language prompts via a linear transformation implemented with an MLP. Meanwhile, MMRL obtains visual tokens from a shared latent space that is a set of learnable tokens, which are initialized by sampling from a Gaussian distribution, and then uses separate linear projection functions (also implemented with MLPs) to generate modality-specific prompts. We will provide more details in the revision to further clarify and release our code to ensure reproducibility.

W2/Q3: There is no discussion of the method's limitations in the main paper, although it is stated to be present in the supplementary material.

R: Thanks for the suggestion. We will move this discussion into the main paper in the revision.

W3: The performance improvement over the state-of-the-art is marginal, raising concerns about the impact.

R: We respectfully disagree with this assessment. VaMP consistently improves upon strong baselines across 11 datasets in both few-shot and cross-dataset generalization settings. In the base-to-novel generalization setting, VaMP outperforms the year-25 state-of-the-art method MMRL by achieving 78.67 (+1.51) on novel classes and 82.37 (+1.17) on the harmonic mean (HM) metric. These gains are non-trivial: they represent average improvements across 11 diverse datasets, and the HM metric, computed as HM = 2 × base_acc × novel_acc / (base_acc + novel_acc), is inherently challenging to improve due to its non-linear nature.

For context, MMRL's improvement over the previous year 2024's SOTA method, MMA, was 0.36 for novel class accuracy and 1.33 for HM, indicating that our results are neither marginal nor negligible. Likewise, in cross-dataset evaluation, the improvement VaMP achieves over MMRL in the source dataset (72.83 vs. 72.03) is comparable to MMRL's improvement over MMA (72.03 vs. 71.00). Moreover, on the target dataset, VaMP's average improvement over MMRL across 10 datasets (67.74 vs. 67.25) parallels MMRL's gain over MMA (67.25 vs. 66.61).

Beyond accuracy, VaMP introduces a principled framework for sample-specific, uncertainty-aware adaptation, which offers broader benefits beyond raw performance, as also recognized by other reviewers. We believe these results and contributions together underscore the superiority of the method.

W4/Q2: Overlaps with prior work. Please clarify the key technical differences from [24] and [15], beyond what has already been published.

R: We respectfully but firmly disagree that our contributions merely overlap with prior work. Firstly, MMRL [15] and MaPLe [24] serve as baseline methods upon which our approach builds, as stated in the preliminary section (Lines 108-109); we do not claim any of them as our own contributions. Second, one key novelty of VaMP lies in the reformulation of prompt learning as a variational inference problem for multi-modal prompt learning (as also recognized by other reviewers). Through this formulation, VaMP enables sample-specific and token-level uncertainty modeling, a capability not offered by any existing methods. We further introduce a class-aware prior derived from semantic prototypes to guide adaptation and enhance generalization. To the best of our knowledge, there is no existing methods providing such a novel framework containing these ideas for multi-modal prompt learning.

Moreover, the originality and rigour of our framework are also widely recognized by other reviewers. Reviewer hikd noted that it represents “a novel framework”, Reviewer EaPS described it as “well-motivated, addressing a clear gap in current multi-modal prompt tuning methods”, “conceptually elegant”, “novel and intuitive,” and Reviewer R47nX confirmed that “the mathematical formulations are well-defined, and the reasoning is sound.” We believe these endorsements underscore the conceptual and technical novelty and depth of VaMP.

W5: The method is only evaluated on a narrow range of models (e.g., CLIP), with no extension to more recent large-scale vision-language models.

R: Our method is architecture-agnostic and can be easily integrated into more complex Vision-Language Models (VLMs), as it only requires access to intermediate transformer layers and the ability to inject prompt tokens. The variational adaptation module is lightweight (MLPs per layer) and does not assume any specific backbone structure. Following this suggestion, we have validated VaMP on several state-of-the-art VLMs that represent the latest advances in CLIP-style architectures, including EVA-CLIP [a], SigLIP [b], and SigLIP 2 [c]. These models share structural similarities with CLIP, allowing quick adaptation within the rebuttal timeframe. As shown in Table A6, VaMP consistently improves performance across all tested VLMs. Notably, these improvements on novel classes underscore VaMP's strong generalization capability, which is an essential factor in prompt learning. The consistent gains across these diverse architectures highlight the effectiveness and generalizability of VaMP.

Table A6: Performance comparison of different methods on ViT-B/16 backbone for base, novel classes, and harmonic mean (H).

Furthermore, we have also applied our method to different tasks and different models, as demonstrated in Table A1 and Table A2 in our reply to Reviewer hikd. The consistent improvements observed across diverse tasks and models clearly illustrate the generalization capability and versatility of our approach, and we hope the reviewer can re-consider the benefits of VaMP.

Additionally, our method is also technically feasible to be integrated with MLLMs, such as QWen-VL and InternVL or other GPT-4V like models. Due to limited time and computational resources during the rebuttal period, we could not conduct additional experiments on such models, but we are confident on outcome given all our experiments executed on all VLMs so far.

[a] EVA-CLIP: Improved Training Techniques for CLIP at Scale. Arxiv 2023.

[b] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.

[c] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. Arxiv 2025.

We appreciate the reviewer's valuable feedback and recognition of VaMP's clear motivation, conceptual elegance, methodological noveltyand thorough empirical validation across diverse benchmarks. We address the reviewer's comments below carefully.

W1: The class-aware prior is only used during training, and replaced by a standard Gaussian at test time—some reflection on the potential impact of this train-test mismatch would be helpful.

R: We appreciate this suggestion. Indeed, class-aware priors are used only during training to regularize the posterior and encourage semantically structured latent spaces. At test time, we revert to standard Gaussian priors due to the absence of labels. While this introduces a train-test mismatch in the prior, the learned posterior remains image-conditioned and captures instance-specific variations. Empirically, we observe strong generalization on novel classes (Table 1) and unseen datasets (Table 2), indicating that the semantic guidance provided during training improves prompt consistency without overfitting to label-specific priors.

W2: Second, although the variational approach introduces uncertainty into prompt learning, the paper doesn't delve into how this uncertainty might be utilized in practice, such as for calibration or confidence-aware decision making, which could be an interesting extension.

R: Thanks for the suggestion. In VaMP, uncertainty is primarily leveraged to enable diverse, instance-conditioned prompt generation during training and inference. While we do not explicitly use the uncertainty estimates for downstream decision-making, we agree this is a promising direction—e.g., using posterior variance for selective prediction, reweighting, or out-of-distribution detection. We will add a brief discussion on this point in the conclusion to highlight it as a potential avenue for future work.

Q1: The current experiments use a fixed 16-shot setting. Have the authors explored how VaMP performs under more limited (e.g., 1-shot) or more generous supervision?

R: Following the suggestion, we evaluated VaMP under the challenging 1-shot Base-to-Novel Generalization setting across 11 diverse classification datasets. The results are reported in Table A5 below. In this demanding scenario, models are trained on base classes with only a single example per class and must generalize to previously unseen novel classes. As demonstrated in Table A5, VaMP achieves superior performance with the highest harmonic mean (71.23) and novel class accuracy (72.48), substantially outperforming existing methods in this cross-category generalization task. Most notably, compared to the current state-of-the-art MMRL method, VaMP demonstrates remarkable improvements with a +4.05% gain on novel classes (72.48 vs. 68.43) and a +2.11% increase in harmonic mean (71.23 vs. 69.12). This significant performance improvement over the previous state-of-the-art highlights the effectiveness of our variational prompt learning approach, which better captures the uncertainty inherent in prompt distributions when transferring knowledge from base to novel categories, demonstrating robust performance and sample efficiency even under such extremely data-constrained conditions.

Table A5: 1-shot Base-to-Novel generalization results on 11 datasets with ViT-B/16 backbone.

Q2: Since the posterior over prompt tokens is conditioned only on the image, have the authors considered whether adding textual input (e.g., class name or prompt template) could further inform the distribution?

R: Our goal is to generate image-specific prompts that adapt to the visual content of each instance, thereby capturing intra-class variability and improving generalization to novel or ambiguous samples. Conditioning on the image is therefore essential for enabling this instance-level adaptation. In contrast, conditioning on textual input such as class names would shift the modeling focus toward class-specific or template-guided prompts, which could reduce the diversity and flexibility of the learned prompts. Notably, the class label is already indirectly incorporated via the prototype-based prior during training, which allows us to guide the posterior distribution without explicitly relying on textual input as a conditioning signal.

Q3: In Section 4.4, the authors mention drawing 10 Monte Carlo samples from the posterior to compute the final prediction. Could the authors clarify whether the same set of sampled prompts is shared across all classes for computing similarities, or whether separate prompts are sampled for each class template (e.g., "a photo of a [CLASS]")?

R: The sampled prompts are image-conditioned and shared across all class templates. That is, for a given image, we draw 10 samples from the posterior $q_φ(z|x)$, and compute cosine similarities between the resulting text features (with each class name inserted) and the frozen image embedding. This design ensures efficiency and avoids leakage of label-specific information into the sampling process.

Q4: In Section 4.3, class prototypes are computed by averaging the posterior means of image embeddings within each class (Eq. 18). Are these computed once before training (e.g., with frozen CLIP features), or are they updated dynamically during training as the posterior evolves?

R: The class prototypes are computed once before training using frozen CLIP image features. They remain fixed throughout training and serve as stable semantic anchors for constructing the class-aware prior. This avoids potential instability from dynamic updates and ensures consistency across training iterations. We will clarify this in our revised manuscript.

We appreciate the reviewers’ valuable feedback and their recognition of our framework’s novelty, robust generalization, and good performance. We have carefully addressed the comments below.

W1: Constructing the class-aware prior depends on class labels to compute prototypes, which may not generalize well to unsupervised or label-scarce settings.

R: Our experiments demonstrate that VaMP performs well in label-scarce settings. In the base-to-novel setting, we only have access to base class labels during training with merely 16 shots per class, representing a highly label-scarce scenario. For novel classes, no label information is available during training. Under this constraint, prototypes are computed only for base classes, yet VaMP achieves the best novel class performance (Table 1), showing better generalization to unseen categories. To further validate this finding, we conducted additional experiments under the extreme 1-shot constraint as suggested by Reviewer EaPS (Table A5), where VaMP continues to outperform existing methods, thereby corroborating its effectiveness under severe label scarcity. Moreover, in the cross-dataset evaluation (Table 2), the model is trained solely on ImageNet using 16 shots per class and directly tested on unrelated datasets without using any label or prototype information, effectively operating in a zero-shot setting. Despite these limitations, VaMP still consistently outperforms prior methods. At test time, we revert to a standard Gaussian prior, ensuring applicability without label access.

We believe the key reason for our method’s strong generalization is its variational formulation. It has been shown in the literature (e.g., [a]) that VAE-like designs in traditional generative models exhibit robust generalization capabilities because the latent variable acts as a structured bottleneck that captures salient, disentangled factors of variation. Similarly, in VaMP, the latent prompt distribution encourages the model to encode task-relevant variations in a sample-specific yet class-agnostic manner, enabling strong zero-shot transfer. This design makes VaMP particularly suitable for real-world applications where labeled data is scarce or expensive to obtain.

Meanwhile, we agree that our approach still requires at least some labeled data to compute meaningful class prototypes, which is a practical assumption and normally easily accessible. It remains an open challenge to obtain reliable prototypes in completely unsupervised scenarios, which we consider as the future work.

[a] Bayesian Invariant Risk Minimization. CVPR 2022.

W2: Extension to other tasks.

R: Following this suggestion, we evaluated VaMP's generalizability beyond image classification task by conducting experiments on open-vocabulary segmentation and action recognition tasks. We integrated VaMP into state-of-the-art frameworks of two tasks: CAT-Seg[a] for segmentation and FROSTER[b] for action recognition, utilizing CLIP ViT-B/16 as the backbone encoder. Both baselines are extensions built upon CLIP, allowing our method to be seamlessly integrated into these approaches.

For open-vocabulary segmentation, we followed CAT-Seg's protocol, training on COCO-Stuff (118K images, 171 categories) and evaluating across multiple benchmarks including ADE20K, PASCAL-Context, and PASCAL VOC with varying category scales (59-847 classes). The results are shown below in Table A1:

Table A1: Performance for open-vocabulary segmentation.

For open-vocabulary action recognition, we adopted FROSTER's base-to-novel evaluation protocol on two widely used video benchmarks, i.e, Kinetics-400 and UCF-101, where models trained exclusively on base classes are required to generalize to unseen novel action categories. The results are shown below in Table A2:

Table A2: Performance for open-vocabulary action recognition. B: Base classes, N: Novel classes, HM: Harmonic mean.

As shown, VaMP achieves consistent improvements across different tasks, demonstrating the scalability of our approach and its strong generalization capabilities.

[b] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation. CVPR 2024

[c] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition. ICLR 2024.

W3/Q1: Analysis on the trade-off between improved generalization and increased latency.

R: We appreciate the reviewer's suggestion and have conducted a more detailed quantitative analysis of the trade-off between inference cost and accuracy under different numbers of Monte Carlo samples S. As shown in Table A3, even with a modest number of samples (e.g., S=5 or S=10), VaMP achieves clear gains in harmonic mean accuracy over the strong MMRL baseline, with only marginal increases in inference time (e.g., 5.3ms/image → 6.1ms/image). Specifically, at S=10, VaMP improves HM from 81.20 to 82.37 with a latency increase of only 0.8ms per image—demonstrating an efficient balance between generalization and overhead. Notably, the performance saturates quickly with increasing S, suggesting that our method remains highly efficient even with limited sampling. This confirms that VaMP can be deployed with low overhead while still leveraging uncertainty modeling for improved generalization.

Table A3: Comparison of inference time

W4/Q4: Scalability to newer multi-modal models like GPT-4V.

R: Our method is architecture-agnostic and can be readily integrated into more complex VLMs, as it only requires access to intermediate transformer layers and the ability to inject prompt tokens. The variational adaptation module is lightweight (MLPs per layer) and does not assume any specific backbone structure. However, due to time and computational resource constraints during the rebuttal period, we were unable to conduct experiments on large-scale GPT-4V like MLLMs, such as QWen-VL and InternVL. GPT-4V is proprietary and we can not apply to it directly. However, technically, it is feasible. We will explore this afterwards and include it in our revision.

Despite these constraints, we have validated VaMP on several state-of-the-art VLMs that represent the latest advances in CLIP-style architectures, including EVA-CLIP [d], SigLIP [e] and SigLIP 2 [f]. These models share similar structural designs with CLIP, enabling rapid adaptation during the rebuttal period. As shown in Table A4, VaMP obtains clear and consistent performance improvements across all VLMs, demonstrating its strong generalization capability to unseen categories—a critical challenge in prompt learning. The consistent improvements across different VLMs further substantiate the effectiveness and generalizability of our approach to different architectures.

Table A4: Performance comparison of different VLMs on ViT-B/16 backbone under base-to-novel generalization setting

[d] EVA-CLIP: Improved Training Techniques for CLIP at Scale. Arxiv 2023.

[e] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.

[f] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. Arxiv 2025.

Q2: Put efficiency analysis in the main paper.

R: Thanks for the suggestion. We will move it to the main paper in revision.

Q3: Qualitative examples to illustrate the generated prompts' structure or diversity.

R: While we are unable to include image visualizations in the rebuttal due to format constraints, we will provide such examples in the revised manuscript. Specifically, we visualize the learned prompt distributions via t-SNE for different image categories. The results show clear separation across categories, suggesting that VaMP generates semantically distinct prompts conditioned on image content. This inter-class diversity is significantly more pronounced than that of deterministic prompt methods, validating the effectiveness of our uncertainty-aware prompt generation.