Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment

We thank Reviewer JaD2 for the review and constructive suggestions. Our response is detailed below.

1. Computational cost

Our method fine-tunes only the final layer of the Gemma-2B model. For training efficiency, we cache hidden states from the frozen LLM layers (L168-169). This dramatically reduces the feed-forward FLOPs from 144.2 to 24.8, which is only marginally higher than vanilla CLIP's 23.5 GFLOPs. To further address feasibility and our strategy's benefits, we experimented with a 128M-parameter LM from MaMMUT (Kuo et al. 2023). Our approach again yielded clear improvements, with comparable GFLOPs (24.4 without caching) to vanilla CLIP.

2. Generalizability to other LLM

We evaluated our approach using the LLaMA3 8B model as the text encoder (batch size 4096, unfrozen last layer). As shown below, our method showed consistent improvement over a single prompt baseline with the same LLaMA backbone.

We thank Reviewer TY9F for the constructive feedback and suggestions.

1. Comparison with [a] DeFo and [b] VDT-Adapter

We will include the discussion on [a] and [b] in the final version. While DeFo [a] and VDT-Adapter [b] use decomposed representations or LLMs, their methods, task formulation, and applicability fundamentally differ from our approach to general image-text retrieval.

DeFo [a] is designed for fixed-category classification and requires fine-tuning. It learns latent textual embeddings and a classification-specific linear layer, rather than handling natural language inputs. This constrains DeFo to predefined classes and makes it unsuitable for zero-shot retrieval. Inspired by DeFo's M-token-per-query concept (M=16 in DeFo) and the reviewer's suggestion, we tested a baseline appending 16 [APT] tokens to the [input_text] (i.e., "[input_text] [APT-1]...[APT-16]"). Our structured prompt method (single [APT-i] per prompt in the template "the [APT-i] of the image means:") performs better.

VDT-Adapter [b] uses a large external LLM (GPT-4) with manual, dataset-specific heuristic prompts (e.g. for FGVC Aircraft dataset, using attributes like "manufacturer" or "engine count") to augment class names for classification, and is not evaluated on retrieval. Applying this to retrieval by augmenting every query with GPT-4 (trillion-parameter scale) would incur significant cost. More importantly, it is unclear how to design general-purpose prompts for retrieval. Although re-captioning the entire LAION pretraining dataset is beyond the scope of this rebuttal and is an orthogonal research area, we augmented Flickr/MSCOCO test queries using an external LLM with a prompt ("Describe the visual aspect in detail"). This did not help the zero-shot retrieval.

2. Comparison with CLIP with Gemma text encoder

To isolate our multi-prompt strategy's benefits from using a larger LLM (Gemma-2B), we note that our Table 1 already evaluates a 'LLM-unfrozen last layer' setup. This uses Gemma-2B by fine-tuning only its final layer (PEFT) and includes baselines with basic last-token and simple prompt-based last-token pooling. We further experimented with standard mean pooling for this Gemma encoder. The results show our method's gains stem from our multi-prompt approach, not just the larger LLM:

3. Additional inference-time usage

Interpretability: Our K individual prompt embeddings and their visual attention patterns (Fig. 2) provide granular understanding of text-visual alignment.

Aspect re-weighted inference: The K distinct [APT] embeddings allow inference-time adjustment. The similarity score $S = \sum_{k=1}^{K} (p_k \cdot q_k)$ (where $p_k$ and $q_k$ are k-th segments of normalized text/image embeddings), can be modulated by weights $w_k$ to $S_{weighted} = \sum_{k=1}^{K} w_k (p_k \cdot q_k)$. Inspired by Figure 2 where different [APT-i] tokens show varied focuses (salient objects, secondary elements, or background), we tested this on ImageNet zero-shot classification (batch size 4096). Up-weighting the first segment ([APT-1], presumed subject focus) with $w_1$=1.25 and slightly reducing others improved the top-1 accuracy from 61.4% to 61.8%.

Dear Reviewer TY9F, We are checking in to see if our rebuttal addresses your questions. We are happy to address any others you may have. Thank you.

We appreciate Reviewer L9Jh for their feedback. Our response is detailed below.

1. Comparison with SigLIP/SigLIP2

While SigLIP and the very recent SigLIP 2 show strong performance, SigLIP's core method (replacing softmax with a sigmoid contrastive loss) and SigLIP 2's additional techniques (e.g., decoder-based pretraining, self-supervised losses, data curation) are largely orthogonal and complementary to our main contribution: a multi-prompt embedding strategy.

Direct comparisons are challenging due to significant differences in training regimes. SigLIP models leverage the much larger, proprietary 10B-scale WebLI dataset (which is not publicly available), larger batch sizes (32k) and longer training (1,250k iterations), compared to our LAION experiments in Table 3 (16k batch size, 500k iterations). To isolate our strategy's benefits in a SigLIP-aligned setting, we conducted new experiments using a sigmoid loss with a 4096 batch size. Our multi-prompt method shows consistent gains in this sigmoid loss setting as well, indicating its benefits are independent of the specific contrastive loss function (softmax vs. sigmoid).

2. Motivation for image-text setting and text-only application

To explore the reviewer's point that our approach could benefit text-only settings, we ran a preliminary experiment on the Semantic Textual Similarity benchmark (STS-B) with finetuning. This initial experiment shows encouraging results for our multi-prompt method over the baseline.

Our primary motivation for the vision-language (VL) framework lies in its advantages for our adaptive prompt tokens. VL contrastive learning allows these tokens to be specialized by optimizing for alignment with visual content, enabling them to capture semantic aspects highly relevant for visual matching. This specialization is demonstrated by our attention visualizations in Figure 2, where different [APT-i] tokens guide the visual encoder to focus on distinct image regions.

3. Prompt-wise attention masking vs. Batching with prefix caching

Both our prompt-wise attention masking and prefix caching with batching efficiently avoid recomputing KV states of the shared [input_text] prefix. While the efficiency of batched prefix caching can depend on system-level optimizations for managing and loading KV caches for batched items, the core computational work is likely similar. Our prompt-wise masking provides a direct implementation, processing K prompts in a single forward pass, and is not reliant on specific device support (e.g. those with limited memory for caching) or framework optimizations for cache handling.