Visual Abstraction: A Plug-and-Play Approach for Text-Visual Retrieval

Q1:Concern of computational latency

We would like to clarify that our method is compatible with smaller, more efficient retrievers, and we have observed that models with significantly fewer parameters (e.g., 400M or 1.5B) still achieve comparable performance (see our response to Reviewer 1ri6, Q1).

In addition, the general description module can be executed entirely offline, prior to deployment. This design ensures that it incurs no cost during online inference. As noted in our response to Reviewer 1FJx (Q1), even when the QA component is omitted, VISA still delivers competitive performance relative to existing VLMs. In such case, only the text retriever operates in real-time, and its computational overhead can be adjusted by choosing a model of a suitable size.

Together, the use of lightweight retrievers and offline gallery processing make VISA practically deployable and efficient, even in resource-constrained settings.

Q2:Image-to-text (I2T) retrieval experiments

We initially did not include text-to-image (T2I) experiments because our primary focus is on supporting diverse user retrieval demands with varying levels of query granularity—a scenario more naturally aligned with T2I settings.

To address this concern, we conducted image-to-text (I2T) retrieval experiments on Flickr30K, MSR-VTT, and Urban1K, using the same retrieval setup and backbones as in the T2I setting. Specifically, for each query image, we first retrieve the top-20 candidate text descriptions. For each candidate, we generate three questions and append the corresponding answers derived from the image. Finally, we re-rank the candidates using text-level retrieval.

These consistent improvements across datasets and backbones confirm that VISA not only enhances T2I retrieval, but also generalizes effectively to I2T retrieval.

Q3:Compared with multimodal retrieval methods that use LMMs

As there are only a few works directly built on LMMs, we have compared our method with RAGVL in Table 1. On Flickr30k, our approach achieves higher performance with R@1 = 85.1 compared to RAGVL’s 84.4 and further improves to 86.1 when using EVA-CLIP. The plug-and-play design does not require end-to-end fine-tuning of large models.

Q1:Integrating VLMs

Thank you for this insightful suggestion. To explore this idea, we integrate SigLIP (as the base model) with EVA-CLIP (18B parameters), which has a significantly higher parameter count compared to the text retrieval model gemma-9B. This hybrid setup yields notable performance gains over SigLIP alone:

Notably, our proposed method achieves even greater improvements on both datasets. This suggests that generative visual abstraction offers more effective enhancement than simply fusing large VLMs. Importantly, VISA maintains its plug-and-play design, requiring no model retraining or architecture modification. Additionally, as detailed in our response to Reviewer 1ri6 (Q1), our method is compatible with smaller, more efficient text retrievers (e.g., 400M and 1.5B parameters), which also achieves strong performance.

Q2:The explanation of 'No' and 'Uncertain' responses

Our core motivation is to abstract visual content into textual descriptions, enabling more effective reasoning and relationship modeling in the text space. In the question-answering module, we distinguish between two types of negative responses: "No" and "Uncertain", each serving a different purpose:

• "No" is used when the question refers to a subject that is not present in the image, or when a specific condition, action, or attribute does not apply. For example, if the image shows a person in red shirt playing basketball and the question asks, "Is the person in red shirt playing football?", the correct response would be "No". This type of response provides explicit negative evidence which helps refine semantic alignment.
• "Uncertain" is used when the question cannot be reliably answered based on the visual content, e.g., asking about the shirt color of someone playing football when no such activity is depicted. In these cases, the response is considered uninformative, and the corresponding QA pair is discarded from downstream processing.

Q3:How VISA manages ambiguous queries where multiple visual interpretations are possible.

VISA incorporates LLMs to manage query ambiguities by leveraging context to disambiguate meanings. For example, in a query like "How to do something in windows?" (as shown in Figure 5 of the main paper), the surrounding context helps the model infer that "windows" refers to the operating system Windows instead of glass windows. In the future, we plan to further enhance this capability with reflection mechanism to check the visual interpretations.

Q1:Efficiency Concerns

Thanks for the suggestion regarding the efficiency of using large text retrievers. Importantly, VISA does not require LLMs with over 7B parameters to work effectively. To demonstrate this, we conducted experiments using smaller text retrievers with 400M and 1.5B parameters, and the results are summarized below:

As shown, even relatively small models such as Stella-400M yield significant performance gains over the baseline without a text retriever. This demonstrates that the performance gains from VISA are not solely due to model size, but rather arise from the integration of visual abstraction into the retrieval pipeline. These findings reinforce the generalizability and flexibility of VISA. In scenarios where inference efficiency is a priority, smaller and more efficient models can be adopted without substantial performance loss. For additional discussion on latency and FLOPs, please refer to our response to Q1 of Reviewer 1FJx.

Q2:Impact of abstraction length

We explored how varying the length of the generated general descriptions affects retrieval performance. Specifically, we modified the prompt for the captioning model to: “Please generate descriptions of the given image in approximately {num} words.” The corresponding results are available at the anonymous URL: https://imgur.com/a/amTI9n5.

Our results show that longer visual captions generally improve retrieval performance, as they encode richer semantic information such as object attributes, spatial relationships, and contextual details. However, we also observe a clear performance saturation beyond a certain caption length. That is, once the caption adequately captures the essential visual content, adding more tokens results in diminishing returns. Importantly, the optimal caption length depends on the complexity of the query. For short-context queries (e.g., Flickr30K), moderately long captions are sufficient to achieve strong performance. In contrast, long-context queries(e.g., Urban1K) benefit from longer, more detailed captions that better capture fine-grained visual elements.

Q3:Using VISA on other cross-modal tasks

We believe that the visual abstraction mechanism is broadly applicable to other cross-modal tasks. For example:

• Video Moment Retrieval: By segmenting a video into temporal clips and generating visual abstractions for each segment, one can match a textual query to the semantic descriptions of these segments, thereby localizing relevant time intervals.

• Text-based Re-identification and Composed Image Retrieval (CIR): This can benefit from generating abstractions for both gallery items and query items (such as image queries in CIR) and comparing them in the language space.

We plan to include this discussion in the next version.

Q4:Image-to-text (I2T) retrieval

We initially did not include text-to-image (T2I) experiments because our primary focus is on supporting diverse user retrieval demands with varying levels of query granularity—a scenario more naturally aligned with T2I settings.

To address this concern, we conducted image-to-text (I2T) retrieval experiments on Flickr30K, MSR-VTT, and Urban1K, using the same retrieval setup and backbones as in the T2I setting. Specifically, for each query image, we first retrieve the top-20 candidate text descriptions. For each candidate, we generate three questions and append the corresponding answers derived from the image. Finally, we re-rank the candidates using text-level retrieval.

These consistent improvements across datasets and backbones confirm that VISA not only enhances T2I retrieval, but also generalizes effectively to I2T retrieval.

Q1: Efficiency Concerns (FLOPs and latency)

Thank you for the valuable suggestions. We include the FLOPs and latency comparisions on the Flickr dataset below. For clarity, we divide the retrieval process into two stages:

• Offline stage precomputes visual features (via VLM) and generates general descriptions (via VISA) for the gallery candidates.
• Online stage performs the text encoding (via VLM), QA-based description refinement and text-level re-ranking (both via VISA) during inference.

In the offline stage, generating general descriptions using large multimodal models (LMMs) indeed incurs high FLOPs (138,000 vs. 335 compared with SigLIP, approximately 400×). However, this process could be executed ahead of time and does not influence the real-time inference latency. In the online stage, inference overhead is significantly reduced through techniques like KV cache and parallel execution, supported by efficient frameworks such as SGLang. In practice, this results in only ~1 second of additional latency on an NVIDIA 4090 server compared to the original SigLIP pipeline.

We acknowledge that VISA introduces a trade-off between efficiency and performance, and it may not be ideal for strict low-latency applications. However, we would like to emphasize two key points:

• Test-Time Augmentation Paradigm. To our knowledge, VISA might be the first work to systematically explore test-time augmentation for VLM-based retrieval. This paradigm aligns with trends in LLM research (e.g., test-time scaling), where huge additional compute yields significant performance gains. Specifically, VISA achieves average R@1 improvements of +2.3% (video) and +6.7% (long-context image retrieval). We also include BLIP-2 comparisons below, showing VISA improves over BLIP-2 by +0.6% (COCO) and +0.5% (Flickr30K) in R@1.
• Flexible Trade-Offs. Even using only the General Description module (without QA) yields strong performance boosts of +2.5%, +1.3%, and +8.3% R@1 across three tasks (see Table 6(a)), offering a lighter-weight deployment option (0.13 second) when needed.

Q2: Comparison to BLIP-2

Thanks for highlighting this important baseline. We initially did not compare against BLIP-2 because our evaluation follows the widely adopted zero-shot retrieval protocol in recent VLM research. In contrast, BLIP-2 is finetuned on COCO and then evaluated on COCO and Flickr, making it less directly comparable to the evaluation setup in Table 1.

Here we include BLIP-2 in our evaluation (referencing Table 5 from the BLIP-2 paper). As shown below, VISA continues to yield improvements when applied on top of BLIP-2. This demonstrates the compatibility and effectiveness of our method even when applied to strong VLMs like BLIP-2.

More broadly, while BLIP-2 is highly competitive for short-query retrieval tasks such as COCO and Flickr30K, our method demonstrates notable improvements on long-text queries and video retrieval tasks as well, across 10 widely-used datasets. This highlights the generality of VISA as a plug-and-play enhancement applicable to a range of backbone models.