RoRA-VLM: Robust Retrieval-Augmented Vision Language Models

W1. Comparing two-stage retrieval with image+query composite retrieval

We appreciate the thoughtful comment from the reviewer and humbly argue that our two-stage retrieval method is more flexible and potentially has better performance and efficiency. The reasons are as follows.

During database construction, we only need to encode images as keys and their entity names as values. Once the main entity in an image is identified, we can simply combine the question with the entity name and search for relevant knowledge in a pure text database. In contrast, using an image+query embedding for searching requires constructing a search index that jointly represents both image content and knowledge, as seen in the knowledge base used in [1]. However, models capable of effectively encoding image+query pairs into embeddings are often not as powerful as models designed for generating embeddings within a single modality. Existing approaches often rely on ad hoc implementations, such as combining CLIP embeddings of images and text, which introduce many design questions and may result in suboptimal performance.

In comparison, our proposed two-stage method is modular and can seamlessly integrate with any state-of-the-art image and text retriever, ensuring adaptability and robustness.

To further demonstrate the advantages of our approach, we implemented two retrieval-augmented baselines that apply the multimodal composite retrieval, PreFLMR [2] and RA-CM3 [1], to empirically compare their performance with our proposed method. Both baselines were implemented using the same backbone models as our approach (Vicuna/LLaVA-1.5) to ensure a fair comparison. PreFLMR employs a multimodal retriever to retrieve fine-grained query-related textual contexts, while RA-CM3 encodes multimodal documents for mixed-modal retrieval.

The experimental results, presented in the table below, demonstrate that while composite retrieval approaches such as RA-CM3 can benefit from richer cross-modal representations, they may struggle to match the adaptability and robustness of our modular two-stage retrieval approach in knowledge-intensive complex reasoning tasks.

Table: Evaluation results in accuracy (%). The best performance is highlighted in bold.

W2. Confliction between visual token refinement and adversarial noise injection

We appreciate the reviewer’s feedback and would like to clarify that the motivations behind visual token refinement and adversarial noise injection are distinct but complementary, working toward the same overarching goal.

Adversarial noise injection during training is designed to help the model effectively leverage visual modality information by comparing retrieved entity images to the query image. This process enables the model to identify query-relevant retrieved documents while filtering out irrelevant retrieval passage noise. Its primary focus is on improving the model's robustness against noisy or irrelevant information in the retrieved documents.

On the other hand, visual token refinement is specifically tailored to support the adversarial noise injection training process. It filters out query-irrelevant content from the input images, ensuring that only query-relevant visual information is retained. By removing distracting elements such as background content or query-irrelevant entities, visual token refinement facilitates more accurate comparisons of entity-level content between images. This design ensures that the model is less influenced by extraneous visual information, improving its ability to focus on the content most relevant to the query.

Thus, while the two components address different aspects of the problem—visual token refinement focuses on image-level filtering, and adversarial noise injection focuses on retrieval-level robustness—they are aligned in their purpose of enhancing the model's ability to handle noise.

W3. Fairness of Experiment Comparison in Table 1

We are sorry for missing the training details of the baselines. To clarify, all the baselines in Table 1 are finetuned on OVEN, InfoSeek and Enc-VQA, respectively, so that we can ensure a fair comparison between our approach and all the baselines

We list the knowledge-intensive pretraining as one of our contributions since previous vision-language models were predominantly pretrained on image-caption datasets such as CC12M. Pretraining solely on image-caption pairs may not be sufficient to align the internal knowledge in large language models (LLMs), such as entity names and entity background knowledge, with the visual representations of entities in images. In Table 4, we perform ablation studies to show the effect of knowledge-intensive pretraining on both LLaVA-v1.5 and our approach: LLaVA-v1.5 v.s. LLaVA-v1.5 w/ WikiWeb2M, RORA-VLM v.s. RORA-VLM w/o WikiWeb2M. Our results indicate that pretraining on image and entity-rich captions can significantly improve VLMs’ performance on information-seeking tasks, highlighting an important direction for advancing VLMs in the future.

W4. Ablation study on hyper-parameters

We appreciate the reviewer’s comment regarding the impact of the parameters k, l, and m on model performance. Due to the time constraints of the rebuttal period, we were unable to conduct ablation studies for all these parameters. However, recognizing the importance of these analyses, we have already included an ablation study on the most critical hyperparameter, m (the number of retrieved knowledge snippets), in Appendix A2 of our submission. This study provides insights into how varying m affects model performance.

For the other suggested ablation studies, we acknowledge their value and will ensure they are included in the revised version of the paper if our submission is accepted.

References

[1] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Retrieval-augmented multimodal language modeling. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1659, 39755–39769.

[2] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5294–5316, Bangkok, Thailand. Association for Computational Linguistics.

I sincerely thank the authors for their detailed rebuttal, which has addressed most of my concerns. I decide to maintain my positive score. Here are some additional comments:

Regarding W1:

Thank you for the detailed explanation and additional experiments. Based on the authors’ response, I understand that multimodal composite retrieval excels in capturing multimodal semantics, while the entity-driven method shows superior performance in entity-centric, knowledge-intensive, complex reasoning tasks. Including explicit clarifications and comparisons in the paper would be beneficial to elucidate the trade-offs between these two approaches.

Regarding W2:

I appreciate the authors’ clarification. However, the statement that “visual token refinement is specifically tailored to support the adversarial noise injection training process” remains somewhat unclear. My understanding is that the two modules are not tightly coupled. Visual token refinement aims at denoising, while noise injection introduces adversarial noise for robust training. Using both during training might potentially interfere with the adversarial learning dynamics. It could be insightful to validate this hypothesis by employing noise injection during training and incorporating visual token refinement exclusively during inference. This additional experiment might help clarify their interplay.

W1. Needs a figure to show 2-stage retrieval

We appreciate the reviewer’s suggestion to include a figure illustrating the two-stage retrieval method in the Introduction section. We agree that a detailed visual representation would enhance the clarity and understanding of the proposed approach. Accordingly, we will add a detailed figure to illustrate the two-stage retrieval process in the revised version.

W2. Discussion of related studies and novelty of our method.

We provide the following discussion to highlight the distinctions and contributions of our method, which will also be included in the revised version of the draft.

Previous studies, such as [1], [2], and the more recent work [4], focus on multimodal retrieval methods that enhance text-only language models by retrieving textual knowledge using visual queries (e.g., an image) to assist in answering visual questions. These approaches primarily aim to improve retrieval quality to support downstream tasks better.

In contrast, our work focuses more on addressing the critical challenge of how to more effectively utilize the retrieved information in retrieval-augmented generation (RAG). While prior work [1][2] and recent models, such as RA-CM3 [5] and Wiki LLaVA, largely rely on the quality of the retrieved passages, they do not explicitly account for the inherent noise and irrelevance introduced by multimodal retrieval processes. Given that the recall@1 of state-of-the-art retrievers on datasets like InfoSeek is still below 0.2, the presence of noisy or irrelevant passages remains a significant limitation for RAG systems. Our method, RoRA-VLM, is the first to directly tackle this issue by proposing a robust solution to reduce retrieval-induced noise. Unlike previous approaches that rely exclusively on textual retrieved information, our framework fully utilizes multiple modalities of retrieved information during the generation process. By enabling the model to learn to distinguish relevant information from irrelevant noise within the retrieved materials, our approach significantly enhances the robustness of the RAG pipeline. This allows the vision-language models to maintain strong performance even when retrieval accuracy is imperfect.

[3] primarily focuses on visual representation learning, where noise is introduced at the embedding level of images and text. In contrast, our method focuses on injecting noises into the retrieved knowledge for generative models. The noise injection process and the corresponding learning paradigm in our approach are fundamentally different from those in [3].

W3. Correspondence between the ablation model variants in Table 2 and the proposed module.

The first row in Table 2 presents the performance of our RoRA-VLM model on InfoSeek. The second row, labeled as RoRA-VLM w/o VK-Refinement, evaluates the impact of our proposed visual token refinement strategy. Specifically, in RoRA-VLM w/o VK-Refinement, all configurations remain the same as in RoRA-VLM, except that the visual token refinement method is not applied to the input image tokens. Since the sequence length of LLaVA is limited to 2048, to accommodate four images and the textual context, we reduce the number of image tokens for each image by employing 2D average pooling, to reduce the number of input image tokens from 576 to 144 and match the number of visual tokens used in our VK-Refinement method.

In the third row, we assess the effectiveness of Adversarial Noise Injection for Robust Augmentation. To demonstrate that the performance improvements of our model come from its ability to distinguish relevant knowledge (associated with the entity in query image) from irrelevant knowledge or noise—and not simply from the availability of additional retrieved knowledge—we conduct an ablation study. In this ablation, we keep the same two-stage retrieval process but only provide the retrieved textual knowledge to the VLM during both training and inference, omitting the retrieved images entirely, so that the model relies solely on textual information to make predictions and cannot leverage visual information as evidence to validate the correctness of retrieval. This variant, labeled RoRA-VLM text-only RAG, ensures that the VLM processes the same textual knowledge as in the proposed approach but without the additional image input. As shown in Table 2, this results in a significant performance drop, which demonstrates that the improvements achieved by our adversarial training stem from the model's ability to effectively filter and focus on relevant knowledge rather than simply benefiting from the additional knowledge.

W4. Ablation of the two-stage retrieval

Following the reviewer’s suggestion, we conducted an additional ablation study to emphasize the effectiveness of our two-stage retrieval approach. Specifically, we performed an ablation experiment using only a single-stage retrieval method. In the single-stage configuration, we utilized the CLIP embedding of the query image to retrieve the most similar entity images in our retrieval database, and thereby obtain the corresponding entity names and background knowledge. This differs from our two-stage approach in that it bypasses the secondary textual retrieval phase, which normally uses the entity name and input query to refine the knowledge selection. Instead, the single-stage method directly employs the retrieved entity background contexts as knowledge snippets for retrieval-augmented generation. We compare this single-stage retrieval method with our proposed two-stage retrieval method in the table below. For a more comprehensive comparison, we also included RA-CM3 for comparison as it employed a single-stage retrieval method.

Table: Evaluation results in accuracy (%). The best performance is highlighted in bold.

From the results, it is evident that our two-stage retrieval method outperforms the single-stage approaches. A likely reason for this superiority is the flexibility and efficiency of our two-stage method. Specifically, during database construction, we only encode images as keys and their corresponding entity names as values. Once the main entity in an image is identified, we can combine the query with the entity name and efficiently search for relevant knowledge in a purely textual database.

In contrast, single-stage retrieval methods require constructing a search index that jointly represents both image content and knowledge, as seen in knowledge bases like [1]. However, models capable of effectively encoding image-query pairs into embeddings often underperform compared to models optimized for embedding generation within a single modality. Existing approaches typically rely on ad hoc implementations, such as combining CLIP embeddings of images and text. These methods introduce various design challenges and can lead to suboptimal performance.

In comparison, our proposed two-stage retrieval method is modular and can seamlessly integrate with state-of-the-art image and text retrievers, ensuring greater adaptability and robustness. We hope this study further demonstrates the advantages of our approach and addresses the reviewer’s concerns.

W5. Refine Figure 2 to include more details of the methodology

We appreciate the reviewer’s comments and suggestions. We agree that adding visual representations would greatly improve the clarity and understanding of our approach. In the revised version, we will include two more figures: one to illustrate the details of the two-stage retrieval process and another to depict the query-oriented visual token refinement process.

References

[1] Gui, Liangke, et al. "KAT: A Knowledge Augmented Transformer for Vision-and-Language." Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

[2] Lin, Weizhe, et al. "Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering." Advances in Neural Information Processing Systems 36 (2023): 22820-22840.

[3] Gan, Zhe, et al. "Large-scale adversarial training for vision-and-language representation learning." Advances in Neural Information Processing Systems 33 (2020): 6616-6628.

[4] PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers. Weizhe Lin, Jingbiao Mei, Jinghong Chen, Bill Byrne

[5] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Retrieval-augmented multimodal language modeling. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1659, 39755–39769.

W1. Shorten the description of Stage-1.

We appreciate the suggestion from the reviewer and admit that Stage-1 is quite similar to the KNN formulation illustrated by the reviewer. However, besides providing the details of the technical design, the discussion of Stage-1 is more about providing the setups of the multimodal-based retrieval augmentation process, such as the source of the retrieval augmentation (i.e., an image database built on 37.6M images from WIT) and the choice of image encoder (CLIP), which are different from relevant previous studies and necessary to help readers gain a better understanding of the problem setup. The following sections also refer to some variable names defined at this stage. Following the reviewer's suggestions, we will shorten the description and move some details to the appendix in the revised version.

W2. How to encode an image into a sequence of visual embeddings using CLIP? do you encode the image patches (n) to the same dimension? Do you feed-forward each patch, separately, to the CLIP model? Do you use the N internal CLIP features for each patch? If so, are you sure that their dimension is d, before the last projection layer? Do you project them with the last visual projection layer as the pooled [CLS] token projected? Please elaborate more on this procedure.

Below, we provide a detailed description of how we encode an image into a sequence of visual embeddings using CLIP, addressing each aspect of the reviewer's concerns.

Image Encoding with CLIP: In the CLIP model, the visual encoder is based on the Vision Transformer (ViT) architecture. Given an image, the visual encoder processes it as a whole and encodes it into a feature representation of shape [576, 1024]. This representation can be interpreted as 576 vectors, each with a dimensionality of 1024. The 576 vectors correspond to patches of the input image, where the image is internally divided into a grid of patches during the encoding process. This division is not explicit; rather, it is an inherent part of the ViT architecture, which computes patch-level embeddings directly through a convolutional embedding layer applied to the full image. The resulting intermediate patch embeddings collectively form the image’s representation in the model’s latent space.

Dimensionality of Visual Embeddings: After passing through the vision transformer (ViT) layers, each patch is represented as a feature vector with a dimensionality of 1024. To further process these features, we utilized the final visual projection layer of the original CLIP model. This projection layer, which is also used for the pooled [CLS] token in the original implementation, is applied to all 576 patch-based feature vectors in our approach. The projection reduces the dimensionality of each feature vector from 1024 to 768. To clarify further, the visual projection layer is part of CLIP’s original implementation. While it is typically applied only to the pooled [CLS] token to produce the image-level feature representation, in our work, we extend its application to all 576 patch-level feature vectors. As a result, the output is a feature representation of shape [576, 768], where 576 corresponds to the number of patches and 768 is the dimensionality of the projected patch embeddings.

After computing the patch embeddings, for each text query, we derive a 768-dimensional vector from the [CLS] token of the CLIP text encoder. We then compute the similarities between the text embedding and the image patch embeddings to select the top-m relevant patches, which are subsequently projected into the LLM's latent space using the LLaVA projector.