MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training

Thank you for your constructive comments. We will solve your problems point by point as follows:

W1: The authors mention that they integrate “Multimodal RAG” in the Abstract and Introduction sections, but it seems that the retrieved content is only limited to image modality, so it would be clearer to point out what type of modalities are considered in this work at the beginning.

A1: Figure 1 clearly introduces the motivation and theme of our work, illustrating that the retrieved content consists of best-match image-caption pairs (including both image and text modalities) to ensure robust generation. This highlights that our method explicitly accounts for the relevance between different modalities throughout the “Multimodal RAG” process.

W2: The title and abstract are highlighting RAG, but the main content of this manuscript focuses on re-ranking of MLLM, which is inconsistent.

A2: Reranking is a key component of the retrieval stage and represents the core contribution of our work. This emphasis is reflected in the main content, as well as prominently in the title, “MLLM is a Strong Reranker”, and the abstract. Additionally, our approach constitutes a comprehensive multimodal RAG method, comprising three stages: retrieval, reranking, and generation. Beyond optimizing the reranking stage, we have also introduced a novel method to enhance the generation stage. To maintain coherence, this narrative is consistently presented in the title and throughout the paper.

The corresponding experimental results for the generation stage are detailed in Table 3, Table 4, and Figure 5, with additional results provided in the Appendix. These results demonstrate the superiority of our proposed method in achieving accurate retrieval and robust generation.

W3: The motivation of using noise distort image is unclear.

W4: The authors resort to an existing method, i.e., VCD, for logit contrastive learning and propose to adopt the difference between logits as a visual correlation weight in the MLE loss, but fail to tell clearly why they do that.

A3&4: As stated in Lines 240-244, we analyze the interleaved challenge in real-world scenarios, which poses the difficulty of determining the optimal number of images to support robust generation. Since it is challenging for the retriever to perform 100% accuracy, irrelevant images included in input prompts would still disturb the generation of MLLM. Therefore, the model should learn to deal with these MNC (Multi-granularity Noisy Correspondence) situations, where retrieved images and captions might carry irrelevant or noisy information as defined in the Introduction. That is why we use noise-distorted images during fine-tuning MLLM for more robust generation.

Although both our method and VCD [1] use contrastive logit calculation for a more robust generation of MLLMs, there are fundamental differences in their implementation and motivation. Our approach employs contrastive logit calculation during training, rather than inference. VCD, by contrast, applies this calculation exclusively during inference and does not involve fine-tuning. Additionally, we introduce two types of noise during training: token-level noise and data-level noise (negatively sampled images). VCD only incorporates token-level noise during inference. By injecting noise at both levels during training, we leverage the $\Delta logits$ as visual correlation weights to reassign the loss for each token, guiding the model to focus on relevant visual elements. Importantly, inference in our method involves standard decoding, not contrastive decoding.

In Lines 245-265, we draw inspiration from VCD, where visual uncertainty amplifies language priors, and contrasting the logits obtained from enhanced language priors with the original logits better highlights visual relevance. However, our motivation extends beyond mitigating irrelevant factors from a single retrieved image to addressing those arising from multiple images. In contrast, VCD focuses on better attending to visual tokens within a single ground truth image. Thus, the motivations underlying our use of contrastive logits differ fundamentally from those of VCD.

Dear Reviewer e7m3,

We wish to express our sincere gratitude for your invaluable feedback! We kindly request that you review our responses and consider updating your assessments accordingly.

We believe that our explanations have thoroughly addressed your concerns. If you have any additional questions or require further clarification, please feel free to contact us. Your final evaluation and potential review update would be greatly appreciated and would significantly contribute to the refinement of our work.

Thank you for your dedication and thoughtful reviews. We look forward to your further response.

Best regards,

Paper#3410 Authors

Thank you for your insightful comments. We will solve your problems point by point as follows:

W1: Reranking with MLLM is actually very time consuming compared to clip, and MLLM without Captions-aware IT in the Table 1 doesn't gain any performance.

A1: We agree that directly using MLLM for inference reranking is indeed more time-costly compared to CLIP. However, the reranking process can be performed in a parallel way by scoring multiple images at the same time. We can also reduce the number of images needed to be reranked to decrease the cost of computation and time (in this paper, the number of reranked images per query is 20). Moreover, as mentioned in Appendix C, we can adopt engineering approaches such as FlashAttention[1], PagedAttention[2], and Prompt Cache[3] to pre-cache the attention calculations for repeated prompts or directly include the repeated prompts in the system prompt to reduce inference latency. We believe that with existing methods, it is feasible to achieve significant computational efficiency improvements for the proposed reranking method through engineering optimizations. However, this is not the focus of this paper. In the subsequent version, we will also consider using smaller-parameter MLLMs to further reduce this cost.

As shown in Table 3, using visual knowledge to assist MLLMs in knowledge-intensive tasks enables more grounded reasoning. Models without RAG perform significantly worse on benchmarks compared to models with RAG. Reranking is a crucial step for all RAG systems. For Multimodal RAG, providing MLLM with all top-k images (k=20 for WebQA) would inevitably introduce a large amount of irrelevant noise, which naturally hinders the model’s ability to accurately utilize information from ground truth images, leading to degraded reasoning performance. This highlights the importance of reranking, as failing to rerank often results in irrelevant images being directly retrieved. Therefore, in scenarios where only a few images are selected from retrieved images, improving accurate recall is crucial. As shown in Table 2, on MultimodalQA, our method achieves the same level of recall performance on Recall@1 as CLIP’s Recall@20 (Table 6), illustrating that the reranking step can bring a significant improvement in retrieval accuracy.

Table 1 compares the performance of models without fine-tuning to those fine-tuned using our proposed fine-tuning method for the reranking model. Since MLLMs are typically trained on general datasets, we used a small instruction-tuning dataset to induce the model’s discriminative ability to conduct the reranking task. The results demonstrate that our fine-tuned model can leverage its advanced semantic understanding and analytical capabilities to comprehend queries, making it a powerful reranker. More importantly, as shown in Figure 3, our method is both resource-efficient and highly generalizable, making it easy to apply in any reranking scenario.

W2: The interpretation and analysis of the experimental results are insufficient. For example, in Table 12, why is the effect of no effect ND better in a single image scenario.

A2: We found that when using LLaVA as a reranker with a natural threshold for single-image tasks, there is a high probability that only one image is retained after filtering the top-N images (N=2 for WebQA). In this scenario, our noise-injected data struggles to take effect as it aims to make MLLMs focus on the correct image from multiple input images. Thank you for pointing this out. In future versions, we will provide a more in-depth analysis and explanation of this result.

W3: The performance improvement seems to depend on Image Caption. I wonder if the author's method is limited if no image caption is provided or if the image caption is poor.

A3: As shown in the example in Figure 1, the captions of the ground truth and the distractors are relatively similar and do not provide additional detailed information (e.g., the details of "the carving on the front" from the query), while similar situations occur frequently in the datasets. However, our method is still able to correctly retrieve both ground truth images, demonstrating that captions and images play a synergistic role in retrieval, both being critically important. This is also why we use MLLM as a reranker, as it can better leverage information from both visual and textual modalities.

[1] Dao T, Fu D, Ermon S, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness[J]. Advances in Neural Information Processing Systems, 2022, 35: 16344-16359.

[2] Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with pagedattention[C]//Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.

[3] Gim I, Chen G, Lee S, et al. Prompt cache: Modular attention reuse for low-latency inference[J]. Proceedings of Machine Learning and Systems, 2024, 6: 325-338.

Thank you for your comments. We will solve your problems point by point as follows:

W1: The framework may lack substantial novelty. The RagVL approach appears to combine an MLLM ranker with elements from existing techniques, including Visual Contrastive Decoding ([1], for fine-grained noise injection) and Robust RAG Training ([2], for coarse-grained noise injection). Using LLMs/MLLMs as rankers (reward models) in Reinforcement Learning from Human Feedback (RLHF) is already a common approach.

A1: As mentioned by Reviewer X8zE in Strength 1:our method "Effectively Mitigating the Multi-Granularity Noisy Correspondence (MNC) Problem: By employing the knowledge-enhanced reranking module, the paper uses instruction tuning to equip MLLMs with stronger semantic understanding and reranking capabilities, enabling more precise selection of visual information relevant to the query. This approach significantly improves retrieval accuracy.". Unlike the reward model in RLHF, which ranks candidate answers generated by LLMs based on human preferences, our reranker is specifically designed for retrieval-augmented generation (RAG). While many studies employ LLMs as rankers in various tasks, such as LLM-as-Judge [6] and RankGPT [7], their primary innovation lies in ranking answers accurately. What's more, to the best of our knowledge, we are the first to utilize MLLMs as rerankers for multimodal RAG, allowing for effective semantic analysis and improved retrieval. Subsequent to our work, NVIDIA proposed MM-Embed[3], which finetunes MLLM as universal retrievers and rerankers to tackle diverse tasks, further highlighting the value and innovation of our contribution.

Although both our method and VCD [1] use contrastive logit calculation, there are fundamental differences in their implementation and motivation. Our approach employs contrastive logit calculation during fine-tuning, rather than inference. VCD, by contrast, applies this calculation exclusively during inference and does not involve fine-tuning. Additionally, we introduce two types of noise during training: token-level noise and data-level noise (negatively sampled images). VCD only incorporates token-level noise during inference. By injecting noise at both levels during training, we leverage the $\Delta logits$ as visual correlation weights to reassign the loss for each token, guiding the model to focus on relevant visual elements. Importantly, inference in our method involves standard decoding, not contrastive decoding.

In Lines 245-265, we draw inspiration from VCD, where visual uncertainty amplifies language priors, and contrasting the logits obtained from enhanced language priors with the original logits better highlights visual relevance. However, our motivation extends beyond mitigating irrelevant factors from a single retrieved image to addressing those arising from multiple images. In contrast, VCD focuses on better attending to visual tokens within a single ground truth image. Thus, the motivations underlying our use of contrastive logits differ fundamentally from those of VCD.

At a coarse-grained level, you noted similarities between our method and Surf proposed by Sun [2]. However, we would like to clarify that our work was publicly released in July 2024 and submitted to ICLR on October 11th, whereas the paper of Surf was first made public on September 21st, 2024. Thus, Surf should be regarded as a concurrent or subsequent work to ours. Furthermore, there are clear distinctions between our methods. Our approach involves negative sampling of the ground truth image, combining the sampled negatives with context for MLLMs during instruction tuning. During inference, our retriever selects Top-K candidates, which are then reranked into Top-2 (e.g., for WebQA) by the reranker. The noise-injected training enables our model to discern differences among candidates and identify relevant visual elements. Conversely, Sun’s Robust RAG Training emphasizes high-quality dataset construction, where positive and negative labels are pre-generated by MLLMs. During inference, their retriever directly passes Top-K candidates to the generator without reranking.

Thank you for your constructive comments. We will solve your problems point by point as follows:

W1: As presented in Table 2, the two approaches introduced in reranking thresholds show improvements in most metrics but exhibit instability.

Q2: The two methods introduced for reranking thresholds have shown improvements across most metrics but also exhibited instability. After incorporating the reranking thresholds, the model’s recall decreased. Please analyze the reasons behind this phenomenon.

A1: The proposed thresholds are designed to filter out irrelevant images, preventing the generator model from being misled by irrelevant image inputs and thereby improving precision. However, since the reranker cannot achieve 100% accuracy, some relevant images with low relevance scores may also be filtered out, resulting in a lower recall but more importantly a significantly higher precision.

We would like to highlight that retrieval accuracy is very important for Multimodal RAG as irrelevant images in the input context will lead to a performance decrease of MLLM during inference. Thus, we adopt two thresholds to filter out images with low confidence after reranking. Although this step might cause a slight decrease in Recall, the removal of irrelevant images brings a positive effect on MLLM reasoning.

W2: Although the introduction of the MLLM reranker resulted in some performance improvement, it requires a substantial amount of additional inference time. When using only CLIP to select the top-K, the time taken is 1.23 seconds, but incorporating the reranker adds an extra 5 to 6 seconds.

Q1: MLLM possesses strong capabilities, but using it as an additional reranker seems to introduce significant computational overhead. Please discuss scenarios where performance improvements might justify the extra computational cost.

A2: We agree that directly using MLLM for inference reranking is indeed more time-costly compared to CLIP. However, the reranking process can be performed in a parallel way by scoring multiple images at the same time. We can also reduce the number of images needed to be reranked to decrease the cost of computation and time (in this paper, the number of reranked images per query is 20). Moreover, as mentioned in Appendix C, we can adopt engineering approaches such as FlashAttention[1], PagedAttention[2], and Prompt Cache[3] to pre-cache the attention calculations for repeated prompts or directly include the repeated prompts in the system prompt to reduce inference latency. We believe that with existing methods, it is feasible to achieve significant computational efficiency improvements for the proposed reranking method through engineering optimizations. However, this is not the focus of this paper. In the subsequent version, we will also consider using smaller-parameter MLLMs to further reduce this cost.

As shown in Table 3, using visual knowledge to assist MLLMs in knowledge-intensive tasks enables more grounded reasoning. Models without RAG perform significantly worse on benchmarks compared to models with RAG. Reranking is a crucial step for all RAG systems. For Multimodal RAG, providing MLLM with all top-k images (k=20 for WebQA) would inevitably introduce a large amount of irrelevant noise, which naturally hinders the model’s ability to accurately utilize information from ground truth images, leading to degraded reasoning performance. This highlights the importance of reranking, as failing to rerank often results in irrelevant images being directly retrieved. Therefore, in scenarios where only a few images are selected from retrieved images, improving accurate recall is crucial. As shown in Table 2, on MultimodalQA, our method achieves the same level of recall performance on Recall@1 as CLIP’s Recall@20 (Table 6), illustrating that the reranking step can bring a significant improvement in retrieval accuracy and thus is necessary for Multimodal RAG.

[1] Dao T, Fu D, Ermon S, et al. Flashattention: Fast and memory-efficient exact attention with io-awareness[J]. Advances in Neural Information Processing Systems, 2022, 35: 16344-16359.

[2] Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with pagedattention[C]//Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.

[3] Gim I, Chen G, Lee S, et al. Prompt cache: Modular attention reuse for low-latency inference[J]. Proceedings of Machine Learning and Systems, 2024, 6: 325-338.