Training-Free Retrieval-Augmented Generation for Knowledge-Intensive Visual Question Answering

Limitations in the contributions. The paper combines existing approaches such as RAG, CLIP, and LLMs, without any added value. Furthermore, for the fine-grained matching step, the framework relies on CLIP which aggregates data into a single token. This limits the ability of captioning detailed and localized information which is crucial for knowledge-intensive VQA. Since the framework is training-free and plug-and-play, I would recommend trying another more fine-grained model of your choice and analyzing its effectiveness.

Lack of evaluation. The evaluation scope is limited to only 2 VQA datasets. Other datasets, such as OK-VQA are also suitable for a visual question-answering setting where external knowledge is needed. The paper argues that KI-VQA “adds additional complexity to the task” compared to OK-VQA, so naturally, I would expect that a model suited for KI-VQA will perform well on OK-VQA. Also, it would be interesting to see how it will deal with standard VQA datasets, such as VQAv2, compared to existing baselines.

Lack of visualizations and comprehensive presentation. The paper lacks more visualizations, plotting attention maps, and qualitative analysis to improve the overall presentation. For instance, presenting visual results, such as outputs with accompanying retrievals and entity verification outcomes, could offer a more intuitive understanding of how the model processes the complex questions.

1)The paper's main contribution is proposing a straightforward, effective knowledge retrieval method. However, the experiments do not convincingly validate its effectiveness. In Table 1, comparisons are unfair, as training-free methods lack additional knowledge inputs, while other works, such as [1] demonstrates that simple retrieval methods can achieve strong results without training MLLMs. The paper should compare with more multimodal RAG approaches that also do not require training. 2)The paper emphasizes the "plug-and-play" nature of the method, but only tests it with LLAMA3-8B, which limits the scope of its effectiveness. 3)The ablation study is insufficient. For example, the paper employs both General Image-Text Matching and Detail Verification methods but does not include an ablation to test these individually. 4)Minor errors: The variable "N" in line 191 should be "M", and "FRA" in Table 1 might refer to "KIRA." Formatting issues include mistake quotation marks in LaTeX, the title "E-VQA" in Table 1 should replace "EVQA," and lines 444 and 445 are too close, affecting readability. Figure 1 is also somewhat inaccurate, as it omits image-text retrieval information in the fine-grained matching. [1] Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

1. The integration of image relevance and question relevance is a common approach in knowledge-intensive VQA. The innovation here is incremental, mainly combining existing retrieval techniques.
2. As a training-free method, KIRA is tested only on LLaMA3-8B and a variant called Bunny-1.1, both based on LLaMA3. This limited scope raises questions about the broader applicability of KIRA to other model architectures.
3. Table 1 presents performance improvements, but comparisons are limited to vanilla models w/o RÅG and KIRA’s own results, without considering other retrieval-augmented generation methods. A more comprehensive baseline, including other RAG-based approaches, would provide a fairer assessment of KIRA’s performance.
4. The paper lacks references to recent advancements in RAG and RAG-based VQA, which could provide valuable context and positioning within the latest developments in the field.

1. The core mechanism is still built on similarity matching between inputs rather than true information comprehension. I would highly doubt the performance when the knowledge domain shifts or knowledge format changes, or on complex reasoning tasks that require deeper understanding.

Wiki articles varies a lot in terms of content and source across different entities, which inevitably introduces bias that cannot be addressed by the proposed pipeline. E.g. Q: Where is this bird found? Given a picture of a Boissonneaua. The question can be fairly easy once the correct entity is retrieved. But a less commonly seen bird with few images or with an info page written in a completely different format is queried, then the model would likely to fail in the first step and produce cascading error.

2. The underlying limitation of training-free is that the performance would highly rely on pre-trained base models. The potential for information gain and performance improvement is limited by these models' pre-trained capabilities, which may vary significantly across different model choices. Though fine-tuning is avoided, hyper-parameters like $\lambda$ still need to be learnt for different datasets.

3. I believe the framework is built on top of some strong assumptions which may limit broader usage of the framework beyond the two benchmark datasets. For example, the framework assumes a very strong semantic alignment between questions and knowledge text, as enforced by initial matching and ColBERT for relevance verification, which means the performance are likely optimized for certain knowledge formats. I appreciate the authors efforts in evaluating the framework on the benchmarks, but the observations and insights might be less valuable to researches working with a different knowledge base or other forms of vqa.

1. The KIRA approach presented in this paper appears more like an engineering solution (CLIP and ColBERT), primarily involving the stacking of existing technologies, and thus demonstrating limited novelty. It also lacks in-depth theoretical analysis.
2. Many existing RAG systems can be applied to different knowledge bases and tasks with simple modifications or prompt engineering, also exhibiting a degree of "plug-and-play" capability. Therefore, the claim in the paper of being the "first plug-and-play" system is debatable.
3. The method offers limited parameter tuning space. KIRA primarily relies on the performance of existing models, possessing few adjustable parameters of its own, mainly focused on the λ parameter in Equation (6). This restricts its adaptability to different datasets and tasks.
4. The paper doesn't thoroughly discuss how the coverage and quality of the knowledge base affect retrieval results. Furthermore, the KIRA framework shows a much larger performance improvement on Encyclopedic VQA than on InfoSeek, suggesting that the method's applicability might be limited to certain scenarios.