Visual Question Answering with Fine-grained Knowledge Unit RAG and Multimodal LLMs

1-The writting can be improved: especially Sec.1. Current Sec. 1 is more like Related work instead of Introduction. The method is easy to understand, but the writing in Sec.3 makes me confusing.

2-Insufficient experiments: only conduct exp with GPT-4o. Should combine proposed KU-RAG with both open-sourced and proprietary MLLMs, and at least 3-5 different MLLMs.

3-The method lacks novelty: this method is more like a combination of multiple engineering techniques. Extracting from KB is widely used in KVQA tasks, knowledge correction chain is CoT.

1. The details of the constructed knowledge base are not provided (e.g., knowledge source, scale, length, etc.).
2. Experiments are limited on GPT-4o, experiments on more open-source MLLMs (e.g., LLaVA) should be included.
3. To validate the effectiveness of the proposed knowledge base, experiments should be conducted on the comparison of the proposed knowledge base with other knowledge bases.
4. The proposed framework is similar to the traditional multimodal RAG [1], which leads to low novelty.

[1] Lin W, Chen J, Mei J, et al. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering[J]. Advances in Neural Information Processing Systems, 2023, 36: 22820-22840.

• Unclear Contribution of Knowledge Units: The knowledge units represent a straightforward implementation for organizing the existing knowledge base with both images and text. While I understand that the knowledge unit serves as the foundation for the proposed RAG framework, the technical contribution in this area is somewhat limited.
• Necessity of KCC: The Knowledge Correction Component (KCC) is designed to replace incorrect retrieval-augmented answers with directly generated answers from the inherent knowledge of MLLMs. However, the primary purpose of RAG is to equip MLLMs with external knowledge. If I understand correctly, KCC merely remedies poor retrieval results and relies on the MLLM's ability to distinguish incorrect results. If the authors could demonstrate the effectiveness of KCC in weaker MLLMs, such as LLaVA1.5[1], its application would be more practical.
• Lack of Baselines: My major concern in the experiments section is the lack of competitive baselines. Although the authors compare with trained and text-retrieval baselines, no baseline is presented for multimodal RAG. The zero-shot GPT-4o baseline already outperforms the "SOTA (trained)" methods in Table 2. If the authors could provide comparisons with the following RAG strategies:

1. CLIP image-to-image retrieval (EchoSight [2])
2. CLIP text-to-image retrieval (InfoSeek [3])

the experimental results would be more persuasive.

• Limited Choice of MLLMs: The authors only conduct experiments with the powerful GPT-4o, raising concerns that the proposed methods may only work with costly, large MLLMs. Could the authors provide results using popular open-source MLLMs such as LLaVA1.5 or LLaVA-Next? References: [1] Improved Baselines with Visual Instruction Tuning [2] EchoSight: Advancing Visual-Language Models with Wiki Knowledge [3]Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

1. Some phrases like "Traditional wisdom" flags the potential usage of LLMs in composing the draft.
2. The design of KCC is still a naive experimental attempt which may not cast a positive impact on real knowledge-based vqa tasks. Despite the promising benchmarking results, the nature of KCC means that error propagation and knowledge conflicts are inevitable when applied to real-world vqa cases.