Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

1. There is a lack of efficiency analysis, especially concerning the average latency per query, number of LMM calls, or token consumption. While efficiency is often an issue with large models, an essential analysis should not be overlooked.
2. The evaluation has some limitations. The paper only tests on the LLaVA-Wild dataset, which contains only 60 queries. In practice, many VQA datasets involving external knowledge, such as A-OKVQA, could serve as more comprehensive evaluation benchmarks.
3. There is insufficient information regarding reproducibility. Sampling parameters, such as temperature and top-p during large model inference, are not provided. Additionally, prompts used with large models (e.g., in Eq. 4 and Eq. 7) are not specified.

Method

1. Complexity: The approach is overly complicated, essentially trading time for performance.
2. Correlated Formulations: How are these extracted? The function $F_{VLM}$ takes an image and two captions, so how can it output correlated formulations? Is $F_{VLM}$ specifically trained for this task?

Experiments

1. Insufficient Experiments: The proposed method is essentially retrieval-augmented generation (RAG), yet the authors barely discuss its relevance to existing RAG works. Furthermore, there are no experimental comparisons between the proposed method and state-of-the-art works, which is unacceptable in research. Given that New Bing (Copilot) is known as an excellent RAG system, it would be more convincing to compare with it.
2. Impact of Open-Set Detectors: How do open-set detectors affect the performance of the proposed method?
3. Generalization Across VLLMs: The proposed method appears to be training-free. Why were experiments only conducted on LLaVA 1.6? It is crucial to validate the generalization of this method across different VLLMs.
4. Open-Set Benchmark: What does the open-set benchmark look like? How is fairness ensured during subjective evaluation?
5. Overclaim: Why is the Chain of Search algorithm considered the answer to “How to search”? There are no quantitative results to support this claim. Additionally, the explanation for “By what to conclude” should be “How to conclude”. “What to conclude” focuses more on content selection.
6. Efficacy of the Proposed Method: The authors use many qualitative results to prove the efficacy of the method, which is not convincing. Sometimes, if VLLMs have never seen the objects before, the captions extracted from VLLMs make it difficult to retrieve results from the web.

Additional Comments

• Political Examples: Using examples related to politics is not advisable. Many VLLMs avoid discussing such topics, which may lead to unfair implications about model performance.
• Overall Assessment: In my view, this paper is not ready for publication due to numerous weaknesses. I vote to reject this paper and hope the authors can further improve their work.

1. The framework appears to be a typical application framework for LLMs that combines captioning, searching, and RAG. This seems to be a combination of A + B + C. The author needs to highlight their contributions.
2. Which search engine is being used? It doesn't seem to be mentioned in the paper.
3. The discussion of factuality, relevance, and supportiveness in the Open-Set Evaluation could be more detailed, as the current descriptions may not fully convey their specific meanings.
4. In the ablation study, providing quantitative results, such as in tabular form, would enhance the clarity and impact of the findings.
5. The experiments could benefit from a clearer explanation of the specific type of score being referenced, as this would help readers better understand the results.

1. The introduction of the experimental section is somewhat careless. For instance, Figure 5 is provided in this paper, but there is no corresponding explanation or discussion of its content within the main text.
2. This paper lacks a detailed description of the open-set dataset used in the evaluation.
3. In Figure 6, the proposed method significantly outperforms GPT-4 web, but there is no clear explanation in the method section that justifies this improvement. Additionally, the paper does not analyze or discuss this result.
4. The experiments conducted are not sufficient, and this paper lacks comparisons with more open-world methods.
5. This paper does not provide information on the computational efficiency or time consumption of the proposed framework, particularly the Chain of Search algorithm, which is noted for its complexity.