MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below.

Comment 1: Inefficient data creation pipeline

Thank you for your advice. In response to your concern, we have developed an automated data curation pipeline. The introduction of this efficient pipeline largely simplify the data annotation process. The data creation pipeline reduces the time of the steps of human browsing the news website and raising the questions, which consumes most of the time when we are collecting data of MMSearch. For future data evolution, all we have to do is to specify the time range for the collected questions and let the pipeline do the work. Only minimal human effort is required to guarantee the quality of the benchmark. We hope this automatic data creation pipeline could address your concern of the difficulty in keeping the benchmark and evaluation dynamic. Please refer to the general responses for our proposed automated methods to further expand the benchmark.

Comment 2: Benchmark size is limited

Thank you for your advice. Here are a few reasons why the current scale is enough:

1. The news domain in MMSearch covers all hot topics available on the mainstream news website like CNN or The Guardian. It includes most daily search scenarios with comprehensive categories of news.
2. Both the two domain data in MMSearch aim to be beyond the internal knowledge of LMMs. Different from other benchmarks, all the data in MMSearch truly examines the searching capability without any leakage.
3. Each instance includes four tasks: end-to-end, requery, rerank, and summarization. A model is required to be evaluated on each task with the task-specific annotated data. Therefore, there are 1,200 questions in total in MMSearch.

The current scale also allows for efficient evaluation. In the end-to-end setting, we need to interact with the search engine and also record the search results and screenshots. The evaluation of 300 questions now takes 5 to 8 hours for the end-to-end task to complete. A larger dataset may harm the development efficiency.

In addition, we have developed an automated data curation pipeline. During the rebuttal phase, we have collected 300 more data from the CNN and The Guardian websites following our designed pipeline. All these data are about news that occured between September 1, 2024 to November 14, 2024, ensuring no overlap with the current benchmark. We believe the automated pipeline could be employed to further expand the dataset at low cost.

Comment 3: Step-wise scores missing

Sorry for any confusion caused. We have included the step-wise scores for the proposed pipeline in Table 2 in the main paper during submission. We list the scores of the three steps: requery, rerank, and summarizaion. We have refined the table caption in Table 2 to make the content of the table more clear.

Comment 2: Lack of complexity hierarchy

Thank you for pointing out! We have implemented a three-tier complexity assessment framework based on the key challenges in the three steps of MMSearch-Engine:

1. Requery difficulty. This concerns the complexity of transforming the original question into an effective search query. Complex cases arise when the question references image content, requiring the LMM to first analyze the visual information and then synthesize it with the text question into a coherent search query. For instance, when a user asks about a landmark shown in an image, the LMM must first identify the landmark through image search and then incorporate this information into a text query about the landmark's specific attributes.
2. Rerank difficulty. This dimension evaluates the challenge of identifying and prioritizing relevant search results. The difficulty primarily scales with the information scarcity. If there are only very limited websites containing useful information, it is more difficult to successfully retrieve and choose the website.
3. Summarization difficulty. This aspect involves both information synthesis and multi-modal reasoning challenges. In cases requiring synthesis, answers cannot be derived from a single source sentence - the LMM must integrate information scattered across different parts of the website. For example, comparing event frequencies across locations (like concert counts between cities) requires gathering and analyzing distributed data. Additionally, some questions demand analysis of both textual and visual website content, sometimes necessitating comparison with input images with images in the website.

Based on these criteria, we have categorized all questions into three difficulty levels, with the following distribution: hard (28%), medium (27.7%), and easy (44.3%). We showcase the evaluation results grouped by the difficulty levels as below:

The result showcases that most LMMs still have trouble handling the hard questions compared with the easy questions. The complete version is in Table 4 in Appendix C. We have included the discussion in Appendix C in the revised version. These complexity annotations will be included in our benchmark release to facilitate future research on model capabilities across different difficulty levels.

Comment 1: Further expand the scale of the dataset

Thank you for your advice. Here are a few reasons why the current scale may be enough:

1. The news domain in MMSearch covers all hot topics available on the mainstream news website like CNN or The Guardian. It includes most daily search scenarios with comprehensive categories of news.
2. Both the two domain data in MMSearch aim to be beyond the internal knowledge of LMMs. Different from other benchmarks, all the data in MMSearch truly examines the searching capability without any leakage.
3. Each instance includes four tasks: end-to-end, requery, rerank, and summarization. A model is required to be evaluated on each task with the task-specific annotated data. Therefore, there are 1,200 questions in total in MMSearch.

The current scale also allows for efficient evaluation. In the end-to-end setting, we need to interact with the search engine and also record the search results and screenshots. The evaluation of 300 questions now takes 5 to 8 hours for the end-to-end task to complete. A larger dataset may harm the development efficiency.

In addition, we have developed an automated data curation pipeline. During the rebuttal phase, we have collected 300 more data from the CNN and The Guardian websites following our designed pipeline. All these data are about news that occured between September 1, 2024 to November 14, 2024, ensuring no overlap with the current benchmark. We believe the automated pipeline could be employed to further expand the dataset at low cost.

Please refer to the general responses for more details of our proposed automated data curation pipeline.

Thank you for the detailed response to my questions, it addressed most of my concerns. I have a further suggestion regarding the validation of the proposed datasets, particularly given the limited number of instances. Conducting robustness experiments, comparing the results with human evaluations, and performing significance testing could help strengthen the findings.

Thank you for your suggestion!

1. We conduct the robustness analysis comprising iterative experiments (n=8) utilizing LLaVA-OneVision-7B with a temperature parameter of 0.6 across three primary tasks: requery, rerank, and summarization. The results, visualized in Fig. 9 in Appendix E.2, demonstrate remarkable stability across all tasks, with minimal variance in performance metrics, suggesting the robustness of the evaluation results.

2. As for the significance testing, we evaluate the correlation between the human evaluation and the automatic evaluation metrics of the requery and summarization tasks. Since the rerank task can be evaluated quantitatively as a classification problem using human annotations as ground truth, we focused our human evaluation on the other two tasks. Three independent annotators assess the quality of the requery and summarization outputs using a score of {1, 2, 3, 4}. Then the score is normalized to a [0,1] range. We compute two correlation coefficients: Pearson's correlation coefficient ($r$) and Spearman's rank correlation coefficient ($\rho$) for the significance testing. The results are:

   Task	Pearson's r	Spearman's ρ
   Requery	0.5413	0.5503
   Summarization	0.8112	0.7944

   The analysis revealed substantial correlations between human and automatic evaluations across both tasks. All correlations were highly significant (p $\textless$ 0.001), providing strong evidence for the validity of our automatic evaluation metrics in aligning with human judgment.

Dear Reviewer mwVo,

We recognize that the timing of this discussion period may not align perfectly with your schedule, yet we would greatly value the opportunity to continue our dialogue before the deadline approaches.

Could you let us know if your concerns have been adequately addressed? If not, please feel free to raise them, and we are more than willing to provide further clarification; if you find that your concerns have been resolved, we would highly appreciate it if you could reconsider the review score.

We hope that we have resolved all your questions, but please let us know if there is anything more.

Thanks.

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below.

Comment 1: The future-proof issue of the benchmark

Thank you for your advice. In response to the concern, we have developed an automatic data curation pipeline. The pipeline automatically obtains latest news from mainstram news websites efficiently. As for the question and annotation, we employ a model pool containing state-of-the-art LMMs to raise the questions and provide the answers. To guarantee the quality of the generated data, we further conduct human verification. The high efficiency brought by the data generation pipeline enables us to update the benchmark in a low cost. Therefore, the regular update of the benchmark could prevent it from being obsolete by substituing the benchmark with recent news.

Please refer to the general responses and Appendix B for more details of our proposed data curation pipeline.

Comment 2: Indication of the sufficiency of the benchmark's coverage

Thank you for pointing this out. The categories of our benchmark are based on the categories of the authority news website like CNN [1] and The Guardian [2]. We assume the categories listed in these popular websites could cover most search scenarios.

Specifically, take CNN as example, the header categories of the website are listed as follow: US, World, Politics, Business, Health, Entertainment, Style, Travel, Sports, Science, Climate, Weather, and Wars. We omit politically sensitive categories (Politics, Wars) and related news from US and World sections to maintain neutrality. We merge Style, Travel, Health, Climate, and Weather into a single "General" category, as these topics typically involve less time-sensitive information. We preserve Business, Entertainment, Sports, and Science as standalone categories and add a "False Premise" category to test model robustness. The knowledge domain categories are derived from these news categories and validated by confirming that even state-of-the-art LMMs have limited prior knowledge in these areas, ensuring the necessity of external information retrieval.

Comment 3: The impact of corruption on the input image

This is a very interesting question. It helps to find out the role of image input in the search process. We consider four kinds of image corruptions: occlusion, gaussian blur, gaussian noise, and color jitter. The corruption is added to all the images. To better understand where the image input affects most, we conduct the experiments respectively on the requery, rerank, and summarization tasks. We test on InternVL2 for both low-resolution and any-resolution settings. The results are shown below:

The observations are as follows:

1. The requery and summarization step highly rely on the image input. When applying image corruption, most results are worse than clear image input.
2. The image input is not important for the rerank step, even bringing interference. The image corruption even brings large improvement to the low-resolution setting, which suggests InternVL2 does not leverage image information in the rerank step.
3. Any-resolution image input is more robust to image corruption than low-resolution image input. Compared to the low-resolution setting, the any-resolution setting suffers less performance deterioration.

[1] https://edition.cnn.com/

[2] https://www.theguardian.com/international

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below.

Comment 1: Small amount of data may cause randomness

Thank you for your advice. Here are a few reasons why the current scale may be enough:

1. The news domain in MMSearch covers all hot topics available on the mainstream news websites like CNN or The Guardian. It includes most daily search scenarios with comprehensive categories of news.
2. Both the two domain data in MMSearch aim to be beyond the internal knowledge of LMMs. Different from other benchmarks, all the data in MMSearch truly examines the searching capability without any leakage.
3. Each instance includes four tasks: end-to-end, requery, rerank, and summarization. A model is required to be evaluated on each task with the task-specific annotated data. Therefore, there are 1,200 questions in total in MMSearch.

The current scale also allows for efficient evaluation. In the end-to-end setting, we need to interact with the search engine and also record the search results and screenshots. The evaluation of 300 questions now takes 5 to 8 hours for the end-to-end task to complete. A larger dataset may harm the development efficiency.

In addition, we have developed an automated data curation pipeline. During the rebuttal phase, we have collected 300 more data from the CNN and The Guardian websites following our designed pipeline. All these data are about news that occured between September 1, 2024 to November 14, 2024, ensuring no overlap with the current benchmark. We believe the automated pipeline could be employed to further expand the dataset at low cost.

Please refer to the general responses for more details of our proposed automated data curation pipeline.

Comment 2: One requery operation may not demonstrate the LMMs' capabilities.

1. Sorry for the confusion caused. In the data annotation phase, we want to make sure at least one search result contains the ground truth answer to the question. So we require annotators to repeatedly adjust their requery to satisfy the condition. In contrast, in the evaluation phase, for fairness, each LMM is given only one chance to provide a good requery. This standardized approach allows for consistent evaluation metrics.

2. We acknowledge the valuable suggestion to explore multiple requery attempts. This could be another evaluation setting where we could better exploit LMM's capability. Following this suggestion, we conducted an additional experiment with LLaVA-OneVision (7B), where we generated 5 requery attempts per question (temperature = 1.0) and selected the best requery based on similarity to human-annotated queries. The result is provided below.

The result showcases that conducting requery for five times could indeed improve the end-to-end performance. This is a valuable insight for future development of a multimodal AI search engine pipeline, and we have included this discussion in our revised manuscript, highlighted in Appendix E.1.

Comment 3: The reasonableness of the score weight given the sequential nature of each step

Thanks for pointing out. We would like to explain the rationality of our weighting scheme from two complementary perspectives:

1. The importance of each task due to the sequential nature is already reflected in the end-to-end score. Detailedly, although it is easy to discern that the upstreamed task is more important, it is difficult to assign a precise weight to each of them. So we do not manually assign the weights but directly focus mainly on the end-to-end score, which implicitly considers their cascaded nature.
2. The individual task weights serve as complementary metrics rather than indicators of relative importance. Relying solely on end-to-end evaluation, while comprehensive, may obscure the performance characteristics of individual components and hinder targeted improvements. We, therefore, maintain independent evaluation of each task, with the weight distribution designed to balance the prominence of the end-to-end metric against the component-level assessments. This dual evaluation strategy enables both system-level optimization and component-specific refinements.

We have included this discussion in Appendix F in our revised version.

During the rebuttal phase, we have collected 300 more data from the CNN and The Guardian websites following our designed pipeline. All these data are about news that occurred between September 1, 2024 to November 14, 2024, ensuring no overlap with the current benchmark. Due to the limited time, we employ the LMM to conduct the question check step. Our manual inspection indicates the generated data meets acceptable quality standards. For the revised manuscript, we will conduct manual verification of all the newly generated end-to-end data and complete the requery, rerank and summarization data generation process.

We evaluate LLaVA-OneVision-7B, Qwen2-VL-7B, and GPT-4o on the newly collected data on the end-to-end task. The results are:

The trend of the model performance aligns with that reported in the main paper, which suggests the high quality of the generated data. The introduction of this automated pipeline could expand the dataset with much less human effort and greatly reduce the cost of data annotation. The future direction is to further refine the process and boost the data quality. We have added the content of this data curation pipeline in Appendix B in the revised version.