LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content

We would like to thank the reviewer for the good and important comments that helped us improve the work significantly. LiveXiv is a live dataset. Thus, during the review period we published in huggingface version 2 anonymously, consisting of 18K samples. We introduced 4 new domains from physics (namely, physics.optics, physics.bio-ph, physics.app-ph, physics.data-an), and added two more LMMs (Pixtral and Molmo-7B). We are now building version 3, which will have multiple generation and filtering models (GPT4o, Claude-Sonnet, Qwen2-VL-72B) to reduce potential biases and increase diversity. We expect to have this version published with all the analyses before the end of the discussion period. In addition, to perform deeper analysis, we created a variant of the first version of the dataset (that was reported in the paper) with opposite roles (Claude as the model that generates the QAs and GPT as the filter model), and validated that the overall performance and ranking remains the same. Lastly, we created a LiveXiv version (denoted as v0) from papers more than 10 years ago. Our experiments on it reassure our findings reported in the paper about the performance of our efficient evaluation showing that it is also valid in the presence of a large time gap between the dataset versions.

The paper's benchmark scope appears limited, with potential for expansion in sample size and domain diversity. LiveXiv is a flexible method that supports various input sources. For version 2 we extended our domains to be 14 (adding 4 additional domains from physics). LiveXiv can accept any arbitrary source of pdf files that we can increase and adapt the diversity as needed. At each version LiveXiv has over 16K questions generated from 4-5.5K images which is on par with all well established scientific datasets: ChartQA dataset - 33K questions (for all sets, while the test-set has only 3K questions). DocVQA dataset - 5K images and 50K images. AI2D dataset - 5K images and 4.5K questions. Overall after two versions of LiveXiv our dataset is larger than the leading scientific multi-modal benchmarks.

The study's narrow focus on Knowledge-VQA tasks limits its broader applicability in addressing data contamination across different domains.

While our proposed benchmark reduces contamination, we are focused on efficient generation and evaluation. We are motivated by using a rapidly and unified growing and adapting data source that can hint at contamination issues. Indeed, we demonstrate our approach on VQA, yet, our approach can be extended beyond it."

The reliance on proprietary LMMs for the benchmark's operation introduces potential variability and a lack of control over the evaluation process

We analyze the diversity of our dataset by the textual content in Section 4.3, and Tables 8,9. Our analysis shows that LiveXiv is diverse in terms of the different questions categories for both tasks. In addition, we will add the difficulty distribution of the question to show how LiveXiv is indeed a challenging dataset for LMMs. We also add a comparison to the first version where we switched the rules and now GPT4o is the filtering model and analyzes the effect of each rule. We can see that overall the relative ranking and performance remains similar. Moreover, we created v0 from papers 10 years ago, and reassured our evaluation is steady across time where the efficient evaluation estimates accurately (correlation 0.97 on average and MAE 2.7pp)

The efficient evaluation method shows promise, its effectiveness and reliability over time and across different versions of the benchmark need further validation.

We demonstrate the robustness of our efficient evaluation method in several scenarios. In the paper we demonstrated the effectiveness of our efficient evaluation method by splitting LiveXiv first version. We already verify the accuracy of the estimation of predicting models in version 2 from version 1, even on new domains introduced in LiveXiv version 2 (we will update Figure 4). We further stress our method by generating version 0 - a version of LiveXiv based on papers from 2010. We also find that our method is robust to predict the v1 from v0 (correlation of 0.97 and MAE of 2.7pp on average). In addition, we also converted the existing multi-modal live bench presented in [1] into a closed set of QA and demonstrated that our method can be applied to other live datasets as well See Section A.3.2 in the appendix for more details.

[1] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models, Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and others, arXiv preprint arXiv:2407.12772, 2024.

If we have adequately addressed all your concerns, we kindly ask you to reconsider your rating. If there are any additional questions or issues, we would be happy to provide further clarification.

We would like to thank the reviewer for the good and important comments that helped us improve the work significantly. LiveXiv is a live dataset. Thus, during the review period we published in huggingface version 2 anonymously, consisting of 18K samples. We introduced 4 new domains from physics (namely, physics.optics, physics.bio-ph, physics.app-ph, physics.data-an), and added two more LMMs (Pixtral and Molmo-7B). We are now building version 3, which will have multiple generation and filtering models (GPT4o, Claude-Sonnet, Qwen2-VL-72B) to reduce potential biases and increase diversity. We expect to have this version published with all the analyses before the end of the discussion period. In addition, to perform deeper analysis, we created a variant of the first version of the dataset (that was reported in the paper) with opposite roles (Claude as the model that generates the QAs and GPT as the filter model), and validated that the overall performance and ranking remains the same. Lastly, we created a LiveXiv version (denoted as v0) from papers more than 10 years ago. Our experiments on it reassure our findings reported in the paper about the performance of our efficient evaluation showing that it is also valid in the presence of a large time gap between the dataset versions.

The work's limitation is relying solely on table and figure captions for dataset creation.

Our generation process consists of two phases, as explained in Section 3.1 (lines 214-224). First we give the model the image and the caption and ask it to generate a detailed description of the image given the image and the caption. Then we call the model again and ask it to generate the questions based on the image and the detailed description. This way, we ensure the model focuses on the content of the image but it has much larger context for the question generation. For Tables, we have the data of the table so using different prompts strategies we force the model to create not just retrieval questions but also arithmetic questions over the table. See Figure 17 for more details.

Human evaluation to avoid hallucinations and errors.

We have performed a human validation study as described in Section 4.2 (lines 440-447). We collected 500 questions from each task (1000 overall) and measured the performance change between LiveXiv and manual curated subset. We saw the difference in performance is less than 2.5% on average. Tables 2, 6 and 7 summarize the results.

The efficiency and robustness of the efficient evaluation method We demonstrate the robustness of our efficient evaluation method in several scenarios. In the paper we demonstrated the effectiveness of our efficient evaluation method by splitting LiveXiv first version. We already verify the accuracy of the estimation of predicting models in version 2 from version 1, even on new domains introduced in LiveXiv version 2 (we will update Figure 4). We further stress our method by generating version 0 - a version of LiveXiv based on papers from 2010. We also find that our method is robust to predict the v1 from v0 (correlation of 0.97 and MAE of 2.7pp on average). In addition, we also converted the existing multi-modal live bench presented in [1] into a closed set of QA and demonstrated that our method can be applied to other live datasets as well See Section A.3.2 in the appendix for more details.

[1] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models, Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and others, arXiv preprint arXiv:2407.12772, 2024.

In Line 235, which LLM do you use here?

For the first two LiveXiv used LLaVA-1.6-34B without image tokens. From v3 and forward LiveXiv is using LLama-3.1-70B.

How do you select the test set in Table 1? Line 458, which 5 models are selected by the algorithm exactly?

For our efficient evaluation we used the following models according to the task: VQA: InstructBLIP-7B, LLaVA-OneVision-Qwen2-7B, Idefics2-8B, Claude-Sonnet, and Qwen2-VL. TQA: InstructBLIP-7B, InternVL2-8B, LLaVA-1.6-34B, Idefics2-8B, and Claude-Sonnet All the models were selected using the criterion introduced in Section 3.3.2 We added the list of models to the paper.

If we have adequately addressed all your concerns, we kindly ask you to reconsider your rating. If there are any additional questions or issues, we would be happy to provide further clarification.

We would like to thank the reviewer for the good and important comments that helped us improve the work significantly. LiveXiv is a live dataset. Thus, during the review period we published in huggingface version 2 anonymously, consisting of 18K samples. We introduced 4 new domains from physics (namely, physics.optics, physics.bio-ph, physics.app-ph, physics.data-an), and added two more LMMs (Pixtral and Molmo-7B). We are now building version 3, which will have multiple generation and filtering models (GPT4o, Claude-Sonnet, Qwen2-VL-72B) to reduce potential biases and increase diversity. We expect to have this version published with all the analyses before the end of the discussion period. In addition, to perform deeper analysis, we created a variant of the first version of the dataset (that was reported in the paper) with opposite roles (Claude as the model that generates the QAs and GPT as the filter model), and validated that the overall performance and ranking remains the same. Lastly, we created a LiveXiv version (denoted as v0) from papers more than 10 years ago. Our experiments on it reassure our findings reported in the paper about the performance of our efficient evaluation showing that it is also valid in the presence of a large time gap between the dataset versions.

The paper's reliance on Claude for question filtering and verification may introduce biases that favor Claude's architecture and methodologies.

Indeed, we mentioned in Section 4.2 (lines 407-411) that relying on Claude as our filtering model might introduce a potential bias. To show the 'impact’ of the bias, we created an opposite version of v1, where Claude is the model that generates the questions and GPT is the model that filters. We see in it that overall, the ranking remains similar and the changes are minor. Indeed, for the generating model there is some advantage as we see that GPT goes down a bit in the ranking (Claude remains first in both cases).

The authors should clarify their method's novelty by comparing it with existing benchmark generation approaches.

As far as we know, we are the first to create a live vision-language dataset at this scale. Both KIWI [1] and LiveBench [2] involve a human in the loop for the dataset generation. The fact they have a human in the loop, limits their scale dramatically into several hundreds of samples, thus making them smaller in an order of magnitude compared to each LiveXiv version. Second, our efficient evaluation strategy is novel and was not taken from other fields. Indeed, the IRT model has been used by other papers in the past but it is just part of our solution, which is very different from previous ones. For example, both tinyBenchmarks [3] and PromptEval [4] consider the static case in which no questions or models are added over time. This significantly differs from our scenario in which we need to select a few models to be re-evaluated and infer the performance of the rest on new batches of questions. Moreover, the dynamic nature of our problem makes our application more challenging compared to scenarios considered by previous works.

[1] KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions, Xu, Fangyuan and Lo, Kyle and Soldaini, Luca and Kuehl, Bailey and Choi, Eunsol and Wadden, David, arXiv preprint arXiv:2403.03866, 2024

[2] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models, Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and others, arXiv preprint arXiv:2407.12772, 2024.

[3] Maia Polo, Felipe, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. "tinyBenchmarks: evaluating LLMs with fewer examples." arXiv preprint arXiv:2402.14992 (2024).

[4] Maia Polo, Felipe, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. "Efficient multi-prompt evaluation of LLMs." arXiv preprint arXiv:2405.17202 (2024).

How to ensure diverse and evolving VQA benchmark generation, rather than generating repetitive questions that test a fixed capability across different timestamps.

We utilize LLM to classify our questions into categories, which will help to maintain the overall statistics of the categories. Thus, we can monitor and maintain the questions distribution to be diverse. Tables 8 and 9 show the overall performance according to the questions’ categories clusters. We can see in brackets at the top row that our dataset is diverse according to the content. For example, for VQA all categories have a similar amount of questions besides arithmetic reasoning which is less common in images. Yet, for TQA the dominant categories are Data-Analysis, Arithmetic and Reading while the other categories are less common since they do not relate to tables. We also added version 2 and we can see the diversity is kept both for the figure content and for the question categories:

How does the domain impact the evaluation scores? We visualize (Figures 5-8) the statistics of the performance of each model with respect to the domain. For TQA the domain doesn’t affect much, since our questions are limited solely to the table content. However for VQA (Figure 5), we can see clear trends of weak models to be worse across all methods while the newest models are performing well and equally across domains. In the middle, the variance in the performance is greater and suggests high sensitivity to the domain.

Could the authors show how models perform on different categories in different domains and see the correlation? We compute the average performance of the models as a function of the domain and question category. However, we did not see any clear trend between the category of the question and the arxiv domain.

If we have adequately addressed all your concerns, we kindly ask you to reconsider your rating. If there are any additional questions or issues, we would be happy to provide further clarification.

We would like to thank the reviewer for the good and important comments that helped us improve the work significantly. LiveXiv is a live dataset. Thus, during the review period we published in huggingface version 2 anonymously, consisting of 18K samples. We introduced 4 new domains from physics (namely, physics.optics, physics.bio-ph, physics.app-ph, physics.data-an), and added two more LMMs (Pixtral and Molmo-7B). We are now building version 3, which will have multiple generation and filtering models (GPT4o, Claude-Sonnet, Qwen2-VL-72B) to reduce potential biases and increase diversity. We expect to have this version published with all the analyses before the end of the discussion period. In addition, to perform deeper analysis, we created a variant of the first version of the dataset (that was reported in the paper) with opposite roles (Claude as the model that generates the QAs and GPT as the filter model), and validated that the overall performance and ranking remains the same. Lastly, we created a LiveXiv version (denoted as v0) from papers more than 10 years ago. Our experiments on it reassure our findings reported in the paper about the performance of our efficient evaluation showing that it is also valid in the presence of a large time gap between the dataset versions.

All questions are generated by gpt-4o, which may introduce issues such as the lack of diversity in questions.

We analyze the diversity of our dataset by the textual content in Section 4.3, and Tables 8,9. Our analysis shows that LiveXiv is diverse in terms of the different questions categories for both tasks. First, we will add the difficulty distribution of the question to show how LiveXiv is indeed a challenging dataset for LMMs. Second, we will also add a comparison to the first version where we switched the rules and now GPT4o is the filtering model and analyzes the effect of each rule. Lastly, version 3 will be composed of 3 participating models, where each time one of them acts as the question generation model and a different model is used for filtering. The data is divided into subsets and we will analyze the effect of each model pair (generation and filtering models) and how it reduces the potential bias of the single model for filtering and generation.

The authors conducted human studies for question filtering but did not validate whether model rankings align with human perception or established benchmarks.

In Section 4.2 (lines 440-447) We describe our human validation study. We performed a validation study over 1000 examples overall (500 for VQA and 500 for TQA). The study showed that the performance change is less than 2.5% on average. Moreover, the relative ranking of the top models remains steady. Tables 6 and 7 describe the performance change for each task in detail. We will add the ranking to tables 6 and 7. Additionally, we compared LiveXiv to 3 relatively similar well known benchmarks, namely ChartQA, DocVQA and AI2D. We compared the relative ranking between those datasets and LiveXiv and we confirmed that overall the ranking remains similar, which hints our model rankings align with human perception.

The data curation relies on proprietary models, which makes the benchmark prone to low reproducibility.

To ensure the maximal reproducibility, each version of our data is publicly available on HuggingFace and our evaluation is based on the open-source evaluation framework “lmms-eval” (https://github.com/EvolvingLMMs-Lab/lmms-eval ), we will also add LiveXiv as an evaluation dataset in "lmms-eval", which will make the evaluations reproducible as possible.

[1] KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions [2] “LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models”

[3] MATHEW, Minesh; KARATZAS, Dimosthenis; JAWAHAR, C. V. Docvqa: A dataset for vqa on document images. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021. p. 2200-2209.‏

4] HENDRYCKS, Dan, et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

Why the GPT-4o performs very badly in Table 1 while it is the model to generate the questions and answers? GPT4o’s low performance is indeed surprising. During the evaluation we tried to explore the root cause by experimenting with various hyper-parameters such as temperature, optimizing the resolution of the visual content to the model and exploring several prompts, but in all of our experiments the results of GPT4o remained similar.