MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

Thank you for your thoughtful feedback and for recognizing the strengths of our work, including the large and diverse MMSCI dataset, the realistic and well-crafted tasks, and the pre-training experiment. We truly appreciate your insights. Below, please find our detailed responses to your concerns.

C1: The MMSCICAP (Scientific Figure Captioning) task seems flawed. Providing only the abstract rather than figure-specific sentences—or even the entire relevant section—may undermine the task’s grounding. Abstracts alone should suffice only for captioning teaser figures; otherwise, the lack of figure-paired text may contribute disproportionately to the task’s difficulty.

Response: Thank you for the suggestions. We also conducted experiments where the entire article was provided to the model, challenging its ability to read the full text and identify information relevant to the figure. Given the extensive length of the full article, we evaluated on the proprietary model with long-context capability. The results are provided below:

While the performance did improve when using the full article content compared to using only the abstract, we want to emphasize that including figure-related content from the article might introduce unfairness. Authors sometimes repeat or closely mirror figure captions within the main text, which could lead to inflated performance metrics that do not reflect genuine model understanding. Given this consideration, we primarily use abstract-based captioning, which we believe could ensure a fairer evaluation while still providing contextual information about the figure's topic and the article's subject matter.

C2: Multidisciplinarity is highlighted as a key feature, yet the experiments lack in-depth analysis on performance variation across disciplines. What factors contribute to these variations across different subjects? For example, why does Astronomy and Planetary Science yield high accuracy while disciplines with more data, like Materials Science, perform less well?

Response: Thank you for the valuable feedback. To gain deeper insights into the data across different disciplines, we engaged domain experts (PhDs in the relevant fields) to analyze our dataset. After reorganizing the original 72 categories into 18 scientific fields, we further prioritized 10 major science domains for detailed analysis.

PhD Expert Evaluation: Specifically, we recruited 30 PhDs as human evaluators from the Prolific platform, which is a well established platform of high quality scores, with verified degrees in 10 major scientific categories that align with our dataset's primary domains: Material Science, Chemistry, Physics, Biochemistry, Environment, Climate Sciences, Earth Sciences, Biological Sciences, Biomedical Sciences, and Health and Medicine.

Our human evaluation protocol covered 10 broad scientific domains. For each domain, we selected 75 questions (25 from each of our three figure-caption matching tasks), yielding a total evaluation set of 750 questions. We recruited 30 PhD-level evaluators through Prolific, specifically 3 PhDs per domain, with verified degrees in their respective fields. Each evaluator provided two types of assessments:

• Performance Score: Evaluators answered the questions themselves, establishing a human performance baseline measured by accuracy (0-100%)
• Quality Assessment: Evaluators rated each question's quality on a 5-point scale (1-5), judging its clarity and whether it effectively tests domain-specific knowledge.

Response 2: I appreciate the comprehensiveness of your evaluation; however, the article does not delve into insights about multidisciplinary differences. While your human evaluation protocol is strong, the final scores alone are insufficient to provide a holistic understanding. Interpretations and discussions are equally important. For instance, why do Astronomy and Planetary Science achieve high accuracy, whereas fields with more extensive datasets, such as Materials Science, perform relatively poorly? Addressing such questions would significantly enhance the value of your evaluation.

We sincerely thank the reviewer for recognizing the comprehensiveness of our human evaluation protocol while raising important questions about interdisciplinary differences. Our analysis reveals notable performance variations across disciplines, particularly between Materials Science (showing performance in the 40-60% range) and Astronomy and Planetary Science, prompting a careful examination of the underlying factors.

Our analysis reveals three key factors contributing to this performance disparity:

Factor 1: The complexity of knowledge domain structure plays a crucial role

Using Web of Science classifications, we find that Materials Science encompasses six distinct specialized sub-fields:

Materials Science 1. Materials Science, Multidisciplinary 2. Materials Science, Ceramics 3. Materials Science, Coatings & Films 4. Materials Science, Composites 5. Materials Science, Characterization & Testing 6. Metallurgy & Metallurgical Engineering

In contrast, Astronomy and Planetary Science operates primarily within a single domain:

Astronomy and Planetary Science 1. Astronomy & Astrophysics

Methodology Note on Field Classification: Our analysis maps 250 Web of Science specialized subject categories to 18 first-level broad subjects, considering only primary relationships with high confidence. We employed a conservative mapping approach, excluding ambiguous cases and avoiding multiple mappings of the same subject. This ensures clear, well-defined relationships in our field complexity analysis.

This structural difference has profound implications for knowledge integration. Within Materials Science, even PhD-level experts naturally specialize in specific sub-areas, as mastering the entire breadth of the field is exceptionally challenging. For example, our dataset includes cases where understanding a single materials characterization figure requires expertise in both crystallography and mechanical properties sub-areas that typically represent distinct specializations even within Materials Science departments. This specialization reality means that the difficulty of question-answering tasks can vary significantly depending on how many distinct sub-areas of expertise are required to fully interpret a single figure or answer a question.

Factor 2: The visual content complexity varies significantly across fields. In Materials Science, figures frequently combine multiple specialized visualization techniques within single panels. For instance, in our dataset, we observe materials characterization figures that simultaneously present atomic-scale microscopy, spectroscopic data, and phase diagrams - each requiring distinct interpretation approaches. This multiplicity of visualization paradigms within single figures creates additional cognitive demands for both human experts and LVLMs.

Factor 3: The depth and breadth of knowledge integration required for accurate interpretation varies substantially. Our analysis shows that Materials Science questions often require simultaneous application of knowledge from multiple specialized domains. This multi-domain integration requirement is evidenced in our benchmark questions, where successful responses frequently depend on connecting concepts across different sub-fields of materials science.

Our fine-tuning experiments on the dataset demonstrate promising results in addressing these challenges. The LVLM models fine-tuned with our dataset MMSci achieved the highest overall multiple-choice accuracy on our benchmark, showing substantial improvements in handling multi-domain knowledge integration across both depth and breadth.

This analysis suggests that while performance variations across scientific fields stem from inherent differences in knowledge structure and integration requirements, they are not insurmountable limitations. Rather, they represent opportunities for targeted improvement in LVLM development, particularly in handling complex, multi-domain scientific content. Our findings indicate that with appropriate training approaches and data resources, LVLMs can enhance their capabilities in processing sophisticated scientific information, even in fields requiring complex multi-domain expertise.

Thank you for your prompt and constructive feedback. We address each of your points below:

Response 1: What I intended to convey is that the information available in the abstract alone is insufficient for specific and meaningful captioning. In other words, solving the problem using just the abstract is inherently impossible. While the full article introduces noise, it at least contains the necessary information to make the captioning task feasible, albeit challenging.

We appreciate the reviewer's insight regarding full article context. While abstracts provide limited information compared to full papers, our results with Qwen2-VL-7B-MMSci, trained on our comprehensive MMSCI dataset, demonstrate significant capabilities that merit careful consideration.

Our experimental results show that Qwen2-VL-7B-MMSci, using only abstract context, achieves performance comparable to proprietary models using full articles. Comparing with Gemini 1.5 (Flash&Pro)'s full-article performance:

• BLEU-4: 6.89 versus 6.94-7.01
• ROUGE-1: 32.62 versus 32.24-32.83
• ROUGE-L: 21.80 versus 19.32-22.02
• BERTScore: 84.33 versus 83.18-83.26

While we observe gaps in Meteor and G-Eval metrics, these differences primarily reflect semantic nuance rather than fundamental comprehension limitations. This performance validates our core contribution: developing vision-language models with deep scientific knowledge across multiple interactive domains through our carefully curated MMSsci dataset. Our approach embeds graduate-level understanding during training, fundamentally reducing the need for extensive manual curation of information for assistant systems.

The genuine knowledge capability demonstrated by our model proves particularly valuable for real-world research assistant systems.

• In practical applications such as scientific literature review, experimental data analysis, and cross-disciplinary research, approaches based on selecting manual full-PDFs and putting them into LLM often face significant constraints. These include context window limitations, processing overhead with complete documents, and retrieval performance challenges in RAG systems. Our approach, focusing on embedded knowledge through training rather than extensive context at inference time, offers a more efficient and lightweight alternative while maintaining competitive performance of applications including AI Agent systems too.
• This efficiency translates directly to real-world benefits. For instance, in interactive scientific question-answering scenarios, our model can provide accurate responses without the latency associated with processing full documents. Similarly, in multi-turn scientific discussions, the model can leverage its embedded knowledge to maintain context coherence without repeatedly accessing full papers.

While we acknowledge the potential benefits of full article context for long-context models, our results demonstrate that robust scientific context comprehension is achievable even with input of limited contextual information based on proper training on our dataset. This finding has significant implications for developing more efficient and deployable scientific AI assistants that can effectively serve graduate-level research needs across multiple disciplines.

We will include both abstract-based and full-article results in our updated paper to provide a comprehensive analysis, though our primary contribution lies in demonstrating how proper training with our dataset enables strong performance even with minimal context.

Table 1: Captioning results with different context, article absract or full article

C4.2: How well do proprietary models perform on this task? Are there any proprietary models that generate reasonable responses worth evaluating?

Response: We present the results of GPT-4o on this task using both 5-shot and 10-shot settings. We generated 10,000 samples via GPT-4o using the same prompt and setting as the other models in Table 5 in the main pages and Gruver et al. for evaluation. The results are shown below with the reported absolute percentage performances from the 10,000 generated sample cases.

As observed, GPT-4o, representing large proprietary LLMs, showed notable limitations in materials generation across validity, coverage, and stability metrics. Its performance was limited - achieving only 1.50% metastability in 5-shot and 4.72% metastability with 0.09% stability in 10-shot settings. In contrast, our LLaMA2-7B-MMSci model, pre-trained on MMSCI dataset, achieved significantly higher results with 64.5% metastability and 8.2% stability, highlighting the crucial role of domain-specific training data in enhancing LLMs for materials research.

C4.3: In Table 5, the “Stable DFT” column numbers for LLAMA models (from Gruver et. al.) appear to not be consistent with numbers reported in Gruver et. al. Why is that?

Response: We would like to clarify that we did not tweak anything but used absolute percentage performances from 10,000 generated samples from the output files in Gruver et al. to report their performances. We reran the experiments using their uploaded outputs, performed the DFT calculations with the same settings, and found the reported absolute percentages are almost identical to their reported performances if we reversely calculate from the stage-wise survival ratios from the previously survived candidates at the earlier steps in Gruver et al.

That is, we reported 2.1% Stability (DFT) for LLaMA2-7B, 4.9% Stability (DFT) for LLaMA2-13B, and 5.3% Stability (DFT) for LLaMA2-70B, and they are almost the same as, respectively:

• 2.1% (Absolute stable (DFT) percentage from 10,000) = 2.10% = 0.964 * 35.0% * 6.2%
• 4.9% (Absolute stable (DFT) percentage from 10,000) ~= 5.23% = 0.955 * 38.0% * 14.4%
• 5.3% (Absolute stable (DFT) percentage from 10,000) ~= 5.26% = 0.996 * 49.8% * 10.6%

Here are the reverse calculations from the reported numbers in Gruver et al.:

LLaMA2-7B in Gruver et al. For 2.1% (Absolute stable (DFT) percentage from 10,000 samples from Gruver et al.) = 2.10% = 0.964 * 35.0% * 6.2%, where:

• 0.964 is the stage-wise valid ratio reported in Gruver et al.
• 35.0% is the stage-wise meta-stable ratio for the previous valid step
• 6.2% is the stage-wise stable ratio for the previous meta-stable step

LLaMA2-13B in Gruver et al. For 4.9% (Absolute stable (DFT) percentage from 10,000 samples from Gruver et al.) ~= 5.23% = 0.955 * 38.0% * 14.4%, where:

• 0.955 is the stage-wise valid ratio reported in Gruver et al.
• 38.0% is the stage-wise meta-stable ratio for the previous valid step
• 14.4% is the stage-wise stable ratio for the previous meta-stable step

LLaMA2-70B in Gruver et al. For 5.3% (Absolute stable (DFT) percentage from 10,000 samples from Gruver et al.) ~= 5.26% = 0.996 * 49.8% * 10.6%, where:

• 0.996 is the stage-wise valid ratio reported in Gruver et al.
• 49.8% is the stage-wise meta-stable ratio for the previous valid step
• 10.6% is the stage-wise stable ratio for the previous meta-stable step

Thank you for your valuable feedback and for recognizing the strengths of our work, including the large and diverse dataset we introduce, the comprehensive evaluation we conduct, and the quality of the paper's writing. Please find our responses to your concerns below.

C1: The tasks are somewhat standard - Figure captioning, and matching figures/sub-figures to appropriate captions. It would have been nice to see some unique tasks created from this nice dataset showcasing the diversity of images/plots. e.g. some variety of interleaved image-text tasks such as Question Answering from images could have been considered.

Response: Thank you for your suggestion. Our dataset contains high-quality and diverse information, including articles, figures, references, and reviews, offering a robust foundation for advancing this field. In the initial efforts on leveraing our dataset, the tasks we present already expose significant limitations in existing multimodal language models (MLLMs).

By focusing on scientific figure comprehension with figure captioning along with bi-directional matching between captions and figures/sub-figures, we identify a critical gap in current MLLM capabilities and provide a targeted benchmark for evaluation and development. However, we acknowledge the potential to expand into additional tasks involving more complex interleaved image-text interactions. This work lays the foundation for such advancements, and we plan to explore these directions in the future.

C2: It would have been nicer to have more detailed analysis of model responses for a few images (3-5) in about 10 domains. Where experts weigh-in on model responses even for just the figure captioning task to evaluate the strengths and weaknesses of models in the different domains. Especially on the variety showcased in Figure-2, e.g. 10 examples from each category in figure-2 analyzed by domain experts.

Response: Thank you for the valuable feedback. We had indeed planned to involve domain experts (PhDs in corresponding fields) to analyze our dataset. After re-grouping the original 72 categories into 18 science fields, we prioritize 10 major science domains for further analysis.

PhD Expert Evaluation: Specifically, we recruited 30 PhDs as human evaluators from the Prolific platform, which is a well established platform of high quality scores, with verified degrees in 10 major scientific categories that align with our dataset's primary domains: Material Science, Chemistry, Physics, Biochemistry, Environment, Climate Sciences, Earth Sciences, Biological Sciences, Biomedical Sciences, and Health and Medicine.

A new sub-benchmark: We highly appreciate your suggestion on additional human evaluation on figure caption generation, but since there were questions on bi-directional matching between captions and figures/sub-figures, we constructed a new sub-benchmark with 10 major categories. For each category, we constructed a new sub-benchmark by selecting 25 questions per setting (75 total) from our three figure-caption matching tasks from the original test set. Thus, the new sub-benchmark consists of 750 questions total, and we have updated human evaluation scores with PhD degrees on 10 broad categories.

Specifically, we recruited 30 PhD-level evaluators through Prolific, specifically 3 PhDs per domain, with verified degrees in their respective fields. Each evaluator provided two types of assessments:

• Performance Score: Evaluators answered the questions themselves, establishing a human performance baseline measured by accuracy (0-100%)
• Quality Assessment: Evaluators rated each question's quality on a 5-point scale (1-5), judging its clarity and whether it effectively tests domain-specific knowledge.

Specifically, for the quality assessment, the human evaluators are asked to assess whether the questions were clear and required an understanding of scientific knowledge within the respective discipline, as outlined in the following metrics:

• Score Point 1. Strongly Disagree: The question is irrelevant or cannot be answered based on the scientific content presented in the figure.
• Score Point 2. Disagree: The question lacks clarity or can be answered without specific knowledge of the scientific content in the figure (e.g., it can be answered with common sense).
• Score Point 3. Neutral: The question is clear but requires only minimal understanding of the scientific content in the figure.
• Score Point 4. Agree: The question is clear, answerable, and requires an adequate understanding of the scientific content in the figure.
• Score Point 5. Strongly Agree: The question is clear, answerable, and effectively evaluates a very deep understanding of the scientific content in the figure.

The evaluation results are shown in the table (next response).

C3: other metrics for captioning task

Response: Thank you for the suggestion. We will BLEU (BLEU-4) [2], CIDEr [3], and also L3Score [4] as recommended in the updated Table 3 in the updated draft. Below we briefly show these added results when grounded on abstract.

Table 1: Captioning results when grounded on paper article.

The BLEU and CIDEr scores are relatively low, likely because the captions are not typical text; they include many scientific domain-specific terminologies and scientific expressions, which pose challenges for these metrics. The L3Score exhibits a similar trend to G-Eval but provides more distinct differentiation. The finetuned models got meaningful improvements on all of these metrics by using our dataset. Especially, we were surprised at great increase on L3Score, so that our model is better than Gemini-1.5 Pro.

C4: The results for the materials science case study

We appreciate your thoughtful questions regarding the material science case study results. We would like to firmly confirm that we followed all the same settings as Gruver et al., but we improve the results further with pre-training of LLMs with our dataset along with fine-tuning, and reported absolute percentage performance, which is much more fair and correct, from 10,000 generated sample cases instead of stage-wise survived ratio for the previous steps. Please find detailed response below.

C4.1: What is the baseline LLAMA-2-7B performance? (without any tuning?) Many numbers in Table 5 and Figure 6 already seem quite high so it is hard to understand what baseline you are starting from and how much room for improvement there was

Response: The LLaMA2-7B and even the proprietary LLMs shows limited performance in crystal material generation without fine-tuning (For detailed zero-shot performance metrics of GPT-4o, please refer to response to C4.2.). Therefore, Gruver et al. proposed a novel way of fine-tuning LLMs for crystal material generation. The performance of LLAMA2 models reported in our Table 5 in the main pages are obtained by further finetuning on mateiral generation data as in Gruver et al's paper. Our experiments show that when the base LLaMA-2-7B model is first further pre-trained on our dataset before fine-tuning for material generation, it achieves superior performance compared to directly fine-tuning the base model. This improvement suggests that pre-training on our data enhances the model's scientific knowledge understanding.

Metrics: The validity check measures the generated materials' correctness in terms of structure and composition, while the coverage metric assesses how well the generated materials compare to an existing collection. These metrics primarily ensure that the generated materials are valid and identify differences from existing ones, but they are less critical. However, to develop meaningful crystal materials, the most important metrics are about stability including metastability measured by M3GNet, and stability measured by DFT, since practically we confront the difficulty on generating stable materials for next generation of materials. Regarding the performances on these two metrics from the previous SOTA models, there are rooms for significant improvement. As shown, our approach significantly enhances metastability and stability, which are the aspects that truly matter in material science.

Thank you for recognizing the novelty of our dataset, the comprehensive evaluations, and the transparency of our benchmarks. Please find our detailed responses to your concerns below.

C1 regarding taxonomy

Response: Thank you for your feedback on our taxonomy. The original taxonomy was established by the authoritative committee of the Nature Communications journals, providing a credible framework for categorization.

However, we appreciate your valuable suggestion to consolidate the categories into broader groupings. To better align with the categorization approaches used in other datasets, we have re-defined the original 72 categories into 18 science fields: Material Science, Chemistry, Physics, Biochemistry, Environment, Climate Sciences, Earth Sciences, Biological Sciences, Biomedical Sciences, Health and Medicine, Engineering, Energy, Mathematics and Computing, Astronomy and Planetary Science, Social Science, Pharmacology, Agriculture, Business and Industry

Additionally, for the purposes of human evaluation, we have selected 10 of these major categories to report on. Regardless of the specific taxonomy used or the number of categories, we believe that the scope of our dataset (encompassing over 131,000 peer-reviewed articles and 742,000 figures from the Nature Communications journals) provides comprehensive coverage to evaluate LLMs' understanding of natural sciences.

C2 regarding benchmark design and question originality:

Response:

We acknowledge that the current dataset uses permutations of existing data. However, our benchmark requires comprehensive scientific understanding through bidirectional tasks:

1. Figure-to-caption matching requires models to select the correct caption from multiple options within the same paper. For example, in Figure 3, models must distinguish between different experimental results and structural characterizations of PbZrO3 membranes by understanding detailed synthesis processes, surface morphology measurements, and atomic-scale structural analyses.
2. Subfigure-to-caption and caption-to-subfigure matching test bidirectional understanding of technical details. Models need to differentiate between various characterization techniques (surface morphology by AFM, atomic-resolution HAADF-STEM imaging, and electron diffraction patterns) and match them with their corresponding technical descriptions.

Regarding the concern about pattern matching: Our tasks specifically challenge superficial matching by using options from the same paper with similar technical depth. Success requires understanding the scientific relationships between different characterization methods and their implications. This is evidenced by:

1. Basic MLLMs performing at chance level on bi-directional figure-caption matching.
2. Performance improving significantly with abstract context (Table 3 in the main pages).
3. Clear performance differences from the large proprietary MLLMs and open-source MLLMs across scientific domains (Figure 4 in the main pages)

While generating new questions could be valuable, current LLMs struggle with reliable scientific question generation, and expert annotation across 72 disciplines would be resource-intensive. Our work provides a foundation with rich peer-reviewed content for future extensions.

C3 regarding whether the benchmark task necessaties the understanding of the scientific knowledge in the figures.

Response: To address the concerns about our human baseline and validate that our tasks require domain expertise, as we mentioned in the above, we conducted comprehensive evaluation with human experts of PhD degree in 10 major science domains.

PhD Expert Evaluation: Specifically, by adopting your advice, we considered 10 broader major scientific categories, and we recruited PhDs from the Prolific platform, which is a well established platform of high quality scores, with verified degrees in 10 major scientific categories that align with our dataset's primary domains: Material Science, Chemistry, Physics, Biochemistry, Environment, Climate Sciences, Earth Sciences, Biological Sciences, Biomedical Sciences, and Health and Medicine.

For each category, we constructed a new sub-benchmark by selecting 25 questions per setting (75 total) from our three figure-caption matching tasks from the original test set. Thus, the new sub-benchmark consists of 750 questions total, and we have updated human evaluation scores with PhD degrees on 10 broad categories.

C4: Concerns about the design of the human baseline, specifically the use of CS graduates instead of domain experts as the benchmark.

Response: In addition to the quality assessment reported above, we also evaluated the actual performance of PhD experts on our benchmark to address the concerns about human baseline design. We conducted a comprehensive evaluation with PhD experts from the Prolific platform across 10 major scientific categories. Our evaluation was designed to establish three key performance metrics:

• In-Ph.D. Degree Domain Expert Performance: Performance of human Ph.D. evaluating questions within their specialty domain of Ph.D. degree
• Out-of-Ph.D. Degree Domain Performance: Performance of human Ph.D. evaluating questions in Science, but outside their expertise of Ph.D. degree
• Overall Cross-Domain Performance: Comprehensive assessment across all domains

Table 2 in Rebuttal: Performance Comparison of Human Ph.D. Experts and Models with a Newly Constructed sub-Benchmark (%)

As demonstrated in Table 2 in Rebuttal, several key insights emerge:

Domain Expertise Impact: PhDs evaluating within their expert domain of Ph.D. degree consistently outperform those working outside their expertise in other Science domains across all settings, confirming our tasks require domain-specific knowledge

Task Difficulty: Even domain experts achieve moderate performance within the recommended time limit (2 minutes per question), indicating the tasks' challenging nature

Scientific Understanding Capabilities: The strong performances of both our fine-tuned sLLM and large proprietary models demonstrate potential as AI research assistants. Notably, our dataset significantly improved sLLM performance, reducing the gap with large proprietary models (see Table 4 in the main paper)

C5: For captioning tasks, the model finetuned on in-domain data achieved significantly higher score on the ROUGE metric yet underperforms its baseline counterpart in FActScore.

Response: Firstly, we updated all results using 200 samples instead of the previous 100 for LLM-based evaluation. Additionally, we fine-tuned both Qwen2-VL-2B and Qwen2-VL-7B models. The results are included in the updated draft, and also presented in the following table.

Table 3 in Rebuttal: Updated figure captioning results using 200 samples for LLM-based evaluation, including Qwen2-VL-7B fine-tuned on our data.

We observed that fine-tuning the larger Qwen2-VL-7B model significantly improved the FActScore, even surpassing GPT-4V and GPT-4o. Regarding the performance degradation of Qwen2-VL-2B after fine-tuning, we manually reviewed the generated captions and found that this may be due to the oracle captions typically following a specific format: "[Summary] a. [subcaption a] b. [subcaption b] ..." This format is concise, only summarizing the essential content of each sub-figure without unnecessary details. Given the limited capabilities of Qwen2-VL-2B, it appears that the model learns the format but struggles to generate accurate and concise content, resulting in degraded performance. In contrast, the more powerful Qwen2-VL-7B model demonstrates notable improvement by effectively learning both the format and the concise, accurate summary of the figure's content.

Q1: Relevance to MMStar [1]

Response: Thank you for bringing this work to our attention. It is indeed relevant, and we will include it in the related work section of the updated draft. This paper also points out that many current datasets contain questions that do not require a deep understanding of the figure to answer. In contrast, our benchmark ensures that a thorough understanding of the images is necessary for answering the questions. Additionally, while MMStar is curated from existing works, we introduce a newly collected large dataset.

Q2: L359: Can authors verify the statement by evaluating the top 5 models on a different evaluator?

Response: Thank you for the suggestion. In the updated draft, we have updated the results using the latest version of GPT-4o-0806 as the evaluator, which is different from the evaluated models.

Q3: Figure 4 is very confusing and the overlapping of performance from different models makes the comparison very hard to understand. Can authors use use a better way to visualize this or use a table instead?

Response: Thank you for the suggestion. We consider updating Figure 4 to include the 10 reorganized subjects instead of the 3 categories and over 50 subjects, as discussed above, to make the figure easier to read.

Q4: For figure 6, the gap between MMSci+MMC4 (Vis+Text) and MMSci+MMC4 (Text) seems to be small. Can authors additional provide standard error? Also, given many figures appear to have small texts, have authors verified the readability of figures after being downscaled to 336^2 (L452 where you stated the use of CLIP ViT-L/14-336)?

Response: Yes, we ran the inference three times using these two models. The average score and standard error are provided in the table, and we can see that the performance increases from using our multi-modal dataset are valid by considering standard error to the average scores.

Yes, we run the inference three times using these two models. The average score and standard error are provided in the following table.

In addition to this table, we found the gains of 3.08% increase on metastability of using our dataset compared to the baseline with the same setting, which is a key metric for model performance. Additional significant gains in in-filling tasks are detailed in Table 11. of the supplementary materials.

Regarding image resolution: While the 336x336 resolution maintains text legibility in most cases, we acknowledge potential readability limitations for very small text. However, this is a common resolution and practice in current MLLMs using this architecture. and we do think it can influence we draw the conclusion.

Q5: An additional experiment that can be very helpful to understand the impact of your training dataset is to evaluate existing VLMs model finetuned on your dataset on other knowledge-based benchmarks such as MMMU (which also examines specific sub-divisions in natural science) to see whether the model is only fitting to the nature communication distribution or generalizing to a variety of distributions that requires retrieving knowledge based on multimodal input. Could authors provide an additional experiment on this?

We appreciate the reviewer's suggestion to evaluate our fine-tuned models on other multimodal knowledge benchmarks, such as MMMU. We are pre-training and fine-tuning a larger model with our dataset, and we will keep trying to post the result within rebuttal period or at least we will include the results of larger models in the updated version.

Thanks for the thorough response! For the rebuttal that

To better align with the categorization approaches used in other datasets, we have re-defined the original 72 categories into 18 science fields: Material Science, Chemistry, Physics, Biochemistry, Environment, Climate Sciences, Earth Sciences, Biological Sciences, Biomedical Sciences, Health and Medicine, Engineering, Energy, Mathematics and Computing, Astronomy and Planetary Science, Social Science, Pharmacology, Agriculture, Business and Industry

Can the authors describe how the original 72 categories were re-defined into 18 fields? This still seems confusing—for example, Biochemistry is considered part of Biological Sciences according to Nature's taxonomy, yet you have separated them into two distinct fields. I am not objecting to the limited number of domains, but it would be better to ensure everything is as concise and consistent as possible. Additionally, the five categories listed in Table 6 should be compared with other works that consider broader domains as categories.

Performance of human Ph.D. evaluating questions within their specialty domain of Ph.D. degree

Upon reviewing Table 2 in the rebuttal, I was surprised that your fine-tuned model not only outperformed all proprietary models but also in-domain Ph.D. evaluators by approximately 2x. This raises additional concerns about the utility of your benchmark:

• What exactly is your benchmark evaluating? Does it assess holistic multimodal scientific understanding or only the specific tasks you outlined? If it evaluates holistic multimodal scientific understanding and your model, along with proprietary models, outperforms in-domain Ph.D. evaluators, this seems to contradict findings from other multimodal knowledge benchmarks such as MMMU, where GPT-4 performs worse than human experts. Could the authors explain this contradiction, or possibly rescope the evaluation?

• In general, one would aim to create evaluations that humans (whether expert or average) perform well on, but not models. Based on the results, however, this appears to be a task where existing models outperform humans, while humans themselves struggle. Could the authors discuss why human experts perform worse than all models reported in the table?

• Fine-tuning models on in-domain data appears to substantially boost performance. In fact, it seems that the fine-tuned 7B model is nearing saturation on this benchmark. Given that there are already more capable open-source models available than the one you used, this raises concerns that the utility of the benchmark could diminish if saturation occurs too soon. Could the authors address this issue and suggest ways to make the evaluation more challenging?

We thank the reviewer for their prompt and insightful feedback. Please find our detailed response below:

Can the authors describe how the original 72 categories were re-defined into 18 fields? This still seems confusing—for example, Biochemistry is considered part of Biological Sciences according to Nature's taxonomy, yet you have separated them into two distinct fields. I am not objecting to the limited number of domains, but it would be better to ensure everything is as concise and consistent as possible. Additionally, the five categories listed in Table 6 should be compared with other works that consider broader domains as categories.

Our re-categorization is based on the Web-of-Science citation index. The 72 subjects are mapped into 18 distinct fields, striking a balance between meaningful consolidation and preservation of important disciplinary distinctions. We have compiled the mapping in Table 1.

Table 1: recategorization of the 72 subjects in the MMSci dataset.

Dear Reviewers, We sincerely thank you for your thoughtful feedback that has helped strengthen our manuscript. Given our comprehensive responses and substantial improvements, we respectfully request your reconsideration of our submission.

Major Enhancements:

1. Rigorous Human Expert Validation

• Conducted large-scale evaluation with 30 PhD experts across 10 scientific fields
• Achieved consistent expert quality ratings (~4/5) validating scientific depth
• Demonstrated clear expertise effects: in-field PhDs (40-50% accuracy) vs. out-of-field PhDs (33-46%)
• Validated task design through 750 expert assessments (75 per domain)
• Confirmed questions require graduate-level domain knowledge through expert feedback
• Demonstrated real-world research assistant potential through expert performance benchmarking

2. Comprehensive Technical Evaluation

• Expanded evaluation framework with comprehensive metrics: BLEU-4, CIDEr, G-Eval, L3Score for robust assessment
• Doubled evaluation sample size to 200 for enhanced statistical validity
• Extended model scale analysis across 2B, 7B, and 26B parameters
• Conducted full-article vs. abstract-based captioning comparison
• Demonstrated Qwen2-VL-7B-MMSci achieves state-of-the-art performance:
  • Multiple-choice accuracy: 87.49% overall
  • Captioning metrics: BLEU-4 (8.29), ROUGE-L (21.80), L3Score (0.447)
  • Competitive performance of Qwen2-VL-7B-MMSci with minimal context requirements to the propriety model with full-article (Gemini 1.5)

3. Enhanced Materials Science Validation

• Conducted comprehensive proprietary model comparisons
• Achieved breakthrough improvements:
  • Metastability: 64.5% (29.2% relative improvement)
  • Stability: 8.2% (54.7% relative improvement)
• Validated results through multiple experimental runs with standard error analysis
• Demonstrated practical impact on materials discovery
• Established new state-of-the-art for scientific AI in materials generation

4. Framework and Methodological Improvements

• Refined scientific taxonomy using Web of Science citation index
• Distinguished MMSci from existing benchmarks through:
  • Graduate-level depth requirement
  • Multi-domain scientific reasoning
  • Comprehensive visual-language integration
• Demonstrated value as pre-training resource through transfer learning results
• Established clear evaluation protocols for reproducibility
• Provided detailed cross-disciplinary performance analysis

These improvements demonstrate how our MMSci dataset advances the state-of-the-art in scientific understanding, enabling LVLMs to develop both broad and deep domain expertise across scientific fields. We would greatly appreciate your reconsideration based on our comprehensive responses to your valuable feedback. Thank you for your time and careful consideration of our work.

Best regards, Authors

We sincerely appreciate your detailed and thoughtful review of our work. We are grateful for your recognition of the dataset’s comprehensiveness, the challenging benchmarks, and the demonstrated improvements through fine-tuning. These aspects are core to our contributions, and we are pleased that they resonate with you. Below, we provide our responses to your feedback and address your questions and concerns.

Concerns regarding Dataset Curation Details: "The paper should provide more detailed information about the data annotation process to enhance transparency and reproducibility."

Response: Thank you for the suggestion. We have now included additional information on the quality control stages. As outlined in Section 3, the dataset content, including article sections and figures, is systematically organized in different places in the original nature communications website, ensuring easy extraction without data loss. The primary task was to split the full figure captions into individual descriptions for each sub-figure. To achieve this, we first conducted a small-scale human verification to identify potential caption formats. Based on these findings, we designed regular expression matching functions to extract the sub-captions. Next, we used the SpaCy package to verify whether the extracted text fragments were complete sentences or meaningful standalone phrases. Finally, we performed additional human verification on a subset of the extracted data to ensure the method's accuracy and reliability.

Concerns about model size comparison.

Response:
Thank you for raising this concern. In our experiments, we prioritized widely used open-source and proprietary models. During the rebuttal period, we also fine-tuned Qwen2-VL-7B on our dataset, resulting in Qwen2-VL-7B-MMSci, and included baseline results from a larger model InternVL2-26B. Noting Qwen2-VL-7B's higher baseline performance compared to InternVL2-26B, we are also fine-tuning Qwen2-VL-72B. If these experiments are completed within the rebuttal period, we will report the results; otherwise, they will be included in the revised version. Additionally, we encourage other researchers to evaluate their models using our benchmark.

Table 1: Multi-choice QA accuracy with additional models (partial results for brevity)

Concerns about LLM Evaluation Sample Size

Response: Thank you for the suggestion. We have added LLM-based evaluation results with a sample size of 200 using the recent GPT-4o-0806 version, which will be included in the revised manuscript. "W/ Abs" and "w/o Abs" indicate whether captioning is grounded on abstracts.

Concerns about Data Leakage

Response: Since all the papers in our dataset are accepted publications from the prestigious Nature Communications journal, we believe there is minimal risk of significant overlap in similar images across the dataset.

We hope these responses have addressed your concerns. If you need any further clarification, please don't hesitate to ask. We value your feedback and look forward to further discussion.