SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

Q1: The creation of the CPT and SFT datasets seems to rely on LLMs. The paper can randomly sample a small subset of the created dataset to check its quality with humans to show the effectiveness of the proposed framework.

Thank you for the suggestion to evaluate the quality of the CPT and SFT datasets with human annotations! In response, we sample random subsets of unfiltered CPT and SFT data and ask human annotators to rate their quality. We then compare the human annotations to the scores from our quality filters to validate the effectiveness of the proposed filtering framework.

Evaluation Setup. For the CPT dataset, 50 entries are sampled from textbook and journal datasets. For the SFT dataset, 50 entries are sampled from SciLitIns. Each entry is independently evaluated by four annotators using the same prompts provided to LLMs. To assess alignment with human evaluations, we compute two measures:

• Human-Human Agreement: Calculated as the average Spearman correlation coefficient across all pairs of annotators.
• Human-Quality Filter Agreement: Calculated as the average Spearman correlation between each annotator’s scores and the quality filter's score.

A higher Spearman correlation indicates a stronger agreement between the compared sets of scores. The results are presented below:

On both datasets, the Human-Quality Filter Agreement is comparable to or even exceeds the Human-Human Agreement. This indicates that the scores generated by our quality filter are closely aligned with human annotations, demonstrating the reliability of the filtering framework in capturing quality as perceived by humans.

We hope this answers your question! Detailed results and analyses are included in the Appendix H.

Q2: The experiment and ablation study is not comprehensive. The paper fails to show that the model is fine-tuned to other existing scientific understanding datasets, such as the Dolma dataset (https://allenai.github.io/dolma/).

R2: Thank you for bringing up the Dolma dataset. We appreciate the opportunity to clarify our position and the distinctions between our dataset and existing resources.

The Dolma dataset, designed as a broad pretraining corpus, includes a wide variety of topics, with its scientific subset identified as the peS2o dataset [1]. To provide a clearer comparison, we outline key characteristics of the peS2o dataset and our scientific textbook and journal dataset below:

While the peS2o dataset offers broad coverage of open-access academic papers, we would respectfully note that such open-access materials (e.g., books and academic papers) may already be included in the pretraining corpus of Qwen [2], as mentioned in the Qwen technical report: “Our dataset is designed to meet these requirements and includes public web documents, encyclopedia, books, codes, etc.”

In contrast, our dataset provides unique value through its curated, high-quality content from copyrighted textbooks and journals, which is specifically tailored to scientific literature tasks. Although we cannot release the dataset directly due to copyright restrictions, we have shared a complete list of the textbooks used, including textbook titles, authors, and other meta-information (See the books.xlsx file in the updated supplementary materials).

Thank you again for your constructive feedback! We have also included references to these papers in the related work section.

[1] Luca Soldaini and Kyle Lo. 2023. peS2o (Pretraining Efficiently on S2ORC) Dataset. (https://github.com/allenai/peS2o)

[2] Qwen Technical Report (https://arxiv.org/abs/2309.16609)

Q1: The CPT corpus (12.7B tokens) is relatively small compared to standard pre-training datasets. The paper acknowledges this limitation but could discuss potential impacts more thoroughly. For example, how this affects the representation from different scientific subject domains.

R1: Thank you for your insightful feedback! We agree that discussing the implications of the relatively small scale of our CPT corpus is crucial for a comprehensive understanding of the work’s limitations.

Specifically, while our dataset has been meticulously curated to ensure high quality and domain relevance, the limited size presents challenges in achieving comprehensive representation across diverse scientific disciplines. For example, certain specialized domains may be underrepresented, potentially limiting the model’s performance on tasks requiring expertise in those areas.

While increasing the dataset size is a logical next step, it is equally important to maintain stringent quality control to avoid introducing noise, which could dilute the benefits of domain-specific data. The promising performance observed in our experiments highlights that a smaller but well-curated dataset can still yield significant gains. However, scaling the corpus effectively while preserving relevance and quality remains a key challenge for future work.

We have included this expanded discussion in the revised Limitations section of the paper. We appreciate your thoughtful suggestion and hope these additions address your concerns.

Q2: How sensitive is the model performance to the quality filtering threshold (currently set at 25%)? Was this choice empirically validated?

R2: Thank you for your constructive feedback. We have conducted empirical experiments to investigate the sensitivity of model performance to the quality filtering threshold. The results are summarized below:

The results show that setting the filtering threshold at 25% achieves the best overall performance, effectively injecting scientific domain knowledge into the model.

• When the threshold is set too high (e.g., 35%), the average quality of the corpus improves; however, the reduction in data volume limits the ability to fully capture domain knowledge.
• Conversely, when the threshold is set too low (e.g., 15%), the increase in data volume includes more low-quality content, which negatively impacts model performance.

As you noted, the trade-off between data quality and quantity is crucial. We have included this analysis in Appendix C of the revised paper to clarify our choice of the 25% threshold. Thank you for highlighting this important aspect!

Q3: The instruction synthesis method uses GPT-4 - have you explored using smaller models or your own models in a bootstrapping approach?

R3: Thank you for your insightful comment! We agree that bootstrapping strategies, such as leveraging smaller models or our own models for data synthesis, could be a promising direction for future improvements. These approaches effectively address data scarcity by iteratively enlarging the dataset, as demonstrated in influential works like [1][2].
We also recognize the critical role of high-quality data in scientific domains, and we are excited to explore this direction in the next version of SciLitLLM. However, given our current computational constraints and the need to prioritize other experiments during this rebuttal period, we may not be able to implement and evaluate this approach in the current submission cycle.

We appreciate your understanding and believe this suggestion will be valuable for guiding the continued development of our model. Thank you for highlighting this important area for improvement!

[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

[2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Q1: Are you planning to release the textbook datasets?

R1: Thank you for your question. We have released a complete list of 73,000 textbooks, including textbook titles, authors and other meta-information (See the books.xlsx file in the updated supplementary materials). Here are the first two rows of the textbook list:

Unfortunately, due to copyright restrictions on the electronic textbooks we use, we are unable to directly release the textbook files. However, with all our respect, we would like to highlight the contribution we’ve made in terms of open-sourcing the data processing pipeline.

For researchers in academia or industry who wish to train private domain-specific models similar to ours, —especially those with access to large private domain copora but limited domain expertise — we hope our open-source pipeline can be a valuable resource. It allows for the construction of private domain models, not only limited to scientific literature.

In fact, the need to build a domain-specific model for understanding scientific literature arose from our own practical challenges. We found that existing research and open-source projects did not directly address our demands, which motivated us to develop the data processing and model transfer pipeline described in this paper. While we are unable to release the textbooks themselves, we would like to highlight that by open-sourcing the complete data processing pipeline, we can provide helpful tools for both academic researchers and industrial practitioners working on similar scenarios in the future.

We hope this clarifies our stance, and we appreciate your understanding.

Q1: The PDF processing pipeline can be improved. Scientific PDF are known to contain complex layout and structures, and previous work have identified that using simple PDF parsers can lead to suboptimal training results (S2ORC, Lo and Wang, and ). However, the author primarily uses a simple PyPDF parser ("Converting these documents using tools like PyPDF2 often introduces formatting and syntax errors, which degrade the quality of the corpus (line 246)"). I'd suggest the authors investigate the text quality issues and check other libraries like papermage, Lo et al.

R1: Thank you for this valuable feedback regarding PDF processing. We compared a few PDF parsing options, focusing primarily on the closed-source Mathpix and the open-source PyPDF2 library. Due to the high costs associated with Mathpix (approximately $80,000 to process 70,000 textbooks), we opted for PyPDF2 along with additional formatting and syntax error correction techniques described in the paper to mitigate common parsing issues.

We appreciate your suggestion to explore alternatives like Papermage [1], and will investigate these in our future work to further improve parsing quality. Given our current computational limitations, however, we may not be able to retrain the model with Papermage-parsed data within the rebuttal period. We hope for your understanding in this regard.

[1] PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific

Q2: The results are not very strong. I'd imagine a domain-specifically distilled model can have a substance gain in performance compared to GPT-4o, especially the instruction fine-tuning dataset is generated via GPT-4o (line 331); however, as shown in table 3, the trained models (SciLitLLM-7B and SciLitLLM-14B) are on par with GPT-4o.

R2: Thank you for your thoughtful feedback. We would like to take this opportunity to clarify the performance of SciLitLLM.

While we agree that domain-specific fine-tuning can provide performance gains in certain cases, we respectfully argue that it is not always feasible for such models to outperform strong general-purpose models like GPT-4o, even within a specialized domain. In scientific knowledge and literature understanding fields, for instance, Table 5 in SciKnowEval [2] — a benchmark for evaluating scientific knowledge in LLMs — shows that GPT-4o outperforms domain-specific models like Mol-Inst and ChemLLM by over 10%, where both models are instruction-tuned on extensive real-world and synthetic data.

Similarly, Table 3 in SciRIFF [3], which specifically evaluates scientific literature understanding, demonstrates that both their 7B and 70B fine-tuned models underperform GPT-3.5T and GPT-4 by significant margins, with the 70B model trailing GPT-4 by over 10%.

In this context, SciLitLLM, despite being comparatively smaller in scale, achieves performance on par with GPT-4o, which we believe is a promising result. Additionally, we would like to highlight the practical value of our work: there are real-world scenarios that require private, domain-specific models and cannot rely on proprietary general-purpose systems like GPT-4o. Addressing these needs is a key motivation behind our research.

We hope these clarifications provide further perspective on the strengths of SciLitLLM, and we greatly appreciate your understanding and thoughtful evaluation!

[2] SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

[3] SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature

Dear Reviewer 7iRT,

Thank you again for your thoughtful feedbacks on our submission, especially for advising us to 1) clarifing the limitations of our current PDF processing, 2) explaining SciLitLLM’s performance relative to GPT-4o, and 3) revising results in Tables 2 and 3 for clarity, 4) justifing the inclusion of SciTulu for fairness. These valuable suggestions have improved the clarity and quality of our work. We hope that these improvements will be taken into consideration.

If our response has resolved your concerns on our paper, we will greatly appreciate it if you could re-evaluate our paper for a higher rating. We are also willing and ready to engage in discussions, if you have any further questions.

Authors