Structure-aware Domain Knowledge Injection for Large Language Models

1. Unclear Experimental Setup: (1) In Open-ended Question Answering: For the LONGBENCH dataset, the division between the training and test sets is unclear. For example, as described in Section A.1, the authors generate a knowledge structure for each passage, but the sum of individual documents—reported as 200+200+150+200+200+200=1150—does not align with the 1350 entries stated on line 781 of this paper, raising concerns about the fairness and consistency of comparisons. Additionally, the reference to “10,476 passages” on line 349 is unclear in terms of how these passages differ from those in the appendix (line 781). Could the authors clarify the dataset composition, providing a detailed breakdown of how the 1350 entries and the 10,476 passages correspond to each other and to the original LongBench dataset? (2) The paper mentions “removing entries with over 0.5 F1-Score similarity to test samples to prevent data leakage.” This raises questions about whether the SSFT data generation process creates questions similar to those in the bechmark’s test set, and if so, whether filtering out sample with an F1 score over 0.5 is sufficient to prevent SSFT from merely serving as data augmentation by introducing similar samples to the test set. Could the authors provide more details on the SSFT data generation process, specifically how similarity to test samples is measured? Additionally, an analysis of the similarity distribution between generated and test samples could help justify the choice of a 0.5 F1-Score threshold. Futhermore, I am also curious about the results about CPT+SSFT, which can measure whether the performance improvement is just because SSFT introduced data augmentation. (3) In the Multiple-choice Tasks, the RAG approach consistently underperforms compared to other fine-tuning methods, which contrasts with findings in existing studies [1][2]. However, the paper does not provide specific settings for the RAG experiments, which could impact the reliability of these comparisons. Could the authors include detailed information about the RAG experimental setup, including the retrieval method, the number of retrieved documents, and how these documents were incorporated into the model’s decision-making process? This additional context would help clarify the reasons for the observed performance discrepancy.

[1] Ovadia, Oded, et al. “Fine-tuning or retrieval? comparing knowledge injection in LLMs.” arXiv preprint arXiv:2312.05934 (2023).

[2] Soudani, Heydar, Evangelos Kanoulas, and Faegheh Hasibi. “Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge.” arXiv preprint arXiv:2403.01432 (2024).

2. Inappropriate Dataset: (1) In the LONGBENCH Dataset: The average passage length in both single- and multi-document settings ranges from 3K to 18K, exceeding the 4K context limit of Llama-2. This makes it unsuitable for the baseline OBQA evaluation, as much of the content is truncated. While the intent might be to showcase the advantage of knowledge structuring on lengthy documents, the baseline’s context limitations lead to an unfair comparison, necessitating a more appropriate choice of long-context models for fair assessment. (2) In the Multiple-choice Task: The paper evaluates on the MMedBench dataset but uses MedTextBooks as the knowledge source rather than MMedC. The reason for not applying SCPT on MMedC is unclear. Comparing training costs across different datasets might not be valid, as smaller, high-quality datasets can yield better training outcomes, making this comparison potentially misleading.
3. Overly Conclusive Experimental Results: On line 468, the scaling law in Equation 3 is derived by fitting only four points from Figure 7. This limited dataset does not provide a sufficient basis for such a definitive conclusion, making the generalization seem premature. Could the authors consider collecting more data points to better validate the scaling law? Alternatively, if further data collection is not feasible, presenting it as a preliminary observation rather than a definitive conclusion would strengthen the reliability of this claim.

Commonality in Code Training: While the paper claims originality in its structure-aware approach, this technique is already prevalent in contexts where training data consists of code or hierarchical data structures. In code training, for instance, it’s common to replace long sequences with structured, layered representations that summarize relationships across hierarchies. This may limit the novelty of StructTuning’s proposed structure-aware methodology.

Scaling Law Verification: The proposed scaling law, while intriguing, relies heavily on extrapolation. The lack of empirical validation for corpus ratios between 5% and 100% significantly undermines this claim. While acknowledging resource constraints, exploring at least a few intermediate data points would considerably strengthen the paper.

Clarity on Knowledge Structure Extraction: The paper would benefit from greater clarity on the knowledge structure extraction process using the specialized 7B model. Specifically, more information on training details and quantitative performance metrics, such as precision and recall against human-annotated structures, would strengthen the analysis. Additionally, including representative examples of extracted structures across different domains would illustrate the model’s effectiveness and adaptability. An ablation study could help evaluate the impact of the extracted structures on downstream tasks like multi-hop QA. For instance, removing different levels of the extracted structure (e.g., top-level categories versus deeper hierarchies) and analyzing the effect on QA performance would clarify whether certain structural elements are critical or if the performance degrades gracefully without them.

Lack of comparison with other knowledge injection methods: The paper would benefit from a more comprehensive comparison with a broader range of knowledge injection techniques beyond the primary comparisons to vanilla CPT+SFT and MMedLM. Including comparisons with methods like knowledge distillation tailored for domain adaptation, recent approaches integrating external knowledge graphs during fine-tuning, and hybrid methods combining retrieval-augmented generation with parameter-efficient fine-tuning would strengthen the paper’s claims of superiority. This expanded evaluation would provide a clearer picture of the proposed method’s advantages in terms of efficiency and domain adaptation capabilities.

Comparison with Retrieval-Based Methods: While RAG is briefly mentioned and experimented with (sec 4.2 & 5). The paper provides very limited details about how this RAG baseline was implemented. It doesn't specify which retrieval method was used (e.g., dense or sparse retrieval and embedding model), the size and composition of the knowledge base, and any preprocessing or indexing steps applied. Additionally, details on retrieval hyperparameters—like the number of retrieved passages and any reranking strategies—would clarify the setup. A more rigorous comparison with state-of-the-art retrieval-augmented methods is needed to properly contextualize the contribution. This is particularly important given the focus on knowledge-intensive tasks.

Small Typo: Table A4, line 2 & line 3's 1st column. P19, line 999, "self-conatined".

The analysis of experimental results is too arbitrary: for example, the author directly fits the regular curve of line 468 after obtaining the performance of the large language model under three experimental conditions: 0.1, 0.3, and 0.5. In the absence of sufficient experimental data, it seems impossible to draw such a universal conclusion. Such an arbitrary conclusion may cast doubt on the reliability of the conclusions of this paper.

The Questions below need to be addressed. Q4 about corpus size is important and needs clarification. Otherwise the paper is in a good shape overall.

After Author Rebuttal: While I appreciate the author responses and my clarification questions have been mostly addressed, the Q4 remains concerning. A main result in the paper is that the proposed LLM+Ours that uses 76M tokens can achieve 50% of the improvement over the baseline LLM compared to LLM+MMed that uses 25.5B tokens. However, "Ours" uses a different knowledge source than "MMed", as Reviewer kHUD also mentioned, and it is unclear whether the performance comes from the fact that the extra knowledge source is high quality. The scaling law predicted results is also not perfectly convincing, and the authors could have run the experiment with the 5% data to show the argued 100% performance. Otherwise, I believe the paper has its merits.