ChemOrch: Empowering LLMs with Chemical Intelligence via Groundbreaking Synthetic Instructions

We sincerely thank you for the thoughtful and detailed feedback. Below we respond to each point individually.

---

W1. Concern about the factual accuracy of generated instruction–response pairs.

A1: Thank you for raising this important concern. We respectfully believe there may have been a misunderstanding about the nature of our data generation pipeline.

To clarify: ChemOrch is explicitly designed to *avoid* relying on free-form LLM outputs for response generation, precisely because of the risk of hallucinations and factual errors. Instead, we adopt a tool-grounded generation framework, where responses are produced through structured tool calls executed by trusted chemistry engines such as RDKit and PubChem.

This approach ensures that outputs are not hallucinated, but rather scientifically derived from verified computations and databases. We further apply multiple safeguards—including error detection, self-repairing logic, and sufficiency validation (Sections 3.2–3.3)—to verify the completeness and correctness of generated responses before they are accepted.

In addition, we conduct extensive human evaluation to empirically validate the quality of instruction–response pairs. As reported in Section 4.3 and Appendix G:

• ChemOrch-generated data achieves 82.64% instruction-following accuracy
• And 85.14% factual correctness, as judged by domain experts

These results, along with a transparent annotation protocol and clearly reported sample sizes, offer strong empirical support that the generated data is of high quality and appropriate for downstream fine-tuning.

We hope this clarifies the design intent of ChemOrch and reassures you that factual accuracy is both a central motivation and a validated strength of our framework.

---

W2. Unclear motivation behind the early stopping mechanism.

A2: Thank you for highlighting this point. We apologize for the lack of clarity.

The early stopping mechanism in ChemOrch is not used to skip or prematurely terminate response generation. Instead, it is purely a computational efficiency optimization, applied only after the system determines that all instruction requirements have been successfully satisfied.

To ensure no part of the instruction is overlooked, early stopping is always followed by a sufficiency validation step. If this check identifies any missing or incomplete outputs, execution resumes and additional tool calls are triggered.

In short:

• Early stopping is never applied at the expense of correctness
• It only eliminates redundant computation when a task is verifiably complete

We will clarify this mechanism further in the revised manuscript.

---

W3. Concerns about human judgment introducing bias in evaluation.

A3: We appreciate your thoughtful observation. We address this concern through a robust, multi-pronged evaluation framework combining both human annotation and LLM-as-a-Judge validation.

---

1. Human Evaluation Protocol – Designed to Minimize Bias

We adopt several measures to reduce subjectivity:

• Fact-based assessment: Annotators were encouraged to consult authoritative chemical resources (e.g., PubChem, textbooks) when uncertain.
• Diverse reviewer pool: Our evaluation involved nine annotators (3 undergraduates + 6 PhD students) from ML, chemistry, and chemical engineering backgrounds.
• Blind and randomized assignment: Evaluation was conducted blindly across models and batches to avoid anchoring bias.

To further validate the internal consistency of our human evaluations, we computed the inter-annotator agreement using Cohen’s Kappa coefficient across three separate batches. As shown below, the Kappa scores indicate substantial to almost perfect agreement:

These results demonstrate that human raters maintained a high degree of consistency across tasks, further supporting the reliability of our human evaluation protocol.

---

2. Necessity of Human Evaluation in Open-Domain Tasks

We respectfully argue that human evaluation is indispensable for open-domain chemistry tasks, for the following reasons:

1. Automatic metrics are insufficient: Metrics like BLEU, ROUGE, or embedding similarity only assess surface-level overlap. They cannot reliably detect serious scientific errors, such as incorrect property values, invalid tool usage, or misapplied reaction mechanisms.

2. Chemistry tasks require expert reasoning: Many tasks involve multi-step logic, adherence to domain-specific rules, and scientific justification. These qualities are hard to quantify using purely automated metrics, especially for open-ended or complex instructions.

3. Human evaluation remains the gold standard: As supported by prior work [1–4], expert human judgment remains essential for evaluating synthetic data and LLM performance in scientific domains. In our study, annotators relied on structured criteria and external references to ensure objectivity.

---

3. LLM-as-a-Judge (High Alignment with Human Judgment)

To further validate reliability, we conducted a human–LLM agreement study across three tasks and two strong models (LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct).

• Binary alignment (Property Prediction): up to 99.75%
• Binary alignment (Tool Usage): up to 97%
• Score-based correlation (General QA): average Pearson r = 0.7955, all statistically significant

These results confirm strong agreement between automated and human judgments. Summary tables below:

---

Binary Accuracy – Property Prediction

---

Binary Accuracy – Tool Usage

---

Together, our inter-annotator agreement, expert-informed annotation protocol, and strong human–LLM alignment provide strong evidence for the robustness and fairness of our evaluation process.

---

2. Score-based Evaluation: General Q&A

These high correlations (r ≈ 0.73–0.86) with highly significant p-values demonstrate that our LLM judge is consistent with human judgments even in open-ended tasks.

---

Q1. What specific tools are required for Chemistry Q&A tasks, and how do they enhance LLM performance?

A4: Thank you for this insightful question.

While LLMs can handle many chemistry-related questions reasonably well, they often fall short when dealing with:

• Uncommon or specialized topics
• Out-of-date information
• Facts that require external validation

To address this, ChemOrch integrates web retrieval as the primary tool for Chemistry Q&A tasks. This allows the model to access up-to-date scientific knowledge from reliable sources, verify its outputs with external evidence to reduce hallucinations, and accurately handle rare or niche chemistry questions.

We will expand the explanation of tool roles in Chemistry Q&A in the revised manuscript to make these enhancements more explicit.

---

[1] Huang, Yue, et al. "Datagen: Unified synthetic dataset generation via large language models." ICLR 2024.

[2] Bran, Andres M., et al. "Augmenting large language models with chemistry tools." Nature Machine Intelligence, 2024.

[3] Long, Lin, et al. "On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey." ACL (Findings), 2024.

[4] Mirza, Adrian, et al. "A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists." Nature Chemistry (2025): 1-8.

---

Once again, we deeply appreciate your careful review and constructive comments. If you feel that our responses have sufficiently addressed your concerns, we would be truly grateful if you would consider revisiting your rating. Thanks you!

We sincerely thank the reviewer for the thoughtful feedback. We believe there may be a potential misunderstanding regarding factual correctness in synthetic data generation: even the most advanced frameworks are hard to achieve 100% accuracy, regardless of domain.

R1. On the nearly impossibility of 100% correctness – even in general domains

State-of-the-art general-domain systems such as DataGen [1] also report non-negligible factual and executional errors. This limitation arises from intrinsic challenges such as ambiguous prompts and edge-case failures.

In contrast, chemical instruction generation is substantially more difficult, as it involves precise validation of molecular structures, reaction pathways, and symbolic representations—often with no clear-cut gold standard. Expecting 100% accuracy in this setting sets a bar not demanded in any comparable domain.

R2. ChemOrch’s correctness is both high and conservatively measured

Despite this intrinsic difficulty, ChemOrch achieves 85.14% factual correctness and 82.64% instruction-following accuracy—measured under strict expert-defined criteria:

• No partial credit for incomplete or partially correct answers
• All property values, structures, and tool calls must exactly match trusted sources
• Annotators referenced authoritative databases (e.g., PubChem) during scoring

These thresholds are stricter than prior work in both chemistry and general-domain synthetic generation, making our results especially robust. As a comparison point, baseline LLM generations in our setting achieve ≤63% factual correctness.

R3. Variation across tasks is expected and inherent to difficulty

In our benchmark, certain tasks such as SMILES conversion achieve near-perfect accuracy because they can be solved deterministically via tool calls. The lower accuracy for the most challenging tasks is expected — even expert chemists may not consistently produce correct answers for these, due to intrinsic complexity, open-ended reasoning, and multiple valid solution paths.

R4. Diversity vs. correctness trade-off

Some prior work achieves higher reported accuracy by restricting to a narrow set of easier tasks (e.g., reasoning tasks [2]), which inevitably sacrifices diversity and coverage. In contrast, ChemOrch supports a wide spectrum of chemistry tasks, including:

• Deterministic structure/property lookups
• Tool-augmented open-ended reasoning
• Multi-step synthesis planning
• Complex property prediction

This broad coverage is essential for training LLMs to handle realistic, heterogeneous chemistry scenarios, even if it naturally lowers the global average accuracy.

R5. Correctness enforcement is built into ChemOrch’s design

In ChemOrch, we further incorporate:

• Error detection and self-repairing logic
• Sufficiency validation before early stopping

This leads to substantial error reduction and makes ChemOrch one of the first pipelines to support scalable, verifiable instruction synthesis for chemistry.

R6. Practical effectiveness: correctness sufficient for LLM tuning

Most importantly, the correctness level achieved by ChemOrch has proven to be sufficient and effective for downstream LLM fine-tuning. As shown in Section 4.4, models trained on ChemOrch:

• Outperform baselines across multiple tasks in current benchmarks
• Exhibit improved tool usage and factual grounding

---

[1] Huang, Yue, et al. "Datagen: Unified synthetic dataset generation via large language models." The Thirteenth International Conference on Learning Representations. 2024.

[2] Zhu, Kaijie, et al. "DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks." The Twelfth International Conference on Learning Representations.

---

We sincerely respect your comment and fully acknowledge the importance of factual correctness in synthetic chemistry data generation. While 100% accuracy is inherently nearly unattainable in both general-domain and chemistry-specific pipelines, we have demonstrated that ChemOrch achieves state-of-the-art performance under strict evaluation standards, offers broad task coverage beyond what prior work supports, and delivers tangible downstream benefits. We would be truly grateful if, in light of these clarifications and evidence, you could kindly reconsider your evaluation of our work. Thank you!

We sincerely thank you for the detailed and thoughtful feedback. We address each point in detail below:

Q1: Concern about potential data leakage. Can you show evaluation on external datasets like ChemLLMBench or Mol-Instructions?

Thank you for raising this important concern. We apologize for any confusion and respectfully clarify that this issue is already addressed in the manuscript (Lines 676–678):

“For the fine-tuning experiments described in subsection 4.5, each of the three tasks (property prediction, tool usage, and molecule captioning) includes 400 samples for training and 400 for testing, with the test data sampled from the original benchmark.”

Specifically:

• For Property Prediction, Molecule Captioning and Tool Usage, test sets are fully sourced from ChemLLMBench.
• In the Tool Usage task, test prompts are derived from real-world function documentation (not model-generated), and molecules are randomly sampled from ChemLLMBench. Evaluation is based on exact output match, ensuring objectivity.

Regarding the General Q&A, we did consider using subsets of MMLU and SciBench. However, the chemistry questions in these datasets are mostly focused on mathematical calculations, with relatively few items testing core chemical knowledge. While we acknowledge that our current Q&A set is generated by ChemOrch (as a necessary compromise), we have nonetheless applied data filtering to ensure quality, as stated in Lines 680–681:

“For all testing sets, we conduct a human evaluation to filter out low-quality data points.”

This filtering follows the rigorous protocol in Appendix E. We will clarify this in the revision to ensure full transparency of our evaluation pipeline.

---

Q2: Comparison to ChemBench

Thank you for this excellent suggestion. While ChemBench is a valuable benchmark for systematically evaluating LLMs’ chemical knowledge and reasoning, our ChemOrch framework offers several clear advantages:

1. Scalable data generation: ChemOrch automatically produces diverse, high-quality chemistry instruction–response pairs at scale, without manual labeling.
2. Broader and practical task coverage: It includes not only Q&A but also complex tasks like property prediction and tool-based reasoning with executable outputs.
3. Targeted for model improvement: ChemOrch can generate data tailored to specific weaknesses, supporting both evaluation and fine-tuning.

We now compared ChemOrch to ChemBench, ChemLLMBench, and Mol-Instructions on two diversity metrics:

These metrics indicate that ChemOrch achieves higher instruction diversity than ChemBench.

---

Q3: The evaluation metrics are unclear. Please define them clearly in the main text.

Thank you for pointing this out. We agree that metric definitions are crucial for clarity. Below are the metrics we use, now explicitly described in the revised manuscript:

• Property Prediction Accuracy: Proportion of correctly predicted properties (range: 0–1). Higher is better.
• Tool Usage Accuracy: Based on whether tool outputs match expected function calls (range: 0–1). Higher is better.
• Molecule Captioning: Score on a 1–10 scale. Higher is better.
• General Chemistry Q&A: Score on a 1–10 scale. Higher is better.
• Instruction Following Rate / Factual Correctness Rate: Percentage of responses rated by humans as fully instruction-compliant or scientifically accurate (range: 0–1).
• Diversity Metrics:
  • Average Pairwise Similarity (APS): Lower is better.
  • Remote-Clique Score: Higher indicates broader semantic coverage.

---

Q4: Better performance on chemistry Q&A doesn't necessarily prove domain understanding or reasoning.

Thank you for the thoughtful critique. We agree that higher benchmark scores do not automatically imply true domain understanding or reasoning.

Our original phrasing—“suggesting enhanced domain understanding”—may overstate this link. In the revision, we now clarify that:

“The performance gains indicate improved model accuracy on diverse and reasoning-intensive chemistry tasks.”

---

Q5: Comparison with ChemCrow?

Thank you for highlighting this. We agree that ChemCrow is a notable framework in the LLM-for-chemistry space. However, its goal and output differ significantly from ChemOrch:

Due to these distinctions, a direct performance comparison is not meaningful. However, we have expanded the related work section in the revision to more clearly articulate how ChemOrch complements systems like ChemCrow.

Q6: Please provide more context in the main text about human evaluation.

Thank you for this helpful suggestion. While the full description of our human evaluation setup was previously placed in Appendix G, we agree that its key elements should appear in the main text. In the revised draft, we clarify that a total of 400 samples were assessed by four Ph.D. students with backgrounds in computational chemistry. The two human-evaluated metrics—Instruction Following Rate and Factual Correctness Rate—are both computed using the following formula:

$$\text{rate} = \frac{\text{Number of samples judged as correct}}{\text{Total number of evaluated samples}}$$

A sample is counted as correct only if it fully satisfies the corresponding criterion: instruction adherence or factual accuracy. If a response contains even a single substantive error—whether it deviates from the instruction or includes incorrect scientific content—it is considered incorrect. This conservative scoring scheme ensures consistency across annotators and avoids inflating the evaluation scores due to partial correctness.

---

Q7: Figure 6 shows that LLaMA benefits more from fine-tuning than Qwen. Do the authors have an explanation for this?

Thank you for this insightful question. We believe the discrepancy arises from two key factors:

1. Ceiling effect:
   Qwen starts from a much stronger baseline, leaving less room for improvement via fine-tuning. In contrast, LLaMA has more room to improve.

2. Data–model alignment:
   The style, tokenization, or instruction formulation of the generated data by ChemOrch may align more closely with LLaMA’s pretraining distribution, making it more receptive to fine-tuning with ChemOrch data.

We have included this discussion in the revised version.

---

Q8: The manuscript references “Transition State Search” as one of the tasks, but no example is provided.

Thank you for pointing this out. We have now added a concrete example to the Appendix. Due to the word count limitation, below is a shortened excerpt:

"instruction": "Calculate the transition state energy using the MP2 method for the E2 reaction given the 3D structure: [...]",
"response": "The transition state energy value for the E2 reaction using the MP2 method, based on the provided data, is -179.132.\n\n### Explanation of the Answer\n\n [...]\n\n### Reasoning Process\n\n1. **Understand the Problem**: [...]"

---

Q9: Table 2 shows Molecule Captioning scores above 5, but the caption says the range is 1–5. Please clarify.

Thank you for catching this inconsistency. The actual scoring range is 1–10, as used throughout our human evaluation. The caption in Table 2 was mistakenly written as 1–5. We have modified it in our draft.

---

Q10: Figure 3 would benefit from a comparison of word count distributions to reference datasets like ChemLLMBench and Mol-Instructions.

Thank you for the excellent suggestion. We are unable to provide the updated Fig3 in response (image is not allowed by the rebuttal policy). However, we include the average instruction word counts for tasks in ChemLLMBench, and Mol-Instructions in the tables below. We have also updated Figure 3 in the revised draft as suggested.

---

ChemOrch

---

ChemLLMBench

Once again, we appreciate your insightful and constructive comments. If you feel that our responses have sufficiently addressed your concerns, we would be truly grateful if you would consider revisiting your evaluation. Thanks you!

We sincerely thank the reviewer for the valuable feedback and constructive suggestions, which have significantly improved the clarity and completeness of our work. Below, we provide detailed responses to each point raised:

---

Q1: The paper includes some ablation, but the difficulty reward model ($M\_{diff}$) is not analyzed. This leaves unclear how insightful the authors' understanding of the framework really is.

A1: Thank you for pointing this out. We did place the evaluation of the difficulty reward model in Appendix F (due to the space constraint). Following your suggestion, we have now moved the relevant results into the main paper. Specifically, we provide both distribution comparisons and human evaluation results (see Figure 9 and Table 7 in the draft). The model’s predictions are well-aligned with human judgments, achieving an overall alignment rate of 87.0%, as detailed below:

These results demonstrate the effectiveness of the reward model in accurately capturing instruction difficulty in the chemistry domain.

---

Q2: Figure 6 lacks error bars, and it's unclear if the performance difference between Qwen Vanilla and Qwen Fine-tuned is statistically significant.

A2: Thank you for this helpful suggestion. In response, we conducted multiple runs for the fine-tuning experiments and computed standard deviations. The results are summarized below:

Additionally, for general Q&A experiments (also reported in Table 4 of the paper), the results with standard deviation over three runs are:

These results confirm that the observed improvements are consistent and statistically meaningful.

---

Q3: While the experiments show improvements, it’s unclear how groundbreaking this framework actually is.

A3: Thank you for this valuable perspective. We recognize that some of the language in the manuscript (e.g., “groundbreaking”) may be overly strong. In the revised version, we will adopt a more objective tone, replacing such terms with more neutral alternatives like “novel,” “systematic,” or “comprehensive.”

That said, we would like to respectfully clarify the contributions of ChemOrch, which we believe represent meaningful and distinctive innovations:

Framework-Level Innovation: ChemOrch introduces the first two-stage chemistry-specialized synthetic pipeline—task-controlled instruction generation and tool-aware response construction—specifically designed to address chemistry-specific challenges such as rule alignment, task decomposition, and verifiability (see Sections 1 & 3, Fig. 2).

Advanced Technical Features:

• Difficulty Reward Model: Enables fine-grained complexity control, aligned with human judgments (Sec. 3.1, Appendix F).
• Tool Decomposition & Distillation: Breaks complex tools into atomic components to facilitate accurate tool use and scientifically valid responses.
• Self-Repair & Sufficiency Validation: Ensures robust error correction and response completeness (Sec. 3.3, Fig. 7), not found in previous frameworks.

Empirical Evidence:

• Human evaluation shows high correctness (85.14%) and instruction alignment (82.64%), with over 3× performance gain over GPT-4o (Sec. 4.3).
• Benchmarks generated by ChemOrch reveal LLM weaknesses in underrepresented tasks that other benchmarks miss (Table 3).
• Fine-tuning on ChemOrch data yields significant accuracy gains in chemistry tasks (Sec. 4.5, Fig. 6, Table 4).

Broader Impact: ChemOrch democratizes access to scalable, verifiable chemistry datasets, lowering barriers to developing high-performing, chemistry-aware LLMs (Appendix A).

---

Q4: What is the computational cost of the proposed framework (e.g., $M_{diff}$ fine-tuning, instruction–response generation)?

A4: Thank you for raising this important point. We provide the following clarifications:

• $M_{diff}$ Training Cost:
  The difficulty reward model was trained on 3,390 examples using 4× A100 (80GB) GPUs. The total training time was under 1 hour. We will add this detail to the appendix in the revision.

• Instruction–Response Generation Cost:

  1. As mentioned in Lines 240–250, the cost of generating an instruction–response pair using commercial APIs (e.g., o3-mini) is highly affordable—typically under $0.05 per pair. For fine-tuning on 100 examples, this translates to < $5, or even < $1 if using GPT-4o, making our framework cost-effective.

  2. For local generation using open-source models, we benchmarked several models using Replicate's cloud inference service. A typical instruction–response pair involves approximately 3.5K tokens, and with 5 concurrent processes, generating 100 pairs generally completes within 1 hour for all tested models. The generation speed and latency are summarized below:

     Model	Token/s	Time (s)
     deepseek-ai/DeepSeek-V3-0324	70.42	56.802
     deepseek-ai/deepseek-r1	24.02	166.528
     meta/meta-llama-3.1-405b-instruct	29.74	134.499
     meta/meta-llama-3-70b-instruct	41.73	95.854

  3. The average token consumption for each module in the pipeline is as follows (also illustrated in Figure 4 of the paper):

     Module	Token Consumption
     Embedding Token	120
     Self-Repairing	231
     Code Script	902
     Tool Distillation	1,095
     Web Search	1,106
     Answer Generation	1,165
     Validation	1,472
     Tool Selection	4,094

Together, these metrics confirm that ChemOrch is scalable, computationally efficient, and practical for both commercial API-based and open-source local deployments.

---

Once again, we sincerely appreciate the reviewer’s thoughtful and constructive feedback. Your suggestions have significantly improved the clarity, rigor, and presentation of our work.

If you find our responses helpful and feel that they address your concerns, we would be truly grateful if you would consider revisiting your evaluation.

Thank you very much!

Dear Reviewer,

As the rebuttal period is coming to an end, we would greatly appreciate it if you could respond to our rebuttal and let us know whether your concerns have been addressed. Thank you very much!

Best,

Authors

We sincerely thank you for the valuable feedback and constructive suggestions. Below, we provide detailed responses:

---

Q1: The paper focuses largely on SMILES inputs and outputs. To what extent can the proposed framework be readily adapted to alternative string-based molecular encodings—such as graph-adjacency lists, tree-structured or JSON representations—and what specific effort or modifications would this entail?

A1: Thank you very much for your thoughtful question. Our focus on SMILES inputs and outputs was motivated by its status as the most widely used and canonical representation in cheminformatics, ensuring broad compatibility and comparability with existing literature. However, we fully acknowledge the importance of supporting alternative molecular encodings, such as graph-based, tree-structured, or JSON representations.

Importantly, ChemOrch can be readily extended to handle alternative molecular formats by simply introducing appropriate conversion functions in the preprocessing stage. For example, to support graph-based representations, one only needs to add a transformation module before the main task. Below, we provide a code snippet illustrating how ChemOrch can seamlessly convert a graph representation:

def graph_to_iupac_name(graph):
    mol = Chem.RWMol()
    atom_idx_map = {}
    
    # Add atoms
    for i, atom_info in enumerate(graph["atoms"]):
        atom = Chem.Atom(atom_info["element"])
        atom.SetFormalCharge(atom_info.get("charge", 0))
        atom.SetIsAromatic(atom_info.get("is_aromatic", False))
        idx = mol.AddAtom(atom)
        atom_idx_map[i] = idx

    # Add bonds
    bond_order_map = {
        "single": Chem.rdchem.BondType.SINGLE,
        "double": Chem.rdchem.BondType.DOUBLE,
        "triple": Chem.rdchem.BondType.TRIPLE,
        "aromatic": Chem.rdchem.BondType.AROMATIC
    }

    added = set()
    for a1, neighbors in graph["bonds"].items():
        for a2, bond_type in neighbors:
            if (a2, a1) in added:
                continue
            bt = bond_order_map.get(bond_type.lower())
            if bt is None:
                raise ValueError(f"Unknown bond type: {bond_type}")
            mol.AddBond(atom_idx_map[a1], atom_idx_map[a2], bt)
            added.add((a1, a2))

    mol.UpdatePropertyCache(strict=False)
    Chem.SanitizeMol(mol)
    return pcp.get_compounds(Chem.MolToSmiles(mol, canonical=True), 'smiles')[0].iupac_name

To verify effectiveness, we tested this approach on 20 randomly sampled molecular graphs and achieved a 100% (20/20) success rate in conversion. This demonstrates ChemOrch's strong flexibility and extensibility for handling various molecular representations beyond SMILES.

We will include these implementation details and practical examples in the revised manuscript. Thank you again for your helpful suggestion.

---

Q2: All accuracy numbers are produced with the LLM-as-a-Judge rubric. Without human or rule-based ground truth, evaluation is vulnerable to the same hallucination, bias, and coherence errors it tries to measure.

A2: Thank you for raising this critical point. We would like to clarify that the evaluation in our study is not solely reliant on automated scoring without validation. In fact, we incorporated human evaluation at several key stages, particularly when assessing the quality of generated instruction–response pairs, where extensive human annotation was performed.

Regarding rule-based ground truth: For simple binary tasks, we agree that rule-based metrics are suitable and, where feasible, we will employed them. However, rule-based methods themselves may introduce inaccuracies (e.g., missing relevant keywords), and recent literature has increasingly adopted the LLM-as-a-Judge paradigm for its broader applicability and contextual understanding.

To further assess the reliability of automated scoring, we conducted a human–LLM agreement study across three tasks and two representative models (LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct):

• Binary alignment (Property Prediction): up to 99.75%
• Binary alignment (Tool Usage): up to 97%
• Score-based correlation (General QA): average Pearson r = 0.7955, all results statistically significant

These results confirm strong consistency between automated and human assessments. Please see the summary tables below:

---

Binary Accuracy – Property Prediction

---

Binary Accuracy – Tool Usage

---

We will further detail our evaluation protocol and present additional human–LLM agreement analyses in the revised manuscript to ensure transparency and rigor. Thank you for highlighting this important aspect.

---

Thank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!

Dear Reviewer,

As the rebuttal period is coming to an end, we would greatly appreciate it if you could respond to our rebuttal and let us know whether your concerns have been addressed. Thank you very much!

Best,

Authors