G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration

We sincerely thank the Reviewer for the valuable and constructive feedback. Below, we address each concern and outline key improvements in the revised manuscript.

---

1. Experimental Validation and Benchmark Fairness

We acknowledge the concerns regarding validation and benchmark selection:

• New Metrics: We have conducted additional experiments beyond Wasserstein distance, we now include Mean Squared Error and Maximum Mean Discrepancy (MMD) metrics for a more comprehensive assessment, https://imgur.com/a/yx228xx.

• Computational Fairness: To ensure fair comparisons, we will explicitly report training times and number of parameters for all baselines, as seen here: https://imgur.com/a/pgdvwWT.

2. Manuscript Clarity and Presentation

We fully agree with the suggested improvements for clarity and readability. Specifically:

• Concise Abstract: The abstract will be shortened, succinctly summarizing existing method limitations and explicitly emphasizing our main contribution (LLM-guided, gradient-free simulator generation). Technical details will move entirely into the main text.

• Formatting Improvements: We significantly reduce itemized/enumerated lists and paragraph breaks:

  • Merged "Current methods" and "General-purpose simulators" into a streamlined narrative introduction.
  • Replaced itemized "Key properties" and "Contributions" with concise narrative descriptions.
  • Removed unnecessary paragraph breaks (\paragraph{}) and lists on page 4.

• Method Earlier in Manuscript: To enhance readability, we move the "G-Sim: Hybrid Simulator Construction" section from page 5 to page 4, preceding related work.

• Improved Fig. 1 Caption: Revised caption for Figure 1 to be self-contained and clear:

  "Overview of G-Sim, an automatic simulator-generation framework integrating LLM-derived domain knowledge and empirical data. G-Sim iteratively: (1) proposes simulator structure using domain-informed LLM, (2) calibrates parameters via gradient-free optimization, and (3) refines through diagnostics (predictive checks, stress-tests, LLM-reflections), enabling robust ‘what-if’ scenario analyses."

• Minor Clarification: Line 284 revised explicitly as: "imbalances (Rauba et al., 2024)."

---

3. Detailed Responses to Reviewer Questions

Parameter Fitting: In Appendix E. 3, we explicitly clarify gradient-free evolutionary optimization (population initialization, mutation strategy, selection criteria, and convergence checks). We minimize MSE using EvoTorch's genetic algorithm (population 200, simulated binary crossover, Gaussian mutation, and tournament selection). Furthermore, we warm-start reusing the best parameters from the previous iteration where possible.

Score Function Clarification: The "Score" quantifies discrepancies between simulated and observed data in our implementation via MSE, as detailed explicitly in Appendix E.3.

Theoretical Guarantees: We acknowledge the lack of rigorous global optimality guarantees but highlight known convergence properties of evolutionary methods towards global minima under suitable conditions [1]. We will explicitly include this discussion, clearly indicating assumptions and limitations.

[1] Rudolph, G. (1994). IEEE Trans. Neural Networks, 5(1), 96-101.

Clarifications on Section 4.3:

• Diagnostic formulation ($\\delta$): Diagnostic discrepancies ($\\delta$) aggregate multiple metrics—predictive accuracy (e.g., MSE, Wasserstein distance) and domain constraints (non-negativity, cyclical patterns). Appendix B.3 explicitly details these diagnostics.

• Threshold $\\varepsilon$ Selection: We implicitly set $\\varepsilon$ via iteration limits. We will clarify this explicitly in the manuscript.

• LLM Feedback Robustness: Empirical validations (App. I.2 iterative refinement logs; App. E.4 prompt templates) show LLM feedback consistently improved accuracy, demonstrating robustness across diverse diagnostic scenarios. This evidence will be explicitly highlighted.

---

4. "What-If" Scenario Comparisons

Baseline methods inherently lack structured mechanisms for performing counterfactual ("what-if") analyses. Thus, such scenarios are uniquely suited for our generative framework, clearly distinguishing G-Sim's capabilities. We will clarify this explicitly in the revision.

---

Summary of Improvements

• Expanded experimental validation.
• Streamlined abstract.
• Improved manuscript readability.
• Clarified parameter-fitting and diagnostics.
• Enhanced clarity of Sec 4.3.
• Discussion of convergence.

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We sincerely thank Reviewer for their insightful and constructive feedback, which significantly helps strengthen our paper. Below, we address each major concern explicitly and outline concrete improvements for the camera-ready version.

---

Baseline Tuning

We appreciate the reviewer’s concern about baseline comparisons. The baselines' hyperparameters are standard, and both the baselines and hyperparameters are similar to recently published works on similar simulation tasks, coming from the Hybrid Digital Twin work (L 245).

---

Structural and Causal Accuracy

The reviewer rightly emphasized the importance of validating structural and causal accuracy. Our intervention experiments implicitly verify structural correctness; simulators that produce accurate predictions under unseen interventions strongly indicate correct causal structure.

Additionally, we now include quantitative evaluations (F1 Score and Structural Hamming Distance - SHD) against their known causal structures. Specifically, the Supply Chain environment perfectly matches ground-truth causal relationships (F1=1.0, SHD=0), while Hospital and SIR environments yield F1 scores above 90% and minimal SHD (≤1.5), demonstrating that G-Sim reliably captures causal relationships. These metrics and their analysis will be prominently featured in the updated manuscript.

---

Gradient-Free Optimization (GFO) vs. Simulation-Based Inference (SBI)

The reviewer's suggestion regarding SBI methods is highly pertinent. Initially, we adopted evolutionary-based GFO primarily for scalability reasons, given its suitability for non-differentiable, stochastic simulator components. Nonetheless, we agree that SBI's uncertainty quantification and robustness are valuable. We have since integrated SBI into G-Sim, testing it on the COVID-19 environment. Preliminary results indicate comparable performance to GFO, with added benefits of uncertainty estimation. A detailed comparison between SBI and GFO, along with explicit recommendations for future work, will be included in the revised manuscript (Sections 1 and 4.2).

---

Contributions

The reviewer correctly identifies that our primary contribution is the innovative integration of existing techniques (LLM-driven structural reasoning combined with GFO).

---

Intervention Modeling

Interventions are directly modeled by explicitly modifying parameters in the transparent, human-readable code modules generated by the LLM, as described in 6.1. Our design ensures robustness under interventions outside the training distributions, providing valuable "what-if" analyses. We will expand on these scenarios in a dedicated appendix, explaining detailed examples to enhance clarity.

---

Prompt Engineering Effort

Prompt engineering required modest domain-specific adjustments, leveraging generalizable, environment-agnostic prompts (provided in Appendix E.4) supplemented with concise, environment-specific descriptions (Appendix F). Our approach minimizes the need for extensive expert intervention. We will explicitly discuss the extent of prompt engineering required in our revision. We optimized the prompts to support and understand the tool calls and iterative workflow.

---

Scalability

We recognize scalability as an essential consideration. Scaling discovery is often an NP-hard problem, and we also find that it is difficult to scale. However, future work can exploit decomposable structures, and we can scale with the number of parameters to optimize, inheriting the same scalability as ES.

We will add an expanded discussion of scalability and practical examples detailing scalability in a new appendix.

---

Ethical Considerations

We appreciate the reviewer's concern regarding ethical implications. Our approach prioritizes transparency, allowing easy inspection, stress-testing, and expert verification of simulator outputs prior to deployment. As explicitly stated in our Impact Statement (L2182-2199), we emphasize the critical role of domain expertise, transparency in assumptions, and rigorous oversight. To further address these concerns, we will include an expanded discussion on ethical considerations, biases, and mitigation strategies in a dedicated appendix.

---

Summary of Revisions:

• Clarified baseline comparisons.
• Added explicit quantitative evaluation of causal and structural accuracy.
• Provided a new experiment comparing GFO and SBI.
• Enhanced explanation and examples of intervention modeling.
• Clarified prompt engineering effort and detailed scalability strategies.
• Included thorough discussion of ethical considerations and mitigation strategies.

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We sincerely thank the Reviewer for their detailed and constructive feedback. We fully agree on the importance of clearly presenting our methodology within the manuscript itself. To comprehensively address these concerns, we will allocate the additional page in the camera-ready version specifically to enhance clarity. Below, we provide a shortened clarification of our method, explicitly referencing relevant sections and appendices, and address each of the reviewer's comments directly.

Clarification of the Proposed Method:

G-Sim automates the generation of domain-specific simulators by integrating structural proposals from Large Language Models and numerical calibration via Gradient-Free Optimization. The process involves an iterative loop between three distinct phases, outlined in Algorithm 1 and described in Section 4 and Appendix E.2:

1. Structural Proposal via LLM (Section 4.1 and Appendix E.4):

   • The LLM generates Python code to outline the simulator’s structural components, guided by domain-specific textual descriptions and available background knowledge.
   • Structural proposals explicitly specify modular Python functions and classes capturing domain-relevant mechanisms, such as epidemic spread dynamics or resource distribution processes. Importantly, the LLM defines the structural templates without numerical parameters, which are left open for subsequent calibration.

2. Parameter Calibration via Gradient-Free Optimization (Section 4.2 and Appendix E.3):

   • Following structural generation, numerical parameters within these LLM-generated modules are independently calibrated using gradient-free methods, such as evolutionary algorithms, to align the simulation outcomes with empirical observations.
   • This parameter optimization step is entirely decoupled from the structural proposals made by the LLM, which we hope clarifies potential confusion between the structural and numerical stages of the simulator generation process.

3. Iterative Refinement via Diagnostic Feedback (Section 4.3 and Appendix E.4):

   • Simulations produced by the above steps are then evaluated against the available empirical data to identify discrepancies or inadequacies.
   • These evaluation outcomes are synthesized into textual feedback (for example, "Simulator lacks weekly seasonality, consider incorporating periodic seasonal modules"), guiding the LLM to refine structural proposals.
   • This iterative loop of evaluation, feedback, and refinement proceeds automatically until the resulting simulator meets the desired levels of accuracy and domain plausibility.

The combination of the symbolic and optimization approaches ensures G-Sim remains robust, transparent and precise, as justified empirically, offering strong generalization capabilities even in out-of-distribution scenarios.

Extent of Automation:

• The iterative refinement loop described above is fully automated once initial contextual inputs are provided, and our experiments follow this approach. However, the framework is flexible enough to optionally accommodate human expertise for interpreting nuanced feedback or integrating additional domain-specific constraints.

Explicit References to Examples in Camera-Ready Version:

• Explicit examples of the generated Python code and the iterative refinement process are already detailed in Appendix I.2. We will provide clearer cross-references to these examples within the main manuscript, enabling the reader to readily access practical demonstrations of structural proposals and refinement iterations.
• Moreover, Appendix F presents detailed prompts alongside the environment specifications provided to the LLM, illustrating the exact inputs guiding structural generation. We will reference these more explicitly in the main text to further clarify how our methodological approach leverages domain knowledge.

Committed Enhancements for Final Manuscript:

• A dedicated additional page in the manuscript clearly describing each component of our method: structural proposals (Appendix E.4), gradient-free parameter calibration (Appendix E.3), and diagnostic iterative refinement (Algorithm 1 in Appendix E.2).
• Explicit references within the main text to practical illustrative examples (Appendix I.2) and to detailed LLM prompts and environment details (Appendix F).
• Clarification on the scope of automation and the optional role of domain expertise.

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.