LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

it would be interesting to see how traditional symbolic regression methods would fare on the dataset... which would make the need for LLM-based discovery stronger, and the dataset more useful.

Thanks for the thoughtful suggestion and constructive feedback. Previous work on LLM-based scientific equation discovery (LLM-SR, LaSR, and SGA) has provided evidence for the benefits of these approaches compared to traditional methods. However, we agree that evaluating traditional non-LLM methods on our benchmark would further demonstrate these advantages. In response to your suggestion, we conducted experiments with PySR (a state-of-the-art non-LLM baseline) on our benchmark datasets during the rebuttal period. The results (shown in table below) demonstrate that PySR often achieves competitive numeric accuracy but considerably lower symbolic accuracy, particularly on non-physics datasets. This confirms that while traditional methods can fit data numerically, they struggle with symbolic understanding due to lack of domain knowledge. We also think that these findings further motivate the value of LLM-based techniques for scientific equation discovery. Thank you again for the constructive feedback. We will surely include this analysis in the final version.

Incoporation of noise Thank you for the comment. We have not explored the imacpt of noise in the our benchmark's data generation. The main motivation of this benchmark was to help community towards building better general LLM-based equation discovery agents that can be leveraged in scientific domains, addressing the challenges of current benchmarks for the emerging LLM-based techniques. We fully agree with the reviewer on the importance of noise in real-world scientific discovery scenarios. This is one of the directions we're considering for future benchmark enhancements.

R2 vs Accuracy to Tolerance Thanks for raising this important question. Based on our analysis (and some of the previous works [Kamienny et al., 2022; Biggio et al., 2021]), R2 is not a good metric for the evaluation of symbolic regression, particularly if we have large-scale synthetic data with very different output scales.

• R2 can be easily saturated by normalized mean patterns (similar to NMSE), often missing prediction nuances at different scales. This is evidenced by the consistently high R2 scores (above 0.999) achieved by most recent methods in benchmarks like SRBench.
• The accuracy-to-tolerance metric (defined in Section 2.3) aggregates point-wise normalized metrics for each datapoint rather than normalizing mean aggregated scores. It also implements a tolerance threshold for the worst point-wise normalized distance, making evaluation more robust to function behavior nuances (e.g., sudden changes or spikes) As the goal of symbolic regression is to learn correct underlying mathematical relations, we should care more about these nuances of function numeric behavior as well as the symbolic mathematical alignment which makes accuracy to tolerance a better metric for numeric precision assessment.

Evaluation of Equation Semantic Similarity with LLMs vs more classical methods Thank you for the comment. We will surely add more details on equation semantic similarity evaluation in the updated version. Regarding the question, we think that restricting the outputs of the equations to specific forms for LLM-based equation discovery won’t be a good idea since one of the main benefits of LLMs is their capabilities in programming and code generation which opens new more flexible representations for equation discovery that was not possible with the previous expression tree based methods and their corresponding symbolic evaluations. While evaluating semantic similarity is non-trivial, our empirical analysis and human study show that LLMs are usually strong at recognizing equation equivalence across different styles and representations. We also think that correlation between symbolic accuracy (computed from GPT-4o) and generalization (computed from OOD data) shown in Figure 6 further support LLMs effectiveness as semantic symbolic evaluators for equation discovery.

---

We hope that most of the reviewer’s concerns have been addressed. We’d be happy to engage in further discussions.

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

A systematic classification of failure cases across different domains would provide valuable insights. Examining where and how models fail—whether due to misidentified variables, incorrect functional forms, or numerical instability

Thank you for the constructive suggestion. We examined the failure cases during rebutal period and noticed that LLM-based models usually don't fail due to the syntax function errors (such as missing variables or dependencies). Instead, most errors come from incorrect terms or combination of terms within the equations, reflecting mistakes due to semantic aspects of discovery for a given scientific problem. We could not identify a uniform pattern of specific mathematical relations that consistently challenge current discovery methods. We believe this is because LLM-based equation discovery involves a complex interplay of domain knowledge, search capabilities with data-driven feedback, and mathematical manipulation skills. To thoroughly study failure case patterns would require investigating all these dimensions. We agree with the reviewer on the importance of this research direction for future work.

Representative Failure Examples Thank you for this suggestion. We will include examples of failure cases for different methods across domains in the camera-ready version.

The paper provides evidence that generalization performance varies across scientific domains, ... However, it does not analyze why this variation occurs.

Thanks for the thoughtful question. We have explored problem complexity across domains (Figure 10 in Appendix shows the distribution by complexity). While biology and physics problems have similar and slightly higher complexity than other domains, we observe a smaller ID-OOD gap for physics but a larger one for biology in the results. This suggests factors beyond mere complexity are at play (like LLM domain knowledge). The reviewer raises an interesting point that would require domain-specific analysis beyond our study's scope. Our primary goal is to develop a benchmark that helps the community build better general LLM-based equation discovery agents applicable across domains. More specialized applications would require domain-focused analyses

Novelty Check...In the LSR-Synth pipeline, the authors use GPT-4o as a novelty evaluator to determine whether a generated equation is distinct from known scientific expressions. However, LLMs may incorrectly classify an equation as novel despite it existing in prior literature. .. this does not guarantee that the novelty check is reliable.

We cannot theoretically guarantee whether GPT-4o evaluation of novelty is reliable, but evaluating novelty is a non trivial task and we think that LLMs (with their vast knowledge about literature) could be a helpful tool for novelty assessment. Our empirical evidence also supports the effectiveness of this approach: the consistently lower performance of discovery methods on LSR-Synth datasets suggests these problems aren't trivial and do contain novel elements (if they were simply known equations, LLM-based discovery models would likely recall them from embedded knowledge and solve them easily). We agree that more rigorous novelty evaluation involving domain experts and specialized scientific literature retrieval tools would enhance the design of benchmark problems. However, this extension would require significant domain knowledge and human expert review which is not feasible for the rebuttal period but valuable for future work.

Dataset Size Reduction in LSR-Transform Here is the detailed breakdown of our filtering process in LSR-Transform: After transformation steps in Figure 3 (Step 4), we obtained 471 transformed problems from the 100 original Feynman problems; During the solvability check with sympy (Step 5), 53 problems were discarded, leaving 418 problems; In dataset refinement (Step 6), we only filtered datapoints to ensure they were within the domain of the new transformed equations, without eliminating any equations at this stage. We then filtered out 307 problems due to significantly higher complexity compared to original problems, resulting in the final 111 problems in LSR-Transform. This is to ensure that the challenging nature of LSR-Transform stems from semantic aspects of discovery rather than from syntactically more complex or lengthy problems (as shown in Figures 4 and 8). We will clarify this process further in the revised version of paper.

Additional Qualitative Examples of Outputs As some reviewers have higlighted examples of Figure 14 helpful, we plan to include more hypothesis examples from different domains in the revision of paper.

---

We hope that most of the reviewer’s concerns have been addressed. We’d be happy to engage in further discussions.

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

• However, it would be good to include more detailed case studies or error analyses to better understand which types of equations or mathematical patterns pose the greatest challenges for current methods.
• While showing that current methods struggle, there's limited analysis of why they fail or which specific reasoning capabilities are lacking

Thank you for the thoughtful suggestion. We examined the failure cases during rebutal period and noticed that LLM-based models usually don't fail due to the syntax function errors (such as missing variables or dependencies). Instead, most errors come from incorrect terms or combination of terms within the equations, reflecting mistakes due to semantic aspects of discovery for a given scientific problem. We could not identify a uniform pattern of specific mathematical relations that consistently challenge current discovery methods. We believe this is because LLM-based equation discovery involves a complex interplay of domain knowledge, search capabilities with data-driven feedback, and mathematical manipulation skills. To thoroughly study failure case patterns would require investigating all these dimensions. We agree with the reviewer on the importance of this research direction and consider it a promising avenue for future work.

The correlation between symbolic accuracy and OOD performance is intriguing. Does this relationship hold equally across all scientific domains, or are there areas where symbolic understanding is less predictive of generalization?

Thank you for this thoughtful question. We have actually explored the correlation between symbolic accuracy and OOD performance across different domains (detailed in Figure 13 Appendix). Our results demonstrate that this positive correlation holds across all four domains. Symbolic understanding appears to be a reliable predictor of generalization capability in equation discovery.

How sensitive is performance on your benchmark to the quality or quantity of provided data? In real scientific scenarios, data is often limited or noisy - have you explored this dimension?

We have not yet explored the impact of noise or data limitations in our current benchmark generation process. Our primary motivation was to develop a benchmark that helps the community build better general LLM-based equation discovery agents for scientific domains, addressing limitations in existing benchmarks for emerging LLM-based techniques. We fully agree with the reviewer on the significance of noise and limited data in real-world scientific discovery scenarios. This is one of the directions we're considering for future benchmark enhancements.

Your benchmark focuses on discovering equations given data and context. Have you considered extending it to evaluate LLMs' abilities to generate plausible hypotheses before seeing complete datasets, which is another important aspect of scientific discovery?

We would like to clarify that our benchmark evaluates SOTA methods that already incorporate a two-phase approach: hypothesis generation followed by data-driven validation. In the first phase, LLMs generate plausible equation hypotheses without seeing the data, which are then refined based on how well they fit the data. We agree that evaluating pure hypothesis generation capabilities is an important dimension of scientific discovery. While our present focus has been on end-to-end data-driven scientific equation discovery, extending the benchmark to explicitly measure hypothesis quality before the data validation is also a valuable direction for future work. We also think that combining mathematical derivation processes with data-driven reasoning represents an interesting avenue for future research that has not yet been thoroughly explored in current methods.

The examples of output hypotheses in Figure 14 provide good qualitative insights

As some reviewers have higlighted this example helpful, we plan to include more hypothesis examples from different domains in the revision of paper.

• Could benefit from more discussion of how its findings might influence future LLM designs to improve scientific reasoning
• Would be nice to see more discussion around if some of the equation transformations in LSR-Transform, while mathematically valid, are meaningful from a scientific perspective

Thank you for the suggestion. We will make sure to add more discussion regarding these points in the camera-ready version.

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

• there's no systematic exploration of whether different methods have domain-specific advantages.

We agree this is an important consideration, but it falls outside our study's scope. Our benchmark aims to help community towards building better general LLM-based equation discovery agents applicable across domains. Domain-specific variations mostly come from the LLM's knowledge rather than the agentic discovery framework itself. Further experiments with different LLM backbones would be helpful to study this question in more depth.

Do you have either intuition or insight into specific criteria for when numeric performance and symbolic accuracy metrics diverge?

Thank you for the thoughtful question. This is indeed a common challenge in equation discovery. Equations with good numeric accuracy may differ significantly in symbolic manner. Conversely, equations with similar symbolic structures can also exhibit significantly different numeric behaviors due to constant/coefficient variations affecting function behavior. This is why our benchmark incorporates both numeric and symbolic metrics for comprehensive evaluation.

The benchmark design assumes that transforming equations (LSR-Transform) and combining known with synthetic terms (LSR-Synth) effectively models scientific discovery. How do you justify that these approaches genuinely reflect how scientists discover new equations in practice, rather than simply creating mathematically challenging problems? Would performance on these tasks predict success in real scientific discovery?

Thanks for raising this important question. Whether performance on these tasks predicts success in real scientific discovery remains an open research question. But we think that our benchmark can be a good starting point to guide models towards that direction, particularly in the task of scientific equation discovery. To ensure our benchmark reflects scientific discovery rather than just creating mathematically challenging problems, we implemented filtering steps in the design of benchmark (detailed in Section 2, Figures 4, 8, and 10) to avoid lengthy problems with excessive mathematical and syntax complexity.