Symbolic Regression with a Learned Concept Library

Thank you for taking the time to review our work! We address your questions inline:

It's not clear how valuable the concept abstraction approach is compared to some "baseline" of simple LLM prompting. For example, one baseline could just use a single concept "This expression is a good formula for symbolic regression" or something like that, and see how it compares. This could perhaps be a direction for future work: try a bunch of simple prompting strategies for combining LLM's with PySR, and report how well each of them work.

Good suggestion! This strategy – fixing a static (but highly domain specific) prompt – usually causes the LLM to regurgitate low performing programs and requires a bigger, slower model to work properly. One of our ablations is very close to the suggested experiment: in Figure 3 Left, we compare against LaSR with no concept evolution (in this case, “the fixed prompt” is just the system prompt) . Nevertheless, LaSR paves the way to try out exciting techniques from the prompt programming [1] / prompt tuning community for solving scientific discovery problems.

I'm not sure what to make of the synthetic dataset. In particular, PySR works so well on AI Feynman alone, but works very poorly without the LLM addition on this synthetic dataset. Why does LaSR work so much better than PySR here, but not help as much on AI Feynman? One hypothesis is that AI Feynman has a lot of easy tasks, and the synthetic benchmark only has tasks right on the edge of what PySR can discover, which LLM incorporation helps push over the edge.

Thank you for pointing this out. Randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. To properly verify the utility of language guidance, we generated the synthetic dataset equations to be within the upper limit of PySR’s performance. To construct this dataset, we first generated equations which, anecdotally, contain many characteristics that PySR struggles with. Then, we verified PySR’s performance on this dataset for 400 iterations to ensure there weren’t trivial bugs in the equations (such as the equation evaluating to zero). This left us with 41 equations, for which we report LaSR’s performance with the same hyperparameters.

Another pessimistic take is that the synthetic benchmark is designed with LaSR in mind. While this still eliminates worries on data leakage, an explanation here would help understand better how LaSR is being helpful.

Thank you for pointing this out. As mentioned before, randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. We deliberately want to pick equations that would be hard for PySR to properly verify the utility of language guidance. We’ve edited the synthetic dataset section to clarify the data construction process, which explains PySR’s poor performance.

Discovering what is known…

Our experiments on synthetic data suggest that we can use LaSR to make discoveries even without significant prior domain knowledge. In addition, since the submission, we have been actively exploring ways to use LaSR in novel discovery tasks. For example, we currently have exciting preliminary results on automatically discovering novel LLM scaling laws using LaSR. Such real-world case studies require deep collaboration with domain experts and are hence outside the scope of the present foundational effort. However, we anticipate many such results in the future.

Lack of Running Time

Our goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time.

This is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for gpt3.5 we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic so the wall time will be substantially different across experiments.

Regardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, and as the hardware to run LLM inference improves.

[1]: https://github.com/stanfordnlp/dspy

Thank you for taking the time to review our work! We address all questions inline:

Claims that need to be fixed.

We would be happy to adjust these claims along the lines you suggest. Thank you for pointing this out!

In line 83, shouldn't it be P(C) instead of P_C?

Good catch! Fixed.

In the mutation operator, it is stated that for each deleted subtree, one adds a single leaf node (lines 111-112).

This isn’t correct. We’ve updated the explanation for delete_subtree on line 111. In PySR’s implementation, the delete_subtree function randomly selects a single node from the subtree, deletes that node, and replaces the deleted node with one of its children (selected randomly).

In the for-loop in line 4 for Algorithm 1, shouldn't it be I instead of N?

Yes, it should be. Fixed in the latest manuscript.

Why not provide a figure in the appendix to describe Concept Evolution (similar to figures 4 and 5)?

Thank you for the suggestion! We added a figure for concept evolution in the appendix. The prompt should be available in the provided repository.

How does that cascade work for PySR? (the one that didn't work)

The PySR cascade operates in a similar fashion to the LaSR cascade. We run PySR for 40 iterations, prune equations that are below a threshold solution, and repeat the process the same number of times as we do for LaSR. Pruning and rerunning PySR should have no effect on the performance but we do this anyway for completeness.

Why use MSE and not the exact correct metric in Section 4.3?

Due to rounding errors of the data, waiting for a MSE error of exactly zero (an exact match) is not ideal.

Why is the difference between PySR and LaSR so large in Table 3?

Thank you for pointing this out. Randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. To properly verify the utility of language guidance, we generated the synthetic dataset equations to be within the upper limit of PySR’s performance. To construct this dataset, we first generated equations which, anecdotally, contain many characteristics that PySR struggles with. Then, we verified PySR’s performance on this dataset for 400 iterations to ensure there weren’t trivial bugs in the equations (such as the equation evaluating to zero). This left us with 41 equations, for which we report LaSR’s performance with the same hyperparameters. We’ve edited the synthetic dataset section to clarify the data construction process, which explains PySR’s poor performance.

Weren't the sentences in Section 4.4 swapped? To me, the first sentence looks more refined than the second. This is because it mentions "waveforms or periodic phenomena," while the second only talks about "scientific phenomena."

The second sentence is, in fact, refining upon the first sentence in some ways. For instance, the second sentence mentions more mathematical operations (multiplications, division, and trigonometric operations) than the first sentence.

This is a common challenge with using LLMs: some parts of the generated concept undergo more refinement than other parts do. And it’s hard to quantitatively contrast two natural language concepts. Nevertheless, this additional generation flexibility is advantageous in many situations because it enables the LLM to abstract similar parts of the sentence between iterations.

In the prompt shown in Figure 5, how much is the excerpt "relation to physical concepts" contributing to the quality of the concepts?

We didn’t find any performance difference including/excluding that phrase from the prompt. In our experience, the instruction fine tuned models are most sensitive to the initial instructions, the formatting instructions, and the bullet point list of concepts/equations.

Thank you for taking the time to review our work! We address questions inline:

Introduction (...) algorithms.

We agree that LLMs might require additional compute, which can raise the cost. However, this is a worthwhile tradeoff for scientific discovery for a couple of reasons:

• The expected reward of finding even one additional equation is extremely high. As LaSR consistently outperforms PySR, the additional cost of running LaSR is worth the tradeoff.
• We are optimistic that the cost of running LaSR will further decrease – not increase – as local language models become faster, smaller, more accurate, and cheaper in the near future [1]. We are even able to run LaSR on a laptop with the latest quantized models!
• LaSR usually considers a larger space of equations than a traditional symbolic method in the same number of iterations. Furthermore, LaSR produces two artifacts: an equation of best fit as well as a library of concepts the method finds relevant to discovering that equation. These provide invaluable context information to a scientist.

(...) black-box environment, the backbone (...) traditional evolutionary algorithms might perform better because they do not have this kind of knowledge as the constraint.

This is exactly the setting of our Synthetic dataset experiment. We procedurally generate novel synthetic equations, and their respective data points, that lie outside the distribution of known equations, and find that LaSR is able to solve more equations than PySR – an ablation of LaSR without language guidance.

The reason LaSR works in such scenarios is because LaSR is a neurosymbolic approach which uses a classical evolutionary search as the backbone and then uses the LLM to direct this search. In the synthetic domain, the novel synthetic blackbox equations generated through the evolutionary search still exhibit abstract mathematical trends, and LLMs can discover these trends, and then use these trends to guide further search.

What's the operation of LLMINIT? If we do not have any prior knowledge, then what will we prompt LLMs to do the initialization? Is it the same when we nothing more to know in advance.

Great question! In the absence of prior knowledge / hints, we fall back to the symbolic initialization function. The full prompt is available in the linked repository.

Have you ever tried to use merely LLMs for the whole process, that would be an interesting topic.

Thank you for the reference! Using an LLM for the whole process would not be ideal in cases where we do not have much prior concrete knowledge about the problem. For instance, in a completely black box setting, in LaSR, the LLM helps guide the evolutionary search using abstract concepts extracted by the classical evolutionary operations while exploring the “local” search space.

Nevertheless, LaSR paves the way to try out exciting techniques from the prompt programming [2] / prompt tuning community for solving scientific discovery problems.

high-dimensional symbolic regression tasks

For high dimensionality problems, the SR community relies on the model distillation paradigm [3]. Here, we first fit a sparse GNN (or any large parametrized network with a sparsity constraint) to the data and then distill each network subcomponent into an independent equation. As LaSR is a drop-in replacement for PySR, a practitioner can use LaSR with this methodology to learn equations in high-dimensional tasks with almost no code changes. We leave this for future work.

Comparison with KAN

Great suggestion! KAN’s [4] take a different, orthogonal approach as compared to our work for symbolic regression. Symbolic regression with KAN’s relies on a two-step model distillation strategy similar to [3]. First, they fit a sparse KAN to the dataset and then they use a “search” component to find the closest fit symbolic function for each learned activation (the search component here is iterative greedy refinement).

Consecutively, the closest direct comparison of LaSR with KAN’s would be to measure the efficacy of greedy iterative refinement compared to LaSR for model distillation. This is an exciting direction for future work!

[1]: https://arxiv.org/abs/2407.21783

[2]: https://github.com/stanfordnlp/dspy

[3]: https://arxiv.org/abs/2006.11287

[4]: https://arxiv.org/pdf/2404.19756

Thank you for taking the time to review our work! We address questions inline:

Could you provide the 7 additional equations that LASR solves ...?

We’ve added a table containing these equations and the ground truth equation to the global PDF for figures.

Are these 7 equations consistent across runs?

Yes, LaSR discovers these equations consistently across runs.

What have been specific contributions of the LLM for them?

Due to the nature of evolutionary algorithms, it is extremely challenging to trace how each operation (whether it be an LLM-based operation or a symbolic operation) interacts with other operations to evolve a candidate pool. Intuitively, the LLM operations help introduce new candidates in the population that respect the natural language hypotheses (whether learned or user provided). LLM-biased candidates that are a good fit are retained by the evolutionary search, while bad candidates are discarded.

We refer the reviewer to the qualitative study in the Appendix of a Feynman equation task, where we compare the equation LaSR finds with the equation PySR – an ablation of LaSR without the LLM operations – finds. Notably, we find that LaSR’s equation is more interpretable, yet is very far from the common formatting of the equation (available on the internet and, possibly, in the training dataset of the LLM).

How can you ensure that the LLM is not implicitly using its prior knowledge ....

We refer the reviewer to Figure 3 (Left) where the ablation of LaSR without the variable names and the hints still outperforms PySR.

We find that more language guidance consistently improves LaSR’s performance (by adding variable names, by adding hints, by using a larger model, or by calling the LLM more frequently).

Furthermore, In Appendix A.4.1, we analyze the equations discovered by PySR and LaSR from data corresponding to Coulomb's law. If LaSR’s performance was rooted in regurgitating memorized data, LaSR’s equation would correspond to the common formatting of Coulomb's law (F= (1/(4 * \pi * epsilon)) q_1.q_2/r^2). Instead, we observe that LaSR’s discovered equation displays a very different format, and needs at least five simplification steps to transform to the commonly seen equation.

Have you conducted ablation studies...

Yes. Figure 3 (Left) depicts an ablation of LaSR without any physics knowledge (variable names or hints). This ablation still outperforms PySR. Also, one of our prompts contains the term “physical concepts.” We’ve verified that inclusion/exclusion of this phrase has no impact on LaSR’s performance.

For equations solved by both LASR and PySR, could you provide the number of iterations required to obtain the correct form using each method?

Figure 3 tracks the number of equations solved after each iteration for PySR, LaSR, and various ablations of LaSR. Our experiments are fully reproducible, and we plan on releasing our logs with our codebase, which should highlight more details.

Could you clarify what constitutes an "iteration" ... There are discrepancies between ... operations will be used?

An iteration is one full run of the SR cycle, and the concept crossover steps (this is best expressed in Algorithm 1). $10^6$ in Figure 1 is the total number of operation calls per iteration. As mentioned in Appendix 3.1, LaSR makes $60000$ operation calls per iteration, $60000$ calls/iteration * $40$ iterations = $2.4 \times 10^6$ calls. We have updated Figure 1 to say “$10^6$ operations” instead of “$10^6$ iterations.” Thank you for pointing this out!

We cannot run LaSR (or PySR) for $10^6$ iterations, as that would take roughly $10^6$ to $10^7$ minutes on an 8 x NVIDIA A100-80GB server node. As mentioned in Appendix 3.1, assuming $p=0.01$, we will make $60000$ calls per iteration so $0.01 * 60000 = 600$ LLM operation calls (roughly) per iteration.

For the data leakage validation, ... black-box datasets in SRBench?

Great question! We wanted to compare PySR’s performance and LaSR’s performance on a set of unknown problems with ground truth equations available, which is why we chose to procedurally generate our own equations. The black box datasets in SRBench do not have ground truth equations.

Could you provide a computation time comparison?

Our goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time.

This is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for gpt-3.5-turbo we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic, so the wall time can be substantially different across experiments.

Regardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, run LaSR on dedicated backend hardware, and as the hardware and software to run LLM inference improves.

(...) examples of "hints" in Feynman equations?

As mentioned in Section 4.4, we use the title of the chapter the equation appears in in the Feynman Lectures on Physics. For instance, the Shockley diode equation is introduced in the “Semiconductors” chapter of the Feynman lectures. Hence, the hint would be “Semiconductors.”

Based on Table 2, (...) Could you explain why higher values were not explored?

As we mention in the limitations sections, our evaluation was constrained by our compute budgets for LLMs and search. Whether these trends hold for higher compute regimes remains to be seen.

Thank you for engaging with us and helping us improve this work! Apolgies for the delay! PySR experiments took a while. We respond to your points inline:

ground truth equations

We’re happy to include the LaSR’s discovered equation and the ground truth equations in the broader artifact release. We did not report on exact match for ground truth equations because neither the equations discovered by PySR nor those by LaSR were close enough to regard as “exact matches.” Instead, we relied on correlation metrics used in the SR community specifically for this circumstance. As shown in Table 3, we find that LaSR’s equations are far closer to the ground truth data as compared to PySR’s equations.

Comparison on wall time instead of # iterations

In our opinion, comparing w.r.t # iterations is a fair experimental strategy for several reasons:

1. In practice, evaluating an equation itself is more expensive than the LLM queries, and thus the # iterations becomes directly correlated to time. For example, you might want to use the discovered expression inside a scientific simulation, which can be very expensive to evaluate. For such problems, the wall clock time is a function of the # iterations alone.

2. As mentioned in our original rebuttal, evaluating on wall clock time is less reliable and reproducible than evaluating on the # iterations.

3. Regardless, LaSR’s experiments took roughly 10 hours on an 8xA100 80GB server node. We ran PySR for 10 hours (capped at 1 million iterations) on the Feynman equations dataset and report results here:

   Feynman Equations	PySR ($1000000$ Iterations, 10 hour timeout)	LaSR ($40$ iterations)
   Exact Solve	59 + 3/100	59 + 7/100

   Overall, we find that, despite running PySR substantially longer than LaSR, LaSR still substantially outperforms PySR. This is because PySR, like other evolutionary algorithms, is susceptible to falling in local minima and converges to this local minima pretty fast. This is why supplementing such “local” search strategies with LLM guidance is useful.

The improvement of LaSR looks marginal

In scientific discovery, finding even one additional equation can be highly valuable. We are encouraged by the fact that LaSR's ablations still outperform the current state-of-the-art algorithm (PySR) under the same conditions.

Differing performance for MSE metric and exact match metric

Exact match is rooted in the symbolic equations sketch discovered, while MSE is based on how well the symbolic equation fits to the dataset. The latter is susceptible to optimization errors, especially since these optimization methods do not guarantee finding a global minima (we use BFGS for all experiments). Also, we evaluate on a challenging version of the Feynman dataset where we add random noise to each equation. This further exacerbates parameter fitting errors.

The MSE threshold metric is necessary for gauging performance per iteration because evolutionary algorithms constantly evolve a pool of candidates and the MSE threshold allows localizing when the best performing candidates start appearing in the candidate pool.

Concerns about memorization / equation consistency

There is a slight miscommunication here. LaSR consistently discovers solution equations for these seven problems. However, the discovered equations do not necessarily follow the same format. For instance, Equation 53 in the Feynman equations dataset defines the heat flow from a point source (an inverse square law with ground truth formulation: $h = \frac{P}{4\pi R^2}$). LaSR finds the following solutions in successive runs with the same hyperparameters:

Both equations fit well to the underlying dataset and reduce to the ground truth. Yet, despite using the same hyperparameters, the functional forms exhibit sharp differences.

The observation ... part of its inference

We are encouraged that LaSR's equations differ significantly in format from the known ground truth equations. It is highly unlikely that the discovered equations are available verbatim online, further reinforcing our hypothesis that LaSR’s performance is not simply the result of regurgitated memorized responses.

Discussion of LaSR ... physical concepts

We’re happy to add relevant discussions to the manuscript. Thank you for the excellent suggestions!