SIRD: Transformers Assisted Step by Step Symbolic Integration

• Dataset generation concerns:
• For procedurally generated datasets, the diversity of questions and the relative difficulty are more critical than the scale of the dataset.
  • Previous work in this area [1] generated a lot of interest and the community found a lot of avenues of improvement. Many of the avenues of improvement were summarized in this review [5]. Specifically, a key problem with the evaluation of [1] was that the comparison with SymPy, Mathematica, and Matlab only targeted a very small fraction of in-domain problems (verbatim: "indefinite integrals of enormously complex functions of a single variable ‘x’ whose integral is a much smaller elementary function containing no constant symbols other than the integers −5 to 5.").
• Does the current procedural generation script expand the diversity of the problem set beyond these narrow constraints?
• Additional Evaluation Dataset:
  • The held-out dataset may not be a sufficiently challenging benchmark for general automated symbolic integration algorithms, given that both the proposed algorithm and [1] achieve over 90% accuracy. A more demanding evaluation could enhance the robustness of the results. Here's are some suggestions for "gold standard" datasets to evaluate the efficacy of this method:
    • The MIT integration bee is an annual competition in which contestants are asked to integrate an equation in the least amount of time. The practice rounds consists of 20 questions. All the previous year questions (and answers) are available at the bottom of this website: https://math.mit.edu/~yyao1/integrationbee.html
    • Are current SOTA methods like SymPy, [1], and the proposed algorithm able to solve some of these problems? Although some mathematical operators (e.g., hyperbolic functions like tanh) may be out-of-domain, a significant portion of these problems should still fall within the scope of this approach.
    • The proposed algorithm should ideally perform as well as SymPy, whereas the baseline [1] is likely to struggle on these more complex, out-of-domain problems.
    • I want to emphasize that the results on this evaluation should not be held against the contributions in this paper. Most of these problems are very challenging for humans without prior training. This task is not a standalone benchmark for general symbolic integration capabilities, but may serve as a valuable test of the algorithm’s overall efficacy.
  • Of course, the authors may prefer other gold-standard datasets for downstream performance evaluation.
• Additional Evaluation Baseline:
  • Symbolic integration necessitates high level planning and reasoning abilities. As such, a comparison with an LLM based baselines would substantially strengthen the paper’s central claims. I suggest evaluating against an instruction-finetuned LLM in a zero-shot program induction settings. ie: Directly prompt an LLM with the integrand and instruct it to derive the solution. A prodecurally generated dataset should ideally not cause any data leakage issues.
• Clarity:
  • I apologize if this sounds harsh, but the paper is poorly organized. A research paper should generally present a series of properly organized logical statements. However, the current manuscript scatters key information across many sections. I highly recommend reviewing [2,3,4]. Here is a (incomplete) list of clarity issues:
    • The introduction doesn't summarize key results. In the bullet points that summarize contributions, there is no mention of a comparison with [1]. This was very confusing on the first pass, as I thought the authors only evaluated against SymPy. It isn't until Page 6 that we see a comparison with [1].
    • Why is the loss formulation in the experiments section? Consider creating a problem statement section that defines the search problem, the neural guided search problem, the loss formulation for the search problem, and the search space.
      • The preliminaries section of these program synthesis papers might be helpful.
        • https://arxiv.org/abs/1906.04604
        • https://arxiv.org/pdf/2006.08381
        • https://arxiv.org/abs/2310.19791
        • https://arxiv.org/abs/2007.12101
        • https://arxiv.org/abs/2409.09359
    • Figures are poorly motivated, and the captions lack depth. I suggest the authors rewrite the captions with a brief description of the experiment, the hypothesis, the relevant findings, and how the relevant findings support the central claim of the experiment.
    • The explanation of integration by parts might be unnecessary within the section on dataset generation. I recommend moving detailed discussions on dataset construction to an appendix and provide a summary in the main manuscript.
    • Expand the related works section to include relevant literature on neural guided search and automated symbolic integration. Specifically, address connections to:
      • Neural Guided Search:
        • https://arxiv.org/abs/1906.04604
        • https://arxiv.org/pdf/2006.08381
        • https://arxiv.org/abs/2310.19791
        • https://arxiv.org/abs/2007.12101
        • https://arxiv.org/abs/2409.09359
      • Automated Symbolic Integration:
        • https://arxiv.org/pdf/2410.02666 (this is concurrent work).
    • Adding an overview paragraph at the beginning of the Experiments section would be helpful. This paragraph should ideally summarizes what research questions each experiment attempts to answer.
    • Table 2: The "Test set" and "Approach" column only list a single value. These columns are unnecessary. I recommend rewriting the caption to contain more information about relevant sections as well.
    • Table 3: Separate each approach into its own column for clarity.
    • Table 4: Clearly indicate the section and method relevant for the observations in this table.
    • L373: The algorithmic details should be moved to an algorithms section within Section 3.
    • Table 5: The symbolic baseline and the neural guided baseline should be explicitly pointed out.
    • L420: Alternating between the abbreviation "LC" and the full citation "Lample and Charton, 2020" is slightly awkward. Consider using only one of them.

[1]: https://arxiv.org/abs/1912.01412

[2]: https://www.cs.utexas.edu/~mooney/cs391L/paper-template.html

[3]: https://www.cs.williams.edu/~andrea/Carla/craft.html

[4]: https://deeplearning.cs.cmu.edu/F22/document/recitation/Recitation7/recitation_7_Paper_Writing_Workshop_F22.pdf

[5]: https://arxiv.org/pdf/1912.05752

1. Does "Number of Samples" in Tables 2 and 3 refer to the number of training samples? If so, why is the accuracy of the Substitution Rule significantly lower, despite having twice the number of samples compared to the Parts Rule?
2. The baseline model, LC, was proposed in 2020 and is somewhat outdated. Why didn't the authors explore more advanced methods to assess the robustness across different base models?

• The reviewer would like to know the responses to Weaknesses.
• On line 323, it says “However, in the test set, we only have a single ground truth label for the function.” Does this mean that the test set is limited to samples with only one correct rule, or does it mean that, even though there are multiple correct rules, only one of them is considered to be the correct answer? If it is the former, the reviewer is concerned about the bias of the samples as a test set, and if it is the latter, concerned about the incompleteness of the evaluation method. Since it is a test set, shouldn't multiple correct rules be given as ground truth, even if it takes a labor effort?
• On line 370, it says “Consequently, we exclude such functions from the test sets of all three datasets before evaluation. Additionally, we remove functions with a sequence length greater than 384 when converted to prefix notation.” However, the reason for this is unclear. Even if there are problems that cannot be handled by the proposed method or the baseline, they should be included in the comparison. In particular, the restriction that series longer than 384 are not included is thought to work in favor of the proposed method, and the fairness of the comparison of methods is a concern.

Have you evaluated how modern LLMs like GPT-4, Sonnet3.5 perform on your dataset?

How does the model perform on real-world integration problems from mathematics textbooks or research papers?

How could you address the limitations of model can not handling hyperbolic functions and their inverses?

The paper presents a well-executed but incremental contribution to symbolic integration. While the dataset creation and integration with SymPy are valuable, the technical approach lacks novelty and has significant limitations. The absence of comparisons with modern LLMs and the restricted handling of mathematical functions make it difficult to assess the true value of this contribution. I recommend rejection but encourage the authors to address these limitations in future work.