SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training

Details and Clarifications on section 4 and figure 5. Thank you for the helpful questions - they will allow us to communicate these important details more clearly. Answer to your points:

• In Sec 4, the input to the model is solely the symbolic equations, and the numeric properties (NCR, Upwardness) are predicted for these symbolic inputs. This aligns with the cross-modal nature of the tasks. As you noted, these properties are defined for 1D functions (D=1). We have updated the text to clearly state this detail.
• The graphs in the interpolatability analysis (Figure 5) correspond to actual functions in the SNIP's latent space. For example, the blue point $Z^s_V$ corresponds to expression $f_s=x^2$, and the bottom-right orange point $Z^d_V$ corresponds to expression $f_d=\cos{(4 x)}+7$. When we interpolate between these points in the latent space (black point $Z_V^{int}$), the decoded expression $f_{int}=1.2\cos{(2.3x)}-0.9\sqrt{9-0.1x}+7$ exhibits a behavior between $f_s$ and $f_d$. This indicates the continuity of the latent space - small movements in the latent vectors result in semantically meaningful changes in the numerical behaviors of the decoded functions. This is a major advantage for symbolic regression, allowing us to transform the combinatorial search into a smoother semantic search by optimizing in the continuous latent space.

Mapping Network and the alternative suggestion. Thanks for suggesting this insightful alternative approach. As noted correctly, one of the motivations for using the mapping network was to integrate with the pre-trained decoder weights from Kamienny et al. 2022, which have strong existing knowledge of expression generation. Cross-attending with encoder outputs (before pooling) could be an interesting approach to explore in future work. However, one potential challenge with this approach is that it would preclude the use of latent space optimization, which provides key benefits for symbolic regression: Performing optimization directly in the higher-dimensional encoder output space (e.g. N x demb = 200 x 512) is significantly more difficult than in SNIP's pooled 512-dim latent space. The continuity and semantics captured in SNIP's latent space allow an efficient yet meaningful search. Many existing neural symbolic regression methods do not incorporate optimization-based search. Those that do, tend to perform it after decoding, which limits the search capabilities. SNIP's main motivation for symbolic regression is to employ latent space optimization which integrates strong neural priors with the strengths of semantic search.

• In Fig 3, it looks like supervised model is going to catch up with SNIP… This is a bit problematic, ideally we would need to see results with more training examples to make sure SNIP remains better asymptotically. The authors may argue that SNIP is much better in the low data regime, I don’t think this is the most important asset of SNIP, as we typically have infinite data in these kind of tasks anyway.

Thanks for pointing this out. We agree that ideally, it would be preferable to show SNIP maintains its advantage asymptotically. However, it is important to consider that SNIP's pre-training phase uses only unlabeled data to learn general symbolic-numeric representations, unlike the supervised baseline model directly trained on labeled data for each specific task. When evaluated on the downstream property prediction tasks in Sec 4, SNIP encounters the same limited labeled examples as the baseline. The fact that the specialized supervised model approaches SNIP’s performance with more data is expected, since it is tailored to the prediction task of this specific property. In fact, SNIP's value proposition is in its flexibility to efficiently adapt to new tasks thanks to its task-agnostic pre-trained representations. This capability makes it complementary to task-specific supervised models. For example, even for CLIP model, despite observing hundreds of millions of paired examples during pre-training, it achieves comparable performance to specialized models on image classification tasks, without necessarily outperforming them.

• I’m not sure the fine-tuning task with 100 examples can be called few-shot. Few-shot usually refers to only a handful of examples. I would instead call this the “low data regime".
• I don’t find the results section of sec 5.4.2 particularly well presented, especially in paragraphs “strogatz”, “black box” and “Feynman” where many numbers are presented which can directly be read from the graph — since the orderings of the models are similar on all three datasets, it would be better to have a more qualitative description extracting the key takeaways.

Thank you for your valuable suggestions. We have addressed each of the points you raised in the updated version.

Symbolic Regression (Continue)

• Figure 11.a, in Appendix E, strongly suggests that SNIP is learning different (independent?) models for different dimensions D. Is there a benefit in training SNIP on many dimensions at the same time? Or should different models be trained on different dimensions?

We appreciate the reviewer's insightful question. Fig 11 does illustrate that SNIP learns clustered representations based on the input dimension. This indicates that the model successfully captures similarities between functions of the same dimensionality during pre-training. The benefit of training a single SNIP model on a range of input dimensions (up to 10D in our case) is that it allows flexible application on datasets with varying dimensionality without needing to train specialized models per dimension. This is a more efficient approach than training separate SNIP variants for each input dimension. Importantly, Fig 11.b and 11.c also show meaningful structure within each dimension's cluster based on the symbolic-numeric essence of the functions. So while distinct dimension representations are learned, the pre-trained encodings still embed cross-modal knowledge.

• The test sets for symbolic regression are small, and somewhat limited. Feynman has 119 functions, many of them redundant, most of them very simple (first order approximations of physical laws). Strogatz has 14, all in low dimension. Evaluating from a larger test set of generated functions (held-out from the training set) would greatly improve the paper.
• Please provide results on a held-out dataset of generated functions, and scaling results for different problem dimensions (D), function complexity (e.g. family of operators used), number of input (N), precision achieved (in the MSE sense).
• How does model performance scale with problem dimension, function complexity, the number of inputs, desired precision?

Thank you for the suggestion. In the original manuscript, we focused evaluations on SRBench given its recognition as the gold standard benchmark and the inclusion of leading GP baselines. However, in response to the reviewer's comment, we have additionally tested SNIP on the held-out in-domain datasets, generated following the data generation protocol in Sec 3.4, consisting of 400 validation functions with different levels of difficulty: number of binary operators ($b$), number of unary operators ($u$), input dimension ($d$), and number of input data centroids ($c$). The detailed results in Fig 15 of App E.5 demonstrate how the models' performance is affected by increasing formula complexity, as characterized by a higher number of operators and input dimensionality. We have provided these additional results in the appendix due to the page limitation, and that SRBench datasets, including Feynman, Strogatz, and Blackbox, constitute the most widely used and challenging open-source benchmark for symbolic regression. In fact, many recent pre-trained models (Biggio et al. 2021, Kamienny et al. 2022) have shown even weaker performance on SRBench compared to held-out synthetic datasets from their pre-training distribution. This highlights the greater difficulty of generalizing to out-of-domain datasets like SRBench.

Encodings:

Alternative numeric encodings. We appreciate the reviewer highlighting recent alternate number encodings like Golkar et al. 2023, and Becker et al. 2023. We agree that discussing the merits of emerging schemes would enrich the background and have acknowledged them in the updated related work. However, analyzing tradeoffs between encoding schemes is not the core focus of our work, and needs further exploration which can be a potential future work. These potential directions will certainly involve challenges and may demand rigorous experimentation to ascertain their efficacy. SNIP's primary contribution lies in pioneering cross-modal pre-training between mathematical symbols and numerics. This unified learning of symbolic and numeric representations is orthogonal and complementary to the choice of encoding scheme.

Skeleton vs. Prefactors We appreciate the reviewer's insightful question. We actually explored this direction initially. However, our preliminary experiments showed limitations of solely using equation skeletons without numeric constants. Specifically, pre-training with just skeletons cannot effectively imprint insights about a function's unique numeric behavior, as each skeleton can manifest vastly different numeric behaviors based on different constants. This hindered capturing cross-modal similarities in the learned representations and led to subpar performance in tasks where this numeric pattern is critical such as prediction of numeric properties or symbolic regression. For example, the convexity of a simple function like $f(x) = C x^2$ depends heavily on the choice of constant $C$ as it is fully convex for $C>0$, and non-convex for $C<0$.

Suggested modifications on dataset size and references. Thanks for reviewing these details. The total number of pre-training examples used for SNIP is around $60M$. This information is now added to the main manuscript for clarity. We have also updated the references for accuracy in the new version.

Property Prediction:

Evaluation metrics, chance level, and results for other properties. We thank the reviewer for these valuable suggestions. To improve the evaluation on function properties we have done the following steps as suggested:

1. To avoid redundancy, we have removed the column for $R^2$, and only provide NMSE.

2. As suggested, we define an accuracy metric, based on the closeness of the predicted values to their corresponding true values. To this end, we first normalize the true values for all of the tasks to the range of $(0, 1)$. Subsequently, we define the accuracy within absolute tolerance of $\tau$ as $Acc_{\tau} = \frac{1}{N_{t}} \Sigma^{N_{t}}_{i=1} \mathbb{1}$ |\hat{p}_i - p_i| \leq \tau $$, where $p_i$ and $\hat{p}_i$ are the true and predicted values for $i$-th example. This metric perceives a prediction as accurate if the predicted value is within tolerance (here, $\tau =0.1$) of the normalized values.

3. For NMSE, the chance level can be considered to be $NMSE=1$ as it corresponds to the predicting average of property without considering input values. For the newly defined accuracy metric, we also consider the base (chance level) as the accuracy achieved when assuming $\hat{p}_i = \bar{p}$ for all examples. We have reported the chance level for each property in the table below.

4. We have updated the results for the NCR and Upwardness metric in Sec 4.2 and Tab 1, and to be comprehensive, we have also added the results for all properties in App C.2 as suggested. For your reference, here are the updated results:

We observe that SNIP predictions have better performance compared with the supervised model in both metrics. The chance level depends on the distribution of values for each property. Also, different tasks have various difficulties, and SNIP's performance can be better in some cases (e.g. NCR) compared to more challenging properties (e.g. Avg y).

Details and Clarifications on the requested sections. Thank you for the helpful questions - they will allow us to communicate these important details more clearly. Answer to your points:

• Section 4: Yes, you're correct. the input to the model is solely the symbolic equations, and the numeric properties (NCR, Upwardness) are predicted for these symbolic inputs. This aligns with the cross-modal nature of the tasks. We have updated the text to clearly state this detail.
• Section 5: For the symbolic regression, the decoder is initialized from the pre-trained weights of Kamienny et al. (2022), and then trained with the mapping network for the expression generation task (with cross-entropy loss). Details of symbolic regression training are provided in pg 20, App E.1, p.4. We have also included a summary in the main text for clarity.

• The experiments performed in Section 4 are conducted on 1-d symbolic expressions. Although this setting is already insightful, I would be curious to see whether similar tests can be conducted on higher-dimensional expressions.

We appreciate the reviewer's insightful question. The current properties we defined and analyzed are primarily intended for 1D examples, as a proof of concept to showcase SNIP's capabilities for cross-modal reasoning between symbolic equations and numeric data patterns. Expanding this analysis to higher-dimensional problems, however, would require defining new properties relevant to such contexts, as most current properties are focused on 1D scenarios. If the reviewer has any further thoughts or suggestions regarding insightful multi-dimensional properties worth exploring, we would highly appreciate the input. Higher-dimensional analysis on new properties would also be time-consuming as we'd need to train the three model variants to predict these new properties in higher-dimensional contexts.

• While Transformer-based approaches to symbolic regression provide advantages in terms of inference time compared to genetic programming approaches, they may suffer from some of the limitations characterizing standard language models. One such limitation could be that they may hallucinate and output a symbolic expression that does not fit the numerical data at all. I would have liked to see some investigations on this aspect, for example an analysis on the extent to which output symbolic expressions are numerically aligned with the input dataset.

Thanks for raising this very important point. As the reviewer noted correctly, most of the Transformers neural symbolic regression models that are trained on large-scale synthetic datasets (e.g., Kamienny et al. 2022, and Biggio et al. 2021) struggle to generate accurate symbolic expressions due to their training objective (cross-entropy loss) being misaligned with equation discovery goals ( fitting data well). They rely on techniques like beam search and constant refinement to mitigate this, but these are not always sufficient. SNIP, however, targets this issue innovatively. Unlike traditional models that depend solely on Transformer logits and post-decoding search strategies, SNIP leverages latent space optimization (LSO) to integrate neural priors with semantic search with the objective of achieving higher fitting accuracy (e.g., $R^2$ score). As shown in Fig 5, SNIP's latent space provides both interpolatability and continuity, where minor adjustments in semantic latent vectors lead to meaningful changes in the numerical behaviors of decoded functions. This is a strong foundation for symbolic regression, allowing us to transform the combinatorial search into a smoother semantic search by optimizing in the continuous latent space.

Based on the reported results in Sec 5.4 and Fig 6, as well as App E.4, we observe that SNIP achieves higher average $R^2$ scores compared to prior neural symbolic regression models. This indicates SNIP's output equations exhibit stronger numerical alignment with the input data. We would also like to note that to improve the numerical alignment of the generated equations, the candidate equations (generated from the decoder) also undergo a constant refinement stage (similar to Kamienny et al. 2022) which optimizes their coefficients based on the given dataset using BFGS optimization. By leveraging both LSO with $R^2$ objective and this refinement strategy, SNIP is able to closely fit the training data.

Novelty of SNIP and discussion on comparisons. Thank you for the comments and thoughtful questions. Please find our answers below.

• Pre-training in Mathematical Expressions: While our techniques draw inspiration from multi-modal representation learning, prevalent in domains like vision-language, their employment to mathematical expressions is unique. Existing Transformer-based models in this domain have been successful in tasks like differential equations and function integration but fall short in integrating symbolic and numeric data insights. Recent Transformer models in symbolic regression, like Kamienny et al. (2022) and Biggio et al. (2021), also only focus on mapping numeric data to mathematical expressions in a supervised, task-specific manner. To the best of our knowledge, our work is the first approach that integrates these domains through a unified, task-agnostic pre-training. This approach is distinct from existing methods and provides a novel perspective in mathematical representation learning.
• Cross-modal Property Prediction Comparisons: During pre-training, SNIP only sees unlabeled data, learning mutual symbolic-numeric similarities in an unsupervised way. When later evaluated on downstream tasks, SNIP encounters the same limited labeled data as baseline models. The comparisons demonstrate SNIP's ability to efficiently adapt to new cross-modal prediction tasks thanks to pre-training - a valuable complement to specialized and task-specific supervised models.
• Symbolic Regression Comparisons: In the SR comparisons, most of the baselines are based on Genetic Programming which does not use any prior knowledge, as they only see one dataset. Transformer-based baselines like Kamienny et al. (2022) employ supervised training on large synthetic datasets for the numeric-to-symbolic generation task of SR. As noted, SNIP's pre-training is unsupervised, focused on learning general symbolic-numeric representations like CLIP learns visual-textual relationships. For the SR task, we follow a similar strategy as Kamienny et al. (2022) to generate data and fine-tune the symbolic decoder in a supervised manner. Subsequently, we exploit SNIP's prior knowledge combined with LSO for enhanced equation generation.

We hope our response addresses your concern. Please do let us know if we have correctly interpreted your concern or if further clarification is needed.

Impact of LSO in the use of SNIP for symbolic regression. Thank you for pointing this out. To clarify, existing Transformer neural symbolic regression models that are trained on large-scale synthetic datasets struggle to generate accurate symbolic expressions due to their training objective (cross-entropy loss) being misaligned with equation discovery goals ( fitting data well). They rely on techniques like beam search to mitigate this, but these are not always sufficient. SNIP, however, targets this issue innovatively. Unlike traditional models that depend solely on Transformer logits and post-decoding search strategies, SNIP leverages latent space optimization (LSO) to integrate neural priors with semantic search. As shown in Fig 5, SNIP's latent space provides both interpolatability and continuity, where minor adjustments in latent vectors lead to meaningful changes in the numerical behaviors of decoded functions. This is a strong foundation for symbolic regression, allowing us to transform the combinatorial search of mathematical equations into a smoother semantic search by optimizing in the continuous latent space. To the best of our knowledge, none of the prior works in this domain have employed such a latent space search strategy. In fact, LSO is an integral component of our method when used in SR. It enables generating equations that leverage the prior knowledge from SNIP pre-training, while also fitting the dataset well.

In response to the reviewer's comment, we conducted additional ablation experiments to further illustrate the impact of LSO in our method (on 400 held-out synthetic validation functions). The results (please see the table) show that performing LSO (with the objective of fitting accuracy) on the SNIP's rich representation space significantly improves the fitting performance of the generated equations compared to the variant without LSO, while maintaining the same level of complexity. We have included this study and additional discussion on the impact of LSO in App E.5.