An Expressive and Self-Adaptive Dynamical System for Efficient Function Learning

We sincerely appreciate your insightful comments. We will address your questions below.

1.The number of parameters
We will report the number of parameters in the Appendix. Due to the character limit, we present parameters for a dataset below. For the PEMS04 dataset: Graph WaveNet: 250,689; MTGNN: 555,169; DDGCRN: 567,109; MegaCRN: 316,225; NP-GL: 188,498; EADS: 94,976.

2.Include NP-GL in Comparison

NP-GL supports inference on a dynamical system, its training is conducted offline on GPUs. According to [1], NP-GL operates with similar power for inference to EADS. Both methods achieve similar inference latency, resulting in comparable inference energy efficiency. However, for training, compared to NP-GL, EADS achieves an average ~$10^{2}$ training speedup, ~ $10^5$ greater energy efficiency.

3.Include Theoretical Computational Complexity
The computing paradigm of EADS is inherently different from that of digital methods, rendering their complexities not directly comparable. Specifically, GPU-based methods execute explicit instructions sequentially, while EADS operates via natural annealing—where electrons (dis)charge capacitors to seek equilibrium. To provide a clear understanding, we define one operation in our DS as a single substantial (dis)charge event of a capacitor, taking ~0.2 ns. For a digital processor, a single instruction execution requires ~0.25 ns. Given that the operation execution times are comparable, we evaluate computational complexity by comparing their total number of operations. Our results show that EADS requires ~$10^3$ operations per inference, while GPU-based solutions require ~$10^5$ operations per inference. We will incorporate detailed discussion.

4.Experiment Details on LLMs
We design our LLM experiments to assess EADS’s ability to learn the complex transformations embedded within each decoder of GPT-2. The entire GPT-2 model is not implemented on EADS; rather, only one decoder is implemented by EADS. Although our system has the potential to implement all decoders, such an extension is beyond the current scope of our work. We will provide details in the manuscript.

5.Include the Settings of EADS
Thanks for your valuable suggestion. We will add a comprehensive section in the Appendix specifying EADS configurations.

6.Scalability and Flexibility of EADS

• Scalability: Dynamical systems have demonstrated strong scalability through single-chip and multi-chip solutions, as evidenced in [2-3]. Specifically, [2] propose a single-chip solution, while [3] explored multi-chip solutions. Overall, the scalability of dynamical system machines is well-founded.
• Flexibility of EADS: Since EADS enables on-device training, its parameters can be dynamically adjusted to suit different problems without hardware modifications, making it highly versatile.

7.Why Is the Method Called Dynamical System
We refer to EADS as a “dynamical system” because EADS is not merely a model—it is a software-hardware co-design implemented using programmable electronic components.

8.Under What Circumstances Is On-Device Training Needed?
On-device training is essential for several reasons:

1. Enabling a New Computing Paradigm: Instant on-device training extends the exceptional computational power of dynamical systems beyond inference to include training, thereby establishing a new AI paradigm with remarkable efficiency.
2. Applications Requiring Real-Time Adaptation or Facing Training Costs:

• By implementing training directly on-device, EADS can update parameters automatically in response to new data in real time without the delays associated with off-device training, ensuring accurate predictions as underlying patterns evolve.
• In scientific computing, the potential of data-driven equation solving is often hindered by prohibitive computational costs. As ML models scale to achieve higher accuracy, training resources may exceed those required by traditional numerical solvers, limiting their practical benefits. Our on-device training directly addresses this challenge.

3. Advancing AI Paradigm Development: Recent research suggests that collocating inference and training on the same hardware can significantly reduce training costs while more closely mirroring biological intelligence [4]. There is also evidence indicating that the human hippocampus functions as a dynamical system with collocated training and inference [5]. Our proposed on-device training method offers a meaningful exploration in this domain.

[1] Extending power of nature from binary to real-valued graph learning in real world

[2] DS-GL: Advancing Graph Learning via Harnessing Nature’s Power within Scalable Dynamical Systems

[3] Increasing ising machine capacity with multi-chip architectures

[4] The forward-forward algorithm: Some preliminary investigations

[5] Attractor dynamics in the hippocampal representation of the local environment

We appreciate your thorough review and constructive feedback. We address each concern in detail.

1.Relation to Equation Discovery
We clarify that our work fundamentally differs from equation discovery: rather than discovering equations, our work focuses on efficiently solving equations with high speed and low computational cost while maintaining high accuracy. EADS works by optimizing its parameters through the proposed on-device training method to capture the joint distributions between inputs and outputs. Once trained, it can efficiently generate solutions for new inputs. In PDE solving, equation discovery methods would attempt to identify the underlying equations, while EADS aims to generate solutions to PDEs under varying inputs (different coefficients or initial conditions) as described in [1]. Importantly, our work extends to solve analytically intractable equation. We will add a subsection in the Related Work to state the relation.

2.Clarification of Learn in Dynamical Systems
We apologize for any confusion. As noted in Q1, learning is indeed misleading since our focus is on solving equations efficiently rather than learning them. We agree that EADS do not learn their inherent dynamics. However, it's worth noting that natural dynamical systems do adjust the parameters within their dynamics. A typical example is the human hippocampus, which functions as a dynamical system and adjusts synaptic strengths (i.e., parameters) between neurons [2]. We will revise the manuscript to replace “learn equations” with “tune the parameters within dynamics.”

3.Formality of language, Sentence Clarifications and Key Term Definitions
We will revise the manuscript to use more precise language.

• Line 327: Revised to “in graph learning tasks that it is originally designed for.”
• Inference Latency: Defined as the time delay between submitting an input to a trained model and receiving its output.
• Hard-to-Define Equation Learning: Revised to “For complex real-world problems where underlying equations are unknown or not readily expressible in closed form.”
• Abstract: Revised to “While modern machine learning methods approximate complex functions, their escalating complexity and computational demands pose challenges to efficient deployment.”

4.Hamiltonian Training and Example
Training Hamiltonian involves optimizing its parameters to capture correlations between nodes. Consider $\mathcal{H} = -\sum_{i\neq j}^{N} J_{ij} x_i x_j + \sum_{i=1}^{N} h_i x_i^2$, with trainable parameters $J,h$. Using a conditional likelihood method [3], the estimated state at equilibrium is: $\hat{x_i} = \frac{1}{2h_i}\sum_{j\neq i}^{N} (J_{ij}+J_{ji}) x_j.$ Then, the MSE loss is minimized via standard backpropagation to optimize parameters. Detailed explanation and example will be included in Section 2.1.

6.EADS Training in PDE Solving
In PDE solving, EADS approximates solutions under varying inputs (coefficients or initial conditions). Training is optimizing system parameters ($P,J,Q$) to capture the correlations between inputs and outputs. In one of the Darcy Flow example [1], we solve the 2D Darcy Flow equation on the unit square with Dirichlet conditions. Each training sample comprises a 16×16 grid of coefficient a(x) and its corresponding solution u(x). EADS learns mapping from a to u, then rapidly generate solutions without iterative solvers. A detailed description will be added in the Appendix.

7.Interpretability and Comparison with Equation Discovery and FEM
As explained in our response to Q1, our method does not aim to derive explicit equations from data. Since the goal and required training data are significantly different between our method and methods in equation discovery methods, it is difficult to incorporate genetic programming based methods in our evaluation for a fair comparison. To provide a reference, a method in the genetic programming-based equation discovery domain that also evaluated on the Burgers' equation achieves an MSE of $4.33×10^{-5}$ [4], while our method achieves an average MSE of $9.37×10^{-6}$ under our evaluation. Regarding the comparison with FEM, our evaluation employs outputs from the FEM as ground truth, so we only compare their efficiency. On average, the inference latency of EADS is at the level of 10e-7 seconds, while the latency of typical FEM generally exceeds 10e-3 seconds. Therefore, the efficiency of EADS is significantly better than FEM, especially on complex PDEs with higher resolutions.

[1] Li, Z., et al. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR.
[2] Wills, T.J., et al. Attractor dynamics in the hippocampal representation of the local environment. Science, 2005.
[3] Wu, C., et al. Extending power of nature from binary to real-valued graph learning in real world. ICLR.
[4] Chen, Y., et al. Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE). Physical Review Research.

We sincerely appreciate your positive feedback and constructive suggestions. Below, we address each of your points in detail to further improve the manuscript.

1.Additional explanations for comparable accuracy experiments

Thank you for this insightful suggestion. In our evaluations, we compare EADS with NP-GL and deliberately selected state-of-the-art digital baselines that have consistently demonstrated exceptional accuracy through rigorous validation in prior research. Although the accuracy differences may appear similar in magnitude, the actual improvements are significant. Specifically, compared to NP-GL, EADS achieves an average MAE reduction of 8.92% on spatial-temporal prediction, a 71.4% MAE reduction in PDE solving, and a 6.32% reduction in PPL on LLM tasks. Furthermore, compared to the best digital baseline, EADS yields a 15.10% MAE reduction on spatial-temporal prediction, and a 4.62% MAE reduction on PDE solving, while substantially reducing computational demands. We will expand our analysis with additional explanations to better highlight the advantages of EADS.

2.Terminology clarification

Thank you for highlighting this potential confusion and for your valuable suggestion. We agree that the term “function learning” offers a more precise description of our work. We will change “equation learning” to “function learning” throughout the manuscript and differentiate our focus from symbolic regression in the Introduction.

3.Add visual diagram for EADS workflow

We appreciate your valuable suggestion. We will include a new figure in the Methodology section to illustrate the overall EADS workflow. As we cannot directly upload figures in the rebuttal, we provide a detailed description of the proposed diagram below. The diagram will be divided into two main components:

• Expressivity Enhancement: We will visualize the hierarchical structure and heterogeneous dynamics with clear illustrations of information flow. This visual will demonstrate how EADS progressively refines the input through multiple processing stages with heterogeneous dynamics.
• Instant On-Device Training: This part will detail the parameter adjustment process. It will depict how intrinsic electrical signals are used as feedback to drive learning, illustrating the feedback loop where the output nodes’ electrical currents guide rapid, on-device parameter updates.

4.Enhanced figure and table captions to highlight key messages

Thank you for recommending more informative captions. In our revision, we will update the captions to explicitly highlight the key performance takeaways:

• Table 1: Spatial-temporal prediction performance in MAE. EADS consistently outperforms all baselines across all datasets (best results in bold).
• Figure 4: Training time and inference latency for spatial-temporal prediction. EADS demonstrates significantly higher efficiency than GPU-based baselines.
• Table 2: Test MAE for PDE solving. EADS achieves superior accuracy compared to all baselines, with marginal improvements over FNO.
• Figure 5: Training time and inference latency for PDE solving. EADS is markedly more efficient than GPU-based baselines.
• Table 3: Test perplexity (PPL) on LAMBADA. EADS significantly outperforms NP-GL.
• Figure 6: Training time and inference latency on LAMBADA. EADS exhibits superior efficiency compared to GPU-based methods.