A Brain-Inspired Machine Learning Paradigm for Nature-Powered Equation Solving

Describing Hopfield networks as a SOTA method is questionable--particularly as they have limited applicability in modern contexts. Furthermore, the conceptual parallels between ising model and physical or brain-like systems have been debated for nearly half a century, but any new insights were not introduced here.

The study presents PDE-solving as a means to evaluate performance, but the task specifics are unclear. It remains ambiguous whether the model is learning PDEs from data (as in a Neural PDE Solver) or using NNs as bases of PDE solutions (akin to PINNs). Overall, the intent to improve upon NP-GL (Wu et al., 2024) is evident, but it did not evaluate PDEs and focused on GNN-related tasks. Thus, describing NP-GL as SOTA is inappropriate. Other comparison methods include outdated techniques like RBFs and SVMs, bypassing contemporary machine learning methods and thus casting doubt on reliability. While comparisons in Table 3 seem more reasonable, the rationale for selecting these methods is unclear.

In Table 2, there is an evaluation of partially replacing Transformer components, but only power consumption is reported, with no performance metrics. This inconsistency raises concerns that only favorable aspects from various tasks are cherry-picked for presentation.

There are also questions regarding the feasibility of electrical circuit implementation. Regardless of whether one uses Transformers or other deep learning methods, their structure is optimized for parallel computation on GPUs, making them suboptimal for custom circuit design. Even with circuit miniaturization and implementation efficiency, can the proposed architecture truly compete with GPUs? Additionally, when combining this architecture with Transformers, how would the parts that the architecture cannot replace be handled?

In order of importance, the paper has the following weaknesses:

• The implementation of the DS-solver is unclear. It is based on analog hardware, and therefore there should be much more details on what the components actually are and what realistic effects were/weren't taken into account in the experiments (see Questions).
• One of the main claims of the paper is that the designed Ising-like physical system allows for high expressivity. Why is this the case exactly? For example, how can we be sure that it can be as expressive as convolutions or attention? Further theoretical or empirical investigation would be needed to make this claim stronger.
• For the transformer benchmarks, if only the first self-attention layer is accelerated, surely this results in an overall latency improvement that is quite marginal (due to Amdahl's law). This makes the result much less impressive, and it would be good to add and end-to-end evaluation of latency for the full model (including I/O times and energy consumption under realistic assumptions). Even then, I am not convinced that you can replace a self-attention layer with your physical system and maintain the same outputs with high precision in a realistic implementation (at least it was not shown precisely how in the paper, and how this depends on device and model parameters).
• The main text refers to implementation details being present in the appendix, but there are very few details there (and none about the language modeling experiments). The fact that the code is not available here is also a problem because of the lack of technical detail throughout the paper.
• What happens in the presence of realistic effects in physical hardware, such as limited resolution, dynamic range, various noise sources and device mismatch. This should be quite central in the discussion, and it is not explained what you considered in your simulations.
• Many results and claims are incomplete. For example, it is written "DS-Solver provides ultra-low power of 1.6W for training, and 326mW for inference". For what task and system size, and what device specifications?
• There is a whole body of work around equilibrium propagation, which was not referred to in the related work or throughout the paper, which is very similar to what is achieved here, with stronger theoretical justification and experimental results. A discussion of the benefits of your approach with respect to equilibrium propagation would be interesting to add.

1. The paper lacks discussion on the impact of shadow node count on performance and guidelines for determining the optimal number of shadow nodes
2. The feedback signal path implementation shown in Figure 4 is oversimplified and requires more detailed explanation.
3. The visualization in Figure 8 lacks a detailed error analysis, especially in the small error range, and it is difficult to see the fit quality.

1.Clarity of the Methodology:While the overall methodology is described, some aspects could be presented more clearly. For example, the physical embodiment of the DS-Solver with EC-Train could be explained in more detail. The equations and derivations related to the node dynamics and training algorithm could be accompanied by more intuitive explanations to aid understanding.

2. There is a lack of theoretical analysis. The article claims to enhance the expressiveness of existing dynamic system AI paradigms, however, there is no theoretical analysis provided to support this claim.