DS-LLM: Leveraging Dynamical Systems to Enhance Both Training and Inference of Large Language Models

Overview

We sincerely appreciate for your insightful and constructive feedback, and we have uploaded a revised version of paper to address your concerns. The contents corresponding to your questions and concerns are labled as Reviewer D like "D-Q1-1". The detailed response to your comments are listed as below:

1. Novelty and Differentiation from prior work (question 1-1):

Thank you very much for your meaningful comments, and we have revised the introduction to clearly highlight the distinctions between our work and the prior NPGL approach. We also add a discussion section in Appendix to clarify our novelty and differentiation from prior work, which is also listed as below.

The prior work NPGL is not a mapping of existing GNNs including those who have attention mechanism. It ignores the architecture of neural networks and directly fits the data with the Hamiltonian, which is a shallow fully connected model without any hierarchy or structure as shown in Eq. (1). The parameter size of this model is determined by the size of input data, making it fundamentally unable to support extremely complex tasks like NLP which cannot fit in the shallow Hamiltonian. Therefore, rather than directly fitting data into a shallow DS model as NPGL does, we propose a novel mapping method that enables the deployment of existing deep neural networks (NNs) to DS machines. This allows us to leverage well-established, mature model architectures, rather than working solely with the DS model itself. To the best of our knowledge, this is the first work to map deep NNs—designed for traditional computing architectures—onto DS machines, with LLMs chosen as a specific application.

2. Key advantages of our approach (question 1-2):

We deeply appreciate for the constructive suggestions. We have highlighted the key advantages of our approach over prior work and other emerging compute alternatives in the Appendix as a discussion section, which are also listed as below:

Prior works (e.g. NPGL): Previous works, such as NPGL, focus on building new models based on the Hamiltonian. While this approach holds promise, it doesn’t directly benefit the computation of existing models like LLMs. In contrast, our approach maps existing models to DS machines, enabling us to transfer well-researched, mature models like LLMs to a powerful emerging computing framework. This allows for dramatic reductions in both training and inference costs, leveraging the power of DS machines without the need to reinvent models from scratch.

Quantum computing:

Quantum computing is a promising avenue but is still constrained by scalability issues and the need for complex error correction. As quantum systems scale, errors and noise increase, demanding advanced error-correction techniques that are not yet fully mature. Existing largest quantum computer has about 1100 qubits [1], which is too small to support LLM computing tasks. Meanwhile, due to the technology requirements, nowadays building and running quantum computers are still very expensive. In the contrast, DS machines are built on CMOS-compatible technologies, which are highly mature and the cost of its fabrication is at the same level of digital processors.

Optical Computing:

The optical computing solutions also rely on special technology and is hard to be integrated with digital systems. Building complex and accurate optical circuits can be a big challenge and very expensive, making the stability and feasibility of optical computer a larger challenge than DS machines.

Computing-In-Memory (CIM):

CIM technology is relatively feasible and CMOS compatible. Existing CIM works only supports inference, achieving up to 100s times acceleration and 2000 times energy reduction [2], which is much lower than our solution especially on energy reduction. The insight behind this is that CIM still follows a traditional instruction-based paradigm, completing computations step by step. In contrast, DS machines are driven by physical processes, specifically natural annealing, which doesn’t require step-by-step control. This allows DS machines to naturally perform complex tasks automatically without extra energy for controlling, achieving a much higher energy efficiency.

Reference:

[1] https://www.ibm.com/quantum/blog/quantum-roadmap-2033

[2] Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference (https://arxiv.org/pdf/2406.08413 )

• Due to limitation of characters, responses to other questions will be shown in next comments.

Thank you for your responses.

I still feel my major concerns are not yet clearly addressed in the updated revision of the papers (main sections not just Appendix). I am not a fan of how the authors boast of such significant speedup and energy efficiency comparing between a mature technology (GPUs) with an emerging one (Dynamical Systems) that still is immature, not fully grounded, and possibly impractical. My concerns are further exacerbated by the fact that when deriving these comparision numbers, the authors have not fully leveraged existing SOTA techniques (such as vLLM [1] which easily improves throughput 2-4x), leading to possibly severly under-reported numbers for GPUs. Specifically,

1. I share similar concerns with Reviewer xU44 on examples of real-world applications and particularly physical practicality of the proposed solution. While in the authors did try to clarify, [2,3] only seemed to present theoretical analysis based on simluations without physical implementations, and [4] showed physical implementations solving MAX-CUT on small graphs and not so impressive (or even advantageous) runtimes. This leads me to question the feasiblity of such solutions and whether such systems would have the performance if physically implemented versus what the authors claim to be theoretically possible.

2. Concerns on unfair comparisions. The authors have not fully leveraged existing SOTA techniques (such as vLLM [1] ), leading to possibly severly under-reported numbers for GPUs. Furthermore, the authors are comparing between a measured performance of GPUs to that of theoretical numbers on paper of an emerging tech. I do not think such comparisions are solid, and I also do not like how they are presented in the paper. For example authors claim 37,545x speedup on inference, this number could possibly easily been cut to more than 1/2 with techniques such as vLLM, and more if accounting for what is theoretically possible versus physically implementable. I do not think it should be a game of what tech presents the most amazing theoretical improvement, and in fact I would appreciate it if the authors responsibly scale back on such claims with full disclosure on the technology's limitations (and that some of the numbers presented might be based on unfair comparison of theoretical v.s. physical).

3. Concerns on scalability. I do appreciate the authors efforts on adding experiments for larger 7B models and understand the limitations of not being able to scale experiments to even larger models. I would think it to be much better even if the authors in the main section of the paper to analyze the cost and performance of scaling on such systems theoretically. Possibly it would also help with visualizations if some key metrics of Table 2 and Table 3 are presented in a plot (with theoretical projections on how such scale to larger models). It would also be helpful if a scaling plot on the cost of scaling (area etc.) or explained in text.

4. Meaningful metrics. I appreciate the authors updates on tokens/s and tokens/kWh. I suggest replacing Table 2, Table 3 (or suggested plots above) with these metrics.

5. No disclosure in the main paper on limitations. Given my concerns above (see 1,2,3), I suggest the authors fully disclose the limitations of such emerging technology responsibly throughout their main paper (including abstract, intro, results, and if possible including an additional section on limitations). Especially I have questions on, (1) are such systems even physically possible to reach what the authors are proposing? (2) if they could, would they even perform to what is theoretically possible as stated?; (3) are there challenges in scaling, and if so are there solutions?; (4) how difficult would it be to implement such a system at scale to what is currently achieved with GPUs? What is the current progress on physically implementing such systems and how do they compare to the scale of GPUs?

With the above stated, I am open to further discussions and open to increasing my ratings if the authors clarifications (on my concerns of limitations, comparisions, and scalability) do get reflected in the main sections of the paper. As of the current state of the paper and authors responses to my concerns, I could not warrant a higher rating.

References

[1] Efficient Memory Management for Large Language Model Serving with PagedAttention

[2] Ising-Traffic: Using Ising Machine Learning to Predict Traffic Congestion under Uncertainty.

[3] Extending Power of Nature from Binary to Real-Valued Graph Learning in Real World.

[4] Experimental investigation of performance differences between coherent Ising machines and a quantum annealer.

We are thrilled to hear that our revisions have addressed your concerns. We deeply value your insightful and constructive feedback and sincerely appreciate your recognition and appreciation of our work, which affirm our efforts in exploring this emerging area. Once again, we thank you for your thoughtful inputs and for acknowledging the potential impact of our contributions!

Thanks for the authors responses.

I have raised my scores significantly from 3 to 6. The authors have greatly revised their draft and provided relevant information in the discussion. Specifically improvements were made:

• Discussions on Limitations: The authors have now avoided reporting comparision numbers directly of the theoretical results on DS Machines with measured performance of GPUs. The authors have modified the main paper (abstract, introduction) to reflect such changes. The authors also provide a new Section 5 (and Appendix A1) to discuss the limitation concerns of DS machines.
• Improved baselines and metrics: The authors have updated to use vLLM as GPU baselines for throughput and energy. The authors have also updated Table 2 to report on more relevant metrics such as token/s and token/KWh
• Concerns on scalability and cost: The authors have provided a better visualization (Figure 7) with theoretical projections on scalaing curve on training/inference performance. The authors also discuss cost of scaling in Table 3.
• Concerns on nonidealities: The authors provide such studies in Appendix regarding trade-offs on Taylor expansion and robustness on circuit non-idealities such as dynamic noise and static offset.

Reasons on not a higher score. Although I find the work to be interesting and quite possibly the first work to customize DS machines to LLMs, I fail to find too much of a distinction in novelty compared with prior work. While the authors state their mapping method to be the main contribution (Section 3.2), I do not find it as a considerable ''novelty'' factor. I do not consider myself an expert in this domain (especially DS machines), thus I have low confidence and also reluctant to offering a higher score.

Based on all the above, updated version of the paper, the good efforts authors took to address my concerns, I have raised my score to 6.

Overview

Thank you very much for your meaningful feedback which helps us a lot to improve the paper. We have updated a revision version of paper to address your concerns and you can find the contents labled as Reviewer C like "C-Q1". The detailed response to your comments are listed below.

1. Limitations of DS machines (weakness 2 and question):

We deeply appreciate your insightful comments and have added a discussion section in the Appendix to address the limitations and challenges associated with DS machines. Below, we outline our responses to the three main concerns raised in your feedback: stability, precision constraints, and scalability.

Stability:

The stability of DS machines is inherently promising due to their fundamental characteristics. Compared to modern GPUs, DS machines consume significantly less power, which reduces the risk of overheating. Additionally, as DS machines rely on a stochastic annealing process rather than deterministic computation, they are naturally more robust to noise and non-ideal factors [1]. To further support this, we included a new experiment to evaluate the impact of noise on the accuracy of DS machines. The results demonstrate that DS machines exhibit high stability and robustness under various noise conditions.

Precision Constraints:

The precision of computation in DS machines is inherently continuous due to their analog nature. However, the precision of Analog-to-Digital Converters (ADCs) and Digital-to-Analog Converters (DACs) used in the system do affect the overall accuracy. Fortunately, ADCs and DACs are well-established technologies with numerous mature solutions that allow for various design trade-offs—for instance, achieving 16-bit precision with lower power consumption or 32-bit precision with higher power consumption. Such trade-offs align with and are theoretically compatible with existing quantization techniques, providing flexibility to balance precision and power efficiency based on the application requirements.

Scalability Issues:

While prior work like NP-GL was designed for small graphs with fewer than 1,000 nodes, the capacity of DS machines can be significantly expanded to handle much larger scales. First, DS machines have demonstrated linear complexity with respect to the number of nodes [2], making them inherently scalable with increased chip area. For context, NP-GL occupies only about 5 mm², whereas modern GPUs like the H100 have a die size of 814 mm², and wafer-scale chips—such as those with up to 46,255 mm²—are emerging [3]. Based on linear complexity, a single-chip solution could theoretically support millions of nodes within a single DS machine.

Second, for even larger models or faster training, multi-chip approaches offer a viable path forward. Existing research has explored multi-chip solutions for DS machines [4], where individual chips perform annealing with periodic synchronization. We are also investigating promising techniques such as deploying models across multiple DS machine chips to achieve pipeline parallelism.

Overall, the scalability of DS machines is theoretically well-founded, offering both single-chip and multi-chip solutions to meet the demands of increasingly large and complex models.

Reference:

[1] Reasons for the superiority of stochastic estimators over deterministic ones: robustness, consistency and perceptual quality

[2] DS-GL: Advancing Graph Learning via Harnessing Nature’s Power within Scalable Dynamical Systems

[3] Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]

[4] Increasing ising machine capacity with multi-chip architectures

2. Comparison under more LLM sizes (weakness 1)

We sincerely appreciate your insightful suggestions. In response, we have included evaluations on two larger-sized models including OPT-2.7B and Llama2-7B to better assess our solution's performance. The results demonstrate that DS-LLM continues to offer a comparable accuracy and significant speedup & energy reduction over traditional GPU-based solutions.

The updated tables are shown in next comments.

Overview:

Thank you very much for your insightful and meaningful feedback, we have uploaded the revised paper to address your concerns and you can find the contents according to the lable for Reviewer B like "B-Q1". We would like to list our detail response to your questions as below:

1. Comparison with implementations for edge devices (weakness 1):

We sincerely appreciate you for your valuable suggestion. We have included a new comparison in the revised version to evaluate our work against low-precision CPUs [1] and emerging edge devices [2] on the Llama2-7B model. This model was specifically chosen as both referenced studies evaluated it in their work, allowing for a direct and meaningful comparison. For power estimation of the CPU work [1], the authors did not provide the exact power consumption of the “4th Generation Intel Xeon Scalable Processors” used. Hence, we conservatively assumed a power consumption of 100W, which is lower than any product in the stated CPU series. The results show that DS-LLM achieves orders of magnitudes higher token generation rate and energy efficiency than the references. This is because both the low precision CPU works and edge devices are still digital computing diagram which follows a step-by-step instructions-based computing, while DS machines don’t need micro instructions and perform continuous natural annealing by analog currents which is extremely fast and energy efficient.

Reference:

[1] Efficient LLM Inference on CPUs (https://arxiv.org/pdf/2311.00502)

[2] Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM (https://arxiv.org/pdf/2409.15654)

2. Hardware limitations (weakness 2 and question 1):

We sincerely appreciate your insightful comments, which help us a lot to improve this work. We have added a discussion section in the Appendix to address your concerns on hardware limitations including scalability and model flexibility.

Scalability Issues:

While prior work like NP-GL was designed for small graphs with fewer than 1,000 nodes, the capacity of DS machines can be significantly expanded to handle much larger scales. First, DS machines have demonstrated linear complexity with respect to the number of nodes [3], making them inherently scalable with increased chip area. For context, NP-GL occupies only about 5 mm², whereas modern GPUs like the H100 have a die size of 814 mm², and wafer-scale chips—such as those with up to 46,255 mm²—are emerging [4]. Based on linear complexity, a single-chip solution could theoretically support millions of nodes within a single DS machine.

Second, for even larger models or faster training, multi-chip approaches offer a viable path forward. Existing research has explored multi-chip solutions for DS machines [5], where individual chips perform annealing with periodic synchronization. We are also investigating promising techniques such as deploying models across multiple DS machine chips to achieve pipeline parallelism.

Overall, the scalability of DS machines is theoretically well-founded, offering both single-chip and multi-chip solutions to meet the demands of increasingly large and complex models.

Model Flexibility Issues:

This work focuses on classic operations in Transformer-based LLMs, acknowledging that modern LLMs may include different operations such as varied activation functions or embedding methods. Despite the diversity of LLM architectures, these operations can generally be categorized as either polynomial or non-polynomial. Polynomial operations, which can be broken down into basic addition and multiplication, are directly mappable to DS machines. Non-polynomial operations, such as exponential function, require transformation into polynomial approximations (e.g., via Taylor expansion) or the addition of auxiliary circuits, which may slightly increase latency depending on their complexity. Fortunately, most high-computational-demand operations, particularly in attention layers and FFNs, are polynomial or even linear. Thus, DS machines offer high flexibility and adaptability for various models.

Reference:

[3] DS-GL: Advancing Graph Learning via Harnessing Nature’s Power within Scalable Dynamical Systems

[4] Wafer-Scale Computing: Advancements, Challenges, and Future Perspectives [Feature]

[5] Increasing ising machine capacity with multi-chip architectures

3.Introduction revision (weakness 3):

Thank you for your helpful suggestions. We have revised the introduction to make it clearer.

*Due to character limitation, other responses wil be listed in next comments.

The discussions over the past few days have been incredibly insightful and have greatly contributed to improving our paper. Much appreciated for the reviewer's suggestions. With the extension of the discussion period, we see this as a rare and invaluable opportunity to further engage with the reviewer and learn from their expertise and wisdom.

Therefore, we would like to take this opportunity to clarify the new contributions of this work compared to prior art (e.g., NP-GL), beyond the mapping of LLM kernels onto DS machines. Specifically, DS-LLM (this work) represents the first effort to enable on-device training on a dynamical systems (DS) machine, thereby extending the computational capabilities of DS machines from inference to training. In contrast, prior works restricted DS machines to inference tasks, requiring training to be conducted on GPUs.

Once again, we deeply appreciate the reviewer’s recognition of the potential of our work in advancing LLM development and their constructive feedback throughout this process. We are always eager to receive any additional suggestions.

Overview

We sincerely thank you for the valuable suggestions, which have significantly helped us improve the quality of this paper. We have uploaded a revised version addressing the points you raised and you can find the contents according to the label for reviwer A like "A-Q1". The specific response to your questions are as below:

1. Examples of real-world applications (weakness 2 & question 1):

Thank you for your suggestion and we have incorporated concrete examples of DS machines being applied to real-world problems in the introduction of the revised manuscript. The applications include traffic predictions in [1], air quality, taxi demand, and pandemic progression in [2], and optimization problems like MAX-CUT in [3].

References:

[1] Ising-Traffic: Using Ising Machine Learning to Predict Traffic Congestion under Uncertainty.

[2] Extending Power of Nature from Binary to Real-Valued Graph Learning in Real World.

[3] Experimental investigation of performance differences between coherent Ising machines and a quantum annealer.

2. Feasibility of DS machines (weakness 1 & question 2):

Thank you very much for this insightful feedback and we have added a discussion section in Appendix to analysis the feasibility of DS machines in detail. While this work is early-stage research exploring the potential of DS machines to meet the growing computational demands of LLMs, we would like to discuss the following key points address their feasibility and future integration:

First, we want to highlight that there are no fundamental challenges to integrate DS machines into existing computing systems. DS machines, though architecturally distinct from digital processors, are built using CMOS-compatible technology. This compatibility ensures they can be integrated seamlessly into existing systems as co-processors (similar to TPUs or NPUs) via interfaces like PCIe. Theoretically, no fundamental hardware adaptations are required.

From a system perspective, though this work is still in early stage, we agree that integration to the existing computing infrastructure is very promising as future exploration. There are a lot of exploration space on developing software toolchain like compilers, optimizing memory management, and pipelining tasks between DS machines and other processors. With all these works and software tools help, we believe DS machines are inherently feasible for integration into digital systems and work as a new type of co-processor like GPUs / TPUs / NPUs. Future work can explore hybrid use cases that combine CPUs, GPUs, and DS machines based on their unique strengths to achieve optimal performance.

Overall, while DS machines are not yet mature, their fundamental feasibility paves the way for exciting opportunities in integration with existing computing infrastructures.

3. Detail of the FEA emulator (question 3):

Thank you for your meaningful suggestion. The FEA software emulator we used is adapted from the one used in BRIM, which has been validated to provide results comparable to actual chip prototypes. EFA is a SOTA simulation approach for circuit-level simulation that incorporates the physical parameters such as capacitance, resistance, and temperature, simulating voltage and current evolution in tiny time steps (100 ps). In essence, it functions as a GPU-accelerated poor-man’s circuit-level simulation tool for Natural Annealing processes.

4. Error of FEA simulation results (question 4):

Thank you for raising this insightful point. While our FEA simulation assumes ideal conditions, it is worth noting that actual chips, including conventional CPUs/GPUs, often encounter non-ideal factors during manufacturing. We have not yet manufactured a prototype for this specific work, but the evaluation results on a prototype for previous work shows a less than 1% error between the simulation and actual chip. The previous prototype was designed for graph problems but fundamentally has the same basic components as this work. Therefore, we believe the judgement can be extended to this work that the circuit-level EFA simulation is a stable approach for DS machines.