FedECADO: A Dynamical System Model of Federated Learning

We would like to thank the reviewer for their comments and hope the following addresses their concerns.

---

Continuous-time ODE

The key innovation of FedECADO over the framework proposed by Agarwal and Pileggi (2023) is its introduction of circuit- and simulation-based techniques to address the unique challenges of heterogeneous computation in federated learning. While the previous framework was designed for traditional distributed optimization, assuming homogeneous and always-available worker nodes, FedECADO is specifically designed to handle heterogeneous clients with varying computational capabilities, learning rates, non-IID data distributions, and availability.

At the core of FedECADO is a continuous-time ODE model, which reframes the challenges of heterogeneous client learning rates as a distributed simulation problem. In this model, each client's sub-circuit is defined by local loss function and dataset. By modeling federated learning as a continuous-time ODE, it becomes apparent that client drift due to heterogeneous client computation (where each client has different learnings) is a result of simulating each client for different time-scales.

Building on this insight, FedECADO maps the federated learning process to an equivalent circuit to enable the use of circuit simulation to handle the challenges of asynchronous distributed updates. Additionally, this circuit model introduces a new way to model the client sensitivity using a Thevenin impedance, a concept well-established within circuit literature. As part of the novelty of FedECADO, we were able to integrate this sensitivity into the central agent step to improve the convergence rate of the overall federated learning process.

The mapping to continuous-time ODEs and circuit models is outside the realm of this community, however, brings new a perspective to address the challenges in federated learning. In this regard, we will add a background section to the supplementary material that covers ordinary differential equations and an understanding of circuit theory to provide the reader with more appreciation on how we derived our methodology.

---

Dataset size

The value of $p_i$ refers to the relative number of samples of each client’s training set. The relative weighting allows us to ensure clients with larger datasets have greater influence on the central agent server.

---

Computational Cost of (33)

The main computational cost of performing FedECADO’s central agent step involves computing the Backward-Euler integration step in equation (33). We leverage a constant sensitivity model for each client to ensure that the left-hand side matrix in equation (33) is constant over the simulation. This allows us to pre-LU factor the large matrix prior to training process. During the training process, each epoch of the central agent step performs a forward-backward substitution of the pre-computed LU factor. This helps reduce the computational complexity of FedECADO as well as the overall runtime. The difference in the computational runtime is highlighted in Appendix D.

We would like to thank the reviewer for their comments.

---

Hessian Approximation

In federated learning with non-IID data distribution, each client’s local loss function is a function of its unique dataset. The Hessian captures the curvature of each loss function at the client’s specific operating point. As a result, different tasks, domains and scenarios will create unique solution landscapes whose curvature can be measured using the Hessian. We use this Hessian to measure how sensitive a client’s model is to changes from the global model during the central aggregation step. This sensitivity helps the server anticipate the impact of model aggregation on individual clients.

We approximate a constant Hessian for each client by sampling multiple data points and averaging their respective Hessians. Future work will improve this approach by using prior work (such as [R2]-[R4]) to improve efficiency and accuracy.

---

Ablation Study

In this work, the server step-sizes are modeled as time-steps for a Backward-Euler integration. These step sizes are adaptively selected based on the local truncation error, a measure of how closely we are tracking the designed ordinary differential equation. As a result, we do not select the step sizes, but rather select the tolerance for the discretization error due to Backward-Euler integration, denoted as $\delta$ in equation (36). We find that the value of $\delta$ does not provide much difference in the final simulation accuracy. We have added a figure, shown in [R1] where $\delta$ is changed by three orders of magnitude (assuming a fixed client step size) for training on CIFAR-10 dataset. From the figure, we notice that the convergence plots remain well-behaved regardless of the choice of $\delta$. Future work will look at designing the value of $\delta$ based on approximating the Lipchitz constant of the gradient-flow updates.

---

Communication Cost

The advantage of FedECADO in terms of communication is that we can handle client updates with heterogeneous client learning rates with only an additional scalar constant being communicated at each epoch. This scalar constant, $\Delta T$, measures the amount of time each client is simulated for.

---

Dynamic Clients

Adding and removing clients during the federated learning process can definitely be mapped to FedECADO's circuit simulation framework. We can model this behavior similarly to the switching of transistors in circuits, where transistors are added in series with clients and are “turned on or off” to connect or disconnect clients from the central agent. Circuit simulators, with their deep history in simulating discrete events, are well-suited for this type of modeling.

---

[R1] https://drive.google.com/file/d/1OZRDtok0Ou6RdELJ2Eb13N7_4TJaU-J3/view?usp=share_link

[R2] Elsayed, M., Farrahi, H., Dangel, F. and Mahmood, A.R., 2024. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. arXiv preprint arXiv:2406.03276.

[R3] Elsayed, M. and Mahmood, A.R., 2022. Hesscale: Scalable computation of hessian diagonals. arXiv preprint arXiv:2210.11639.

[R4] Yao, Z., Gholami, A., Keutzer, K. and Mahoney, M.W., 2020, December. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) (pp. 581-590). IEEE

Thank you for your comments.

---

Novelty of FedECADO

Regarding novelty, FedECADO is inspired by distributed circuit simulation, where sub-circuits are independently simulated and recombined using waveform relaxation (White and Sangiovanni-Vincentelli, 2012). Our approach extends waveform relaxation by incorporating inductors and constant sensitivity models to accelerate convergence. The key innovation lies in establishing a connection between federated learning and circuit simulation, allowing simulation principles to be adopted to federated learning.

FedECADO extends the circuit model of distributed optimization presented in (Agarwal and Pileggi, 2023) by introducing multi-rate integration and chord-based modeling to address the unique challenges of federated learning. Unlike in distributed optimization, where the worker nodes are homogeneous and always available, federated learning can have heterogeneous client computation capabilities where computational nodes are not available simultaneously and exhibit varying local learning rates. The multi-rate integration method is a key innovation of our work that can handle these new challenges. Importantly, the circuit model provides a new perspective to these challenges (such as framing heterogeneous client learning rates as asynchronous communication), which enables the development of intuitive and effective heuristics. FedECADO provides the first bridge between these fields, enabling new research directions where physical principles and simulation methods can drive federated learning solutions.

---

Optimization Plots and Update on Table 1

To the reviewer’s request, we have added the plots for training CIFAR-10 in the following link (in addition to the supplementary): https://drive.google.com/file/d/1Ye1AiKFR2c8Xz2Tmps6x387WG8l0xlte/view?usp=share_link

We also would like to address the inconsistency between the text and Table 1. We have an unfortunate typo in the previous version and have have updated Table 1 (specifically FedRS, FedDecorr, and FedExp) to match the given plots.

The convergence plots for this CIFAR-10 example is provided in the link below:

https://docs.google.com/document/d/e/2PACX-1vTamA9cixTmdcZecNeVjy7PZ9i-NuQi9c53d0zG2wPzjTg2tuoF4K_aGFIVZgLt096D0189JNK235ht/pub

---

Background on circuit details

We understand that the inspiration for FedECADO (namely circuit-based ODEs) is not within the realm of the community. We plan on adding additional supplementary information that gives a primer on circuit physics and simulation in the appendix.

We would like to thank the reviewer for their comments.

---

Novelty of FedECADO

FedECADO builds on the distributed optimization method, ECADO, and introduces key innovations (multi-rate integration and chord-based modeling) to address the unique challenges in federated learning. Unlike in distributed optimization, where the worker nodes are homogeneous and always available, we tackle heterogeneous client computation capabilities where computational nodes are not available simultaneously and exhibit varying local learning rates and non-IID datasets. We believe the novelty of our approach lies in establishing a connection between federated learning and circuit methods, which offers a new perspective on these challenges. For example, we re-interpret heterogeneous client learning rates as sub-circuits simulating for different time-periods. This allows us to take inspiration from circuit methods for distributed simulation. While advancements in general optimization have made the connection with circuits to inspire fast-convergent methods [R1]-[R3], FedECADO is the first work that introduces circuit knowledge to federated learning. This provides insights that are easily attained from the circuit model and simulation practices.

---

Aggregate Sensitivity Model

Could the reviewers clarify their specific concerns about aggregate model sensitivity?

We would like to emphasize that in federated learning with non-IID data, each client’s local loss function has a unique solution landscape shaped by its data distribution. The Hessian captures the curvature of the loss function at the client’s specific operating point and local training data. We use the Hessian to estimate the first-order sensitivity of a client’s local model to changes in the global model parameters. This sensitivity is used to anticipate how aggregation steps will affect individual client states. To approximate a representative Hessian for clients, we sample multiple data points and compute an averaged Hessian. Future work in FedECADO can also extend this by approximating the Hessian via [R4]-[R6]

---

Multi-Rate Integration

Could the reviewers clarify their concern regarding multi-rate integration?

To clarify, we are not considering asynchronous client updates where clients begin with different global model versions. Instead, using the circuit model, we interpreted the challenge of heterogeneous client learning rates as clients simulating their local models for different time scales. To remedy this, the central server performs multi-rate integration by accepting the latest client updates, similar to FedAvg. Then, rather than directly aggregating these updates as in FedAvg, FedECADO first applies a linear interpolation/extrapolation operator to synchronize the simulation time scales across clients. The central agent step is then computed using a Backward-Euler integration step. This approach has a process similar to the aggregation step in FedAvg, but adds the additional linear operation, $\Gamma(\cdot)$, to address the challenge of heterogeneous client computation, and uses a Backward-Euler step to maintain numerical stability.

Additionally, in FedECADO, each client also receives the updated global state as shown in Algorithm 2 in the main text.

---

Improving the background on circuit-based framework

As part of improving the readability, we will include a section in the appendix that provides a deeper background on ODEs and circuit analysis.

---

Formatting

Thank you for the suggestion. In the following revision, we will improve the layout of Tables 4 and 6 as well as the equations.

---

Update on Table 1 and Optimization Plots

Thank you for pointing out the inconsistency in Table 1. This was an unfortunate type in the previous version and is updated to reflect true classification accuracy as shown in the response for Reviewer xyRN.

---

[R1] Boyd, S., Parshakova, T., Ryu, E. and Suh, J.J., 2024. Optimization algorithm design via electric circuits. Advances in Neural Information Processing Systems, 37, pp.68013-68081.

[R2] Agarwal, A., Fiscko, C., Kar, S., Pileggi, L. and Sinopoli, B., 2022. ECCO: Equivalent Circuit Controlled Optimization. arXiv preprint arXiv:2211.08478.

[R3] Yu, Y. and Açıkmeşe, B., 2020. RC circuits based distributed conditional gradient method. arXiv preprint arXiv:2003.06949.

[R4] Elsayed, M., Farrahi, H., Dangel, F. and Mahmood, A.R., 2024. Revisiting scalable hessian diagonal approximations for applications in reinforcement learning. arXiv preprint arXiv:2406.03276.

[R5] Elsayed, M. and Mahmood, A.R., 2022. Hesscale: Scalable computation of hessian diagonals. arXiv preprint arXiv:2210.11639.

[R6] Yao, Z., Gholami, A., Keutzer, K. and Mahoney, M.W., 2020, December. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data) (pp. 581-590). IEEE