FedDES: A Discrete-Event Simulator For Large-Scale Federated Learning

1.Limited Discussion of Scalability Bounds:The paper doesn't thoroughly explore the upper limits of FedDES's scalability；No clear discussion of memory requirements or computational complexity. 2.Validation Scope:Evaluation focuses mainly on Vision Transformer with CIFAR-10;Could benefit from testing with more diverse models and datasets; Limited exploration of edge cases or failure scenarios 3.Network Modeling:While network delays are considered, there's limited discussion of complex network topologies or dynamic network conditions; The paper could benefit from more detailed analysis of how network conditions affect simulation accuracy

There are several significant weaknesses in the current version of the paper.

The paper claims that FedFS can help accelerate FL developing and debugging cycles. The key advantage should be providing modular design and clear interfaces, allowing the FL developers to interact with the simulator and to plug in their own FL protocols, optimization strategies and datasets. However, this paper does not present any of these aspects.

In section 4, it states that Section 3.1 formalizes FedDES’s state machine for FL event management. But this is presented in a descriptive manner rather than formally. As a result, the subsequent theorems and proofs lack solid theoretical foundation and depth.

I also have the reservation whether this work, developing a FL simulator, falls into the scope of ICLR.

• My major concern is the contribution of the paper. At a high level, this is a simulator with the aim of generating almost accurate results after real-world deployment. Yet, it remains unclear what the real-world deployment environment means.
• Next, the authors claim that FedDES generates event logs in a few seconds. However, the contribution here seems to be incremental. It is a must for every simulator, not only FL, to log data in detail for faster debugging.
• The related work seems to be a bit off either. Section 2.1, which is tiltled FL aggregation strategies, describes many methods that differ from each other far beyond aggregation themselves. In addition, although the FL algorithms' categories appear intuitive, the authors fail to capture the key components that differentiate different categories and affect their implementations of the FL simulator. Moreover, given the rapid development of FL research, the citations inside are far from enough (now is ~2 per strategy).
• In conventional FL setups and in most of the machine learning tasks, the trainers have to follow the protocols set by the algorithms. I failed to find something novel unless I missed something.
• The pseudocode in Algorithm 1-3 is not helpful. This is because most of the lines capture how the clients/ server time the training progresses without contextualizing what the real problem is there.
• The theorems proved in Section 4 seem to be trivial since all the simulators need to follow the rules as specified by the algorithms.
• In Section 5, the implications of three divergence measures are almost the same. The authors need to provide performance comparisons with the other FL simulators in terms of the 3D things. The current evaluations seem to be narrow.
• In Section 5, the definitions of real-world deployment are still missing. It also remains unclear whether FedDES can simulate a given type of GPU simulation results or whether it is suitable for a family of hardware devices.
• In Section 5, beyond purely comparing the fast implementation speed of FedDES, the authors also need to verify its correctness by testing the same client population, neural network models, and local datasets across different simulators.