FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices Using a Computing Power-Aware Scheduler

Dear Reviewer vABe,

First and foremost, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful comments. Our responses to your comments are as follows.

Comparison with Tiered FL algorithms

The primary distinction between tiered FL algorithms and FedCompass lies in their targeted FL settings and client scale. Tiered FL algorithms are specifically designed for cross-device FL settings, where the client count can range from hundreds to thousands and even more. In contrast, FedCompass is designed for cross-silo FL scenarios and focuses on optimizing the use of dedicated computing resources at each silo, typically involving client numbers on the order of ten.

The FedAT algorithm aims to develop a communication-efficient asynchronous FL method (Chat et al., 2021). It features synchronous updates within tiers and asynchronous updates among them. The method employs random client sampling within tiers and rough tier assignment broadly based on client response latency. However, such design choices are not well-suited for cross-silo FL, where all clients are expected to participate in each round of training. This fundamental difference in settings makes it difficult to directly compare FedCompass with tiered FL methods. Moreover, the granularity of tiers in tiered FL algorithms is generally coarse. In scenarios with a small number of clients, it's likely that only one tier is formed, effectively making the algorithm the same as the FedAvg algorithm (McMahan et al., 2017).

In terms of the resilience to client speed changes, we would like to first elaborate on what this means in the context of cross-silo FL. Unlike cross-device FL where a client experiences frequent changes due to the unreliability and intermittent computing of the devices, for cross-silo FL, each client usually has dedicated and reliable computing resources on HPC or cloud. It is possible that the clients have significant yet occasional speed changes due to some auto-scaling behaviors. FedCompass is designed to be responsive and resilient for those significant speed changes as showcased by the ablation study results in Appendix G.2. However, tier FL methods usually lack response to sudden speed changes and may experience performance degradation.

Unclarity in Section 3.2

Thanks a lot for pointing this out and we agree that the previous presentation lacks clarity. Now we have rewrote the paragraph for describing the server side algorithm of FedCompass by focusing on the higher-level idea instead of the low-level details (last two paragraphs at the end of page 5). We believe that now it is a more clear presentation of the algorithm.

Summary of changes in the paper in response to Reviewer vABe

(1) Rewrite Section 3.2 for better presentation clarity, and have the descriptions focus more on higher-level design ideas.

(2) Add few sentences at the first and last paragraphs of Appendix G.2 to further explain the resilience of FedCompass to client speed changes in practical cross-silo FL settings.

Chai, Z., Chen, Y., Anwar, A., Zhao, L., Cheng, Y., & Rangwala, H. (2021, November). FedAT: A high-performance and communication-efficient federated learning system with asynchronous tiers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1-16).

McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282). PMLR.

Dear Reviewer DhHF,

First and foremost, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful questions and comments. Our responses to your comments and questions are as follows.

Unconvincing motivations

The straggler issues in cross-device FL are mainly caused by the unreliability of the clients due to intermittent compute availability and connectivity. On the other hand, in cross-silo FL, though each client is assumed to have reliable and dedicated resources, the straggler issues come from the disparity/difference of the computing power among clients. In practical settings, there usually exists a substantial gap in computing capability between cross-silo FL clients.

Firstly, the increasing adoption of specialized ASICs like those from Cerebras, SambaNova, and Graphcore, particularly by large institutions engaged in FL projects such as national laboratories, is noteworthy. At the other end of the spectrum, institutions like hospitals, also interested in utilizing federated learning, may only have access to standard consumer GPUs or solely CPU-based computing resources. This diversity in the types of accelerators and computing machines used by cross-silo FL clients inevitably leads to substantial disparities in computational capabilities.

Secondly, disparities in computing power are also evident even among clients using the same type of accelerator, such as GPUs. For instance, HPC clusters built with different generations of GPUs and associated support hardware, like networking components, can exhibit significant differences in training capabilities. This is further elaborated in the following NVIDIA post (https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/).

Considering these disparities among cross-silo clients' computing resources, it becomes crucial to address resource utilization efficiently. The aim is to prevent resource wastage, which can occur by having faster clients idly wait for slower ones and underutilizing the available resources. A typical scenario in HPC clusters illustrates this: when a user requests a node with 8 GPUs through a job scheduler like Slurm, they are billed for a fixed number of compute hours, regardless of whether all 8 GPUs are fully utilized throughout the job duration.

Using CIFAR-100 as datasets

In our experiments, we selected the MNIST and CIFAR-10 datasets to evaluate the performance of FedCompass. These two datasets are widely recognized and frequently utilized in FL research, as evidenced by Ma et al. (2022), who noted their prevalent use in the field. According to a survey presented in their study (Figure 9), MNIST is used in 32% of FL experiments, while CIFAR-10 is used in 28%. CIFAR-100, in contrast, is seldom chosen for such experiments. Beyond these traditional datasets, we also assessed FedCompass using two datasets from FLamby (Terrail et al., 2022), a recently introduced benchmark specifically designed for cross-silo FL. These datasets are significantly larger, at 444MB and 9GB, respectively. We believe this diverse range of datasets, both in terms of size and complexity, adequately demonstrates the versatility and efficacy of FedCompass in various scenarios.

Limited technical novelty

At first glance, the concept of assigning varying local steps to clients based on clients' computing speeds might seem straightforward. However, in practice, this approach encounters several challenges and edge cases. For example, these include synchronizing clients when there is no prior knowledge of their computing speeds, addressing minor and significant fluctuations in client speeds, and handling scenarios where there is a substantial disparity in client computing power (e.g., exponential client computing speed distributions).

FedCompass not only addresses these practical challenges effectively but also provides theoretical proof of convergence and demonstrates robust performance through an extensive series of experiments. Therefore, we believe that this paper possesses sufficient novelty, underscoring its contribution to advancing FL methodologies.

Ma, X., Zhu, J., Lin, Z., Chen, S., & Qin, Y. (2022). A state-of-the-art survey on solving non-IID data in Federated Learning. Future Generation Computer Systems, 135, 244-258.

Terrail, J. O. D., Ayed, S. S., Cyffers, E., Grimberg, F., He, C., Loeb, R., ... & Andreux, M. (2022). Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620.

Integrating other server optimization strategies

Algorithm 1 only presents a simple demonstration of the FedCompass algorithm without combining it with other server-side optimization strategies. If the individual client update is needed to combine FedCompass with other strategies such as FedAdam, FedAdagrad, and FedYogi (Reddi et al., 2020), the server can simply buffer each client’s update instead of adding them up. Therefore, it only requires some minor changes in the implementation.

Adding ablation study on CIFAR-10

Sure! We have now added ablation study results for $Q_{\min}/Q_{\max}$ on the class partitioned CIFAR-10 datasets in Table 3 as well. The results are consistent with those on the MNIST dataset, namely, FedCompass can outperform FedBuff (Nguyen et al., 2022) with a large range of $Q_{\min}/Q_{\max}$ values.

Typos

Thanks for the reminder! We have fixed it in the revised paper.

Summary of changes in the paper in response to Reviewer DhHF

(1) Cite the survey paper to further explain our choice of experiment datasets in the first paragraph of Section 5.1

(2) Add results for ablation study on CIFAR-10 in Table 3.

(3) Fix typos in equation 6.

Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., ... & McMahan, H. B. (2020). Adaptive federated optimization. arXiv preprint arXiv:2003.00295.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Dear Reviewer DhHF,

Thanks for your reply and we would like to further clarify your concerns more specifically.

Unconvincing Motivation

While MNIST and CIFAR-10 are relatively small datasets, our experimental design is still challenging. We use two partition strategies, resulting in each client holding only a few classes. Further, this differs from cross-device FL scenarios where there are thousands of clients and each client may hold data of 3 to 4 classes. In cross-silo FL, there are only around 10 clients and each client only holds 3 to 4 classes, and this makes the problem much more challenging. As illustrated in the x-axis of Figures 12 to 17, experiments on the CIFAR-10 dataset usually take 1 (asynchronous algorithms) to 6 hours (asynchronous algorithms) to finish, indicating a significant saving of more than 5 GPU hours per client compared to synchronous methods like FedAvg. This translates to a total of 100 GPU hours saved if there are 20 clients. Additionally, we incorporate two datasets (444MB and 9GB, both are larger than CIFAR100 - 161MB) from the FLamby cross-silo benchmark (Terrail et al., 2022) to further demonstrate FedCompass’s effectiveness. Therefore, we believe that our experiment design is still well-aligned with our motivation.

CIFAR-100 Datasets

In the experiment setting of this paper, each speedup/accuracy number is obtained from the average of 10 independent runs, taking 10 to 60 hours per client on CIFAR10 or CIFAR100 in total. Consequently, obtaining all accuracy/speedup numbers on CIFAR100 could exceed 10,000 GPU hours. Due to the limited time available for discussion, acquiring comprehensive CIFAR100 results is currently unfeasible, and we request your understanding in this matter. Nonetheless, we still believe that our findings on the heterogeneous CIFAR10 dataset and the two FLamby datasets could highlight FedCompass’s efficiency and accuracy advantages over synchronous and other asynchronous methods, especially given the fact that CIFAR10 and CIFAR100 actually have the same dataset size (~161 MB) and the used two FLamby datasets have even larger sizes.

Limited Technical Novelty

We think there may exist some miscommunication or misunderstanding in terms of the theoretical contribution of this paper. Here is what we really want to express in terms of our convergence analysis part, and we also updated the responses the reviewer UVPW as well to further clarify things.

(1) The main focus/contribution of our convergence analysis section is deriving the convergence behavior of our newly proposed semi-asynchronous algorithm FedCompass using the commonly-used assumptions and obtaining further insights, instead of trying to get rid of any commonly-used assumptions such as bounded staleness.

(2) Though we are not the first ones to study the convergence behavior of the general asynchronous FL algorithms or FL algorithms with different client local steps, this paper actually first studies the convergence behavior of a semi-asynchronous FL algorithm with a scheduler assigning different numbers of local steps to clients, and we really believe this is still a good contribution to the FL community.

We apologize for any previous confusion and look forward to further discussions.

Terrail, J. O. D., Ayed, S. S., Cyffers, E., Grimberg, F., He, C., Loeb, R., ... & Andreux, M. (2022). Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv preprint arXiv:2210.04620.

I really appreciate the authors for their efforts to address the concerns and clarify the unclear aspects. But I have the remaining concerns below.

1. Unconvincing motivation: Now, I understood there might be a significant gap in computing time between cross-silo FL clients, even if they have sufficient computing resources. However, I think the gap is more meaningful in large models/datasets, and it is unlikely to be a critical issue in small models/datasets such as MNIST and CIFAR-10, which are the main experimental results of this paper.

2. CIFAR-100 dataset: As the number of classes is large, more diverse heterogeneous scenarios can be simulated in FL. Therefore, I was just curious about the scalability of the proposed method with respect to the number of classes. Since this work addresses cross-silo FL where each client has sufficient computing power, I think this analysis would be valuable. I also found that the CIFAR-100 dataset has been adopted in many well-known studies such as FedDC, FedDyn, FedMD.

3. Limited technical novelty: I admit this work tackles a practically important problem in cross-silo FL. However, I found that the authors mentioned the convergence analysis is not a core contribution (in the response to Reviewer UVPW). Considering this, the main contributions of this work seem not to lie in machine learning algorithms but rather in the simple assistance for implementation. Therefore, I am not sure if this is above the acceptance threshold bar of ICLR.

Therefore, for now, I will keep my original score.

Dear Reviewer 7w8f,

First and foremost, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful questions and comments. Our responses to your comments are as follows.

Unclear description in Section 3.2

For Section 3.2, we now rewrote the paragraph for describing the server-side algorithm of FedCompass by focusing on the higher-level idea instead of the low-level details (last two paragraphs at the end of page 5). We believe that now it is a more clear presentation of the algorithm.

How FedCompass alleviates the bias towards faster clients

Compared with other asynchronous FL algorithms such as FedAsync (Xie et al., 2019) and FedBuff (Nguyen et al., 2022), FedCompass alleviates the bias towards faster clients by using group aggregation to reduce global aggregation frequency and avoids the frequent application of staleness factors. We would like to further elaborate this via a simple example:

Suppose that client A is 4 times faster than client B. In FedAsync, if A and B perform the same number of local steps in each local training round, then client A will perform 4 times more local training rounds than client B. When client B’s model arrives, the global model has already been updated four times using A’s model, so B’s local model will become very stale/outdated, and another penalty staleness factor (e.g., say 0.25) will be applied to B’s model during the aggregation. In such a case, B not only performs fewer updates, but its model also experiences an additional penalty factor to further cause the global model to drift away from it. For FedBuff, although it has a buffer to reduce the global update frequency, it is very possible that the buffer contains several local models from the same fast clients and slower clients may also experience some staleness factor. However, for FedCompass, we are grouping the arrival of clients, so in this example, A will perform 4 times more local steps than B and they will arrive almost simultaneously at the server for a group aggregation. In this case, no staleness factor (i.e., that 0.25) is applied to B, so the drift towards A is mitigated compared to other methods.

Of course, from the above description, it is clear that FedCompass only mitigates the drift caused by the stateless factor but does not eliminate it, as client A still performs 4 times more steps than B. The drift can be further mitigated by combining FedCompass with other strategies such as normalized averaging (Nova) proposed in FedNova (Wang et al., 2020). Nova basically normalizes the client’s local gradient according to the number of local steps to mitigate objective bias towards faster clients. Now we also include the experiment results for this combination (FedCompss+N) in the last row of Tables 1 and 2 of the revised paper (which is also the response to another review’s comments). However, it is noteworthy that there is still a tradeoff here: though Nova can mitigate the objective bias, it also diminishes the updates from the faster nodes and may sometimes even lead to slower convergence of the global model.

Questions about equilibrium

Equilibrium means that the number of existing groups and the group assignments do not change. Take the scenario in Figure 1 as an example. Initially, there are two groups {1,2,3} and {4,5}. At 12:00, when clients {1,2,3} finish local training together, the Compass scheduler assigns them to the group of clients 4 and 5 and now all clients are within the same group. Then, we can see that all the clients will continue the training following the same grouping (assuming no drastic changes in the computing speed of clients during training), and the number of local steps performed by each client is proportional to the computing speed of each client.

Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint arXiv:1903.03934.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 7611-7623.

Different behaviors for clients arriving before and after $T_{\max}$

All clients within the same group are expected to arrive at almost the same time for a group aggregation. Therefore, if a client arrives earlier than the latest arrival time $T_{\max}$, then the server will wait for other pending clients (or $T_{\max}$, whichever is earlier) for a group aggregation. Then the clients can obtain new training tasks with the most updated global model. However, if a client arrives later than $T_{\max}$, it means that all other clients within the same group have already finished the group aggregation after waiting until $T_{\max}$. Therefore, there is no point in waiting and the server will immediately assign new training tasks to the client. We think the confusion is mainly caused by the unclarity in the original draft and we hope the revised version conveys the idea more clearly.

Restrictive assumptions for convergence analysis

The first three assumptions are commonly used in convergence analysis of non-convex functions in federated learning and distributed learning literature (Stich, 2019; Li et al., 2019, Yu et al., 2019, Nguyen et al., 2022). In terms of the bounded heterogeneity and staleness assumption, it is valid and reasonable in cross-silo FL, as each client is considered to be reliable. We would also like to clarify that the main focus/contribution of our convergence analysis section is deriving the convergence behavior of our newly proposed semi-asynchronous algorithm FedCompass using the commonly-used assumptions and obtaining further insights (such as the impacts of different hyperparameters/setting on convergence), instead of trying to get rid of any commonly-used assumption such as bounded staleness. Additionally, this paper actually first studies convergence behavior of a semi-asynchronous FL algorithm with a scheduler, and we really believe this is still a good contribution to the FL community.

Citations to other related works

Thanks a lot for providing those additional papers. We find them relevant and useful and have cited them in suitable places in the revised paper.

Summary of changes in the paper in response to Reviewer 7w8f

(1) Rewrite Section 3.2 for better presentation clarity, and have the descriptions focus more on higher-level design ideas.

(2) Add one sentence at the end of Section 3.1 to explain the property of equilibrium - fixed number of existing groups and group assignments.

(3) Cite additional papers at suitable places.

Stich, S. U. (2018). Local SGD converges fast and communicates little. arXiv preprint arXiv:1805.09767.

Li, X., Huang, K., Yang, W., Wang, S., & Zhang, Z. (2019). On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189.

Yu, H., Yang, S., & Zhu, S. (2019, July). Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 5693-5700).

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Hyperparameter problems for FedBuff and FedCompass

First, thanks for the reminder to adopt FedBuff instead of FedBuffer for consistency with the original paper (Nguyen et al., 2022). We have changed all FedBuffer to FedBuff in the revised paper now.

Second, we would like to answer why we do not simply use $K=10$ as the original FedBuff paper does, especially given that their paper mentions that “Our extensive empirical evaluation finds that $K=10$ is a good setting across benchmarks and does not require tuning”. This is because all the experiments in FedBuff are in cross-device FL settings, and have at least thousands of clients in all experiment settings (CelebA dataset: 9343 clients, Sent140: 660120 clients, and CIFAR-10: 5000 clients). However, that statement does not hold for cross-silo cases, as we have less than or equal to 10 clients in most experiments. When the number of clients becomes small, the users have to decide the buffer size themselves instead of directly using $K=10$.

Then, we would like to describe how we decide the value of $K$ for different numbers of clients. We use the class partitioned MNIST dataset with {5, 10, 20} clients and normal distribution as the client heterogeneity setting. For 5 clients, we search $K$ among values {2,3}, for 10 clients, we search $K$ among {2, 3, 5}, and for 20 clients, we search $K$ among {3, 5, 10}. We select $K$ with the highest average accuracy of three independent runs. According to the table below, we finally select $K=3/3/5$ for 5/10/20 clients. For Fed-IXI, we choose $K=2$ as there are only three clients. For Fed-ISIC2019 with 6 clients, we choose $K=3$ as we use this buffer size for both 5 and 10 clients. Additionally, we would like to mention that from the training process visualization figures (Figure 2 and more in Appendix F), we can clearly see that there is a non-marginal gap between the FedCompass curve and the FedBuff curve in many cases, especially for the relatively complicated datasets, CIFAR10 and Fed-ISIC2019.

Finally, regarding the hyperparameter tuning, we would like to elaborate on the reason why we state that FedCompass requires easier tuning. For FedCompass, it has three algorithm-specific hyperparameters: $Q_{\min}$, $Q_{\max}$, and $\lambda$, or equivalently, $Q_{\max}$, $Q_{\min}/Q_{\max}$, and $\lambda$. For FedBuff, it requires $Q$ and $K$. $Q_{\max}$ in FedCompass and $Q$ in FedBuff are both used to define the amount of computational work we want each client to do in each round, so they cancel each other out. The ratio $Q_{\min}/Q_{\max}$ is irrelevant to the number of clients, and we empirically show in the ablation study that a large range of values can lead to faster convergence than FedBuff (we now also added new ablation study results on the CIFAR-10 datasets to address another reviewer’s request, and the observation remains on the more difficult CIFAR-10 dataset). Similarly, for $\lambda$, its impact on the convergence is also minor and a number slightly above 1 is fine. On the other hand, for $K$ in FedBuff, though their paper mentions $K=10$ is a good choice when the client numbers are large, there does not exist any rule of thumb for selecting $K$ when the number of clients is small in cross-silo settings, so we had to do some small preliminary experiments to choose $K$ for different numbers of clients. For FedCompass, we simply choose $Q_{\min}/Q_{\max}=0.2$ and $\lambda=1.2$ for all experiments (which are not even optimal in some cases), and FedCompass still outperforms FedBuff all the time with a non-marginal gap.

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., & Huba, D. (2022, May). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics (pp. 3581-3607). PMLR.

Dear Reviewer UVPW,

First and foremost, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful questions and comments. Our responses to your questions are as follows.

Comparison between FedCompass and FedNova

The main focus of FedNova (Wang et al., 2020) is how to eliminate objective inconsistencies when the number of local steps for clients is different (via adjusting weighting factors). In practice, FedNova is typically implemented in the following two synchronous manners: (1) Each client performs the same number of local epochs, thus different number of local steps due to different data sizes. In this case, the algorithm suffers a lot from client computing speed variations like the FedAvg algorithm. In the worst case, the slowest client may have the largest dataset, and the whole training process can be extremely slow. (2) Setting a universal completion time in each round and letting each client perform local updates until the completion time is reached. This approach eliminates the long waiting time and also has a simple implementation process, but it requires the universal completion time to be manually selected for different training tasks, model sizes, and average client computing capabilities.

On the other hand, FedCompass is designed to be an asynchronous federated learning algorithm, and focuses on how to employ a scheduler to have asynchronous clients arrive almost simultaneously. This is achieved by assigning different numbers of steps dynamically using real-time performance metrics. This design choice, while marginally complicating the implementation, significantly streamlines the experimental setup, as users only need to provide a range of desired local update steps in each round (which is more intuitive to users) and do not need to empirically estimate a suitable universal completion time. There is a tradeoff between ease of implementation and usage convenience.

Furthermore, we would like to clarify that FedNova and FedCompass are not analogous but actually “orthogonal” to each other. As FedCompass involves different client local steps in each round, it can be combined with FedNova’s normalized averaging strategy as a manner to further mitigate the drifting to faster clients. To further emphasize the FedNova paper and explain the orthogonal relationship between FedCompass and FedNova, we now provide a new set of experiment results for their combination in the last rows of Tables 1 and 2 (FedCompass+N) of the revised paper (as well as all result tables and figures in the Appendix).

Wang, J., Liu, Q., Liang, H., Joshi, G., & Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33, 7611-7623.

FedCompass’s behavior with speed changes

Different from cross-device FL where each client may experience frequent speed changes due to power constraints, the operation of other concurrent applications, or network connectivity issues. In practice, a cross-silo FL client usually has dedicated computing resources such as HPC with a job scheduler or some cloud computing facilities—where significant changes in computing speed are less frequent, generally arising from events like auto-scaling. Our experiment design in Appendix G.2 is an overestimate of the practical cases, where each client has a 10% probability in each round to resample a “speed” from the client speed distribution (a simulation for auto-scaling behaviors), and we think that this setup could be a reasonable simulation for speed variations in practical cross-silo FL settings. Of course, in the worst-case scenario where all client speeds change dramatically all the time, then the scheduling algorithm of FedCompass will fail to synchronize the client arrival as desired, since the previous speed information becomes obsolete. In such a scenario, the training will go completely asynchronously and FedCompass will downgrade to the FedAsync algorithm (Xie et al., 2019).

Convergence analysis [updated]

(1) The main focus/contribution of our convergence analysis section is deriving the convergence behavior of our newly proposed semi-asynchronous algorithm FedCompass using the commonly-used assumptions and obtaining further insights (such as the impacts of different hyperparameters/settings on the convergence), instead of trying to get rid of any commonly-used assumptions such as bounded staleness.

(2) Though we are not the first ones to study the convergence behavior of the general asynchronous FL algorithms or FL algorithms with different client local steps, this paper actually first studies convergence behavior of a semi-asynchronous FL algorithm with a scheduler, and we believe this is still a good contribution to the FL community.

Summary of changes in the paper in response to Reviewer UVPW

(1) Add experiment results for combining FedCompass and normalized averaging (Nova) to further clarify the “orthogonal” relationship between FedCompass and FedNova in Tables 1 and 2.

(2) Add a few sentences at the end of the first paragraph in Appendix E.2 to explain the choice of buffer size K and the reason why we do not simply choose $K=10$.

(3) Add sentences at the end of Appendix G.2 to explain that our experiment design is aimed to mimic the occasional yet substantial changes in speed that may arise in real-world cross-silo federated learning environments as a result of auto-scaling. We also point out the limitations of FedCompass if all clients experience dramatic and very frequent speed changes.

Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint arXiv:1903.03934.