CO2: Efficient Distributed Training with Full Communication-Computation Overlap

Dear reviewer,

Thanks for the comments and questions. Please see below for our response to your concern.

Q1: How many runs for each task? Conducting the experiment only once may not yield persuasive results

A1: We thank for the helpful comments. All runs in the convergence experiments were executed five times with distinct random seeds to ensure robust results. And for the throughput experiments, the presented throughput results are averaged values extracted from iterations 100 to 200 for each experiment, which is in order to reduce fluctuations. We will emphasize these experiment details in the forthcoming rebuttal revision version of the paper.

Dear reviewer,

Thank you for your valuable and constructive feedback, which contributes a lot to the enhancement of our paper. Below, we provide our responses addressing your concerns. And all the revisions will be updated in the forthcoming rebuttal revision version of the paper.

Q1: The trade-off between convergence and throughput benefits

A1: Thanks for the helpful comments. The omission of throughput results in Table 1 and 2 stems from (part of) our convergence experiments being conducted on 8 A100 nodes connected through a high-speed RoCE RDMA network. And the investigated model sizes were relatively small, which mitigates the discernibility of CO2's speed advantage. As rightly pointed out by you and other reviewers, we recognize the significance of showing the trade-off between convergence and throughput benefits for CO2, we have addressed this gap by providing the missing throughput data in the below Tables. The presented throughput results represent averaged values extracted from iterations 100 to 200 for each experiment. Given the extensive nature of our experiments, convergence outcomes in Table 1 and 2 originate from two distinct clusters. Specifically, the IC and ALM tasks were executed on an A100 cluster equipped with RoCE RDMA high-speed inter-node connections, featuring 8 nodes and a total of 64 GPUs. Conversely, the SS, PC, and BLM tasks were conducted on a 3090 cluster with a standard TCP/IP Ethernet inter-node connection, comprising 4 nodes and 32 GPUs. Upon observing tasks trained on the A100 platform, it is evident that the throughput of CO2 surpasses other counterparts, although with a slight advantage. This advantage becomes more pronounced on tasks trained on the 3090 platform. It is reasonable to infer that this advantage will be amplified on larger clusters, especially those with slower inter-node connections.

Prior to commencing large-scale convergence experiments, we meticulously tune the values of $\tau$ for each task using a simple grid search strategy within the range of {1, 3, 6, 12, 24, 48, 96, 192} to reach a balance of accuracy and throughput. Hyperparameter tuning experiments are conducted on smaller versions of models when the model to be trained is large, owing to the associated high time and resource costs. We start the tuning of $\tau$ from 1 to larger candidate values given the consideration of high-speed communication on the corresponding platform. It is worth noting that, in addition to CO2, other methods such as Local-SGD/Adamw and SlowMo also require tuning of $\tau$. In practice, we exclusively tune $\tau$ for CO2 and employ the same tuned $\tau$ values for Local-SGD/Adamw and SlowMo. This is because of the similar role played by $\tau$ in these local-updating methods. Employing identical $\tau$ values for different methods on the same task ensures a fair comparison of their convergence and training throughput.

* For IC and SS tasks, we present the throughput results in "images/s", the image resolution is 224x224; For PC task, we present the throughput results in "point clouds/s", each point cloud has 1024 points.

* For ALM and BLM tasks, we present the throughput results in "tokens/s".

Q4: The compatibility with parallelism technologies

A4: We appreciate the comprehensive assessment provided. Given the additional memory footprints associated with asynchronous updates, the compatibility of CO2 with various parallelism techniques becomes an indispensable consideration, especially in the context of large-scale model training. During the implementation of the algorithm, we systematically investigated its compatibility with ZeRO-series optimizers. Currently, ZeRO-1, ZeRO-2, and ZeRO-3 are integrated into the inner loop training of CO2, with the exclusion of the outer loop. We conducted thorough experiments to validate the effectiveness and efficiency of these integrations. In particular, with respect to ZeRO-3, which entails more sophisticated communication during both forward and backward passes, we had considered its compatibility with CO2. Given CO2's local-updating behavior in the inner loop, continuous gradient synchronization is deemed unnecessary at each step. As a result, communication operations for CO2 with ZeRO-3 involve only two all-gather collective communication primitives at the beginning of the forward and backward passes, respectively, which differs from the original implementation of ZeRO-3.

Concerning tensor parallelism, sequence parallelism, and their integration with CO2, while we have not implemented them at this moment, their mechanisms appear non-conflicting with CO2's inner loop training. Tensor parallelism divides large matrix multiplications across multiple devices, and sequence parallelism divides the input sequence into chunks, processing them simultaneously on different devices for efficient training of long sequences. We will try to integrate them with CO2 in the future work.

Q2: The memory overheads of CO2

A2: Thank you for the comprehensive review. As an asynchronous method, CO2 leverages the advantages of a high overlap ratio facilitated by asynchronous communication, while contending with an augmented memory footprint due to the one-step asynchronous delay. Here we present a preliminary comparative analysis of the memory footprint of the methods discussed in the paper, using the widely adopted Adamw optimizer as the baseline and excluding considerations of mixed precision. Local-Adamw, a variant of Adamw, conducts local forward and backward passes and updates parameters without synchronizing gradients. It shares the same memory footprint as Adamw but exhibits lower communication frequency and diminished accuracy performance. SlowMo, can be seen as a variant of Local-Adamw, introduces slow momentum calculation and outer model updates to enhance accuracy. Despite maintaining the same communication frequency as Local-Adamw, SlowMo incurs an additional memory footprint (twice that of the model itself) due to the introduced slow momentum and outer updates. In comparison, CO2 introduces a one-step-delay outer momentum and outer model updates to overlap communication with multiple steps of local-updating computation, enhancing communication-computation overlap. This improvement comes at the cost of an additional memory footprint (twice that of the model itself) compared to SlowMo, attributed to asynchronous operations.

Despite the heightened memory overhead introduced by CO2, it enables full overlap (in the best case) of communication and computation, particularly advantageous for large clusters with limited inter-node connections. Techniques such as ZeRO-1, ZeRO-2, and ZeRO-3 have been incorporated to alleviate redundant memory footprint issues associated with optimizer states, gradients, and model parameters within the inner loop training of CO2. These measures could mitigate the adverse effects of CO2's augmented memory overheads. Furthermore, some ongoing efforts are dedicated to refining the CO2 algorithm in subsequent versions, with a focus on mitigating additional memory overheads.

Q3: The non-decay learning rate consideration

A3: Thank you for bringing up this point. Indeed, the assumption of a decaying learning rate when designing the staleness gap may be considered somewhat restrictive. We acknowledge that our experiments did not thoroughly explore various types of learning rate schedules. To address this concern, we conducted a comparative experiment on GPT-2 (Small) using the CyclicLR schedule provided in PyTorch with the "triangular2" policy (see pytorch link and github link), training for 100K steps. The test results and their comparisons with CosineAnnealingLR are summarized in the table below:

The experimental outcomes consistently affirm the favorable impact of the staleness gap penalty technique on training performance, regardless of the choice between CosineAnnealingLR and CyclicLR. While the enhancements in perplexity (PPL) may not be substantial, they exhibit a reliable positive trend. It is noteworthy that PPL values using CyclicLR surpass those using CosineAnnealingLR, potentially owing to suboptimal learning rate tuning. Despite the cyclic nature of the learning rate in CyclicLR, the staleness gap penalty continues to exert its positive influence. This could be attributed to the periodic decrease in the learning rate during CyclicLR's training iterations, facilitating the efficacy of the staleness gap penalty. It is pertinent to acknowledge that both tested schedules incorporate a learning rate warm-up during the initial 4000 iterations, which also represents a form of increasing learning rate. From a technical standpoint, we posit that there may exist specific learning rate schedules where the staleness gap penalty technique may not exhibit optimal performance. In such instances, we advocate exploring alternative decaying learning rate schedules or considering the option to disable the staleness gap penalty.

Q3: How $\tau$ is set in Table 1?

A3: Thanks for the valuable comments. We agree that describing the tuning procedure and the setting of $\tau$ in experiments are helpful for understanding the algorithm. In our experiments, we tune the values of $\tau$ for each task using a simple grid search strategy within the range of {1, 3, 6, 12, 24, 48, 96, 192} to reach a balance of accuracy and throughput. The tuning experiments are conducted on smaller versions of models when the model to be trained is large, owing to the associated high time and resource costs. We start the tuning of $\tau$ from 1 to larger candidate values given the consideration of high-speed communication on the corresponding platform. It is worth noting that, in addition to CO2, other methods such as Local-SGD/Adamw and SlowMo also require tuning of $\tau$. In practice, we exclusively tune $\tau$ for CO2 and employ the same tuned $\tau$ values for Local-SGD/Adamw and SlowMo. This is because of the similar role played by $\tau$ in these local-updating methods. Employing identical $\tau$ values for different methods on the same task ensures a fair comparison of their convergence and training throughput.

We list the values of $\tau$ in all the convergence experiments in the paper as below:

Q4: How about the end-to-end training performance (i.e., time to accuracy)?

A4: Thanks for the valuable question. Actually, we had conducted experiments on GPT-2 (Small, Medium and Large) to show the time to accuracy performances of CO2. The resulted curves are presented in Table 7 in the Appendix of the paper. Note that the experiments are performed on the A100 platform with high-speed inter-node connections. Even with the high-speed connections, CO2 still exhibits the fastest convergence speed on relative time compared to other existing methods.

Q5: How to choose $\tau$ in a new distributed GPU cluster?

A5: Thank you for your question. We use grid search to select $\tau$. Specifically, we employ a simple grid search strategy within the range of {1, 3, 6, 12, 24, 48, 96, 192} to determine the value of $\tau$, aiming to achieve a balance between accuracy and throughput. To reduce computational demands during hyperparameter tuning for large models, we first conduct experiments on smaller model versions and then apply the hyperparameter to larger models. Similar to CO2, other methods such as Local-SGD/Adamw and SlowMo also need the tuning of $\tau$. In our practical implementation, we first fine-tune $\tau$ for CO2 and subsequently apply the same tuned $\tau$ values for Local-SGD/Adamw and SlowMo. Standardizing $\tau$ values across different methods for a given task ensures a fair and meaningful comparison of their convergence behavior and training throughput.

Dear reviewer,

Thank you for your valuable and constructive feedback, which contributes a lot to the enhancement of our paper. Below, we provide our responses addressing your concerns. And all the revisions will be updated in the forthcoming rebuttal revision version of the paper.

Q1: How about comparing with SSP-like algorithms in terms of theoretical convergence bound and empirical scaling efficiency?

A1: Thanks for bringing this up. The stale-synchronous parallel (SSP) framework as described in [ref2] aims to mitigate the straggler problem with relaxed synchronization. Specifically, SSP allows faster workers to perform more updates than slower ones, reducing the waiting time of faster workers. However, to maintain model consistency and ensure convergence, SSP imposes a staleness bounded barrier that limits the iteration gap between the fastest and slowest workers. Comparing with CO2, their differences mainly lie in these three aspects:

1. The SSP framework as described in [ref1] and [ref2] is built upon the Parameter-Server architecture. While CO2 as well as the Local-SGD and SlowMo algorithms investigated in the paper all based on the All-Reduce architecture.
2. The SSP-like algorithms indeed introduces asynchronization into its training procedure, however, it does not allow the communication to be overlapped with computations. And its staleness is not punished, which will leads unsatisfactory convergence performances. As comparison, the asynchronous communication in CO2 makes the communication can be overlapped with multiple-step local updates, which hugely improves the training efficiency. And the staleness gap penalty technique allows to mitigate the negative effects on convergence results. These imply the scaling efficiency and convergence performance of SSP-like algorithms will be worse than CO2 in practice.
3. The convergence analysis section in [ref2] indicates that the convergence rate of SSP in normal cases is $O(\frac{1}{\sqrt{T}})$ . Comparing with CO2, when the total steps $T\tau$ is sufficiently large, i.e, $T\gg G^3\tau^3$, the convergence rate of CO2 is $\mathcal{O}\left(\frac{1}{\sqrt{G T \tau}}\right)$. So theoretically, the convergence rates of these two algorithms are similar.

Q2: 100% scaling efficiency is over-claimed

A2: We appreciate the thoughtful suggestion. Indeed, the original concept behind the design of CO2 aims to achieve a full overlap of the communication with multi-step local computation, the length of which is determined by the parameter $\tau$. However, it is acknowledged that an excessively large $\tau$ may have a detrimental impact on convergence performance, introducing a trade-off between the overlap ratio of communication and computation and the overall convergence. In previous experiments, we have observed that an appropriately chosen $\tau$ can achieve a high level of overlap (100%) while maintaining satisfactory accuracy results. We fully understand the concerns raised and will address them by refining the relevant statements in the paper during the rebuttal revision, aiming for a more balanced depiction of the scalability claims.

Q4: Detailed breakdowns of computation and communication times

A4: Thank you for the valuable comments. We conducted quantitative measurements on the time allocated to local computations and asynchronous communication. This assessment specifically targeted CO2 with different $\tau$ configurations on an A100 cluster equipped with 128 GPUs, employing a slower TCP/IP inter-node network. Adhering to the settings outlined in the paper, utilizing TransNormer-LLM (7B), we maintained consistency in our experimental conditions. The measured duration for a single-step local computation was approximately 0.109s, while an all-reduce communication incurred a cost of 1.566s. Subsequently, we computed and calculated the communication/computation overlap ratio for various $\tau$ values, presenting the results in the Table below.

Q5: Why CO2 obtains consistently lower perplexity than AdamW?

A5: The elements that have an impact on accuracy in CO2 are multiple sides: asynchronous operation and local updating may hurt the convergence as them both introduce noise into the training procedure, while staleness gap penalty and outer momentum clipping may improve the convergence. The resulted accuracy is a mixture of all these elements. Furthermore, in our experiments, we found that in some cases, moderate values of $\tau$ (i.e., the number of local steps) have a regularizing effect, which leads to an improvement on the accuracy results.

Dear reviewer,

Thank you for your valuable and constructive feedback, which contributes a lot to the enhancement of our paper. Below, we provide our responses addressing your concerns. And all the revisions will be updated in the forthcoming rebuttal revision version of the paper.

Q1: Describe the key distinctions from prior work in more detail

A1: The main contribution of the Cooperative SGD paper is to unify a variety of local-update SGD algorithms, such as local SGD, elastic averaging SGD, and decentralized parallel SGD, under the proposed unified Cooperative SGD framework. It also provides a unified convergence analysis for the methods under this framework. As one of the by-products, it proposes to add an auxiliary variable $z$ on the elastic averaging SGD, which serves as a local copy of the model. It can be seen as a preliminary approach that enables the asynchronous communication of model averaging to overlap with computation. However, our method leverages more sophisticated asynchronous communication design to improve the convergence. The differences between our method and the Cooperative SGD are threefold: 1) The CO2 algorithm is built upon SlowMo, which involves slow momentum and outer updates to improve the convergence; 2) CO2 leverages more sophisticated asynchronous communication design together with the outer momentum and outer updates to hide more communications; 3) CO2 also introduces staleness gap penalty and outer momentum clipping to improve the convergence as well as training stability.

Q2: The values of $\tau$ for each experiment and the description of the tuning procedure

A2: Thanks for the valuable comments. We agree that describing the tuning procedure and the setting of $\tau$ in experiments are helpful for understanding the algorithm. In our experiments, we tune the values of $\tau$ for each task using a simple grid search strategy within the range of {1, 3, 6, 12, 24, 48, 96, 192} to reach a balance of accuracy and throughput. The tuning experiments are conducted on smaller versions of models when the model to be trained is large, owing to the associated high time and resource costs. We start the tuning of $\tau$ from 1 to larger candidate values given the consideration of high-speed communication on the corresponding platform. It is worth noting that, in addition to CO2, other methods such as Local-SGD/Adamw and SlowMo also require tuning of $\tau$. In practice, we exclusively tune $\tau$ for CO2 and employ the same tuned $\tau$ values for Local-SGD/Adamw and SlowMo. This is because of the similar role played by $\tau$ in these local-updating methods. Employing identical $\tau$ values for different methods on the same task ensures a fair comparison of their convergence and training throughput.

We list the values of $\tau$ in all the convergence experiments in the paper as below:

Q3: Which stage of ZeRO have you used for the TransNormer experiment?

A3: Thanks for the question. We use ZeRO stage 2 in the experiments of TransNormer-LLM, will add this information into the paper later.

Q6: How do other models considered in the paper (e.g., ResNets or ViTs) scale?

A6: Thank you for your inquiry. We specifically focused on assessing the scalability of TransNormer-LLM (7B) because the other models examined in our study are relatively small, with the number of parameters for each being smaller than 1B. As you rightly pointed out, smaller models tend to derive less benefit from communication optimizations. The communication overheads associated with all-reducing these small models are minimal, rendering the communication tails often negligible. Additionally, for computer vision (CV) task models such as ResNet or ViT, we typically do not scale them to very large sizes, although there is ongoing research exploring the scalability of ViT to 22B.

To effectively demonstrate the scalability advantages of CO2, we conducted experiments specifically on GPT-3 (13B), a well-known language model with autoregressive transformer architecture. The results are shown in the below Table, includes throughput comparisons with SlowMo to address Q7. The results reveal that on GPT-3 (13B), CO2 consistently achieves higher throughput on platforms with different communication conditions.

Q7: How do other communication-efficient (e.g., SlowMo) methods scale on the fast and slow network?

A7: Thank you for your valid concerns. While Local-SGD/Adamw generally demonstrates commendable scalability performance, our investigation revealed that their convergence behaviors often fall short, as outlined in Table 1 and Table 2 in the paper. So we choose to only provide the detailed comparison between CO2 and SlowMo, considering their performance on both fast and slow networks. The experiments are conducted on GPT-3 (13B), are summarized in the Table corresponding to Q6. The results indicate that CO2 exhibits moderate advantages in terms of throughput compared to the communication-efficient method, SlowMo. Notably, these observed advantages will be more pronounced with larger clusters.

Q8: How much communication/computation overlap is actually achieved by CO2, particularly at scale?

A8: Thank you for the valuable feedback. Evaluating the communication/computation overlap is indeed crucial to substantiate the core claims of our paper. Notably, Table 3 in the manuscript provides a comprehensive overview of the scalability ratio of CO2 with varying $\tau$ across distinct communication platforms, offering a comparative analysis against Adamw. This scalability ratio can be considered as a metric for assessing the communication/computation overlap.

To enhance our analysis, we conducted quantitative measurements on the time allocated to local computations and asynchronous communication. This assessment specifically targeted CO2 with different $\tau$ configurations on an A100 cluster equipped with 128 GPUs, employing a slower TCP/IP inter-node network. Adhering to the settings outlined in the paper, utilizing TransNormer-LLM (7B), we maintained consistency in our experimental conditions. The measured duration for a single-step local computation was approximately 0.109s, while an all-reduce communication incurred a cost of 1.566s. Subsequently, we computed and calculated the communication/computation overlap ratio for various $\tau$ values, presenting the results in the Table below.

Q9: On the gradient bucketing

A9: I appreciate the reviewer's observation, and I agree with the noted concern. The depiction of gradient bucketing in the paper requires refinement, and we will address this in the revised version. It's important to clarify that the primary objective of gradient bucketing is to reduce the number of communication operations during gradient synchronization by employing larger tensor buffers in each operation. Communication/computation overlap is another system optimization build upon gradient bucketing.

Q4: Implementation details and discussions on memory overheads

A4: Thank you for bringing this to our attention. We acknowledge the lack of details in our paper regarding the implementation of asynchronous communication and the analysis of memory overheads. In our implementation, the asynchronous communication operations are intentionally conducted on a new CUDA stream dedicated solely to communication. At the end of the inner loop, a collective communication primitive of NCCL all-reduce, is launched among all ranks within the communication group. Subsequently, a waiting operation is executed to ensure the completion of the last launched all-reduce from the preceding outer loop, allowing the subsequent computation operations to proceed.

To facilitate asynchronous communication, a separate weight buffer is essential to store the stale version of the model. While this results in increased memory footprints compared to the synchronous version, it enables communication to overlap with multiple local steps. Specifically, in comparison to the SlowMo algorithm, CO2 incurs an additional memory footprint, twice that of the model itself, to support asynchronous communication and outer updates. To address redundant memory footprint concerns, techniques such as ZeRO-1, ZeRO-2, and ZeRO-3 have been integrated, alleviating issues related to optimizer states, gradients, and model parameters within the inner loop training of CO2. These measures aim to mitigate the negative impact of CO2's heightened memory overheads. Ongoing efforts are also dedicated to refining the CO2 algorithm in subsequent versions, with a specific focus on minimizing additional memory overheads.

Q5: Add the observed variances

A5: Thank you for bringing this to our attention. We have organized the experimental outcomes and presented the variance results on accuracy and throughput below. In terms of convergence results, Local-SGD/Adamw consistently exhibits the largest variances, due to lack of compensation measures for accuracy. SlowMo and CO2 demonstrate only slightly larger variances than SGD/Adamw, as they both integrate an outer loop to enhance convergence. Between CO2 and SlowMo, CO2 exhibits smaller variances due to the application of staleness gap penalty and momentum clipping.

Regarding throughput, it is evident that all methods experience larger variances on larger clusters. This is reasonable as communication on larger clusters tends to be more unstable and fluctuates frequently. On large clusters, such as those with 64 or 128 GPUs, CO2 with different settings of $\tau$ demonstrates slightly larger variances than the baseline Adamw, regardless of the speed of inter-node networks.

Dear reviewer,

We appreciate your valuable and constructive feedback, which significantly contributes to the improvement of our paper. In response to your concerns, we have provided detailed responses, and all the revisions will be incorporated into the upcoming rebuttal revision version of the paper.

Q1: Tone down and better contextualize the claims of "perfect 100% scalability"

A1: We appreciate the thoughtful suggestion. Indeed, the original concept behind the design of CO2 aims to achieve a full overlap of the communication with multi-step local computation, the length of which is determined by the parameter $\tau$. However, it is acknowledged that an excessively large $\tau$ may have a detrimental impact on convergence performance, introducing a trade-off between the overlap ratio of communication and computation and the overall convergence. In previous experiments, we have observed that an appropriately chosen $\tau$ can achieve a high level of overlap (100%) while maintaining satisfactory accuracy results. We fully understand the concerns raised and will address them by refining the relevant statements in the paper during the rebuttal revision, aiming for a more balanced depiction of the scalability claims.

Q2: How to select $\tau$?

A2: Thank you for your question. We use grid search to select $\tau$. Specifically, we employ a simple grid search strategy within the range of {1, 3, 6, 12, 24, 48, 96, 192} to determine the value of $\tau$, aiming to achieve a balance between accuracy and throughput. To reduce computational demands during hyperparameter tuning for large models, we first conduct experiments on smaller model versions and then apply the hyperparameter to larger models. Similar to CO2, other methods such as Local-SGD/Adamw and SlowMo also need the tuning of $\tau$. In our practical implementation, we first fine-tune $\tau$ for CO2 and subsequently apply the same tuned $\tau$ values for Local-SGD/Adamw and SlowMo. Standardizing $\tau$ values across different methods for a given task ensures a fair and meaningful comparison of their convergence behavior and training throughput.

Although grid search is used at present, there are more potential efficient strategies. One innovative suggestion is to perform a pre-run warmup with $\tau=1$ for a limited number of iterations. During this warmup phase, metrics like computation time and all-reduce communication time are collected to calculate an optimal number of steps for local computation ($\tau^*$). Then in formal runs, $\tau$ can be initialized as $\tau^*$, and adjusted based on accuracy results. Besides, varying $\tau$ values at different stages of training, similar to learning rates, is also an interesting idea. These ideas could be explored in future research related to CO2, we have not deeply investigated them currently.

Q3: How does the paper improve upon prior works which tune the communication frequency (see above for some references)?

A3: We thank for the insightful comments. The prior studies referenced above introduce adaptive communication frequency (i.e., the value of $\tau$) schedules on the native Local-SGD method. As for our work, the proposed CO2 leverages more sophisticated asynchronous communication design and outer updates on the top of naive Local-SGD, and further proposes staleness gap penalty and momentum clipping to improve convergence and training stability.

We acknowledge that these adaptive schedules for $\tau$ are likely to yield a more favorable trade-off compared with using a constant $\tau$. However, these adaptive schedules can be seamlessly integrated into our CO2 algorithm. Exploring and incorporating such adaptive strategies is a promising direction for enhancing CO2 in the future research.

Dear reviewer,

We thank for the reviewer's comprehensive evaluation on our paper. Below, we provide our responses addressing your concerns. And all the revisions will be updated in the forthcoming rebuttal revision version of the paper.

Q1: The idea of overlapping communication and computation is not new. The comparison with previous works are not enough, like Wang et al, 2020. Overlap-Local SGD

A1: We thank for the valuable comments. We will include the Overlap-Local SGD method in our comparisons. The method is also able to overlap communication by local computations via adding an anchor model on each node. The differences between our method and the Overlap-Local SGD are threefold:

1. Overlap-Local SGD achieves communication-computation overlap on the top of naive Local-SGD, whose convergence performance is normally not good enough. On the other hand, the CO2 is built on SlowMo, which involves outer loop includes outer momentum and outer updates to improve the convergence of native Local-SGD.
2. As mentioned by the reviewer, CO2 introduced staleness gap penalty and outer momentum clipping to prevent divergence and improve training stability, which has been verified effective via extensive experiments.
3. The Overlap-Local SGD is only tested on the CIFAR-10 image classification task using ResNet-18, while CO2 has been widely verified on multiple tasks on both CV and NLP fields, including image classification, semantic segmentation, 3D point cloud reconstruction, autoregressive language modeling and bidirectional language modeling.

To better comparing the performance of Overlap-Local SGD and CO2, we reproduced Overlap-Local SGD on its open-source implementation and ported it into our codebase. We run the experiments in paper using Overlap-Local SGD and present the results as below. We also list the throughput results and the settings of $\tau$ in all the convergence experiments we have conducted.

* For IC and SS tasks, we present the throughput results in "images/s", the image resolution is 224x224; For PC task, we present the throughput results in "point clouds/s", each point cloud has 1024 points.

* For ALM and BLM tasks, we present the throughput results in "tokens/s".

Q2: About the convergence analysis

A2: We thank for the comprehensive review. The convergence analysis was performed on CO2 with Local-SGD used in the inner loop. Staleness gap penalty and momentum clipping are not yet considered in the current convergence analysis. This is because we take these two techniques as plug-ins, a convergence analysis without them is more clean and easy to follow. The proof derivation in SlowMo is taken as a reference when we do convergence analysis for CO2, we will follow the suggestions to add the citation of it in the beginning of the proof.