We thank the reviewer for their comments. We’re glad that the motivation and intuition behind ACCO came through clearly, and we appreciate your recognition of the "solid" empirical results and wall-clock improvements demonstrated in our LLM pretraining experiments. We address your points and suggestions in detail below.

W1: “Only comparing to DPU”

We would like to clarify that in Section 4.3, we also compare ACCO to the Weight Prediction method introduced in [8, from ACCO], so the claim of comparison only to DPU is not accurate.

Regarding Streaming DILOCO: it serves a different purpose and is designed for settings with sufficient communication bandwidth to allow full rematerialization of optimizer states — as discussed in lines 35–39 of the introduction. In contrast, ACCO specifically targets bandwidth-limited environments where such rematerialization is not feasible.

That said, if rematerialization were allowed, ACCO could potentially be combined with DILOCO (which is a local SGD method). Exploring such a hybrid is an interesting direction, but it falls outside the scope of this work.

W2: “The description of the algorithm is very confusing”

We respectfully disagree, particularly regarding the DPU variant. As written in the algorithm section, gradients are computed using $\theta^{t-1}$, while parameter updates are applied to $\theta^t$. This is precisely what those equations convey.

The buffers alternate between steps, and the implementation is consistent with the written description. For further clarity, we have included the code with the submission and have also provided a code snippet in our response to Reviewer S8XN.

W3: “What happens if local batch sizes are smaller?”

As shown in Equation (ACCO), line 164, local gradients are aggregated via AllReduce across all workers, so updates occur based on the global batch, not per-worker mini-batches. Thus, even if a local batch is small (e.g., size 1), the overall update still reflects the aggregated contribution from all workers.

We address scenarios with imbalanced local batch sizes in Section 4.6. Our current model assumes that nodes process batches at a similar speed, leading to balanced partitions and stable training. While one worker computing less (due to heterogeneity or imbalance) is possible, we find ACCO remains stable under such conditions in practice.

Regarding the comment that the method may be “highly sensitive to variation in mini-batch size,” we would appreciate clarification. If this refers to intra-node batch imbalance, AllReduce ensures global aggregation still proceeds correctly. If it refers to variability over time, our assumption of stationary GPU behavior ensures consistent gradient dynamics under fixed batch size.

“Minor issue: The all-reduce expression is unnecessarily complex”

We intentionally provide the full all-reduce expression because we are not simply averaging gradients $g_i$, but rather computing $g_i + \tilde{g}_i$. This distinction is essential to the algorithm, and precision in notation is important — a point we believe aligns with the reviewer’s earlier concern in W2.

Simplifying the expression would risk omitting key aspects of the computation.

Q: “ADAM analysis?”

Deriving theoretical convergence results for Adam is more complex than for SGD, which is why we chose to focus on the latter to build intuition and provide a clear framework. While the analysis for Adam is beyond the scope of this work, we emphasize that ACCO+Adam empirically matches standard Adam training behavior - as demonstrated in Figures 5–9 - which strongly supports its effectiveness in practice. We appreciate the reviewer’s interest and view theoretical extensions to adaptive optimizers as valuable future work.

Concerning the running buffers, it’ll probably add some variance for very small batch sizes (not common for LLMs training), but should lead to similar convergence for reasonable batch sizes.

Thanks for the reply.

Re: DiLoco

DiLoco is designed to run at extremely low communication bandwidth, since you only need to sync every few hundred steps. This is the only provenly production ready distributed training algorithm in deployment, https://www.primeintellect.ai/blog/opendiloco. I believe avoiding comparison with it makes the argument weaker.

In terms of combining the two methods, what additional advantages do you see it brings? Seems both of the methods are self-sufficient and could be in contest with each other?

Re: Local Batch Size

I am referring to the estimation of $\tilde{\theta}_{t+1}$, not the final updates. Apologize if there was confusion.

Re: Adam

It's fair to skip the theoretical analysis due to complexity. However, can you provide some direct comparison beyond the training loss? For example, the difference between updates, if you were to run either pure Adam or ACCO+Adam?

We thank the reviewer for their thoughtful and encouraging feedback. We're especially grateful for your recognition of ACCO’s strengths as a practical, major advancement, and high-quality contribution, and address your suggestions below.

Minor formatting issues

Thank you for pointing these out. We will correct all typos and formatting issues as requested. We will also add back the guidelines of the checklist.

Minor formatting issues in the Appendix

We will make all requested changes, including adding the missing reference text for Figure 13 and Algorithm 1, and relocating the figure to appear between Sections B.1 and B.2.

Q1: “How are data loaded and handled in heterogeneous hardware?”

We follow standard practice by partitioning the dataset evenly across workers. In heterogeneous environments, this data allocation could be adapted to reflect differences in compute capacity: thus, the dataset is loaded as needed. For heterogeneous hardware, it would make sense to indeed allocate the data accordingly to the worker speed. While we do not explore this in the current work, we agree it is an important consideration and appreciate the suggestion.

Q2: “Experiments beyond LLMs?”

We have tested ACCO on CIFAR-10 using the DLA model, SGD, and the cosine scheduler — the model reaches 95.3% both with ACCO+SGD and vanilla SGD, without any hyperparameter tuning. We plan to release the CIFAR-10 code to support further experimentation. Thanks for the suggestion.

We sincerely thank the reviewer for their thoughtful and detailed feedback. We appreciate the time and effort taken to engage with our submission which “targets an important problem”. Below, we address each point raised, structured according to the review:

W1. “Lack of practical motivation for reducing communication overhead”

While we agree that some clusters achieve high throughput (e.g., ~97% as reported in Gemini [arXiv:2312.11805]), this is not the case in all data centers, and indeed many datacenters in high-end academic, government, and other settings do not have the most expensive connectivity. Indeed, this work was performed on a relatively high-end and large-scale academic cluster.

We also note that with methods like ACCO becoming prevalent, even the largest-scale industry players can potentially save on some of the expensive infrastructure. ACCO is designed to be simple, optimizer-agnostic, and portable - requiring no infrastructure-specific tuning or assumptions. As the reviewer notes, our method would be especially beneficial for lower-bandwidth settings, and we appreciate the suggestion to better highlight this use case. We will revise the introduction and Section 2 to explicitly frame ACCO as a low-infrastructure-cost alternative suited for less-optimized clusters than Google.

W2: “Scalability to larger models and 3D parallelism”

We thank the reviewer for raising this important point. ACCO modifies only the optimizer step and operates independently of how gradients are computed. Therefore, it is compatible with tensor and pipeline parallelism, and does not interfere with 3D parallel setups. ACCO can be integrated post-gradient computation, making it orthogonal to parallelization strategies. In fact, it may be seen as two models alternating computation on shared threads, and thus naturally extends to larger scales. That said, optimized 3D parallel implementations would benefit from further engineering integration, which we consider an exciting future direction.

W3: “Unclear presentation of the algorithm”

We acknowledge this and will revise the paper for clarity. Specifically:

• We will consistently use "computation stream" and "communication stream" throughout the paper.
• We will replace $\nabla_i$ with $g_i$ and $\tilde\nabla_i$ with $\tilde g_i$ to unify notation.
• You're correct in your assumption. The beginning of Section 3.1 clearly defines each worker’s local batch as $N_i$, $\xi_k$ as a sample and $\nabla F_i$ as the local gradient estimate on node i. $\tilde g_i$ are computed over separate halves (each of size $N_i$), yielding an effective batch of size $2N_i$. We will clarify this language further.
• Figure 2 is illustrative of how gradient computation overlaps with communication; it is not a full computational graph. ACCO uses standard PyTorch gradient flows - no custom computational graph is introduced.

W4: “Figure 3 misleading due to optimizer precision”

Figure 3 is adapted from Figure 1 of (ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, 2020, Rajbhandari et al.) . We will revise the caption to clearly note that the optimizer state is stored in higher precision, and ensure visual clarity by adapting the figure in the final version and referring to the Table 1 of our paper which clarifies this.

W5: “DDP baselines may be poorly tuned; Figure 4 unclear”

We appreciate this feedback. We tuned DDP hyperparameters first (see Table 4 of the Appendix for a list of the hyperparameters that we tuned only on DDP, and line 46 from Section 1), and then applied ACCO using the same settings.

Figure 4 illustrates the case where communication is not fully optimized - this was intentional, to highlight potential overlap gains in typical academic setups. While we agree that well-optimized implementations can mask communication overhead, our contribution is algorithmic, not engineering-based. ACCO remains applicable in such contexts and is complementary to techniques like gradient bucketing. Our aim is not to outperform state-of-the-art communication kernels, but to provide a simple algorithmic handle for improving overlap where tuning is limited or ineffective.

W6: “Slow network (100 Gbit/s) is not representative”

We agree this is lower than some of the most high-end infrastructure. However, it reflects what is available in many settings - from startups, government institutions, academic settings, and even a broad range of cloud providers. Our results aim to show that even in these constrained environments, ACCO can provide significant speedup. We will better contextualize this in the revision.

W7: “No runtime results for H100s”

Due to excellent intra-node bandwidth on H100s (which is available on our cluster), we do not expect ACCO to show a runtime improvement. Our experiments on H100 are intended to demonstrate convergence and functional correctness, not performance gains. We will clarify this in the experimental section.

W8. “Statistical significance of results”

For experiments in Section 4.3, where we expected variance, we averaged over 3 runs. We did not observe significant differences in training behavior between ACCO and DDP. While a broader statistical analysis is valuable, the cost of large-scale LLM training is prohibitive. That ACCO matches DDP so closely in loss curves across all settings provides strong evidence of its stability.

W9. “Lack of comparison to standard gradient bucketing overlap”

Gradient bucketing and ACCO are complementary: bucketed all-reduce can still benefit from being scheduled one step earlier via ACCO. We agree this relationship could be more clearly explained and will add the following clarifying sentencesin Section 2 to explicitly distinguish ACCO's goal from intra-step overlapping techniques: "ACCO is complementary to standard gradient bucketing: while bucketing overlaps communication within a backward pass, ACCO overlaps across steps by shifting communication earlier. Both techniques can be combined for additional benefit."

W10: “Claim of 25% speedup unclear in Figure 7"

Thank you for this observation. Indeed, the training accuracy reached by DDP at iteration 5000 is matched by ACCO around iteration 2800 (better than announced). We will clarify this in Section 4.4.

W11: “Terminology: ‘mirror’ → ‘match’”

We will update "mirror" to "match" for clarity.

Q1: "Is large-scale LLM training communication-bound?"

In the highest-end internal industry clusters, communication may not be the bottleneck until cross-data center training is needed. But in a wide range of existing clusters and settings that we are aware of, LLM training is indeed communication-bound: these include startups working on cloud providers, government research clusters, some of the highest-end national academic clusters, or even the growing number of actors trying to train LLMs over machines connected through the internet (INTELLECT-1 Technical Report, 2024, Sami Jaghouar et al.).

These environments — including ours — commonly suffer from costly inter-node all-reduce, especially as model size grows. Figure 4 illustrates this overhead, and ACCO offers a simple way to mitigate it without infrastructure changes. We also emphasize that expensive communication infrastructure is built in the highest-end datacenters exactly to overcome these issues. Work towards communication-efficient learning can thus benefit the next generation of highest-end HPC clusters.

Q2: "How would ACCO handle 3D parallelism?"

ACCO applies after gradient computation and does not interfere with model partitioning. It is compatible with tensor, pipeline, and expert parallelism. A careful implementation would be required for full integration at scale, which we view as promising future work.

Q3: "Why does Figure 4 show poor all-reduce performance? Would ACCO still help after tuning?"

The figure reflects an untuned or modestly tuned academic setup. With tuning, some overlap is achieved, but not fully — particularly for early layers. ACCO still helps by shifting communication one step earlier, thus improving overlap even in optimized scenarios.

Q4: "What speedup does ACCO yield on higher-bandwidth networks?"

If communication cost is negligible and the optimizer step can be fully overlapped, then ACCO provides no timing benefit. However, ACCO still maintains correctness and convergence, and can be safely deployed without penalty.

Q5: "What is the runtime performance on the 8×H100 setup?"

We observed similar runtime between ACCO and DDP due to the efficient intra-connect. Our goal with this experiment was to verify convergence, not performance.

Q6: "Is the loss difference statistically significant?"

No. For training loss, curves are nearly identical. Minor variation in test accuracy falls within expected noise. We have not observed divergence or instability in any experiment.

Minor Corrections and Clarifications

• The mention of a limitations discussion at the end of Section 4.6 was mistakenly omitted. We will add a proper limitations section in the final version.
• We will revise Figure 3, fix terminology inconsistencies, and improve the clarity of all figures and equations as requested and described above.

We thank the reviewer for recognizing that our work tackles an important problem and for highlighting the strengths of our submission, notably the "comprehensive experiments" and "convergence guarantees." We now address the points raised for improvement in detail:

Q1/W1: “Why not ZeRO-2 or ZeRO-3 with ACCO?”

We appreciate the reviewer’s insightful suggestion. Our choice to focus on ZeRO-1 was driven by clarity and simplicity in isolating the core effects of ACCO. Importantly, integrating ACCO with ZeRO-2 or ZeRO-3 does not affect the final training accuracy but alters the memory footprint, communication latency, and training time. ACCO is fully compatible with these variants: ZeRO-2/ZeRO-3 shard the optimizer and gradient states across devices, and in this setup, ACCO operates by using a sharded model and gradients on the computation stream. While memory benefits are preserved, the computation stream incurs additional communication overhead to synchronize updates, which introduces slowdown.

To reflect this, we will revise Table 1 to explicitly report memory usage under ZeRO-2 and ZeRO-3 by dividing the parameter and gradient buffer memory by the number of devices, consistent with the optimizer state sharding.

We have conducted preliminary experiments on CIFAR-10 using two GPUs and observed a similar runtime slowdown when applying ACCO or not with ZeRO-2/ZeRO-3 on the computation stream. These results confirm our expectation. Nonetheless, we agree that optimizing these variants in conjunction with ACCO is a promising research direction and plan to investigate efficient implementations as future work.

W2: “Only the proofs for Adam in ACCO?”

We thank the reviewer for raising this important point. It is a deliberate choice to restrict our analysis to SGD, grounded in clarity and scope. The convergence of adaptive optimizers like Adam is a subtle and complex topic; indeed, works such as “A Simple Convergence Proof of Adam” (2020, Deffossez et al.) focus entirely on that challenge and are quite involved despite the name.

Our goal in this paper is to introduce a practical and effective algorithm. Accordingly, our analysis of ACCO with SGD serves as a sanity check, providing intuition and formal reassurance that ACCO preserves training dynamics. Extending these guarantees to Adam is out of scope for this paper, but is an exciting direction for future work.

That said, our empirical results show that ACCO reproduces Adam’s behavior extremely well (see Figure 5 and 7). This is strong evidence that ACCO’s core mechanism remains stable and effective even with adaptive optimizers. Additionally, to reinforce the generality of our method, we include results using SGD on CIFAR-10 with the DLA model and cosine scheduler, where ACCO+SGD achieves 95.3% accuracy, matching vanilla SGD.

We will clarify this design choice more explicitly in the final version of our manuscript.

Q2: “Maximise GPU throughput while halving batch size?”

We fully agree with the reviewer’s assessment. In terms of computation: each sample is processed only once, so FLOPs per sample are effectively unchanged compared to DDP. While ACCO introduces a small overhead by sequentially computing gradients on two partial batches, this overhead is not a concern as long as GPU utilization remains saturated.

Q3: “How does ACCO handle noisy or non-i.i.d. gradients?”

This is an excellent theoretical point. The robustness of ACCO to noise and non-i.i.d. data is explicitly addressed in Proposition 3.2, where the variance term $ \sigma^2 $ captures stochastic noise. Non-i.i.d. effects introduce correlations in gradient noise, which appear in the covariance structure of the same bound. Consequently, ACCO’s sensitivity to noise is in line with standard SGD under similar assumptions.

Empirically, we observe that ACCO outperforms delayed SGD (DPU) and matches the performance of standard SGD even in regimes with increased gradient variance (see Section 4.6 and Figure 9), providing strong practical validation of our theoretical insights.

We sincerely thank the reviewers for their thoughtful and constructive feedback, as well as for recognizing the significance of ACCO as a clever and novel work in addressing well-known and critical communication bottlenecks in large-scale model training.

W1: Complex implementation

We believe that the implementation does not add extensive complexity and can also be modularized in future software libraries. We propose a simple code on CIFAR10 to convince the reviewer of the simplicity to use ACCO where net and net_ would be two CNNs that mirror $\theta$ and $\tilde\theta$ and optimizer_ and optimizer would apply ACCO updates:

for batch_idx, (inputs, targets) in enumerate(trainloader):
    inputs, targets = inputs.to(device, non_blocking=True), targets.to(device, non_blocking=True)

    if batch_idx % 2 == 1:
        with torch.cuda.stream(stream1):
            outputs = net(inputs)
            loss = criterion(outputs, targets)
            stream1.wait_event(event1)
            loss.backward()
        with torch.cuda.stream(stream2):
            copy_model_grads(net_, net) # g_i is initialized with tilde g_i
            event1.record()
            optimizer_.step()
    else:
        with torch.cuda.stream(stream2):
            zero_grad(net_) # set tilde g_i to 0
            outputs = net_(inputs)
            loss = criterion(outputs, targets)
            loss.backward()
        with torch.cuda.stream(stream1):
            optimizer.step()

    stream1.synchronize()
    stream2.synchronize()

We’ve run an experiment on CIFAR-10 using the DLA model, SGD and the cosine scheduler — the model reaches 95.3% both with ACCO+SGD and vanilla SGD. However, we agree that a highly optimized implementation would be necessary to perform a PR in PyTorch, but this is beyond the case of this paper.

W2: "Proofs for adaptive optimizers?"

We agree that extending the theoretical analysis to include adaptive optimizers like AdamW would further strengthen the paper’s contributions, and we view this as an important and valuable direction for future work, particularly in the context of local SGD and large-scale distributed training. However, such proofs are much more involved and we view our SGD proof rather like a sanity check. See our answer to Reviewer 2aEN.

W3: “Batch size might influence the learning rate or BatchNorm”

Regarding the learning rate, it's important to highlight that one of the primary goals of ACCO is to avoid unnecessary grid search or hyperparameter tuning. Therefore, we intentionally did not fine-tune this hyperparameter during our experiments. Furthermore, we believe the effective batch size is the one happening between two successive updates of $\theta$. We would be happy to add a clarification in the appendix explaining that the equivalent batch size in ACCO corresponds to the number of samples processed between two successive updates of $\theta$, and that training dynamics are adjusted accordingly.

As for Batch Normalization, if it posed an issue, a potential solution would be to share the buffers and adapt the update rules to reflect the two-stage nature of ACCO. Therefore, we conducted additional experiments on CIFAR-10 using architectures such as DLA, which rely on BatchNorm, and observed no degradation in performance. For models like LLMs, this should not be an issue. Specifically, both ACCO+SGD and standard SGD achieved an accuracy of 95.3%. To support reproducibility, we had included a SLURM script in Appendix B.3 for reviewers who may wish to run ACCO or a baseline method of their choice.

Q1: "Beyond one-step delays?"

ACCO is currently designed to handle a one-step delay, which aligns with typical scenarios involving overlap between communication and computation. However, extending ACCO to handle larger or more variable delays - as might occur in less stable network environments or more complex parallelization schemes - is an interesting direction for future work. Since ACCO is derived from an ADMM-style reformulation involving auxiliary variables $\theta=\tilde\theta$, one natural extension would be to introduce multiple auxiliary variables to model longer delays. We plan to explore this line of research in future work.

Q2: "3D parallelism?"

ACCO integrates naturally with model parallelism approaches such as tensor and pipeline parallelism. Since ACCO modifies only the optimizer step - not the forward or backward passes - it remains agnostic to how gradients are computed. Therefore, tensor and pipeline parallelism can be implemented as usual, and ACCO simply applies its update rule afterward. In a fully 3D-parallel setup, ACCO’s two-stream design remains applicable and effective, as it operates independently of the specific gradient computation strategy. See the code above.

Q3: "Redundancy?"

There is no redundant computation between the first and second stages of ACCO. The gradients computed in the first stage (on half the mini-batch) are accumulated and reused. The second stage computes gradients on the remaining half using the updated parameters ($\tilde\theta$), and these are added to the previously accumulated gradients. Thus, the gradient computation is distributed across two steps, but no work is wasted. See the code above.

Q4: "Does ACCO change optimal hyper-parameters?"

In all our experiments, we first tuned the hyperparameters for the standard DDP baseline to ensure optimal performance, and then used the exact same settings for ACCO. We observed strong results without needing to retune. While it's possible that ACCO could benefit from additional hyperparameter tuning and potentially outperform the baseline further, this was not the focus of our work. A key advantage of ACCO is that it introduces no new hyperparameters, simplifying adoption in practice. See Table 4 in the Appendix for a list of the hyperparameters that we tuned only on DDP and line 46 from Section 1.

Q5: "Failure prone environment?"

This is a valid and important point. ACCO does not introduce any new fault tolerance mechanisms beyond what is already provided by the underlying DDP infrastructure. It does not exacerbate failure modes, but it also does not specifically improve resilience to worker failures. We will clarify this limitation in the final version of the paper to guide practitioners who may consider ACCO for large-scale, long-running training jobs. Thank you for pointing this out.