We thank Reviewer GGaM for the thoughtful feedback and constructive comments. We have responded and addressed every weakness and question in detail. Meanwhile, for weaknesses and questions that have been addressed with additional explanations or experiments in the rebuttal, we will incorporate the corresponding discussions or results into the revised version of the paper. Please see the one-by-one responses below.

[Weakness 1]

[Answer 1] We conducted experiments on Cluster 1 with 16 GPUs to measure GPU utilization under different microbatch sizes. Specifically, we adjusted the microbatch size by changing the pipelining degree $R$ (a larger $R$ results in a smaller microbatch). VanillaEP represents the original mini-batch without pipelining. We used the NVIDIA SMI tool to sample the utilization of all GPUs every 5ms and averaged the values over 1000 iterations. Additionally, we reported the standard deviation of all GPU utilization values to illustrate the degree of data variability. The experimental results are as follows:

Based on the above results, we have the following conclusions:

(1) According to the comparison of average GPU utilization, smaller microbatch sizes may result in lower GPU utilization (e.g., when training GPT2-Tiny-MoE with $R=4$ using FlowMoE), but in most cases, GPU utilization is nearly identical to that without pipelining (e.g., when training BERT-Large-MoE, LLaMA2-MoE and DeepSeek-V2-S using FlowMoE). This is because the actual tensors involved in GPU computation remain sufficiently large for larger MoE models, and GPU SM resources are still utilized efficiently. In other words, the microbatches introduced by FlowMoE's pipelining do not waste GPU SM resources for large MoE models;

(2) According to the standard deviation of all GPU utilization values, compared to vanillaEP, FlowMoE maintains relatively stable GPU utilization during training (e.g., the standard deviation is even less than 6% when training BERT-Large-MoE and LLaMA2-MoE). This is because FlowMoE overlaps computing and communication tasks, and the GPU rarely enters idle states waiting for communication. In contrast, vanillaEP's communication tasks must be executed separately, resulting in low GPU utilization during communication task execution.

We will add the above discussion and results on the GPU utilization in the Appendix if the paper is accepted.

[Question 1]

[Answer 2] ●Firstly, smaller batch sizes are more likely to result in lower GPU utilization. We measured GPU utilization with different batch sizes when training different MoE models using FlowMoE on Cluster 1 with 16 GPUs. Similarly, we used the NVIDIA SMI tool to sample the utilization of all GPUs every 5ms and averaged the values over 1000 iterations. The measurement results are as follows:

The results demonstrate that smaller batch sizes are more likely to lead to lower GPU utilization, e.g., when training GPT2-Tiny-MoE or BERT-Large-MoE with a batch size of 2. Moreover, GPU utilization remained nearly unchanged when training LLaMA2-MoE or DeepSeek-V2-S. This is because these two MoE models have a larger number of parameters, which keeps the tensor dimensions involved in GPU computation sufficiently large, ensuring that SM resources are still efficiently utilized even with a lower batch size.

●Secondly, when using a large number of experts with a small number of activated experts, the utilization of GPUs will vary significantly. GPUs hosting frequently activated experts have higher utilization, while those with rarely activated experts experience underutilization.

We also evaluated the impact of the number of activated experts on GPU utilization when using FlowMoE on Cluster 1 with 16 GPUs for a large number of experts. Specifically, we constructed BERT-Large-MoE-w by increasing the number of experts per GPU from 2 to 8 and adjusting the number of activated experts by modifying the capacity factor $f$, where a larger $f$ indicates more uneven token routing by the gate function and fewer activated experts since most tokens are routed to popular experts. The detailed configurations of BERT-Large-MoE-w are as follows:

We used the NVIDIA SMI tool to sample the utilization of each GPU every 5ms and averaged the utilization of each GPU over 1000 iterations. We reported the maximum and minimum GPU utilization in the table below:

The above results show that, with a large number of experts, the fewer the number of activated experts, the more unbalanced the computation load on the GPU and the greater the difference in utilization between GPUs.

We will add the above discussion and results on the GPU utilization in the Appendix if the paper is accepted.

[Question 2]

[Answer 3] The Bayesian optimization (BO) process used to adjust the AR chunk size ($S_p$) is not a critical path in the training loop and can be made asynchronous with respect to the training loop. However, BO is crucial to FlowMoE. Meanwhile, FlowMoE's performance is not sensitive to BO hyperparameters, and users do not need to spend time carefully designing these parameters.

Specifically, BO is only executed during the initial profiling phase (typically the first ~80 iterations), during which it continuously suggests and samples a small number of ($S_p$, per-iteration time) pairs (8 by default). Then, BO predicts an approximate optimal $S_p$ based on these sampling pairs and uses it for subsequent training. All training iterations during the initial profiling phase use the existing FlowMoE pipeline and run normally. Therefore, we agree with the reviewer that BO can be redesigned to piggyback on standard iterations (e.g., using a background thread or dedicating evaluation to a shadow process, with BO and the main training loop process exchanging only sampled values and updated $S_p$) without blocking training, thereby further reducing even this minimal overhead. We view this as a valuable direction for future research.

I'd like to thank the authors for the detailed reply. I also want to point out that nvidia-smi tool only reports the percentage of time that any SM is active, therefore, it can inflate the result since other SMs can be non-active. To get the accurate GPU utilization that considers all SMs, I'd like to refer the authors to nvidia's DCGM tool and consider revising the evaluation method. Thanks.

Thank you once again for taking out time to review our paper. We will utilize the limited time remaining in the discussion to make our best effort to remeasure GPU SM utilization using NVIDIA's DCGM tool and attach the results.

Dear Reviewer GGaM,

Thank you very much for your suggestion. Since the GPU SM utilization measurement function in the NVIDIA DCGM tool is not enabled for consumer-grade GPUs (e.g., RTX series GPUs), we used the CUPTI tool to measure SM utilization. In particular, according to [1-2], DCGM also needs to call CUPTI when measuring SM utilization, therefore CUPTI can obtain more accurate SM utilization. The following three tables show (1) the change in average GPU SM utilization with different microbatch sizes (Table 1), (2) the change in average GPU SM utilization with different batch sizes when using FlowMoE (Table 2), and (3) the maximum and minimum GPU SM utilization for a large number of experts with different numbers of activated experts when using FlowMoE (Table 3) in Cluster 1 with 16 GPUs. Our observations show that although the numerical utilization values measured using different tools may vary, the conclusions obtained remain consistent. We will add this measurement result and the related discussion if accepted.

Table 1

Table 2

Table 3

References:
[1] NVIDIA Corporation, NVIDIA Developer DCGM Documentation User Guide.
[2] NVIDIA Corporation, Open Source: DCGM GitHub Repo.

We thank Reviewer YDn2 for the thoughtful feedback and constructive comments. We have responded and addressed every weakness and question in detail. Meanwhile, for weaknesses and questions that have been addressed with additional explanations or experiments in the rebuttal, we will incorporate the corresponding discussions or results into the revised version of the paper. Please see the one-by-one responses below.

[Weakness 1]

[Answer 1] We appreciate the opportunity to clarify the innovation and impact of our work.

●Firstly, the MoE architecture has become one of the dominant architectures for modern LLMs, including LLaMA4 (Meta, 2025), DeepSeek-V3 (2024), Grok-1 (xAI, 2024), and DBRX (Databricks, 2024). As MoE models become more prevalent, optimizing their distributed training is both critical and urgent. We believe FlowMoE will make a meaningful contribution to the development of this field.

●Secondly, existing pipelining techniques focus on overlapping of All-Reduce (AR) and computing, or All-to-All (A2A) and expert computing, but leave two key issues unresolved: (1) optimal coexistence of AR and A2A, and (2) a unified pipeline scheduling scheme for all major MoE tasks. FlowMoE explicitly addresses these two issues. Therefore, FlowMoE is not an optimization of existing pipeline technologies but proposes a new pipeline acceleration scheme, representing a fundamental innovation.

●Thirdly, we introduce a formal task-dependency pipelining model and prove that our scheduling strategy strictly reduces or matches per-iteration time compared to sequential or partially pipelined scheduling (see Theorems 1 and 2). This mathematical foundation distinguishes our method from heuristic-driven pipeline strategies, going beyond standard engineering optimization.

●Fourly, FlowMoE not only focuses on system-level implementation but also emphasizes algorithm design for the pipelining scheme and AR chunk based priority scheduling mechanism (e.g., Algs. 1 and 2 in Sec. 4.2). These contributions clearly extend the impact beyond the specific domain of MoE training optimization.

●Finally, our submission is highly consistent with the NeurIPS infrastructure track (e.g., libraries, improved implementations and scalability, distributed solutions). FlowMoE provides an efficient distributed solution that improves the implementation of MoE training systems. Additionally, the AI community has been extensively focusing on research to accelerate distributed training through pipelining techniques [1-7], and we believe that FlowMoE's contributions represent a fundamental advancement in this important and evolving field.

[Weakness 2, Question 3, Question 4]

[Answer 2] To deepen the theoretical analysis, we analyze (1) the convergence of our scheduling method and (2) FlowMoE's sensitivity to Bayesian optimization (BO) configurations. Since the above two theoretical analyses also address Questions 3 and 4, we provide a unified response here.

(1) Convergence of the proposed scheduling method.
FlowMoE only changes the execution order of computing and communication tasks without affecting model convergence. We denote the samples processed in each pipeline as one microbatch, and the number of microbatches is equal to the pipelining degree $R$.

Specifically, during the backpropagation of the $l$th transformer block in one iteration, the gradients from all $AT_r^l$ and $E_r^l$ ($1 \le r \le R$) are accumulated and summed. Once the gradient from $AT_1^l$ is also accumulated, the All-Reduce chunk communication task for the MHA and gate is started. Similarly, when the gradient of $E_1^l$ is accumulated, the expert parameters are updated. This effectively prevents the impact of gradient staleness on convergence from early parameter updates. Additionally, to ensure that the accumulated gradients are equivalent with and without pipelining, the loss of each microbatch is scaled by $\frac{1}{R}$, i.e., $\frac{loss^{(r)}}{R}$ ($1 \le r \le R$), and the detailed derivation follows:

●The loss and gradient when performing backpropagation using the entire mini-batch are expressed as follows:

which is currently used by all mainstream MoE training frameworks, e.g., Tutel, DeepSpeed-MoE and Megatron-LM.
●When dividing mini-batch into $R$ microbatches, the number of samples in each microbatch is $b=\frac{B}{R}$. The loss for each microbatch is:

Then, the scaled loss for each microbatch is:

The cumulative loss of all microbatches is:

According to Eqs. 1 and 4, we can obtain:

This analysis shows that the gradient update strategy is numerically equivalent with and without pipelining, and FlowMoE does not introduce gradient staleness. The only difference between them is the optimization of task scheduling order during execution, which does not compromise model convergence during training.

(2) Sensitivity analysis of FlowMoE to BO configurations.
FlowMoE is insensitive to BO hyperparameters, which can be manually adjusted.

Firstly, the BO search space should be fixed as (0MB, max tensor size to be communicated per transformer block] to ensure that the optimal AR chunk size is included. Thus, the BO configuration only involves two hyperparameters: the acquisition function and surrogate model. Then, an oversized AR chunk size will affect the prioritization of A2A tasks and may prevent them from being started in time, while an undersized AR chunk size will cause excessive startup overhead. Therefore, there must exist a unique optimal solution that maximizes training speed. For this single-peaked, smooth objective function with a fixed search space, BO consistently converges near the optimum, largely unaffected by hyperparameter variations. The table below shows that different BO settings have minor effect on FlowMoE performance when training BERT-Large-MoE on Cluster 1.

*EI: Expected Improvement
*Surrogate Model use Gaussian Process Regression (GPR) with different kernel functions.

We will add the above theoretical analysis in the Appendix if accepted.

[Weakness 3, Question 1]

[Answer 3] ●Firstly, we added a new recent baseline, FSMoE[8]. FSMoE focuses on pipelining for expert computing and communication tasks within the MoE layer, and it maximizes computation-communication overlap within the MoE layer by reasonably scheduling inter-node and intra-node communication tasks. The table below shows the comparison of average per-iteration time of FlowMoE and FSMoE on Cluster 1.

* $S_1$ is the speedups of FlowMoE over FSMoE.

The results show that FlowMoE still achieves better performance gain compared to FSMoE. This is because FlowMoE implements pipeline scheduling for all major MoE-related tasks and introduces BO to adaptively adjust the AR chunk size to maximize computation-communication overlap across the entire transformer block. It is worth noting that the scheduling strategies of FSMoE and FlowMoE are orthogonal and can be combined to further accelerate MoE training.

●Secondly, Table 9 in Appendix D.2 also shows the superiority of BO over other auto-tuning optimization strategies in FlowMoE.

[Question 2]

[Answer 4] Yes, we have compared BO, Grid Search, and Random Number Generation in Appendix D.2. The results from Table 9 show that using BO to adjust the AR chunk size achieves the shortest per-iteration time among the three methods.

References:
[1] Pipeline Parallelism with Controllable Memory. NeurIPS, 2024.
[2] Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models. NeurIPS, 2024.
[3] SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training. NeurIPS, 2022.
[4] Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learning. NeurIPS, 2019.
[5] Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training. NeurIPS, 2018.
[6] PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization. ICML, 2025.
[7] Pfeife: Automatic Pipeline Parallelism for PyTorch. ICML, 2025.
[8] FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models. ASPLOS, 2025.

We thank Reviewer gKp7 for the thoughtful feedback and constructive comments. We have responded and addressed every weakness and question in detail except for providing evaluation results for larger-scale GPU clusters. We sincerely apologize for this, because it is very difficult and unrealistic for us to obtain larger-scale hardware resources and measure results within one week. But we will continue to seek larger-scale hardware resource in the future to demonstrate the scalability of FlowMoE. In addition, for weaknesses and questions that have been addressed with additional explanations or experiments in the rebuttal, we will incorporate the corresponding discussions or results into the revised version of the paper. Please see the one-by-one responses below.

[Weakness 1]

[Answer 1] ●Firstly, to stress-test FlowMoE's performance, we measured the training performance of different training frameworks on two scaled-up MoE models (LLaMA2-MoE-L and DeepSeek-V2-M), both of which are close to the memory upper limit of Cluster 1. Specifically, we constructed LLaMA2-MoE-L by increasing the number of transformer blocks in LLaMA2-MoE from 32 to 64. We constructed DeepSeek-V2-M by reducing k to 1 while keeping 7 transformer blocks. The detailed configurations of the two scaled-up MoE models are as follows:

Then, the average per-iteration time using different frameworks on these two MoE models is shown as follows (FasterMoE is OOM in any cases):

* OOM: out-of-memory
* $S_1$, $S_2$ and $S_3$ are the speedups of FlowMoE over ScheMoE, Tutel and vanillaEP, respectively.

The results above show that FlowMoE still achieved the best training performance among all baselines on larger models.

●Secondly, verifying the scalability of FlowMoE on a larger-scale GPU cluster would indeed be an added strength. However, we sincerely apologize that obtaining a larger-scale hardware resource and evaluating the results within a week is very difficult and unrealistic for us. Nevertheless, our current clusters and models are already sufficient to demonstrate the effectiveness of FlowMoE's pipeline mechanism. We will continue to seek larger-scale hardware resource in the future to demonstrate the scalability of FlowMoE.

●Thirdly, we agree with the reviewer that any claim of “trillion-parameter scalability” is indeed inappropriate in our paper since we lack experiments on trillion-parameter models. The sentence: "The parameter size of modern large language models (LLMs) can be scaled up to the trillion-level via the sparsely-activated Mixture-of-Experts (MoE) technique to avoid excessive increase of the computational costs", we had in the abstract may have caused misunderstanding. We apologize for that. Thereby, we will modify this sentence to: "The parameter size of modern large language models (LLMs) can be scaled up to a massive scale using the sparsely-activated Mixture-of-Experts (MoE) technique to avoid excessive increases in computational costs." to rectify this issue.

[Question 1]

[Answer 2] Yes, FlowMoE avoids the stale gradient updating problem, and FlowMoE's scheduling mechanism only changes the execution order of computing and communication tasks without affecting the convergence of the model.

We denote the samples processed in each pipeline as one microbatch, and the number of microbatches is equal to the pipelining degree $R$.

Specifically, during the backpropagation of the $l$th transformer block in one iteration, the gradients from all $AT_r^l$ and $E_r^l$ ($1 \le r \le R$) after backpropagation are accumulated and summed. Once the gradient from $AT_1^l$ is also accumulated, the All-Reduce chunk communication task for the MHA and gate function in the $l$th transformer block is started. Similarly, when the gradient of $E_1^l$ is accumulated, the expert parameters are updated. This effectively prevents the parameters of MHA, the gate function, and the expert from being updated early, thereby avoiding gradient staleness. Additionally, to ensure that the accumulated gradients are equivalent with and without pipelining, we scale the loss calculated for each microbatch by $R$, i.e., $\frac{loss^{(r)}}{R}$ ($1 \le r \le R$), where $loss^{(r)}$ is the loss calculated using the samples from the $r$th microbatch, and the detailed theoretical derivation is as follows:

●The loss and gradient when performing backpropagation using the entire mini-batch are expressed as follows:

which is currently used by all mainstream MoE training frameworks [1,2,3].
●When dividing mini-batch into $R$ microbatches, the number of samples in each microbatch is $b=\frac{B}{R}$. The loss for each microbatch is:

Then, the scaled loss for each microbatch is:

The cumulative loss of all microbatches is:

According to Eqs. 1 and 4, we can obtain:

This analysis shows that the gradient update strategy is numerically equivalent with and without pipelining, thereby FlowMoE does not introduce gradient staleness. The only difference between them is the optimization of task scheduling order during execution, which does not compromise model convergence during training. We will add this discussion in the Appendix if accepted.

[Question 2]

[Answer 3] Please let us clarify that our measurement process actually does cover the entire cluster-wide energy usage, including both the communication cost and computing cost.

We use the NVIDIA SMI tool to sample the real-time power of each GPU in the cluster every 5ms, then integrate these sampled data over time to calculate the energy consumption of each GPU during the training process. We sum the energy consumption of all GPUs to obtain the total energy consumption, then divide it by the number of GPUs and the number of iterations to obtain the average per-worker (per-GPU) energy consumption as shown in Table 6. It is worth noting that the power reported by the NVIDIA SMI tool includes both communication and computing costs. According to the official documentation of NVIDIA SMI [4], the energy consumption measured by the NVIDIA SMI tool covers the entire GPU card's power consumption, where communication power includes (1) PCIe communication costs, (2) NVLink/NVSwitch communication costs, and (3) collective communication costs (e.g., NCCL All-Reduce and All-to-All). In addition, some representative works [5,6] also used the NVIDIA SMI tool to monitor GPU communication cost. Therefore, our measurement results effectively reflect the total cluster-wide energy usage, compatibly providing a fair comparison with the literature.

References:
[1] Tutel: Adaptive Mixture-of-Experts at Scale. MLSys, 2023.
[2] DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. ICML, 2022.
[3] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053, 2019.
[4] NVIDIA Corporation, NVIDIA System Management Interface (nvidia-smi) User Guide.
[5] Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. IEEE TPDS, 2019.
[6] AUGEM: Automatically Generate High Performance Dense Linear Algebra Kernels on x86 CPUs. SC, 2013.

We thank Reviewer hxBa for the thoughtful feedback and constructive comments. We have responded and addressed every weakness and question in detail. Meanwhile, for weaknesses and questions that have been addressed with additional explanations or experiments in the rebuttal, we will incorporate the corresponding discussions or results into the revised version of the paper. Please see the one-by-one responses below.

[Weakness 1]

[Answer 1] Prior works [1–3] focus on non-MoE models and primarily consider pipelining between computing and All-Reduce (AR) communication. In contrast, FlowMoE is specifically designed for MoE models, where All-to-All (A2A) communication is introduced due to the sparsely activated expert layers. Compared to works [1-3], FlowMoE has two fundamental novelties: (1) the co-existence and scheduling of both AR and A2A communication tasks, and (2) the pipelining of computing tasks (MHA, gating and expert) with two heterogeneous communication tasks (AR and A2A). In particular, to the best of our knowledge, FlowMoE is the first work to pipeline these five tasks together.

In addition, as shown in Fig. 2b, in the backpropagation of the MoE model, the expert computing of the $l$th transformer block depends on the combining A2A of the $l$th transformer block, the dispatch A2A of the $l$th transformer block depends on the expert computing of the $l$th transformer block, and the MHA computing of the $(l+1)$th transformer block depends on the dispatch A2A of the $l$th transformer block. However, the AR communication of the $l$th transformer block only depends on the $l$th transformer block MHA computing and gating. Therefore, the A2A communication is not equivalent to the AR communication. Works [1-3] do not consider these complex dependencies and cannot be directly used to solve the pipeline problem of distributed MoE training. The scheduling mechanism of FlowMoE is essentially an efficient solution for scheduling multi-type tasks in distributed MoE training.

[Weakness 2]

[Answer 2] BO auto-tuning is an indispensable part of FlowMoE and is absolutely necessary. While FlowMoE's efficient performance depends on the introduced BO process (Table 10 in Appendix D.2 demonstrates this conclusion), but it is less sensitive to the hyperparameters used in BO, which are of lower importance and can be manually adjusted.

Firstly, regarding your point on providing insights, we clarify that: An oversized All-Reduce (AR) chunk size will affect the prioritization of All-to-All (A2A) tasks and may prevent them from being started in time. In contrast, an undersized AR chunk size will result in excessive startup overhead. Therefore, neither an oversized nor undersized AR chunk size can achieve the shortest per-iteration time, and there must exist a unique optimal solution that maximizes training speed. The BO optimizer balances these two competing effects by sampling and learning from actual iteration times, efficiently finding an AR chunk size​ that maximizes overlap without incurring excessive overhead. This tuning is crucial because the optimal trade-off point varies across models, GPU interconnect topologies, and MoE configurations. We elaborate on these in Appendix D and Theorem 2, where we show theoretically (and empirically in Table 9/10 and Fig. 4) that BO-tuned configurations consistently reduce iteration time compared to fixed or grid-based alternatives. We will clarify this explanation further in the main text if accepted.

Then, in FlowMoE, the objective function for per-iteration time regarding the AR chunk size is a single-peaked, smooth objective function with a fixed search space. For this objective function, the process of using BO to find the optimal solution is insensitive to the two hyperparameters (acquisition function and surrogate model) of BO, and BO always converges and approaches the optimal solution. The following table shows the comparison results of average per-iteration time when training BERT-Large-MoE with different BO parameter configurations on Cluster 1. The results indicate that BO hyperparameters have a minor impact on FlowMoE performance, and different BO configurations lead to similar iteration time.

*EI: Expected Improvement
*Surrogate Model use Gaussian Process Regression (GPR) with different kernel functions.

[Question 1]

[Answer 3] No, FlowMoE will not obtain slower per-iteration time in any case. We analyzed three cases:

(1) The communication task time is much longer than the computing task time. When the communication time is very long, the time of the All-to-All (A2A) communication task alone completely covers the computing task time. In this case, the All-Reduce (AR) chunk based priority scheduling mechanism will fail because the AR chunk cannot be inserted into the A2A communication task. The performance of FlowMoE will be the same as ScheMoE, Tutel, and FasterMoE, but better than vanillaEP due to the hidden computing task time.

(2) The computing task time is much longer than the communication task time. When the computing time is very long, all the communication task time can be covered by the computing task time. In this case, FlowMoE will outperform ScheMoE, Tutel, and FasterMoE due to the hidden AR task time, and better than vanillaEP because it further hides the A2A task time.

(3) The communication task time is comparable to the computing task time. In this case, FlowMoE outperforms all baselines because it maximizes the overlap of multi-type tasks.

In summary, in all cases, the performance of FlowMoE is greater than or equal to that of ScheMoE, Tutel, and FasterMoE, and is always better than that of vanillaEP.

[Question 2]

[Answer 4] ●Firstly, we conducted a detailed analysis on how FlowMoE adapts to heterogeneous GPU clusters. (1) For clusters with mixed GPU types, since All-Reduce (AR) and All-to-All (A2A) tasks can only begin once the slowest GPU completes its corresponding computing task (other GPUs will be idle in waiting time after completing their computing tasks), the task timeline for each iteration of FlowMoE and other baselines is determined by the slowest GPU. In this case, FlowMoE can maximize the computing/communication overlap of the slowest GPU, and its performance still outperforms the baselines. Table 12 in Appendix G.1 also demonstrates the same conclusion when comparing different frameworks on GPU clusters with heterogeneous computing power. (2) For clusters with bandwidth asymmetry, since AR and A2A tasks are collective communications, all GPUs start and finish communication tasks simultaneously regardless of whether their bandwidths are the same. Therefore, the task scheduling timeline is the same for all GPUs. In this case, FlowMoE still achieves better performance than the baseline due to its efficient pipeline solution.

●Secondly, it is not necessary to conduct the BO retuning every time there is a hardware change. We have designed a re-Bayesian tuning mechanism for FlowMoE in Appendix G.2. Specifically, we define a re-execution threshold $\delta$. We denote the iteration time corresponding to the near-optimal AR chunk size $S_p$ predicted by the last BO as $\hat{\mathcal{F}}(S_p^{best})$. If the change of the current iteration time $T$ compared to $\hat{\mathcal{F}}(S_p^{best})$ is more than $\delta$, i.e.,

the BO will be re-executed to find a new $S_p$. This is based on the understanding that there is only one optimal AR chunk size to minimize the iteration time for the same hardware cluster (see [Answer 2] for detailed analysis). Therefore, when $T$ and $\hat{\mathcal{F}}(S_p^{best})$ are close, it indicates that the hardware conditions have not changed or have only slightly fluctuated, and there is no need to readjust the AR chunk size. However, when $T$ and $\hat{\mathcal{F}}(S_p^{best})$ differ significantly, it indicates that the hardware conditions have changed significantly, and the optimal AR chunk size has also changed. At this point, the BO needs to be re-executed to update the AR chunk size.

●Thirdly, FlowMoE's performance is sensitive to the BO process. Table 10 in Appendix D.2 compares the per-iteration time of FlowMoE using BO auto-tuning and different fixed AR chunk sizes. The results show that randomly selected fixed AR chunk sizes are likely to severely limit FlowMoE's performance. Therefore, BO auto-tuning is an indispensable part of FlowMoE and can maximize its performance. Although FlowMoE's performance is sensitive to the BO process, it is insensitive to the hyperparameters used in BO and they can be manually adjusted (see [Answer 2] for detailed analysis).

Thank you for your rebuttal. Some of my concerns (i.e., BO tuning) are addressed, but I still think the novelty of this paper is limited given that there're lots of related works in this filed.

First, it's true that some [1–3] of prior works focus on non-MoE models. But it's unclear to me why the pipelining of computing tasks (MHA, gating and expert) with communication tasks (AR and A2A) is fundamentally different and complex, given this sequential execution dependency. Second, there are also some prior works mainly focusing on computation-communication overlapping for MoE models, for example:

• Zhang, Shulai, et al. Comet: Fine-grained computation-communication overlapping for mixture-of-experts. https://arxiv.org/pdf/2502.19811v1
• Jiang, Chenyu, et al. Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping. MLSys 2024
• Punniyamurthy, Kishore, Khaled Hamidouche, and Bradford M. Beckmann. Optimizing distributed ml communication with fused computation-collective operations. SC 2024.

Therefore, I will maintain my score.