Thanks for your valuable and constructive feedback. We will explain your concerns point by point.

Weaknesses:

Q1: concerns about the title and claims

A1: Thanks again for your insightful advice. We adjust the title to "Rethinking Memory and Communication Costs for Efficient Large Language Model Training in Data Parallelism" to better fit the study of this paper. As stated in the global rebuttal (Author Rebuttal), PaRO can be used with other n-D parallel strategies to accelerate the training of large models.

We will update the description of performance improvements with more specific and accurate language, like "PaRO improves the training speed of LLMs by up to 266% that of ZeRO-3 as basic DP strategy".

Q2: For the experiments:

a. ZeRO-1 is missing

b. It would be better to compare PaRO-CC with the Hierarchical Communication Strategy in MiCS.

A2:

a. We conducted a set of experiments comparing PaRO(NII) with ZeRO-1, using the 7B model on 32 GPUs. With the same effective batch size, the throughput of NII was found to be 48.7% higher than ZeRO-1. For detailed data, please refer to Table 1 in the PDF.

b. The difference between PaRO-CC and MICS's Hierarchical Communication Strategy (MiCS-HCS) is that we overlap inter-group and intra-group communication, and we do not need to rearrange data blocks, avoiding unnecessary memory operations. Details can be compared between Figure 2 in the PDF and the MiCS paper. And the collective communication time with 128 GPUs of PaRO-CC, MiCS-HCS and Ring are 162, 183 and 288 ms separately.

Q3: Figure 1 is a bit complicated.

A3: We have redrawn the figure, please refer to Figure 1 in the PDF.

Questions:

Q1: What's the difference between partial-parameters training and PEFT?

A1: To tradeoff between training efficiency and statistical performance, model training may have diverse requirements of trainable parameters in different scenarios. The ratio (trainable-parameters/parameters) of partial-parameters and PEFT is 1/16 and 3/1000 separately. This difference will result in significant differences in memory usage and communication data volume, and the most suitable strategy is also different. Please refer to section 3.1.1 in our transcript and Table 9 in the Appendix for more information.

Q2: If the scope of this paper is for LLM as in the title, could you please add more related work to elaborate the current state of LLM training?

A2: As mentioned in Weaknesses A1 and global rebuttal (Author Rebuttal), we will adjust the title and give more description of PaRO usage in LLM training.

Q3: Can you elaborate the difference between PaRO-CC and the Hierarchical Communication Strategy in MiCS?

A3: Same as Weaknesses A2.b.

Limitations:

Q1: How to effectively integrate with current 3D parallelism (or other model parallel techniques) in LLM is a big concern, which may limit its application.

A1: As a future work, we currently working on further utilising PaRO in n-D parallel training for more large-scale LLM training. We find it is effective in two scenarios in the latest n-D parallel training:

1. Scenario 1 is when there is a difference in bandwidth between intra-group and inter-group communication, which is common in GPU clusters. n-D hybrid parallelism usually performs intra-node TP and inter-node PP in a subgroup of nodes, and nesting DP or PaRO in the outer groups. However, the grouping criteria is no longer one machine per group, but rather based on the network topology of the cluster. For example, in a multi-layer switch scenario, grouping can be classified based on the lower-layer switches. For another example, when training with multiple Nvidia SuperPods or similar infrastructures, one SuperPod can be used as a group.

2. Scenario 2. In a vast number of GPU scenario, n-D parallelism usually uses TP + PP + DP for efficient training of over ten thousand GPUs. Following the same principles as discussed earlier, PaRO could provide a more flexible strategy as an alternative to DP in such cases.

Please refer to global rebuttal (Author Rebuttal) for more detail.

Thank you again for taking the time to review our paper. We hope we have addressed the reviewer's questions and the reviewer is willing to raise their score. Please let us know if we can provide any additional clarification or information.

Reason 3 is reasonable to me, because Deepspeed-ulysses is another widely used technique, where PaRO can be applied (but the specific strategy is still not clear).

It does not propose a new parallelism strategy but leverages the existing parallelism techniques, where new parallelism techniques like PaRO or Sequence parallelism are not included

I don't think this claim is correct. Although Alpa doesn't propose any new parallelism strategy explicitly, most existing strategies including PaRO and Sequence parallelism lie in the solution space of Alpa. The main difference is, Alpa uses an op-level space while PaRO uses a model-level space (only sharding batch dimension), which means PaRO has a much smaller space so that all solutions can be enumerated while Alpa needs an ILP to search for the optimal solution. There may be some tiny differences in the details, but conceptually, PaRO is a subset of Alpa.

Given my concerns about integrating with other methods are partially addressed and the systematic implementation should be helpful to the community, I'm willing to increase my score from 4 to 5. But I still think the paper needs to be improved regarding the above discussions. More experiments about integrating with other methods to prove its efficiency would be appreciated.

Thanks for your postitive and constructive feedback. We will explain your concerns point by point.

Weaknesses:

Q1: Do you have much longer sequence length, for example, 4K or 8K?

A1: We are currently conducting experiments with longer sequence lengths, but we have not had time to obtain the experiment results. Theoretically, longer sequence lengths should not significantly increase communication volume (only parameters and gradients are communicated in DP strategy) and are anticipated to result in similar performance improvements.

Q2: The latest Megatron supports more efficient training schemes. It would be helpful to show if PaRO can still achieve significantly higher training performance even when considering these methods.

A2: PaRO could be an alternative to DP in the latest n-D parallel training and advance the training effiency. Please refer to global rebuttal (Author Rebuttal).

Questions:

Q1: How PaRO determines the group size (G)? Is it a hyper-parameter?

A1: The PaRO will automatically partition groups according to the nodes, making full use of the communication advantages within the nodes. It also supports other group sizes, such as using the lower-layer switch as the grouping basis in multi-layer switch networks, and of course, custom group sizes can also be defined.

Q2: In the experiments, do other well-known computation optimizations (e.g., FlashAttention) are applied?

A2: In order to avoid introducing more factors, we did not use optimization schemes such as flash-attn and quantization. But for the LLaMA-65B model, we activated checkpointing to ensure successful training.

Q3: How can users enable PaRO? Could you provide an example of the programming interface?

A3: We provide the open-source release of our code in paper. To enable PaRO, you just need to add "paro_strategy": "NIG" in the zero_optimization dict in the original ds_config.json, and start the training the same way as with deepspeed. Configuration examples can be found in the README.md file in the code repository.

Thank you again for taking the time to review our paper. We hope we have addressed the reviewer's questions and the reviewer is willing to raise their score. Please let us know if we can provide any additional clarification or information.

Thanks for your positive and constructive feedback. We will explain your concerns point by point.

Weaknesses:

Q1: Figure.1 & 2 a bit hard to understand.

A1: We have redrawn the figures, please refer to Figure 1 & 2 in the PDF.

Q2: Experiments in Part 4.2 do not seem exhaustive.

A2: We choose the best strategy for specific training scenarios based on the guidelines and verify it in our experiments.

Q3: Comparison with standard collective communication strategies other than "the ring topology" is not done.

A3: I appreciate you bringing this up.

The Butterfly topology sends and receives large data blocks each time, which makes it difficult to fully utilize the bandwidth and prone to delay jitter. Its performance is not as good as the Ring topology.

Compared to the Ring topology, the Tree topology in NCCL speeds up communication primity locally. We focus on the huge difference in communication bandwidth between inter-group and intra-group interactions. Group communication is utilized to minimize inter-group communication and maximize intra-group communication. The communication can utilize either a ring or tree structure.

The Moshpit topology is a group communication strategy for allreduce that solves cluster instability and node failure problems through an iterative averaging protocol. Our algorithm combines multi-level sharding of large models and fully utilizes bandwidth between and within machines through a grouping strategy to improve communication.

Questions:

Q1: line 228: “Assuming that communication and computation can be fully overlapped, 𝑇 can be approximately regarded as the maximum of 𝑡𝑐𝑜𝑚𝑚. and 𝑡𝑐𝑜𝑚𝑝.” Could you be more specific when this assumption is actually valid in practice?

A1: When training transformer architecture models, the next layer can prefetch parameters while the previous layer is computing, allowing computation and communication to happen simultaneously. Here we simplify and assume perfect overlap, meaning the training time for a single step is determined by the longer of the two, computation or communication.

Q2: line 237: “the formula of 𝑇” Do you mean max{𝑡𝑐𝑜𝑚𝑝.,𝑡𝑐𝑜𝑚𝑚.}?

A2: Yes. We will make it a numbered formula in our manuscript for better clarification.

Q3: line 287: (throughput indicator) It not clear at first that this is the “TPS indicator” (otherwise not defined) in the subsequent figures, I would advice indicating it at this stage.

A3: Yes, throughput indicator(log(1/T)) is the “TPS indicator” in figures. We will declare this in the subsequent version.

Q4: Fig 7: Small variations between the losses for the different methods are observed, although they should be mathematically identical, why is that?

A4: Thanks for pointing this out. We have verified the values of the model parameter, gradient, and optimizer state before and after an update step, and the errors of different strategies remain within the normal machine precision range (1e-6). Small variations between the losses for different methods, despite their mathematical equivalence, can be attributed to floating-point representation errors in computers. These truncation errors accumulate during different steps of the calculations. As a result, the outputs may show slight discrepancies. However, these factors do not impact the statistical performance during training or the consistency in convergence behavior. The phenomenon is also consistent across ZeRO-1/2/3.

Thank you again for taking the time to review our paper. We hope we have addressed the reviewer's questions and the reviewer is willing to raise their score. Please let us know if we can provide any additional clarification or information.