SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models

We are deeply grateful for your support of our work, and we provide detailed responses to your comments as follows:

Q1. The paper would benefit from quantifying the potential speed-up in set-ups that do not suffer communication bottlenecks.

Thanks for the comment. We plan to add more details to the final draft.

SPD targets the communication bottlenecks incurred by all-reduce operation after self attention. In Figure 7, we utilize 100% percentage of SPD block to total number of blocks as maximum speed-up case. In this case, all communications after self attention are eliminated while all communications at the end of the block (after feed forward network) are remaining. However, using the case of removing all the communications in model which doesn’t suffer from communication bottleneck is impossible since the proposed block structure and sensitivity measuring of SPD cannot be exploited in this case. In conclusion, instead of measuring latency of all communications removed cases which is not realizable, we used latency from removing all communications after self attention which is the best case in realizable solution.

Q2. The way sensitivity is measured looks arbitrary. Why don’t I measure sensitivity by applying SPD to individual blocks or applying SPD to the blocks at the beginning of the model?

Thank you for the question. When measuring the sensitivity of a block to SPD, we apply SPD to consecutive blocks starting from the selected block and extending toward the last block. This approach allows us to evaluate the worst-case impact of SPD on the block’s output while ensuring that its input activations remain numerically identical to the original, unaffected by prior SPD modifications. By analyzing the perplexity difference between a given block and the next block, we can isolate the effect of SPD at that specific layer, excluding any influence from subsequent layers. This method provides a fast yet precise and independent measurement of each block's sensitivity. The sensitivity analysis reveals that the impact of SPD varies significantly across different layers. For instance, in Qwen2.5-7B (Reviewer USNW - Q2), the sensitivity ranking of blocks from least to most sensitive is observed as (26-24-5-25-22-17-11-20-21-18-12-10-7-14-16-15-4-23-19-8-2-13-9-3-6-1-27-0). This highlights that some layers are highly resilient to SPD, whereas others are more sensitive, underscoring the importance of a structured approach in determining which blocks to apply SPD while minimizing its impact on model accuracy.

Q3. The authors suggest B2B distillation. HG includes modifynig the architecture. The motivation for me to use them is unclear.

Thank you for the comment. While B2B and HG may not provide a perfect accuracy without loss, they offer valuable options for users seeking more aggressive optimizations. Additionally, HG does not modify the model architecture but rather permutes weights before distillation, consequently having same shapes of parameters. This simple adjustment helps improve accuracy after distillation in a distributed setting, making it a useful technique for enhancing performance without structural changes to the model.

Q4. Figure 1.b is misleading. You are suggesting architecture in Figure 2.

Thank you for the comment. Figure 1.b is a conceptual illustration showing the removal of sync operations, while Figure 2 presents the proposed block structure designed to achieve this without accuracy loss. To clarify the distinction, we will revise the title of Figure 1.b to "With elimination of sync-point in attention output", emphasizing that it does not represent SPD itself but rather illustrates the conceptual effect of removing sync points.

Q5. How does SPD interplay with other latency reduction techniques (quantization, pruning, block skipping)? Do I actually need SPD if I already have all of them?

SPD is an orthogonal approach focusing on communication efficiency to other LLM optimizations targeting computation efficiency, e.g. efficient attention, quantization, pruning. This means that while computation has improved, communication is not optimized, making synchronization a larger bottleneck than before. This leads to a greater proportion of time spent in end-to-end model execution. In this circumstance, SPD can addresses this issue by reducing the number of sync points rather than just making them faster.

Q5.a. How much does sync-point dropping affect the communication-computation gap? One would expect that the first all-reduce in the block is a major bottleneck.

Since SPD is optimizing communication channel, after SPD, the percentage of communication time channel is reduced. However, the difference between first all-reduce (after self attention) and second all-reduce (after feed-forward layer) are negligible since they use same tensor size for communication and they cannot be overlapped in both cases.

Thanks for your support on our work:

Q1. Insufficient details about how to set hyper-parameters

Thank you for the comment. While we will add more details on hyper-parameters to the final draft, we like to point out one observation we made during our study about τ1 and τ2. We observed that there are distinct points where the sensitivity metric (perplexity difference with consecutive SPD blocks) sharply increases as SPD is applied to more sensitive blocks. These points served as natural candidates for setting τ1 (over 0.05 perplexity difference) and τ2 (over 10 perplexity difference), allowing us to categorize blocks into insensitive, sensitive, and extremely sensitive groups. This approach strikes a balance between maintaining model accuracy and achieving communication reduction, as described in Section 4.2. The results in Section 5 show that the thresholds are robust across different configurations, minimizing accuracy degradation while maximizing the benefits of SPD. We will add this details in the final version of paper.

Q2. It remains unclear how well the method generalizes to modern LLM architectures featuring GQA and MoE.

We are happy to share more experimental results around the modern LLM architectures. In general, the blocks and algorithms of SPD can be applied to any model or system, as long as tensor parallelism (TP) can be applied.

1. GQA:

The each head of key and value can be firstly divided across GPUs and the matching queries can be followed as in TP. We provide more results of Qwen2.5-7B on zero-shot tasks (same as the presented results of Figure 7) in the Table below. The results show that SPD can be successfully applied to LLM architectures with GQA.

Table: Qwen2.5-7B 4-GPUs distributed inference average accuracy on zero-shot tasks (ARC, HellaSwag, LAMBADA, PIQA, SciQ, WinoGrande). Notations are same as in Figure 7 (‘ZS’: applying zero-shot dropping. ‘ZS+B2B’: applying ZS on ISBs and block-to-block distillation to the other remaining SBs. ‘ZS+B2B+HG’: applying ZS on ISBs, B2B on SBs and B2B with head grouping initialization to the other remaining ESBs.)

2. MoE:

Due to the space limit, we would appreciate if the reviewer can check out our response to Reviewer e4AF - Q3.

Q3. Is it possible to integrate this approach with sparsification techniques such as sparse attention?

Thank you for the question, and the our answer is yes. SPD is an orthogonal approach focusing on communication efficiency to other LLM optimizations, e.g. sparse attention, quantization, pruning. Sparse attention can be easily integrated into distributed attention weights distributed in each devices. The distributed attention weights are derived from query and key which are came from partial linear Q, K projection layers divided along the head dimension. Therefore, diverse sparse attention techniques working within each head dimension of partial attention weights are easily applicable with SPD settings.

Q4. How does the method affect accuracy on tasks that require handling long context lengths? + Q5. These zero-shot tasks evaluated typically generate a single token. What is the impact on generation accuracy when the model is required to produce a longer sequence?

Thank you for raising important question and we conducted extra experiments to find answer to the question. The Table below shows Longbench-Multinews (summarization) benchmark [1] on Llama-2-7b-chat. Similar with Figure 7 and 8 of the paper, in generation task, SPD effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference.

Table: Longbench-Multinews Rouge-L score on Llama-2-7b-chat 8-GPUs distributed

[1] https://github.com/THUDM/LongBench/tree/main/LongBench#summarization

[2] Liu et al, Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML 2023

[3] Agarwal et al, CHAI: Clustered Head Attention for Efficient LLM Inference, ICML 2024

[4] Agrawal et al, exmy: A data type and technique for arbitrary bit precision quantization, 2024

[5] Dong et al, Towards Low-bit Communication for Tensor Parallel LLM Inference, NeurIPS 2024

[6] Nvidia, https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

Q1. This paper claims to improve the scalability of TP, but no direct scaling evaluation is shown.

Thank you for the comment. While we haven’t directly measured the scalability of TP with SPD, we can intuitively understand that SPD would lead to better scaling efficiency as GPU counts increase since SPD surgically targets the communication overhead (which is a key barrier to high quality scalability). The question is how much better and it also depends on many complex factors including the compute/communication ratio as discussed with Reviewer e4AF - Q4. We couldn’t directly measure the scalability due to the time and resource limit during this rebuttal period, but plan to add detailed scalability measurements in the final draft.

Q2. The method might have different performance on different models.

Thank you for the important comments. For more data point, we experimented with Qwen2.5-7B Reviewer USNW - Q2 and Longbench-Multinews (summarization) on Reviewer USNW - Q4. In short, our new results indicate SPD shows the comparable effectives as it showed on the paper.

Q3. Lack of impact: A safe threshold for a negligible accuracy loss is 50% SPD results in 25% reduction of the overall TP volume.

Thank you for the chance to discuss the impacts. Although a 25% reduction may seem modest at first, achieving this reduction with no additional cost is significant. Most model-level optimizations, such as quantization or sparsity, require preprocessing steps or specialized hardware to realize performance gains. In contrast, SPD selectively removes sync point for blocks by simply not performing all-reduce, effectively removing 25% with negligible accuracy loss. Furthermore, in larger model (Llama2-70B), 70% of SPD results in 35% reduction. These results demonstrate that SPD offers a very low cost approach to improve efficiency in large-scale distributed inference. Also, note that SPD provides a scalable communication that adapts to different budget constraints while minimizing accuracy loss.

Q4. What is the definition of latency in the evaluations?

Thank you for the chance for clarification. The latency reported is end-to-end model latency when generating one token on prompting (time to first token) with batch size 1 and prompt length 128 on A100 by following the metrics utilized in existing optimizations [2,3] . On Figure 2, we will add the definition of latency. On Figure 7, to provide the intuitive comparison of reduced latency, we displayed the speedup with improved latency of SPD from latency without SPD. For the decoding latency, since the operations are bottlenecked to memory, the communication optimization including SPD and existing works [4,5] does not give great improvement compared to prefill stage. We will add more details to the final version.

Q5-1. 10GB bandwidth is too low.

Since the GPU only supports interconnect with NVLink (300GB/s) and without NVLink (10GB/s), there are no more selections that we can test with.

Q5-2. For high-bandwidth (HBW) scenarios, the latency improvement is around 5% without accuracy loss. The performance gain in practical case seem to be moderate.

General-purpose infrastructure, such as GPU servers, is designed for high performance across all aspects, featuring hardware support that minimizes communication overhead. As a result, with HBW, the latency improvement from SPD may appear moderate. However, in specialized infrastructures, particularly inference-focused hardware, cost constraints often limit HBW interconnect support compared to GPUs. In such environments, where communication bandwidth is a more significant bottleneck, SPD can play a crucial role in reducing sync overhead and improving overall efficiency. This makes SPD particularly valuable for inference infrastructure with limited interconnect performance.

Q5-3. If the paper wants to claim TP becomes practical in this scenarios, it should compare with other parallelisms.

Thank you for the comments. We design SPD for TP since TP is the most widely used techniques in distributed inference. Therefore, we focus our evaluation on TP to specifically address its communication bottleneck. Also, TP would work with other parallelisms in a hybrid mode (please see Reviewer e4AF - Q3). We plan to explore the suggestion as a future direction.

Q6. Why the SPD blocks have to be consecutive blocks from the last block?

Please refer to Reviewer J4A1 - Q2. In short, measuring SPD sensitivity with consecutive blocks enables isolating a block’s impact by analyzing perplexity difference. We will add more details to the final version.

Reference: Please refer to the references of Reviewer USNW.

Thanks for your thoughtful advices to our work. Here are the replies to your concerns.

Q1. The evaluation is limited to LLaMA2 and OPT models. Testing on more recent models could validate its generalizability.

Due to space limit, we couldn’t share our new results here. Please refer to the displayed result of Qwen2.5-7B on Reviewer USNW - Q2.

In short, we’ve observed comparable effectiveness of SPD on Qwen2.5-7B, further validating its generalizability to recent models. Thank you for your understanding.

Q2. The evaluation benchmarks are logprob-based tasks. It would be appreciated if more generation based or reasoning tasks will be included.

Again, due to space limit, we would appreciate if you can check out the displayed result of Longbench-Multinews (summarization) benchmark on Llama-2-7b-chat on Reviewer USNW - Q4.

In short, we’ve obtained positive results on generation tasks with SPD, showing performance comparable to the logprob-based benchmarks.

Q3. Would SPD work on MoE architectures or hybrid parallelism setups?

Thank you for the chance to elaborate how SPD works with MoE and hybrid parallelisms. The blocks and algorithms of SPD can be applied to any model or system that tensor parallelism (TP) can be applied.

• Applying on MoE architectures
  • Token Routing (Top-k Experts Selection)
    • In SPD, the routing network (softmax over token scores) can be replicated across GPUs following TP. The overhead with the replicated routers is negligible since they have much fewer parameters and require much less computation than the expert network. The router in each GPU assigns each token to the top-k selected experts having same decision per devices. GPUs begin expert computations immediately with local routing decisions.
  • Expert Feedforward Layers (Sparse Computation)
    • Each expert is a fully connected feedforward network (FFN). The selected experts process only their assigned tokens, with evenly distributed to all GPUs. After computation, the expert outputs are processed through next FFN layer, then requiring final all-reduce communication between GPUs.
  • Residual connection and bias in SPD block
    • As in dense models, skip connections and bias connections can be applied using the same structure presented in Figure 3.
• Applying on hybrid parallelism setup
  • Data parallelism
    • Copying a SPD model replica in GPU setting can make data parallelism work with SPD. For example, when we have 8-GPUs, we can set 4-GPUs distributed SPD model as one replica and set the copied replica to the other 4-GPUs so that 2 replicas are running as data parallelism setup.
  • Pipeline parallelism
    • Pipelining each parallelized execution of SPD (a fraction of attention and FFN of one block in a single GPU branch) can make pipeline parallelism work with SPD. For example, we have 4-GPUs distributed SPD block. In this case, we can divide a fraction of attention operation and FFN operation of one GPU to a pair of GPUs. As a result, 4 x 2 - GPUs server can work for pipeline parallelism setup and SPD eliminates all-reduce operation of former GPU in each 4-GPU-pairs.
  • Combined data and pipeline parallelism
    • One pipeline parallelized model can be set as a replica for data parallelism and the copied replica can make data parallelism setup with pipeline parallelized setup. This can finally make hybrid parallelism working with SPD ({TP+SPD} + data parallelism + pipeline parallelism)

Q4. For recent hardware e.g. H200/H100, the communication speed is actually very fast. The gain of removing an AR may be incremental.

Thank you for your comments. We first like to share how H100 is compared with A100 on computation and communication.[6]

• H100 offers 3.17× higher computation performance (on FP16 TFLOPS)
• H100 offers 1.5× faster NVLink interconnect for device-to-device communication

Hence, we can easily say that overall inference latency will go down thanks to stronger HW capability. However, we’d like to highlight that since compute has grown even faster, communication becomes an even larger bottleneck than before. In the Table below, we projected how compute vs. communication ratio changes for Llama2-7B 8-GPUs distributed inference when we move from A100 to H100, and it clearly shows that now the communication takes about 1.8x larger portion of the overall latency (i.e. from 15% to 27%), thus making SPD would be even more important. We hope this address your concern around how SPD will help with the recent hardware advancements.

Table: H100 projected computation/communication ratio compared to A100 on HBW (300GB/s) interconnect (Llama2-7B 8-GPUs)

Reference: Please refer to the references of Reviewer USNW.