Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping

Thanks for your feedback on our paper! Below we provide a few clarifications on how our experiments demonstrate practical gain from ladder-residual and its compatibility with other parallelism.

In the paper, some experiments are conducted under TP16, where inter-node communication cost is stronger on 8-gpu per node clusters.

We agree with the reviewer that tensor parallelism is not commonly conducted across nodes due to the heavy communication. Our TP16 experiment is intended to demonstrate the potential of Ladder-Residual in scenarios where cross-node model parallelism becomes necessary. Generally, larger models are served with TP+PP (for example DeepSeek 671B model). However, it should be noted that with newer generation GPUs (B200 NVL72) and the recently announced (Rubin NVL576), it will become very feasible to run these models with purely TP since these systems expand the NVLink (high bandwidth) domain from 8 GPUs to 72 and 576 GPUs respectively.

We also want to note that Ladder-residual is compatible with multi-dimensional parallelism, please see our response to Reviewer 9rvM for further details. Therefore, even if we want to combine TP with other parallelism, ladder-residual can still be applied to accelerate the TP part. Furthermore, despite we didn't explore it in this work, we believe ladder-residual can accelerate other heavy-communication parallelism like EP as well.

It's possible that the advantage of this paper only happens in large TP settings, which may be not often used.

Table 1 and Figure 2 investigates the benefit of ladder-residual when intra-node TP (2, 4 and 8 GPUs) is used, which is the most common and flexible inference-time parallelism approach. In this regime, ladder-residual consistently show a large speedup against the standard Transformer and beats the parallel attn-mlp alternative until the batch size is very large. Figure 4 also shows that under the intra-node TP setting, Ladder-residual can reliably push the Pareto-frontier between latency and throughput.

Thanks for the valuable feedback and the questions. Below we discuss how our paper is novel, why it’s compatible with other parallelism, address the concern on scaling, and provide analysis on the change of activations.

We also thank the reviewer for the suggestions on the presentation. We will incorporate them in the next draft.

---

Novelty over parallelizing MHA and MLP blocks

First, Ladder-residual is a fundamentally different architecture from parallelizing MHA and MLP blocks, with distinct motivations. Ladder-residual is designed to decouple computation and communication to enable their overlap. In contrast, the goal of parallelizing MHA and MLP blocks is to combine computation for speedup. Due to the reduced depth, parallelizing MHA and MLP blocks also saves communication but is not its original focus. The speedup of Ladder-Residual comes from overlapping all (except last) communication where parallelizing MHA and MLP focuses on accelerated matmul and as a byproduct, it cuts 50% of the communication.

---

Combining with other communication optimization methods

Yes. A key advantage of Ladder-Residual is that it is hardware-agnostic and does not rely on custom kernels, making it fully compatible with other low-level optimization methods. However, since Ladder-residual hides latency of all communication except the last one, further applying these techniques can only accelerate the final communication which won’t achieve much gains.

---

Comparison with other frameworks

We focus on inference speedups in this paper and gpt-fast and vLLM are highly optimized for inference. We choose gpt-fast to demonstrate the inference speedups but it should be noted that we still achieve training speedups (around 5-8%) when combining TP+FSDP.

---

Compatibility with other parallelism

It is possible to use Pipeline Parallel (PP) with the Ladder architecture: just before the pipeline boundary, we wait for the async AllReduces to complete, and send 3 tensors to the next pipeline stage: residual, current_mlp_output tensor and current_attention_output. It should be noted that this is still pretty cheap since generally the P2P communication during inference is latency bound and can be implemented easily using the batch_isend_irecv API (https://pytorch.org/docs/stable/distributed.html#torch.distributed.batch_isend_irecv).

def forward(
   self,
   previous_attention_out: Tensor,
   previous_mlp_out: Tensor,
   residual: Tensor,
   attention_handle,
   mlp_handle,
) -> Tensor:
   attention_handle.wait()
   residual = residual + previous_attention_out

   current_attention_out = self.attention(self.attention_norm(residual))
   current_attention_out = all_reduce(current_attention_out, async_op=True)

   mlp_handle.wait()
   residual = residual + previous_mlp_out

   current_mlp_out = self.feed_forward(self.ffn_norm(residual))
   current_mlp_out, mlp_handle = all_reduce(current_mlp_out, async_op=True)

   if is_last_layer_on_pp_stage:
       attention_handle.wait()
       mlp_handle.wait()

   return current_attention_out, current_mlp_out, residual, attention_handle, mlp_handle

For Data Parallelism or FSDP, the cross-layer dependency doesn’t pose any complications. Therefore Ladder-residual can be seamlessly incorporated in a multi-dimensional parallelism training paradigm. Our training framework already supports TP+FSDP for ladder residual model training and can be easily extended to support PP as illustrated by the above code snippet.

---

Performance and scaling

In our response to Reviewer Mhub, we provide experiments where we increase the number of parameters of the Ladder-Transformer by 30%. We show that at both 1B and 3B scale, the 30% larger ladder-Transformer has both higher accuracy and higher TPS than the standard-Transformer. This confirms our architecture truly pushes the Pareto frontier.

It’s difficult to conclude the scaling trend from just two sizes and in the future we hope to run models of more sizes to study this more carefully.

---

Activation difference analysis

Our motivation is that, given in a trained standard Transformer, the activation differences between modules are small, replacing with the Ladder architecture won’t lead to large degradation. We don’t care too much if this property is still held during training, and it’s difficult to have a theoretical guarantee due to unpredictable training dynamics.

To further verify this intuition, we analyze block similarity (between consecutive attention and MLP modules) in both original Llama-3.1-8B-Instruct and Hybrid-Ladder-8B-16L-retrained. The visualization can be found at This Anonymous GitHub link. Overall there is a slight decrease in activation similarity after adapting to Ladder-residual but the similarity remains high overall (above 0.9 at most of the layers).

Thanks for your time and the feedback! We want to clarify that in a lossy efficient method (where a more efficient architecture is proposed to approximate the original one), trading accuracy for efficiency is common and we provide a good trade-off. It’s difficult to have one-size-fits-all as some users might prefer higher efficiency while others prefer higher accuracy.

However, to make things more clear, we provide additional results that achieve both higher accuracy and higher throughput (token-per-seconds) below. This was accomplished by training a slightly larger ladder model for the train-from-scratch experiment and employing an improved post-training pipeline for the post-training-adaptation experiment.

For Table 3, Ladder-Transformer has a gap of 1 to the Standard-Transformer.

In our train-from-scratch experiments, we previously reported that the Ladder-Transformer slightly underperformed compared to the Standard-Transformer of the same size. As shown in Table 1 from the paper, Ladder-residual can offer at least 30% speedup for any model size. To further explore the accuracy-throughput trade-off, we increased the parameter count of the Ladder-Transformer by 30%. This allows us to answer the question: For two models with similar throughput, how does accuracy compare?

Importantly, increasing the Ladder-Transformer’s size by 30% leads to less than a 30% decrease in throughput. As a result, the scaled-up Ladder models still achieve higher tokens-per-second (TPS) than their smaller Standard counterparts. We report both accuracy and TPS below following the format of Table 3 in the paper. (Note: the “1B” and “3B” models in the paper correspond to 1.2B and 3.5B parameters respectively; here, we write out the exact values for clarity.)

As shown above, the Ladder-Transformer achieves higher average accuracy, lower perplexity, and significantly higher TPS at both the 1.2B and 3.5B sizes.

For Table 4, although the average score of Hybrid-Ladder-8B-16L-retrained is comparable to Llama-3.1-8B-Instruct, in most of the evaluation categories (6 out of 9) the hybrid ladder is worse than the baseline.

We retrain the hybrid-ladder-8B-16L model with the same data (3B tokens, all open-source datasets), but this time with logit distillation (KL divergence loss with the logits from the original Llama-3.1-8B-Instruct). Following the format of Table 3 in our paper:

Now we are better on 5 out of 9 tasks while being 1.5 points higher for the average score, and all tasks we are worse at are within a very small margin (< 1 point). This confidently demonstrates Hybrid-Ladder-8B-16L-retrained can be a drop-in replacement to Llama-3.1-8B-Instruct, achieving 23% speedup with no accuracy lost.

We also thank the reviewer for the feedback on clarity and the formatting issue, we will incorporate these in the next version.