StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training

Thanks for reading our paper. We would like to address your concerns as follows:

1. About USP. The main contribution of USP is to efficiently combine Ring Attention and Ulysses. As we discussed in Section 2.1.2, Ulysses is orthogonal to ring-style sequence parallelism, and our work specifically focuses on optimizing the performance of ring-style SP. As such, USP lies beyond the scope of this paper. Nevertheless, we are open to exploring potential combinations of StarTrail and Ulysses in future work on hybrid parallelism.

2. About LoongTrain. The most relevant aspect of LoongTrain to our work is its design of Double-Ring Attention. However, Double-Ring Attention pursues a fundamentally different objective. It partitions GPUs into $C$ outer rings and $P/C$ inner rings. The model first passes Keys and Values through each inner ring for one full round, followed by a single iteration through the outer ring. In this setup, the communication volume per inner ring round is $C2BNH$, repeated $\frac{P}{C}$ times, and the outer ring adds $\frac{P}{C}2BNH$. Thus, the total P2P communication volume becomes $2CBNH \times \frac{P}{C} + \frac{P}{C}2BNH = 2\frac{C+1}{C}PBNH$, which is slightly larger than the original Ring Attention ($2PBNH$). By contrast, StarTrail achieves a substantially smaller overall communication volume than Ring Attention. As stated in Section 4.5.2 of the LoongTrain paper, Double-Ring Attention improves computation-communication overlap at the cost of slightly increased total communication volume. StarTrail, on the other hand, simultaneously provides lower communication volume, better overlap, and improved communication locality, at the expense of a modest increase in memory usage. In summary, Double-Ring Attention explores a different tradeoff space in ring-style communication and does not aim to reduce communication volume. For this reason, we believe it is not appropriate to include it as a baseline, and a direct comparison would not be meaningful.

3. Writing. Thank you for your helpful suggestions. We will reverse the order of Figure 2 and Figure 3 and adopt clearer notations to enhance readability. Figure 3 serves as an overview of the entire system, with its components described in detail in Section 3. To further improve clarity, we will add an overview subsection following Section 3.1 to guide readers through the system structure in the revised version.

Thanks again for your thoughtful feedback. We sincerely hope we have addressed your concerns.

Thanks for reading our paper! We would like to address your concerns as below:

1. Related Work. First, we would like to clarify that StarTrail focuses on optimizing the communication pattern of ring-style sequence parallelisms, and the currently most widely used baseline is Ring Attention. Other related works have focused on different aspects of sequence parallelism. For instance, we have discussed DistFlashAttention in Section B.3. DistFlashAttention proposes two main contributions. The first is a modification of the communication pattern to address the load imbalance issue in causal models. However, a more elegant solution, the Zigzag dataloader, has been proposed to address this problem. By simply modifying the data loading process, we can balance the load across GPUs. We have implemented the Zigzag dataloader in StarTrail, as discussed in Section A.1. The second contribution of DistFlashAttention involves reordering recomputation steps, which we have also implemented as a feature in StarTrail. StarTrail demonstrates further performance improvements by incorporating both of these techniques into the baseline. Therefore, we believe it is not necessary to include DistFlashAttention as an evaluation baseline. Nonetheless, we are happy to explore the integration of StarTrail with other optimization techniques in future work.

2. Collective communication. We would like to clarify that we have discussed how the cost of collective communication scales with the team size $C$ or the total number of GPUs in Section 3.2.2, line 221. As shown in Equation (2), the additional collective communication volume is much smaller than the reduced P2P communication volume, provided that $C<\sqrt{P}$. Moreover, we would like to emphasize that the condition $C<\sqrt{P}$ is a feature, rather than a limitation to be addressed. As discussed in Section 3.2.1, the valid range for $C$ is $1 < C < \sqrt{P}$. In our experiments, with $P=64$, $C$ should be greater than 1 and less than 8. When $C=1$, StarTrail degenerates to Ring Attention. When $C=8$, there are no rings; each GPU in a team first gathers Keys and Values from the other 7 teams, performs attention computation, and applies reduce-scatter to obtain the final result. The optimal choice of $C$ depends on the system configuration, and StarTrail achieves the best performance at the sweet spot that balances reduced P2P communication and overlapping collective operations with matrix multiplications.

3. Topology of hardware. We would like to clarify that our evaluation includes four different cluster topologies to demonstrate that StarTrail consistently outperforms the baseline regardless of hardware configuration. As shown in Table 1, our experiments span 2 GPU types, 4 topologies, and 2 types of inter-node connections, covering most common configurations used in modern Transformer training workloads. We would also like to emphasize that StarTrail does not rely on keeping sub-rings within a single node to achieve performance gains. We consider two possible scenarios, and we refer to inter-node bandwidth as $W_l$ and intra-node as $W_h$. In the first, all ring-style communication occurs within nodes. In this case, the communication overhead is $(\frac{P}{C^2}-1)(\frac{2CBNH}{PW_h} + L)=\frac{(P-C^2)2BNH}{CPW_h} + (\frac{P}{C^2}-1)L$. In the second scenario, where sub-ring communication spans multiple nodes, the communication is bottlenecked by inter-node bandwidth, yielding $(\frac{P}{C^2}-1)(\frac{2CBNH}{PW_l} + L)=\frac{(P-C^2)2BNH}{CPW_l} + (\frac{P}{C^2}-1)L$. As discussed in the paper, StarTrail reduces communication overhead by approximately $\frac{W_h}{W_l}C$ and $C$ in these respective cases. As for less favorable environments, it becomes even more difficult to overlap communication with computation, thus making StarTrail more beneficial by reducing communication volume. For more complex topologies, communication is still constrained by the lowest-bandwidth links in the ring, and the corresponding theoretical analysis remains similar. We would also like to clarify that our goal is to address challenges in common long-sequence training settings, while more exotic or complex cluster topologies are beyond the scope of this work.

Thanks again for your suggestions, and we sincerely hope we have addressed your concerns.

Thanks for reading our paper and providing constructive suggestions! We sincerely appreciate your feedback and would like to address your concerns as follows.

1. Bandwidth in the analysis. In our theoretical analysis, we assumed a single bandwidth $W$ to simplify the presentation. However, we are happy to extend the analysis to incorporate both inter-node and intra-node bandwidths. Specifically, we denote the inter-node bandwidth as $W_l$ and intra-node bandwidth as $W_h$. For Ring Attention, as illustrated in Figure 2, the overall communication bandwidth is constrained by the bottleneck $W_l$, and the communication overhead can be expressed as $(P - 1)(\frac{2BNH}{PW_l} + L) = \frac{2BNH(P-1)}{W_lP} + (P - 1)L$.

   For StarTrail, we consider two possible scenarios. In the first case, all ring-style communication occurs within nodes. The corresponding communication overhead is $(\frac{P}{C^2}-1)(\frac{2CBNH}{PW_h} + L)=\frac{(P-C^2)2BNH}{CPW_h} + (\frac{P}{C^2}-1)L$. In the second case, where sub-ring communications span across nodes, the bandwidth is again limited by $W_l$, leading to a communication cost of $(\frac{P}{C^2}-1)(\frac{2CBNH}{PW_l} + L)=\frac{(P-C^2)2BNH}{CPW_l} + (\frac{P}{C^2}-1)L$. As in our main analysis, StarTrail reduces communication overhead by approximately $\frac{W_h}{W_l}C$ and $C$ in the two cases, respectively.

2. Topology and related communication. We would like to clarify several key concepts introduced in the paper:

   • Teams: Groups of $C$ GPUs that perform all-gather and reduce-scatter operations; for example, 2 GPUs per team as shown in Figure 3(b).
   • Subrings: Smaller rings used for ring-style communication, each comprising $P/C^2$ GPUs (e.g., 4 GPUs in Figure 3(b)).
   • Team groups: Collections of subrings that are connected via all-gather and reduce-scatter. In Figure 3(b), there are two team groups: the first includes teams 0–3 and the second includes teams 4–7. Communication between team groups only occurs during the initial P2P communication phase (Figure 6, Section A.2). In subsequent iterations of ring-style and collective communication, no inter-group communication occurs.

3. The value of C. As detailed in Section 3.2.1, the parameter $C$ satisfies $1 < C < \sqrt{P}$. In our experiments with $P=64$, valid values for $C$ are in the range (1, 8). When $C = 1$, StarTrail reduces to Ring Attention. When $C = 8$, rings are eliminated entirely: each GPU gathers Keys and Values from the other 7 teams, performs attention computation, and applies reduce-scatter for final aggregation. This scenario introduces large-scale collective communications that cannot be overlapped with matrix multiplications, leading to performance degradation—hence we did not include it in our main figures. However, we are happy to include this case in the appendix of the revised version. The optimal choice of $C$ depends on system characteristics, and StarTrail achieves the best performance at the sweet point of the tradeoff between the reduction of P2P communication and the collective operations, which can be partially overlapped with matmuls.

4. Convergence. StarTrail is mathematically equivalent to FlashAttention (and thus to Ring Attention). As described in Algorithm 1, StarTrail computes attention scores block by block, preserving intermediate values $lse$ and $O$ and updating them iteratively. The only difference is that StarTrail computes $C$ intermediate scores in parallel and merges them using reduce-scatter, rather than updating scores sequentially as in FlashAttention. These approaches are mathematically equivalent. Therefore, StarTrail maintains the same convergence behavior as Ring Attention while improving runtime. That said, we are willing to provide convergence curves in future versions for completeness.

5. Hybrid Parallelism. We address this concern from three perspectives.
   First, sequence parallelism (particularly Ring-style) operates independently of other forms like tensor or pipeline parallelism and is typically deployed across nodes. This placement makes it more challenging to overlap communication with computation, which further motivates the need for StarTrail.
   Second, emerging workloads—such as multi-modal models or long chain-of-thought reasoning—require sequence lengths far exceeding 128k tokens, necessitating higher degrees of sequence parallelism.
   Third, even at small degrees of CP(SP) (e.g., 8), StarTrail delivers improved performance with small team sizes (e.g., 2), as demonstrated in Figure 10(a) and (b). Thus, we believe StarTrail is necessary to support the scaling of modern Transformer models with increasingly longer sequences.

6. Bandwidth in evaluations. The benefit of StarTrail is related to the communication bandwidth: the lower the bandwidth, the more significant the advantage. So the necessity of optimizing the communication can be weakened if the communication can be well-covered by computation. However, as we have discussed in the last paragraph, sequence parallelisms are usually placed between nodes in real scenarios so that parallelisms that require more communication, like tensor parallelism, can be kept within the nodes. This makes it hard for CP(SP) to be overlapped. To evaluate bandwidth impact, we included two types of interconnects: $8 \times 400$ Gbps InfiniBand (which is among the highest inter-node connection bandwidths and typically deployed on the newest NVIDIA DGX systems) and 100 Gbps Ethernet (common in cloud A100 setups), as shown in Table 1. Experimental results show that StarTrail significantly outperforms Ring Attention even under high-bandwidth settings like 8*400 Gbps InfiniBand, underscoring its practical effectiveness.

7. Typo. We apologize for the typo in Figure 8. The y-axis should indeed be normalized to 1. We will correct this in the revised version.

Thanks again for your thoughtful suggestions and careful reading of our paper. We sincerely hope that our responses have addressed your concerns.

Thank you for the comments from Reviewer xk9c and also author.

After reading your discussion, the discussion from my side, and also the LoongTrain paper, I have a new question.

• According to the HuggingFace Transformer Flops Calculator and Figure 7 of your paper, I find that for 128k sequence, StarTrail can achieve MFU ≈ 12%
• However, under the same setting, the LoongTrain paper shows that they can achieve nearly MFU ≈ 40% as shown in Figure 14 of their arxiv paper.
• Meanwhile, your implementation of LoongTrain is 20% slower than StarTrail.

What are your thoughts on the numbers?

Thank you very much for your thoughtful and constructive feedback, and for recognizing the contributions of our work.

StarTrail is a system-level optimization designed for long-sequence Transformer training. It modifies the order of computation while preserving equivalence with the original attention mechanism. As the paradigm of sequence parallelism has largely converged to Ulysses-style and Ring-style Attention, we focus in this work on optimizing the performance of the Ring-style variant, which is widely applicable in practice. That said, we are also interested in extending StarTrail to other attention mechanisms, such as Linear Attention, and appreciate the suggestion to consider broader applicability.

We acknowledge that the hierarchical structure of StarTrail may be challenging to follow upon first reading. In response to your comments, we will revise the manuscript to provide clearer and more detailed explanations of the process, and will move essential content from the appendix into the main text to enhance clarity.

Once again, thank you for your valuable insights and for helping us improve our work.