We are grateful for the constructive feedback and insightful comments. Please find our answers below.

The paper lacks a clear, explicit description of the mechanism used to enforce the subspace constraint on the projection matrices.

We thank the reviewer for pointing this out. In practice we enforce the constraint with an inexpensive matrix multiplication:

$W_{projected} = WUU_k$.

We will clearly mention this in the revision.

Ablation with $k$

In our experiments, we observed that compression ratios as high as 200× can be achieved under certain hyperparameter configurations with only minimal performance degradation. However, we chose to use a more conservative 100× compression (with $k = 40$) as the default and gain consistent performance across a variety of setups (different datasets, model sizes etc.), showcasing its robustness. This choice also supports our practical goal of enabling pipeline-parallel training over internet-grade bandwidth. In response to the reviewer’s suggestion, we now include an ablation study that varies $k$ (and thus the compression ratio). The results, shown below, are for a 1B model trained on BookCorpus. We thank the reviewer for this insightful suggestion.

While a mechanism to update the shared subspace using the Grassmann manifold is proposed, it lacks a thorough sensitivity analysis.

We thank the reviewer for pointing this out. Our observation is that the grassman updates matter towards mid/end training (we have provided an ablation in Fig.14), and very infrequent updates (every 500th iteration) are enough. We would add a more thorough ablation in the supplementary.

The paper's narrative flow could be improved. The separation of the methodology (Section 4) from its theoretical justification (Section 5) can be disruptive; integrating the theory more directly into the method's description could make the core ideas clearer.

We appreciate the reviewer’s input. We intentionally separated Section 4 (method) from Section 5 (theory) so that readers can first grasp the overall procedure without being slowed by mathematical details. In the camera‑ready, we will improve the flow by adding forward pointers in Section 4 (e.g., “See Sec. 5.2 for the formal proof of Step 3”) and brief intuitive summaries inside Section 5. This integrates the theoretical justification more tightly with the algorithm while still keeping the main text accessible to readers who prefer a math‑light first pass.

Could Subspace Networks be combined with methods that induce low-rank structure in self-attention matrices or with other techniques used in DDP to achieve even greater communication savings?

This is a very insightful question. Our subspace compression approach is indeed orthogonal to, and compatible with, other compression techniques, including those used in data-parallel (DDP). To demonstrate this, we conducted an experiment using a hybrid setup that combines DDP and pipeline-parallel PP training. In this configuration:

1. We apply low-rank compression and top-K compression to weight gradients to mimic typical DDP compression strategies.

2. Simultaneously, we apply subspace compression across the pipeline-parallel segments.

We train a 1B model in this setup with 80Mbps links. This combined setup yields strong end-to-end communication savings while preserving convergence as shown in below results.

We also include results with Swarm parallelism [1], where our method is layered on top of standard Swarm compression (e.g., quantization).

These results highlight that Subspace Networks can be composed with other compression schemes (e.g., low-rank, quantized gradient exchange) to enable efficient training across heterogeneous environments with varying bandwidth constraints.

We thank the reviewer for this excellent suggestion and will expand on these findings in the supplementary material.

How does this constrained optimum relate to the stationary points of the original, unconstrained problem? Could the subspace constraint potentially prevent the model from reaching a better (e.g., lower loss) stationary point that exists outside the predefined subspace?

This is an intriguing question. Our central argument for the overlap, and comparable performance, between the constrained and unconstrained optima is that the final solutions for $W_{p1}$ and $W_{p2}$ already reside in the low‑rank (constrained) sub‑space. We support this claim theoretically (Theorem E.2, G.1) and empirically: as shown in Fig. 16, we analyzed the rank distributions of the projection matrices in multiple frontier open‑source models (DeepSeek, Llama, Qwen, and OLMo, spanning 4 B to 70 B parameters) and found them all to be highly rank‑deficient.

Because the true solution lies within the constrained space, projected gradient descent naturally converges to a first‑order stationary point that contains a feasible low-rank solution. While it is conceivable that alternative training strategies could discover higher‑rank optima with lower loss, our experiments and every set of publicly available weights we have examined indicate that state‑of‑the‑art models consistently converge to solutions inside the proposed low‑rank manifold.

In Figure 4, the 16-layer model appears to reach a similar training loss as the 8-layer model in much fewer training time.

This is an intriguing question and we thank the reviewer for noticing this. Although the 16‑layer model requires more computation, (which is visible by lower TPS in Fig. 4), it also reduces the loss more quickly per training iteration. Similar behaviour can be seen in the publicly available OLMo training curves. We invite the reviewer to inspect the public WandB logs for OLMo, which exhibit the same trend. (We omit direct links here to comply with NeurIPS rules.) Because the larger model needs fewer iterations to reach a given loss, its wall‑clock time to a similar loss can be shorter despite the higher per‑step compute cost.

Minor suggestions:

Thank you for pointing these out. We will address all of the above.

[1] - Ryabinin et al. - Swarm Parallalism

We thank the reviewer for the positive comments and intriguing questions. Please find our answers below.

It would provide more clarity if the measurements of the computation and memory is clearly stated in the start of section 6

Thank you for the helpful suggestion. We have included detailed memory and computation analysis in Supplementary Sections K and L (see Tables 3 and 4) and show that ours incur extremely minimal overhead. As per the reviewer’s recommendation, we will also add a brief summary of these measurements at the beginning of Section 6 to improve clarity.

Can you provide more background introduction of the centralized method and explanation

We appreciate the reviewer’s concern. In our setting, the centralized model refers to the same architecture as the decentralized version, but deployed on a high-performance GPU cluster, where each transformer block (residing on a separate GPU) is connected to the others via high-bandwidth, low-latency links. This setup allows near-seamless communication between layers. In contrast, our decentralized model places each transformer block on a node connected via low-bandwidth links, such as internet-grade connections. This introduces significant communication constraints, particularly for activations and gradients, which our method is specifically designed to address.

The strong alignment in performance between these two regimes highlights the effectiveness of our compression scheme in bridging the performance gap caused by bandwidth limitations. We will expand this explanation in the revised version to clarify the distinction and significance of the centralized baseline.

The words in the plot is too small.

Thank you for pointing this out. We will fix this.

Why the decentralized with no compressed is not visible when the x-axis is training steps? Would the loss be similar as the proposed method regarding the training steps?

The decentralized uncompressed model exhibits identical training dynamics to the centralized model when plotted against training steps, as the architecture and optimization procedure are the same. The only difference lies in the communication bandwidth between GPUs. As a result, its loss curve completely overlaps with that of the centralized model in figures where the x-axis is training steps, which is why it is not separately visible.

However, it is important to note that, as shown, the tokens-per-second (TPS) of the decentralized uncompressed model is significantly lower due to the high communication cost over low-bandwidth links. This results in much slower convergence in wall-clock time, which is where our proposed method provides a clear advantage by enabling efficient training under bandwidth constraints.

What is the intuition behind the use of F norm?

This is indeed a great question. Below we summarize the rationale behind this choice.

1. Relation to principal angles
   For two $k$‑dimensional sub‑spaces $\mathcal U,\mathcal V$ with orthonormal bases $U$ and $V$,

=(|UU^{\top}-VV^{\top})|_{F} =\sqrt{2\sum \sin^{2}\theta_i},$$ where $\theta_i$ are the principal angles. Therefore, minimising $d_F$ therefore minimises the **average squared mis‑alignment** across all $k$ directions. 2. **Rotational invariance** Because the metric relies only on the projection operators $UU^{\\top}$ and $VV^{\\top}$, any orthonormal basis that spans the same sub‑space yields **exactly the same value**. This ensures the distance is defined for sub‑spaces themselves, not for a particular choice of basis. 3. **Average‑case discrepancy** (instead of worst‑case) While the spectral norm captures only the largest principal angle (worst‑case deviation), the Frobenius norm **sums over all angles**, giving an overall or average measure of how much the two sub‑spaces differ. This is the relevant notion of closeness when we care about preserving information in every direction, not just the single worst one. # The experiments involve training across different locations, but there’s no study on how the number of machines affects training speed or results. Thank you for raising this point. To address it fully, could you please clarify what configuration you have in mind? Specifically, are you referring to increasing the number of nodes in a data‑parallel (DDP) setup? In that case, the results would be the same since that would be akin to training with a larger batch size. If the reviewer has a different setup in mind, we would like to respectfully ask for more information so we can run the appropriate ablations and report their effects on throughput and convergence. # How the initial subspace setup of $U_k$ affects training. We initialise $U_k$​ by drawing an isotropic Gaussian matrix and taking its top‑k principal components via SVD. After orthonormalising $U_k$​, the specific directions and the initial standard deviation no longer influence convergence. # How does $k$ affect training. In our experiments, under specific hyperparameters, we achieved compression ratios as high as 200× without much performance degradation. Nonetheless, we used a more conservative 100× ratio ($k=40$) by default for stable performance across setups. This ratio also meets our practical goal: training in a pipeline-parallel setup even when nodes are connected via internet-grade networks. To address the reviewer’s request, we now include an ablation study varying k (and therefore the compression ratio). The results are shown below for a 1B model over BookCorpus. We thank the reviewer for the suggestion. | Method | Perplexity | Compression| |--------------------------|------------|------------| | Centralized | 18.35 | - | | Decentralized (K = 40) | 17.63 | 100x | | Decentralized (K = 20) | 18.31 | 200x | | Decentralized (K = 10) | 20.28 | 400x | | Decentralized (K = 5) | 34.81 | 800x | We agree that exploring the true upper limits and deriving more intuitive heuristics (potentially tied to model architecture or task requirements) is an exciting direction for future research. # Analyses on how the subspaces drift in the experiments is helpful. We agree that reporting subspace drift would be valuable, and we will add these statistics in the revision. # What isn’t clear is exactly at what bandwidth threshold it starts to outperform other approaches—answering this would require comparing multiple methods across a range of bandwidths Thank you for this important question. As requested, we provide TPS comparing our method with alternative approaches across a range of bandwidths. We observe that top‑k sparsification allows up to a 10× compression and 8‑bit quantization enables around 4× compression before significant degradation in convergence. Note that these findings are consistent with those reported in [1]. Accordingly, we fix those compression ratios for the baselines. We exclude SVD‑based low‑rank method as it is extremely slow. Our method achieves up to 100× compression without compromising convergence. This allows it to outperform other methods even when the available bandwidth drops to 80Mbps. The following results (to be included in the supplementary) show that our approach demonstrates a consistent TPS compared to other methods across the range of bandwidths, making it highly practical for pipeline‑parallel training over internet-grade connections. We appreciate the reviewer’s suggestion and will include the table and performance plots in the supplementary. | Method | TPS @ 100 Gbps | TPS @ 1 Gbps | TPS @ 800 Mbps | TPS @ 500 Mbps | TPS @ 80 Mbps | |--------------|----------------|--------------|----------------|----------------|----------------| | Baseline | 602 | 378 | 333 | 222 | 36 | | Top-K | 612 | 532 | 521 | 451 | 129 | | Quantization | 608 | 499 | 487 | 415 | 83 | | Ours | 624 | 612 | 610 | 603 | 592 | [1] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training

We are thankful to the reviewer for the continued engagement and the thought provoking question.

On distributing the nodes across a varying number of geographical regions

In the bandwidth-limited, steady-state regime where (i) per-layer compute and intra-region transfers costs are negligible, and (ii) throughput is limited by inter-region links, the slowest inter-region link bottlenecks and sets tokens-per-second (TPS). Once the pipeline is saturated, changing the number of regions $d$ does not meaningfully change TPS assuming a standard number of micro batches.

---

Analysis

Let:

• $n$: total layers, partitioned across $d$ regions
• $p$: bytes in activations that must cross a region boundary (after compressing activations & activation gradients)
• $B_i$: bandwidth of inter-region link $i$; let $B_{\\min}=\\min_i B_i$
• $m$: number of microbatches (pipeline chunks)
• $\alpha$: scheduling constant ($\\alpha \\approx 1$ for GPipe; $\\alpha \\approx 2$ for 1F1B)

With communication time $\\gg$ compute time, the stage service time is dominated by the worst boundary:

$$T_{\text{stage}} \approx \max_i \frac{p}{B_i} = \frac{p}{B_{\min}}.$$

In steady state (pipeline filled),

$$\\text{TPS} \\approx \\frac{B_{\\min}}{p} \\times \\frac{m}{m+\\alpha(d-1)}.$$

Typically $m \\gg d$, yielding,

$$\boxed{\text{TPS} \approx \frac{B_{\min}}{p}\ \ \text{(independent of } d\text{)}}.$$

---

However, If $m$ is too small, more regions increase bubble overhead. Nonetheless, in typical pipeline parallel settings, $m$ is chosen to saturate the pipeline.

Therefore, under the stated assumptions (bandwidth-dominated, saturated pipeline, isolated links), TPS is governed by $B_{\\min}/p$. Thus, the number of geographical regions $d$ is not a primary determinant of throughput. We will clarify these assumptions and include the above analysis in the appendix to make this explicit and try our best to add an ablation. Thank you for the valuable suggestion.

We thank the reviewer for the insightful comments and feedback. We address the reviewer's concerns below.

Novelty, the low-rank property is a well-known phenomenon, prior work already uses low-rank projection.

We thank the reviewer for their feedback and would like to clarify key points that distinguishes our contribution.

1. Low‑rank structure of activations is not a well known property

While the community has long recognized that weight matrices and their gradients often exhibit low effective rank (enabling techniques such as LoRA, PowerSGD) the same assumption does not hold for activations or activation gradients. In transformers, activations are inflated in rank because PE and TE are re‑added at every layer. This resulted in an obscure view in earlier studies, leading them to report accuracy drops when activations were compressed beyond ≈10%, concluding that aggressive activation compression is infeasible [1,2]. Another contributing factor to this is the exponential error accumulation (Theorem D.1).

2. Our key insight: remove embedding components, reveal a hidden low‑rank subspace

We show that once PE and TE contributions are removed, the residual activations lie in the span of the layer’s output‑projection matrices, which is a provably low‑rank subspace.

Concretely: 1) PE can be subtracted inside each block because they are deterministic. 2) TE are batch‑dependent and high‑rank. We decompose them into a fixed high‑rank component and a learnable low‑rank component that can be compressed efficiently.

This decomposition exposes a low‑rank structure in activations and their gradients that, to our knowledge, has not been identified or exploited in prior work.

3. Practical impact: efficient pipeline‑parallel training

By isolating the compressible subspace, we achieve activation and activation‑gradient compression well beyond the previous rates, without measurable loss in accuracy, enabling end‑to‑end pipeline‑parallel training with dramatically lower bandwidth. Previous methods focused on weight/gradient compression could not unlock this regime. Also the network provably converges to a feasible solution with constrained optimization

We hope this clarifies that our contribution goes beyond applying a “standard” low‑rank trick: it uncovers and leverages a previously overlooked low‑rank structure in activations.

Comparisons with other MP or DP methods.

1. Comparison against other DP methods

Because our algorithm is tailored to MP training, it is not directly comparable to DP approaches. DP assumes each full model replica lives on a high‑capacity cluster, which is less restrictive than splitting the computational graph across the internet, which is the setting we target. Existing decentralized or communication‑efficient DP methods address slow bandwidth between clusters, not between individual model layers, making that problem comparatively easier.

Our method is therefore orthogonal and complementary to DP communication‑reduction techniques and can be layered on top of them. To illustrate this, we present an additional experiment below: we applied top‑k and low‑rank compression to the weight gradients (replicating a DP scenario) while simultaneously using our scheme to compress activations and activation gradients. Under an 80 Mbps link between devices, we achieved a substantial increase in TPS with no loss of convergence.

2. Comparison against other MP methods

As noted in our previous response, activation compression in model‑parallel (MP) pre‑training has generally been considered harmful to convergence, so we are not aware of any published method that successfully compresses both activations and activation gradients during pre‑training.

The recent study [2] (which does not propose a new algorithm) benchmarks 8‑bit/4‑bit quantization and top‑K sparsification. We already include these baselines, and a vanilla low‑rank SVD variant, in Fig. 6, where our method significantly outperforms them. We would also like to add a note on why our method outperforms low-rank SVD methods.

Why our algorithm outperforms low‑rank SVD?

1. Embedding removal & constrained factorization of the projection matrices expose a strictly lower‑rank subspace; naïve SVD sees the full high‑rank tensor.

2. No per‑batch SVD: we replace the O(d³) decomposition with a single matrix multiplication derived from our constrained optimization, yielding substantial runtime savings.

3. Convergence guarantees as our method can be formulated as a projected gradient descent problem.

Further, [3] proposes compressing activation deltas during fine‑tuning, but this technique is infeasible for large‑language‑model (LLM) pre‑training: 1) Memory overhead. It must store the full activations for every training example, which is infeasible at LLM scale. 2) Requirement for multi epoch training. Error elimination requires many epochs, and the method fails to converge in the few‑epoch regimes typical of large‑scale training, where workloads often cover less than a single pass over the dataset. A different research thread ([4,5,6]) focuses on memory efficiency by compressing activations within each module and decompressing them later to compute weight gradients. Because they leave the inter‑device traffic unchanged, these methods do not address communication efficiency and are therefore not comparable to ours.

We did identify a very recent workshop paper [7] that uses column masking to compress activations. However, it achieves only up to 10× compression, whereas our technique reaches 100×. For completeness we have added a comparison: their convergence degrades sharply at an aggressive compression rate, while ours remains stable. We will add this to supplementary.

If the reviewer is aware of additional MP activation‑compression algorithms, we would be grateful for citations. We will try our best to provide ablation tables for such methods in the discussion period and include in the supplementary material.

We further conducted an experiment with Swarm parallelism [8]. Swarm’s primary goals are fault tolerance and dynamic volunteer participation rather than communication efficiency. Swarm implementation relies on standard activation and gradient quantization for compression. Our technique is therefore is orthogonal to swarm too and can be layered on top of Swarm to further reduce bandwidth. To demonstrate this, we have conducted an additional experiment that applies our compression within a Swarm setting; the results are provided below.

Is rank collapse consistently observed across different models? Even if so, might the optimal rank dimension vary with model type, architecture, size, or training scheme?

Yes. As shown in Fig. 16, we analyzed the rank distributions of the projection matrices in multiple frontier open‑source models (DeepSeek, Llama, Qwen, and OLMo, spanning 4 B to 70 B parameters) and found them all to be highly rank‑deficient.

In our experiments, we were able to achieve compression ratios of up to 200× under certain hyperparameter settings, with minimal impact on performance. However, we adopt a more conservative 100× compression (using $k=40$) as the default to ensure stable performance across all of our configurations. Note that we use the gain consistent performance with this value across all our ablations (different datasets, model sizes etc.). This setting also aligns with our practical objective: enabling pipeline-parallel training over internet-grade connections. In response to the reviewer’s suggestion, we now include an ablation study varying $k$ (and thus the compression ratio). Results are reported below for a 1B model trained on BookCorpus. We sincerely thank the reviewer for this valuable feedback.

What does the loss curve look like if training continues until full convergence?

We have already trained models to full convergence using the compute-optimal 1:20 model-to-token ratio from the Chinchilla scaling law [9] in Appendix F. We achieve even better convergence than centralized models.

What is the overhead of the subspace-projection?

We have already provided these measurements in Appendix L.1. The overhead is approximately 1%, indicating a negligible computational burden.

[1] - Bian et al. - Does Compressing activations help model parallel training?

[2] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training

[3] - AQ-SGD

[4] - Shamshoum et al. - CompAct

[5] - Chen et al. - ActNN

[6] - Evans et al. AC-GC

[7] - Ramasinghe et al. - Beyond Top-K

[8]- Ryabinin et al. - Swarm Parallalism.

[9] - Chinchilla

Thank you for the encouraging comments and constructive feedback. We address each of the reviewer’s specific concerns below.

My primary request is to include baseline methods that cover pipeline parallelism or existing model parallelism strategies (swarm parallalism)

We thank the reviewer for the suggestion. Swarm implementation relies on standard activation and gradient quantization, and multiple node integration for computation pipes, for compression and increased throughput. Our technique is therefore orthogonal and can be layered on top of Swarm to further reduce bandwidth. To demonstrate this complementarity and address the reviewer’s concern, we have conducted an additional experiment that applies our compression within a Swarm setting; we train a 1B parameter model in swarm, with and without our compression. Results are summarized below and we will add the loss curve in the revised supplementary. As illustrated in, with our compression, swarm achieves a remarkable >11x throughput gain.

Please note that activation and activation‑gradient compression during model‑parallel pre‑training has long been viewed as harmful to convergence, making it difficult to find suitable baselines. The recent survey paper [1], which proposes no new algorithm, evaluates 8‑bit / 4‑bit quantization and top‑k sparsification for MP. We already include these baselines, along with a vanilla low‑rank SVD variant, in Fig. 6; our method consistently outperforms them.

[2] introduces an activation‑delta compression scheme for fine‑tuning, but it is impractical for LLM pre‑training for two reasons. (1) Memory overhead: the method must cache full activations for every training sample, which is infeasible at LLM scale. (2) Multi‑epoch dependence: the accumulated error is only removed after many epochs, so the approach fails to converge in the few‑epoch regimes typical of large‑scale training, where workloads rarely exceed a single pass over the data. A different research thread ([3,4,5]) focuses on memory efficiency by compressing activations within each module and decompressing them later to compute weight gradients. Because they leave the inter‑device traffic unchanged, these methods do not address communication efficiency and are therefore not comparable to ours.

Upon further inspection, we did find a very recent workshop paper [6] that applies column masking to compress activations, but it achieves at most a 10 × reduction. In contrast, our technique delivers up to 100 × compression. We ran an experiment to validate the efficacy of our method using a 1B model on BookCorpus. The results are shown below after 5k iterations. Note that their method’s convergence is notably inferior to ours at 100x compression rate.

If the reviewer is aware of any other MP activation‑compression algorithms, we would greatly appreciate the citations. We will do our best to add ablation results for those methods during the discussion period and include them in the supplementary material.

[1] - Rudakov et al. - Activations and Gradients Compression for Model-Parallel Training

[2] - Wang et al. AQ- SGD

[3] - Shamshoum et al. - CompAct

[4] - Chen et al. - ActNN

[5] - Evans et al. AC-GC

[6] - Ramasinghe et al. Beyond Top-K