From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics

We sincerely thank you for your detailed and insightful feedback, which helps us significantly improve the clarity and rigor of our paper. We address each of your points below.

Response to Question 1

Our theoretical analysis does not rely on strong data assumptions—only the mild condition

$

\boldsymbol{v} := \frac{\sum_{i=1}^n y_i \left(\sum_{j=1}^s X_{i,j}\right)}{\left\| \sum_{i=1}^n y_i \left(\sum_{j=1}^s X_{i,j}\right) \right\|_2} \neq 0.

$

which holds in very general settings. Thus, the analysis is largely task-independent. To empirically demonstrate this generality beyond our synthetic task, we conducted new experiments on the Wikitext-103 dataset using a standard 2-layer Transformer with GeLU activation and residual connections. To visualize the result, we define the following aggregation index to reflect our results. According to the definition 1 of the article. A neuron is condensed if its alignment with the dominant direction exceeds:

$$\left| \left\langle \frac{\boldsymbol{W}\_k}{\| \boldsymbol{W}\_k \|}, \boldsymbol{v} \right\rangle \right| \ge 0.95$$

The condensation rate is defined as follows:

\text{Condensation Rate} = \frac{\text{\\# condensed neurons}}{\text{total neurons}}

We plot the condensation rate of the first layer of decoder.

Our findings on this 2-layer Transformer confirm that the same two-stage learning dynamics emerge, directly addressing the concern about whether our results generalize to larger networks and more complex tasks. Specifically, we observed that the $W_V$ and MLP weights condense first, followed by the $W_Q$ and $W_K$ matrices, consistent with our theory.

Response to Question 2

By "extremely overparameterized," we refer to the empirical observation that modern LLMs have far more parameters than needed to fit their training data. This is supported by scaling laws, as shown by Kaplan et al. [1, Section 3.2], where larger models consistently yield better performance despite having orders of magnitude more parameters than data points.

Response to Question 3

Dropout remains an important regularization technique in modern language models. For example, the Qwen-7B/72B technical reports [3] explicitly state in Section 2.3 that a dropout rate of 0.1 is consistently applied after both attention and feed-forward layers.

Response to Question 4

Rank-collapse refers to the degeneration of parameter or attention matrices into low-rank states. In our work, this specifically describes the collapse of the $W_Q$ and $W_K$ matrices. Similar phenomena have been discussed in prior works such as [2]. Regarding the dichotomy assumptions, its absence means that Theorem 1 holds almost everywhere, rather than implying the dynamics necessarily degenerates or diverges. In other words, Theorem 1’s conclusions apply broadly without requiring this assumption.

Response to Question 5

We emphasize in-context learning (ICL) because recent studies in ICL provide well-developed dynamical frameworks under the sequential prediction paradigm, which inspire task-agnostic analysis. This perspective informs our approach and allows our dynamical analysis to remain independent of specific tasks, which we view as a key contribution. We agree that Line 106 introduces novel terminology and will relocate it outside the Related Work section in the revision.

Response to Question 6

In the linearized model, our dynamics (10) exhibits a high degree of symmetry, which induces useful conservation laws. However, purely symmetric setups can lead to degeneracies—for example, certain measurements may vanish due to exact cancellations. To avoid this, we rely on initial asymmetries to break such degeneracies and make the conserved quantities informative. As training progresses, the dynamics tend to restore symmetry through parameter evolution. This motivates our introduction of the cohesion condition to ensure that the system evolves toward a symmetric structure in a controlled way.

Response to Question 7

As mentioned in our response to Question 1, we do not require any specific assumptions on the dataset. Our analysis is general and can be applied to various tasks, including binary classification, without constraints on the scaling of $X$ and $y$. This flexibility is one of the strengths of our approach.

Response to Question 8

The notation $\dot{\boldsymbol{W}}$ is shorthand for the time derivative of the matrix, i.e., $\frac{\mathrm{d}}{\mathrm{d} t} \boldsymbol{W}$. We will clarify this in the manuscript to avoid confusion.

Response to Question 9

Our analysis can be extended to all loss functions for which the condition

$$l'(0) < 0.$$

holds. In such cases, equation (7) will remain unchanged.

Response to Question 10

Although we omit the activation function in the second part of the analysis, this is acceptable because, at this stage, the entries for $W_V$, $W_1$, and $W_2$ remain small. For activation functions like $\tanh$, the absence of a second-order term in its Taylor expansion means the model remains approximately linear with respect to $W_V$, $W_1$, and $W_2$. Therefore, omitting the activation function simplifies the analysis without affecting the dynamics, as the higher-order terms of $\tanh$ are negligible during this stage.

To further address the reviewer's concern, we conducted additional experiments by removing the activation function on synthetic dataset. The condensation rate, defined as the ratio of condensed neurons to total neurons, showed that the dynamics remained consistent with the original experiment, with condensation occurring at similar time points. The minor misalignment is due to adjustments made to the initialization for training stability.

Response to Question 11

The index in equation (15) corresponds to the sample index, which is defined in Section 3 of the paper.

Response to Question 12

Thank you for your insightful comment. We define the first and second stages based on the degree of parameter aggregation and the change of norm. Specifically, the transition between stages is marked by the following:

• In the first stage, we observe minimal change in the Frobenius norm of $W_Q$, while the norm of $W^{[2]}$ increases significantly.
• The second stage begins when the norm of $W_Q$ starts to increase, which coincides with a more substantial change in model behavior. To visualize this, we provide the Frobenius norms of $W_Q$ and $W^{[2]}$ at various epochs in the table below. As shown, the norm of $W_Q$ remains static until epoch 13, where it begins to grow abruptly by an order of magnitude, marking a clear and sudden transition to the second phase.

Response to Question 13

To evaluate the robustness of the stage timing and the analysis across different seeds, we've conducted five additional runs with seeds 1843, 1984, 2000, 2025, and 2048. The timing of the stage transitions has been measured by tracking the condensation rate of $W_Q$. The results demonstrate a high consistency in the transition between stages, even across different initializations.

The table below shows the condensation rates for $W_Q$ at different epochs, with the corresponding mean and standard deviation:

As seen in the table, the condensation rate shows a clear progression across epochs, with minimal variation between runs. The standard deviations indicate that the transitions are stable, and there is a high level of reproducibility in the dynamics of the system for different seeds.

This consistency reinforces the robustness of the stage timing in the training process.

Reference

[1] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[2] Noci, Lorenzo, et al. "Signal propagation in transformers: Theoretical perspectives and the role of rank collapse." Advances in Neural Information Processing Systems 35 (2022): 27198-27211.

[3] Bai, Jinze, et al. "Qwen technical report." arXiv preprint arXiv:2309.16609 (2023).

Dear Reviewer t9Wk,

We hope this message finds you well! Thank you for your insightful review of our paper. As for the concerns about our work in the review, we have provided very specific and detailed responses. Have these responses resolved your concerns? If you have any further questions about our work or responses during this discussion period, please do not hesitate to contact us. We sincerely look forward to receiving your further comments on our responses.

Once again, thank you for your valuable time and feedback!

Best regards,

Authors

We sincerely appreciate your insightful comments, which have greatly contributed to improving the quality of our work. Below is a point-by-point response to your feedback.

Weakness 1

"Is condensation consistently observed beyond synthetic datasets?"

Response: We confirm the ​​universality of condensation dynamics​​ through new experiments on ​​Wikitext-103​​ (natural language corpus) using a standard 2-layer decoder-only transformer with GeLU activation and residual connections. This demonstrates the phenomenon’s independence from specific data distributions.

Experimental Results

1. Condensation Rate

We define the following aggregation index to reflect our results. According to the definition 1 of the article. A neuron is condensed if its alignment with the dominant direction exceeds:

$$\left| \left\langle \frac{{W}\_k}{\| {W}\_k \|}, {v} \right\rangle \right| \ge 0.95$$

The condensation rate is defined as the number of condensed neurons divided by the total number of neurons.

2. Key Result We plot the condensation rate of the first layer of decoder.

On Wikitext-103, all key matrices rapidly condense (rates approach 1.0), confirming the phenomenon's generality. The observed temporal separation, where linear parameters condense earlier than attention parameters, also provides strong support for our multi-stage dynamics theory.

Weakness 2

"Unconventional architecture: no residual connections, tanh instead of ReLU/GELU"

Response: We address Weakness 2 from two perspectives: experimental validation and theoretical justification.

1. Experimental Setting

As demonstrated in our response to Weakness 1, experiments using a 2-layer, 1-head decoder-only Transformer on Wikitext-103—which employs ​​GeLU activations​​ and ​​residual connections​​—confirm that our core findings generalize to architectures with modern activation functions and residual pathways. It shows the generality of our results to multi-layer gelu activation functions.

2. Theoretical Development

On the versatility of activation functions

Our theoretical results apply to all activation functions that satisfy the following conditions:

$$\sigma(0) = 0 ~ \text{and} ~ \sigma'(0) \neq 0.$$

This condition encompasses GeLU (validated experimentally) and excludes only functions with undefined or zero derivatives at the origin (e.g., ReLU).

Stage 1 dynamics remain consistent​​ under this condition. The empirical loss expansion retains the same asymptotic form:

$$\mathcal{L} ({\theta})= \frac{1}{n} \sum\_{i=1}^n \left[ 1- \varepsilon^3 \left( \sum\_{j=1}^s \frac{1}{s} y\_i X\_{i,j} \bar{{W}}\_V \bar{{W}}^{[1]} \bar{{W}}^{[2]} \right) + o(\varepsilon^3) \right],$$

yielding identical leading-order dynamics:

$$\frac{\mathrm{d} \bar{{\theta}}}{\mathrm{d} \bar{t}} = \nabla\_{\bar{{\theta}}} \left( {v}^{\intercal} \bar{{W}}\_V \bar{{W}}^{[1]} \bar{{W}}^{[2]} \right)$$

So we get the same result for general activation function in the first stage of training.

Stage 2 analysis​​ relies on the assumption that after Stage 1 convergence to a linear critical point, higher-order perturbations remain negligible. This holds for all σ satisfying the above condition, preserving our theoretical conclusions.

Discussion about residual connections

While residual connections warrant deeper exploration, we provide preliminary theoretical insights. Consider a single-layer Transformer with residuals:

$$f\_{{\theta}} (X) = (X + Attn (X) + FFN(Attn(X))) W\_{\text{proj}}.$$

The empirical loss expands as:

$

\mathcal{L} ({\theta})= \frac{1}{n} \sum\_{i=1}^n \left[ 1- \varepsilon \left(y\_i X\_{i,s} \bar{{W}}\_{proj} \right) + o(\varepsilon) \right].

$

Therefore, the first stage of analysis will be much simpler, since the leading order dynamics will be a linear ode governing ${W}\_{proj}$. Next, following the same principles as our second-stage analysis, we assume that the model reaches the critical point of the linear model dominated by $W\_{proj}$ and decompose the empirical loss as follows:

$$\mathcal{L} ({\theta}) \approx \frac{1}{n} \sum\_{i=1}^n \mathcal{L}\_{1,i}({\theta}) \mathcal{L}\_{2,i} ({\theta}) \mathcal{L}\_{3,i} ({\theta})$$

where

$$\begin{aligned} \mathcal{L}\_{1,i} &= e^{-y\_i X\_{i,s} W\_{proj}} \\\\ \mathcal{L}\_{2,i} &= e^{-y\_i Attn(X\_i)\_s W\_{proj}} \\\\ \mathcal{L}\_{3,i} &= e^{-y\_i FFN(X\_{i,s}+Attn(X\_i)\_s) W\_{proj}} \end{aligned}$$

By making reasonable assumptions, we can analyze the dominant dynamics at the moment. At this moment, the change of $W\_V$ will be most significant, rather than $W\_Q$ and $W\_K$. After all linear model changes stop, $W\_Q$ and $W\_K$ in the attention module begin to change. This ​​sequential learning phenomenon​​ aligns with the module-specific update ordering observed in our original work. Future work​​ will formalize this analysis for multi-layer architectures, but current results suggest residual connections preserve the core mechanistic principles we describe.

Response to Question 1:

To address the reviewer's concern about the generalizability of our findings on condensation, we provide the following clarifications. We have indeed confirmed that the condensation phenomenon holds across the various dimensions queried by the reviewer. These results are empirically and theoretically demonstrated in our responses to Weaknesses 1 and 2, as summarized below:

• Residuals & Multi-Layer Models: Addressed by our extended theoretical analysis (Weakness 2) and empirically verified with a 2-layer model on Wikitext-103 (Weakness 1).
• Alternative Activations (GELU/ReLU): Covered by our general theoretical proof and validated with GELU experiments (Weakness 1).
• Beyond Toy Datasets: Confirmed via large-scale experiments on Wikitext-103 (Weakness 1, Table 1).

Response to Question 2:

We appreciate these technical inquiries and address them sequentially below.

What is the exact initialization used for this experiment?

Given any trainable parameter matrix $W \in \mathbb{R}^{d\_1 \times d\_2}$, where $d\_1$ and $d\_2$ denote the input and output dimensions, respectively, its elements are initialized according to a normal distribution:

$$W\_{i,j} \sim \mathcal{N}\left( 0, d\_{1}^{-2\gamma} \right),$$

where $\gamma$ is the initialization rate. Specifically, the initialization scale decreases as $\gamma$ increases. Note that $\gamma = 0.5$. In our experiments, we take $\gamma = 0.8$.

Could the authors elaborate what language task it is simulating?

Our constructed dataset is designed to simulate algorithmic reasoning tasks, similar to those found in works like [arxiv:2306.00802, 2401.08309]. The primary goal is to create a controlled environment that isolates a model's ability to learn fundamental structural relationships between tokens, a core capability required for more complex language understanding.

How the cosine similarity matrices of Fig 1 are calculated? What are the numerical values on y-axis in Fig 1?

• Calculation Method: For parameter matrix ${W} \in \mathbb{R}^{n \times d}$:

  • Row-wise normalize: $\tilde{{W}}\_i = \frac{{W}\_i}{\\| {W}\_i\\|\_2}$
  • Compute $\tilde{{W}} \tilde{{W}}^{\intercal}$

• Y-axis Values: The numerical values on y-axis in Fig 1 is about 4-4.5 which corresponds to relative early stage of training.

It looks like $W^{[1]}$ is no longer in the condensation state at the end of training in Fig 1 - is this expected?

Yes, this is entirely expected and a key part of our findings. Our results show that training proceeds in stages:

• Stage 1 (epoch 0- 15): ${W}\_V$, ${W}^{[1]}$ and ${W}^{[2]}$ first condense into a low-complexity state to capture the task's simplest patterns.
• Stage 2 (epoch 15-50): Other components, like ${W}\_Q$ and ${W}\_K$, begin to condense.
• Furthur Stage (epoch 50+) To achieve a lower loss and fit the task more accurately, the model must develop more complex and specialized representations. This requires the parameters to differentiate and move out of the condensed state, as observed for $W^{[1]}$.

Is there a chance that training for longer time beyond the epochs reported leads to a sudden change in the model, somewhat similar to the grokking phenomenon?

This is an insightful question. While we did not observe the specific delayed generalization dynamic that defines grokking, the staged learning process we identify could be related. The transition from a condensed, low-complexity state to a differentiated, high-complexity one might be a foundational mechanism that precedes phenomena like grokking. We see this as a very promising avenue for future investigation.

Response to Question 3:

We confirm

• Eq 10 contains a notation error. The correct term is ​${v}^{\intercal} \bar{{W}}\_V$.
• Proposition 1 contains a notation error. The correct term is $X\_i \in \mathbb{R}^{s \times d\_m}$.

Response to Question 4:

This is an excellent point. We agree that cosine similarity does not capture matrix scale, and we appreciate the opportunity to provide direct empirical evidence.

To demonstrate the different scaling dynamics, we report the Frobenius norm of $W\_Q$ and $W^{[2]}$ at key training epochs below.

As the data shows, the norm of $W\_Q$ is remarkably stable during the first training phase (e.g., from epoch 0 to 13). The subsequent sharp increase in its norm precisely marks the transition into the second learning phase we described. This behavior contrasts sharply with $W^{[2]}$ whose norm grows significantly within the first phase. This empirically confirms our claim that different model components exhibit distinct dynamic behaviors.

We would like to express our sincerest gratitude for your thorough and highly constructive review.

We were very encouraged by your positive assessment of our paper's quality, clarity, significance, and originality. We respectfully ask if you might be willing to reconsider the negative overall evaluation, in light of your positive scores on these individual aspects. We hope that following detailed responses have fully addressed your concerns and now better align the paper's overall contribution with the strengths you identified.

Response to Weaknesses

We sincerely appreciate these structural observations and will implement the following revisions:

$m$ is not defined prior to definition 1.

We will reposition ​​Definition 1​​ to immediately follow Section 3.1.

$W^{[1]}$ and $W^{[2]}$ are being referenced prior to being properly defined and cannot be as easily deduced as $W\_Q$, $W\_K$ and $W_V$.

We will adjust the statement in L37 to “We delineate different training dynamics for parameter matrices in Transformers.” and adjust the statement in section 4.

The placement of the experimental results (Section 4) is inappropriate.

We will relocate experiments to follow Section 5 (Theoretical Analysis) and add new introductory paragraph: "We now empirically validate our theoretical findings regarding stage-dependent dynamics (Section 5). These experiments specifically test two predictions: (1) initial condensation phenomenon, and (2) bifurcation of training dynamics into distinct stages."

Section 5.3 is referencing experiments/empirical evidence that don’t seem to be included in the paper.

We apologize for the lack of experimental evidence and add the following:

Empirical Validation of Assumption 1

Empirical Validation Framework

We quantitatively evaluate both conditions in Assumption 1 using custom-defined satisfaction metrics which reflects how many neurons satisfy the hypothesis:

• Assumption 1.1 satisfied rate = $\frac{|A\_1|}{d\_m}$ where A\_1 = \left \\{ i \in [d\_m] \mid \boldsymbol{W}^{[2]}\_i \boldsymbol{W}\_{\mathbf{v}} \boldsymbol{W}^{[1],i} >0 \right\\}
• Assumption 1.2 satisfied rate = $\frac{|A\_2|}{d\_m^2}$ where A\_2 = \left\\{ (i,j) \in [d\_m] \\times [d\_m] \mid \langle \boldsymbol{W}^{[2]}\_i \boldsymbol{W}^{[1],i}, \boldsymbol{W}^{[2]}\_j \boldsymbol{W}^{[1],j} \rangle > 0 \right\\}

2. Key Results: Rapid Convergence to Full Satisfaction

Quantitative measurements across initial training epochs reveal:

The satisfaction rates start near 50% at initialization, as expected for random vectors. They then rapidly converge, surpassing 95% by 0.1 epochs and reaching a stable 100% by 0.3 epochs. This state of full satisfaction persists throughout the condensation phase we analyze.

Conclusion

This empirical evidence confirms that Assumption 1 is not merely a theoretical convenience but a condition that is reliably and quickly met in practice. This validates the foundational premise of our condensation analysis, directly addressing the reviewer's concern.

The lack of theoretical hypothesis verification and larger model experiments.

We will verify the theoretical hypothesis and provide experimental evidence for a larger transformer.

1.Validation of Dynamics Separation Assumptions

We provide direct empirical validation of Assumption 2 through quantitative measurement of parameter dynamics during training. Our analysis reveals clear separation behavior consistent with the theoretical assumptions.

Empirical Validation Method

We quantify the dynamical separation phenomenon using the relative change rate metric:

$$\text{relative change rate} = \frac{\\|\boldsymbol{W}\_{t+1} - \boldsymbol{W}\_{t}\\|}{\\|\boldsymbol{W}\_{t}\\|}$$

This measures the relative magnitude of parameter updates between training steps.

Key Results: Separation at Epoch 13

Measurements at the critical transition point (epoch 13) demonstrate:

Critical observations:

1. Attention parameters ($\boldsymbol{W}_Q,\boldsymbol{W}_K$) exhibit large update magnitudes $\approx 1e-2$
2. Linear path parameters ($\boldsymbol{W}_V,\boldsymbol{W}^{[1]},\boldsymbol{W}^{[2]}$) show minimal updates ($\le 1e-4$)

This empirical evidence directly supports Assumption 2 which validates the core claim that $\boldsymbol{W}_Q,\boldsymbol{W}_K$ evolve while $\boldsymbol{W}_V,\boldsymbol{W}^{[1]},\boldsymbol{W}^{[2]}$ remain quasi-static.

2. Larger transformer experiments

We confirm the ​​universality of condensation dynamics​​ through new experiments on ​​Wikitext-103​​ (natural language corpus) using a standard 2-layer decoder-only transformer with GeLU activation and residual connections. This demonstrates the phenomenon’s independence from specific data distributions.

Experimental Results

Condensation Rate:

We define the following aggregation index to reflect our results. According to the definition 1 of the article. A neuron is condensed if its alignment with the dominant direction exceeds:

$$\left| \left\langle \frac{\boldsymbol{W}\_k}{\| \boldsymbol{W}\_k \|}, \boldsymbol{v} \right\rangle \right| \ge 0.95$$

The condensation rate is defined as the number of condensed neurons divided by the total number of neurons.

Key Result We plot the condensation rate of the first layer of decoder.

Experimental results show that $\boldsymbol{W}\_V$ $W^{[1]}$ parameters converge quickly (at epoch 0.075), while $\boldsymbol{W}\_Q$ converges significantly later than the former (at epoch 0.15). This shows the applicability of our results on real language tasks.

Response to Question 1

Yes, the anchor function is formulated as a classification task, following common practice in prior works (e.g., [arxiv:2306.00802, 2401.08309]). We use cross-entropy loss to align with standard transformer training and better approximate downstream language tasks.

Response to Question 2

Thank you for pointing this out, and we apologize for the confusion caused by the caption. You are correct that the model consists of a single layer. Our intention was not to present cosine similarities across layers, but rather to show the cosine similarities of different parameter matrices.

We clarify the computation of cosine similarity of the prarameter matrix. Given a parameter matrix $\boldsymbol{W} \in \mathbb{R}^{n \times d}$, we compute the cosine similarity matrix as follows:

• Normalize each row: $\tilde{\boldsymbol{W}}_i = \frac{\boldsymbol{W}_i}{|\boldsymbol{W}_i|_2}$

• Compute the cosine similarity matrix: $\tilde{\boldsymbol{W}} \tilde{\boldsymbol{W}}^\top \in \mathbb{R}^{n \times n}$

This results in a matrix where each entry represents the cosine similarity between a pair of rows in $\boldsymbol{W}$, i.e., between two parameter vectors.

Response to Question 3

Yes, Assumption 2 is verified in our experiments. As shown in our response to Weakness 5, the key-query stunting and criticality conditions are satisfied around epoch 13.

Response to Question 4

Yes, we address this by introducing the concept of effective rank as a soft measure of matrix rank. Specifically, for a parameter matrix $\boldsymbol{M}$ with singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_n$, we define:

$$\text{erank}(\boldsymbol{M}) = \sum_{i=1}^n \frac{\sigma_i}{\sigma_1}$$

This quantity captures how concentrated the spectrum is, with lower values indicating stronger rank collapse. We compute the effective rank of several parameter matrices across different training epochs, as shown below:

We observe a sharp drop in effective rank beginning around epoch 13, which corresponds to the second stage of training. This provides clear evidence that rank collapse emerges during this phase, in line with the theoretical predictions of Theorem 3.

Response to Question 5

We have included additional experiments on the WikiText dataset, which show that the same training stages also emerge in real-world settings.

We deeply appreciate your rigorous feedback, which has helped us significantly strengthen the theoretical grounding of our work.​​ We respectfully ask if you might be willing to reconsider the negative overall evaluation, in light of your positive scores on quality, clarity, significance, and originality. Below are point-by-point responses to your concerns.

Response to Question 1: Practical Probability of Non-Degenerate Initialization

"What is the probability that non-degenerate initialization conditions hold in practice?"

Our Response: We thank the reviewer for this insightful question. Our non-degenerate condition holds with probability 1 under standard initialization schemes (i.e., almost surely). To address this rigorously, we have added a formal proof in Appendix A of our revised manuscript, which we summarize below.

Theoretical Justification (Probability Measure Perspective)

Proof Strategy:
Our proof strategy is to demonstrate that the complementary (failure) event has a probability of zero. The non-degenerate condition, as specified in Definition 3, fails if either of the equalities it seeks to avoid occurs.

Part 1: Proof for Definition 3.1

The first condition for failure is $\\|\boldsymbol{W}^{[2]}(0)\\|\_2 = \\|\boldsymbol{W}\_{\boldsymbol{v}}(0)\\|\_2$. This is equivalent to the event $\\|\boldsymbol{W}^{[2]}(0)\\|\_2^2 - \\|\boldsymbol{W}\_{\boldsymbol{v}}(0)\\|\_2^2 = 0$.

Let's define $D\_1 := \\|\mathbf{W}^{[2]}(0)\\|\_2^2 - \\|\mathbf{W}\_{\mathbf{v}}(0)\\|\_2^2$. The weights in $\boldsymbol{W}^{[2]}(0)$ and $\boldsymbol{W}\_{\boldsymbol{v}}(0)$ are initialized by sampling from Gaussian distributions. As a result, $D\_1$ is a continuous random variable, as it is a sum and difference of squared random variables drawn from continuous distributions. For any continuous random variable, the probability of it taking on a specific value is zero.

Therefore, $\mathbb{P}(D\_1 = 0) = 0$. This means the failure event almost surely does not occur, and consequently, Definition 3.1 is satisfied with probability 1.

Part 2: Proof for Definition 3.2

The second condition (Case 1) for failure, assuming $\\|\boldsymbol{W}^{[2]}(0)\\|\_2 > \\|\boldsymbol{W}\_{\boldsymbol{v}}(0)\\|\_2$ is $\\|\dot{\mathbf{W}}\_{\mathbf{v}}(0)\\|\_2^2 - \\|\dot{\mathbf{W}}^{[2]}(0)\\|\_2^2 - \\|\mathbf{W}^{[2]}(0)\\|\_2^2 D\_1 =0$.

Let's define $D\_2 := \\|\dot{\mathbf{W}}\_{\mathbf{v}}(0)\\|\_2^2 - \\|\dot{\mathbf{W}}^{[2]}(0)\\|\_2^2 - \\|\mathbf{W}^{[2]}(0)\\|\_2^2 D\_1$. The continuous distribution characteristic of standard initialization schemes renders $D\_2$ absolutely continuous. This property necessarily implies $\mathbb{P}(D\_2 = 0) = 0$ almost surely. Analogous reasoning applies to Case 2 ($\\|\mathbf{W}^{[2]}(0)\\|\_2 < \\|\mathbf{W}\_{\mathbf{v}}(0)\\|\_2$).

Response to Question 2: Probability of Condensation Condition (Assumption 1)

"In what probability is the condensation condition (Assumption 1) met? Please provide theoretical evaluation or empirical evidence."

Our Response: Empirical measurements demonstrate ​​Assumption 1 holds with probability 1 during the early condensation phase​​ (≤0.3 epochs), satisfying the preconditions for our theoretical analysis.

Empirical Validation of Assumption 1

1. Empirical Validation Framework

To quantitatively evaluate Assumption 1, we define "satisfaction rates" that measure the proportion of neuron pairs satisfying each of its two conditions during training.

• Assumption 1.1 satisfied rate = $\frac{|A\_1|}{d\_m}$ where A\_1 = \left \\{ i \in [d\_m] \mid \boldsymbol{W}^{[2]}\_i \boldsymbol{W}\_{\mathbf{v}} \boldsymbol{W}^{[1],i} >0 \right\\}
• Assumption 1.2 satisfied rate = $\frac{|A\_2|}{d\_m^2}$ where A\_2 = \left\\{ (i,j) \in [d\_m] \\times [d\_m] \mid \langle \boldsymbol{W}^{[2]}\_i \boldsymbol{W}^{[1],i}, \boldsymbol{W}^{[2]}\_j \boldsymbol{W}^{[1],j} \rangle > 0 \right\\}

2. Key Results: Rapid Convergence to Full Satisfaction

Quantitative measurements across initial training epochs reveal:

The satisfaction rates start near 50% at initialization, as expected for random vectors. They then rapidly converge, surpassing 95% by 0.1 epochs and reaching a stable 100% by 0.3 epochs. This state of full satisfaction persists throughout the condensation phase we analyze.

Conclusion

This empirical evidence confirms that Assumption 1 is not merely a theoretical convenience but a condition that is reliably and quickly met in practice. This validates the foundational premise of our condensation analysis, directly addressing the reviewer's concern.

Response to Question 3: Theoretical Contribution of Dynamics Separation Stage

"The theoretical results in Proposition 3 appear self-contained without needing further proof or repetition."

Our Response: We thank the reviewer for this sharp observation. We respectfully clarify that Proposition 3 is a non-trivial consequence of Assumption 2, not a restatement. Our core contribution is to prove how the state described in the assumption leads to specific, separated gradient behaviors in the full, coupled system.

1. What is Assumed vs. What is Proven

• Assumption 2 (The Precondition): We assume that after the initial condensation stage, the system enters a specific state where:
  • The linear subsystem (involving $\boldsymbol{W}\_V, \boldsymbol{W}^{[1]}, \boldsymbol{W}^{[2]}$) has effectively reached a critical point.
  • The attention score dynamics (involving $\boldsymbol{W}\_Q, \boldsymbol{W}\_K$) remain quasi-static.
• Proposition 3 (The Proven Consequence): We then prove that if the system is in this state, the gradients of the total empirical loss $\mathcal{L}(\boldsymbol{\theta})$ with respect to the different parameter groups will separate and scale differently. Specifically, we prove that $\nabla_{\boldsymbol{W}\_V, \boldsymbol{W}^{[1]}, \boldsymbol{W}^{[2]}} \mathcal{L} \rightarrow \mathcal{O}(\delta^2)$ and $\nabla_{\boldsymbol{W}\_Q, \boldsymbol{W}\_K} \mathcal{L} \rightarrow \mathcal{O}(\delta)$.

2. The Non-Trivial Derivation: From Assumption to Proposition

The key to our proof is the novel decomposition of the loss function, which is motivated by our multi-stage dynamics framework:

$$\mathcal{L}(\boldsymbol{\theta}) \approx \frac{1}{n}\sum\_{i=1}^n \underbrace{\mathcal{L}\_{1,i}(\boldsymbol{\theta})}\_{\text{linear dynamics}} \cdot \underbrace{\mathcal{L}\_{2,i}(\boldsymbol{\theta})}\_{\text{attention dynamics}}$$

This decomposition allows us to analyze the total gradient using the chain rule. The gradient for $\boldsymbol{W}\_V$, for instance, is not assumed to be small. Instead, we derive its structure:

$

\frac{\partial \mathcal{L}}{\partial \boldsymbol{W}_V} &= \frac{1}{n}\sum_{i=1}^n \left( \frac{\partial \mathcal{L}_{1,i}}{\partial \boldsymbol{W}_V} \mathcal{L}_{2,i} + \mathcal{L}_{1,i} \frac{\partial \mathcal{L}_{2,i}}{\partial \boldsymbol{W}_V} \right) \\ &= \frac{1}{n}\sum_{i=1}^n \frac{\partial \mathcal{L}_{1,i}}{\partial \boldsymbol{W}_V} (1 + \mathcal{O}(\delta^2)) + \mathcal{O}(\delta^2) \quad \text{(by Stage 1 condensation)} \\ &= \underbrace{0}_{\text{critical point}} + \mathcal{O}(\delta^2) \quad \text{(via Assumption 2.2)}

$

Conclusion

Our significant theoretical contribution is thus two-fold: we propose a novel dynamics separation framework, and we provide the rigorous proof (Proposition 3) that technically validates it by connecting the assumed system state (Assumption 2) to the concrete, observable gradient dynamics.

Response to Question 4: Validation of Dynamics Separation Assumptions

"Are there theoretical or empirical evidence supporting Assumption 2? No empirical evidence was found in the paper."

Our Response: We provide direct empirical validation of Assumption 2 through quantitative measurement of parameter dynamics during training. Our analysis reveals clear separation behavior consistent with the theoretical assumptions.

Empirical Validation Method

We quantify the dynamical separation phenomenon using the relative change rate metric:

$$\text{relative change rate} = \frac{\\|\boldsymbol{W}\_{t+1} - \boldsymbol{W}\_{t}\\|\_F}{\\|\boldsymbol{W}\_{t}\\|\_F}$$

This measures the relative magnitude of parameter updates between training steps.

Key Results: Separation at Epoch 13

Measurements at the critical transition point (epoch 13) demonstrate:

Critical observations:

1. Attention parameters ($\boldsymbol{W}\_Q,\boldsymbol{W}\_K$) exhibit large update magnitudes $\approx 1e-2$
2. Linear path parameters ($\boldsymbol{W}\_V,\boldsymbol{W}^{[1]},\boldsymbol{W}^{[2]}$) show minimal updates ($\le 1e-4$)

Conclusion

The significant difference in relative change rates provides direct empirical confirmation of the dynamics separation phenomenon postulated in Assumption 2, validating the foundation of our Stage 2 analysis.

Dear Reviewer S1TU,

We hope this message finds you well! Thank you for your insightful review of our paper. As for the concerns about our work in the review, including "whether conditions are true in practice", "theoretical results in Proposition 3", "empirical evidence to support Assumption 2", etc., we have provided very specific and detailed responses. Have these responses resolved your concerns? If you have any further questions about our work or responses during this discussion period, please do not hesitate to contact us. We sincerely look forward to receiving your further comments on our responses.

Once again, thank you for your valuable time and feedback!

Best regards,

Authors