Mastering Syntax, Unlocking Semantics: A Mathematically Provable Two-stage Learning Process in Transformers

Thank you sincerely for your thoughtful comments! We understand that your primary concern lies in aligning our theoretical modeling with practical syntax-semantic analysis. We sincerely regret any confusion caused by our introduction of the syntax, semantics or easy-to-fit component $\mathcal{P}$, hard-to-fit component $\mathcal{Q}$. We respectfully argue that these theoretical modelings are indeed motivated by experimental findings and are estabilished to provide a rigorous mathematically analysis for this stage-wise learning phenomenon. In addition to the current empirical validations, as you suggested (e.g., syntax benchmarks like BLiMP), we have designed more experiments with specific quantitative metrics about the syntax and semantics to further support our theory.

We strongly encourage to read General Response for detailed clarification of our motivation regarding the statements on syntax and semantics, as well as further experimental evidence (primarily addressing your concerns in Weaknesses, Question 1 and 3)! Below, we do our best to address your additional questions adequately.

Q2: Have you considered alternative hypotheses regarding what the easy/hard-to-learn components of examples are, besides syntax and semantics? It is important to rule these out before concluding that the model is learning syntax and semantics, specifically, in that order.

A: Thanks for your question! As you mentioned, there are some empirical studies which have already examined various aspects of language. This provides a strong foundation to support that syntax and semantics are representative and well-understood components of linguistic features. Moreover, we would like to emphasize that the core contribution of our research lies in offering a theoretical understanding that explicitly characterizes the model's learning dynamics across features of differing complexity. Syntax and semantics serve as proxies for easy-to-fit (syntax) and hard-to-fit (semantics) features. The focus is not to claim these are the only such features but rather to demonstrate the general mechanism of how the model learns features with varying difficulty levels.

In addition, we encourage to read the "Re-clarification for our motivation and theoretical establishments" part in General Response, which maybe crucial for your concerns!

Q4: You cite (Vaswani, 2017) for the transformer model, but shouldn't this be (Vaswani et al., 2017)?

A: Thank you for pointing out this citation mistake. We will make the correction in the formal version!

Q5: How do you train the model using which Figure 5 was made?

A: Thanks for your question! Our transformer model is based on the GPT-2 architecture with 12 layers, 12 attention heads, and 768-dimensional embeddings. Limited by computation resources and training data, we use "AutoModelForCausalLM.from_pretrained("gpt2")" to load a pretrained version from the HuggingFace model hub, not training from scratch. In order to observe the optimization process of transformers, we train the model for 200 epochs using the AdamW optimizer with a batch size of 32, with Counterfact and HotpotQA dataset. All experiments are conducted using a single 24GB NVIDIA GeForce RTX 3090.

Once again, we sincerely thank you for the detailed review and constructive feedback! We hope that our clarifications satisfactorily address your concerns and we welcome any further discussions that may be helpful to the evaluation.

In fact, your proof from line 2241 to line 2243 is wrong. You want to show:

$\max\left(\log(2) - \frac{1}{2} ( \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W), \log(2) + \frac{1}{2}( \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W) \right) \le \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W$

It would be much clear to see the problem if simply denote $S = \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W$ and the above process can be rewritten as:

$$\max\left(\log(2) - \frac{1}{2} S,\ \log(2) + \frac{1}{2} S \right) \le S$$

which is WRONG based on your current conditions (especially, when $d$ goes infty where S goes to zero, this inequality is wrong). It might be correct for small $d$ where $S$ has a lower bound of $cons. \times \log(2)$, but it will make your argument in Theorem 2 become wrong, since you can not make $d$ sufficiently large.

In a summary, from line 2238 to line 2243, you first find a larger upper bound (the max thing) and make a mistake on the following inequality which gives your current upper bound. The intermediate step is wrong, and that makes your argument for theorem 2 incorrect.

Thank you for your feedback, before I go to the rest of the feedback, I still want you to further clarify this process, since I think it is wrong. To avoid any misunderstanding, let me explain my question in detail:

From lines 2238 to 2243, you want to derive the key components to get the upper bound for $K_{t_1}^1(\overline{W}_{t_1})$.

Here you first derive the upper bound for $K_{t}^1(W^*)$:

$$K_{t}^1(W^*) \le \max(\log(1+e^{-x}), \log(1+e^{x})) \le \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W, \ \text{where}\ x=\epsilon_{W,1}+\sqrt{d}/L\log(d) \epsilon_W$$

(Pay Attention, the middle part has a lower bound $log(2)$!!! you can more clearly find that in line 2241)

Later based on this property, you goes to line 2288 and have:

$K_{t_1}^1(\overline{W}_{t_1})\le K_{t_1}^1(W^*) + C$

and then based on the result from line 2238 to line 2243, you have:

$$K_{t_1}^1(W^*) + C \le \max(\log(1+e^{-x}), \log(1+e^{x})) + C \le \epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W +\frac{1}{\sqrt{\log(d)}}$$

which gives you the upper bound for $K_{t_1}^1(\overline{W}_{t_1})$.

Based on this, you write your argument in Line 394 is "upper bounded by an o(1) term which converges to zero as the dimension d goes to infinity" where the upper bound refer to $\epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W +\frac{1}{\sqrt{\log(d)}}$.

However, as I explained above, the upper bound already has a lower bound which is shown in your process "line 2238 to line 2243"

$$\epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W +\frac{1}{\sqrt{\log(d)}} \ge \max(\log(1+e^{-x}), \log(1+e^{x})) +\frac{1}{\sqrt{\log(d)}} \ge \log(2) +\frac{1}{\sqrt{\log(d)}}$$

and as $d$ goes to infty, this cannot go to zero!!!

More precisely, this implies the upper bound of Theorem 2 aligns with Theorem 1 on the same constant scale, which cannot converge to zero as the dimension $d$ goes to infinity. Can you provide more explanation on this. Thank you!

Moreover, I will check the rest of the comprehensive feedback in detail, and generate the feedback more carefully for them later. Thank you for your detailed feedback.

Dear Authors,

I acknowledge the significant effort you have invested in your manuscript, and I truly appreciate the dedication reflected in your work. However, I would like to ask for clarification on whether you are able to address the critical issue in your proof I have outlined above.

If the issue cannot be resolved, I may need to reject your work, as this concern pertains to what I understand to be the central contribution and result of your paper.

Best regards,

Reviewer 7oPM

Thank you so much for your further clarification. Meanwhile, I believe you should first modify your manuscript, since if you change your bound, the "theorem 2" and "messages behind theorem 2" should be rewritten and some attractive result becomes invalid, such as "the network $h$ learns elementary knowledge like $\mathcal{P}$, marking the so-called elementary stage". I think all the reviewer should be notified by this key modification.

Moreover, if you wish to use a Taylor series expansion at 1 to derive the upper bound, similar to the process outlined in Theorem 1, it should also be possible to determine a lower bound for $K_t^1(W^*)$. Based on your recent feedback, I believe the lower bound would be:

$$K_t^1(W^*) = \frac{1}{N}\sum_{n=1}^N \log(1+\exp(-y_L^n f_t^1)) \gtrsim \log(1+e^{-1}) - (\epsilon_{W,1} + \frac{\sqrt{d}\log(d)}{L} \epsilon_W).$$

as $d$ goes to infty, this can give you a lower bound for $K_t^1(W^*)$, which is $\log(1+e^{-1})$. Similar to your argument in Theorem 1, network $h$ shouldn't be assumed to effectively learn the elementary knowledge $\mathcal{P}$. I was wondering how would you interpret this?

Q4: In the analysis, one key difference between the two learning stage is the constant learning rate and the annealing learning rate. Is that crucial to the proof and do you observe the same effects in practice?

A: Thanks for your question! Annealing the learning rate is indeed crucial to enable the model to acquire specialized knowledge in the later stage without undermining the elementary knowledge. We would like to analysis the learning rate annealing strategy theoretically and practically!

• Theoretically, in deep learning models, the complexity of knowledge required varies at different stages. A larger learning rate enables the model to quickly learn the simple characteristics of the data distribution. And then a smaller learning rate helps fine-tune the model to capture fine-grained specialized features without disrupting elementary knowledge. This stage-wise learning has been evaluated in our theory and empirical validations! Furthermore, a high learning rate during the initial phase helps the model escape shallow suboptimal solutions, while a smaller learning rate in later phases allows for more precise parameter adjustments, increasing the likelihood of finding better solutions.

• Practically, a large learning rate in later stages would result in excessive parameter updates, disrupting learned patterns. Annealing mitigates this issue, making the training process smoother. In addition, learning rate annealing works well with various optimizers (e.g., Adam, SGD) and delivers significant improvements across different tasks, including image classification, natural language processing, and reinforcement learning.

Q5: L2 normalization is hardly used in the actual training of Transformers. How do you view this difference between the proof and the practice?

A: Thanks for your question! We acknowledge that there is a certain gap between theory and practice, including the use of a simplied transformer architecture and L2 normalization. In theoretical analysis, L2 normalization plays a crucial role as it limits the unbounded growth of parameters, reduces model complexity and enhances the controllability of finite-time convergence analysis.

Moreover, in the practical training of transformers, L2 regularization (typically implemented as the 'weight decay' parameter in optimizers like AdamW) remains a commonly used technique. Its value is generally set within a relatively low range (such as $10^{-5}$ to $10^{-2}$), aligning with our assumptions on hyperparameters, where $1/\lambda = \mathcal{O}(\sqrt{\log d})$, as stated in Assumption 1.

In total, incorporating L2 normalization provides significant benefits for theoretical analysis, while its small magnitude in practical training ensures minimal divergence between our theory and real-world implementations.

Once again, we sincerely thank you for the detailed review and constructive feedback! We hope that our clarifications satisfactorily address your concerns and we welcome any further discussions that may be helpful to the evaluation.

Thank you sincerely for your thoughtful comments! We are delighted that you found our paper provides rigorous proof and insightful spectral characteristics analysis. To further address your concerns regarding the transformer-specific theory, we have made more clarifications that our theoretical modeling and proof are both specifically tailored to transformers. We strongly encourage to read General Response for further clarification of our motivation regarding syntax and semantics, as well as additional experiments! Below, we do our best to address your questions adequately.

Q1: The difference between syntax and semantics in this work is basically one is easier to learn than the other as a feature [...] Right now, even though the proof is done on Transformers, it really feels like a general conclusion that should be held on a much broader class of models.

A: Thanks for your question! We would like to make more clarifications that our theoretical modeling and proof are specifically tailored to transformers. In more detail:

• Training Prompt Structure. As suggested by the general ICL regime, for the $n$-th prompt, input samples $x_1^n, \cdots, x_{L-1}^n$ and query $x_L^n$ are drawn randomly and independently from the same data distribution. The input-label pairs are stacked to form a training prompt $P^n=\left(x_1^n, y_1^n, \cdots, x_{L-1}^n, y_{L-1}^n, x_L^n\right)$, with prompt length $L$. In summary, we explore the training dynamics of transformers, considering training them to hold ICL abilities using $N$ training prompts as introduced above. This prompt structure is specifically designed to align with the properties of attention layers in transformers, yet is largely irrelevant to other model architectures.
• Proof Specific to Transformers. We investigate how model weights are optimized and updated, as well as how the training loss evolves during the training process. Theoretically, we primarily adopt the signal-noise decomposition technique to analyze the characteristics of differences in activations and network output under various activation and weight schemes (such as using noise weight to compute activation and signal weight to compute attention score, which is a crucial aspect). Further details are provided in the Proof sketches in Appendix Remark 6-9. In summary, the stage-wise training dynamics and the signal-noise decomposition technique are highly inspiring for other models, yet variations in training data, activation modes and model architectures pose significant challenges in characterizing the changes in activations and model outputs, and further in the final finite-time convergence results.

Q2, Q3: I would also suggest shifting the focus of this paper away from the syntax/semantics contrast. The syntax/semantics aspect is more suitable to be one of the additional findings in the empirical verification part [...] The empirical verification part of this work should be more detailed. Right now, there are no quantitative metrics about the syntax or semantics accuracy after each stage.

A: Thank you for suggestion! We sincerely regret any confusion caused by our introduction of the syntax, semantics or easy-to-fit component $\mathcal{P}$, hard-to-fit component $\mathcal{Q}$. We respectfully argue that these theoretical modelings are indeed motivated by experimental findings and are established to provide a rigorous mathematically analysis for this stage-wise learning phenomenon. In addition to the current empirical validations, we have designed more experiments with specific quantitative metrics about the syntax and semantics to further support our theory. We strongly encourage to read General Response for further clarification!

Thank you sincerely for your support and insightful comments! We strongly encourage to read the General Response first and return here for further clarifications! Below, we do our best to address your questions adequately.

Q1: The generative process description for the data is somewhat unclear. Based on the current explanation, it seems that the Bayes optimal classifier might not need to rely on semantic features; syntactic features appear sufficient to solve the task in an asymptotic setup [...] Figure 3 and the reference to Li et al. (2019) suggest that my interpretation is incorrect. E.g., in Li et al there’s a stochastic choice in generating features for each example (would 'easy' or 'hard' for this case)., but it is not what is happening here. This needs to be explained better.

A: Thank you for your thoughtful consideration! As you mentioned, it is indeed natural to further consider the existence of stochastic noise in generating samples. This implies that for certain samples, the prediction task depends solely on features from $x_1$ (aligning with your statement that "the Bayes optimal classifier might not need to rely on semantic features,", yet this case is beyond our current scope). For other samples, the prediction task depends solely on features from $x_2$. The remaining cases, which we really focus on, are those where features from both $x_1$ and $x_2$ are simultaneously present.

Our theoretical modeling of the data structure, the case where both $x_1$ and $x_2$ are present in a sample, better benefits our goal of exploring how transformers acquire different types of knowledge during the dynamics of training. Specifically, completing the composite nonlinear classification task requires the model to grasp knowledge related to both components $\mathcal{P}$ and $\mathcal{Q}$. This further accelerates our conclusion: when features of varying difficulty coexist in the training data, the model exhibits a clear preference in learning, showing a stage-wise learning process for easier and harder features.

We sincerely hope the above analysis satisfactorily address your concerns!

Q2: I’m unclear on the rationale for assuming a block-diagonal structure (Eq. 2). In realistic settings, one wouldn't typically know which features are hard or easy in advance.

A: Thanks for your question! In deep learning models, the actual structure of weights and the interactions between features are highly complex, making it nearly impossible to directly analyze the characteristics of a fully developed practical model. The assumption of block-diagonal structure is not a strict description of the actual training process but rather a theoretical abstraction used to study whether the model can effectively decompose tasks within different feature subspaces.

Referring to feature learning theory, we indeed do not need to know in advance which features are simple and which are complex. The distinction between $X_1$ and $X_2$ lies in theoretical abstraction: for a $2d$-dimensional feature space, simple feature dimensions are grouped into $X_1$, and complex ones are grouped into $X_2$. After such abstraction and reordering, the features can theoretically form a block structure (where features exhibit a certain degree of independence in the two subspaces). Correspondingly, we assume the weight matrix is block-diagonal, which allows us to more clearly observe how the network learns the distributions of feature subspaces $X_1$ and $X_2$ through distinct weights $W$ and $V$, respectively. This makes theoretical analysis feasible and helps us better understand the behavior of the model in separating feature learning processes.

In total, we model the learning process of different types of features based on assumptions with certain practical foundations, and we acknowledge that this simplification represents a general limitation for the theoretical analysis of weight optimization. Meanwhile, it is indeed a promising direction to explore more practical weight structures, and we hope to provide theoretical techniques and insights to inspire future work!

Q5: The decision to link optimization parameters (such as the learning rate in the second stage) to the data generation procedure seems unrealistic. Is the idea that this selection approximates parameters typically chosen through hyperparameter tuning? This paper demonstrates that specific parameters enable the model to acquire hard-to-learn knowledge in the later stage without undermining the easy-to-learn knowledge, but it doesn’t show that this outcome holds across a wide range of parameters or those commonly used in practice.

A: Thanks for your question! We acknowledge that our theoretical results are established on a relatively restricted setting (with annealing the learning rate and some hyperparameter order requirements), yet this is required for rigorous mathematical proof.

(1) We would like to point that our theory aligns well with empirical observations in practice. For example, models tend to learn easy-to-fit features (such as syntactic information) in the early stages of training, while hard-to-fit features (such as semantic information) are gradually learned in later stages. This dynamic is consistent with actual training processes and empirical intuition, which validates the relevance of our theoretical analysis.

(2) Furthermore, our research provides new insights into the model’s learning process beyond the rigorous stage-wise learning theory. The choices of learning rate and hyperparameters suggest that

• Learning rate scheduling: Use dynamic adjustment strategies, such as a larger initial learning rate for the learning of easy-to-fit features and then a smaller learning rate for hard-to-fit features.
• Initialization: Use stable small-variance initialization schemes based on the data dimension.
• Prompt optimization: Adjust the input sequence length, especially in high-dimensional scenarios where sufficient context is necessary.

In summary, while our work involves certain assumptions, its core value lies in offering a theoretical perspective on understanding the dynamics of model learning. It also provides a theoretical basis for optimizing practical strategies, such as hyperparameter tuning and learning rate scheduling.

Q6, Q7: I would discourage the authors from using terms semantics and syntax as these seem to be misleading [...] More broadly, it is unclear to me if the two stages are results of the explicit choices in defining the generative process and the stage of learning, or would emerge under more broader range of settings.

A: Thank you for suggestion! We sincerely regret that there is some confusion for the introduction about syntax, semantics or easy-to-fit component $\mathcal{P}$, hard-to-fit component $\mathcal{Q}$. We respectfully argue that these theoretical establishments are indeed insightful and reasonable, rather than purely constructive. We strongly encourage to read General Response for further clarification!

Once again, we sincerely thank you for the detailed review and constructive feedback! We hope that our clarifications satisfactorily address your concerns and we welcome any further discussions that may be helpful to the evaluation.

Regarding (b): More experiments with quantitive metrics of syntax and semantics. In our current experiments, due to the selection of datasets, the syntactic errors are limited to tokens like "of", "the", "The" and "a", etc. Thus, we only need to check if these words appear in the predicted answers and manually determine whether they comply with the syntactic rules. It is not necessary to check every sample, as this is difficult to achieve on large-scale datasets. Furthermore, as suggested by reviewers, the metric to evaluate syntax and semantics, indeed motivates us to conduct new experiments! We found that there are some syntax benchmarks like BLiMP, which is generally used to detect syntax in the model's output. Therefore, we are considering training the model simultaneously on multiple syntax datasets (e.g., BLiMP) and semantic datasets (e.g., SICK) of similar scale, and observing whether the model shows stage-wise preferences for different tasks. In this case, we can use the individual average loss on the syntax dataset and semantic dataset as metrics to evaluate the model's capabilities, achieving a human-free evaluation.

Once again, we sincerely thank all the reviewers for the detailed review and constructive feedback! We hope that our clarifications satisfactorily address the concerns and we welcome any further discussions that may be helpful to the evaluation.