Consensus Is All You Get: The Role of Attention in Transformers

We thank the reviewer for the time devoted to reading our paper and providing feedback.

Why are the results important for the community? Is there any implication for real-world applications?

The paper answers the following question, what is the role of attention? The role is to bring the tokens together, i.e., to achieve consensus. In practice, our results show that transformers cannot be too deep since the information contained in the tokens disappears as they converge to a location that is independent of where the tokens start. Our results also show that a transformer has to be deep enough since, otherwise, the last token (used to predict the next token) is not sufficiently influenced by the tokens that precede it which would lead to a next token prediction that is independent from the query/context. Our results then raise the following question, what is the optimal depth of a transformer? This is a question we are trying to answer.

While experiments are important, I believe that the paper wastes too much space on the experimental side.

As can be seen by the different reviews and our replies, there are different opinions about experiments. We hope to reach a compromise by reducing the space devoted to experiments while providing more informative experiments (e.g., randomization over multiple prompts as suggested by other reviewers).

This novelty is completely lost in the paper, as there is no mention of it in the main paper after the introduction.

We will address the reviewer's concern by using the space freed by reducing the number of experimental plots to succinctly describe the techniques used in each proof and highlight their control theoretic origin.

We thank the reviewer for reading our paper and providing feedback.

While there is no section titled "related work", after each theorem there is a remark titled "closest result available in the literature" where we provide a detailed comparison with the most relevant work. We would be happy to consider any paper the reviewer finds relevant.

The reviewer wrote "The theoretical approaches in this paper give a sense of direction, but the exact goal is not clearly presented. From my perspective, the theories are limited by the provided assumptions and seem to hold only under specific conditions. As a result, this reviewer feels that the experimental support is limited and not clearly aligned with the theory."

In the subsection "contributions of the paper" we wrote "The main contribution of this work is to provide a number of results (...) showing that all tokens converge to a single cluster thereby leading to a collapse of the model." This is the goal of the paper. We are happy to expand the goal's description.

We interpret the second sentence as: the stated assumptions are too strong. Could the reviewer tell us which assumptions are too strong so that we can address them? Please see the reply to reviewer aNPn where we explain how specific assumptions can be relaxed.

All the experiments support the conclusions of the theorems, i.e., they show evidence of consensus. The experiments either satisfy the theory's assumptions or do not. When the assumptions are satisfied, there is perfect agreement with the theory. When they are not, the experiments show that consensus still holds. This does not mean the results are wrong, it means that consensus can be proved under weaker assumptions

Having the definition of E in the appendix was an oversight that will be corrected.

The reviewer wrote "The experiment is designed as a simulation (...) However, this setup does not accurately simulate the behavior of a large language model. It is unclear why the authors did not use larger models (...)"

The experiments in Figure 3, and Figures 5-7 were performed on the GP2-XL model and are not simulations. Since this is a theoretical paper, experiments do not require large models as we do not seek to compare performance metrics. The experiments serve two purposes: 1) illustrate the theorems proved in the paper; 2) show that conclusions hold under weaker assumptions.

The reviewer wrote "This reviewer speculates that the outdated GPT-2 XL, trained on a limited corpus with fixed token lengths, may easily generate meaningless tokens when iteratively fed its own outputs. This may not accurately reflect the model collapse described in the theory."

The experiments do not show GPT-2 XL providing meaningless tokens, they show that all the tokens converge to the same token as predicted by the theory. The function E is only zero when all the tokens are the same. In Figures 3, 5, 6, 7, and 8 we see the function E converging to zero which indicates that all the tokens converge to consensus, i.e., perfect agreement between experiments and theory.

The reviewer wrote "This reviewer believes this paper can be improved by presenting its motivation and goals better. Additionally, scaling up and modernizing the experiments instead of just using GPT2-XL would support the claims effectively (...) If the authors address the concerns, I am willing to raise my score."

We will revise the paper to provide more intuitive explanations for the theoretical results and better describe the objectives. We hope to have convinced the reviewer that: 1) theoretical claims are proved theoretically, not experimentally; 2) there is perfect alignment of theory with experiments. We will also provide more experiments to address the concerns raised by multiple reviewers and would be delighted by a raised score.

Why the continuous concept is necessary for deriving the equation from (4) to (5). We can interpret (4) as a discrete-time dynamical system. Equation (5) is a differential equation, i.e., a continuous-time dynamical system. Continuous-time models enable the use of tools developed in physics, mathematics, and control theory. Corresponding tools for discrete-time either do not exist or are much more difficult to use.

Why do the trained weights converge faster than the random ones? Reviewer aNPn had a similar question, please see the answer we provided.

A traditional element layer normalization in Transformers does not follow the norm-scaling formula described in eq.(2). Eq. (2) has been used in concrete transformers, see the citation in the paper. See also the reply to reviewer aNPn who had a similar question.

Lemma 4.2 does not exist in the reference Geshkovski et al., 2023a. Lemma 4.2 appears in versions 1-3 of the arXiv preprint, but in the latest version is now Lemma 6.4. This will be corrected.

$y_1$ in Eq.(18) should be $y_0$? We will rectify this typo on the final version.

We thank the reviewer for the time devoted to reading our paper and providing feedback.

We were not aware of the referenced papers. All but the second paper discuss training dynamics whereas we study already trained transformers. Hence, they do not seem relevant. The second paper shows that the evolution of tokens along a transformer can be interpreted as an instantiation of gradient descent. We cannot see any direct connection to our work but we will be studying this paper out of intellectual curiosity.

The reviewer states that the contribution is "somewhat too incremental" and supports this opinion with "similar claims have been shown in previous paper" and "still requires some strong condition (...) like U still need to be identity". The latter statement is factually incorrect. Section 4.2 discusses the case where the matrix U is not the identity. The reviewer can find in Assumption 4.4 that we only require U to be symmetric and time-invariant. See also our reply to reviewer aNPn where we explain that the time-invariance assumption can be relaxed.

We would also like to politely disagree with the incremental characterization of our work and with "similar claims have been shown in previous paper". Theorems are written as implications, say a implies b, where a is called the antecedent and b the consequent. Two theorems are not similar because they have the same consequent. The strength of two theorems offering the same consequent is measured by how weak the antecedent is. A weaker antecedent means the implication applies to a larger class of systems and is thus a stronger result. Here is one example, before people could travel by aircraft, the US could be reached from Europe by boat in several weeks. Once air travel began, the US could be reached from Europe in several hours. In both cases the consequent is the same, the US can be reached from Europe, but these scenarios cannot be characterized as being "similar".

The reviewer writes "The authors haven't detailly compared the techniques between them and previous literature to show why they can achieve relaxed conditions." It is not clear to the authors what is meant by a detailed comparison of techniques. An intuitive comparison is provided in the 3rd paragraph of the introduction. Intuitively speaking, some prior work used a mean-field model that resulted in a partial differential equation whose solution was interpreted as a distribution. We used ordinary differential equations that enabled us to use control theoretic techniques such as Input-to-State Stability as well as existing results on consensus on spheres. Existing work that also used ordinary differential equations did not use control theoretic techniques. We would be happy to expand this intuitive comparison in the final version of the paper and, in particular, highlight which control theoretic techniques were used in each proof, if this is what the reviewer has in mind.

We are more than happy to change the paper's title.

We agree with the criticism related to the need to use multiple prompts. We will provide such experiments in the final version.

The matrix V_\eta is defined in the second paragraph of Section 2.2 as the value matrix. Please let us know if additional explanations regarding this matrix are needed.

We could not understand the remark «Seems all the "[0, \infty]" are "[0, \infty["».

We should have made this clear in the paper. We follow Bourbaki's notation (introduced in Éléments de mathématique) and write [a,b[ to denote the interval including the point a and excluding the point b. This is done so that there is no confusion between the ordered pair (a,b) and the interval ]a,b[ that excludes the points a and b. We did find a few instances of sets in the appendix using the notation (a,b) and we will correct them.

Not including the definition of E in the main part of the paper was an oversight of our part. It will be rectified.

Not defining attractivity was an oversight, it will be corrected.

The reviewer asks about the "Key difference in technique". This is the first paper that uses control theoretic techniques such as Input-to-State Stability or draws inspiration from results available in the control literature on consensus on spheres. As we mention in the reply to the last reviewer, we propose to comment on which control theoretic techniques are used in each proof so as to provide deeper insights into our approach. We hope this change addresses the concerns of both reviewers.

We thank the reviewer for the time devoted to reading our paper and providing feedback.

We were not aware of the paper Wu et al. (2024). Upon reading it we found that it confirms the results in our paper since it provides several scenarios of rank degeneration, i.e., consensus. It also provides an example where consensus does not occur but this example requires the query and key matrices to be constant and equal to zero which does not happen in practice.

We will be happy to add a conclusion/discussion section as suggested.

Layer normalization

Thank you for this question, we should have included a discussion about this in the paper. At an intuitive level, any normalization technique has the objective of restricting the tokens to a compact space, hence the results should not depend on how the normalization is done. At the technical level, we can show that layer normalization (removing the mean, dividing by standard deviation, multiplying by a learned scale, and adding a shift) corresponds to projecting on a suitably translated sphere when the learned scale parameters are all equal and non-zero. Hence, our results apply to this case. We suspect the same holds for arbitrary scale parameters but we need to perform a more careful analysis to ensure the topology/geometry of the resulting compact space is homeomorphic/diffeomorphic to that of a sphere.

Matrix U time-varying

Our proof technique is based on projecting the dynamics along the eigenvector of U corresponding to the largest eigenvalue to obtain a scalar differential equation that is easier to analyze. When U is time varying, the eigenvectors of U are time varying. This has two consequences: 1) the consensus point is now time-varying; 2) the scalar ODE has one additional term coming from the time derivative of the eigenvector (this derivative is zero in the time-invariant case). Provided the eigenvector changes slowly, all the tokens will asymptotically converge to the time-varying eigenvector and consensus is still reached. We are happy to discuss this extension in the final version if the reviewer finds it useful.

Theorem 3.2, tokens starting outside an hemisphere

The conclusions of Theorem 3.2 still hold when the tokens don't start in an hemisphere but enter an hemisphere at some future time. We proved that hemispheres are invariant (once the tokens enter one, they cannot leave) and thus the conclusion follows. In our experiments we always observed consensus, independently of the initial condition.

Experiments

Defining the function E in the appendix was an oversight that we will correct. We will also be happy to average the results and provide confidence intervals in the final version.

Collapse is less prominent for random weights/Removing periodicity seems to reduce collapse.

This is an interesting question. Our educated guess is that convergence is exponential when the matrices are constant (this is supported, under additional assumptions, by Theorem 6.3 in version 4 of https://arxiv.org/abs/2312.10794). One may speculate that the rate of convergence is related to the time derivative of the matrices. Constant matrices, zero derivative, faster convergence, random matrices, highest derivative, slowest convergence. The periodic case seems to lie in between the constant and random cases but we do not have a good explanation for this.