On the Emergence of Linear Analogies in Word Embeddings

Referee imGd

We thank the referee for their comments and questions.

Weaknesses

1. Polysemous words, hierarchy, etc are not so neatly explained by this model (but this is a common limitation in prior work as well).

Polysemous words could be studied theoretically in our framework by ‘merging’ pairs of semantically distinct words (i,j) into a polysemous word k, by summing their corresponding rows and columns in the PMI matrix, such that P(ell)= P(i)+P(j) and P(w, ell)= P(w,i)+P(w,j) for any word w. It deserves a study of its own, but we expect our results - which are extremely robust to perturbations - to hold if the fraction of polysemous words is small.

Modeling the presence of hierarchical attributes is a nice problem for the future; we hope our contribution will stimulate future studies in that direction.

2. The simple co-occurence based models mentioned (glove, w2v, etc) are rather old by today’s standards.

See below.

Questions

1. What can this tell us about popular contemporary LLMs?

See reply to referee Zp5k to weaknesses 4+9: linear analogies are also observed in LLMs. They are argued to be functionally important, as they can mechanistically explain LLM fact retrieval in some cases (Merullo et al., ACL 2024). How they emerge is not fully understood, and our work is a first step in seeking to build this understanding. We will discuss this literature in the introduction.

2. Does this analysis give us any insight into how we can develop better word embeddings or improve LLM design today?

It was shown with SENSEBERT that if a machine (BERT in that case) is trained by predicting the meaning (a linguistic notion; e.g. cat belongs to the “animal” category) of a word and not only the word itself, performance is improved (Levine et al., ACL 2020). It would be very interesting to check if LLMs perform better if instead of predicting the next token, they are trained to also predict the attributes of the associated word (see below for a practical proxy to these attributes).

3. Is it possible to do some kind of binary latent factor analysis to discover the actual attributes that might underpin the existing words?

As a proxy for attributes, we could simply consider the projections of word embeddings into the top eigenvectors of the PMI matrix. As discussed in the text and above, attributes are somewhat mixed in these eigenvectors. However, linear combinations of attributes may be enough for practical purposes, such as the extension of SENSEBERT discussed just above. We will discuss these interesting points in the conclusion.

Referee vqG4

We thank the referee for their useful comments.

Weaknesses

1. Simplified Assumptions: The model makes strong simplifying assumptions, including binary semantic attributes and independence of attribute effects.

See response to referee Zp5k: we had already provided an extension to tertiary semantic attributes in the appendix. Extension to n-valued attributes is straightforward. In response to both referees, we have extended our theory to the case of correlated attributes, see a sketch of the formalism above, which will appear in Appendix in the final paper.

2. Generalizability and Realism: The assumption of all possible combinations of attributes existing in the vocabulary is unrealistic.

(i) Our model is robust to sparsification (i.e. subsampling the vocabulary). In that case, all possible combinations of attributes do not exist in the vocabulary. Thus, we have already tested that this assumption can be relaxed. (ii) Clearly, some attributes do not enter in the semantic meaning of a given word. For example, a word can be neither masculine nor feminine. This can be readily modeled in our framework by considering tertiary attributes with a “neutral” category, each time a given attribute does not apply to a given word. We will add a discussion on these points.

3. Potential Degeneracies in Parameter Choices: The model’s predictive accuracy and analogy results are sensitive to the choice and distribution of parameters (e.g., semantic strength distributions ).

The notion that there exists a distribution of semantic strength is not a bug in our opinion, but a nice feature of our approach. It explains why distinct linear analogies do not appear at the same dimension of embedding, which is not captured by previous models.

Referee Zp5k

We thank the referee for their comments and criticism, they are useful to clarify the manuscript.

Weaknesses:

We respond to the distinct points raised by the referee below:

1. Summary of referee comment: The mathematical notation and statements used in the paper lack clarity, particularly in sections 4, 5, and 6.

We will improve the flow and clarity, particularly in the highlighted sections. The $\delta$ refers to the standard Kronecker delta, so $\delta_{a_i^{(k)},1}$ evaluates to 1 whenever $a_i^{(k)}$ is 1, and is otherwise zero. We note that the appendix also offers a more pedagogical treatment of several of the calculations, a review of Kroenecker products, and an explicit worked example for $d=2$. We will draw the reader’s attention to this earlier in the manuscript.

2+6+7. Summary of referee comment: The model makes simplifying assumptions: (i) binary variables don’t capture multi-valued (or continuous) semantics, (ii) there may be correlations between semantic attributes.

Points 2+6 criticize the assumption of binary variables. Please note that our model can be generalized to multi-valued semantics. In fact, we had already performed an extension to tertiary variables in the appendix, and our findings carry through largely unchanged. We will highlight this result earlier in the paper. The extension to n-valued variables is straightforward, and may represent an approach to the continuous limit. We hope our contribution inspires such lines of inquiry.

Point 2 also points to the fact that semantic variables might be correlated. This is an excellent comment, which led us to generalize our model to correlated attributes. We briefly sketch this generalization below in answering Q1.

3. Validation on a single test set (Wikipedia), and limitations of applicability more broadly to other languages or corpora.

The presence of linear analogies our theory seeks to explain has been demonstrated in numerous other languages, including more morphologically complex ones such as Finnish (Venekoski and Vankka, ACL 2017) and Chinese (Su and Lee, ACL 2017) when using the appropriate scale of morphologically aware tokenization, such as FastText (Bojanowski et al., ACL 2017). We will make sure to cite these references in our introduction, to highlight the generality of the problem we are addressing.

The objective of our contribution is to generate a robust, minimal theoretical framework that explains several key aspects of word embeddings. Other work in the realm of theory building (e.g. Arora et al. TACL 2016) also uses a single dataset and language (English Wikipedia). Comparing quantitatively multiple languages is certainly an interesting endeavour, but it presents a work on its own. It should not obscure the presentation of the new formalism introduced here in its purest form.

4+9. Summary of referee comments: This work addresses primarily a feature of context-free word-embedding models. To what extent does this generalize to LLMs?

The semantic content of language exhibits both elements that are independent of context, and others that depend on it (HP Grice 1975, Pustejovsky, J., 1995). Strictly speaking, our theory is making statements about the former, since it studies context-free embeddings like word2vec. Concerning LLMs, they must necessarily also capture this context-free element in their internal representations, and so our work has some bearing on them. Bommasani et al. 2020 showed that such context-free embeddings can be obtained by pooling the representation of words across many contexts. This approach could be used to test our views in LLMs, as we will discuss in the conclusion.

The work of Hernandez et al. 2024 ICML is also very pertinent: The authors find that the internal representations of LLMs obey linear analogy (up to a context-dependent linear projection), satisfying relations like $W_r (\text{Miles Davis} - \text{Billy Joel}) = \text{Trumpet} - \text{Piano}$, where the relational matrix $W_r$ is induced by the words “plays the instrument” (and thus depend on context). Thus, linear analogy structure survives in the inner representations of LLMs in a contextual way. Other work by Merullo et al., ACL 2024, indicate that these linear analogies are important for fact retrieval by LLMs. Understanding the emergence of contextual linear analogies remains a theoretical challenge; the ideas presented in our work are possible handles to tackle it. We will discuss these works in conclusion.

5+8. Limited comparison between our work and existing methods.

The limited comparison with existing methods that the reviewer points out reflects the novelty of our work: we are making predictions where other approaches did not, including on how the quality of linear analogies depends on the dimension of the embedding considered. To our knowledge, there are no alternative predictions in the literature on that. We simply emphasize in the text that some key assumptions of previous methods do not hold (such as the isotropy of the PMI matrix).

Questions:

1. Could correlated semantic attributes bias the model’s predictions or its spectral structure?

We thank the referee for this interesting point. It has prompted us to consider an extension to the model that includes correlated attributes, which we will include in the appendix of the paper. The PMI matrix is robust to correlations.

In brief, in this extension equation 3 takes the modified form: $P(i,j) = P(i)P(j) \left( \prod_k P^{(k)}(i,j) \right) \left( \prod_{k, k’} P^{(k, k’)}(i,j) \right)$ where the $P^{(k,k’)}(i,j) = 1 + s_{k,k’} \alpha_i^{(k)}\alpha_j^{(k’)}$ introduces correlation between attributes $k$ and $k’$ of strength $s_{k,k’}$. This formulation approximately preserves normalization when the $s_{k,k’}$. The subsequent analysis of the PMI matrix carries through in the same manner, resulting in an identical factorization with equation 12, except that the matrix $D$ is no longer diagonal. All of the following results (1,2,3,4) hold, along with the emergence of linear analogies at dimension $d$. The main difference is that the eigenvectors of the PMI matrix in this case do not map one-to-one onto semantic attributes, but this has no impact on the linear analogies.

2. Has the model been evaluated beyond English corpora (e.g., on morphologically rich or logographic languages such as Chinese or Finnish)?

See our response to weakness 3 above.

3. In the era of contextualized word representations (e.g., BERT, GPT), what is the broader significance of studying static word embeddings?

See our response to weakness 4+9 above.

How sensitive is the model to the choice of the PMI transformation (e.g., using shifted PMI, SVD truncation, or subsampling strategies)?

(i) Shifted PMI adds a constant offset to the PMI matrix. This is a rank-1 perturbation to the PMI matrix and therefore has minimal effects on its spectral properties, as we checked. (ii) The choice of the embedding space dimension $K$ already implements SVD truncation. (iii) Regarding subsampling: we test a subsampled (sparsified) vocabulary in figure 2, panel f in the model. We have checked that subsampling by limiting the vocabulary also does not affect our results on Wikipedia, as we will mention in the text.

4. Could you provide insights or explanations regarding the five weaknesses previously identified?

See above.

5. The model assumes attribute independence. Could you provide examples illustrating that such independence is satisfied within the Wikipedia dataset?

As we highlight in our response to question 2 and will add to the paper, our theoretical model is robust to including a correlation between attributes.