A Single Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces

We sincerely thank the reviewer for their time, effort, and thoughtful feedback on our work. The detailed comments and constructive suggestions have provided valuable insights, helping us to clarify and improve the presentation of our contributions. Below, we address each point raised to ensure our responses are as clear and comprehensive as possible.

Much of the confusion in the feedback appears to stem from a misunderstanding of the term Semantic Field Subspace. To ensure a clearer understanding, we recommend that the reviewer start with our response to this comment. Beginning with this response will provide essential context, addressing several misconceptions and clarifying the intended meaning.

Marjor Comments

4. From what I can gather, each "Semantic Field Subspace" is simply a set of embeddings that the SAFARI algorithm found to be particularly semantically coherent. If this is the case, the name "Semantic Field Subspace" is somewhat confusing as it suggests a relation with fields and vector spaces that is not evident in the paper. I suggest the authors consider different nomenclature. I have similar concerns about the "Semantic Field" name.

• Thank you for bringing this to our attention. We appreciate your feedback and agree that we should have provided a more thorough explanation in the paper.
• "Field" in "Semantic Filed Subspace" and "Semantic Field" is not the mathmatical term "field" in math or physics. "Semantic Field" is originally a term in linguistics, which refers to a lexical set of words grouped semantically, just like the set of embeddings that SAFARI found to be particularly semantically coherent as the reviewer mentioned.
• The term "Subspace" here is indeed used in the mathematical sense, consistent with the reviewer's understanding. However, SAFARI is not limited to discovering sets of semantically coherent embeddings. In Line 9 of the SAFARI Algorithm (Algorithm 1), we perform SVD on the aforementioned set of embeddings to construct a subspace (Algorithm 2) to compute Semantic Shift. The input for computing Semantic Shift is the subspace itself—specifically, the row space and singular values derived from SVD. Therefore, the subspace is a fundamental component of our algorithm.
• The Semantic Field Subspace is not merely a set of vectors but an actual subspace within the embedding vector space. This subspace generalizes the semantic meaning beyond the individual vectors, as evidenced by our experiments, which demonstrate that the subspace itself encapsulates the semantics effectively.

1. On line 160, the authors define the "interpretable semantics" of an embedding vector $f_{int}(v)$. Their definition is based on the previously defined "close neighborhood" of said embedding vector, $\mathcal{N}(v)$. In particular, they define $f_{int}(v)$ as any subset of $\mathcal{N}(v)$. This implies that there are several "interpretable semantics" for each embedding vector. Furthermore, their notion of "interpretable semantics" is simply the intersection between the semantics of close neighborhood embeddings. It's unclear how this is "interpretable."

• Firstly, we would like to clarify that $f_{int}(v)$ is not $*any*$ subset of $\mathcal{N}(v)$. Selecting arbitrary elements from $\mathcal{N}(v)$ is unlikely to yield a clear meaning.
• Regard of what is interpretable, the reviewer is correct in stating "there are several 'interpretable semantics' for each embedding vector." And we are unsure about which one should be chosen to 'interpret' the embedding vector.
• We regard the process of refining meanings as interpretation. As intersections are performed, the semantics become increasingly refined and clearer, ultimately leading to unambiguous semantics. Of course, if we continue performing intersections, we would eventually reach an empty set.
• Thus, it is evident that that we cannot choose other $v$s randomly (or any subset of $\mathcal{N}(v)$). This is why we employ hierarchical clustering in the algorithm. It enables us to perform intersections with other semantically similar embedding vectors in a structured manner.
• An example of "Apple" is illustrated in Figure 1. Initially, there may be several interpretable semantics for "Apple," making its meaning ambiguous—or polysemiotic. As more neighbors are added (corresponding to the hierarchical clustering in our algorithm), the semantics become increasingly refined and clearer. Gradually, it becomes possible to determine whether this "Apple" refers to the fruit or the tech company. This progression from ambiguity to clarity is what we regarded as interpretation.
• In the revised version of the paper, we will polish this part in the main paper.

• Before addressing specific comments, we would like to clarify a potential misunderstanding related to mathematical notation that may have arisen due to differing conventions or our presentation. It appears that the reviewer has interpreted several definitions or terms in our paper with the prefix "any", which was not our intention.
• Specifically, if the symbol $\forall$ does not appear in a definition or term, it should not be interpreted with the prefix "any" or "for all". For instance, $\mathbb{S}$ is not intended to represent "all" possible subspace of $\mathbb{R}^d$ without constraints, and $\mathcal{C}$ is not intended to denote "all" possible set of vectors. Misinterpretations of this nature could lead to unintended understandings of the work.

I do not think I have expressed my first concern clearly. The author write (from line 187).

Let $\mathbb{S}$ be a subspace of $\mathbb{R}^d$. The semnatics of $\mathbb{S}$ is defined as: $F_{sem}(\mathbb{S}) = \bigcap_{v \in \mathcal{C}^*} f_{sem}({v})$ where $\mathcal{C}^*$ is the set of embedding vectors in $\mathbb{S}$ that that minimizes the following symmetric difference: $\mathcal{C}^* = argmin_{\mathbb{C} \subset \mathbb{S}} {\bigcup_{v' \in \Set{C}} f_{sem}(v') - \bigcap_{v' \in \Set{C}} f_{sem}(v')}.$

I believe that the idea that the authors want to convey is that $F_{sem}$ can retrieve the semantics for any subspace of $\mathbb{R}^d$. Is that correct?

• No. $F_{sem}$ is not intended to retrieve the semnatics for any subspace of $\mathbb{R}^d$. The process begins with the vector set $\mathcal{C}$, which is not arbitrary. $\mathcal{C}$ is first introduced as an input of $F_{sem}()$, the function used to construct Semantic Field, as extended from Definition 2. We would like to clarify that $\mathcal{C}$ is derived from the neighbor set of a certain vector $v$.
• Additionally, the vector $v$ we mention in Section 3 is not randomly chosen within the space. Our discussion pertains to an embedding space defined as a vector space to utilize a broader array of mathematical tools. However, this does not imply that all concepts applicable to vector spaces are valid here, as embedding spaces are not fully understood. An obvious difference is that an embedding space is not that "dense", meaning not every vector within them is meaningful for analysis. Randomly selecting a vector from this space is likely to yield one lacking significance.
• This also further limits the elements of $\mathcal{C}$ to obtain $\mathcal{C}^*$. In our study, only embedding vectors that can be mapped into entities are considered since it is an embedding space of an Entity Linking model. Therefore, the algorithm we provided, in practice, is more akin to searching than optimization.
• To summarize, $F_{sem}$ is not supposed to retrieve the semnatics for any subspace of $\mathbb{R}^d$. The domain of this function is limited, because $\mathbb{S}$ does not mean a subspace without any constraints.

So, let us choose $\mathcal{C} = \\{\vec{0}\\}$.

• As a result, any set containing only a single vector (e.g., $\\{\vec{0}\\}$) cannot be considered a valid $\mathcal{C}$. Moreover, the vector $\vec{0}$ has no practical significance in real-world applications, as it does not correspond to any meaningful entity. Consequently, $\vec{0}$ is not regarded as a valid $v$ in this study.
• We hope these clarifications contribute to a clearer understanding of our intentions and approach. We are grateful for the reviewer's feedback and welcome any additional questions or suggestions.

Given all the merits mentioned in above, there is one relatively major defects: There is a lack of comparison, which is a common defects shared by many many papers. To this work, especially: There is a lack of baseline models: The authors compared the classification accuracy and training time of the proposed model against SVM, KNN, Random Forest and BERT on the AG-News dataset. The only notable baseline models is the BERT model. All others, SVM, KNN and random forest, cannot be regarded as formal baselines. You need to find more published models to serve as baselines. Now that you adapt BERT, why not further include RoBERTa, Electra and XLNet? More published baseline models are required to make the comparison more convincing.

• We appreciate the suggestion and agree that adding more comparisons could be beneficial. Initially, our focus was on comparing with non-deep learning models due to their explainability.
• BERT was included primarily to illustrate that this is not a trivial classification task. However, we acknowledge that incorporating additional deep learning models, such as RoBERTa, Electra, and XLNet, could enhance the persuasiveness of our results. We will update this section accordingly.

There is a lack of comparison scenario: The authors only conducted classification accuracy and training time comparison on the AG-News dataset. I think that is not enough. I am not familiar with this dataset, so I think the authors may provide a better background introduction to benchmark dataset used. Also, the authors may consider using more than one dataset for accuracy and training time comparison. In addition, beyond accuracy, how about F-1 score, precision, and recall comparison. Accuracy is sometimes not enough to fully evaluate the performance of a model.

• AG-News is a widely used text classification dataset (Madaan et al. 2024; Wei et al. 2021; Zaheer et al. 2020). We appreciate your suggestion and agree that including more background information would be beneficial.

Reference

• Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y. and Gupta, S., 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.
• Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M. and Le, Q.V., 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
• Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L. and Ahmed, A., 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33, pp.17283-17297.

Weakness

1. although the paper starts off with a generic motivation, it remains unclear how the proposed approach would tackle, e.g., mentioned problem of polysemiotic words.

• This aspect is handled through Entity Linking when obtaining the embedding. We would like to emphasize that the primary motivation of this paper is not to address the issue of polysemiotic words.
• For further details, please refer to The Clarification of Motivation in To All Reviewers.

2. some assumptions seem to be made without justification. Unless I'm missing something, hierarchical clustering only ever assigns a point to a single cluster. This does not play well with the key motivation that words are polysemous and meaning determined by their use in context.

• Hierarchical clustering is not used to address the polysemiotic words problem.
• Please refer to The Clarification of Motivation in To All Reviewers.

3. the experiment section is not very clear about what embeddings are used.

• Plese refer to Unclear Experiment Setting in To All Reviewers.

4. limited to text embeddings

• Thank you for your insightful suggestion. Testing the method on other types of embeddings, such as image or graph embeddings, is indeed an intriguing idea. We will include this as a potential direction for future work.

Q1: How do you disambiugate polysemous words in the clustering? It seems that the method only assigns a word to a single cluster?

• This part is done by Entity Linking when we get the embedding. The clustering algorithm is not for this purpose. Please refer to The Clarification of Motivation in To All Reviewers.

Q2: What embeddings are used for the experiments? Are they just tf-idf vectors? word2vec? contextualized embeddings from a language model?

• They are entity embeddings from BLINK.
• Please refer to Unclear Experiment Setting in To All Reviewers for more details.

Before addressing the specific queries, we would like to gently remind Reviewer qNrg of a possible misunderstanding regarding the distinction between singular values/vectors and eigenvalues/eigenvectors. We understand this may be a minor oversight, but mistaking singular values/vectors for eigenvalues/eigenvectors could indeed create challenges in comprehending the remainder of our work. We believe this may have influenced several of the comments provided. We hope this clarification proves helpful as we proceed with addressing the individual queries.

Line 265: In the example, it is unclear to me how Fig. 4 is formed after just 3 iterations. In Algorithm 1, Line 5, the merging happens in a cluster-wise fashion, there are 8 merges in Fig. 4.

• Thank you for your comment regarding Figure 4. We would like to clarify that this figure serves as a toy example aimed at providing conceptual illustration. It is not intended to represent a case study and is not meant to correspond to the exact iterations of the algorithm.

$\sigma_i([A_x | O]) = \sigma_i(A_x)$: this isn’t true as there are additional 0 eigenvalues for the matrix on the left.

• We would like to respectfully remind the reviewer that the terms in question are singular values, usually represented by $\sigma$, instead of eigenvalues, conventionally represented by $\lambda$. Zero singular values are typically ignored in the context of matrix perturbation and many other practical applications because they do not contribute to the structure of the matrix.

How is Equation 8 proved after Theorem 2? The approximation is an upper bound on the original value.

• We would like to kindly remind the reviewer that on the left side of Equation 8, it is $\tilde{F}_{sem}(c_x,c_{new})$ rather than $F_{sem}(c_x,c_{new})$.
• Furthermore, using bounds as an approximation is a common, practical, and effective practice in matrix perturbation theory. This approach forms the foundation of some of the most fundamental and well-known theorems in the field, such as the Davis-Kahan Theorem and Weyl's Theorem.

The presentation of the experimental section needs to be improved. The section should start by stating the overall goal of the empirical results and analysis experiments. Currently, the reader is confused about several setup issues.

• Please refer to Unclear Experiment Setting in To All Reviewers.

$r$,$r_1$,$r_2$ are not defined in Eq. 13.

• We would like to clarify that the $r$, with or without subscripts, denotes the rank of a matrix. We believed this is evident given the context of a discussion closely related to singular values and vectors. However, we appreciate the reviewer's attentiveness, and we will make sure to explicitly define these terms to enhance clarity.

If the paper is posed as a new hierarchical clustering method, it should present results of dendogram accuracy on clustering datasets.

• Thank you for your question. No, the paper is not.
• For further details, please refer to The Clarification of Motivation in To All Reviewers.

Weakness

W1. The experiment section needs some more details, It is missing details on how the baselines are adapted in the work, the hyperparameters chosen, and how the dataset for the experiment is constructed.

• We sincerely appreciate your insightful feedback regarding the experimental configurations.
• We have realized that one issue with the writing in the experiment section is that the shared experimental settings are only listed in Section 5.1. As a result, it may be easy for readers to overlook them when reviewing the subsections for the remaining experiments. Additionally, we acknowledge that we did not clearly specify the configurations of the baseline models, particularly regarding the training process, as also pointed out by ReviewerT896. This lack of clarity could potentially lead to misunderstandings and make the experiments appear less supportive.
• To address these concerns, we will provide a more detailed explanation of how the baseline models are utilized to ensure clarity.

W2. The intuition behind the semantic shift in eq (7) is lacking. It seems quite a big jump from an abstract definition of semantic fields to a concrete definition based on the differences of their eigen-values and eigen-vectors.

• Thank you for your query regarding the intuition behind Equation 7. In our approach, we approximate a Semantic Field Subspace using singular values (representing importance) and singular vectors (representing basis). When the semantics of a given subspace change, the basis of this subspace, or the importance of the basis, would change accordingly.
• For instance, if we randomly collect embeddings from the contexts of the word 'Apple' and use these embedding vectors to construct a subspace, two significant bases (or two sets of significant bases) may emerge: one capturing the meaning of Apple as a fruit and the other capturing its meaning as a technology company.
• If we subsequently add more embeddings from contexts related to fruits, the importance (singular values) of the fruit-related basis will adjust, along with changes in the directions of the bases (singular vectors). Similarly, adding embeddings from tech-related contexts would impact the tech-related basis in the same manner.Hence, Equation 7 provides a straightforward method to quantify these changes.

Q2. There are many equations in the paper, the ones that are not referenced later probably does not need a label.

• We appreciate your observation regarding the numerous equations in the paper. We acknowledge this and will carefully review the equations to remove labels from those that are not referenced. We believe that this revision will enhance the clarity and readability of the initial theoretical section, making it easier to follow for our readers.