Towards an Explainable Comparison and Alignment of Feature Embeddings

We thank Reviewer KUAe for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: (Our Numerical results are shown in this link)

1- Comparison with existing embedding alignment techniques

A distinction of our proposed alignment approach is that SPEC-Align operates directly on kernel matrices, enabling cluster-level comparison between encoders without requiring pointwise embedding alignment. The standard Procrustes and Wasserstein distance-based alignments usually enforce alignment in the sample level, which is a stronger alignment requirement compared to SPEC-align. This cluster-centric perspective is particularly advantageous when the embeddings have different topologies that should not be forcefully aligned or when only cluster-assignment consistency (rather than pointwise matching) is needed for the downstream task. We will include this discussion in the revised text.

2- Difference kernel matrix and differential kernel matrix

We thank the reviewer for pointing this out. We will update the paper and consistently use the term “difference kernel matrix” to avoid any confusion.

3- Non-spectral clustering methods such as hierarchical clustering or DBSCAN

While the current SPEC framework follows spectral clustering for kernel-based embedding comparison, extending non-spectral clustering methods such as hierarchical clustering or DBSCAN [1] to perform cluster-based embedding comparison will be an interesting direction for future exploration. We will discuss this future direction in the conclusion section.

[1] Ester et al., "A density-based algorithm for discovering clusters in large spatial databases with noise" KDD 1996

4- SPEC-align computational complexity and scalability in sample size

Following Proposition 5.1, we showed that SPEC-align can be computed using the power method on a matrix with size $(d_1+d_2)\times (d_1+d_2)$, and the gradient calculation can be run with only $O(\max\lbrace n_B, (d_1+d_2)^2\rbrace)$ computations ($O((d_1+d_2)^2)$ is the cost of one run of the power method), considering a batch size $n_B$ and embedding dimensions $d_1$ and $d_2$. We will explain the complexity of SPEC-align in the revised text.

We thank Reviewer 7vC9 for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: (Our Numerical results are shown in this link)

1- API endpoints for the embeddings used in the work

We would like to clarify that we have used open-source embeddings in our numerical experiments from their main repositories. Specifically, the embeddings discussed in the paper are in the following repositories, for which we will include the links in the revised paper:

DINOv2 downloaded from: https://huggingface.co/docs/transformers/en/model_doc/dinov2

CLIP downloaded from: https://huggingface.co/docs/transformers/en/model_doc/clip

SWAV downloaded from: https://huggingface.co/lixiangchun/imagenet-swav-resnet50w2

RoBERTa downloaded from: https://huggingface.co/docs/transformers/en/model_doc/roberta

E5 downloaded from: https://huggingface.co/intfloat/e5-base-v2

Inception V3 downloaded from: https://huggingface.co/docs/timm/en/models/inception-v3

2- alignment_loss_weight parameter in Section B.4.1

We thank the reviewer for pointing this out. The alignment_loss_weight parameter refers to the $\beta$ hyperparameter in Equation 6, which we have set as $0.1$ (mentioned in the Appendix) in addition to the parameters of the OpenCLIP Github repository. We will make this clear in the revision.

3- Additional experiments for text embeddings’ alignment

We thank the reviewer for the suggestion. To further address the reviewer's comment, we aligned the CLIP text embedding with the T5-XL model. In Figure 6 of the rebuttal link (last page), we can observe that the CLIP kernel has become more similar to T5-XL, and the SPEC-diff is also decreasing.

T5-XL model huggingface: https://huggingface.co/google/t5-v1_1-xl

4- role of $\beta$ penalty on the alignment

In our experiments, we conducted a grid search for $\beta$ values ranging from 0.05 to 1. During our experiments, we observed that when $\beta$ is large ($\beta$ > 0.5), the consistency between CLIP text and image alignment decreases, and aligning should occur gradually. For our experiment, we ultimately used $\beta$ = 0.1.

We thank Reviewer iKx5 for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: (Our Numerical results are shown in this link)

1- Quantitative evaluation of the SPEC method We thank the reviewer for the suggestions. Following the suggestion, we have analyzed the cluster distributions using violin plots to visualize normalized distances between data points. The plots also suggest that one embedding can cluster the points more strongly. Additionally, we performed KMeans clustering on the embedding features and computed normalized mutual information (NMI) between SPEC labels and KMeans clusters. The results indicate that one embedding reaches a stronger alignment and correlation with the KMeans labels.

2- Referring to the Appendix figures in the main text

We thank the reviewer for pointing this out. We will move the referrals to Figures 8, 9, and 12 to the Appendix in the revision.

3- Experimental Validation of Assumptions: Conditions 1 and 2

We analyzed Conditions 1 and 2 in the experiment of Figure 2 in the main text. The numerical results are presented in the Rebuttal Figures 1 and 2.

4- References in Sections 1 and 3

We thank the reviewer for pointing this out. We will cite references related to the spectral clustering, kernel PCA, alignment of embeddings, and explainability of PCA approach to embedding analysis in the revised sections 1 and 3.

5- SPEC method vs. using classification accuracy for embedding comparison

We appreciate the reviewer’s point on using the labeled samples by embedding models to explain their differences. We note that the selection of the label set can significantly influence the result of this approach. On the other hand, the SPEC method performs unsupervised learning to identify the soft-labels (i.e., eigenvectors) for performing the comparison of embeddings. Therefore, the SPEC method can avoid any biases that might be introduced by using a given label set. Also, we highlight that the SPEC method performs a soft clustering, which is different from a hard labeling of the samples assigning each data point to only one label. We will discuss these points in the revised introduction and conclusion.

We thank Reviewer hTAA for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: (Our Numerical results are shown in this link)

1- Experiments on GPT-4o generated text data

We generated a dataset using GPT-4o, covering categories including professions, emotions, genders, actions, objects. We used the bellow prompt to generate. In the revised manuscript, we will explain the generation process.

The prompt for generating the text dataset: “You are an expert in text-to-image models. Text-to-image models take a text prompt as input and generate images. Your task is to generate a prompt describing a person in [Profession] [Emotion], and [Gender] performing [Action] with [Object].

2- Additional experiment for real text datasets

To further address the reviewer’s comment, we also validated our approach on a large-scale real text dataset: WikiText-2. We split the dataset into 10K samples, each containing 100 tokens. Then, we used SPEC to compare CLIP and RoBERTa embeddings. The results can be found in Figure 2. We observed that RoBERTa better clustered Military Operations, Species Biology, Historical Figures, and Music, while CLIP embeddings more strongly clustered Sports and Science.

We also examined the distribution of pairwise distances within each cluster to verify that one embedding successfully captured these clusters while the other was less inclined to do so. Also, we ran the K-means clustering algorithm 50 times on each of the embedding's features and computed the averaged (across the 50 runs) Normalized Mutual Information (NMI) between the K-means labels and the SPEC-identified labels. The results demonstrate that one embedding achieved considerably stronger alignment with KMeans labels.

3- Evaluation on SPEC-diff

To show that SPEC-diff effectively measures embedding similarity in SPEC-Align, we considered the SPEC-Align of CLIP to DINOv2 and tracked the SPEC-diff scores while plotting their kernel matrices. The results in Figure 3 show that as the kernel matrices look more similar, the SPEC-diff value decreases. The results suggest the SPEC-diff's utility for quantifying embedding alignment in SPEC-Align.

4- Explanation on the statement in Line 156

We will make the explanation clear that since the multiplication order in $C=\frac{1}{n}\Phi^\top \Phi$ and $C=\frac{1}{n}\Phi \Phi^\top$ is flipped, they share the same non-zero eigenvalues, and those eigenvectors are in one-to-one correspondence.

5- The choice of Gaussian vs. cosine similarity kernel functions

The choice of kernel function determines the similarity measure used for the clustering algorithm. The cosine similarity looks only at the angle between the vectors, while the Gaussian kernel concerns the Euclidean distance between the input points. The choice of the kernel function depends on the input embeddings and their induced geometry in the embedding space. As we discussed in the text, our analysis can be efficiently applied to both kernels.

6- Relationship to Laplacian spectral clustering

In the revised draft, we will make the connection to Laplacian spectral clustering more clear. One main difference between the proposed SPEC clustering and Laplacian spectral clustering is the usage of kernel matrix in SPEC (similar to Kernel PCA) vs. the Laplacian (kernel minus the diagonal degree matrix) in spectral clustering. We will explain this difference in the draft.

7- Additional results on embedding alignment using SPEC-Align

We aligned the CLIP text embedding with the T5-XL model. In Figure 6 of the rebuttal link (last page), we observe that the CLIP kernel has become more similar to T5-XL, and the SPEC-diff is also decreasing.

8- Symmetry of SPEC-diff

We would like to clarify that SPEC-diff is a symmetric measure because it is defined as the spectral radius (eigenvalue with maximum absolute value) of the difference of the two kernel matrices. Since the definition concerns the spectral radius (and not the maximum eigenvalue), the SPEC-diff is symmetric with respect to the embedding order. We will make this point clear in the writing.

9- Clarification on kernel values

We would like to clarify that the paper’s main results suppose a normalized kernel function where for every $x$, we have $k(x,x)= \langle \phi(x), \phi(x) \rangle=1$, i.e. $\Vert \phi(x) \Vert = 1$. Following Cauchy-Schwarz inequality, $| k(x,x’)|\le 1$ holds for every $x,x’\in\mathcal{X}$. We will make this point clear in the revised draft.

10- Typos and writing improvements

We thank the reviewer for pointing them out. We will correct them in the revised draft.

11-Change from CLIP to OpenCLIP

For alignment experiments, we utilized the widely-cited open source OpenCLIP github repository. We note that in Table 1, the CLIP backbones are the same.