When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product

We would like to thank Reviewer m3ub for the constructive and thoughtful feedback on our work. We are glad that the reviewer finds our work to propose a “novel and practical technical combination”. Below is our response and clarifications regarding the questions and comments in the review:

1- Quantitative comparisons on clustering results for Figures 1 and 3

We thank the reviewer for pointing this out. Originally, we decided to limit the evaluation of Figure 1 in the introduction to the qualitative comparison of kernel heatmaps. To address the reviewer’s concern, we have measured the clustering score, Adjusted Rand Index, Adjusted and Normalized Mutual Information (ARI, AMI and NMI), and the results are in the following table, indicating the better clustering of the Kronecker product fusion in comparison to each of the CLIP and DINOv2 embeddings.

2- Theoretical analysis of Kronecker product in semantic information fusion

We appreciate the reviewer’s point on the theoretical analysis of the role of Kronecker product in combining the semantic information of the two embeddings. As explained in the text (Propositions 1 and 2), the Kroncker product fusion leads to the product of the two kernel functions. A simple but somewhat insightful example to understand how the semantic features can be merged is when we have a 2-dimensional feature vector $[x,y]$ where each of the two embeddings can capture the information of only one of the coordinates, i.e. $\psi_1([x,y]) = f(x)$ and $\psi_2([x,y]) = g(y)$ for mutivaraibel functions $f$ and $g$. Then, the Gaussian kernel fusion of the two embeddings will lead to the following kernel functions

$k_{fuse}([x,y],[x’,y’]) = \exp(\frac{||f(x)-f(x’)||^2 + ||g(y)-g(y’)||^2}{\sigma^2})$

As seen, while neither of the embeddings contained the full information, the kernel function of the Kronecker product captures the role of both the coordinates. We will discuss this simple example in the revised text to better highlight how the fused embedding can capture the semantic information of both the embeddings.

3- Comparison with training-based fusion methods

We appreciate the reviewer’s comment regarding the comparison with training-based fusion approaches. To clarify, the baselines GATE and ATTN included in our study are indeed training-based fusion methods. However, we understand that the reviewer’s comment could more generally refer to a training-time integration, retraining one embedding model from scratch while conditioning on the other, potentially enabling lower inference-time complexity.

Regarding inference efficiency, it is important to note that, assuming sufficient memory, our method allows for the two embedding models to operate in parallel, thereby avoiding increased inference latency. To support this, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The individual CLIP and DINOv2 models required 73.6 ms and 392.1 ms per batch, respectively. Our parallel RP-KrossFuse pipeline achieved a total inference time of 413.1 ms, which is only slightly higher than the slower of the two models. Notably, the fusion step itself adds just 0.14 ms of overhead.

Also, a core advantage of our training-free fusion approach is its potential for improved out-of-distribution (OOD) generalization. Since RP-KrossFuse does not rely on training over a specific dataset, it avoids overfitting to a particular data distribution. In contrast, training-based methods often show sensitivity to their training data. For instance, Table 5 of [1, SLIP] demonstrates a 7.2% performance gap on ImageNet linear probing: 65.4% when trained on CC3M, 73.7% on CC12M, and a drop to 66.5% when trained on YFCC15M. This highlights the dependence of training-based fusion models on the choice of pretraining data. We will clarify these points and include the new results in the revised manuscript.

4- Practical applications of the proposed method

We thank Reviewer 9JZK for the suggestions on showing the practical application of our proposed embedding fusion. We agree that the application of KrossFuse fusion to VLMs could be highly interesting. In the rebuttal period, we did not manage to finish the visual instruction tuning of LLAVA-v1.5-7B. However, we performed another experiment on the application of the framework in guided diffusion models as explained in the following paragraph.

To demonstrate the value of the KrossFuse fusion in conditional image generation, we used it to adapt and improve the Vendi score diversity guidance proposed in [2]. The Vendi score guidance method guides the reverse diffusion process using the Vendi diversity score of the generated data. In the original implementation in [3], the authors used the CLIP embedding for computing the Vendi score in diversity guidance, which we updated to the Kronecker product of DINOv2 and CLIP.

In the experiment, we generated ImageNet samples using a class-conditional Diffusion Transformer (DiT-XL/2) with both the CLIP-based Vendi score guidance from [2], using CLIP as the feature extractor, and the KrossFuse embedding (CLIP fused with DINOv2). We found that the diversity of the generated samples, measured with Recall [3] and Coverage [4] scores, increased with the KrossFuse embedding, while the quality scores Precision [3] and Density [4] also improved over the original CLIP-based Vendi score guidance. We will include this experiment in the received manuscript.

References:

[1] Mu et al. Slip: Self-supervision meets language-image pre-training. ECCV 2022.

[2] Askari et al. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024.

[3] Kynkäänniemi et al. Improved precision and recall metric for assessing generative models. NuerIPS 2019.

[4] Naeem et al. Reliable fidelity and diversity metrics for generative models. ICML 2020.

We would like to thank Reviewer tf3Z for the constructive and thoughtful feedback on our work. We are glad that the reviewer finds our work to address “a problem of potentially high significance” and the design choices “well-motivated” . Below is our response and clarifications regarding the questions and comments in the review:

1- Writing improvements

We thank Reviewer tf3Z for pointing out the needed citations and information in Tables 1,2 and the captions of Figures 4,5. We would like to clarify that the notation $\psi_{1,2}$ denotes the KrossFuse embedding of embeddings $\psi_{1}$ and $\psi_2$ which we will state in Theorem 1. The term “properly aligned” refers to how alignment is supposed to be measured in a cross-modal embedding (e.g. via cosine similarity or Euclidean distance). We will make the term clearer in the revision.

2- Range of kernel values in the KrossFuse fusion

We appreciate the reviewer’s comment on the potential negative values in the kernel inner products and whether they fit into our argument for the product of kernel functions. As already pointed out by the reviewer, this concern does not apply to the Gaussian (RBF) kernel, which always takes positive values. We note that in all our clustering experiments, we have used the Gaussian kernel, which ensures non-negative values.

In the case of the cosine similarity kernel, the kernel inner product may indeed take negative values. However, we note that for both CLIP and OpenCLIP, the cosine similarity is always non-negative on MS-COCO and ImageNet samples (as also shown in reference [1]). On the other hand, the cosine similarity for DINOv2 can take negative values on ImageNet samples.

The question raised by the reviewer can be viewed as follows: What is the interpretation of the product kernel fusion when kernel similarity scores take negative values? Our answer is that in such cases, the product kernel fusion distinguishes sample types based on the orthogonality of their embedded vectors. Please note that we do not claim that this is the universally correct way to define similarity for all embeddings, but rather that it is a natural consequence of fusing kernels via multiplication. As such, the effectiveness of the product kernel fusion depends on whether the orthogonality condition is proper for the fused embeddings, i.e. should clustering within the embedding space be performed based on the orthogonality of the vectors? If an embedding has such a property, then applying the framework with cosine similarity can improve the numerical results. If this is not the case, the framework can instead be used with Gaussian, Laplace, or (normalized) even-degree polynomial kernels, all of which yield positive values bounded by 1, where our intuition on similarity score will directly apply.

3- The value of random projection dimensionality $l$ in the experiment

We would like to clarify that we have used dimension $l=3000$ for all the datasets except $l=5000$ in the case of ImageNet. We will make this clear in the revision.

4- Fairness of RP-KrossFuse comparison with embedding of different dimensions

If we have correctly understood the reviewer’s comment, it concerns the fairness of comparing embeddings with different dimensions. We respectfully disagree with the notion that such comparisons are inherently unfair. For example, DINOv2 and CLIP already have different embedding dimensions (768 for DINOv2 and 512 for CLIP), yet their comparison is widely accepted in the literature. Following the same logic, it would be inconsistent to deem a comparison between RP-KrossFuse and these individual embeddings as unfair solely due to dimensional differences.

To further clarify our motivation for using random projection in RP-KrossFuse, our goal is to reduce the high dimensionality of the original Kronecker fusion (nearly 800,000 for CLIP and DINOv2) only for the sake of computational efficiency in downstream applications. If enough compute and memory were available, we would even prefer to use the full KrossFuse representation over the projected one. As highlighted in our ablation studies, a lightweight random projection with computational cost $O((d_1 + d_2)l)$ and projection dimension $l = 5000$ is sufficient to preserve the performance of full KrossFuse in the ImageNet case.

Continuing our response, if we are constrained to reduce the fused embedding to a bounded dimension, e.g., $l = 512$ to match the lower of the original embedding sizes, then using a pure random projection may not improve the signal-to-noise ratio (SNR). Under such dimensional constraints required by the reviewer's fairness criterion, we would instead apply PCA dimensionality reduction, which provides better SNR at the cost of increased computational cost. Of course, performing PCA directly on the full Kronecker fusion (size $\approx$800,000) is computationally prohibitive. Therefore, in our rebuttal experiments, we applied PCA to reduce the RP-KrossFuse embedding (from 5000 to 512 dimensions) using ImageNet training data. The resulting linear-probe accuracy was 83.7% for the 512-dimensional PCA-reduced embedding, compared to 84.1% for the original RP-KrossFuse at 5000 dimensions. For reference, the linear-probe accuracies of the original CLIP and DINOv2 image encoders are 73.2% and 83.3%, respectively.

To summarize, we apply random projection in RP-KrossFuse solely to reduce computational cost for downstream tasks. We do not believe that exact dimension matching is a strict requirement for fair comparison between embeddings. However, if dimensionality parity is required, we recommend using PCA for compression, as it offers improved SNR, although at a higher computational cost. We will include this discussion in the revised manuscript.

5- Modality Gap for the fused embedding vs. original CLIP embeddings

To address the reviewer’s comment, we have measured the modality gaps (as defined in [1]) of the fused embeddings over the MS-COCO dataset. We made the measurement with our used RP-KrossFuse fused embedding with dimension $5000$. The results are in the following table.

6- Limitations

To address the reviewer’s comment, we will discuss the limitations of the KrossFuse framework in more detail, including the potential cross-validation needed for parameter $C$ and $l$. However, we would like to highlight the following results in the Appendix: the text-image retrieval and clustering results, the text classification results, that extend the evaluation beyond only the image classification task. Still, we will discuss the further evaluation of the KrossFuse framework in non-image domains in the limitation section.

7- Typo in the bound of Theorem 1

We thank the reviewer for catching the typo in the theorem’s equation regarding the power of $B$. We will correct it in the revision.

8- Question 1 (Results for $L_2$-normalized DINOv2)

To answer the reviewer’s question, we tested the performance of the $L_2$-normalized DINOv2 embedding and the numerical results are in the following table (continuing Table 1 in the main text). As seen, the numerical performance on linear-probe classification results are mostly similar between normalized and unnormalized DINOv2.

9- Question 4 (Experiments beyond self-supervised embeddings)

To address this question, we performed zero-shot and linear-probe classification experiments on ImageNet1K by applying RP-KrossFuse to the ImageNet-22k-supervised-trained image model DeiT-III [2] .The numerical results are in the following table:

References:

[1] Liang et al. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 2022.

[2] Touvron et al. Deit iii: Revenge of the vit. ECCV 2022.

We would like to thank Reviewer 9JZK for the constructive and thoughtful feedback on our work. We are pleased to hear that the reviewer finds our work “well-written” and the proposed method to be “sound” . Below is our response and clarifications regarding the questions and comments in the review:

1- Experiments on the recent SigLIP embedding

In the Appendix C.6, we have already presented the numerical results of the zero-shot and Linear-probe classification accuracy of SigLIP embeddings on ImageNet. We will move these results to the main text to address the reviewer’s comment. In the rebuttal period, we also ran more experiments on recent embedding models MobileCLIP[1] and OpenVision[2], and the results support the method:

In the revised paper, we will add these results to the main text.

2- Application of the proposed method on text-to-image diffusion models

We thank Reviewer 9JZK for the suggestion on showing the practical application of our proposed embedding fusion to diffusion models. To demonstrate the value of the proposed fusion in conditional image generation, we used our approach to adapt and improve the Vendi score diversity guidance proposed in [3]. The Vendi score guidance method guides the reverse diffusion process using the Vendi diversity score of the generated data. In the original implementation in [3], the authors used the CLIP embedding for computing the Vendi score in diversity guidance, which we updated to the Kronecker product of DINOv2 and CLIP. In the experiment, we generated ImageNet samples using a class-conditional Diffusion Transformer (DiT-XL/2) with both the CLIP-based Vendi score guidance from [3], using CLIP as the feature extractor, and the KrossFuse embedding (CLIP fused with DINOv2). We found that the diversity of the generated samples, measured with Recall [4] and Coverage [5] scores, increased with the KrossFuse embedding, while the quality scores Precision [4] and Density [5] also improved over the original CLIP-based Vendi score guidance. We will include this experiment in the received manuscript.

References:

[1] Vasu et al. Mobileclip: Fast image-text models through multi-modal reinforced training. CVPR 2024.

[2] Li et al. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprint (2025).

[3] Askari et al. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024.

[4] Kynkäänniemi et al. Improved precision and recall metric for assessing generative models. NuerIPS 2019.

[5] Naeem et al. Reliable fidelity and diversity metrics for generative models. ICML 2020.

We would like to thank Reviewer MP5Z for the constructive and thoughtful feedback on our work. We are glad to hear that the reviewer finds our work’s motivation “well-justified” and our proposed method to be “theoretically grounded” . Below is our response and clarifications regarding the questions and comments in the review:

1- The method’s application to recent embeddings models

We appreciate the reviewer’s suggestion regarding the application of our fusion method to more recent embedding models. We would like to emphasize that RP-KrossFuse is a general method that can be applied to fuse any combination of unimodal and cross-modal embeddings. Our focus on CLIP and its variants is due to CLIP’s widespread adoption and integration in many AI and machine learning frameworks.

Having said that, we agree that evaluating the method on newer cross-modal embeddings is valuable. In fact, we have already included results on the more recent SigLIP model in Appendix C.6 as part of our ablation studies. Additionally, during the rebuttal period, we conducted further experiments on the reviewer’s suggested models, OpenVision and MobileCLIP. Below, we report the Linear-Probe and Zero-Shot classification results on ImageNet. These results also support the effectiveness of RP-KrossFuse when applied to more recent embedding architectures. We will include these results in the revised paper.

2- Role of different training data distributions of the embedding models in their Kronecker product fusion

We fully understand the reviewer’s point regarding the differing training data distributions of CLIP and DINOv2, which naturally result in different behaviors when clustering or distinguishing between samples. In fact, this observation results in a key motivation behind our proposal to fuse these embeddings using the Kronecker product. As discussed in the introduction, the kernel similarity score of our fused representation is the product of the individual similarity scores from the two embeddings. This means that two samples will be distinguishable in the fused embedding space as long as either CLIP or DINOv2 assigns them sufficiently small similarity scores.

Given that CLIP and DINOv2 are trained on different datasets, it is expected that they will specialize in distinguishing different semantic or visual concepts. This diversity enables the Kronecker fusion to effectively combine their complementary strengths. For example, Figure 1 in the paper shows that CLIP is more effective at separating traffic sign categories, while DINOv2 performs better at differentiating between dog breeds. When we apply Kronecker fusion, the resulting representation could successfully distinguish both traffic signs and dog breeds, which neither model could fully achieve on its own.

Therefore, we argue that the difference in training data distributions between the embedding models is not a limitation, but rather could be a motivation for fusion. The Kronecker product allows us to construct a fused embedding that inherits the union of the discriminative capabilities of the models. We will revise the introduction to make this intuition more clear, and we also kindly refer the reviewer to Figures 8 and 10 in the Appendix, which provide additional experiments supporting this point across more diverse sample types.

3- Comparison with the training-based fusion methods

We appreciate the reviewer’s comment regarding the comparison with training-based fusion approaches. To clarify, the baselines GATE and ATTN included in our study are indeed training-based fusion methods. However, we understand that the reviewer’s comment refers to a training-time integration, retraining one embedding model from scratch while conditioning on the other, potentially enabling lower inference-time complexity. We would like to raise two key points in response:

First, regarding inference efficiency, it is important to note that, assuming sufficient memory, our method allows for the two embedding models to operate in parallel, thereby avoiding increased inference latency. To support this, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The individual CLIP and DINOv2 models required 73.6 ms and 392.1 ms per batch, respectively. Our parallel RP-KrossFuse pipeline achieved a total inference time of 413.1 ms, which is only slightly higher than the slower of the two models. Notably, the fusion step itself adds just 0.14 ms of overhead.

Second, a core advantage of our training-free fusion approach is its potential for improved out-of-distribution (OOD) generalization. Since RP-KrossFuse does not rely on training over a specific dataset, it avoids overfitting to a particular data distribution. In contrast, training-based methods often show sensitivity to their training data. For instance, Table 5 of [1, SLIP] demonstrates a 7.2% performance gap on ImageNet linear probing: 65.4% when trained on CC3M, 73.7% on CC12M, and a drop to 66.5% when trained on YFCC15M. This highlights the dependence of training-based fusion models on the choice of pretraining data.

We will clarify these points and include the new results in the text.

4- Unitary transformation in Appendix C.2 experiments

We would like to clarify that the unitary transformation mentioned in Appendix C.2 was applied solely for the cross-modal clustering experiments reported in that subsection. None of the experimental results presented in the main text or other parts of the appendix involve this transformation.

The motivation for applying a unitary transformation in Appendix C.2 is due to the known “modality gap” between the image and text embedding spaces of CLIP, as discussed in previous work [2]. To enable meaningful cross-modal clustering via spectral clustering, it was necessary to learn and correct this modality misalignment. The unitary transformation serves this purpose by aligning the modalities in a shared space before clustering.

We emphasize that this step is limited to the cross-modal clustering setting and was not used in any of the other experiments in the paper. We will make this more clear in the revised paper.

5- "concatenation-only" baseline

We thank the reviewer for raising the point about the concatenation-only baseline. Theoretically, concatenating two embeddings results in a kernel similarity function that corresponds to the sum of the individual kernels. This implies that two samples will have high similarity in the fused space only if both original embeddings deem them similar, effectively modeling the intersection of the embeddings’ concepts. In contrast, our proposed Kronecker fusion leads to a product of the kernel similarities, which enables distinguishing two samples as long as either embedding distinguishes them, modeling the union of their concepts.

In terms of numerical performance, we note that the concatenation baseline is expected to perform comparably to the stronger of the individual embeddings in supervised classification settings. This is because the concatenated vector retains all entries from both embeddings, and the limited VC dimension of the linear probe helps prevent overfitting.

However, in unsupervised or weakly supervised scenarios such as clustering, the product-based Kronecker fusion can better capture the complementary structure from both embeddings. This often results in improved performance, as it reflects broader concept coverage. To empirically validate this, we conducted several experiments comparing the concatenation baseline with RP-KrossFuse across various tasks:

1. For supervised learning tasks like ImageNet linear probing, both methods perform comparably (concatenation: 83.6%, RP-KrossFuse: 84.1%), which is theoretically expected.

2. In multimodal classification tasks, RP-KrossFuse shows clear advantages over the concatenation on MVSA dataset [3] (Sentiment Analysis on Multi-view Social Data):

   • MVSA-Single: 74.3% vs 71.3%
   • MVSA-Multiple: 66.6% vs 63.7%

3. Most notably, in unsupervised clustering tasks, RP-KrossFuse significantly outperforms concatenation across the datasets. We have measured the clustering score, Adjusted Rand Index, Adjusted and Normalized Mutual Information (ARI, AMI and NMI), and the results are in the following table.

References:

[1] Mu et al. Slip: Self-supervision meets language-image pre-training. ECCV 2022.

[2] Liang et al. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 2022.

[3] Niu et al. Sentiment analysis on multi-view social data. MMM 2016.

We would like to thank Reviewer kcnR for the constructive and thoughtful feedback on our work. We are delighted that the reviewer finds our work to provide “an effective solution” and “written in a decent way”. Below is our response and clarifications regarding the questions and comments in the review:

1- Referrals to the Appendix in the main text

We thank the reviewer for pointing this out. In the revision, we will keep the more important numerical results on the zero and few shot classification and clustering performance in the main text and reduce the Appendix referrals to only one time at the end of the numerical results section.

2- The role of random projection dimension $l$ in the RP-KrossFuse method

We appreciate the reviewer’s comment regarding the importance of the random projection dimension $l$ in the RP-KrossFuse method. We would like to clarify that we have used dimension $l=3000$ for all datasets except $l=5000$ in the case of ImageNet. In the revised version of the paper, we will clearly state the value of $l$ alongside the tables and figures.

To further explain the role and selection of dimension $l$, we note that a key challenge of the Kronecker product fusion is the resulting high dimensionality: the fused representation has size $2d_1d_2$ for embeddings of dimensions $d_1$ and $d_2$. For example, combining CLIP and DINOv2 embeddings leads to a fused dimension close to 800,000 ($2 \times 512 \times 768$).

To address this, we introduce random projection in RP-KrossFuse, which is not to ensure fairness in comparison with the original embeddings, but to reduce the computational overhead to meet the computational budget. If computationally feasible, we would even prefer to use the full Kronecker product without random projection. However, this would incur significant memory and compute costs.

Also, we highlight that our proposed RP-KrossFuse scheme reduces the computational complexity from $O(d_1d_2 l)$ (naive random projection) to $O((d_1 + d_2)l)$, enabling more efficient fusion. To empirically validate this efficiency gain, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The original KrossFuse pipeline required 491.3 ms per batch and 18.406 GB of peak memory, while RP-KrossFuse (with $l = 5000$) achieved comparable performance with just 413.1 ms per batch and a significantly reduced memory footprint of 3.39 GB. Notably, the fusion step’s overhead was reduced from 22.0 ms to only 0.14 ms. For reference, the baseline CLIP and DINOv2 models individually required 73.6 ms and 392.1 ms per batch, respectively.

3- “a larger linear layer with more parameters could be applied in the linear probing experiments, leading to better performance compared to training a small linear layer on the CLIP embedding”

We would like to clarify in the linear probe classification experiments, we did not use any non-linear activation functions applied to the random projection features. Therefore, in the comparison with the CLIP model, even if we first apply a linear transformation (denoted by $g_w$) to transform the CLIP features to a higher dimension and then apply the linear probe (denoted by $f_{\theta}$) to the linearly-transformed features, the composition $g_w \circ f_{\theta}$ still remains to be a linear function of the CLIP features, and therefore the linear-probe classification accuracy will not improve over applying it directly to the CLIP embedding.

4- Question on the kernel map $\phi$

We thank the reviewer for their question regarding the kernel map $\phi$. In our experiments, we evaluated two kernel functions:

1. The normalized linear (cosine similarity) kernel, implemented as $\phi(x) = \frac{x}{\Vert x \Vert_2}$. This implementation is straightforward and directly normalizes the input embeddings.
2. The Gaussian (RBF) kernel, for which we used random Fourier features as described in Appendix Section A.4. We would like to emphasize that our implementation of the Gaussian kernel map $\phi$ does not simply apply RP-KrossFuse to a low-dimensional vector of precomputed random Fourier features. Instead, we designed the implementation to unify the generation of random Fourier features and the random projection in a single step. This unified approach effectively simulates operating in a feature space as if the number of Fourier features were infinite, while still being computationally efficient. This design allows RP-KrossFuse to retain the expressivity of the Gaussian kernel without incurring the cost typically associated with large numbers of explicit random features.