With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You

We would like to extend our sincere appreciation to the reviewer for their insightful comments and positive evaluation of our work. We are encouraged by the recognition that our approach requires less training data to achieve strong alignment, making it particularly well-suited to low-resource settings. We are likewise gratified that the method’s versatility as a regularization technique, enabling seamless integration into existing alignment frameworks, was noted. Finally, we appreciate the acknowledgement of our comprehensive experimental suite, which spans a wide range of benchmarks.

1.1 & 1.2. STRUCTURE with Smaller Networks

We focus on large models because they are readily available and are not being updated within our framework, making them essentially an initial feature extraction. Additionally, we would like to highlight that Figure 11 of Appendix I.6. (referenced in the main paper on L225) includes experiments with a base ViT.

In response to the Reviewer’s feedback, we conducted additional experiments of two model combinations, which can be considered much smaller. Specifically, in the first table, we align a ViT-Tiny with 5.72 million parameters (even smaller than a ResNet18) and an all-MiniLM-L6-v2 with 22.7 million parameters. In the second table, we align a ViT-Tiny with 5.72 million parameters and an all-mpnet-base-v2 with 109 million parameters. Both combinations are trained using linear alignment functions. Results show that adding STRUCTURE still yields a benefit even for these smaller model combinations.

We would like to highlight two key aspects of these results. First, the alignment score, in terms of mutual kNN, for these pairs of models is significantly lower than for the other (bigger) combinations. Specifically, 0.09 for DINO_v2 ViT-Tiny+all-MiniLM-L6-v2 and 0.10 for DINO_v2 ViT-Tiny+all-mpnet-base-v2, which are much lower than the 0.29 for DINO_v2 ViT-L+RoBERTa. Second, as the Platonic Representation Hypothesis suggests, the convergence of the models to the same representation increases with scale and performance. Thus, size itself has a significant influence on the relative performance of the aligned models, as more similar models facilitate their alignment, and enforcing hierarchical relationships becomes more meaningful. However, even when the models are less similar and respectively smaller, the proposed regularization yields consistent improvements.

ViT-Tiny+all-MiniLM-L6-v2:

ViT-Tiny+all-mpnet-base-v2:

2. STRUCTURE on Three or More Modalities and Additional Two Modal Cases

First, we would like to discuss the extension of the proposed regularization to aligning more than two modalities. As regularization is only calculated on the pretrained representations of each individual modality, thus not relying on any cross-modal data, extending this approach involves computing the regularization in each modality separately and adding an additional term to the loss function. Specifically, Eq. 8 would get a third term for the new modality:

Thus, regardless of the alignment objective function used, the extension of regularization is indeed straightforward and we will add a discussion in our manuscript.

In addition to text and vision modalities, we would like to point out the additional experiments on the audio and text domains in the original submission in Appendix I.3. In addition, in response to the reviewer’s comment, we have now conducted additional experiments on the biological domain using only 19,900 paired human single-cell image-transcriptome data. In this experiment, we linearly align a UCE $1$ encoder for the transcriptome data and a masked autoencoder (MAE) trained on single-cell images for the image data. We split the 19,900 paired samples into 100 training samples and 19,800 test samples to simulate the low data setting. Results below show that our regularization also helps in the biological domain, yielding relative improvements of 43.1% on the retrieval task and 30.3% on the classification task, even if the Platonic representation hypothesis is not guaranteed in the biological domain.

$1$ Universal Cell Embeddings: A Foundation Model for Cell Biology, bioRxiv 2023

3. N in Eq. (2)

The variable N in the default setting of STRUCTURE itself represents the size of the batch, typically set to around 4,096 random samples (approximately 6% of the entire dataset), which allows for a sufficient estimate of the true data generation distribution while maintaining computational efficiency. In response to the reviewer’s feedback, we conducted experiments to demonstrate how the choice of sample size affects performance. Results are given when aligning RoBERTa with a ViT-G using linear alignment. Results show that the benefit of the regularization increases with increased sample size, but also show that it’s efficiency, since with even 128 samples, there is a clear improvement.

4. Understanding of Hierarchical Level L

The hierarchical level $L$ in Equation 4 can be understood through the lens of a random walk on the data manifold. Specifically, the row-normalized similarity matrix, $P\_X$, can be interpreted as the transition matrix for a random walk on a graph where data points are nodes and similarity defines edge weights. In this context, the value $P\_X(i, j)$ represents the probability of transitioning from sample $i$ to sample $j$ in a single step. The hierarchical levels then correspond to the length of this random walk:

(1) l \= 1 (1-hop): represents the probability of a direct transition between two samples. Preserving this level enforces consistency in the immediate local neighborhood structure. It ensures that if sample $j$ is a close neighbor of $i$ in the original space, it remains so in the aligned space.

(2) $l \> 1$ (l-hop): the matrix power $(P\_X)^l$ computes the probability of reaching one sample from another in exactly $l$ steps. This captures progressively larger structural patterns. For $l=2$, it considers paths through one intermediary. This enforces consistency at the level of local clusters or "neighborhoods of neighborhoods." Two samples that are not direct neighbors but belong to the same tight cluster will have high 2-hop probabilities. As $l$ increases, we capture more global, long-range connectivity across the data manifold.

By enforcing consistency between the multi-step transition probabilities of the original space $(P\_X)^l$ and the aligned space $(P\_A)^l$, the STRUCTURE regularizer ensures the alignment function preserves the intrinsic geometry of the data manifold at multiple scales. This prevents the "over-warping" that can occur in unregularized alignment, where local neighborhoods might be preserved but the global arrangement of clusters is distorted.

While our experiments show that setting $L=1$ is often sufficient, this is because perfect 1-hop consistency mathematically implies perfect $l$-hop consistency for all $l$. Therefore, strongly regularizing the most local structure provides a powerful, indirect constraint on the entire manifold geometry. Explicitly including higher levels ($L\>1$) simply makes this constraint more direct, which can provide additional benefits on fine-grained tasks where subtle, multi-hop relationships are more critical to preserve.

We will expand the discussion about it in the revised manuscript.

5. About Modality Gap

We measured the modality gap for three different alignment setups for a ViT-Giant and RoBERTa Large, trained on COCO with a linear alignment function. The modality gap was measured for the two image-text datasets considered in our submission (i.e., COCO and Flickr) to ensure that the comparison is influenced solely by the data itself and not by any other artifacts introduced by the evaluation, such as a zero-shot template, as would be necessary for the classification datasets. Interestingly, the results show that the modality gap indeed decreases when first selecting the most similar layers and then further when applying the proposed regularization. This aligns with the findings from the respective paper, that the modality gap indeed has a relationship with downstream performance. We will include these findings in the revised manuscript.

We would like to extend our sincere appreciation to the reviewer for their positive assessment and their feedback. We are pleased that the manuscript’s clarity and rigor were noted, and that the reviewer found the idea of preserving hierarchical relationships for alignment technically sound and compelling our results strong and the proposed regularization effective and modular. We thank the reviewer for being open to increasing their score if their concerns are addressed.

1. The authors should discuss how to effectively use the hierarchical relation. Ideally, the alignment effect would increase as l increases and gradually converge.

Regarding the contribution, we would like to emphasize that the main contribution is broader than hierarchical relationships imposed in the regularization and involves the proposition of a regularization strategy that acts as a geometric prior inferred from the pretrained encoder that can enhance different alignment strategies in low data regime.

In the original version, we investigated the effect of increasing l on coarse-grained datasets (STL10, CIFAR10, Food101, and CIFAR100). The results are presented in Table 7 of the Appendix. There, we found that for some coarse-grained datasets, such as CIFAR10 or STL, the influence of the number of hops considered is less impactful. In response to the Reviewer’s question, we have now conducted more comprehensive experiments on additional datasets for different levels in the table below. On datasets that can be regarded as more fine-grained, such as Pets or Flickr, we find that enforcing higher-order interactions has a positive impact and slightly improves the performance.

It is important to note that even if one only considers a single hierarchical level, the modal still enforces consistency between higher-order relationships, as a perfect similarity graph on the lowest level also yields perfect relationships on all higher levels. Thus, increasing the number of hops is a more direct way of enforcing these higher relationships, but the lower levels still enforce them indirectly.

2. It is also suggested to discuss l-hop when first introduced, which would make it easier for readers to understand.

We will thus revise paragraphs L143-144 and include a more intuitive description of the l-hops. Specifically, we will add the following sentences after L144: “Intuitively, the similarity graph can be thought of as a Markov chain where the hops refer to the transitions between different states. Thus, the $l$ hops represent the probability of transitions reachable with $l$ hops.” Additionally, we will discuss l-hop in more detail when it is first introduced.

We would like to extend our sincere appreciation to the reviewer for their thorough and encouraging evaluation. We are grateful by the reviewer’s recognition of the well written manuscript and motivation of our work, the novelty of our methodology, and strong empirical results which convincingly demonstrate the efficacy of our methodology. We appreciate the reviewer’s willingness to increase their score, provided that our response addresses their concerns and questions.

1. This paper is very similar in spirit to FuseMix $1$ from CVPR 2024 but it was never mentioned in this work $...$ .

As the reviewer noted, although FuseMix shared a similar motivation, its underlying methodology differs from our proposed approach, especially in terms of where the contribution is focused. FuseMix aims to improve the training efficiency by using latent space augmentations. In contrast, our framework suggests selecting the most similar layers for alignment and introduces hierarchical preservation regularization to improve performance in data-limited settings. Thus, the two methods are complementary to each other and can be used jointly.

In response to the reviewer’s feedback, we conducted additional experiments: (1) comparing FuseMix to linear alignment enhanced with our methodology (layer selection + STRUCTURE regularization R_S​); and (2) enhancing FuseMix itself by incorporating our methodology into FuseMix to show complementarity. The results show that (1) the proposed framework (Similar + R_S) yields significantly better results than the FuseMix approach in the same setting, specifically leading to an average relative improvement of 64.2% on the considered datasets. Furthermore, the results show that (2) when integrating parts of the proposed framework, such as the layer selection and the regularization, with FuseMix, there is a 64.6% relative improvement compared to the original FuseMix approach. This further demonstrates the flexibility and wide applicability of STRUCTURE. We will add FuseMix results in our manuscript and discuss it in the introduction and related work section.

2. The authors do not acknowledge limitations or potential negative societal impact of their work.

We would like to point the reviewer to Appendix A of the original submission, which includes a discussion of the limitations. We did not include negative societal impacts, as the outlined methodology enables multimodal alignment even in limited data scenarios, while being computationally efficient. Thus, we are not aware of any negative societal impacts that would directly result from this work, as noted in L823-L825. To highlight these sections, we will include a reference to them in the main paper.

We thank the reviewer for the positive assessment, detailed review and the insightful questions. We appreciate reviewer’s recognition of the clear, well-structured presentation and strong writing, the effectiveness and design of our methodology, and the variety and extensiveness of our experiments. We are encouraged by the reviewer’s belief that our method has the potential for wide adoption, and we are grateful for their willingness to reconsider increasing their score based on our response. We will include all suggestions in the final version of our manuscript.

Evaluation Setup:

Throughout the response, we linearly align RoBERTa and ViT-G and report accuracy for classification and recall@1 for retrieval, unless otherwise stated.

Major questions:

1. Class-wise Errors and Additional Metrics

We chose the same evaluation setup as for CLIP, including benchmarks and evaluation metrics. However, we fully agree that using a metric that takes into account the class distribution is beneficial.

We report (1) the macro F1 score and (2) the standard deviation of the per-class F1 score to analyze the variance across classes. Results in terms of macro F1 score show that the performance, using our proposed framework, brings significant performance gains.

When comparing the standard deviation of the per-class F1 score, we observe that there is indeed a large variation in scores across classes. For CIFAR100, Food101, and ImageNet, the spread increases when using STRUCTURE, which is expected since the classes represented in the alignment dataset (here, COCO) will experience a performance increase, whereas the others will likely remain the same due to the lack of coverage.

As the reviewer suggested, we further looked for classes that are entirely missclassified. Without the proposed framework, 520 out of the 1000 ImageNet classes were completely misclassified. When using the proposed framework, 168 classes that were previously entirely unrecognized now achieve non‑zero scores (with an average per-class F1 score of 18.6), while 35 new classes that have previously been partially correctly classified (with an average per-class F1 score of 3.9) are now misclassified.

2. Additional Ablations

The intuition behind these specific choices is that we rely on the same distance function used within the standard contrastive objective, and by normalization, we remove any constant offset in the embedding space, which could otherwise dominate raw inner products or cosine similarities. Centering ensures that the subsequent similarity graphs reflect relative geometry.

We further experimented with different normalization schemes and distance functions. Overall, our method demonstrates robustness to these changes, highlighting that the primary benefit stems from the idea of regularizing hierarchical relationships.

3. Paired vs. Unpaired Samples for STRUCTURE

We evaluated the influence of using paired or unpaired samples for computing STRUCTURE. Specifically, for all experiments, we keep the alignment dataset the same but we vary the samples on which we compute the regularization. In the default setting (called paired) we compute the regularization on 5k samples that are also used to train the alignment function. In the second experiment, suggested by the reviewer, we compute the regularization on 10k samples that are disjoint from the alignment dataset where 50% of samples are used to compute regularization for the vision model, and other 50% for the language model (called unpaired). Results show that computing structure on unpaired samples yields comparable benefits as using paired samples. This shows that the structural prior that STRUCTURE maintains is generalizable to the whole original space.

4. Fixed vs. Random Subset for STRUCTURE

By default, for efficiency, STRUCTURE is computed on the current batch of samples, rather than on the entire dataset. We evaluated the influence of using a fixed same-size subset for computing STRUCTURE. Specifically, we compare the use of a fixed random sample from the dataset (fixed) with the standard approach of using the batches to compute the regularization term (random). Results show that the regularization yields very similar results regardless of whether it uses a fixed or a random subset of the data.

Minor Weakness:

1. Additional Baseline: ASIF

Unlike our method, ASIF $1$ does multimodal matching without learning. We now include ASIF as a baseline, and the results demonstrate that our method substantially outperforms it, achieving relative improvements of 32.9% on the COCO retrieval task, 32.9% on the CIFAR-100 fine-grained classification task, and 5.6% on the CIFAR-10 coarse-grained classification task.

2. Related Work

We have revised the related work section to provide a broader context for our work within the field of representation alignment $b, c, d, e$ . This updated section now positions the "Platonic representation hypothesis" within this more general framework.

Minor comments:

2. Definition of Resource Constrained

We use at least 1000 labeled samples in our experiments because, in the vision-language domain, this scale is often considered “resource-constrained” relative to the large datasets typically used. However, our method does not require 1000 labeled samples to be effective. For example, as shown in our response to Reviewer y6jJ (Q2), the results show that STRUCTURE remains beneficial with only 100 labeled samples in the biological domain.

4. Additional Experiment on “Train-test data distribution shift”

We conducted additional experiment of “Train-test data distribution shift” in which we added a mix of 10 samples for each class in each dataset. Results show that this consistently improves performance across all datasets, rather than introducing any noise.

5. Unifying Scale Up and Scale Down Experiment

We did not unify the scaling up and down experiments because they rely on different data used to train the alignment functions. Specifically, for scaling down, we use COCO, and for scaling up, we add LAION samples to COCO to increase the sample size while maintaining the core as the same as in the other experiments. Furthermore, we wanted to keep them separate as they investigate different behaviours of the proposed method. Specifically, the scaling-down investigation examines how little data is needed to improve the standard approach using standard contrastive methods. In contrast, the scaling-up investigation explores where the influence of regularization diminishes when sufficient samples are available.

6. Similarity Distribution of Mutual kNN

In general, for a given model pair, the most suitable layers are the layers with the highest mutual kNN score. However, we did not find a rule where these most suitable layers are found across different model pairs. Since the rebuttal policy forbids us from uploading images, we have added a table below that shows a subset of results. In particular, we aligned each of the last five layers from the vision and language encoder (RoBERTa and ViT-L) and evaluated their respective performance on the STL dataset. Results show that the most suitable layers are 24/22, and they perform significantly better than the others. Additionally, we report the alignment score in terms of mutual kNN in brackets for each combination as the similarity distribution. We will add comprehensive results across each layer as a heatmap in the revised manuscript.

1&3&7. Comments

We have updated the manuscript in response to comments 1, 3, and 7.