Learning Shared Representations from Unpaired Data

We thank the Reviewer for their thoughtful and encouraging feedback. We are pleased that you found the problem setting important and the theoretical motivation clear and well-explained. We also appreciate your recognition of the ablation study as a meaningful bridge between theory and empirical results. Below, we respond in detail to your comments and suggestions.

Q.1a: Additional modalities. We appreciate the Reviewer’s interest in the broader applicability of our method. In response, we have added results on a new multiview benchmark, the Handwritten dataset [1], which can be viewed as a tabular domain. This dataset consists of two handcrafted feature sets derived from images: (1) Karhunen–Loève coefficients (K); and (2) pixel averages over 2x3 windows (P), without the use of foundation models or pretrained embeddings. These experiments are thus fundamentally different from those in the main paper, and they provide evidence that SUE generalizes to non-vision-language settings. The table below extends Tab. 1 from the main paper and shows that SUE significantly outperforms the contrastive baseline on this dataset as well.

Notably, while our experiments mainly focus on vision-language and vision-vision settings, these domains were selected for their publicly available benchmarks, controllable pairing levels, and intuitive visualization, which support clear comparisons and understanding. We agree that extending SUE to other domains such as medical imaging is a compelling future direction. This requires further and deeper analysis, and we anticipate that SUE's ability to leverage unpaired data will provide clear advantages in those settings.

Q.1b: Unpaired scaling. We appreciate the reviewer’s suggestion to extend the scaling experiment in Fig. 5 to observe whether and when performance plateaus. However, Fig. 5b already uses the full available dataset, and unfortunately, no additional unpaired samples are available for this benchmark. Importantly, even at this point, SUE already significantly outperforms the baseline. Notably, performance continues to improve with the amount of unpaired data throughout the observed range. We will clarify this observation in the revised manuscript and view investigating the saturation regime as an important direction for future work with larger or synthetic datasets.

Q.2: Text encoder. Thank you for the observation. To address this point, we extended our encoder ablation study in App. A.4 to include a stronger and more recent text encoder, GTR [2]. The results are provided in the table below, extending Tab. 7 from the paper.. Interestingly, absolute performance is slightly lower. We hypothesize that the Mini-LM was already sufficient given the strength of the vision encoder (DINOv2), or alternatively, that GTR is less aligned with the image representations. In either case, the core effectiveness of our method appears robust to the choice of text encoder.

Q.3: Downstream task results. We thank the reviewer for this observation. This paper addresses the question: "What can be achieved when only a small number of pairs are available, alongside large unpaired datasets?" In this setting, standard paired models fall short. While SUE is not intended to compete with models such as CLIP trained on 400M image-text pairs, it offers a compelling alternative for low-pair regimes where such resources are unavailable. In this context, the downstream task performance achieved by SUE is significantly beyond what was previously possible. We will revise the manuscript to better highlight this setting and provide additional context for interpreting downstream results.

Q.4: Scaling. We thank the Reviewer for raising this important point. As discussed in previous responses, our work specifically addresses the underexplored and practically relevant setting where only a small number of paired samples are available alongside large amounts of unpaired data. In this scenario, existing contrastive methods like CLIP, which require hundreds of millions of pairs, are not applicable. Instead, our goal is to ask: What is achievable in the absence of large-scale paired datasets? In this regime, SUE offers a strong solution and significantly outperforms existing baselines (see additional baseline in the general comment). We agree that investigating whether performance continues to improve with more unpaired data is an important direction for future work.

Q.5: $k \geq r$ assumption. In our method, the original data from each modality lies in $\mathbb{R}^{d_1}$ and $\mathbb{R}^{d_2}$, respectively. These are first reduced to a common dimension $k$ via Spectral Embedding. Then, CCA is applied to project the data from $\mathbb{R}^d$ to a shared latent space of dimension $r$. Since CCA performs a projection onto the top $r$ canonical directions, we require $k \geq r$ to ensure the projection is well-defined. This assumption is standard in CCA-based methods.

Q.6: MMD network. The MMD residual network is intended as a final refinement step between two approximately aligned, equi-dimensional distributions. For simplicity and efficiency, we chose to apply the residual network only to $Y$, projecting it to better match $X$. An alternative would be to symmetrically transform both $X$ and $Y$ into a shared space that minimizes the MMD between them. While this could potentially improve alignment, we found the simpler asymmetric approach sufficient in our setting and left such architectural variations for future exploration.

Q.7: Universal manifold. The central assumption in our paper is that similarity graphs constructed from unimodal foundation models are sufficiently aligned across different modalities. We support this through extensive experiments in both vision-language and vision-vision domains. As shown in Fig. 2, the random walks and spectral embeddings exhibit strong cross-modal similarity. Furthermore, Sec. 5 empirically validates this assumption by demonstrating that SUE performs effectively across multiple tasks. This assumption is also supported by recent works that report consistent structural alignment between modalities, including text-image pairs [3, 4, 5, 6], as well as more diverse combinations such as audio-text, LiDAR-text, and timeseries-text [7]. While our results affirm the utility of this alignment in current domains, extending SUE to additional modalities is a promising direction for future research and will require further analysis.

Q.8: proven ability. Thank you for pointing this out. We will add a citation to [3], which directly demonstrates and discusses the semantic representation capabilities of unimodal models, supporting our claim.

Finally, thank you for pointing out these paper formatting improvements. We will include them in the revised version.

[1] Duin. Multiple Features. UCI Machine Learning Repository, 1998.

[2] Ni et al. Large dual encoders are generalizable retrievers. ACL 2022.

[3] Huh et al. Position: The platonic representation hypothesis. ICML 2024.

[4] Fan et al. Scaling language-free visual representation learning. arXiv 2025.

[5] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.

[6] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.

[7] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.

Thank you for the detailed rebuttal. I am convinced that the method is interesting and has a lot of potential. The additional experiments support the claim that SUE can be applied to other settings. However, as the other reviewers noted, the experimental section is lacking. I suggest the authors take a look at the paper "Assessing and Learning Alignment of Unimodal Vision and Language Models" by Zhang et al. (CVPR 2025). Like SUE, SAIL uses two pre-trained encoders from each modality and aligns them. Since both methods use contrastive learning, SAIL may require more paired data than SUE but this needs to be evaluated. Nevertheless, the SAIL paper's experiments are very comprehensive, involving image and text encoders of various sizes and model types which is nit given here. As such, I maintain my weak acceptance score.

I appreciate the additional last-minute experiments by the reviewer. The improvement over the latest SOTA model SAIL is indeed meaningful and further strengthens the results. In light of this, I change my initial assessment and increase the score to 5 (a more fitting score would be 4.5 as explained below). However, I strongly encourage the authors and appeal to more rigor my including some of the more extensive comparative studies provided in SAIL for a potential camera-ready version (using more model architectures, model sizes and downstream experiments such as few-show classification). I also agree with the other reviewers that the authors should include a limitation section talking about how the model might not work for weakly correlated modality pairs from other domains such as biology (e.g., single-cell images and gene expressions).

We thank the Reviewer for their detailed and constructive review. We are glad that you appreciated the intuitive justifications provided for our three-stage methodology. We also value your positive remarks on our visualizations, which aim to offer clearer insight into the embedding space, and on our extensive experimental evaluation across diverse domains, which supports the effectiveness of our approach. Below, we address your comments, add new results, and clarify specific points.

Q.1. We thank the reviewer for pointing out these relevant and timely works. In response, we have incorporated a comparison to CSA [3], which builds upon and improves over ASIF [2], itself based on [1]. Accordingly, we include all three works in the related work section and compare directly against CSA, the strongest among them.

As shown in our updated results below, SUE significantly outperforms CSA in the extremely low-pair regime we focus on. This highlights a key distinction: while CSA leverages limited paired data, it fails to operate in the setting we address, where the number of available pairs is extremely small, and unpaired data plays a central role.

Therefore, although these prior works explore related goals, to the best of our knowledge, ours is the first to demonstrate successful shared representation learning from almost exclusively unpaired data. We will emphasize this distinction and the expanded comparison in the revised version.

The table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Notably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \min\{d_1, d_2\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.

We will include these results in the revised version of the manuscript.

Q.2. The paper includes an ablation study on how changing the unimodal encoders affects the retrieval performance, in App. A.4. It is referred to from Sec. 5.1.

Q.3. The architecture details for all components, including the MMD network, are provided in Appendix D.8. As discussed in Section 4.1, the MMD network is trained entirely on unpaired data, which is available in large quantities. It does not rely on paired supervision, so the limited number of pairs is not a limiting factor here. However, MMD alignment (especially using a residual network) assumes that the input distributions are already roughly aligned. For this reason, the MMD step acts only as a final fine-tuning stage, building on the alignment achieved by the previous pipeline components. As shown in Table 2 and Appendix A.1, MMD alone fails without this prior structure, reinforcing its role as a lightweight refinement rather than a standalone solution.

Finally, thank you for pointing out these minor issues. We will address them in the revised version.

[1] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.

[2] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.

[3] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.

We thank the Reviewer for their insightful review. We are pleased that you found the paper to be well-written and addressing an important topic. We appreciate your recognition of the clear motivation behind our problem setting, as well as the value of our ablation studies and visualizations, which aim to validate our approach across both vision-language and vision-vision tasks. Below, we respond to your comments and provide further results and clarifications where needed.

W.1. The assumption made in the paper, that similarity graphs, derived from unimodal foundation models are sufficiently aligned across different modalities, is supported in the paper with extensive experiments in the vision-language and vision-vision domains. As shown in Fig. 2, the random walks and spectral embeddings exhibit strong cross-modal similarity. Furthermore, Sec. 5 empirically validates this assumption by demonstrating that SUE performs effectively across multiple tasks. This assumption is also supported by recent works that report consistent structural alignment between modalities, including text-image pairs [1, 2, 3, 4], as well as more diverse combinations such as audio-text, LiDAR-text, and timeseries-text [5]. Notably, while text and images may each capture modality-specific nuances, prior work demonstrates that their global semantic structures remain well-aligned.

W.2. Similarly to our answer above, perhaps surprisingly, we show that such a unified latent semantic manifold exists to some extent. While text and images may each capture modality-specific nuances, they still describe the same semantic objects, and thus share their global semantic structure. This assumption is supported by previous work [1, 2, 3, 4, 5].

W.3. Thank you for this comment. In Section 4.1, we support the argument that similar random walks imply aligned eigenfunctions by referencing Theorem 21 from [6]. Specifically, the result shows that if two embeddings $f,g$ of a common manifold $\mathcal{M}$ have bounded distortion and bounded Ricci curvature, then the eigenfunctions of the associated Laplace-Beltrami (or diffusion) operators on $f(\mathcal{M})$ and $g(\mathcal{M})$ are close in the $L_{\infty}$ sense. For completeness, we will add the theorem to the appendix.

This provides a rigorous justification for our assumption in the setting where the observed data lie on geometrically similar manifolds. While these assumptions may not hold perfectly in all practical cases, they offer a strong theoretical foundation for why alignment via spectral structure is both plausible and effective, especially when supported by empirical evidence such as the graph similarities shown in Fig. 2.

W.4. We appreciate the Reviewer’s interest in the broader applicability of our method. In response, we have added results on a new multiview benchmark, the Handwritten dataset [7], which can be viewed as a tabular domain. This dataset consists of two handcrafted feature sets derived from images: (1) Karhunen–Loève coefficients (K); and (2) pixel averages over 2x3 windows (P), without the use of foundation models or pretrained embeddings. These experiments are thus fundamentally different from those in the main paper, and they provide evidence that SUE generalizes to non-vision-language settings. The results (see table in W.5 below) show that SUE significantly outperforms the contrastive baseline on this dataset as well.

Notably, while our experiments mainly focus on vision-language and vision-vision settings, these domains were selected for their publicly available benchmarks, controllable pairing levels, and intuitive visualization, which support clear comparisons and understanding. We agree that extending SUE to other domains such as medical imaging is a compelling future direction. This requires further and deeper analysis, and we anticipate that SUE's ability to leverage unpaired data will provide clear advantages in those settings.

W.5. Following the Reviewer’s suggestion, we have compared SUE to an additional baseline below. Specifically, we have added a comparison to CSA (proposed by Reviewer M2P3). CSA [5] is a recently introduced method designed to match CLIP performance while reducing reliance on paired data. The table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Importantly, SUE significantly outperforms this new baseline in the extremely low-pair regime we focus on. We hope that the additional comparison provides better context for SUE's performance.

Notably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \min\{d_1, d_2\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.

W.6. We thank the reviewer for raising this important point. The tradeoff between the amount of unpaired and paired data may be extracted from in Fig. 5b and Tab. 1. For instance, on the Flickr30k image retrieval task, the contrastive baseline achieves an R@10 of approximately 15 using 500 paired examples. SUE reaches comparable performance using around 15,000 unpaired samples. Importantly, as the amount of unpaired data increases, SUE's performance continues to improve, without requiring additional paired supervision. We will clarify this insight in the revised manuscript.

Q.1. Please see our response to W.4.

Q.2. We agree with the reviewer that the MMD-based alignment step is most effective when the modality-specific embeddings are already roughly aligned. As shown in our experiments (e.g., Tab. 1 and Tab. 2), this initial alignment is relatively strong, which allows the final MMD step to refine rather than create alignment. We also confirm the reviewer’s intuition that MMD alone is insufficient to bridge the modalities without prior alignment: our ablation study in Tab. 2 demonstrates that using MMD without the spectral embedding and CCA steps results in significantly worse performance.

Q.3. Please see "Data Preparation" paragraph in App. D.7.

Q.4. We thank the reviewer for the suggestion. We have computed mean Average Precision (mAP) results, and summarized them in the table below. SUE outperforms the baselines under this ranking-sensitive metric as well, further validating the effectiveness of our approach.

Q.5. We thank the reviewer for raising this point. If we understand correctly, the question refers to the use of informative example mining to improve the performance of contrastive learning. While this is a powerful technique in fully supervised settings, it is not applicable in our scenario. Our work explicitly focuses on the low-pair regime, where only a small, fixed set of aligned pairs is available, and additional pair selection or mining is not possible.

This setting reflects real-world constraints in weakly supervised multimodal learning, where collecting or identifying informative pairs is either expensive or infeasible. Therefore, the contrastive baselines we compare against operate under the same limited supervision, and our claim that contrastive learning requires an order of magnitude more pairs holds under this constraint.

[1] Huh et al. Position: The platonic representation hypothesis. ICML 2024.

[2] Fan et al. Scaling language-free visual representation learning. arXiv 2025.

[3] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.

[4] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.

[5] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.

[6] Bérard et al. Embedding riemannian manifolds by their heat kernel. GAFA 1994.

[7] Duin. Multiple Features. UCI Machine Learning Repository, 1998.

We thank the Reviewer for their thoughtful and positive feedback. In particular, we are encouraged by the acknowledgment of the importance and originality of the problem we tackle, and the clarity and motivation underlying our methodology. We also appreciate your recognition of the extensive empirical validation of our approach, as well as the insightful ablation studies that demonstrate the contribution of each component. Below, we address your comments in detail, add new results, and clarify the points you raised.

W.1

Baseline comparison. Following the Reviewer’s suggestion, we have compared SUE to an additional baseline below. Specifically, we have added a comparison to CSA (proposed by Reviewer M2P3). CSA [1] is a recently introduced method designed to match CLIP performance while reducing reliance on paired data. The table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Importantly, SUE significantly outperforms this new baseline in the extremely low-pair regime we focus on. We hope that the additional comparison provides better context for SUE's performance.

Notably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \min\{d_1, d_2\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.

We will include these results in the revised version of the manuscript.

References. We thank the reviewer for pointing us to relevant references. Both will be cited and discussed in the revised related work section. However, similar to the other methods already mentioned in the related work, these approaches assume access to additional supervision or large-scale paired training data. For example, [2] incorporates class-level labels into the alignment process, whereas our setting is label-free. [3] depends on multimodal foundation models trained on hundreds of millions of paired examples, which falls outside the minimal-pair regime we consider. As such, while informative, these methods are not directly comparable to SUE.

AE ablation. Our framework operates on pre-computed semantic embeddings (e.g., from Sentence-Transformers or DINOv2), not raw input data. In this setting, applying SimCLR or related contrastive methods is non-trivial: these methods are designed for raw data and rely heavily on stochastic augmentations to define positive/negative pairs, an approach that does not translate naturally to pre-extracted embedding spaces. Moreover, the contrastive paradigm assumes access to meaningful augmentations or label structure to guide the loss. In contrast, the autoencoder serves as a natural and neutral baseline in the embedding space, offering a non-spectral method that reconstructs input embeddings without introducing alignment or graph structure. This allows us to isolate and highlight the specific contribution of our spectral learning component.

W.2. Coarse-Grained Alignment: We thank the reviewer for this insightful observation. Our work addresses the question: "What can be achieved when only a small number of cross-modal pairs are available alongside large unpaired datasets?" In this low-pair regime, standard paired models are inapplicable or ineffective. SUE provides a principled alternative that leverages the intrinsic structure of each modality for alignment, achieving downstream performance way beyond what was previously possible.

We agree with the Reviewer that our approach, by prioritizing global manifold structure through spectral embeddings, may naturally trade off some fine-grained detail, in favor of broader semantic alignment. We view this as an inherent and interesting property of our method, and believe that a deeper investigation of this trade-off between global alignment and local fidelity is a valuable direction for future work. We will revise the manuscript to clarify this point and provide better context for interpreting these qualitative results.

Q.1

Feature extraction details. For the text modality, we use Sentence-Transformers (e.g., all-MiniLM), which produce fixed-length sentence embeddings via mean pooling as implemented in the library. For the image modality, we extract features using DINOv2, where the [CLS] token serves as the image embedding vector by default. We will add these to the technical details appendix.

On the potential confounding effect. We acknowledge that pre-trained encoders introduce semantic biases that may facilitate cross-modal alignment, as reflected in Fig. 2. However, rather than viewing this as a hidden confounder, we consider this semantic preprocessing a deliberate and essential design choice. By embedding modalities into meaningful feature spaces, SUE can effectively leverage unpaired data for alignment. Attempting alignment from raw pixels and tokens without such semantic abstraction would demand far more paired supervision, contradicting our goal of minimal pairing.

Information density and alignability. Regarding the comment that images contain higher information density than captions, our method capitalizes on the semantic abstraction of these encoders. Both DINOv2 and Sentence-Transformers distill inputs into embeddings, emphasizing relevant semantic content while reducing irrelevant details such as background noise. This abstraction actually enhances alignability, as supported by the similar graph structures observed in Fig. 2, demonstrating that both modalities capture analogous semantic relationships conducive to spectral alignment.

Q.2. Please see our response to W.1.

Q.3 Modality gap. Thank you for your interest in our discussion of the modality gap. This finding is not limited to the UMAP visualization in Fig. 14, it holds consistently across datasets and embedding dimensions. It arises from two core components of our method: (1) CCA produces whitened outputs, ensuring that the representations for each modality are centered at the origin, which implies zero modality gap by definition; and (2) the MMD objective penalizes distributional discrepancies, further reducing any residual mismatch between modalities. We have verified this behavior across all datasets and will include the corresponding visualizations in the revised version. Due to space constraints, many of our extensive experiments were placed in the appendix; we will highlight key results, including the modality gap analysis, more prominently in the main paper.

Finally, thank you for pointing out these minor issues. We will address them in the revised version.

[1] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.

[2] Leeb et al. Partial alignment of representations via interventional consistency. ICLR 2025.

[3] Wang et al. Omnibind: Large-scale omni multimodal representation via binding spaces. ICLR 2025.

I have raised my score to reflect the authors' strong rebuttal, which directly addressed several of my concerns by adding a several new empirical results and discussion. However, I believe fully integrating the additional discussion, clarifications, and results may benefit from a resubmission. Nevertheless, the work is very promising.