The Indra Representation Hypothesis

Dear Reviewer 7Tyf,

We sincerely appreciate you for providing thoughtful and constructive comments.

We are glad to see you recognize our idea as novel and elegant, providing a thought-provoking new lens to view the properties of embeddings of foundation models.

Below, we address your remaining concerns.

W1&Q1 Scalability & Complexity

[Complexity of Exact Computation] We agree with you that constructing exact Indra representations requires a computational complexity of $\mathcal{O}(n^2d)$ and a memory complexity of $\mathcal{O}(n^2)$ for a dataset with $n$ samples and embedding dimension $d$. This quadratic scaling potentially limits the direct applicability of the exact Indra representations to large-scale datasets.

[Our Research Focus] However, we respectfully clarify that the goal of this work is not to propose a new application-driven multi-modal alignment method. Rather, our focus lies in understanding the fundamental nature of the representations learned by existing unimodal foundation models. Specifically, we aim to investigate the forms these representations tend to converge to and to identify their ultimate convergent targets.

In this paper, we argue that unimodal foundation models converge toward a form of Indra representation, which implicitly reflects the relational structure underlying reality (i.e., The Indra Representation Hypothesis). All our experiments are designed to validate the effectiveness of this form.

[Application-Oriented Solutions] Moreover, the scalability concern is addressable in practice. In the literature, there exists a rich body of work on approximating pairwise distances efficiently. For example, approximate nearest neighbor search (e.g., FAISS, HNSW), landmark-based approximation (e.g., K-means centroids, random subsampling), hashing-based methods (locality sensitive hashing), and sparsified graph constructions (as suggested by Reviewer 6ioa). From an application view, these techniques can be readily adapted to approximate the Indra representation at scale without sacrificing its structural interpretation.

[Experimental Validation] To evaluate the effectiveness of approximated Indra representations, we adopt a simple landmark-based approximation via random subsampling. We assess this approach on two tasks: image classification on the CIFAR-100 dataset and image-text matching on the large-scale CC3M dataset. For CIFAR-100, we use the standard data split provided by torchvision.datasets and employ logistic regression (linear probing) to evaluate classification accuracy. For CC3M, we sample 300,000 image-text pairs for evaluation. We extract image features with DINOv2 and text features with RoBERTa. For each image, we compute mean CLIP scores with the top-$k$ candidate texts and compare the performance of the original features against that of the approximated Indra representations.

Due to space limitation, we kindly refer you to our response to Reviewer yaBd (W1 & Q3) for the experimental details and results.

W2&Q2 Distance metric

We believe there is a misunderstanding regarding the role of enriched category theory in our method. The theory is not intended to guide the selection of an optimal cost function for practical applications. Instead, its purpose is to establish that the Indra representation obtained through our construction is both faithful and complete with respect to the chosen cost function. The Yoneda Lemma indeed cannot prescribe an optimal cost function for the downstream tasks.

In practice, it is not feasible to define a universally optimal cost function across different tasks. As our experimental results demonstrate, angular distance may outperform Euclidean distance in some settings, while the opposite may be true in others. This highlights that the choice of cost function should be task-dependent, and the theory is not meant to replace empirical validation.

Furthermore, not all distance functions are valid cost functions in our framework. To be used as a cost function for the Indra representation, the cost function should satisfy the following properties:

Reflexivity: $d(x, x) = 0$
Non-negativity: $d(x, y) \in [0, \infty)$
Triangle inequality: $d(x, z) \leq d(x, y) + d(y, z)$

In summary, enriched category theory is not used to justify the method post hoc, but to guarantee structural faithfulness and completeness within a mathematically grounded setting.

W3& Baselines

Existing alignment methods, e.g., [50], need additional modules on top of frozen backbones, such as linear projections and adapters, and require substantial fine-tuning on large-scale datasets to learn alignment functions. Once trained, these models become explicitly aligned and are thus more appropriately compared to models like CLIP, which are also designed with explicit alignment.

In contrast, our work aims to investigate the convergent forms and targets of representations learned by unimodal foundation models. Our objective fundamentally differs from that of alignment-driven methods: while they actively design and train new modules to enforce alignment, our approach passively explores whether frozen models already contain a more suitable internal representation that supports better cross-modal alignment.

We uncover that even without any additional training, unimodal foundation models may admit a more optimal representational form than their raw output features. This key difference in motivation, methodology, and training requirements makes a direct empirical comparison with those methods unfair and potentially misleading.

To the best of our knowledge, this is the first work to demonstrate that frozen representations without any task-specific tuning can achieve improved alignment performance through our proposed representation method, not only within a single modality, but also across vision-language and speech-language modalities.

Q1 Kernel approximation methods

We kindly remind the reviewer that the primary goal of this paper is to investigate the representational forms that unimodal foundation models converge to, and to identify their ultimate convergent targets, rather than to accelerate the proposed method through approximation.

In this work, we hypothesize that these models tend to converge toward a form we define as the Indra representation. Accordingly, all our experiments are designed to validate the effectiveness of this representation, demonstrating that it consistently achieves better alignment than the original model outputs without any additional training.

In fact, using approximation techniques (e.g., kernel approximation methods) to construct the Indra representation and empirically demonstrate its performance primarily serves as a validation of the Indra representation hypothesis introduced in this paper.

Q3 Structural myopia & self-attention

Prior studies on representation convergence often treat the representations produced by foundation models as isolated carriers of information. They analyze convergence purely based on model outputs in a pointwise or instance-level manner. This perspective overlooks the underlying structural relationships embedded within the broader data manifold. In contrast, we argue that model outputs do not directly represent the final converged form, but rather provide the initial scaffold upon which convergence should be understood. We contend that truly convergent representations must be structure-dependent.

While both our method and self-attention involve pairwise relationships, they differ fundamentally in nature and purpose. Our method constructs a representation for each sample based on its static, global relationship to others in the dataset, using a predefined cost function. This results in a complete, non-parametric embedding that faithfully captures the sample's position within the overall data geometry. The representation is invariant across contexts and tasks, and fully determined by the relational structure of the data itself.

In contrast, self-attention models dynamic, context-dependent interactions among tokens, where the strength of each relationship is learned via parameterized query-key-value projections and varies across different inputs and layers. Self-attention embeddings are not faithful to the original data geometry; instead, they adapt to downstream objectives and are shaped by the model's learned parameters.

In short, our method is grounded in global structure and faithful representation, while self-attention captures how a token should change based on its current context and specific tasks.

L1 Limitation section revision

Thank you for the thoughtful comments. Our experimental results validate the hypothesis by demonstrating improved alignment performance using frozen unimodal foundation models. We believe it is also meaningful to explore this hypothesis in the pretraining stage, to investigate how learning dynamics evolve when the training objectives align with the representational convergent targets.

We have addressed related concerns in the responses above and will revise the limitations section accordingly. In particular, we will clarify our focus on theoretical insights and empirical findings, explicitly acknowledge the computational challenges associated with computing exact Indra representations, and include a discussion of scalable approximation strategies along with the corresponding experimental results.

We hope the above response adequately addresses your concerns. Once again, thank you for your thoughtful feedback and for helping us improve the quality of our paper.

We thank the reviewer for the thoughtful feedback and would like to clarify a few points that may have been misunderstood:

1. Our primary goal is not to propose a new alignment method, but rather to demonstrate that the proposed Indra representations exhibit stronger convergence behavior than the original network outputs.

Specifically, we study the representation convergence problem, where we investigate the nature of the representations to which unimodal foundation models tend to converge. Through experiments across various modalities, we show that the original network outputs may not be the most suitable endpoints for representation convergence. Instead, Indra representations serve as more consistent and convergent targets.

That said, we agree with the reviewer that including comparisons with standard alignment methods such as learnable linear projections is valuable for contextualizing our findings. In response, we have now included comparisons with the linear projection baseline [50]. The classification and matching results are presented in Tables 1, 2, and 3, respectively.

As shown in our results, the learnable Linear Projection baseline improves alignment performance compared to the original network outputs, but still underperforms relative to our method. Notably, it achieves stronger results on the image-to-text (I->T) task, because it explicitly projects image features into the text space. However, this forced alignment comes at a cost: aligning image representations to a different modality degrades their utility for classification, particularly under noisy conditions. This indicates reduced robustness and suggests that linear projection may compromise the integrity of the original modality's semantics.

2 We would like to clarify that canonical correlation analysis is not a non-parametric, training-free alignment method.

CCA computes explicit linear projections that map two datasets into a shared latent space. These projection parameters are derived by solving a generalized eigenvalue problem based on sample covariance matrices, which have a closed-form solution. Although it doesn't involve gradient descent, CCA still involves fitting to the training data and requires estimating projection parameters that depend on the input data.

We thank the reviewer for suggesting a concrete method for us to compare, and we have added comparisons with CCA. The results are shown in Table 4, including comparisons with different numbers of canonical components (e.g., CCA-100 refers to using 100 canonical components).

Please refer to the next page for the continuation of our response.

3 We respectfully emphasize that our method addresses a fundamentally different research question from the standard alignment approaches referenced by the reviewer.

Methods such as learnable projections and CCA are explicitly designed to optimize alignment between modalities through training or closed-form solutions. These methods aim to answer the question: “How can we better align two representations?”

In contrast, our work investigates the convergent behavior of unimodal foundation models. Specifically, what form their representations tend to converge to. We ask whether the proposed Indra representations can serve as the convergent targets, and naturally yield better alignment without requiring any additional training or projection. That means, our work seeks to answer a distinct question: “Should Indra representations be considered the final convergent targets of unimodal foundation models?”

4 The reviewer’s comment may reflect a misunderstanding of the role that enriched category theory plays in our work.

Our proposed Indra representation is motivated by the metaphor of Indra’s Net: each sample is defined not in isolation, but through its relational pattern with other samples. The enriched Yoneda embedding formalizes this idea. It shows that an object in an enriched category is fully and faithfully represented by its morphisms to and from all other objects, with respect to the given enrichment. In our setting, an object is fully determined by how it relates to all other objects in a category given the cost function. In other words, this theory tells us we can understand an object best by understanding its interactions with all other objects.

We respectfully disagree with the claim that this theoretical structure is merely justificatory. While the completeness result may be partially intuited via classical metric geometry, the enriched Yoneda embedding generalizes beyond metric spaces: any suitable monoidal structure can serve as the enrichment. This flexibility is crucial for extending beyond metric embeddings, and the enriched framework ensures that the relational representation is coherent and principled in all such cases.

Indeed, metric spaces can be viewed as a special case of enriched categories. While classical metric geometry can ensure that certain representations are complete with respect to a chosen distance function, the enriched Yoneda embedding goes further. It is not merely about comparing objects or measuring distances. It’s about reconstructing an object’s identity entirely through its relationships with others, which is central to the philosophy and implementation of our approach. In contrast, standard metric embeddings only glimpse part of that.

We hope the above response adequately addresses your concerns. Once again, thank you for your thoughtful feedback and for helping us improve the quality of our paper. Please feel free to let us know if any part of our response remains unclear or if you have any further questions!

Dear Reviewer 6ioa,

We sincerely appreciate you for providing useful suggestions and for recognizing our idea as novel and compelling.

Below, we address your remaining concerns.

W1&Q2&Q3 Complexity

[Complexity of Exact Computation] Due to space limitations, we kindly refer you to our response to Reviewer yaBd (W1 & Q3) for the complexity analysis.

[Application-Oriented (Scalable) Solutions] The scalability concern is addressable in practice. In the literature, there exists a rich body of work on approximating pairwise distances efficiently. For example, approximate nearest neighbor search (e.g., FAISS, HNSW), landmark-based approximation (e.g., K-means centroids, random subsampling), hashing-based methods (locality sensitive hashing), and sparsified graph constructions. From an application view, these techniques can be readily adapted to approximate the Indra representation at scale without sacrificing its structural interpretation.

[Experimental Validation] Due to space limitations, we kindly refer you to our response to Reviewer yaBd (W1 & Q3) for the corresponding results and analysis.

[Runtime Analysis] The table below reports the runtime for constructing Indra representations on the CC3M dataset using a landmark-based approximation with varying numbers of landmarks. All experiments are conducted on an NVIDIA L40S GPU, and each setting is run 20 times to obtain the average runtime.

[Theoretical Guarantees] Approximate solutions, such as landmark-based methods, may not guarantee uniqueness, completeness, or structure preservation, unless the selected subcategory of landmarks is sufficiently large or dense to distinguish all objects in our sample category. In such cases, we may consider leveraging categorical tools, such as Kan extensions, to extend the behavior of objects defined on the subcategory to the entire category, thereby approximating the full Indra representation. However, this direction lies beyond the scope of the current paper. We will explore this in our future work.

W2 Intuition on captured relations

Thank you for the helpful suggestion. The types of relations captured by our method are influenced by both the choice of cost function and the foundation model, and are not deterministic. To better understand these relations, we perform visualization analyses using t-SNE and clustering algorithms. Due to rebuttal policy constraints, we are unable to include qualitative visualizations at this stage, but we will incorporate them in the revised version of the paper.

W3&Q1&Q5 Other downstream tasks & robustness

We additionally report image classification results on the CIFAR-10, CIFAR-100, and Office-Home datasets under various distance metrics. For CIFAR-10 and CIFAR-100, we use the standard data splits provided by torchvision.datasets. For Office-Home, we evaluate classification accuracy across four domains: Art (A), Clipart (C), Product (P), and Real-World (R), using an 80/20 split for training and testing. Across all datasets, we adopt logistic regression (linear probing) to assess the quality of the extracted representations.

To investigate the robustness of noisy representations, we inject Gaussian noise into the features with varying standard deviations $\sigma \in \{0.0, 1.5, 3.0, 5.0, 7.0\}$. For each noise level, we perturb the features accordingly and train a linear classifier on the noisy representations. This allows us to assess how classification performance degrades as the feature representations are increasingly corrupted by noise.The corresponding results are presented below.

The results clearly show that stronger backbone models (e.g., DINOv2) lead to better performance for Indra representations across all noise levels. For instance, on CIFAR-100 with $\sigma=0.0$, Indra (Euclidean) achieves 91.93% accuracy using DINOv2 features, compared to 85.64% with ConvNeXt and 80.09% with ViT. This performance gap persists and even widens under higher noise; at σ=7.0\sigma=7.0σ=7.0, Indra (Euclidean) with DINOv2 maintains 58.67%, while ConvNeXt and ViT drop to 30.25% and 32.74%, respectively. In addition, as Gaussian noise increases, our Indra representations consistently retain higher classification accuracy compared to the original features, highlighting their robustness. The performance gains of Indra representations hold across multiple backbone architectures (ViT, ConvNeXt, DINOv2), indicating the broad applicability of the proposed method.

W4&Q6 Baselines

The cited methods such as [50] are application-driven with a primary goal of improving cross-modal alignment. They achieve this goal by training additional modules on top of frozen backbones, such as linear projections and adapters. As a result, these methods typically require substantial fine-tuning on large-scale datasets to learn optimal alignment mappings. Once trained, they become explicitly aligned models and are therefore more appropriately compared to models like CLIP, which are also designed with explicit alignment.

In contrast, our work is exploratory in nature, with the main focus on investigating what kind of representation unimodal foundation models inherently converge to. While prior works suggest that different unimodal models exhibit convergence in their output representations, they generally assume that the last-layer outputs themselves constitute this convergent form. We challenge this assumption and hypothesize that there exists a more structured and alignment-friendly representation, i.e., what we propose as the Indra representation. To validate this hypothesis, we compare Indra representations with the original model outputs, showing that Indra representations lead to better alignment performance without any model retraining. It is therefore clear that our objective fundamentally differs from that of those methods. While they proactively design and train new modules to enforce alignment, our approach seeks to passively uncover whether existing, frozen models already contain a more suitable representation structure when viewed from a relational perspective. In summary, while these alignment methods achieve improved performance through training additional layers, our method reveals that even without training, unimodal foundation models may admit a more optimal representational form than their raw outputs. This key difference in motivation, methodology, and training requirements makes a direct empirical comparison with those methods unfair and potentially misleading.

Q4 Qualitative examples

Thank you for the constructive feedback. We have conducted qualitative comparisons demonstrating that Indra representations yield more semantically aligned matches than the original embeddings. Due to rebuttal policy constraints, we are unable to include qualitative examples at this stage, but we will incorporate them in the revised version of the paper.

We hope the above response adequately addresses your concerns. Once again, thank you for your thoughtful feedback and for helping us improve the quality of our paper.

Dear Reviewer 6ioa,

Thank you very much for your time and effort in reviewing our paper.

As the discussion deadline approaches, we would greatly appreciate your thoughts on whether our responses have addressed your concerns. Your feedback is extremely valuable to us, and we would be grateful for any further comments or suggestions you might have.

Thank you once again for your time and consideration!

Sincerely,

The Authors

Dear Reviewer yaBd,

We sincerely appreciate you for providing helpful suggestions and for recognizing our idea as fresh and distinguishable from traditional approaches.

Below, we address your remaining concerns.

W1&Q3 Limitation discussion

[Complexity of Exact Computation] Constructing exact Indra representations requires a computational complexity of $\mathcal{O}(n^2d)$ and a memory complexity of $\mathcal{O}(n^2)$ for a dataset with $n$ samples and embedding dimension $d$. This quadratic scaling potentially limits the direct applicability of the exact Indra representations to large-scale datasets.

[Application-Oriented (Scalable) Solutions] The scalability concern is addressable in practice. In the literature, there exists a rich body of work on approximating pairwise distances efficiently. For example, approximate nearest neighbor search (e.g., FAISS, HNSW), landmark-based approximation (e.g., K-means centroids, random subsampling), hashing-based methods (locality sensitive hashing), and sparsified graph constructions (as suggested by Reviewer 6ioa). From an application view, these techniques can be readily adapted to approximate the Indra representation at scale without sacrificing its structural interpretation.

[Experimental Validation] To evaluate the effectiveness of approximated Indra representations, we adopt a simple landmark-based approximation via random subsampling. We randomly select $m < n$ landmarks as representative samples, reducing the computational complexity to $\mathcal{O}(nmd)$ and the memory complexity to $\mathcal{O}(nm)$. We assess this approach on two tasks: image classification on the CIFAR-100 dataset and image-text matching on the large-scale CC3M dataset.

For CIFAR-100, we use the standard data split provided by torchvision.datasets and employ logistic regression (linear probing) to evaluate classification accuracy. To assess the robustness, we inject Gaussian noise with different levels of standard deviation $\sigma$ into the extracted features.

For CC3M, we sample 300,000 image-text pairs for evaluation. We extract image features with DINOv2 and text features with RoBERTa. For each image, we compute mean CLIP scores with the top-$k$ candidate texts and compare the performance of the original features against that of the approximated Indra representations.

Results under varying numbers of landmarks are presented below.

The experimental results demonstrate that the approximated Indra representations still achieve better classification and alignment performance for real-world applications. Specifically, the Indra representations consistently outperform the original representations under increasing levels of Gaussian noise, particularly at higher noise levels, highlighting their superior robustness. Furthermore, increasing the number of landmarks may make improvements in classification accuracy.

W2 &Q1 Proof of Theory 2

Thank you for carefully reviewing the proof. You are absolutely right, our statement that “the cost function $d$ satisfies the identity property” should be revised to “the cost function $d$ satisfies the T₀ separation axiom.” The identity property, which requires that $d(X_i,X_j)=0$ does not imply $X_i=X_j$. We will revise the theorem and proof accordingly to incorporate a stricter condition on the cost function.

Q2 Downstream tasks

We additionally report image classification results on the CIFAR-10, CIFAR-100, and Office-Home datasets under various distance metrics. For CIFAR-10 and CIFAR-100, we use the standard data splits provided by torchvision.datasets. For Office-Home, we evaluate classification accuracy across four domains: Art (A), Clipart (C), Product (P), and Real-World (R), using an 80/20 split for training and testing. Across all datasets, we adopt logistic regression (linear probing) to assess the quality of the extracted representations. To evaluate robustness, we inject Gaussian noise into the features with varying standard deviations $\sigma$. The corresponding results are presented below.

L1 Limitation section revision

Thank you for the constructive comments. We have addressed these potential issues in the discussion above and will revise the limitations section accordingly. Specifically, we will explicitly acknowledge the computational challenges of computing exact Indra representations and include a discussion of scalable approximation strategies, supported by corresponding experimental validation.

We hope the above response adequately addresses your concerns. Once again, thank you for your thoughtful feedback and for helping us improve the quality of our paper.

Dear Reviewer yaBd,

Thank you very much for recommending acceptance!

We truly appreciate your recognition of our novelty and your thoughtful feedback. Your comments are highly valuable, and we will do our best to revise the manuscript accordingly. Once again, thank you for helping us improve the quality of our work!

Best regards,

The Authors

Dear Reviewer enB4,

We sincerely appreciate you for providing constructive comments and are glad to see you recognize our idea as highly novel and intellectually compelling.

Below, we address your remaining concerns.

W1 Practical utility

Thank you for providing constructive suggestions. We would like to clarify that the primary goal of this work is not to demonstrate that our proposed Indra representations outperform well-aligned models such as CLIP or GPT-4o in practical applications, nor is it to argue that such models are insufficient.

Instead, our core contribution is to investigate what form the learned representations of large-scale unimodal foundation models converge to, after training on unimodal data. Prior studies have shown that such models tend to converge in representation space, but they do not explain what the representations ultimately converge to. Our work offers a new representation convergence hypothesis and provides empirical results to validate this hypothesis.

The practical utility of our proposed Indra representation lies in its ability to improve cross-modality alignment without the need of additional adapter or large-scale finetuning. In domains like medicine or other specialized fields, there may not exist multimodal foundation models, nor is it always feasible to train them. In such settings, leveraging existing unimodal models through our Indra representation offers a compelling solution for cross-modal tasks.

As demonstrated in our experiments, the proposed Indra representation improves performance not only in unimodal tasks, but also across vision-language and speech-language modalities. Furthermore, the added classification experiments show that our representation also enhances classification accuracy and robustness (kindly refer to our response to Reviewer yaBd (W1 & Q3)).

In summary, our work is not intended to replace well-aligned models but to uncover a foundational mechanism that explains how unimodal models can be structurally leveraged or adapted for cross-modal settings, especially where aligned models are unavailable or infeasible.

W2 Notation

We appreciate the reviewer’s feedback regarding the use of numbered notes. We understand that this stylistic choice may come across as informal or inconsistent with conventional academic norms. We have revised the manuscript to adopt a more standard academic style. We thank the reviewer for highlighting this and will ensure the final version adheres to expected standards.

W3 Scalability and practicality

[Application-Oriented (Scalable) Solutions] The scalability concern is addressable in practice. In the literature, there exists a rich body of work on approximating pairwise distances efficiently. For example, approximate nearest neighbor search (e.g., FAISS, HNSW), landmark-based approximation (e.g., K-means centroids, random subsampling), hashing-based methods (locality sensitive hashing), and sparsified graph constructions (as suggested by Reviewer 6ioa). From an application view, these techniques can be readily adapted to approximate the Indra representation at scale without sacrificing its structural interpretation.

[Experimental Validation] To evaluate the effectiveness of approximated Indra representations, we adopt a simple landmark-based approximation via random subsampling. We randomly select $m < n$ landmarks as representative samples, reducing the computational complexity to $\mathcal{O}(nmd)$ and the memory complexity to $\mathcal{O}(nm)$. We assess this approach on two tasks: image classification on the CIFAR-100 dataset and image-text matching on the large-scale CC3M dataset.

For CIFAR-100, we use the standard data split provided by torchvision.datasets and employ logistic regression (linear probing) to evaluate classification accuracy. To assess the robustness, we inject Gaussian noise with different levels of standard deviation $\sigma$ into the extracted features.

For CC3M, we sample 300,000 image-text pairs for evaluation. We extract image features with DINOv2 and text features with RoBERTa. For each image, we compute mean CLIP scores with the top-$k$ candidate texts and compare the performance of the original features against that of the approximated Indra representations.

Results under varying numbers of landmarks are presented below.

The experimental results demonstrate that the approximated Indra representations still achieve better classification and alignment performance for real-world applications. Specifically, the Indra representations consistently outperform the original representations under increasing levels of Gaussian noise, particularly at higher noise levels, highlighting their superior robustness. Furthermore, increasing the number of landmarks may make improvements in classification accuracy.

We hope the above response adequately addresses your concerns. Once again, thank you for your thoughtful feedback and for helping us improve the quality of our paper!

Dear Reviewer enB4,

Thank you very much for your time and effort in reviewing our paper.

As the discussion deadline approaches, we would greatly appreciate your thoughts on whether our responses have addressed your concerns and if they have helped in re-evaluating our work. Your feedback is extremely valuable to us, and we would be grateful for any further comments or suggestions you might have.

Thank you once again for your time and consideration!

Sincerely,

The Authors