Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations

We thank the reviewer for the comments and suggestions. Please find the following answers to each comments.

Weaknesses 1. CentroBind computes the average of the inputs in the batch as the anchors. However, means are sensitive to outliers. When there are outliers in a batch, the performance will be influenced. Particularly, when the batch size is small, it will also influence the performance.

Thank you for pointing this out. We have also considered this perspective, recognizing that the mean is not robust to outliers. One alternative approach is using the medoid, which is more robust to outliers.

However, despite the limitations of the centroid, we have reasons for continuing to use it:

1. The centroid is the minimizer of average L2 distances to the embeddings, making it suitable for the goal of multimodal alignment.
2. Identifying outliers in positive pairs is challenging without prior domain knowledge (e.g., data distribution). Otherwise, detecting outliers would require numerous samples for a specific semantic representation, which is impractical.

To address this concern, we conducted additional experiments on synthetic datasets with manually controlled quality (providing prior knowledge on modalities) in Figure 2, 6, 7, and Figure 8. Even in scenarios with very low-quality modalities (Figure 8) that can be considered outliers, CentroBind performs comparably to the weighted average method that rely on prior knowledge of modality quality, implying robustness of modality imbalance problem.

Weaknesses 2. The experiments are not sufficient. There are many unified multimodal representation methods. Why do the authors only compare FABind? More recent baselines should be included.

We have noted that in both academia and industry, fixed-anchor-bind models such as ImageBind and LanguageBind are the de facto standard used by many researchers and practitioners. Therefore, we focused our analysis and refinements on these approaches. However, we agree that including more baseline comparisons is always helpful, and we have provided additional baseline results incorporating other alignment methods. Please refer to our response to Reviewer YKP7 for further details.

Weaknesses 3. I consider this paper to be incremental work, as it makes only very minor changes to FABind.

In this paper, we first identify the fundamental limitations of alignment methods such as ImageBind and LanguageBind. We generalize these methods under the category of fixed-anchor-bind methods and provide rigorous theoretical analyses of their limitations. Based on these analyses, we propose a simple yet effective modification to the approach, allowing the anchors to be dynamic rather than fixed, which alleviates many of the identified limitations. This also opens the door to further refinements, such as weighted or attention-based averaging. We believe that modeling does not have to be complex; in fact, simplicity is best, especially when grounded in sound rationale and backed by empirical evidence. Therefore, we respectfully disagree with the notion that our contribution is incremental.

Weaknesses 4. There are no visualizations in the experiments. Features before and after CentroBind should be visualized to better demonstrate its effectiveness.

Thank you for your comment. Please refer to our response to Reviewer BBKI for the visualizations.

To address weakness 2, we have included UniBind, AudioCLIP, ViT-Lens as the baseline approaches.

---

Table 2

Table 2 presents the classification accuracy for zero-shot cross-modal sarcasm detection and speaker classification tasks. CentroBind achieves significantly outperform top-1 accuracy compared to all baseline methods in both tasks. Our additional baselines, UniBind, AudioCLIP, and ViT-Lens, perform poorly in the zero-shot cross-modal experiment.

To address weakness 1, we include Table 4 to compare our approach with other methods along with Figure 8 (weighted average, random anchors, and prior knowledge of the given modality).

Table 4

Table 4 presents the classification accuracy corresponding to the experiments illustrated in Figure 8. In the experiments depicted in Figures 8a and 8b, the information level varies such that $X_1$, $X_2$, and $X_3$ are highly noisy, while $X_4$ is highly informative. Conversely, in Figures 8c and 8d, $X_1$ and $X_2$ are uninformative, whereas $X_3$ and $X_4$ are high-quality modalities. Each modality is selected as the fixed anchor modality in turn, and we compare CentroBind with FABind, weighted average, random anchor, and random anchor with intra-learning methods.

When modality quality is imbalanced, the centroid-based approach may produce suboptimal dynamic anchor constructions, allowing other methods, such as weighted averages, to achieve better results in specific scenarios. Despite this, CentroBind consistently performs either better or on par with weighted average methods, showcasing its robustness to modality imbalance. Furthermore, the random anchor method also achieves competitive performance. These observations suggest that the choice of dynamic anchor generation method may have limited impact on final performance, as long as all encoders are appropriately trained during the process.

• We sincerely thank the reviewer for their comments on the use of medians and stability in training. We have updated Figure 8 and Table 4 to include results for a median-based dynamic anchor approach.
• Based on valuable feedback from the reviewers, we now explore a variety of dynamic anchor computation methods, including weighted averaging, random choice (with and without intra-learning), as presented in Table 4. We specifically thank the reviewer for suggesting the inclusion of the median-based approach, making the paper more comprehensive.
• To reflect the diversity of anchor computation methods, we have revised the manuscript to explicitly acknowledge alternative strategies. Additionally, we provide a clear rationale for selecting the centroid-based approach before introducing it as our primary method.
• Finally, we reiterate that a key contribution of our work lies in the theoretical analysis of fixed anchor limitations and their replacement with dynamic anchors, facilitating intra- and shared information learning across modalities. The median-based approach can be an extension of this analysis, and we thank the reviewer for highlighting this point.

Discussion on the Median-Based Results

• The median-based anchor was computed coordinate-wise. Specifically, for a median-based anchor, we compute the $j$th coordinate of the $i$th anchor as $a_{i,j} = {\rm Median}(z_{1,i,j}, z_{2,i,j}, \cdots, z_{M,i,j})$, where ${z}_{m,i,j}$ denotes the $j$th coordinate of the embedding for the $i$th sample in modality $m$.
• Comparative results of the median-based method with other approaches are presented in Table 4. For these experiments, we intentionally introduced extreme outliers, as detailed in Appendix C.
• As expected, the median approach occasionally outperforms other methods in such extreme cases. However, the differences are not significant, and CentroBind often achieves better overall performance.
• In this paper, we chose to focus on centroid-based anchors, as the centroid represents the geometric center, which aligns closely with the objective of this paper: achieving effective multimodal alignment in embedding spaces.
• Furthermore, we think that outliers are uncommon in embedding spaces when the encoders are well-trained. We include an extended discussion of this topic in Appendix C.

Table 4

Dear Reviewer ioPy,

Thank you for your effort in reviewing our work. We hope you have had the chance to review our responses and new updates on experiments. As the discussion period is drawing to a close, we would greatly appreciate it if you could confirm whether our updates have fully addressed your concerns.

Thank you again for your time and thoughtful review.

Best regards, The Authors

We would like to thank the reviewer for their time and effort in reviewing our paper and valuable comments. Please find the following answers to the comments and questions.

Weaknesses 1. To me, the claim of a closer representation to Huh et al's "Platonic representation" is not quite evident. I tend to believe the authors are also not very positive about this aspect from their statement in line 361 that the representation is "likely closer" to the Platonic representation.

Huh et al.’s “Platonic Representation” hypothesis posits that “neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.” While it is difficult to prove this directly, we believe that dynamic computation of anchors based on sample-wise mutual information is more likely to achieve such a representation than fixed anchor-based methods. This is because, CentroBind, by utilizing dynamic anchors, is likely to retain a more comprehensive representation of all modalities.

Weaknesses 2. This is not a weakness of the paper per se. Audio, video and text are the usual modalities that are discussed in multimodal representation. It would be interesting to study datasets with additional modalities such as VIDIMU: Multimodal video and IMU kinematic dataset on daily life activities using affordable devices (https://zenodo.org/record/8210563), which has inertial sensors.

Thank you for suggesting this dataset. Incorporating different modalities, such as kinematics, would indeed be an interesting avenue for future research.

Question 1. Line 220: The authors may clarify why equation (4) shows that FABind does not guarantee shared information between non-anchor modalities.

Solving Equation (4) requires maximizing the mutual information between the anchor modality and other modalities, while the mutual information between non-anchored modalities is not explicitly considered in the objective function. Consequently, there is no guarantee that mutual information among non-anchor modalities will increase. In other words, the solution to Equation (4) only needs to maximize the shared information between the anchor and the other modalities.

Question 2. Are there any constraints on the batch B from which the anchor embeddings are derived?

Similar to any other fixed-anchor-bind method, CentroBind also needs each sample in a batch to have paired representation (embeddings) among different modalities. However, it does not need to be fully paired across all modalities.

Question 3. It is not clear which part of the analysis specifically shows that using a fixed anchor affects the representativeness of all modalities (P4).

Thank you for pointing out this potentially confusing part. Our intention in claiming P4 is to emphasize that a single modality cannot capture the unique information provided by other modalities. To be representative of all modalities, one must leverage all modalities collectively, which is precisely the motivation behind multimodal learning. Otherwise, unique information present in other modalities may be overlooked. We have updated the paper to include this reasoning on page 5.

Question 4. How would acc(All) perform in Fig 2 (a)?

acc(All) refers to the classification accuracy achieved when embeddings from all modalities are utilized. Specifically, this involves training and evaluating a decoder using the concatenated embeddings of all modalities as the input to the decoder.

We would like to thank the reviewer for their comments and suggestions. Please find the following responses to the comments.

Limited Baseline Comparison. While the limitations of fixed anchors are well-articulated, additional baselines would help to further justify the centroid-based approach. For example, sampling a modality as an anchor randomly or choosing an anchor based on the minimum or maximum similarity score could provide useful comparisons to strengthen the motivation for centroid-based anchoring.

Thank you for your comment on the different dynamic anchor generation methods. In the revised version, we have compared various baseline methods for dynamic anchors, such as the weighted average (which requires additional knowledge about the datasets) and random anchors. We briefly include these details in Section 4.1 and provide a more thorough explanation in Appendix C.1. The results indicate that, despite not requiring additional information (e.g., quality or distribution) about the dataset and downstream tasks, CentroBind performs comparably to the weighted average, which we consider a "genie case" assuming prior knowledge of data distributions for each modality.

We also refer to our responses to Reviewer 2XLP for further clarification.

Positioning Among Related Works. Although the paper’s primary focus is to demonstrate improvement over FABIND, it would benefit from a broader positioning among recent multimodal works [1, 2, 3]. Several studies have explored inter- and intra-modality information; a comparative discussion here would contextualize the contributions more effectively. Including a simple baseline of early or late fusion across modalities for both synthetic and real-world datasets would be insightful. Additionally, it’s unclear why Centro BIND does not utilize all modalities in Table 2 for a more comprehensive comparison.

Thank you for the suggestion. We have revised the related work section to include the suggested references, as well as additional relevant works detailed in Appendix A. Specifically, we have expanded the discussion to introduce literature on multimodal learning, including supervised multimodal learning and self-supervised learning, extending beyond the previous focus on multimodal alignment perspectives.

Comparison with Individual Modalities. Since the work emphasizes intra-modality information, a comparison with individual modality baselines and an ensemble of modalities, especially in Table 2 for evaluation, is essential. This would underscore how effectively Centro BIND leverages intra-modality information compared to single-modality models.

Thank you for your comment on the experiment. Table 2 presents the cross-modal zero-shot classification accuracy, which is a standard task for evaluating unified representation spaces. Since the primary objective of our work is to build a unified space for multiple modalities, we focused on comparing CentroBind with the state-of-the-art method FABind, which is specifically designed for multimodal learning in unified spaces. Therefore, we believe that adding unimodal cases to Table 2 would diverge from its intended purpose. Instead, we have included a comparison with unimodal learning in Figure 2, represented by acc($Z_1$). We have analyzed this synthetic experiment in terms of intra-modal information in Section 4.1.

Clarification on FABIND Setup. Could the authors clarify if FABIND uses pre-trained encoders in the experiments, or are the encoders randomly initialized? This detail would help to understand the experimental setup more clearly.

We utilized pre-trained encoders for the real-world dataset experiments, as noted in the first paragraph of Section 4.2. For the synthetic experiments, we conducted extensive comparisons between randomly initialized encoders and pre-trained encoders, as shown in Figures 2, 6, 7, and 8.

---

Table 4

Table 4 presents the classification accuracy corresponding to the experiments illustrated in Figure 8. In the experiments depicted in Figures 8a and 8b, the information level varies such that $X_1$, $X_2$, and $X_3$ are highly noisy, while $X_4$ is highly informative. Conversely, in Figures 8c and 8d, $X_1$ and $X_2$ are uninformative, whereas $X_3$ and $X_4$ are high-quality modalities. Each modality is selected as the fixed anchor modality in turn, and we compare CentroBind with FABind, weighted average, random anchor, and random anchor with intra-learning methods.

When modality quality is imbalanced, the centroid-based approach may produce suboptimal dynamic anchor constructions, allowing other methods, such as weighted averages, to achieve better results in specific scenarios. Despite this, CentroBind consistently performs either better or on par with weighted average methods, showcasing its robustness to modality imbalance. Furthermore, the random anchor method also achieves competitive performance. These observations suggest that the choice of dynamic anchor generation method may have limited impact on final performance, as long as all encoders are appropriately trained during the process.

We have compared several dynamic anchor generation methods in the following Table 4.

Table 2

Models with asterisk are evaluated in different experimental settings (encoder backbone, pretraining datasets).

Table 2 presents the classification accuracy for zero-shot cross-modal sarcasm detection and speaker classification tasks. CentroBind achieves significantly outperform top-1 accuracy compared to all baseline methods in both tasks. Our additional baselines, UniBind, AudioCLIP, and ViT-Lens, perform poorly in the zero-shot cross-modal experiment.

Dear Reviewer YKP7,

Thank you for your time and effort in reviewing our work. We hope you have had the chance to review our responses and clarifications. As the discussion period is drawing to a close, we would greatly appreciate it if you could confirm whether our updates have fully addressed your concerns.

Thanks!

Best regards, The Authors

We thank the reviewer for their time and effort in reviewing our paper and comments. We answer each weaknesses as follows.

Weaknesses 1. I believe that the comparative experiments and ablation studies are not sufficiently. The paper only compares CentroBind and FABIND methods, lacking additional comparisons with other recent multi-modal alignment frameworks, which leads me to question the validity of this comparison.

We would like to point out that FABind methods are among the most widely used approaches and represent the current state-of-the-art in multi-modal representation learning. As an additional experiment, we are working to include the most recent approach, UniBind, as an additional baseline.

Weaknesses 2. There could be more exploration into how CentroBind performs under different data distributions and noise levels, which could provide further insights into its robustness.

Thank you for your comment. We have included additional experiments where we vary the information density of different modalities to the extreme and compare three settings: weighted averaging of each modality, random anchors, and equal averaging (i.e., vanilla CentroBind). Please refer to our response to Reviewer 2XLP for more details.

Weakness summary. CentroBind's scalability and practicality for large datasets require further consideration. In addition, broadening experimental comparisons and analysis would help address potential limitations and position the method within the broader multimodal research.

Our set of experiments includes:

1. Experiments involving multiple modalities with varying levels of quality and informativeness (Figure 2)
2. Experiments involving many more modalities (Figures 6 and 7)
3. Additional experiments using more extreme cases of modality qualities to test the limits of the approach (Figure 8)
4. Experiments on real-world benchmark datasets and subsequent downstream task evaluations (Table 2)

Through these experiments, we believe we have sufficiently demonstrated the scalability and practicality of our approach across various settings and scenarios. We are also working on adding another baseline result using the UniBind [1], which is one of the latest methods with an available codebase, and AudioCLIP [2].

[1] Lyu et al., UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All

[2] Guzhov at el., Audioclip: Extending clip to image, text and audio.

We have conducted experiments on synthetic dataset generated according to certain data distribution to address Weaknesses 2.

Table 4

Table 4 presents the classification accuracy corresponding to the experiments illustrated in Figure 8. In the experiments depicted in Figures 8a and 8b, the information level varies such that $X_1$, $X_2$, and $X_3$ are highly noisy, while $X_4$ is highly informative. Conversely, in Figures 8c and 8d, $X_1$ and $X_2$ are uninformative, whereas $X_3$ and $X_4$ are high-quality modalities. Each modality is selected as the fixed anchor modality in turn, and we compare CentroBind with FABind, weighted average, random anchor, and random anchor with intra-learning methods.

When modality quality is imbalanced, the centroid-based approach may produce suboptimal dynamic anchor constructions, allowing other methods, such as weighted averages, to achieve better results in specific scenarios. Despite this, CentroBind consistently performs either better or on par with weighted average methods, showcasing its robustness to modality imbalance. Furthermore, the random anchor method also achieves competitive performance. These observations suggest that the choice of dynamic anchor generation method may have limited impact on final performance, as long as all encoders are appropriately trained during the process.

We would like to thank the reviewer for their comments and suggestions. Please find the following responses to the questions and comments.

Weaknesses 1. A tSNE visualization of the learned space could be strengthen the claims.

Thank you for your suggestion. We have attached the t-SNE and UMAP visualizations of the learned feature spaces of the synthetic dataset for FABind and CentroBind here. We have included the plots in the revised manuscript.

Weaknesses 2. Maybe more detailed experiments with existing embedding models could be conducted to generalize the results. But the paper is fine also as it is now. :)

Thank you for the suggestion. We strongly agree with this comment, and we will add more baseline results, such as UniBind [1] and AudioCLIP [2]. Once the results are ready (in a few days), we will update the paper and response to the reviewers.

[1] Lyu et al., UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All

[2] Guzhov at el., Audioclip: Extending clip to image, text and audio.

Minor comment 1. Supplementary materials should have been submitted together with the paper.

Thank you for pointing this out. The revised version now includes appendix.

Minor comment 2. The Saxon genitive should be avoided in scientific writing, although I know that both ChatGPT and Grammarly suggest to use it. However, I suggest the authors to remove all the Saxon genitives from the paper.

Thank you for the comment. We have revised the paper accordingly.

Question 1. Could the authors better explain how they dynamically adjust the centroids? I mean, after an iteration, or step by step also inside the iteration?

The dynamic anchor is generated for every pair of samples in each iteration. Specifically, in an iteration with a minibatch dataset $B$, we generate $|B|$ dynamic anchors for each pair of samples. After the iteration is completed and the encoders are updated, the dynamic anchors for the subsequent iteration will differ due to these updates. As a result, the dynamic anchors are continually adjusted throughout the training process, adapting to the evolving embeddings generated by the encoders.

Question 2. How would this model scale up to multiple (more than three) modalities?

Similar to FABind, CentroBind is capable of learning from multiple modalities, including more than three modalities. Due to our limited computational resources, we chose to conduct experiments with three modalities for the real-world dataset case. However, we also explored scenarios involving more than three modalities through synthetic experiments. Figure 2 presents the results for four modalities, while Figures 6 and 7 in the appendix illustrate cases involving six and eight modalities, respectively. These experiments demonstrate that CentroBind successfully constructs a unified representation space for more than three modalities.

We have updated our paper based on the reviewer's comment in Weaknesses 1,2, and 4 in the following Table. Table 2 and 5 summarize comparison of baseline methods and CentroBind on a real-world dataset.

---

Table 2

Models with asterisk are evaluated in different experimental settings (encoder backbone, pretraining datasets).

Table 2 presents the classification accuracy for zero-shot cross-modal sarcasm detection and speaker classification tasks. CentroBind achieves significantly outperform top-1 accuracy compared to all baseline methods in both tasks. Our additional baselines, UniBind, AudioCLIP, and ViT-Lens, perform poorly in the zero-shot cross-modal experiment.

Dear Reviewer BBKi,

Thank you very much for your positive feedback on our work! We greatly appreciate your time and thoughtful review.

Best regards, The Authors

We would like to thank you for your comments and suggestions. We have improved the paper based on the comments and questions. We response to each weaknesses as follows.

Weaknesses 1. Robustness of anchor generation.

Thank you for highlighting the modality imbalance problem. You are correct that CentroBind does not explicitly account for the quality of each modality, which could, in theory, lead to worse performance compared to other dynamic anchor generation methods. To address this concern, we conducted the synthetic experiment shown in Figure 2 under a modality imbalance scenario, where the first modality is the worst, and the fourth modality is the best. In this case, CentroBind outperformed other baseline methods.

In response to your comments regarding weighted averages and other dynamic anchor generation methods, we performed additional experiments with extremely unbalanced information densities across modalities. Details of these experiments (Figure 8) have been added to Appendix C.1. In such imbalanced scenarios, weighted average and CentroBind exhibit similar performance, suggesting that CentroBind is robust to modality imbalance. We hypothesize that this robustness arises because dynamic anchor generation has limited influence on the final result, provided there is no fixed anchor or a frozen encoder, as in FABind.

Although CentroBind demonstrates robustness to modality imbalance, we believe addressing this issue is beyond the scope of our paper. Solving the modality imbalance problem often requires additional knowledge, such as information about downstream tasks, which contrasts with the representation learning setting.

Finally, we note that in practice, binding methods are often trained on diverse datasets containing varying information densities within the same modality. Assuming the information density is unknown, CentroBind provides a reasonable approach for anchor generation, as it represents an optimal point for multimodal alignment in Euclidean space.

Weaknesses 2. Effect of different modal weights.

Please find the above response to weaknesses 1.

Weaknesses 3. CentroBind aims to maximize both intra-and inter-modality information, but is there an information bottleneck effect (e.g. information may be compressed in high-dimensional embedding Spaces).

Thank you for the interesting question. We acknowledge that we have not explored the information bottleneck problem in the context of representation learning and have not conducted related experiments. While this is indeed an interesting direction for future research, we believe it lies outside the scope of our current paper. This issue is not specific to our method but is a broader concern that applies to all representation learning approaches. Addressing the information bottleneck problem would require a more generalized study that goes beyond the focus of our work.

Weaknesses 4. Does CentroBind suffer from potential optimization pitfalls when multiple modalities are strongly correlated or there is obvious synergy information between modalities?

Thank you for the question regarding dependency among modalities. We believe that strong correlations between modalities are always beneficial for CentroBind in extracting representations from multi-modal samples. Our reasoning is as follows:

1. The fundamental motivation for using multi-modal data is to capture a comprehensive representation shared among multiple modalities. This implies that samples from each modality should share some part of the true underlying representation, such as the concept of a Platonic representation [1]. We think that strong correlation implies that modalities share the core representation strongly, which would be helpful for multimodal learning;
2. When modalities are correlated, they naturally contribute more weight to the centroid during dynamic anchor generation. Intuitively, making the anchor embedding align more closely with highly correlated modalities is advantageous, as their shared information is more likely to reflect the true representation compared to the information from uncorrelated modalities.

To demonstrate this further, in line with Weakness 1, we conducted experiments where modality quality varies. Specifically, sub-figures (c) and (d) in Figure 8 of the Appendix depict cases where two modalities are highly correlated while others are not. In these scenarios, CentroBind performs well and is even competitive with the weighted average method, which relies on additional prior knowledge of modality quality.

[1] Huh et al., The Platonic Representation Hypothesis.

Weaknesses 5. Boundary conditions for theoretical analysis: The theoretical analysis in the paper explores the advantages of CentroBind, but are there certain boundary conditions or assumptions under which CentroBind might lose its advantage? For example, does this approach still work when the data is unevenly distributed or the amount of data is severely imbalanced?

The theoretical analysis of CentroBind is based on the lower bound of its objective function, as established in Theorem 1. Notably, this bound is derived in a general setting without making any assumptions about the data distribution. In fact, the bound is sample-wise valid, meaning our analysis holds for any data distribution.

While the bound itself is distribution-independent, we acknowledge that the gap between the objective function and its lower bound likely depends on the underlying data distribution. A theoretical characterization of this gap (e.g., assessing its tightness under specific distributional assumptions) would indeed be an interesting direction for future work. However, we consider this to be beyond the scope of our current study, as our focus is on scenarios where the quality of modalities and data distribution are unknown.

Weaknesses 6. Loss function Optimization: CentroBind's loss function combines multiple InfoNCE loss terms, but does this design affect the convergence speed or stability of the model?

When a dataset contains $M$ modalities, the loss function of CentroBind comprises $M$ InfoNCE losses, whereas FABind (e.g., ImageBind) uses $M-1$ InfoNCE losses. Thus, the number of loss functions is comparable between the two methods. The primary distinction lies in the anchor generation strategy: CentroBind employs dynamic anchors, while FABind uses fixed anchors. We believe dynamic anchor generation enables faster convergence since aligning embeddings with their centroid is geometrically easier (in terms of distance) compared to aligning them with fixed embeddings.

To support this claim, we have included a plot of the training loss curves for CentroBind and FABind in Appendix C.1 (page 19). This plot illustrates that CentroBind's loss converges to its saturation point faster than FABind's loss.

Regarding the temperature parameter in the InfoNCE loss, we have not explicitly explored its effect on the convergence rate, as this was deemed beyond the scope of our study. However, to the best of our knowledge, previous works such as [2], [3], and [4] have investigated the role of the temperature parameter in InfoNCE loss. These studies demonstrate that the temperature influences the local separation and uniformity of the embedding distributions.

[2] Wang et al., Understanding the Behaviour of Contrastive Loss

[3] Zhang et al., How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning

[4] Koromilas et al., Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses

We have updated our paper based on your comment on Weaknesses 1,2, and 4 in the following table. Table 4 summarizes the comparison of dynamic anchor generation methods shown in Figure 8.

Table 4

Table 4 presents the classification accuracy corresponding to the experiments illustrated in Figure 8. In the experiments depicted in Figures 8a and 8b, the information level varies such that $X_1$, $X_2$, and $X_3$ are highly noisy, while $X_4$ is highly informative. Conversely, in Figures 8c and 8d, $X_1$ and $X_2$ are uninformative, whereas $X_3$ and $X_4$ are high-quality modalities. Each modality is selected as the fixed anchor modality in turn, and we compare CentroBind with FABind, weighted average, random anchor, and random anchor with intra-learning methods.

When modality quality is imbalanced, the centroid-based approach may produce suboptimal dynamic anchor constructions, allowing other methods, such as weighted averages, to achieve better results in specific scenarios. Despite this, CentroBind consistently performs either better or on par with weighted average methods, showcasing its robustness to modality imbalance. Furthermore, the random anchor method also achieves competitive performance. These observations suggest that the choice of dynamic anchor generation method may have limited impact on final performance, as long as all encoders are appropriately trained during the process.

Dear Reviewer 2XLP,

Thank you for your effort in reviewing our work! We hope you have had the chance to review our responses and clarifications. As the discussion period is drawing to a close, we would greatly appreciate it if you could confirm whether our updates have fully addressed your concerns.

Thank you again for your time and thoughtful review.

Best regards, The Authors

We would like to thank the reviewers for their valuable suggestions on our paper. To address the comments on including more baselines, we have added additional experimental results and comparisons with more baseline models, including AudioCLIP [1], UniBind [2], and ViT-Lens [3].

Specifically, the following updates have been made:

1. We have added Table 2, which includes baseline methods (UniBind, AudioCLIP, and ViT-Lens) and reports their classification accuracy for cross-modal sarcasm and speaker classification tasks. As shown in Table 2, CentroBind significantly outperforms these additional baseline methods.
2. To improve the readability of Figure 2, we have included Table 3, which provides a clearer presentation of the associated results.
3. We have added Table 4 to enhance the readability of Figure 8. Table 4 presents a comparison of various dynamic anchor generation methods, including FABind (anchored on highly informative modalities), weighted average, random anchor with intra-learning, and random anchor.
4. We have introduced Table 5, which provides conventional classification results (not zero-shot or cross-modal evaluations) for FABind, UniBind, AudioCLIP, ViT-Lens, and CentroBind.

[1] Guzhov et al., Audioclip: Extending Clip to Image, Text and Audio, ICASSP 2022

[2] Lyu et al., UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024

[3] Lei et al., ViT-Lens: Towards Omni-modal Representations, CVPR 2024

Table 5

Models with asterisk are evaluated in different experimental settings (encoder backbone, pretraining datasets).

Table 5 summarizes the classification accuracy of FABind, UniBind, and CentroBind for both single-modal and multimodal settings, using embeddings extracted by the binding methods and a simple decoder for classification. CentroBind generally outperforms other methods in sarcasm detection, while UniBind shows better performance in speaker classification, likely due to its use of LLM-augmented descriptions as additional modality. Notably, CentroBind consistently surpasses FABind. Additionally, CentroBind can incorporate LLM-augmented descriptions as an extra modality, which could further enhance its performance.

For reference, the table also includes sarcasm detection results from FactorCL, SimMMDG, AudioCLIP, and ViT-Lens. While ViT-Lens achieves higher accuracy due to larger backbone encoders and extensive pretraining, it can be considered a variant of FABind. Applying CentroBind's dynamic anchor method to ViT-Lens or integrating Vision Transformers into CentroBind could further boost performance.

We have compared several dynamic anchor generation methods in the following Table 4.

Table 4

Table 4 presents the classification accuracy corresponding to the experiments illustrated in Figure 8. In the experiments depicted in Figures 8a and 8b, the information level varies such that $X_1$, $X_2$, and $X_3$ are highly noisy, while $X_4$ is highly informative. Conversely, in Figures 8c and 8d, $X_1$ and $X_2$ are uninformative, whereas $X_3$ and $X_4$ are high-quality modalities. Each modality is selected as the fixed anchor modality in turn, and we compare CentroBind with FABind, weighted average, random anchor, and random anchor with intra-learning methods.

When modality quality is imbalanced, the centroid-based approach may produce suboptimal dynamic anchor constructions, allowing other methods, such as weighted averages, to achieve better results in specific scenarios. Despite this, CentroBind consistently performs either better or on par with weighted average methods, showcasing its robustness to modality imbalance. Furthermore, the random anchor method also achieves competitive performance. These observations suggest that the choice of dynamic anchor generation method may have limited impact on final performance, as long as all encoders are appropriately trained during the process.

Table 3

Table 3 compares classification accuracy between FABind and CentroBind in both unimodal and multimodal settings, with additional visualization in Figure 2. The experiments are conducted using both randomly initialized and pre-trained backbones while varying the anchor modality. For each scenario, we train five decoders: four for individual modalities and one for the concatenated embeddings of all modalities. $\times$ denotes the experiment without binding method. The results in Table 3 highlight that CentroBind significantly outperforms FABind with randomly initialized backbones, demonstrating its robustness to low-quality backbones.

We have updated our paper and summarized the important changes related to the experimental results.

---

Table 2

Table 2 presents the classification accuracy for zero-shot cross-modal sarcasm detection and speaker classification tasks. CentroBind achieves significantly outperform top-1 accuracy compared to all baseline methods in both tasks. Our additional baselines, UniBind, AudioCLIP, and ViT-Lens, perform poorly in the zero-shot cross-modal experiment.