Towards Out-of-Modal Generalization without Instance-level Modal Correspondence

We deeply thank reviewer KNt4 for your acknowledgment of our novelty and contribution. Here we carefully address each concern and hope that you will find our reply satisfactory.

Q1: Justification of limited effectiveness.
A1: We want to stress that OOM generalization is still a very challenging task where we have 1) no supervision at all and 2) no prior relationship between IM and OOM data.

• Therefore, the baseline "aligned" is for reference where we have numerous data with full supervision.
• Directly comparing our method with "aligned" would be inappropriate.
• Although the gap is discouraging, this gives us opportunities for future research. We believe when closing such a gap, we would be approaching AGI.

Q2: Comparisons with SOTA unsupervised uni-modal methods.
A2: Thanks, since an unsupervised method that is modality-agnostic is hard to find, therefore, we employ a very competitive method Momentum Contrast (MoCo) as a baseline.

• For the semi-supervised case, we use MoCo combined with EntMin to train OOM data.
• For the unsupervised case, we directly employ MoCo. The result on MSR-VTT with R@1 is shown below:

  Setting	Method	Aud	Lan	Vid
  Semi	MoCo+EntMin	20.5	21.1	23.4
  Semi	COX	23.3	23.4	26.5
  Unsup	MoCo	10.5	10.2	12.6
  Unsup	COX	13.5	16.5	15.2

  • We can see that under both scenarios, the performance improvement of COX is still significant, which justifies the effectiveness of COX. We will incorporate such a baseline into our revision. If there are any other baseline methods to be incorporated, please let us know.

Dear Reviewer KNt4,

We really appreciate your acknowledgement of our contribution, which makes us feel it is our obligation to improve the quality of this paper as much as we can.

Therefore, we are eager to know if there are any further questions or unclearity left. We would be glad to have further discussions if there are any suggestions or questions.

Thank you again for your valuable comments, we couldn't have improved our paper without your help.

Best,
Authors.

Dear reviewer KNt4,

• Comparison to SOTA baselines in each uni-modality.

  • Thanks for the suggestion, here we consider the two most well-studied modalities, vision and language, and choose two recent SOTA baselines for comparison.
  • For vision, we choose FreeMatch which conducts balanced semi-supervised learning under the FixMatch framework.
  • For language, we choose MixText which conducts mixup augmentation for semi-supervised training.
  • Here we employ them under the OOM generalization setting, and show the result below:

    Vision	NYU-D	Language	MSRVTT
    FreeMatch	45.2	MixText	21.2
    COX	52.3	COX	23.4

    • Even though the two baselines were effective under their original setting, their performance is still limited when applied to a challenging multimodal dataset with scarce knowledge. As we can see, COX still shows very effective performance compared to them, again justifying the benefits of leveraging IM data.

• Ablation of combining COX for vision tasks.

  • Thanks a lot for the suggestion, it is indeed an important benefit of the OOM generalization study.
  • Here we show the result below:

    Setting	Method	NYU-D
    Unsup	MoCo	15.7
    Unsup	MoCo+COX	23.8

    • Delightfully, we observe significant performance improvement on RGB of NYU-D dataset, this is because COX unleashes the potential label information from IM data to enhance label prediction of OOM, i.e., RGB here.
    • Such a finding implies that COX can extract knowledge from other modalities to enhance new ones, which is the main goal of this study.

Thank you again for your valuable and insightful opinions, we are really glad to achieve some additional findings that justify our method. We will carefully conduct experiments to enrich the final version of this paper. Without your help and effort, we couldn't reach this achievement.

Kind regards,
Authors.

Dear reviewer KNt4,

Thank you for your previous recognition of your contribution, we are really glad that our effort has helped address your concerns. Moreover, we have carefully revised our paper with the following aspects based on your suggestions:

• Incorporating additional baseline MoCo in the main comparison. (Table 3)
• Justifying the effectiveness of COX by showing its benefits under various correspondence levels. (Figure 7)
• Testifying the superiority of COX over competitive uni-modal baselines. (Table 5)
• Enhancing IM performance of MoCo by leveraging COX. (Table 6)

Before the discussion ends within a day, we hope to know whether we can help with any additional concerns that affect your rating. If there are any additional suggestions or modifications that you would recommend, please don't hesitate to let us know.

Again we are deeply grateful for your help and taking your valuable time, looking forward to hearing from you soon.

Sincerely,
Authors.

We thank reviewer oDE6 for your constructive suggestions. We have tried our best to answer all your questions, hope you find it satisfactory.

Q1: Explanation of sampling $p(x^{\mathrm{O}}|x^{\mathrm{I}})p(x^{\mathrm{I}})$ without correspondence.
A1: As discussed in [1], the VIB framework can be applied in an unsupervised manner by assigning each OOM data to one cluster centroid, thus sparing the need for correspondence.

• In our practice, we conduct a warm-up phase to initialize the centroid using the modality agreement rule.
• Further, the cluster is enhanced by learning class-wise mapping across modalities.

[1] Alemi, A.A., Fischer, I., Dillon, J.V. and Murphy, K., 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410.

Q2: Justification of assumption $p(V, Y|X^{\mathrm{O}}, X^{\mathrm{I}})=p(V|X^{\mathrm{I}})p(Y|X^{\mathrm{I}})$.
A2: We want to stress that we consider the case where correspondent modalities are related to only one instance, and each instance is different from others, which is the same setting as CLIP and various multi-modal studies.

• Therefore, the assumption ("$a1=a2=a$") proposed by the reviewer is quite different where two instances have the same modality data, which creates a many-to-one correspondence, but we consider one-to-one correspondent.
• As a result, such an assumption introduces a population shift which violates our problem setting.
• In our setting, it is sufficient to infer the labels $Y$ and latents $V$ by giving $X^{\mathrm{I}}$, because $X^{\mathrm{O}}$ is redundant. Hence, our assumption holds.

Q3: Explanation of applying classifiers to different modalities.
A3: Applying the classifier across different modalities is based on the modality connection.

• Once we build the connection that maps IM data to OOM data, we can form the IM data in a feature space that is similar to OOM data.
• Since IM data is fully known, we can obtain the near-optimal classifiers in such a space.
• Therefore, the optimal classifiers can be applied to OOM data.
• As we show in Figure 5: As connection closes the modality gap, applying classifiers from IM to OOM data achieves improved performance.

Q4: Comparison with SOTA unsupervised uni-modal baselines.
A4: Thanks, since an unsupervised method that is modality-agnostic is hard to find, therefore, we employ a very competitive method Momentum Contrast (MoCo) as a baseline.

• For the semi-supervised cases, we use MoCo combined with EntMin to train OOM data.
• For the unsupervised case, we directly employ MoCo. The result on MSR-VTT with R@1 is shown below:

  Setting	Method	Aud	Lan	Vid
  Semi	MoCo+EntMin	20.5	21.1	23.4
  Semi	COX	23.3	23.4	26.5
  Unsup	MoCo	10.5	10.2	12.6
  Unsup	COX	13.5	16.5	15.2

  • We can see that under both scenarios, the performance improvement of COX is still significant, which justifies the effectiveness of COX. We will incorporate such a baseline into our revision. If there are any other baseline methods to be incorporated, please let us know.

Q5: Related works.
A5: Thanks for providing such an interesting work, we have carefully included the reference and discussed its relationship with our work.

Dear Reviewer oDE6:

We are deeply grateful for your insightful comments. We have made a maximum effort to address each question by providing both theoretical and empirical justifications, such as:

1. applicability of VIB without correspondence,
2. practicality of our assumption,
3. explanation of applying classifiers across modalities,
4. comparison with SOTA baseline.

We hope our answer could clarify your concerns. If any questions remain, please don't hesitate to let us know. We will be more than happy to have a detailed discussion. Thanks again for your valuable contribution to this paper, without which our paper couldn't have been further improved.

Warm regards,
Authors.

Dear Reviewer oDE6,

Thank you for your prompt reply. We are happy to address your remaining questions.

• Assumption $p(V, Y|X^{\mathrm{O}}, X^{\mathrm{I}})=p(V|X^{\mathrm{I}})p(Y|X^{\mathrm{I}})$ that implies $X^{\mathrm{O}}$ can be inferred from $X^{\mathrm{I}}$.

  • We want to clarify that such an assumption is not based on inferring $X^{\mathrm{O}}$ from $X^{\mathrm{I}}$.
    • For example, given dog bark audio, we can know it corresponds to an instance dog. Similarly, given a dog image, we can know it is from a dog, too. So when given dog bark, we assume we can infer the label dog with or without the dog image.
    • On the other hand, here we do not assume we can identify dog bark from a dog image.
  • In our method, we use the VIB framework to connect $X^{\mathrm{O}}$ and $X^{\mathrm{I}}$, which are both observable.
    • Theoretically, given two observable variables, it is feasible to map from one to another using VIB [1], including supervised and unsupervised scenarios [2]. Moreover, as stated in [3], it is reasonable to learn such mappings that connect different modalities.
    • Practically, real-world applications from [4] conduct cross-modal translations to map across medical modalities such as magnetic resonance images (MRI) and electrocardiograms (ECGs), which again justifies our methodology.

• Explanation of "an unsupervised method that is modality-agnostic is hard to find".

  • Let's consider the three essential aspects of machine learning: data, model, and optimization:

    • The limitation of finding a modality-agnostic method is mostly because of the data aspect, where many effective data-augmentation techniques cannot be applied to different modalities. Hence, many popular SSL methods based on image data are not applicable.

    • Therefore, we consider MoCo because of two of its advantages:

      • Momentum update of model,
      • Contrastive learning for optimization,

      which are still very effective for representation learning, thus the superiority of COX over MoCo can well justify its effectiveness. We will complete the baseline comparison with MoCo and update it to the revision soon.

We deeply appreciate your follow-up discussion, we hope the above clarification can address your remaining concerns. If there are any further questions left, we are here to provide answers with our maximum effort.

[1] Alemi, A.A., Fischer, I., Dillon, J.V. and Murphy, K., 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410.
[2] Slonim, N., Atwal, G.S., Tkačik, G. and Bialek, W., 2005. Information-based clustering. Proceedings of the National Academy of Sciences, 102(51), pp.18297-18302.
[3] Lu, Z., 2023. A theory of multimodal learning. Advances in Neural Information Processing Systems, 36, pp.57244-57255.
[4] Radhakrishnan, A., Friedman, S.F., Khurshid, S., Ng, K., Batra, P., Lubitz, S.A., Philippakis, A.A. and Uhler, C., 2023. Cross-modal autoencoder framework learns holistic representations of cardiovascular state. Nature Communications, 14(1), p.2436.

Thank you for your response! I have two follow-up questions: (i) The assumption that XO can be inferred from XI seems overly strong. Could you please clarify or elaborate on this? (ii) Could you explain what you mean by "an unsupervised method that is modality-agnostic is hard to find"?

Dear Reviewer oDE6,

We are grateful for your follow-up comments, we have tried our best to address all the remaining questions. We hope that our reply can clearly and effectively provide an explanation for our study. As the discussion phase is ending soon, we are eager to know whether our answer is satisfactory for you. If there are any other factors that affect your opinions about this paper, we hope you could kindly let us know.

Again thank you for your help and effort, we sincerely acknowledge your contribution for sparing your valuable time to reply to us.

Kind regards,
Authors.

Dear Reviewer oDE6,

Thank you for your initial comments. However, we have not received any follow-up from you, and your opinion regarding our justification remains unclear. If our responses are unsatisfactory, we kindly request that you specify the concerns that remain unresolved, as reviewers are encouraged to provide detailed feedback.

Given that your rating is still negative, we are curious why no additional questions have been raised, despite our careful efforts to address all your concerns.

We look forward to hearing from you and deeply appreciate your time and effort in providing further justification for your rating.

Sincerely,
Authors

Dear reviewer oDE6,

We thank you for reviewing this paper. However, we never received any further comments from you. So we could not make any progress with you if we do not know what is your opinion.

Our discussions with all other reviewers have been successful, and we have achieved a consensus. We are still hoping to know whether we have effectively addressed your concerns since you are the last one. Hope we can have the chance to hear from you before the discussion ends in a day, and we would be deeply appreciated.

We understand it has been a long time, so here we summarize our previous reply:

• We provided intuitive justification for our assumption from both theoretical and practical aspects.
• We explained our choice of baseline.
• Moreover, we have provided additional empirical experiments by comparing COX to uni-modal methods. (Table 5)

If you can spare a little time to take a look, it would be a big help for us. We just hope we can work together to ensure a fair judgment for this paper.

Best wishes,
Authors.

We appreciate reviewer PuJV for your positive opinions. Here we carefully take all your suggestions and address all concerns.

Q1: Discussion on related works.
A1: Thanks for pointing out. We have carefully discussed the difference between them and our work. Please see the revised paper.

Q2: Definition of prediction uncertainty.
A2: Thanks, we have stressed this denotion in the revision: The "prediction uncertainty" denotes the data with uncertain prediction, i.e., low accuracy.

Q3: Relationship between COX performance and IM perceptor qualities.
A3: Here we explain and justify our claim:

• The quality difference between ImageBind and LanguageBind is they use different modalities for binding and have different parameter scales.
• To further justify this claim, we leverage various CLIP as IM perceptors for RGB and Language modalities, and show the OOM performance on Depth data using the NYU-D dataset:

  	ViT-S/16	ViT-B/16	ViT-L/16
  COX	46.4	50.9	52.4

  • We can find that the performance of COX is dependent on the employed IM perceptors, therefore, it is essential to enhance IM perceptors to employ COX.

Q4: Typos.
A4: Thanks for pointing out, we have carefully refined our paper and polished all unclear notations.

Q5: Justification of "IM perceptors can help OOM generalization".
A5: In our comparison, the baselines such as ERM, SSL, and EntMin only learn from OOM data without any IM knowledge. Contrastively, our method leverages the knowledge from IM data:

• Achieving superior performance than baselines.
• Enabling unsupervised OOM generalization without any correspondence.

To further provide intuitive justification, we show the benefits of COX by varying the number of correspondence in OOM data on MSR-VTT:

• We use ERM and EntMin as baselines, and vary the correspondence number to 90%, 60%, 30%, and 10%, as shown below:

  Method	90%	60%	30%	10%
  ERM	68.4	60.1	46.9	38.2
  EntMin	69.2	61.6	48.5	39.4
  COX	69.4	63.9	55.1	48.8
  $\Delta$	0.2	2.3	5.6	9.6

  • First of all, we observe that COX brings more benefits when correspondence is more scarce.
  • This is because sufficient correspondence can maximally uncover the knowledge of OOM data. As correspondence gets less, the knowledge that can be explored from correspondence decreases.
  • However, COX leverages the knowledge from IM data which brings more benefits compared to using OOM data alone.

Dear Reviewer PuJV:

We really appreciate your insightful opinions that greatly helped us improve this paper. We carefully took every single question and tried our best to answer them, including related works and study on IM perceptor quality. If there are any concerns unresolved, we would be glad to have further discussions.

Thanks again for taking your valuable time and contribution on this paper. Even though it is anonymous, we want you to know that you are deeply appreciated. We are looking forward to hearing from you soon.

Sincerely,
Authors.

We thank reviewer Zg1q for your valuable opinions. We have carefully addressed all your comments and hope that you will find our response satisfactory.

Q1: Typos and mathematical formulations.
A1: Thanks for pointing out, we have carefully clarified all mathematical formulations:

• IM data $X^{\mathrm{I}}$ and OOM data $X^{\mathrm{O}}$;
• Eq. 3:

$

I(X^{\mathrm{O}}; X^{\mathrm{I}})&=\int dx^{\mathrm{O}}dx^{\mathrm{I}} p(x^{\mathrm{O}}, x^{\mathrm{I}})\log\frac{p(x^{\mathrm{O}}, x^{\mathrm{I}})}{p(x^{\mathrm{O}})p(x^{\mathrm{I}})}=\int dx^{\mathrm{O}}dx^{\mathrm{I}} p(x^{\mathrm{O}}, x^{\mathrm{I}})\log\frac{p(x^{\mathrm{O}}|x^{\mathrm{I}})}{p(x^{\mathrm{O}})},

$

For details, please check the revised paper.

Q2: Explanation of lower-bound derivation from Eq. 3 to Eq. 4.
A2: In Eq. 3, we first replace $p(x^{\mathrm{O}}|x^{\mathrm{I}})$ with $\int dv p(x^{\mathrm{O}}|v)p(v|x^{\mathrm{I}})$, where $p(v|x^{\mathrm{I}})$ can be modeled by the IM perceptors, and $p(x^{\mathrm{O}}|v)$ can be esitmated via a decoder $q(x^{\mathrm{O}}|v)$, then we have:

$

\int dx^{\mathrm{O}}dx^{\mathrm{I}} p(x^{\mathrm{O}}, x^{\mathrm{I}})\log\frac{\int dv p(x^{\mathrm{O}}|v)p(v|x^{\mathrm{I}})}{p(x^{\mathrm{O}})}= \int dx^{\mathrm{O}}dx^{\mathrm{I}}dv p(x^{\mathrm{O}}, x^{\mathrm{I}})\log\frac{p(x^{\mathrm{O}}|v)}{p(x^{\mathrm{O}})/p(v|x^{\mathrm{I}})}

$

Then, based on $\int dx^{\mathrm{O}} p(x^{\mathrm{O}}|v)\log p(x^{\mathrm{O}}|v)\ge \int dx^{\mathrm{O}} p(x^{\mathrm{O}}|v)\log q(x^{\mathrm{O}}|v)$, and since the denominator stays the same, we can follow the same derivation as VIB and obtain:

Dear Reviewer Zg1q,

We want to express our appreciation for your valuable suggestions, which greatly helped us improve the quality of this paper. We are also glad that you think our work is novel and insightful. We have made our maximum effort to address all your concerns about detailed proof, non-negativity assumption, and method effectiveness.

Your further opinions are very important for evaluating our revised paper and we are hoping to hear from you soon. Thank you again for your effort and constructive opinions.

Best,
Authors.

Dear Reviewer Zg1q,

We have carefully taken all your previous comments into consideration, which are very constructive and helpful. Your effort and contribution to this paper is deeply appreciated.

After careful formulation and reply, we have provided:

• details of our derivation,
• justification of our assumption,
• description of the training process,
• empirical analysis of the anchor point method.

We are eager to know whether our reply could be helpful and want you to know that we are here to make our maximum effort to improve our paper. Thank you again for taking your time and nothing would be more delightful to hear your further opinions.

Sincerely,
Authors.

Dear Reviewer Zg1q,

We again thank you for the effort you put into reviewing. We understand that you might be busy at this time, and we would deeply appreciate it if you could spare a little time to take a look at our rebuttal.

We all hope that the interactive discussion is effective so that there is no misunderstanding or unclarity left between the reviewers and authors. So we cherish this opportunity to discuss with you and have sufficient communication before a final decision is made.

Since the discussion phase is ending, we sincerely hope we can hear from you soon so that we can provide our best effort and help to address your questions.

Kind regards,
Authors.

We thank Review 2YMo for your constructive comments. Here we address all your concerns with careful justifications.

Q1: 1) Practical assumption for optimal classifiers; 2) Experiments on classifiers with varied accuracy levels.
A1: Thanks for the suggestion.
Since we have sufficient labeled IM data, it is possible to achieve near-optimal classifiers:

• Like most studies in AI/ML, we rely on ERM as the theoretical guarantee to approximate the optimal hypothesis class.
• The IM percetors have been pre-trained on numerous data, further fine-tuned on well-known multi-modal datasets.
• Therefore, assuming the IM perceptors are near-optimal classifiers is reasonable.

Still, we acknowledge that real-world applications could suffer from low-data problems where we have both limited IM and OOM data:

• To answer the question, we conduct experiments on MSR-VTT by using IM perceptors fine-tuned with different numbers of IM data, and show the result below:

  	10%	40%	70%	100%
  Aud Acc.	65.5	72.8	81.6	89.4
  Lan Acc.	70.2	77.8	86.6	92.3
  Vid (OOM)	18.4	28.9	36.7	48.8

  • We can see that the OOM performance is significantly affected by the accuracy level of IM perceptors.
  • Therefore, improving the performance of IM perceptors is vital for OOM generalization using COX.
• We also emphasize that our goal is generalizing to rare OOM data without correspondence or labels, therefore, such a challenging situation is orthogonal to our problem.

Q2: The reason for performance improvement by using IM perceptors.
A2: Thanks. Here we justify both theoretical and experimental aspects:

• Theoretically, we showed that the knowledge of one modality contains "commonality" and "uniqueness".
  • Without correspondence or labels, the exploration of both commonality and uniqueness is largely hindered, leading to poor results.
  • Through our VIB-based modality connection, we can maximally discover the commonality shared across modalities, providing essential guidance to understand OOM data.
  • Further, COX leverages the guidance to explore uniqueness using semi-supervised techniques, thus surpassing baselines that neglect such guidance.
• Experimentally, we vary the number of correspondence in OOM data from MSR-VTT to justify the benefits brought by COX:
  • We use ERM and EntMin as baselines, and vary the correspondence number to 90%, 60%, 30%, and 10%, as shown below:

    Method	90%	60%	30%	10%
    ERM	68.4	60.1	46.9	38.2
    EntMin	69.2	61.6	48.5	39.4
    COX	69.4	63.9	55.1	48.8
    $\Delta$	0.2	2.3	5.6	9.6

    • First of all, we observe that COX brings more benefits when correspondence is more scarce.
    • This is because sufficient correspondence can maximally uncover the knowledge of OOM data. As correspondence gets less, the knowledge that can be explored from correspondence decreases.
    • However, COX leverages the knowledge from IM data which brings more benefits even with less correspondence.

Q3: Study on enhancing OOM generalization by increasing modalities.
A3: Thanks for the suggestion, here we conduct experiments on MSR-VTT by increasing the number of IM modalities as 0, 1, and 2:

• For 0 IM modalities, we have the basic ERM method, for 1 IM modality, we conduct COX based on the prediction confidence instead of modality disagreement criterion, and for 2 IM modalities, we have the standard COX.
• The result is shown below:

  	0	1	2
  OOM performance	38.2	40.5	48.8

  • We can see that as the number of IM modalities increases, the OOM performance also improves.
  • Therefore, we can justify that using more IM modality data can benefit the OOM performance.
  • Although well-known multi-modal datasets only have limited modality numbers due to the expense of acquiring modalities, using 3 modalities can provide insights for OOM generalization. We plan to discover the scaling performance on more modalities in the future.

Q4: Typos.
A4: Thanks for pointing out, we have carefully refined our paper and eliminated all the typos.

Dear reviewer 2YMo,

We really appreciate your efforts to help improve this paper. We have carefully addressed all the mentioned concerns, such as the analysis of the IM perceptors and the effect of modality numbers.

Having further discussions really helps to achieve consensus and clarify misunderstandings, so we are eager to know if are there any remaining questions. We are more than happy to try our best to address them.

Best,
Authors.

Dear Reviewer 2YMo,

We want to show our gratitude again for your contribution to this paper, which has given us many insightful suggestions to improve our paper. We are looking forward to hearing from your further opinions about our rebuttal. We cherish the opportunity to polish our paper through this interactive discussion and hope you can spare a little time to take a look.

If there are any further questions left, please feel free to let us know. We sincerely hope we could help clarify any further misunderstandings. Thank you again for your help and support, we truly appreciate it.

Warm regards,
Authors.