Learning Compact Semantic Information for Incomplete Multi-View Missing Multi-Label Classification

Thanks for your constructive reviews and suggestions. Below, we will address each of your questions.

Q1: Some minor mistakes should be revised, the reference of equation 14 in line 205 is incorrect and it should be capitalized in line 374: “In table 1”.

Thanks for your correction. We have corrected the incorrect reference to Eq. (14) in line 205, capitalized “Table 1” in line 374, and addressed other minor errors throughout the manuscript to ensure consistency with academic standards.

Q2: The code is not provided. To improve reproducibility, implementation details and code should be provided.

Thanks for your suggestion. The code will be made publicly available upon acceptance to support reproducibility.

Q3: I noticed that in the experimental results on the complete dataset in Figure 8, there are only results of UPDGD on the Corel5k and Pascal07 datasets. Could you provide more results on other datasets?

We sincerely apologize for this and we have provided the complete experimental results on the five datasets without any missing views and labels in Figure 1 of the PDF document (provided via the anonymized link). It can be observed that our Model (COME) achieves superior performance across six metrics compared to other methods on most datasets.

Q4: Is the data still incomplete in the inference phase? I know that views and labels are not complete in the training, so is this still the case when inferencing?

We still maintain the view’s incompleteness in the inference phase, but we do not use any missing settings for labels as the evaluation criteria for real performance.

Q5: In Figure 2(b), is this label distribution common? Are the patterns found in a single dataset universal?

Thanks to your questions. The label distribution of the other datasets is summarized in Figure 2 of the PDF document. We observed that the number of labels per sample varies across datasets and within individual datasets. This inherent variability makes the Top-K pseudo-labeling strategy [1] unsuitable for such scenarios.

Q6: From Figure 4, why is the negative impact of missing labels smaller than that of missing views? More discussions should be added.

Thank you for your discussion. Intuitively, in the fusion stage of multi-view representations using a Mixture of Experts (MoE), the learned weights reflect the relative importance of different view representations within the fused representation. The absence of key view representations causes rapid degradation in the quality of multi-view joint representations, which poses significant challenges to multi-label classifiers. Additionally, we introduce a dual-branch soft pseudo-label imputation strategy to mitigate the multi-label missing problem. We present the ablation results of the “dual-branch soft pseudo-label imputation” strategy under varying missing label rates. The results of COME without dual-branch soft pseudo-label imputation strategy are summarized in the following table:

And the results of COME with dual-branch soft pseudo-label imputation strategy are summarized in the following table:

For clarity, the results are presented in Figure 3 of the PDF. The experimental results demonstrate that the proposed strategy effectively mitigates the negative impact of missing labels, particularly under a high missing rate of 70%.

References:
[1] Class-Distribution-Aware Pseudo-Labeling for Semi-Supervised Multi-Label Learning. NeurIPS 2023.

Anonymous link:
https://anonymous.4open.science/r/6513/rebuttal.pdf

We greatly appreciate your thoughtful and detailed feedback, and we will address your questions one by one.

Q1: It would be beneficial for the authors to clarify through experiments why dual-branch model is preferred compared to a single-branch structure.

Thank you for your suggestions. We carried out experiments on the Pascal07 dataset to investigate the differences between dual-branch and single-branch architectures. The results are summarized in the following table:

The experimental results demonstrate that the dual-branch architecture exhibits superior performance compared to its single-branch counterpart.

Q2: Some details need to be improved, such as the typo “we suppose all the task-relevant information contained by multi-view shared information.” in line 97. I suggest the authors to polish it to help improve the fluency.

Thank you for your suggestion. In line 97, the original sentence has been revised to the following in the manuscript: “We suppose that all the task-relevant information is contained in the multi-view shared representation.”.

Q3: Equations in the Appendix should not appear in the main text commonly, such as Eq. (14).

I am sorry for the serious typos. The corresponding equation is Eq. (6) rather than Eq. (14). We have updated this in the revised manuscript.

Q4: Please present the comparative experiments using hard labels and soft pseudo-labels.

Thank you for your suggestion. We compared ​the soft pseudo-labeling strategy with ​the hard pseudo-labeling strategy and summarize the results in the follwing table:

The experimental results demonstrate that the soft pseudo-labeling strategy exhibits superior performance compared to the hard pseudo-labeling approach.

Q5: The cross-view reconstruction in Eq.(8) does not use the joint representation z. Could you explain it? Additionally, is the joint representation z utilized in the classification tasks? I think the authors should provide a detailed clarification on this.

Thank you for your question. Combining the Mixture of Experts (MOE) with Eq. (15), we derive Eq. (17). This implies that the mutual information between a given view and others can be enhanced through cross-view representation reconstruction, thereby improving semantic consistency. Moreover, the joint representation z has indeed been utilized in multi-label classification tasks, and we have supplemented this in the revised manuscript.

Q6: In Figure 7, the hyperparameters and appear to tiny influence on the experimental outcomes. Could the authors elaborate on the empirical or theoretical rationale behind this observed insensitivity?

Thank you for your question. We conducted sensitivity analysis experiments to investigate the influence of $\lambda_1$ and $\lambda_2$ on five datasets by varying their values. Subsequently, we identified hyperparameter configurations that ensured stable model performance and refined the intervals for detailed analysis. These results are visualized in Figure 4 of the PDF document (available via the anonymized link). The experiments show that the model maintains stable performance across a wide range of $\lambda_1$ and $\lambda_2$, demonstrating high robustness to these hyperparameters.

Q7: The difference in Figure 4(b) is very small. Is this caused by using pseudo labels? How does the model perform at different missing rates if the pseudo-label module is disabled?

Thank you for your valuable discussion. We conducted ablation experiments on the pseudo-labeling strategy under varying label missing rates. The performance of COME without this strategy is summarized in Table 1 of the provided PDF document. And the results of COME with the proposed strategy are presented in Table 2 of the PDF. For a clear observation, we show the results in Figure 3 of the PDF. The experimental results demonstrate that the proposed strategy mitigates the negative effects caused by label missing.

Q8: Figure 3 does not seem to be mentioned in the text. The font styling in Figure 6 is inconsistent.

Thanks to the suggestion. The manuscript will be revised to include extended annotations for Figure 3 and to ensure font style consistency in Figure 6.

Anonymous link:
https://anonymous.4open.science/r/6513/rebuttal.pdf

Thank you for your valuable review. We will address your questions one by one.

Q1: Three important hyperparameters in equation 13, $\lambda_1$, $\lambda_2$, $\beta$, need the subsequent experiment to analyze their impact.

Thank you for the suggestion. In Appendix D, we investigate the impacts of hyperparameters $\lambda_1$ and $\lambda_2$ across three datasets. Experimental results indicate that our method is robust to hyperparameter variations. For visual clarity, Figure 4 in the PDF (provided via the anonymized link) illustrates heatmaps of the average precision (AP) on two datasets. Additionally, in Section 4.3, we analyze the hyperparameter $\beta$, which balances view-shared and view-specific information. When the $\beta$ is too small, the model tends to focus excessively on view-specific representations, hindering its ability to learn effective shared representations. Conversely, when $\beta$ is too large, the model overemphasizes the consistency of shared representations, thereby compressing an excessive amount of information.

Q2: The authors should check the full text carefully for grammatical errors, such as ‘consitent’ in line 107, “random” in line 116, to meet the high quality requirements of ICML.

Thank you for the correction. The typo “consitent” has been revised to “consistent” and “random” in context has been updated to “randomly” to ensure grammatical accuracy. We thoroughly reviewed the manuscript to detect and fix spelling errors, ensuring compliance with ICML’s high quality requirements.

Q3: In line 112, “As shown in Fig. 4”, what does that mean? You may want to say “Fig. 1”.

We apologize for the confusion. We want to illustrate the performance degradation of inadequate contrastive learning by “Fig.1” rather than “Fig.4”.

Q4: Subfigure needs captions as well, such as that in Fig. 2.

Thank you for emphasizing the need for clearer figure captions. In the revised manuscript, we have incorporated detailed captions for each subfigure in Fig. 2.

Q5: In line 319, “further technical” should be revised to “and further technical”.

Thank you for your suggestion. We have revised the phrase “further technical” to “and further technical” in the manuscript as recommended.

Q6: Why did the author use two separate models for pseudo-tag generation and padding?

Some previous works that train the model using pseudo-labels generated by the same model may lead the model to accumulate error once incorrect pseudo-labels are generated [1,2]. To address this issue, we propose a dual-branch soft pseudo-label generation strategy for missing label imputation. Experimental results on Pascal07 dataset demonstrate that the dual-branch architecture exhibits superior performance and enhanced robustness compared to its single-branch counterpart. As shown in the following table:

Q7: In Eq. 11, whether the selection of threshold value is empirical or not?

Thank you for your question. For the upper bound of the threshold $\tau_h$, we simply set it to 0.5, which is conventionally used as the threshold in classification tasks. We have conducted analysis experiments to investigate the influence of $\tau_l$ on five datasets. For pascal07 dataset, we varying the value of $\tau_l$ from [0, 0.4] with an interval of 0.1. The results are summarized in the following table:

From the results, we simply set the $\tau_l$ as 0.25 for Pascal07 dataset. Intuitively, during the initial phases of training, a more stringent threshold is employed to ensure the high quality of pseudo-labels, while the threshold is gradually relaxed to extend the range of generated pseudo-labels.

References:
[1] Dual-Decoupling Learning and Metric-Adaptive Thresholding for Semi-supervised Multi-label Learning. ECCV 2024.
[2] Debiased self-training for semi-supervised learning. NIPS 2022.

Anonymous link:
https://anonymous.4open.science/r/6513/rebuttal.pdf

Thank you for your valuable suggestions! Below, we will address each of your questions.

Q1: Providing comprehensive experimental details (e.g., dataset-specific hyperparameters) or releasing the source code would significantly enhance the transparency of this work.

Thank you for your suggestion. We will release the code and pre-trained checkpoints upon acceptance. For reproducibility we build on the open source implementation of COME.

Q2: Some statements are not clear enough: on Equation (16) and line 625, “Since we adopt the MoE fusion strategy to model $p(z\vert X^V)$, we have:” which is not very clear.

We sincerely apologize for the confusing. Similar to previous work [1], we propose to factorise the joint variational posterior as a combination of unimodal posteriors, using a Mixture Of Experts (MOE), therefore we have Eq. (16).

Q3: In the paper, the pseudo label generation models are able to effectively mitigate label missingness. However, beyond the ablation studies, I recommend further investigation into the efficacy of this claim.

Thank you for your suggestion. We conducted an in-depth investigation of the dual-branch soft pseudo-labeling strategy under varying label-missing rates on the Pascal07 dataset, with the corresponding four metrics ​presented in Figure 3 of the PDF document ​provided via the anonymous link. The experimental results demonstrate that the dual-branch soft pseudo-labeling strategy exhibits superior performance under elevated label-missing conditions, particularly when the missing rate is 70%.

Q4: (Line 88) The period before “(Federici et al., 2020; Tasi et al., 2020)” should be removed. Double check for this manuscript is crucial.

We appreciate the reviewer’s careful attention to detail. The period before the citations in line 88 has been removed in the revised manuscript to adhere to proper punctuation conventions. Thank you for highlighting this oversight.

Q5: I suspect that the dual-branch design could incur high computational costs (e.g., training time or memory consumption). However, the authors do not provide runtime efficiency with other models. Including such experiments would strengthen the practical relevance of this work.

Thank you for your constructive suggestion. We conducted a fair comparison between COME and three other deep models capable of simultaneously addressing both missing views and missing labels under identical experimental protocols, and report their training time and inference time (Unit: seconds) in the following table:

As indicated in the table, COME requires a prolonged training period on both datasets. The additional computational overhead is primarily attributed to the dual-branch architecture in our method. In future work, we plan to explore efficient alternatives to the dual-branch soft pseudo-label generation strategy to optimize computational efficiency.

Q6: Why do the authors introduce some methods that can not handle the missing views and labels simultaneously?

Thank you for this important question. Due to the scarcity of methods in iM3C that simultaneously address missing views and labels, we incorporated approaches handling single missing scenarios (e.g., missing views or labels) as comparative benchmarks. Specific details are provided in Appendix C. Moreover, the experimental results demonstrate that models handling individual missing scenarios exhibit suboptimal performance, further underscoring the inherent challenges of iM3C tasks.

References:
[1] Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models. NeurIPS 2019.

Anonymous link:
https://anonymous.4open.science/r/6513/rebuttal.pdf