Enhance Multi-View Classification Through Multi-Scale Alignment and Expanded Boundary

W: Weakness; Q: Question

W1. We supplement results with the general prototypes on four datasets in the ablation study, which are shown in Table 1. For convenience, we also provide the results of the model with the expanded boundary. From the results, we can find that the model using the general prototype loss exhibits inferior performance compared with the models using the expanded prototypes, which means the expanded boundary could further improve the prototype-scale alignment module.

Table 1: Supplementary ablation analyses on multi-scale module.

W2. We provide the elaborate complexity analysis of MAMC, which comprises representation extractors, multi-scale alignment module, and joint classifier three parts. The representation extractors need $\mathcal{O}(NVd)$ complexity, the instance-scale alignment needs $\mathcal{O}(N^2V^2d)$ complexity, the prototype-scale alignment needs $\mathcal{O}(4NVd+NVKd)$ complexity, the joint classifier needs $\mathcal{O}(NVd)$ complexity. Then, the overall complexity is $\mathcal{O}(N^2V^2+6NVd+NVKd)$.

Q1. The t-SNE visualization is obtained by concatenating the representation vectors from all the views. Specifically, one instance would be encoded as several view representations, which are then concatenated as a final representation vector. The final representation vector is input into the t-SNE algorithm to obtain the visualization. To provide better visualization of the comparative models, we apply the hyperparameter settings provided in their official codes, which would present satisfactory results.

Q2. As you mentioned, MAMC can be regarded as a robust model to handle uncertainty by integrating fuzzy set theory, which quantifies how much the samples belong to a class. And the differences between the comparative models and MAMC lie in the intentions of the models and the tools to deal with the uncertainties.

Firstly, the intentions of the comparative models and MAMC are different. For the comparative models, they delve into improving the inferior logits in the decision phase but MAMC focuses on learning clear boundaries in the representation space. In our opinion, the data uncertainties will hurt the representation learning and decision phases, which implies that modifying the logits might be sub-optimal. Therefore, we focus on learning clear decision boundaries in the representation learning phase, which may handle the data uncertainties more directly.

Additionally, with respect to the tools, comparative models are based on Dempster-Shafer Theory of Evidence to quantify uncertainty, which is a generalization of the Bayesian theory to subjective probabilities [1]. MAMC combines with fuzzy set theory which adopts the learnable membership function to represent varying degrees of uncertainty [2]. In summary, the comparative models focus on learning ideal subjective probabilities while MAMC aims at learning appropriate membership functions.

[1] Evidential deep learning to quantify classification uncertainty. NeurIPS, 2018.

[2] Robust TSK fuzzy system based on semisupervised learning for label noise data. IEEE TFS, 2020.

Q3. Thanks for your check, we will revise the wrong symbols in the final version.

W4-1. In our results, the losses decrease steadily and converge between 50 and 800 epochs across all datasets. However, as we are unable to provide the loss convergence figures in the rebuttal, we will include a detailed convergence analysis in the final version of the paper.

W4-2. Thanks for your advice. We will add the related references/websites of our employed datasets in the final version of our paper, which are now listed as follows:

(a) HandWritten: http://archive.ics.uci.edu/ml/datasets/Multiple+Features

(b) PIE: https://www.cs.cmu.edu/afs/cs/project/PIE/

(c) YoutubeFace: https://www.cs.tau.ac.il/wolf/ytfaces/

(d) Animal: https://www.cs.ucf.edu/~xmzhang/datasets/

(e) CCV: http://www.ee.columbia.edu/ln/dvmm/CCV/

(f) 100Leaves: https://archive.ics.uci.edu/ml/datasets/One-hundred+plant+species+leaves+data+set

(g) Hdigit: https://archive.ics.uci.edu/ml/index.php

(h) Scene15: http://www-cvr.ai.uiuc.edu/ponce_grp/data/scene15/

W3.

$N$ is the number of instances,

$K$ is the number of classes,

${\hat{p}}_i$ is the final prediction from Eq. (15),

${\hat{p}}_{ij}$ is the $j$-th element of ${\hat{p}}_i$,

${\hat{y}}_i$ refers to the one-hot vector label of the $i$-th instance,

and ${\hat{y}}_{ij}$ is the $j$-th element of ${\hat{y}}_i$.

W2-2. Feature fusion could be divided into several phases, such as early fusion and late fusion. Our model adopts late fusion, with a focus on addressing feature heterogeneity to facilitate more effective fusion. To tackle the feature heterogeneity arising from different data distributions, our model is designed to minimize the distances between similar samples across different views. By utilizing multi-scale alignment modules, similar samples are gathered together, reducing the distances between them and aligning the data distributions across views. Furthermore, the embedded expanded boundary adaptively gathers more similar samples, further minimizing distribution differences. By reducing these distribution discrepancies across views, the representations can be more easily fused in the late fusion phase. To provide a clearer and more comprehensive analysis, we present the results of an ablation study in Table 2. The findings indicate that the multi-scale alignment module could enhance the quality of the fused representations, effectively mitigating feature heterogeneity.

Table 2: Ablation analysis on the multi-scale module.

W2. We have selected a range of advanced comparative methods proposed in the past three years for our analysis. Among these, ETMC [1] is a well-known multi-view classification method that has been widely referenced and further developed by subsequent studies [2-3]. Besides, most comparative models focus on realizing trustworthy classification and could address data uncertainties which is also one merit of our model. And we have conducted the relevant experiments to handle noisy data in Appendix C.1. Therefore, these comparative methods are appropriate and representative. To offer a more comprehensive comparison, we also conducted additional experiments using some recent multi-view classification methods [4-5], with results presented in Table 1. Furthermore, we supplement the related works in the past three years and discuss the relevance between the comparative models and the proposed method below to demonstrate the effectiveness of our method, which will be added in the final version of the paper.

Investigation of related work: Existing multi-view classification methods can generally be divided into two categories: feature fusion-based and decision-based methods. Feature fusion-based methods focus on effectively combining features from all views. MV-HFMD [4] explores feature fusion by introducing a novel fusion scheme and mutual distillation to adapt neural networks for multi-view classification. Mmdynamics [6] dynamically assesses the informativeness of both feature-level and view-level data across different samples, ensuring trustworthy integration of multiple views. However, these methods often neglect feature heterogeneity, which undermines the effectiveness of feature fusion. For the decision-based methods, they are dedicated to making more reasonable decisions. ETMC [1] introduces a new paradigm for multi-view learning by dynamically integrating different views at the evidence level, thereby enhancing classification reliability. RCML [2] attends to that conflictive information in different views, and proposes a conflictive opinion aggregation strategy that exactly models the relation of multi-view common and view-specific reliabilities. MVAKNN [3] utilizes the Dempster–Shafer theory to integrate information from each view, successfully extending adaptive KNN to a multi-view setting. IPMVSC [5] constructs a multi-view fuzzy classification model inherited from the natural interpretability of fuzzy rule-based systems to realize interpretable classification. Many methods of this kind interact with the multiple views only in the decision phase, which cannot well integrate the information from all the views. In contrast, our model aligns view features during the representation learning phase and combines them in the decision phase, allowing for more effective and interactive information integration across all views.

Table 1: Classification results for supplemented comparative models.

[1] Trusted multi-view classification with dynamic evidential fusion. IEEE TPAMI, 2023.

[2] Reliable conflictive multi-view learning. AAAI 2024.

[3] Multi-view adaptive k-nearest neighbor classification. IEEE TAI 2023.

[4] Multi-view Classification Using Hybrid Fusion and Mutual Distillation. WACV 2024.

[5] Multi-view fuzzy classification with subspace clustering and information granules. IEEE TKDE, 2023.

[6] Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. CVPR 2022.

W1. The multi-scale alignment module in this paper is the carrier for the expanded boundary to enhance the ability to tackle feature heterogeneity. Though the multi-scale alignment module is commonly used to improve discriminability and semantic consistency of representations, existing works rarely focus on further enhancing these abilities through learning effective boundaries. To address this gap, we propose the novel expanded boundary as one embedded component of the multi-scale alignment module to reinforce the discriminability and semantics consistency while clearing the ambiguous boundaries. Additionally, we provide theoretical guidance to ensure the alignment loss is both reasonable and reliable. Ablation studies demonstrate that our approach could further improve discriminability and semantic consistency by showing better model performance.

Dear Reviewer 9Q5P，

Thank you for your time. We would like to kindly remind you that we have responded to your valuable feedback on our submission . If you have any further questions, comments, or concerns, we would be more than happy to address them.

We deeply appreciate the time and effort you have dedicated to reviewing our work and providing constructive feedback, which has been immensely helpful to us.

Please feel free to reach out if there is anything else we can clarify. Thank you again for your contribution to improving our research.

Best regards,

Authors.

W1. Three to five loss terms are common in multi-view learning since the construction loss and the cross-entropy loss are the common terms in the existing works [1-4]. Except for these terms, we only introduce two losses, namely the instance-scale alignment loss and the prototype-scale alignment loss for addressing the feature heterogeneity and the information redundancy. Besides, we simply set the penalty coefficients of the construction loss and the cross-entropy loss as 1 to alleviate the burden of hyperparameter tuning, which is also more convenient for attending to the effects of the proposed modules.

[1] Reliable conflictive multi-view learning. AAAI 2024.

[2] Triple-granularity contrastive learning for deep multi-view subspace clustering. ACM MM 2023.

[3] Trusted multi-view classification with dynamic evidential fusion. IEEE TPAMI, 2023.

[4] Self-supervised information bottleneck for deep multi-view subspace clustering. IEEE TIP 2023.

W2. Thanks for your suggestion. We drew Figure 4 in 3D by referring to [1], which might lack consideration for better readability. Following your suggestion, we now provide the elaborate results of the parameter sensitiveness analyses in Table 1 - Table 4 since we are unable to provide the revised 2D figures directly in the rebuttal. From the results, we observe that model performance initially improves and then declines as the hyperparameters increase. This indicates that all the hyperparameters contribute to model performance, validating the role of each loss term and module. Specifically, from Tables 1 and 2, we find that the model performs best when both $\alpha$ and $\beta$ are properly tuned. This suggests that the instance-scale alignment and prototype-scale alignment modules work together to enhance overall model performance.

Table 1: Hyperparameter analyses for penalty coefficients on HandWritten.

Table 2: Hyperparameter analyses for penalty coefficients on Hdigit.

Table 3: Hyperparameter analyses for temperature coefficients on HandWritten.

Table 4: Hyperparameter analyses for temperature coefficients on Hdigit.

[1] "Triple-granularity contrastive learning for deep multi-view subspace clustering." ACM MM 2023.

W3. We have tried our best to find the source codes of the above papers for discussion and comparison, such as searching in GitHub and contacting with the authors. However, we only found source codes of [1] and [3]. Here, we provide the discussions about these references in the following, and the comparative results on classification are shown in Table 5.

Discussion and comparison:

(1) MV-HFMD [1] delves into feature fusion and incorporates a novel fusion scheme and mutual distillation to repurpose off-the-shelf neural networks for multi-view classification. From the results in Table 3, MV-HFMD shows a very outstanding performance, which implies the significance of effective feature fusion in multi-view learning. However, it may neglect to address the feature heterogeneity that plays an important in facilitating the feature fusion, so the model performance is somewhat limited when compared with our model.

(2) SIBMSC [2] aims at learning the minimal sufficient latent representation for each view to obtain common information with the information bottleneck theory. It mainly focuses on the inter-view common information but does not consider the inter-cluster or inter-class information. However, our model focuses on both information. Additionally, we need to point out that this paper may not focus on multi-view learning from a fuzzy view.

(3) IPMVSC [3] proposes an anchor and graph subspace clustering strategy to discover and represent the actual latent data distribution for each view separately, and constructs a multi-view fuzzy classification model inherited from the natural interpretability of fuzzy rule-based systems. Therefore, this work focuses on using fuzzy rules to realize an interpretable classifier. However, our model attends to the feature-level issues and applies the fuzzy set theory in the representation space. Besides, from the results, we could also find IPMVSC does not work well on large datasets since the model makes it difficult to select the appropriate hyperparameters to balance the overfitting problem and model performance as the author stated [3], but our model works well on both small and large datasets.

Table 5: Comparisons of classification results for supplemented comparative models and our model.

Dear Reviewer p7J9，

Thank you for your time. We would like to kindly remind you that we have responded to your valuable feedback on our submission . If you have any further questions, comments, or concerns, we would be more than happy to address them.

We deeply appreciate the time and effort you have dedicated to reviewing our work and providing constructive feedback, which has been immensely helpful to us.

Please feel free to reach out if there is anything else we can clarify. Thank you again for your contribution to improving our research.

Best regards,

Authors.

W5. We find there is only UMDL that encounters the out-of-memory (OOM) issue. The main reason is that the model applies some complex algorithms in the pre-processing phase, such as the singular value decomposition algorithm, which is infeasible for large datasets. And we present the complexity and computational overhead analyses of all the models in Table 7 and Table 8.

Table 7 lists the complexity of all the models. Since the different models may introduce different and complex notations, it is hard to provide detailed complexity for all the models in a unified manner. For convenience, we only adopt $N$ as the data size to calculate the complexity for all the models in Table 7, and we can find that our model has the same level of complexity as some comparative models.

Table 8 provides detailed computational overhead analyses. Because MACs are defined as approximately twice the FLOPs, we only provide MACs in Table 8. Besides, we also list the concrete running time and Parameters for all the models. For a fair comparison, we set the same batch size and training epochs for all the models. From all the statistics from Table 8, we could find that our model is not cumbersome, but is competitive with part advanced models.

Table 7: The complexities among different models.

Table 8: The computational overhead comparisons among different models.

W1. Thanks for your check, we will revise the wrong symbols in the final version of the paper.

W2. The construction loss and the cross-entropy loss are two common losses in multi-view learning [1-3]. And we follow some works [2], [4] to fix the penalty coefficients of these terms to be one. By fixing the coefficients, we can get deep insights into the other two losses that correspond to the proposed modules. Meanwhile, the complexity of hyperparameter search will also be mitigated to some degree.

[1] Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE TNNLS 2023.

[2] Triple-granularity contrastive learning for deep multi-view subspace clustering. ACM MM 2023.

[3] Self-supervised information bottleneck for deep multi-view subspace clustering. IEEE TIP 2023.

[4] MMatch: Semi-supervised discriminative representation learning for multi-view classification. IEEE TCSVT 2022.

W3. Because of the inadmissibility to provide revised figures in rebuttal, we present the results composed of the mean accuracy and variance in Table 1-Table 6. The revised Figure 5 will be presented in the final version of the paper.

Table 1: Noise experiments on Animal

Table 2: Noise experiments on CCV

Table 3: Noise experiments on HandWritten

Table 4: Noise experiments on Hdigit

Table 5: Noise experiments on PIE

Table 6: Noise experiments on Scene

W4. Thanks for your correction. Our original intention was to use "methods" for indicating the comparative methods in the same row but seemed to create ambiguity. And we will revise the ambiguity in the final version of the paper.

Dear Reviewer 1xxb，

Thank you for your time. We would like to kindly remind you that we have responded to your valuable feedback on our submission . If you have any further questions, comments, or concerns, we would be more than happy to address them.

We deeply appreciate the time and effort you have dedicated to reviewing our work and providing constructive feedback, which has been immensely helpful to us.

Please feel free to reach out if there is anything else we can clarify. Thank you again for your contribution to improving our research.

Best regards,

Authors.