Trusted Multi-View Classification via Evolutionary Multi-View Fusion

Q2: At the methodological level, what is the significance of concatenating pseudo-views with the original views? In principle, it seems that the pseudo-view already contains the information of the original view

Re: The pseudo-view already contains the information of the original view. However, its quality still limited due to the view imbalance problem. Moreover, the searched pesudo view will be optimized with other multiple views for trusted fusion. In this process, the problem of imbalanced multi-view learning exacerbates because as the searched pesudo view contains a disproportionate amount of information compared to individual views. Hence, we enhance each view within the fusion architecture by concatenating the fusion architecture's decision output with its respective view.

The strategy is inspired by cortico-thalamocortical circuit, where the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus. Besides, from the perspective of characteristic among early fusion and late fusion, multi-view features are typically integrated through neural networks to achieve late fusion representations. However, this approach may result in the loss of certain raw information within individual views. On the other hand, early fusion methods combine features at an earlier stage but face challenges such as feature heterogeneity and high sample complexity. To address the limitations of both approaches, we innovatively extract late fusion information from the fusion architecture discovered in the first stage. This information is then incorporated into the second stage, where it is concatenated with the original views to generate pseudo-views. This strategy bridges the gaps in both early and late fusion methods, leading to improved results.

Based on above analysis, we concatenate pseudo-views with the original views. The experiment results (See Table3 that is part of Table 5 in the original manuscript) show that the strategy is very effective. TEF$_{2}$ directly uses the pseudo-view without concatenating with the original views for trustworthy fusion.

Table 3

Thank you again for professional comments. Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1 and Q1: Could you please describe in detail the differences from other multi-modal NAS methods, such as the DC-NAS (Divide-and-Conquer Neural Architecture Search for Multi-Modal Classification) mentioned in the manuscript?

Re: This is a very good comment. The main difference between TEF and other multi-modal NAS methods is that the pesudo view searched by TEF will be enhanced by concatenating the fusion architecture's decision output with each view within the fusion architecture while ones searched by other methods are not. The strategy alleviates the view imbalance problem, which allows us to obtain better quality pesudo view than other multi-modal NAS methods. This has been verified by the empirical results that TEF armed with the our obtained pesudo view achieves a better performance than others. The results are shown in Table 1 and 2, where TEF and TEF$_{1}$ denote that TEF is armed with our pseudo-view and one induced by other NAS methods, respectively. That is, our pseudo view is enhanced by concatenating it with the fusion architecture's decision output, while the compared one is not. Table 1 is part of Table 5 in the original manuscript. Table 2 is provided by conducting more experiments.

Table 1

Table 2

Additionally, we want to clarify that the paper aims to further release the potential of the trusted multi-modal learning methods (TMML) by introducing high-quality pesudo view into TMML and breaking its late-fusion limitation. We formulate the pesudo view generation as a population-based multi-view neural architecture problem in the framework of TMML. It is natural that other multi-modal NAS methods are easy to be coupled into TEF. We indeed do not pay more focus on the NAS design, however the pesudo view enhancement is novel and original.

W2: Additionally, I'm uncertain whether the proposed method can be applied to large-scale … I am open to increasing the score upon receiving a well-justified explanation.

Re: As you mentioned, evaluating the fitness of all $N$ chromosome vectors indeed requires too much computing time at each generation. In fact, this still is open problem in population-based search methods. Fortunately, evolutionary multi-view learning has provided many accelerated solutions such as fitness caching (FC), search guided by core structure (CS). In this paper, we use these strategies to accelerate the TEF, and find that the version of TEF armed with FC and CS is very high-efficiency. Obviously, new developments of the acceleration techniques in the evolutionary computation community can be readily integrated into the TEF, achieving a more higher-efficiency implementations of TEF if more effort is made. Table

Additionally, it is hard to afford the overload that each modality backbone network is also searched. The researchers provide a tradeoff between effectiveness and efficiency by fixing the backbone network and only searching the fusion strategies, such as [1-5]. In this paper, we follow the manner for dealing with large-scale end-to-end multimodal datasets. Moreover, in the era of big modal, training time seem not be limitation for algorithm application. One algorithm can be thought to be good if the inference time is within the user's tolerance range in terms of efficiency. TEF is the kind of algorithm.

Last, we would like to explain why chose evolutionary NAS. The NAS methods can be roughly three categories: gradient-based NAS (GNAS), reinforcement learning-based NAS (RLNAS), and evolutionary NAS (ENAS). GNAS requires a predefined search space and substantial memory, RLNAS relies on extensive computational resources, ENAS offers advantages such as global search capability, flexibility, and parallelization. This makes it particularly suitable for handling complex multi-view tasks and large search spaces.

[1] Core-structures-guided multi-modal classification neural architecture search, IJCAI,2024

[2] DC-NAS: Divide-and-conquer neural architecture search for multi-modal classification, AAAI,2024

[3] BM-NAS: Bilevel multimodal neural architecture search, AAAI,2022

[4] Deep multimodal neural architecture search, ACMMM,2020

[5] MFAS: Multimodal fusion architecture search, CVPR,2019

Dear Reviewer a8gx,

We thank you for your thorough review of our paper and for providing constructive feedback that has significantly contributed to its improvement. Your insights have been invaluable in helping us refine our work.

We sincerely hope that our responses have sufficiently addressed the issues you highlighted in your review and follow-up comments. As the author-reviewer discussion period approaches its end, please do not hesitate to let us know if there is anything further we could do to improve your impression and final rating of our work.

Dear Reviewer a8gx,

We are pleased to address your concerns. And thank you very much for raising the score.

Best regards,

the authors.

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

Q1: The paper indicates that introducing a single pseudo-view is effective; have there been any attempts to use multiple pseudo-views to further enhance the results?

Re: Yes, we have considered this and conducted experiments on the NUS dataset. The table below shows the differences between not introducing any pseudo-views and introducing 1, 2, 3, or 4 pseudo-views. Introducing a single pseudo-view typically results in better performance, while introducing two or more pseudo-views does not significantly improve outcomes. This is because introducing one optimal pseudo-view has already enabled sufficient feature interaction, and additional pseudo-views may lead to conflicts or feature redundancy.

Q2: Is this method a general concept? Can other trustworthy fusion methods also adopt similar strategies to improve model performance?

Re: This method can be viewed as a plug-and-play module, suitable for conventional reliable multi-view classification methods. By introducing a pseudo-view generation framework, other methods can effectively address the common issues of traditional multi-view methods: these methods mostly use late fusion techniques, tend to merge the local features of each view, and overlook the global feature interactions between views. The introduction of pseudo-views not only compensates for this deficiency but also significantly enhances the overall performance of the model by integrating global features. Therefore, other credible fusion methods could consider adopting a similar strategy to achieve comparable improvements in model performance. We integrated this method into the classic TMC framework, and the results displayed in the table below illustrate its effectiveness. Compared to the original TMC, the enhanced TMC_TEF model demonstrates significant performance improvements across seven datasets. Specifically, the improvements on the PIE, Scene15, and NUS_1 datasets are 4.18%, 12.7%, and 10.43%, respectively.

[1] Trusted Multi-View Classification 2021 ICLR

Q3: What do you believe is the fundamental reason behind achieving state-of-the-art results?

Re: The main reasons include: First, most credible fusion methods employ a late fusion strategy, which limits the interaction of information between views, consequently leading to suboptimal utilization of multi-view data. To overcome this challenge, we adopted an evolutionary multi-view architecture search method, creating a high-quality fusion architecture to serve as a pseudo-view. This architecture abandons the traditional constraints of late fusion, allowing features to fully interact and effectively aggregate global features rather than just focusing on local features. Second, the introduction of pseudo-views may exacerbate the imbalance issue in multi-view learning, as the pseudo-views contain more information relative to individual views, which could potentially weaken model performance. To mitigate this issue, we enhanced each view within the fusion architecture by concatenating the decision outputs of the fusion architecture with their corresponding views, thereby enhancing the efficacy of the pseudo-views and further optimizing the overall performance of the model.

The authors have addressed my concerns by conducting more experiments and providing more explanations. I would like to increase my score.

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1,Q1: Can you provide more justification or empirical evidence showing that learning imbalance within pseudo-view-guided multi-view classification significantly impacts model performance?

Re: One of the main topics in multi-view learning is how to effectively integrate heterogeneous information from different views. However, although multi-view learning aids in comprehensively understanding the world by integrating information from various senses, most models often fail to satisfactorily achieve multi-view collaboration and do not fully utilize all views. Additionally, while it is anticipated that multiple input views would enhance model performance, we actually find that although multi-view models outperform single-modality models, their potential is still not fully realized [1] [2] [3]. Particularly, the introduction of pseudo-views can exacerbate this imbalance issue, where easily trainable views may suppress the potential of pseudo-views, leading the entire model to suboptimal performance. In Figure 3 of the paper, we demonstrate the performance difference before and after addressing the imbalance issue with pseudo-views, showing noticeable changes. Furthermore, we have added results from seven datasets commonly used in TMVC, leading to a consistent conclusion that the learning imbalance in pseudo-view guided multi-view classification has a significant impact on model performance.

W2,Q2: The benchmarks used in the paper seem different from those used in prior TMVC studies. Were the TMVC methods evaluated under the same conditions as your proposed TEF? Clarifying this would address concerns about fairness in experimental comparisons.

Re: We have clearly stated in Appendix A.4 that the other TMVC methods and TEF are evaluated under the same conditions. Specifically, these methods utilize the same data processing workflows, dataset partitioning strategies, and consistently employ a 128-dimensional view dimension. Additionally, we have meticulously adjusted the parameter settings of the other TMVC methods to ensure they achieve better performance. Based on these fair comparison conditions, the results ensure the fairness of the evaluation.

To fully demonstrate the advantages of our method and further verify its fairness, we also conducted evaluations on seven benchmark datasets commonly used by other TMVC methods. Using the same data partitioning strategy as Xu et al. (2024), we repeated the experiments ten times randomly and calculated the average results and standard deviations. The results show that our method exhibits significant advantages across all seven datasets.

Dear Reviewer jfaY,

We thank you for your thorough review of our paper and for providing constructive feedback that has significantly contributed to its improvement. Your insights have been invaluable in helping us refine our work.

We sincerely hope that our responses have sufficiently addressed the issues you highlighted in your review and follow-up comments. As the author-reviewer discussion period approaches its end, please do not hesitate to let us know if there is anything further we could do to improve your impression and final rating of our work.

Best regards,

The authors.

Thanks for your reply. I am sorry for the late response.

Firstly, I found your response somewhat confusing due to the inadequate citation of sources. It was difficult for me to trace each reference to its corresponding article, which caused considerable difficulty in understanding your points.

To better understand the development in this field, I carefully reviewed the literature related to [1]. You just mentioned early feature interactions, so I would like to ask how the method presented in [2] differs from yours. I was also surprised to find that you cited experimental data from this work but did not provide a direct comparison with it. Additionally, there are other relevant works, such as [3], which seem to bear some relation to your study. Given its publication timeline, I am unsure whether this work was available before your submission. For now, I believe it would be sufficient for you to clarify the similarities and differences between [2] and your approach.

I was quite perplexed by your mention of the evolutionary optimization pseudo-view method. The adaptive function is directly determined by downstream performance (classification accuracy), which means it is entirely supervised. This raises a tricky question: could there be a more detailed division of the training set, or further "evolution" across multiple dimensions? In any case, since the evolution is ultimately supervised, it seems that this work is quite incremental. If the pseudo-view evolution were based on other self-supervised or unsupervised metrics, I believe it would offer more insightful contributions.

[1] Xu et al. Reliable Conflictive Multi-view Learning, AAAI 2024

[2] Huang H et al. Trusted Unified Feature-Neighborhood Dynamics for Multi-view Classification, arXiv'24

[3] Fu et al. Core-Structures-Guided Multi-Modal Classification Neural Architecture Search, IJCAI'24

Thank you for professional comments. We have tried our best to address your questions and revised our paper by following suggestions from all reviewers.

W1: I am interested in your concatenation operation. Why is the late-stage fusion information only concatenated with the pseudo-views and not with the original views? What effects would concatenation have?

Re: In the first stage, evolutionary neural architecture search generates an optimal feature interaction fusion architecture. In the second stage, the method produces a high-quality pseudo-view. However, the training difficulties between the single-view training architecture and the feature interaction fusion architecture exacerbate the imbalance problem, leading to insufficient training of the feature fusion architecture. Auxiliary post-processing features generated in the first stage can assist in the training of the second stage to achieve the desired performance. It is evident from the table that the use of feature concatenation and its placement significantly impacts the performance of the TEF architecture. TEF^0 introduces a pseudo-view framework, TEF^1 introduces pseudo-views without concatenation, TEF^2 performs concatenation before the classification layer, and TEF^3 concatenates within the original view. Comparing TEF^3 and TEF^1, TEF^3 shows a significant performance improvement on all datasets, including a 9.44% gain on YouTubeFace. Furthermore, the performance of TEF^3 surpasses that of TEF^2, better preserving and utilizing multi-view information when concatenating at the original view stage rather than after view fusion but before classification.

W2: Why did you choose evolutionary computation for generating pseudo-views? Have you considered gradient neural architecture search or reinforcement learning?

Re: We have comprehensively considered three neural architecture search (NAS) methods and ultimately chose evolutionary neural architecture search (eNAS) for the following reasons: Compared to gradient-based NAS, which requires predefined search spaces and substantial memory, and reinforcement learning-based NAS, which relies on extensive computational resources, evolutionary NAS offers advantages such as global search capability, flexibility, and parallelization (Liang et al., 2021). This makes it particularly suitable for handling complex multi-view tasks and large search spaces.

W3: Can early fusion features not assist in training? Why was the final choice made to use late-stage features to address view imbalance?

Re: We conducted an analysis on the impact of extracting features from various layers for view enhancement, using the Reuters3 dataset. Specifically, we concatenated features from the last four layers of the fusion architecture to assess their final effect in the TEF (Tensor Ensemble Framework). Assuming the fusion architecture consists of n layers, we extracted features from the n-th, (n-1)-th, (n-2)-th, and (n-3)-th layers, and also set a baseline that did not use any features for comparison. The experimental results, as shown in Figure 6, indicate that the closer the layer is to the final one, the better the performance. For instance, concatenating features from the n-th layer resulted in a 0.59% improvement in accuracy compared to concatenating features from the (n-3)-th layer. This is because the deepest layers of the fusion architecture contain the most complex and advanced representations of the model, which can capture more in-depth patterns and semantic information from the data. Enhancing the original view with multi-view representations post-fusion helps improve the expressiveness of the view and prevents information loss during the fusion process. This approach facilitates better outcomes in multi-view fusion architectures and alleviates the multi-view imbalance issue to some extent.

Thank you again for professional comments. Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.