Navigating Conflicting Views: Harnessing Trust for Learning

The main weakness of the paper the overall quality and clarity of the presentation: in trying to turn a 22-page paper into a 12-page one without removing/compressing any content, the authors made some choices/trade-offs that negatively impact the paper's readability. In its current form, after a reasonable Abstract and Introduction, the paper goes straight into 3.5-page theory section, followed by related work (which should precede it) and then the experimental section. The main contributions of the paper (Algorithms 1 & 2, and even the definition of the novel metric, MVAGT) are relegated to the 10-page APPENDICES, which also include many other details.

In this reviewer's opinion, the paper's quality & readability would be greatly improved by moving most of Section 2 to the APPENDICES. The new structure would be: Introduction, Related Work, Intuitive end-2-end running example (which will re-user some parts of the current Section 2), Algorithms & Metrics, and Experiments. In this form, the authors will have all the space required to INTUITIVELY explain the key concepts and to compare & contrast with existing approaches. Even though this requires adding a new section, it is worth doing, as it will greatly impact the paper's impact & penetration.

For the Related Work section:

• 1st pgf: please explain intuitively (i) why don't noise-robust representation "fully resolve conflicts", and (ii) the main ideas behind cluster-based, self-representation-based, and partially view-alligned approaches
• 2nd pgf: please compare-and-contrast TMC vs your approach
• 3rd pgf: please explain intuitively why (i) CMVC "may inadvertently lead incorrect predictions converge to incorrect ones", and (ii) why this is NOT the case for your approach

For the new "Intuitive end-2-end running example" section:

• please modify the title/term of TDTF, as it is confusing because it uses the word TRUST twice in a 5-gram; may change to "Trust Fusion based on Trust Discounting" or to something else that is easy to parse & understand.
• the new section could be based on the existing Titanic domain, but it should have at least three examples/instances: (i) one in which there is no conflict, (ii) one in which your approach solves the conflict, and (iii) one in which your approach fails to solve the conflict. If (ii) and (iii) have more then one major use-case, you should add one example for each of them
• Table 1 is a good start for a single example, but it should be expanded to all the examples mentioned in the bullet above. It should also explain (rather than rely on the reader to deduce) how the top part of the table is created from Figure 1 (one sentence would suffice), and how the bottom part is computed.
• please provide an intuitive explanation of BCF and how it is applied to the expanded version of Table 1 (which will include 3+ examples)
• please do NOT intermix the BCF story with A-CBF and ABF. If it is important to compare-and-contrast, do it at the end of this new section or add it to the related work section (in which case it should follow rather than precede this new section). Otherwise, relegate it to the Appendix.
• please explain how the values in the to-be-expanded Table 2 (to 3+ examples) are computed
• you should apply algorithms 1 & 2 to the examples in tables 1 & 2, and discus the results

Major Issues

Method

1. Proposition 2 introduces an important point: the prediction uncertainty for conflicting views is higher than that obtained through traditional evidence-based fusion, which can help flag incorrect predictions in multi-view fusion with conflicting views. However, there are some issues. In equation (7), the authors use KL divergence as a regularization term for incorrect evidence. During optimization, particularly on the feature-based datasets used in this study, this term may overlook the influence of correct predictions. This often drives evidence for all classes towards zero, resulting in a high uncertainty score even for trusted predictions. The slight increase in uncertainty (as shown in Table 2) does little to improve the identification of incorrect predictions.

2. If I understand correctly, this work aims to assess the trustworthiness of each view’s prediction using referral networks. When distrust is indicated, it implies a conflict with the ground truth, rather than with other views. How can the method theoretically guarantee an improvement in classification performance when dealing with conflicting views, especially if the referral networks themselves have poor performance?

3. Based on the points above, I would like to discuss with the authors how these two propositions theoretically guarantee the effectiveness of the proposed method.

Experiments

The experimental analysis needs to be more comprehensive. Specifically:

1. In Table 2, the results show that the proposed approach enhances the reliability of evidence-based fusion methods (i.e., BCF, A-CBF, and ABF). However, the experiments lack a direct comparison of these methods with and without the proposed approach in identical settings.

2. The selection of comparison methods is limited. It is recommended to include additional evidence-based multi-view methods, such as TMDOA [1], CCML [2], and TMNR [3].

3. All six datasets used are feature-based. To introduce more challenging tasks, consider adding multi-view datasets with text or image data. Additionally, incorporating real-world cases could visually demonstrate conflicts among different views and showcase the effectiveness of this method.

Clarifications

1. Line 322 states: “However, this approach may inadvertently lead correct predictions to converge towards incorrect ones, potentially jeopardizing model stability.” Please provide support for this point.

Minor Issues

1. In Table 1, the trust score for Dolphin (referral) is 0.90, but Line 179 states, “we know the trust score for Dolphin’s opinion is 0.95.” Please clarify this discrepancy.

[1] Liu W, Yue X, Chen Y, et al. Trusted multi-view deep learning with opinion aggregation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(7): 7585-7593.

[2] Liu Y, Liu L, Xu C, et al. Dynamic Evidence Decoupling for Trusted Multi-view Learning[J]. arXiv preprint arXiv:2410.03796, 2024.

[3] Xu C, Zhang Y, Guan Z, et al. Trusted Multi-view Learning with Label Noise[J]. arXiv preprint arXiv:2404.11944, 2024.

1. The paper presents an incremental innovation in the field of multi-view learning by introducing a trust-based discounting mechanism that leverages subjective logic for uncertainty estimation. While the approach offers a novel perspective, the methodological contribution appears to be primarily theoretical, with the actual implementation relying on existing techniques such as the loss functions from the Trusted Multi-View Classification. This suggests that the paper may not offer substantial improvements over current methods in terms of loss function optimization.

2. In terms of experimental validation, while the paper does provide a comprehensive set of evaluation metrics and tests the method on six real-world datasets, there is room for expansion. The datasets used may not fully capture the challenges associated with uncertainty in multi-view learning. For example, Caltech101 dataset, which has the highest number of categories among the tested datasets, also happens to be the one where your model underperforms. This raises a concern about whether the model may struggle with datasets that have a large number of categories. It would be beneficial to include larger and more diverse datasets to better reflect the complexities and uncertainties inherent in multi-view classification tasks. This would provide a more robust assessment of the method's effectiveness and its ability to handle increased data volume and the number of views.

3. The motivation behind the paper is somewhat broad, and it does not clearly differentiate itself from existing methods that also address view conflicts, such as ECML in comparative methods. The paper could benefit from a more detailed discussion on how its approach differs from and potentially outperforms these methods, especially in scenarios with a large number of categories or high-volume data.

4. The paper lacks a detailed description of the view-specific neural network's architecture and implementation, which is crucial for understanding the model's performance and reliability. Without this information, the reproducibility of the method is compromised, and it is difficult to assess the specific contributions of the proposed method to the overall performance.

5. The ablation study is limited to hyperparameter tuning and does not provide a thorough analysis of the key components of the proposed method. A more comprehensive ablation study that systematically removes or modifies key components would provide clearer insights into their individual contributions to the model's performance. This could include varying the trust discounting factor or the influence of the evidence network on the final predictions.

The first weakness that stands out is the placement of the algorithm in the appendix. Understanding the method, as well as details about the training process are deferred to the end of the paper, making comprehension somewhat difficult. The process is key to understanding the paper and its results, and placing it in the appendix is a significant flaw, in my view.

The placement of the related work interrupts the flow of the paper - in my view, the context for MVCs and where this paper's contributions fit should come earlier.

The ablation study, while welcome, feels out of place without the inclusion of the algorithm within the body of the paper. The process loses significant clarity in doing so, and much of the results become harder to parse without it.

The original methods add significant time - with the exception of MGP, the author's methods take significantly longer to train.

The diversity of the instances of the datasets is not mentioned, making it difficult to know how well the trust mechanism works in diverse contexts.

There are minor spelling/grammatical errors, but none that affect understanding.