Bayesian Inference for Correlated Human Experts and Classifiers

Yes, the claim was basically supported by results in Section 7.3 and 7.4. But it seems the authors make a specific niche setting under this topic, and no other existing algorithms can be directly used to be compared with their method. They modify two existing methods as benchmarks, but it is hard to know whether the benchmarks are suitable.

Strengths:

• Well structured paper, easy to follow
• The proposed algorithm seems novel in the field, adding new knowledge to the community

Weaknesses:

• The reported results need to be further enhanced, e.g. MCMC and ML classifier design and influence to the final performance were not reported.

The proposed method is based on a Bayesian framework, and its effectiveness is empirically validated on four datasets against two baseline methods. The results clearly demonstrate that the proposed method consistently outperforms both baselines across all datasets in terms of the trade-off between the average number of expert queries and classification error. Additionally, the authors claim that their Bayesian model is well-calibrated, as evidenced by low expected calibration error (ECE) metrics in Table 1 and Figure 4.

Strength

• The manuscript is well-written.
• Figure 1 helps significantly in understanding the overall pipeline.

Weakness

• The proposed approach appears rather straightforward and incremental. Indeed, the authors themselves acknowledge in Section 2.2 that this paper extends the work of Showalter et al. (2024) by incorporating correlations among experts' predictions. Consequently, the primary novelty lies in modeling the latent probability $z$ using a hyperprior. However, I find this hierarchical Bayesian approach to be quite standard practice within the community. While there is certainly nothing problematic about utilizing established methods, for a submission to a top-tier conference like ICML, I would expect a more innovative methodological contribution. Therefore, I suggest the authors consider submitting their paper to a journal such as Knowledge-Based Systems, where incremental methodological advancements combined with practical significance are well-received. For Bayesian experimental design part, a similar setting has been investigated in [1], albeit with slight differences. [1] Bayesian Optimization for Building Social-Influence-Free Consensus, https://doi.org/10.48550/arXiv.2502.07166
• This paper appears to conflate the concept of consensus with majority voting. It is important to clarify that Bayesian consensus is not inherently restricted to a utilitarian (majority-based) approach. I am not entirely convinced that utilitarian aggregation is universally appropriate, especially in high-stakes contexts requiring careful deliberation, such as X-ray classification tasks. In the context of medical diagnostics like X-ray analysis, human expert labels should be treated as informed advice rather than definitive ground truths. Definitive ground truths in such scenarios are typically only attainable via invasive procedures, such as surgery, which are prohibitively costly. Hence, expert labels serve as practical proxies. The primary motivation behind aggregating multiple expert opinions is to enhance the reliability and robustness of these proxy judgments. Adopting a majority-vote strategy aligns with the "wisdom of crowds" principle. However, an egalitarian approach may be more suitable in situations where avoiding the risk of overlooking critical indicators is paramount. Given that individual doctors possess varied experiences and specialized expertise, aggregating multiple expert assessments serves primarily to mitigate the risk of missing rare but significant findings. In this egalitarian scenario, if any expert identifies an abnormality, the consensus decision should categorize the image as abnormal, independent of the majority opinion. Fortunately, the approach described in this paper seems sufficiently flexible to accommodate both utilitarian and egalitarian aggregation schemes. I encourage the authors to further develop their methodology to explicitly support a general framework, allowing practitioners to select the appropriate social-welfare functional based on their specific decision-making needs.

Yes.

I have concerns regarding the motivation for Consensus Prediction. Specifically, given the assumption that pre-trained classifiers are provided, it is unclear why expert consultation is necessary for prediction. Furthermore, it raises the question of why pre-trained classifiers cannot themselves be considered experts.

Furthermore, after framing this problem, the Bayesian framework is rather natural, and I do not consider the proposed methodology innovative and significant.

Also, they assume all experts cost the same, which isn't very practical.

The main claim made by this submission is that the proposed Bayesian model is appropriate and effective at tackling the proposed problem. I believe this claim to be well-founded, both from the modelling perspective (with similarities to many similar approaches and well-motivated modifications) and the empirical perspective (with its experiments).

Strengths

Regarding the Setting I think the proposed setting is very interesting, and to my knowledge novel.

Weaknesses

Regarding Presentation of Equations I think the presentation of some equations (e.g. Eq. (4) and Eq. (5), and throughout Appendix A) may be improved by including (in words) a brief description. For example, Eq. (4) may be read as "the predictive distribution of $y_*$ can be obtained as a nested expectation, where first correlations between experts are considered"

Regarding Novelty There have been several methods in the Bayesian combination literature that attempt the modelling of correlated experts (e.g., [1], [2]), and the entropy-based active learning approach is classical. That said, I think this is balanced by the interesting problem setting.