Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers

We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a “strong motivation” (HmXf), being “well-written and easy to follow” (eJLm), offering “novel insights” (HmXf) and “provid[ing] new tools” (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds “a major theoretical result” (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. Due to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted.

Dear Reviewer,

Thank you for reading our work with such attention to detail. We greatly appreciate the depth of your review and the specific suggestions you made, which have improved the manuscript considerably.

Q: “Geirhos (2020), in Section 2.2, already seems to provide a confidence interval for EC. The paper lacks a discussion of existing related work in this direction”

A: Thank you for this comment. You are right, Geirhos et al. indeed provided a basic variant of confidence intervals. But these confidence intervals were computed in a suboptimal way. They were based on the expected agreement $p_{exp}$ rather than the actual accuracies of the two classifiers, $p_1$ and $p_2$. This creates an ambiguity because, for example, $(p_1 = 0.5, p_2 = 0.5)$ as well as $(p_2 = 0.5, p_1 = 0.99)$ both produce the same expected agreement of $p_{exp}$ = 50% but different confidence interval widths. Moreover, these confidence intervals were necessarily centered around 0 because they were constructed only for independent observers. Our approach now enables calculating confidence intervals by making use of the observed accuracies, and for two dependent observers. We will add a section discussing this important difference in the paper. We appreciate your suggestion, which clearly improves our paper by making the difference to prior work more explicit!

Q: “The analysis assumes a ground truth EC of 0.5, which [...] leads to the widest confidence intervals. I recommend (1) adding a link to Appendix D in the main text [...]; (2) either moderate the strength of the conclusions drawn, or (preferably) expand Figure 3 to include a range of ground-truth EC values, indicating where existing studies and results fall along the EC spectrum”

A: Thank you for this actionable suggestion. We agree that this would improve the manuscript and have implemented (1) and (2). In fact, the EC values in most existing studies hover around 0.2 to 0.4, at which point the confidence intervals are very similar to the ones depicted, only the lower bound changes a bit. Unless you insist, we would stick with depicting the largest CIs, since we feel like otherwise we would be downplaying the importance of the result: Since we want to be a voice of caution, displaying the worst case scenario seems reasonable. We are more explicit about this now, though. Does this resolve your issue?

Q: “In Section 4.1, Figure 4 reports the average EC of individual models, averaged, among others, across humans”

A: You are absolutely correct, we are aware of this issue and delighted that you noticed it. In fact, an earlier draft of the manuscript included the figure you requested in the appendix, but we chose to exclude it out of fear that it would confuse less astute reviewers. Individual observers do indeed have somewhat different values, but our main message is not affected, and this is a consequence of the specific experiment itself, not our methodology. We have now included such a figure. However, the CIs in figure 4 are informative in the sense that if one accepts the paradigm, these are conservative CIs, and they are still very large: Every other choice of aggregation would make confidence intervals even larger.

We have numbered your clarifications 1 to 7:

C1: Yes, the figure would indeed look different, as the upper and lower bounds are not symmetric. In the context of the experiments by Geirhos et al, the lower bound is not of much interest because all values are positive, but we now include such a figure in the appendix.

C2: Yes, thank you for catching this typo.

C3: Yes, we now make the independence assumption more explicit. Additionally, we changed the notation in the proof to show underlying rather than observed distributions. Technically, both variants are correct here because, in this proof, both distributions are identical. We nevertheless changed it in line with your suggestion, which we appreciate because this change improves clarity.

C4: Yes, exactly. By “condition” we mean one intensity level within a corruption, so for 3 corruptions with 2 intensity levels each, you’d have 6 conditions.

C5: You are absolutely correct, thanks for the suggestion, we call it a Bernoulli distribution now.

C6: Good point, we explicitly state the Null hypothesis now.

C7: Thank you for the question, we may have glossed over some details here and assumed an unreasonable familiarity with Bayesian statistics. The problem here is the following: Our observers had a certain accuracy, the uncertainty of which depends of course on the number of trials (i.e. if you get 6/10 trials right, I’m less certain that your true latent accuracy is 60% than when you got 600/1000 right). When we model the independent observers, we need to account for our uncertainty about their accuracy, so instead of assuming some fixed accuracy, we draw it from a distribution, which is derived like this: We assume a binomial observer, i.e. an observer that draws each response to a sequence of binary questions from a Bernoulli distribution with an (unknown, ground-truth) parameter $p$, so the whole of its responses were drawn from a binomial distribution. Initially, we know nothing about the observer, so our prior, capturing our knowledge about $p$, is uniform over {0,1} (which is commonly considered the most conservative prior, so is adequate here). After having observed $\alpha$ incorrect responses and $\beta$ correct responses, the posterior distribution $P(p | data) = \frac{P(data | p) P(p)}{P(data)}$ is given by $Beta(\alpha, \beta)$ (because the Beta distribution happens to be the conjugate prior for the binomial distribution, i.e. is the closed-form solution for the posterior, see https://en.wikipedia.org/wiki/Conjugate_prior, where this is the first example). In line 221, there is actually a typo, because we accidentally inverted $\alpha$ and $\beta$, which we have now fixed - thank you for directing our attention here!

Thank you for the spelling- and notation-suggestions, we have implemented most of them. As exceptions, we still use $\kappa$ to denote EC, because it is indeed Cohen’s $\kappa$ (specifically, over the correct / incorrect responses). For this reason, we also kept responses $y$ instead of correctness $r$ in proposition 2 — where we now improve clarity by making this explicit thanks to your suggestion. Lastly, we also kept the superscript notation $P^{(1)}$ instead of $P_1$ to be consistent with the notation for responses, where we have a superscript j for the classifier and a subscript i for the trial, $y^{(j)}_i$. This is not our invention but has been done in other papers as well (e.g., in Gwet, 2016). Having said that, if you insist, we will make this change for the final manuscript version.

Regarding your discussion points (which are great points, so we partially address them now in the discussion section):

1. Yes, to achieve a more precise resolution with EC, we would need way more trials, which will eventually be prohibitively expensive for human observers. A possible remedy would be to design experiments cleverly, by selecting stimuli that bring out the differences efficiently. Also, EC can not only be used for comparisons between humans and DNNs, but also to compare different DNNs with each other, in which case the slow convergence is less of a concern. As multimodal models become increasingly popular and we are no longer limited to pure classification outputs, more sophisticated methods of comparison will have to be developed, but for the near future, most (M)LLM evaluations employ multiple-choice paradigms, to which EC is applicable.

2. This is an excellent question and suggestion. Indeed, given that EC can be separated into two factors, copy probability and accuracy-mismatch term, the uncertainty can be similarly decomposed. We are currently working on this based on standard error propagation methods, and preliminary results support our previous recommendations (e.g., that the accuracies of the classifiers should ideally be held close to 50% to maximize the information gain about the copy probability). Although this could be an interesting addition, we fear it goes beyond the scope of the current manuscript. We would defer a thorough analysis of this uncertainty decomposition to future work.

We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a “strong motivation” (HmXf), being “well-written and easy to follow” (eJLm), offering “novel insights” (HmXf) and “provid[ing] new tools” (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds “a major theoretical result” (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. Due to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted.

Dear Reviewer,

Thank you for providing valuable feedback on our work. We were delighted to read that you found our manuscript well-written and easy to follow and agree that being able to compute confidence intervals is useful. We believe that addressing your concerns has greatly improved the manuscript, especially figure 1 (see below).

Q: “The work is fairly incremental”

A: You are right to point out that bootstrapping to generate confidence intervals would not be novel. However, our contributions go beyond that: At the moment, there is influential work that uses EC to draw widely believed conclusions about behavioral alignment between ML models and humans. But the metric itself is still poorly understood. At this point, non-incremental, “disruptive” work (e.g. proposing yet another poorly understood metric) would probably not advance the field. Instead, we believe it to be crucial to improve interpretations of the existing measure. The novel contribution we make lies in presenting a well-interpretable, generative model. This allows understanding EC better (as a scaled copy-probability) and, as a practical application, to predict the width of confidence intervals in future experiments (as agreed upon by other reviewers). This will enable future work to interpret and use the metric correctly. Without our work, there is a real risk of people overinterpreting differences in rankings made with error consistency and drawing wrong conclusions.

Q: “The interpretation of the main results in Figure 1 is problematic. It was claimed that because the error bar was substantial, the gap between network v.s. human and human v.s. human should shrink. However, it could also be the opposite, that is, the gap was actually larger than what was implied in the difference in the mean of Cohen’s Kappa.”

A: Thank you very much for pointing this out, you are absolutely correct. The phrasing in Figure 1 was indeed wrong—be assured that we never interpret our results erroneously in the text; this was a graphical error in how we represent our results in Figure 1, not an error of substance. What we meant to say was of course that the uncertainty in the measurement is revealed, so the likelihood that the gap was found by chance increases. Thus, our main message does not change. Of course, we have now fixed the figure, to say “gap size uncertain”, and apologize for the mistake. Thanks again for raising this concern!

Q: “The analytical results in Proposition 2 and Section 3.2 seem to be based on relatively restrictive assumptions. [...] This needs to be clarified.”

A: Thank you for bringing this point to our attention, we were indeed not sufficiently clear. Actually, these assumptions are not restrictive: Since we place no constraints of any kind on the marginal distribution (e.g. we do not enforce it to have a certain shape), any overall distribution attainable for observer 2 can be represented as a weighted combination of the marginal of observer 1 and this latent marginal of observer 2; its whole purpose is to “cancel out” the deviation from the empirical marginal introduced by the marginal of observer 1. The misunderstanding might arise from the fact that when we talk about the distributions, we mean the (binary) distributions over correctness here, not the distributions over responses. We have now revised this point in the manuscript . Does this clarify our reasoning?

Q: “Another test [...] would be to test if one EC is larger (or smaller) than another EC. [...] Would this be possible?”

A: Thank you for this excellent question! We agree that testing the significance of a difference in EC-values is even more relevant than just rejecting a null-hypothesis of independence. Indeed, doing this is possible with our methodology. In the simple case with full data, we have responses to N images from humans and two models. We can then bootstrap jointly from these responses, compute ECs for both human-machine pairings, and compute a difference. Doing so for each bootstrap run, we obtain a distribution over those differences, which we can then evaluate as usual. We are currently looking into this and will add an example of such a calculation to our paper. Moreover, with our generative model, we can also evaluate differences even without access to the trials themselves, by first using the model to generate sequences of trials and then running the bootstrapping. Thank you very much for raising this point; discussing and answering it has made our submission stronger.

Q: “Practically, what would be an effective procedure for achieving [uniform marginal distributions of human responses]?”

A: Thank you for this question. What we had in mind was to control the difficulty of trials for humans by standard procedures, e.g. running pilot experiments to find stimuli sets on which human participants achieve 50% accuracy, or using adaptive procedures like staircase paradigms. Alternatively, there are various standard techniques in psychophysics that are used to modulate difficulty of fixed stimuli, such as decreased presentation time or using masking patterns. In any case, controlling the marginal distribution of one of the two classifiers is sufficient. We have elaborated on this point in section 2 of the manuscript. Does this answer your question?

Q: “Can the authors please explain in more detail how the bootstrapping procedure in Section 3.1 was applied?”

A: Thank you for raising this point. We have amended the manuscript to elaborate further on the details of bootstrapping. In Section 3.1 specifically, we apply the bootstrapping approach to the data from [1], which is a bit more complex than a standard experiment: Essentially, they collected multiple trial-sequence-pairs for every pair of observers. One might be tempted to concatenate all these sequences to then calculate one final kappa for each pair of observers. However, since the sequences were obtained from experiments with differing difficulties, doing so would overestimate the EC. So instead, one has to average the ECs themselves. When we bootstrap data, we of course have to follow the same procedure to obtain correct confidence intervals. Does this explanation sufficiently clarify our bootstrapping procedure?

We hope that our responses addressed your concerns satisfactorily, perhaps allowing you to reconsider the score you have given our submission. Of course, we would be very happy to engage in further discussion.

[1] Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., & Brendel, W. (2021). Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34, 23885-23899.

We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a “strong motivation” (HmXf), being “well-written and easy to follow” (eJLm), offering “novel insights” (HmXf) and “provid[ing] new tools” (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds “a major theoretical result” (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. Due to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted.

Dear Reviewer,

Thank you for your review. We were delighted to read that you believe our work "provides new insights" and that our new model is a "major theoretical result". We welcome your suggestions to provide supporting explanations. For that, we have written explicit algorithms for the procedures we describe in sections 3.1 and 3.2, which we would include in the camera-ready version upon acceptance. This does indeed facilitate the understanding of the bootstrapping procedure, which other reviewers also asked about, so thank you for the valuable suggestion.

We would like to thank all reviewers for their valuable feedback. We very much appreciate their assessment of our work as containing a “strong motivation” (HmXf), being “well-written and easy to follow” (eJLm), offering “novel insights” (HmXf) and “provid[ing] new tools” (1TAb). It connects to previous work (SWie, 1TAb, eJLm) and adds “a major theoretical result” (1TAb) which improves our understanding of the popular Error Consistency metric. We have implemented most of the reviewers’ suggestions in the newest version of the manuscript, and hope our responses will clarify all remaining points. The reviewers’ feedback has greatly improved the submission. Due to the rebuttal rules this year, we cannot upload a new version of the manuscript, but can only promise to make changes in the camera-ready version, should the paper be accepted.

Dear Reviewer,

Thank you for taking the time to review our paper. We appreciate that you found our paper to be clearly motivated, and believe that our generative model can be useful. Based on your feedback, we have amended the manuscript to elaborate further on the details of bootstrapping. In summary, we follow standard procedures as suggested by e.g. [1], which are very robust to parameter choices, see below for a more detailed explanation.

Q: “The assumptions of bootstrapping are not discussed. [...] For example, when the sample of humans is too small and cannot represent the population of interest”

A: While you are right that a fundamental requirement for our approach to work is a well-conducted study, like the one by Geirhos et al. (2021), our method is concerned with something else: Once we have an empirical measurement for one pair of observers, how can we estimate the uncertainty in that measurement? The point of bootstrapping is orthogonal to the question whether the sample of human observers is representative, which is, of course, also an interesting question, but not one we address.

Q: “Why is EC modelled with $p_{copy}$? Is it the only choice?”

A: Thank you for this question. In principle, it is possible that other generative models of EC exist, but our model based on $p_{copy}$ has the significant advantage of being very interpretable, providing us with an intuition for what Error Consistency really means. This is itself a contribution of our work: Previously, the metric was not well understood, so there was a risk of people using it incorrectly. Our model also allows us to simulate trial data, which is very useful for planning experiments and for the analyses we conducted here (e.g. we can estimate confidence intervals for Brain-Score models without access to most of their data). Does this answer sufficiently address your concerns? The definition of EC, which you also asked about, is outside of our control: It is, as you write, an established metric already, which measures a quantity of interest that is not captured by the percentage of correct answers given by two classifiers.

Q: “What are the takeaways we can get from analyzing with the confidence intervals but cannot with point estimates?”

A: When ML models are evaluated regarding their behavioral alignment with humans, confidence intervals are crucial for proper evaluation of the measure. For example, say we have two models, each with EC = 0.5 (close to the alignment between humans). Critically, if the confidence interval of one was very wide, say [0.3, 0.7], and that of the other was very narrow, say [0.49, 0.51]. Our interpretations are substantially different in the two cases: For the first, we could assume that the high EC value came from random fluctuations (especially in our context, where multiple classifiers are evaluated and the best one is considered), and it is not ruled out that the ground truth is that this first model has a relatively low alignment. On the other hand, we could be reasonably certain that the second model is well aligned with human behavior; the results are certainly not due to measurement noise. Thus, analyzing EC with confidence intervals substantially changes interpretations of point estimates here: They tell us whether observed differences in the point estimates are meaningful or not.

Details of Bootstrapping

Thank you for pointing out that our explanation of bootstrapping may have been too short. We have elaborated more on this point now, in summary: Given two sequences of length N from two observers with indices ${0,...,N-1}$, we first draw N indices randomly, with uniform probability and replacement. We then obtain bootstrapped trials by selecting the observers’ trials at the chosen index positions, yielding two new sequences of trials. From these new sequences, we can calculate new estimates of EC. Repeating this procedure multiple times yields distributions of EC values, thereby quantifying the uncertainty in the measurement. To address your concern about parameters (which bootstrapping requires very few of), we have also added a paragraph justifying our parameter choices. Basically, there is only one variable parameter, which we typically denote as M, the number of bootstraps. While higher M is always better, we find that a value of 10,000 is sufficient, and have added a figure to the appendix which illustrates how bootstrapped EC values converge as a function of M. The other choices, such as sampling N items with replacement for a sequence of length N and drawing uniformly is how a non-parametric bootstrap is performed, see [1].

[1] Efron, B., & Tibshirani, R. J. (1994). An introduction to the bootstrap. Chapman and Hall/CRC.