A Unifying Normative Framework of Decision Confidence

We thank the reviewer for their constructive feedback and literature suggestions. We will incorporate them into our paper.

We agree entirely that confidence could be different when more than 2 choices are available. In fact, there is a recent work showing that confidence is better modeled by the difference in probability of the top 2 choices as opposed to the posterior belief of the top choice, when three options are presented (Li 2020). Notably, we predict that this is very well modeled by the SoftMax function, where the probability of the least possible option converges to zero. Nonetheless, testing the model on such an experiment would be a very interesting follow-up work on our model.

We also fit the posterior entropy of the belief to subjects’ behavior in experiment 4.1. All AIC values were higher than those of other models (Table 1, rebuttal pdf). This is consistent with this model's poor performance on the three-choice dataset explained above (Li 2020).

Regarding modeling with Markov Processes, we agree that other frameworks are also available. However, the core concepts of all of these models are the same: Markovian system and Bayesian Inference.

Reference:

-"Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis," Hsin-Hung Li & Wei Ji Ma, Nature Communication 2020

We thank the reviewer for bringing up great points about testing the model in more and better experiments. We agree that these directions need to be discussed in the paper, and we will do so in the final version, which allows one more page. Current experiments, mostly including only 2 choices and 1 step of action selection, are insufficient to flesh out different aspects of confidence and its models. We believe the generalizability of our framework makes it a great candidate for testing and modeling confidence in more complicated setups. Future experiments could go in multiple directions. Here are a couple of examples:

1. Confidence assessment in a sequence of actions instead of one action: One important aspect of our model, which is not present in other models and even most experiments, is the ability to assess confidence in a sequence of actions (trajectory). Multiple common experimental setups in the field can be used to study confidence of a sequence of actions (whole decision) if confidence assessment is added to them:

1.1 Two-step task: One potentially interesting experiment is assessing the confidence of humans in the two-step task (Daw 2011). The two-step task is a well-known experiment in neuroscience that involves two binary choices in each trial. However, confidence has not been assessed in that experimental setup yet. It is interesting to see how the involvement of multiple actions for each decision affects confidence and whether such an effect is compatible with our model’s prediction.

1.2 Maze navigation with perceptual cues (odor navigation): In one of the classic papers on confidence in neuroscience, Kepecs et al. used the waiting time of the mice for the “expected incoming reward” as a measure of their confidence (Kepecs 2008). Combined with the odor navigation task, one can assess the confidence of mice when multiple actions are involved (e.g. navigating the maze and sampling). Studying the role of different regions, such as the orbitofrontal cortex and hippocampus, in this confidence assessment could be an exciting area of research, too.

2. Confidence in multiple choices (with different values): As also pointed out by another reviewer, two-choice tasks are too simple to distinguish between various models. Therefore, some researchers have started focusing on confidence assessment on multiple choices. For example, one study has shown that the difference between the probability of the top two choices explains confidence better than the posterior probability of the most likely state when three choices are presented (Li 2020). In that study, the reward was the same for all options. It is interesting to see how that works when the rewards of different choices are different. Notably, the softmax function can model the difference between the two best choices, by eliminating the least possible option. Therefore, we expect that such an experiment confirms our framework’s predictions.

Regarding additional results, we added another analysis in experiment 4.2, looking at the confidence rating based on the choice value difference (figure 1 one rebuttal pdf). While the confidence increases with the increase in the choice value difference in experimental data and both tested models, the predicted growth rate in our model is significantly closer to the experimental data. Specifically, our model predicts very slow growth of confidence with the value difference increase, just like the experimental data.

Regarding different strategies vs. one for confidence assessment, the literature points out both individual differences (Navajas 2017) and parameters that affect confidence, such as attention, confirmation bias, and various types of noises (sensory, motor, and decision) (Khalvati 2019, Li 2020). Notably, Bayesian frameworks could incorporate these effects, but the model would have more parameters. With the increased parameters affecting decision-making and confidence, one could expect even more significant differences between subjects’ behaviors.

References:

-"Neural correlates, computation and behavioral impact of decision confidence," Adam Kepecs, Naoshige Uchida, Hatim A. Zariwala & Zachary F. Mainen, Nature 2008

-"Model-based influences on humans’ choices and striatal prediction errors," Nathaniel D. Daw, Samuel J. Gershman, Ben Seymour, Peter Dayan, and Raymond J. Dolan, Neuron 2011

-"The idiosyncratic nature of confidence, " Joaquin Navajas, Chandni Hindocha, Hebah Foda, Mehdi Keramati, Peter E Latham 5, Bahador Bahrami, Nature Human Behavior 2017

-"Bayesian inference with incomplete knowledge explains perceptual confidence and its deviations from accuracy", Koosha Khalvati, Roozbeh Kiani, Rajesh P.N Rao, Nature Communication 2019

-"Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis," Hsin-Hung Li & Wei Ji Ma, Nature Communication 2020

We thank the reviewer for their feedback and questions.

Our framework is normative because the agent maximizes the reward and information entropy of the policy (eq 9). We presented our approach somewhat backward to make it more intuitive for confidence representations. As other reviewers also noted, we should emphasize in the paper that eq 7 seems arbitrary and heuristic until we reach eq 9. Importantly, maximizing entropy in the fitting is actually mainly related to epistemic uncertainty, where the agent does not know the underlying model and makes minimal assumptions about it.

In response to your questions:

1- Our intention with this sentence was just to communicate that the standard in the field is that confidence is mainly mathematically defined only for scenarios where different choices have the same potential reward. This is likely true because most perceptual decision-making paradigms (which is most frequently where in the past confidence has been defined) feature two choices which will give the exact same reward if they are the correct choice, given the perceptual stimulus, for the trial. We point this out because the goal of our model is to generalize to paradigms which have asymmetric distributions of reward which are much more closely related to real life scenarios.

2- A symmetric reward function is meant to denote the situation where any choice will give the same reward if it is the correct choice while an asymmetric reward function means that is the subject is correct, the reward they will receive depends on the choice they picked. As an example, the perceptual decision-making task in Section 4.1 has trials with both symmetric and asymmetric reward functions. The subject must choose if they think the stimulus was tilted to the right or left based on a brief picture of it they saw; in symmetric trials, they will get a reward of 1 if they correctly choose the direction the stimulus was tilted, while in asymmetric trials, they could get a reward of 3 if they choose right and that is the direction of the stimulus they were shown versus only getting a reward of 1 if they choose left and that is the direction of the stimulus they were shown. In this case, the asymmetric reward would encourage someone who was more unsure of their perception to choose the direction with the higher reward because that might be the option with the greater expected reward.

3- We evaluate the acting part to fit some of our model's parameters, specifically the internal and external observational noise. There is more information in lines 255-269 where we talk about fitting the subject's choices (to fit these subject’s choices we had to evaluate the acting part).

4- If you refer to Equation 10, this is the equation that we use for confidence in the later sections on experimental data; the priors from the first experiment are used to calculate the expected reward (in the exponent) of this equation. For the first experiment the equation we used for expected reward was $E_{s}[r(s,a)]= \int p (s│z)r(s,a)ds$ where $p(s│z)$ is the posterior probability of the state given the observation and so is defined as $p(s│z) \propto p(s) p(z|s)$ where $p(s)$ is the prior.

5- Observation likelihood doesn’t not include the prior information given to the subject about the trial while perception confidence does. So observation likelihood is $p(z|s)$ which is just the likelihood of the observation for that trial given the state the subject choose while perception confidence is $p(s)p(z|s)$ which combines the likelihood of the observation with the given prior information (which results in a Bayesian posterior, so this is the Bayesian confidence hypothesis).

6- We didn’t fit any parameters for experiment 2, however, we did fit some for experiment 1. The details of our fitting methods can be found in lines 248-275. We fit 3 different parameters: internal observational noise, external observation noise, and the confidence cutoff (which we used to binarize the continuous values of confidence we generated to the “high” or “low” assessments of confidence that subjects gave). External observational noise was fit using gradient descent while the other two parameters were fit using grid search with maximum likelihood estimation. Internal and external observational noise were fit on subjects’ choices while confidence cutoffs were fit on subjects’ confidence assessments.

7- The POMPD formulation is not strictly “necessary” for experiment 2 as the subject are tasked with only making one decision as opposed to the string of decisions represented in the POMDP framework. Our goal was to define a model that could generalize across many decision-making paradigms so even if the complexity is not fully utilized in experiment 2, it is included as a general framework for modeling decision-making tasks that are more complex than experiment 2.

We appreciate the reviewer's constructive feedback and comments.

Concerns: About the results, we would like to emphasize that in the first experiment (4.1), subjects were explicitly instructed to report their perceptual confidence (lines 245-247). However, half of them reported their decision confidence (line 287). That’s why we talked about overriding. Notably, even the observation of different effects of prior and value on the subject’s confidence was important and interesting enough that the original paper was published in a prestigious venue in the field of psychophysics a few years ago. That paper was only about the different effects of prior and payoff on various subjects, and it didn’t talk about decision confidence nor a normative model for that. Our model explains that observation in a systematic way. In other words, there was no normative model for that subset of subjects, and we propose one in this paper.

For the second experiment, we added another analysis. This result provided more evidence in favor of our model. As demonstrated in Figure 1 of the rebuttal pdf, we looked at how confidence rating changes based on the difference in the value of presented choices. While the confidence increases with the increase in the choice value difference in experimental data and both tested models, the predicted growth rate in our model is significantly closer to the experimental data. Specifically, our model predicts very slow growth of confidence with the value difference increase, just like the experimental data.

Regarding the exponential function, which looks like a heuristic by itself, we agree with the reviewer that our framework is based on the soft-Q learning principle, especially as a normative model. We chose this narrative as we found it more intuitive for the concept of confidence. We will add more explanation upfront about expression 7 and emphasize that it seems arbitrary until we reach the soft-Q learning loss function.

Questions: 1-Sorry for the typos; The sentence should be “To assess our model’s ability to predict subjects’ confidence reporting behavior, we compared the fit of our soft optimality model to the fit of expected value ratio model.”

2- The reviewer is right; line 148 should be replaced with “Consequently, the probability of observing trajectory τ in an optimal agent is p(τ |O = 1, b1), which should be maximum for a trajectory that is generated by the optimal policy π∗".