Large Language Models Are Not Robust Multiple Choice Selectors

The writing and presentation need more improvement, e.g., I think the proposed PriDe is quite intuitive but the authors introduce too many unnecessary notations ($d_i, o_i, g_i, x_i, ....$) before getting into the real introduction of the method, which makes the reading difficult.

We would like to clarify the reasons for introducing these formal notations before the proposed method. We attempted to introduce the proposed method first and then intersperse or supplement the introduction of the permutation-based baseline on which our method relies. However, we found that this compromised the integrity of the writing content. We also found that without introducing these formal notations, the writing would become very repetitive and lengthy (we would have to repeatedly use the same terms to avoid ambiguity). Additionally, we believed that formal notations could aid in deriving general solution forms. All these above led us to adopt the current formal notations and writing logic.

If you believe there is a better way to present or structure the content, we would greatly appreciate it and be open to taking your suggestions!

The proposed method basically follows estimate-then-mitigate, which is somewhat similar to the calibrate-before-use (Zhao et al. ICML 2021) paper, though this one targets a different setting and is not directly applicable to MCQs. But it would be interesting to compare the differences and know if calibrate-before-use can also help with MCQ sensitivity.

We are willing to discuss the difference from Contextual Calibration (Zhao et al.). We make a preliminary attempt to adapt Contextual Calibration to MCQ debiasing as follows:

1. For each test sample, we use the default input and obtain the prediction distribution $\mathbf{p}$.
2. We then replace all the options with the same content-free text: the null string '', N/A, or [MASK], as in Zhao et al., and estimate the model's prediction distribution over the option IDs, denoted as $\mathbf{p}_0$ (we use all the content-free texts and take the average of their $\mathbf{p}_0$).
3. We use $\mathbf{p}/\mathbf{p}_0$ after normalization as the "calibrated" prediction distribution, as done in Zhao et al.

So, from the perspective of implementation, Contextual Calibration is similar to PriDe. The key difference lies in how we estimate $\mathbf{p}_0$, which we refer to as "prior" in our work. The results are shown below (gpt-3.5-turbo-0613, 0-shot ARC, for a quick verification):

We find that Contextual Calibration fails to mitigate selection bias (RStd) and may impair model performance (Acc). It implies that the "prior" ($\mathbf{p}_0$) estimated by Contextual Calibration cannot reflect the model's selection bias in MCQs and may also be hard to interpret.

Thanks for your constructive comments! We address your concerns or questions as follows.

The proposed method requires sampling test samples first to estimate the prior, which may introduce another dimension of sensitivity of the selection of the test samples. The accuracy of this estimation might vary based on the quality and representativeness of these samples.

Good question! While we do test sample sampling mainly out of the need of experiments, in practice we usually expect the estimated prior to be less sensitive to estimation sample selection. We here supplement statistics in the 5 runs of PriDe (0-shot, $\alpha=5\%$). We present the mean (as reported in our paper) ∆ RStd and ∆ Acc as well as their best/worst values (averaged over all the 20 LLMs):

We observe that the selection of estimation samples may introduce slight fluctuations in debiasing results, and even in the worst-case scenario, PriDe still leads to a notable debiasing performance (decrease in RStd and increase in Acc). Therefore, we believe PriDe's sensitivity to the selection of estimation samples lies within an acceptable range and does not obscure its merit (effectiveness and efficiency).

I understand the procedure of cyclic permutation and full permutation, but how are they used as the debiasing methods? Do the authors take the best result of the permutations as the prediction?

Cyclic and Full Permutation can be viewed as having a debiasing effect, as they involve swapping options and averaging prediction distributions over different permutations. They can intuitively mitigate the model's bias for option IDs or options' ordering positions. This is similarly done in recent work [1] and [2], where they swap candidate responses to mitigate GPT-4's evaluation bias.

For Full Permutation, there is only one possible permutation set (i.e., all possible permutations). For cyclic permutations, there might be multiple possible permutation sets (as long as we ensure one pairing between each option ID and option content). The selection of cyclic permutation sets is not our focus, as our method PriDe can be directly combined with any reasonable cyclic permutation set. In the main text, we use the simplest and most intuitive set for Cyclic Permutation, e.g., $\{ (1,2,3,4), (2,3,4,1), (3,4,1,2), (4,1,2,3) \}$ for 4-option MCQ tasks. We show in Section 3.1 and Figure 16 in Appendix F that selecting other cyclic permutation sets leads to similar debiasing results.

[1] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[2] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

The authors use the balance of recalls and Rstd as the major metrics throughout the paper. Can the authors formally define this? I didn't immediately get it.

Sure! The recall of an option ID $d_i$ is defined as:

\mathrm{Recall}(d_i) = \frac{ \\# (\text{correct answer is } d_i \ \\&\text{ prediction is } d_i) }{ \\# (\text{correct answer is } d_i)} \times 100 \\%,

while RStd (Std of recalls) is:

$$\mathrm{RStd}=\mathrm{Std}( \\{ \mathrm{Recall}(d_i) \\}\_{i=1}^n) = \sqrt{\frac{\sum_{i=1}^n (\mathrm{Recall}(d_i) - \mu)^2}{n}}, \text{where } \mu = \frac{1}{n} \sum_{i=1}^n \mathrm{Recall}(d_i).$$

Our motivation of using this measurement is illustrated in Section 2.2.

Dear Reviewer VDjF,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Thanks for your positive comments! We address your concerns or questions as follows.

It seems that PriDe achieves comparable performance with simple baselines when the computation cost is not limitated. In application scenarios, we always first estimate the prior without concerning the computation cost, then apply this prior to serve applications. It would be better if PriDe could have a higher upper boudn performance.

(This question is similar to the one raised by Reviewer CGVT, so we use the same answer)

When the budget is sufficient, using more permutations does yield better debiasing effects and performance improvements. As discussed in Section 4.3, this is akin to "mixture of experts" or "model ensemble". Our method, on the other hand, provides a computation-efficient alternative. We believe this could be beneficial for debiasing in scenarios with constrained/limited computational resources, such as platforms like the HuggingFace LLM Leaderboard, where a large number of models need to be evaluated on numerous benchmarks.

The generalization analysis indicates that the bias for a certain model is consistent across different tasks. Could you further demonstrate this with more statics or results? It would also help to enhace the claimed interpretability.

Of course! Here we compute the L1 distance (due to its intuitiveness, as in our response to Reviewer CGVT) between the estimated priors from different domains to illustrate PriDe's cross-domain generalization (0-shot, averaged over all the LLMs, $\alpha=5\%$; priors' L1 distance is averaged over 5 runs).

We think these priors' gaps are usually marginal, which could verify that for a certain model, its prior for option IDs is similar across domains.

Dear Reviewer G9aZ,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Thanks for your positive comments! We address your concerns or questions as follows.

It would be better to cite "Leveraging large language models for multiple choice question answering" or other related papers when mentioning the Option-Order Sensitivity problem since they have found the problem earlier than the work of this paper.

We appreciate your suggestion! We will add the citation in the revision.

It would be better to analyze more technicals, including self-consistency.

In Section 2.6, we experimented with simple prompting strategies, considering their popularity in recent research, to observe whether they have a positive impact on debiasing (finding that they do not). We did not explore too much into prompting engineering, for the following reasons:

1. Our empirical analysis in Sections 2.3-2.5 cannot motivate us to work on prompting engineering, i.e., we intuitively believe that prompting engineering is not the fundamental means of debiasing (and may also be tricky).
2. Complex prompting strategies (such as self-consistency) are designed primarily to enhance model performance rather than to debias. Moreover, they typically rely on powerful but often commercial, closed-source LLMs like ChatGPT, Claude, and PaLM, making them less applicable to open-source LLMs like LLaMA.
3. Complex prompting strategies are often expensive, especially when involving much sampling or heuristic filtering.

We also supplement the results of Self-Consistency on ARC (this benchmark has a small scale, suitable for quick verification). We employ gpt-3.5-turbo-0613, sample 10 Chain-of-Thought paths, and then vote on the predicted results.

As expected, Self-Consistency improved Acc. However, like other prompting strategies, it cannot mitigate selection bias and even somewhat amplifies it (RStd increases), which is inconsistent with our goal of debiasing. We believe that investigating the impact of prompting strategies on LLMs' behavioral bias would be an intriguing research problem.

Dear Reviewer rhkS,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Thanks for your positive comments! We address your concerns or questions as follows.

When the compute budget is unbounded, the proposed method sometimes has a slight accuracy disadvantage compared to full perm.

(This question is similar to the one raised by Reviewer G9aZ, so we use the same answer)

When the budget is sufficient, using more permutations does yield better debiasing effects and performance improvements. As discussed in Section 4.3, this is akin to "mixture of experts" or "model ensemble". Our method, on the other hand, provides a computation-efficient alternative. We believe this could be beneficial for debiasing in scenarios with constrained/limited computational resources, such as platforms like the HuggingFace LLM Leaderboard, where a large number of models need to be evaluated on numerous benchmarks.

In deriving the method, there are a few key assumptions, e.g. Prior for option IDs depends mostly on q. Is it possible to empirically verify this assumption?

We removed the dependency on $x^I$ in $P_\textrm{prior}$ because it could be a minimally strong assumption necessary for our derivation. We are also pleased to empirically verify this assumption, that is, whether swapping options (i.e., different $x^I$ w.r.t. $I$) would change the derived $P_\textrm{prior}$? If the answer is no, then our assumption makes sense.

In our main text, we use the cyclic permutation set $\{ (1,2,3,4), (2,3,4,1), (3,4,1,2), (4,1,2,3) \}$ for 4-option MCQ tasks. Our verification contains the following steps:

1. Modify the default-ordered options as $(1,2,4,3)$ or $(4,3,2,1)$.
2. Use the corresponding cyclic set $\{ (1,2,4,3), (2,4,3,1), (4,3,1,2), (3,1,2,4) \}$ or $\{ (4,3,2,1), (3,2,1,4), (2,1,4,3), (1,4,3,2) \}$ to derive $P_\textrm{prior}'$.
3. Check if $P_\textrm{prior}'$ is close to $P_\textrm{prior}$. We use the L1 distance as measurement (averaged over all the test samples), due to its intuitiveness: $d(\mathbf{p}, \mathbf{q})= \sum_i |p_i - q_i|.$

We can see that the difference between $P_\textrm{prior}'$ (i.e., estimated with a different permutation set) and $P_\textrm{prior}$ is marginal (their L1 distance is quite small), which we believe could validate the soundness of our assumption.

Dear Reviewer CGVT,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors