We sincerely appreciate the reviewer's constructive comments and insightful questions. We have addressed the issues outlined below and look forward to further discussions. Please review our responses and let us know if there are any remaining concerns.

Q1. Motivation 1 in Section 2.2.

The purpose of this subsection was to motivate the extraction of the "bias vector." However, we understand the potential for confusion it may cause. We have made appropriate modifications to the manuscript to address this issue. Thank you for bringing this to our attention.

Q2. Justification for AOI with alternative auxiliary options

When collecting answers from humans, including an ''I don't know'' response can improve data quality [1]. Because the models were more likley to show selection bias when they were incorrect, we hypothesized that offering an ''I don't know'' option would improve the quality of the responses provided by the model.

To support our claim, we have already included an ablation experiment in Table 4 of the manuscript to demonstrate the effect of alternative auxiliary options. Specifically, we compared the "I don't know" option with "I know the answer." Here, we additionally tested the "None of the above" option, and the results are reported below. Overall, our "I don't know" AOI consistently outperforms the alternatives across all four metrics. Notably, other option contents may degrade performance for Mistral and Bloomz. We have updated Section 6.2 of the manuscript to include the results.

[1] Converse, Jean M., and Stanley Presser. 1986. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, CA: Sage.

Q3. Impact of BNP on different task performance

We evaluated Llama-3's performance on two general NLP tasks—Sentiment Analysis and Text Summarization—by pruning 8, 16, and 32 nodes. For Sentiment Analysis, we used the "Multi-class Sentiment Analysis Dataset" [1], and for Text Summarization, we used the "CNN/DailyMail Dataset" [2]. The results are presented in the tables below, with the top table corresponding to Sentiment Analysis and the bottom table to Text Summarization.

We observed a slight decline in performance as more nodes were pruned; however, the degradation was not severe enough to significantly affect general linguistic performance. Given that our method is specifically designed for multiple-choice question (MCQ) tasks, we believe that a minor decrease in performance on general NLP tasks is not a significant concern.

[1] https://huggingface.co/datasets/Sp1786/multiclass-sentiment-analysis-dataset

[2] https://huggingface.co/datasets/abisee/cnn_dailymail

Q4. Why is the final layer particularly sensitive to bias?

Figure 3(b) and Figure 8 present detailed layer-wise analyses across multiple models, consistently showing bias concentration in final layers. This aligns with established understanding that later transformer layers handle higher-level semantic tasks while earlier layers process more basic features. The effectiveness of final layer pruning (demonstrated in Tables 1-2) empirically validates our focus on this layer. We hypothesize this is because the final layer directly maps to output probabilities, making it particularly susceptible to systematic biases in token selection. Section 2.2 provides mathematical analysis showing how final layer parameters interact with bias vectors. While expanding this analysis could be interesting future work, our current results already demonstrate the practical utility of targeting this layer.

Dear reviewer fF6T,

As the discussion period is coming to an end, we wanted to kindly follow up on the rebuttal responses we provided and encourage any further feedback or clarification you might have. We would greatly appreciate your thoughts on our responses to ensure we have adequately addressed your concerns.

Thank you again for your time and dedication to reviewing our work. Please let us know if there are any additional points you would like us to address.

Thank you.

Best,

the Authors.

2. Clarification on the definition of Selection Bias and its relation to metrics.

With all due respect, we believe the reviewer may have interpreted the definition of Selection Bias differently. Here, we will clarify its definition and address all the questions raised by the reviewer.

2.1 Where is the "dataset" in the definition of Selection Bias?

In all previous works, Selection Bias has been discussed in the context of specific datasets [1][2]. In our definition, we also defined Selection Bias as the discrepancy between the model's selection under the original choice ordering (of the dataset) and the expected (average) model selection over all possible choice orderings (of the dataset). Consequently, the Selection Bias metrics—RSD, RStd, and our proposed CKLD—are inherently conditioned on the dataset being evaluated. This framing is natural, as it accounts for the diversity in question difficulty and format across different datasets. To provide clearer definitions, we have revised Section 2.1 of the manuscript and highlighted the modifications in blue. Thank you for bringing this to our attention.

[1] Chunjie Zheng et al, "Large Language Models are Not Robust Multiple Choice Selectors", ICLR 2024

[2] Sheng-Lun Wei et al, "Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models", ACL 2024

2.2 Addressing the questions

Q1: "a robust metric should provide similar scores across different datasets, as seen with RStd and RSD"

Datasets vary in difficulty and question format, both of which impact Selection Bias. For example, a model may exhibit lower Selection Bias on easier datasets because its confidence in selecting the correct answer outweighs its preference to specific choice options. Thus, Selection Bias metrics should yield different scores for different datasets. Furthermore, contrary to the reviewer's claim, the RSD values reported in Table 1 of the manuscript differ significantly across the three datasets. Similarly, as shown in an ICLR 2024 Spotlight paper [1], gpt-3.5-turbo's RStd values for ARC-Challenge, MMLU, and CSQA were 3.3, 5.5, and 2.2, respectively. These results further underscore the variability of Selection Bias metrics across datasets.

[1] Chunjie Zheng et al, "Large Language Models are Not Robust Multiple Choice Selectors", ICLR 2024

Q2: "the authors likely made an error in their code because while RSD appears consistent under different ground truth distributions, Rstd varies"

It is unlikely that we made an error in the RStd code as the core part is taken from the code repository of [1] (code repo). We reveal our RStd and RSD code below.

CHOICES = "ABCDEFGH"

def rstd(preds, labels):
    report = classification_report(preds, labels, output_dict=True, zero_division=0)
    recalls = []
    for choice in CHOICES :
        if choice in report.keys():
            recalls.append(report[choice]['recall'] * 100)
    return np.round(np.std(recalls), 4)
    
def rsd(preds, labels):
    report = classification_report(preds, labels, output_dict=True, zero_division=0)
    accs = []
    for choice in CHOICES :
        if choice in report.keys():
            choice_corr = [1 if pred == label and label == choice else 0 for pred, label in zip(preds, labels)]
            choice_support = [1 if label == choice else 0 for label in labels]
            acc_choice = sum(choice_corr) / sum(choice_support) if sum(choice_support) != 0 else 0.0
            accs.append(acc_choice)
    avg_acc = sum(accs)/len(accs)
    acc_var = [(x-avg_acc)**2 for x in accs]
    rsd_acc = np.sqrt(np.mean(acc_var)) / acc if acc != 0 else -1
    return np.round(rsd_acc, 4)

[1] Chunjie Zheng et al, "Large Language Models are Not Robust Multiple Choice Selectors", ICLR 2024

Q3: "if a classifier randomly outputs one of the options with a 0.25 probability, under CKLD, the model would not receive a good selection bias score because it deviates significantly from the dataset’s ground truth distribution."

Yes, that is correct! CKLD measures the divergence between the choice and the answer distribution but does not factor in the model's performance. This is why we emphasized in lines 304–305 that "it is important to refer to multiple metrics for a robust assessment." We believe that performance-based metrics (e.g., RSD) and distribution-based metrics (e.g., CKLD) are complementary and should be used together to ensure comprehensive evaluation.

Q4: "The authors state that RSD gives the lowest score when the selection rate for 'A' is 0.25, as if this is problematic."

The problem of RSD is that it gives the lowest score regardless of the answer choice distribution in the dataset. Ideally, the distribution of the unbiased prediction should roughly be similar to the dataset's answer choice distribution.

We sincerely appreciate the reviewer's constructive comments and insightful questions. We have addressed the issues outlined below and look forward to further discussions. Please review our responses and let us know if there are any remaining concerns.

1. Clarification on the Embedding Difference Analysis in Figure 3(b)

1.1 Four choices are enough to span the 50 tokens in Figure 3(b).

We appreciate your thorough review of our preliminary analysis. To address your concern regarding soundness, we reference one sample from the ARC-Challenge dataset, which was involved in generating Figure 3(b):

Which statement best compares single-celled and multi-celled organisms?
(A) Tissues in a single-celled organism are like the cells in a multi-celled organism.
(B) The nucleus in a single-celled organism is like the skin of a multi-celled organism.
(C) Organelles in a single-celled organism are like the organs in a multi-celled organism.
(D) The cytoplasm in a single-celled organism is like the nervous system in a multi-celled organism.

In this sample, the Llama-3 tokenizer tokenizes the four answer choices into 92 tokens. Such samples with lengthy answer choices contribute to the difference in the early token locations of Figure 3(b). We also conducted a sanity check on the code and confirmed that no error exists in our analysis in Figure 3(b).

1.2 Clarification on the meaning of Embedding Difference

Thank you for raising the insightful point that "final layer embeddings must differ due to different input choice orderings." While your point is valid, we would like to clarify that the key takeaway from Figure 3(b) is "the embedding differences are more prominent in the final decoder layers than earlier layers".

In our analysis, we computed the difference between the average embeddings of "correct" and "incorrect" questions (refer to $\mathbf{b}_\mathbf{x}$ in Figure 4(a) and Equation (1)). This eliminates the semantic information of the sample itself, while the factor contributing to the "incorrectness" remains in the difference. Additionally, by averaging across multiple samples, we further smooth out the effect of the differences in input text. Hence, we can infer that the embedding difference reflects the choice-ordering factor that causes incorrect responses, i.e., the Selection Bias.

1.3 Further Supporting Analysis

To further substantiate our claim that the embedding difference reflects Selection Bias, we present an additional analysis that compares the average intra-difference within the "correct" question set and the average inter-difference between the "correct" and "incorrect" question sets. Specifically, for each dataset, we compute:

where $\mathcal{Z}\_+, \mathcal{Z}\_-$ are the "correct" and "incorrect" embedding sets, respectively.

If the embedding difference does NOT reflect Selection Bias, the average intra-difference within the "correct" embeddings ($\mathbf{d}\_\text{intra}$) should be comparable to the average inter-difference between "correct" and "incorrect" embeddings ($\mathbf{d}\_\text{inter}$). However, as shown in the table below, $\mathbf{d}\_\text{inter}$ consistently exhibits higher values than $\mathbf{d}\_\text{intra}$. This observation suggests that the embedding difference captures information correlated with the "incorrectness" of certain choice orderings, thereby reflecting Selection Bias.

1. May you report how many samples are able to cover 50 tokens with 4 options and how many of them not?
2. I am not saying there is no difference on the final decoder layer or similar to this. Of course, the bias problem may be reflected in the final decoder layer but this is a trivial claim. Every difference in the earlier layers may exist in the final layers because of the residual streams. It doesn't show the final decoder layer makes a real contribution. In summary: I criticize your claim which is "Selection bias stems from the final decoder layers". This is simply not supported and is very likely to be wrong.
3. Could you please provide the standard deviations for the mean experiment as well? This mean difference is not meaningful without stds. Besides, even though the stds are low, how do you attribute these results to selection bias? If you repeat this experiment for different layers, you may observe a similar pattern. You cannot make big claims by showing a correlation with 3 mean values. Something more convincing and possibly a causal relationship should be shown.

2.1 Yes, the model selection bias will be evaluated in a dataset but the selection bias is something the model has or not. It's a model's feature. The dataset here is only for estimating the model's selection bias performance. In some datasets model can perform higher bias, in other datasets it can be lower. If you were able to test a model's selection bias on all possible questions in the universe, then you would get the actual selection bias. The definition shouldn't be a function of dataset D. It should be an expectation over all possible datasets.

2.2 "a robust metric should provide similar scores across different datasets, as seen with RStd and RSD". This claim is TRUE for your proposed synthetic models because that model's bias performance doesn't change from one dataset to another dataset. It will always make some portion correct and the other portion does randomly. However, for real models, the selection bias can be different from one dataset to another because the model for instance can have more selection bias in medical questions than mathematical questions. That's why these two metrics can output different values in different datasets. However, this difference should reflect the model's actual selection bias and should be independent of the choice distribution of the dataset.

2.3 "the authors likely made an error in their code because while RSD appears consistent under different ground truth distributions, Rstd varies". You don't need to run ANY experiments to get RSTD results with the proposed synthetic classifier. You can calculate it analytically. The recall of option (i) is 0.5 + 0.5 x (predictor's probability of sampling option i). The first element 0.5 comes from the fact that the model predicts 50% of questions correctly no matter what and the second element comes from the probability that the model would predict correctly (by chance) in the questions whose answer is option i. However, your plots show that, for a given classifier, RSTD outputs differently in different dataset distributions. Therefore, there has to be a problem in the code because as you see above the recall of option (i) is not a function of the choice distribution of a dataset. Lastly, you only give the code for calculating the std of recalls but it is likely that how to calculate recall is wrong (still don't need to run a simulation for that).

2.4) If a classifier outputs uniformly random then there is no selection bias. This also follows your definition. For any given option order, the model's probability to choose any content is the SAME therefore there is no bias. Mathematically, $E[\text{output o}] = E[\text{output o} | \text{any choice order of outputs}] = 1/(\text{total possible outputs})$

2.5) "Ideally, the distribution of the unbiased prediction should roughly be similar to the dataset's answer choice distribution." This is ideal for a good performant classifier, not a classifier that doesn't have selection bias. You make the same mistake. Please, do not confuse the selection bias and the model performance. There can be a dummy model that would give terrible performance without having a bias to any option.

I agree that "performance-based metrics (e.g., RSD) and distribution-based metrics (e.g., CKLD) are complementary and should be used together to ensure comprehensive evaluation" but your proposed metric cannot replace RSD or RSTD.

Q2.1

Yes, we understand your concern. However, as you may recognize, it is impractical to evaluate a model's Selection Bias across the entire data distribution. Hence, all prior studies on Selection Bias have used dataset performance as a proxy for estimating the level of Selection Bias exhibited by a model.

Q2.2~2.3

To our understanding, the recall for option (i) is NOT "0.5 + 0.5 x (predictor's probability of sampling option i)". Rather, it is $0.5 + 0.5 \times P(\hat{y}= i | y = i)$. The last probability function, $P(\hat{y}= i | y = i)$, is conditioned on the ground truth label, indicating that recall inherently depends on the dataset choice distribution.

For the recall function in our implementation, we use the scikit-learn package, which is identical to the codebase used in the ICLR 2024 Spotlight paper [1]. Also, we have re-verified the evaluation code to ensure it does not contain any errors.

Nonetheless, we greatly appreciate your insightful comment regarding the importance of evaluation metrics being agnostic to dataset distribution to isolate the pure effect of Selection Bias. However, our findings suggest that none of the currently available metrics fully achieve this goal (RStd relies on recall, RSD relies on accuracy etc.). We hope this encourages further research to develop metrics that can address this limitation.

[1] Chunjie Zheng et al, "Large Language Models are Not Robust Multiple Choice Selectors", ICLR 2024

Q2.4~2.5

As stated in lines 261–262 of the manuscript, the experiments in Figure 5 are designed to demonstrate the impact of different data characteristics (i.e., choice distributions) when evaluated using each metric. Hence, we would like to clarify that the synthetic scenario—where half of the predictions are correct and the other half are randomly decided—does not assume a specific model. Rather, the synthetic predictions $\hat{Y}$ are considered to be generated by an arbitrary model with an unknown level of selection bias.

And yes, we did not claim that CKLD should replace RSD or RStd. As stated in lines 304–305 and outlined in Appendix A.4, our intention is to add an additional dimension to the evaluation of Selection Bias metrics.

We sincerely appreciate the reviewer's constructive comments and insightful questions. We have addressed the issues outlined below and look forward to further discussions. Please review our responses and let us know if there are any remaining concerns.

Q1. Section 2 lacks details and needs clarification.

• "Is Figure 2 evaluated based on zero-shot or in-context learning?""

Figure 2 is derived with the zero-shot setting. We have updated the Figure 2 caption in the manuscript to make this clear. Thank you for pointing it out.

• "Are there any scaling trends?""

We extracted the "RSD / CKLD" values from Table 3 and conducted additional experiments with a larger variant of Llama-3, as summarized in the table below. Overall, larger models tend to exhibit lower Selection Bias across the three datasets. However, this trend is largely dependent on the specific dataset being evaluated.

• "Are the open-weight LMs and black-box LMs evaluated using the same criterion? If not, does the evaluation criterion matter?""

In Figure 2, the open-weight LLMs (Llama3, Bloomz, Mistral) are evaluated based on choice token probability and black-box LLMs (Claude3-Sonnet) are evaluated based on Jaccard similarity of the outputs. While this distinction does not undermine the core findings, we acknowledge that it could influence evaluation results. We added a clarification on the selection bias evaluation criteria in the Figure 2 caption.

Q2. Demonstrate BNP on the black-box settings with open-weight models

We tried applying BNP to the open-weight model parameters and evaluated the models in black-box settings. The results are presented in the tables below. The impact of BNP appears to be mixed in this setup, while showing greater effectiveness on particular datasets, such as CommonsenseQA.

Q3. Rationale for using AOI to mitigate Selection Bias

When collecting answers from humans, including an ''I don't know'' response can improve data quality [1]. Because the models were more likley to show selection bias when they were incorrect, we hypothesized that offering an ''I don't know'' option would improve the quality of the responses provided by the model.

[1] Converse, Jean M., and Stanley Presser. 1986. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, CA: Sage.

Q4. How AOI affects the distribution of the choices

The distributional effect of AOI is already presented in Section 6.3, Figure 7. The dark-blue bar can be compared with the light-blue bar to see its effect. In all three dataset tasks, the choice distribution becomes closer to uniform when AOI is applied.

Q5. it is unclear what $\mathbf{z}$['A'] + $\mathbf{z}$['_A'] means.

'_A' is a token that represents "A" with a space in front of it, whereas 'A' is a one-character token. Since these two represent the same thing, we aggregate their logits $\mathbf{z}$ for accurate evaluation. We have updated the manuscript by including further explanations in Appendix A.2.

Q6. Why does an LLM need to match the ground truth label ratio?

Consider a scenario in which an LLM exhibits a bias toward selecting option 'A'. In cases where the LLM is uncertain about the correct answer and resorts to random selection, it is more likely to choose 'A', resulting in a skewed overall choice distribution that diverges from the ground truth distribution. In contrast, an unbiased LLM would select options uniformly under uncertainty, producing a choice distribution that more closely aligns with the original ground truth distribution. Therefore, the extent to which an LLM's predictions match the ground truth distribution can serve as a proxy for measuring Selection Bias. We included this discussion in Appendix C.1 of the manuscript.

Q7. Statistical significance testing by permuting choices

Thank you for the excellent idea. As suggested, we conducted a significance test by running the experiment eight times with randomly permuted choices. The mean performance values for all three datasets are presented in the tables below, with standard deviations shown in parentheses. All values were statistically significant, with t-test p-values below 0.001. We have also updated the manuscript by adding the results in Appendix D.1.

Q8. Performance on larger model families

We also evaluated our methods on Llama-3-70B-Instruct, with the results presented in the tables below. While the model's baseline performance is already exceptionally high, we observe best performance when applying BNP and/or AOI in all three datasets.

Dear reviewer zisy,

As the discussion period is coming to an end, we wanted to kindly follow up on the rebuttal responses we provided and encourage any further feedback or clarification you might have. We would greatly appreciate your thoughts on our responses to ensure we have adequately addressed your concerns.

Thank you again for your time and dedication to reviewing our work. Please let us know if there are any additional points you would like us to address.

Thank you.

Best,

the Authors.

Q4. Limited number of datasets and the method addresses only one form of bias.

We have conducted experiments on four different datasets: ARC-Challenge, MMLU-Redux, CommonsenseQA, and HellaSwag (Appendix). Considering that the ICLR 2024 Spotlight paper [1] reported results on only three datasets, we believe that using four datasets provides a sufficient basis to demonstrate the effectiveness and robustness of our proposed approaches.

Furthermore, we acknowledge that LLMs are subject to various types of biases. However, the focus of our work is specifically on Selection Bias. Our goal is not to address other forms of bias, such as demographic or cultural biases, but rather to study and mitigate Selection Bias in the context of multiple-choice question-answering tasks. Also, the reviewer has mentioned the work of Mikula et al. (2024) without reference to a specific paper. Could you please provide which paper you are referring to?

We sincerely hope this provides additional context and clarifies the scope of our study.

[1] Chunjie Zheng et al, "Large Language Models are Not Robust Multiple Choice Selectors", ICLR 2024

Q5. Impact of data subset size on performance

The size of the training set we use to extract the bias vectors has minimal impact on the overall performance. Also, since we are demonstrating a zero-shot inference task, the size of the test dataset does not affect the general trends observed in the results.

We sincerely appreciate the reviewer's constructive comments and insightful questions. We have addressed the issues outlined below and look forward to further discussions. Please review our responses and let us know if there are any remaining concerns.

Q1. Pruning model weights can influence model behavior in unforeseen ways.

Thank you for your insightful point. First, we would like to note that extensive research has been conducted on LLM parameter pruning for efficient modeling or general-purpose applications [1][2]. However, our approach involves pruning only a very small fraction of the model parameters. For example, in the case of Llama-3, which has 8 billion parameters, we prune just 32 nodes—approximately 0.05% of the total model size.

Furthermore, we evaluated Llama-3's performance on two general NLP tasks—Sentiment Analysis and Text Summarization—by pruning 8, 16, and 32 nodes. For Sentiment Analysis, we used the "Multi-class Sentiment Analysis Dataset" [3], and for Text Summarization, we used the "CNN/DailyMail Dataset" [4]. The results are presented in the tables below, with the top table corresponding to Sentiment Analysis and the bottom table to Text Summarization. We observed a slight decline in performance as more nodes were pruned; however, the degradation was not severe enough to significantly affect general linguistic performance. Given that our method is specifically designed for multiple-choice question (MCQ) tasks, we believe that a minor decrease in performance on general NLP tasks is not a significant concern.

[1] https://huggingface.co/datasets/Sp1786/multiclass-sentiment-analysis-dataset

[2] https://huggingface.co/datasets/abisee/cnn_dailymail

[3] Ma, Xinyin et al. "Llm-pruner: On the structural pruning of large language models." NeurIPS 2023.

[4] Dong, Harry et al. "Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation." COLM 2024.

Q2. Marginal performance improvements

To test the significance of the improvements, we conducted a significance test by running the experiment eight times with randomly permuted choices. The mean performance values for all three datasets are presented in the tables below, with standard deviations shown in parentheses. All values were statistically significant, with t-test p-values below 0.001. We have also updated the manuscript by adding the results in Appendix D.1.

Q3. AOI provides the best overall results; this is expected, given similar behavior observed in models with the SQuAD v1 and SQuAD v2 datasets.

Thanks for your comment. It is not clear to us which specific work is being referred to here. If [1] is the paper in question, we would like to note that it emphasizes the importance of models recognizing unanswerable questions but does not specifically discuss the effect of including an "I don't know" option for multiple-choice question-answering tasks.

[1] Rajpurkar, Pranav et al. "Know What You Don’t Know: Unanswerable Questions for SQuAD." ACL 2018.

Dear Authors,

Thank you for your detailed response to my review. However, I remain sceptical about the proposed method. As mentioned in my review, pruning model weights can lead to unpredictable behavior, which you have not empirically evaluated (see, for example, The Super Weight in Large Language Models, Yu et al., 2024). Additionally, in tasks like QA, models often rely on multiple spurious correlations for answers (see for example, Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models, Mikula et al., 2024. Addressing one does not guarantee that the model will not exploit others. Altering an LLM to fix a single "niche" issue is, therefore, not convincing. Moreover, adding an additional option to the model is a straightforward modification. The authors have also missed prior work with similar findings (e.g., LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS, Kirichenko et al., 2023 and SURGICAL FINE-TUNING IMPROVES ADAPTATION TO DISTRIBUTION SHIFTS, Lee et al., 2023).

Based on these points, I will maintain my scores as they are.