Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs

Thank you for your thoughtful and constructive review. We appreciate your recognition of the paper’s contributions, including the importance of the research question, the soundness of the causal analysis, and the clarity of presentation. Your comments have been very helpful in identifying areas for clarification and improvement.

• On randomness and latent vs. observed bias - We intended latent bias to refer to meaningful, underlying patterns learned during pretraining, while observed bias reflects how these patterns are expressed in outputs. Our assumption was that training randomness influences the latter without substantially altering the former. Our assumption was motivated by the idea that randomness in weight initialization, data order, and optimizer behavior typically introduces noise around stable learned representations, rather than reshaping the core behavioral patterns acquired through pretraining. That said, as you noted, variability across seeds could potentially reflect more directional-seeming influences. However, our analysis in Step 1 (Section 5.1) directly addresses this by quantifying the variance across runs. We show that, despite some fluctuation, the overall bias patterns remain largely consistent. We will revise the paper to clarify this assumption and analysis, acknowledge its limitations, and discuss how randomness may sometimes interact with model behavior more systematically.

• On model differences beyond pretraining - We agree that architectural and size differences between OLMo and T5 complicate attributing bias differences solely to pretraining. To address this, as noted in the general response, we added experiments using Mistral-7B and Llama-2-7B models, which are more comparable. These replicate our main findings, and we will include them in the revision to strengthen our conclusions.

• Lack of theoretical framing - Thank you for highlighting this. These are important and fascinating directions that we are excited to reflect on more explicitly. While it may be beyond the scope of this work to develop a full theory connecting specific data properties to individual biases, we agree that the paper would benefit from some discussion of these possibilities. In the revised version, we plan to reflect on potential contributing factors. For example, we plan to include the following statement in the discussion: “While our work does not model specific causal links between training data properties and individual biases, several factors such as corpus composition, linguistic framing, tokenization choices, filtering decisions, and sampling strategies may plausibly influence the emergence of cognitive biases during pretraining (Feng et al., 2023; Silva et al., 2021; Navigli et al., 2023; Chang et al., 2024). Exploring these connections is a promising direction for future work.” Such additions will help frame promising avenues for future work.

• Lack of discussion on implications and mitigation strategies - Concerning the implications of our work, there are several promising mitigation directions worth discussing. Since our results indicate that pretraining is the primary source of bias, early-stage interventions may be especially effective. For example, instance attribution methods could help identify and filter out harmful training examples, cognitive bias scores could be monitored during pretraining to enable early intervention, and small-scale pretraining runs on alternative datasets could guide better data selection. At the same time, instruction-phase interventions remain highly relevant, as they are more practical to apply and can still meaningfully influence how pretraining biases are expressed. Techniques such as incorporating self-helping prompts during instruction tuning (Echterhoff et al., 2024), like how chain-of-thought prompting aids reasoning, could help reduce surfaced biases. Another avenue involves curating instruction datasets to promote internal consistency and minimize exposure to contradictory or biased patterns. Additionally, our findings raise the possibility that instruction tuning alone may be insufficient, and that stronger post-training alignment methods, such as variants of RLHF that reward unbiased rather than merely preferred outputs, may be necessary. We will revise the concluding section to incorporate these directions.

Thank you again for your valuable feedback. Your comments have directly improved the clarity and scope of our work, and we hope this response addresses some of the concerns.

Thank you for the thoughtful and encouraging review. We're especially grateful for your recognition of our causal framework, the clarity of our writing, and the significance of our findings.

We fully agree that generalizability is critical. As detailed in our general comment, to address this limitation, we added experiments on additional models based on Mistral and Llama2, finetuned on Tulu 2 and ShareGPT. Despite being trained in uncontrolled settings, these models replicate our core finding: pretraining is the primary driver of bias. We hope this stronger empirical support encourages you to consider raising your score.

Thank you again for your constructive feedback. This highlight has helped us improve both the depth of the paper, and we’d be glad to hear if they address your concerns.

Hi reviewer! A quick reminder that the discussion period ends on 6/10. Please make sure to read and respond to the authors' rebuttals -- It'd mean a lot to the authors who have put in a lot of efforts into their work!

Thank you for the detailed and thoughtful review. We're glad you appreciated the structure of the study, the clarity of definitions and visualizations, and the experimental rigor.

• On model and dataset scope - As detailed in our general comment, we added experiments on six additional models based on Mistral and Llama2, finetuned on Tulu 2 and ShareGPT. Despite being trained in uncontrolled settings, these models replicate our core finding: pretraining is the primary driver of bias. We hope this stronger empirical support encourages you to consider raising your score.

• On bias type granularity and scenario variance - We agree this is an important and interesting direction. A complete table of per-bias results is included in Appendix D, allowing such fine-grained analysis. Our preliminary analysis found that most individual biases follow the overall clustering pattern, with no consistent deviations. One minor exception was Status-Quo Bias (a tendency to prefer the current state over change), where Flan-models (0.01, 0.19) showed slightly stronger bias than Tulu-models (-0.02, -0.23), though this difference had high variance. Regarding scenario-level analysis, including full results is infeasible due to the large number of scenarios and the difficulty of drawing clear patterns when considering each bias individually. We will clarify this in the analysis section. We will include this individual biases analysis in the revised paper and expand the discussion to note that some bias types may be more or less sensitive to pretraining vs. finetuning. We will also clarify that while we aggregate across biases, this may mask variation, and cases like Status-Quo Bias highlight where divergence can still emerge. Thank you for raising this point.

• On instruction dataset similarity - You raise an important point. While the instruction datasets (Flan, Tulu2) are often treated as distinct, we did not include a formal content analysis. In our case, the instruction datasets were Flan and Tulu 2. Tulu 2 was explicitly constructed to improve upon and deviate from Flan, aiming to reduce known limitations and increase logical performance. We will clarify this motivation in the paper and include a brief comparison of dataset characteristics to better support the cross-tuning setup. Additionally, as described in the general comment, we expanded our experiments to include models finetuned on ShareGPT, a dataset derived entirely from real user interactions. This further increases the diversity of instruction data and strengthens the validity of our conclusions.

Thank you again for your insightful feedback. These points have helped us improve both the clarity and depth of the paper. We’d be glad to hear if they address your concerns.

Thank you for the thoughtful review and for highlighting the clarity of our methodology, the insightful analysis, and the importance of our framework for understanding bias in LLMs.

• On model and dataset scope - As detailed in our general comment, we added experiments on additional models based on Mistral and Llama2, finetuned on Tulu 2 and ShareGPT. Despite being trained in uncontrolled settings, these models replicate our core finding: pretraining is the primary driver of bias. We hope this stronger empirical support encourages you to consider raising your score.

• On mitigation strategies - While our primary goal was to isolate the source of bias, we agree that discussing how these insights can inform mitigation is important. We will expand the discussion to suggest concrete directions. Given that our findings point to pretraining as the primary source of bias, mitigation efforts at this stage may be especially impactful. For example, instance attribution techniques could help identify and remove training examples that disproportionately contribute to bias. Monitoring cognitive bias scores during pretraining could enable early intervention by stopping, reverting, or adjusting training when problematic trends are detected. Additionally, small-scale pretraining experiments on different datasets or data compositions could help identify sources that lead to reduced bias. Instruction-phase interventions, while operating later in the pipeline, offer greater flexibility and lower resource demands, making them attractive for practical deployment. For example, following the success of self-helping prompts in reducing bias (Echterhoff et al., 2024), one promising approach is to integrate such prompts during instruction tuning, similar to how chain-of-thought prompting improves reasoning. Another direction is to curate instruction data to encourage logically consistent, unbiased behavior and reduce contradictory examples. While exploratory, we believe these ideas can help translate diagnostic insights into actionable interventions.

Thank you again for your constructive suggestions. We believe these additions significantly strengthen the paper, and we’d be happy to hear if they resolve your concerns.