Language Agents Mirror Human Causal Reasoning Biases. How Can We Help Them Think Like Scientists?

We thank the reviewer for the insightful review. We are happy the reviewer finds this interesting and timely, with well-written presentations, strong analyses, and offers valuable insights to the community.

We address the reviewer’s concerns below.

Do not see how the InfoGain Oracle establishes that the conjunctive case is not harder for LLMs… Observed performance gap could simply reflect task difficulty rather than a disjunctive bias.. not convinced that LLMs inherently favor disjunctive reasoning

We thank the reviewer for the astute question. While InfoGain oracle reasons optimally, it spends the same amount of compute on both (conj & dish) settings. This shows that when perfect reasoning is possible, one setting is not computationally harder than the other. We refer to the (systematic) absence of perfect reasoning as a “cognitive bias”.

We can also consider the possibility that due to bounded computational resources, one task may be harder. One way to control for potential inherent task difficulty is to normalize performance relative to a random baseline. We plot the “exploration progress” (between 0 and 1) and random-normalized exploration progress here: https://imgur.com/a/8LPHKLR (details in figure, higher = better):

We observe in Fig 10b that for the high-performing models (e.g. GPT-4o, deepseek-reasoner, deepseek-chat), a disjunctive bias is still present. For completeness, we also include a table below for each model’s progress at end-of-episode.

We will add the above results to the Appendix section, and soften L131-124 to “this is likely not due to the conjunctive case being inherently computationally harder, see appendix [XX] for discussion”. Please let us know if you have any other thoughts on this!

Do reasoning models think more scientifically than non-reasoning ones?

We do not wish to make strong claims about “reasoning” in token sequences and “thinking scientifically” [Shanahan 2022]. For instance, recent works have found that intermediate “reasoning” tokens need not be correct to arrive at the correct final answer [Stechly 2025].

It is certainly intriguing that DeepSeek R1 performs differently. It appears that DeepSeek-R1 does well making inferences on fixed data (both Figure 5 and 6 are inference-only tasks, rather than explore-then-inference tasks), but not well on exploration (Fig 4). We ran additional evaluations using the GPT-o-mini series of reasoning models on the 8-objects envs, and found that setting a high “reasoning effort” does not improve overall performance, and leads to worse exploration (https://imgur.com/a/tsL29hf). Overall, the role of intermediate “reasoning” tokens are to our knowledge not well-established, and suggestive of exciting future work.

For completeness, we also provide a few qualitative examples of intermediate "reasoning" token traces below for Deepseek-R1 and GPT-4o. Both will be added to the appendix of the final paper.

Can this be addressed by simple instruction tuning

While this might be addressed by instruction tuning on this particular single task, instruction tuning cannot feasibly cover all possible causal reasoning scenarios. Whether instruction tuning on e.g. a subset of blicket tasks can enable OOD generalization in other causal settings is an exciting future work.

We did not explore instruction tuning as our goal is to probe the inherent biases in LM responses. Analogous to how cognitive scientists study biases in humans, they do not train humans to do better on their tasks, as it would obscure rather than reveal the underlying naturalistic behaviour patterns they aim to understand. For now, we believe the Blicket test serves as a representative probe of zero shot causal reasoning abilities with tight experimental control and relevant human data.

[Shanahan 2022] Shanahan, M. "Talking About Large Language Models”: 10.48550." arXiv preprint arXiv.2212.03551 (2022).

[Stechly 2025] Stechly, Kaya, et al. "Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens." arXiv preprint arXiv:2505.13775 (2025).

We thank the reviewer for the interesting points. We are glad to hear the topic is important, our approach is interesting, and that the results point to promising future work.

We address the reviewer’s concerns below, and also discuss the more philosophical points raised.

the paper does not include (as far as I can tell) a human baseline for the results

While we currently do not have human data in our blicket game (sec 3), we do compare LM directly with published human data in section 4. The findings in section 4 corroborates our finding in section 3 (of the models' "disjunctive bias").

We can try to run a small pilot study on human adults with the time we have in the remainder of the week, and add the results if it is feasible (it is very time consuming to properly conduct a human experiment). All in all, while we agree that comparison with humans is an exciting direction, we think the LM results in this work stand on their own merits, and paves the way for subsequent works which comprehensively studies human vs. LMs on this task.

the utility of [the blicket] task in particular

The blicket detector task captures fundamental causal structures—disjunctive (OR) and conjunctive (AND) relations—that are ubiquitous in real-world reasoning and decision-making. These structures exist in many tasks across domains: e.g. in chemistry games like Alchemy, combining elements follows conjunctive rules; in tool use, some tools work independently (disjunctive) while others require multiple components (conjunctive). The list goes on.

The construct validity lies in the task's ability to probe models' understanding of these compositional causal primitives that structure so many real-world problems, in a tightly controlled experimental setting that is already well-established from human studies. While we do not wish to make claims about how the model might “think” [Shanahan 2022], poor performance on our task does mean LMs acts in a way that is sub-optimal -- which has implications for how we should employ LMs in settings where autonomous information gathering and making inferences on them is important [e.g. Yamada et al 2025].

methodological details a little hard to understand

We apologize for the lack of details. We kindly point the reviewer to the Global Response, in which we have extensively detailed the infoGain baseline agent (A.1), and our hypothesis sampling agent (A.2). We will add the additional information into the Appendix of the final paper for better clarity.

Does response length == “amount of reasoning”?

We did not intend to equate response length with “amount of reasoning”. We used quotes to refer to how this analogy is used commonly in the literature [Guo 2025, Zeng 2025]. While longer response does literally mean more FLOPs spent, it is unclear whether it maps onto the illusive concept of “reasoning” [Stechly 2025]. We are happy to soften this sentence and provide this context in-text.

As a somewhat related point, the reviewer may find interesting that we ran additional evaluations using the GPT-o-mini series of reasoning models on the 8-objects envs, and found that setting a high “reasoning effort” does not improve overall performance, and leads to worse exploration (https://imgur.com/a/tsL29hf). Overall, the role of intermediate “reasoning” tokens are to our knowledge not well-established, and suggestive of exciting future work.

[Shanahan 2022] Shanahan, M. "Talking About Large Language Models”: 10.48550." arXiv preprint arXiv.2212.03551 (2022).

[Yamada et al 2025] Yamada, Yutaro, et al. "The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search." arXiv preprint arXiv:2504.08066 (2025).

[Guo 2025] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[Zeng 2025] Zeng, Weihao, et al. "Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild." arXiv preprint arXiv:2503.18892 (2025).

[Stechly 2025] Stechly, Kaya, et al. "Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens." arXiv preprint arXiv:2505.13775 (2025).

We are deeply grateful for the very detailed response. We are encouraged that the work is well written, explores interesting + relevant questions with solid experimental design, and that the results are in sufficient depth with meaningful findings.

We hope to address additional concerns below.

Test-time sampling method used to eliminate the causal reasoning bias is highly tailored to the Blicket test in particular, not clear even on its theoretical contribution,

Our point with the hypothesis sampling is simply to demonstrate that the principle is a useful one: rather than directly prompting the LM to act, we can make it act more rationally (here measured by its ability to eliminate hypotheses effectively and make causal inferences) by making it externalize its belief explicitly for us to control. While not explored, this can potentially be useful generally in settings where we wish to sample functions [Romera-Paredes et al 2024]; future work can also explore controlling its belief e.g. toward certain types of ideas about the world. All in all, we believe using test-time sampling to overcome the LM's prior is a general principle, and our result is just one demonstration of this (which in our case takes the form of a hypothesis distribution over binary python functions).

Clarity comments

We apologize for the clarity issues. We have added extensive details in the Global Response about the Oracle InfoGain agent (A.1) and our hypothesis sampling agent (A.2). We will also add this to the Appendix of the final paper.

Choice of LMs

We select a mix of models to span several key axes—reasoning capability, capacity, openness, and architectural family—to explore how each factor may relate to hypothesis elimination and exploration behaviour. The GPT family are some of the highest-performing, state-of-the-art LMs available. The 4o and 4o-mini variants also allow us to see the effect of model size with similar architectures. The DeepSeek family are state-of-the-art LMs in their own right, allowing us to compare "reasoning model" vs. a non reasoning model, along with full access to the reasoning content. We also study a few small-to-medium sized models, including QwQ (medium sized reasoning model from qwen), and the gemma3 series of models (<30B). We will include a more detailed description of each model in the Appendix of the final paper.

Motivation behind the different prompt variants in Appendix B.2

We constructed the human default and human conjunctive variants to be similar in information as what is presented to human participants in psych Blicket test experiments. In particular, human conjunctive makes use of the term "blicketness", which [Lucas et al 2014] used to encourage the conjunctive interpretation (as "blicketness" can be interpreted as an additive quantity). Finally, we designed the math definition variant to provide a rigorous definition of what a blicket is, minimizing any semantic relation to disjunctive or conjunctive interpretations.

Please refer to Figure 10 to see the variance between prompts: the multiple points with the same shape and colour differ in their system prompts.

Discussion of how the human bias may have been injected into the models

It is certainly an interesting area of study. However, we do not wish to make strong claims here as the “human-like” cognitive bias discovered in this paper is correlational. We can briefly speculate on potential hypothesis for this similarity: one of which is the human data, for instance, it could be that the vast majority of text on the internet are spoken in a way where cause have singular or independent probability to bring about effect (e.g. fever resulting from a virus), rather than multiple causes conjunctively bringing about effect (e.g. heart attack may only result if a person has high blood cholesterol, AND a particular genetic susceptibility). Alternative hypotheses could be that the (English) language may have an implicit disjunctive structure. To properly study these hypotheses requires intervening on e.g. the training data which makes for interesting (albeit difficult) subsequent works.

We thank the reviewer for the response. We are glad to hear the work is relevant, the human comparison is interesting, and the mitigation approach is a promising direction.

We address the reviewer’s concerns below.

The paper lacks information and is very difficult to read. In addition, it includes typos and grammatical mistakes in multiple sections(Lines 161 and Lines 171 are a few examples).

We apologize for the grammatical mistakes. We have fixed the all raised typos (including from other reviewers). Please see our global response for additional details on the InfoGain agent baseline (A.1) and our hypotheses sampling agent (A.2). Please see below for examples about human biases, and more details about evaluations. We will add all new information to the final version of this paper. We hope the reviewer finds this helpful and let us know if there is anything else unclear so we can properly address it.

While the paper mentions disjunctive and conjunctive biases in humans, this needs to be clearly articulated with examples so that one can follow better.

We provide this excerpt from [Lucas et al 2014], which was one of the first cog sci papers that used both conjunctive and disjunctive rules in blicket detectors:

“We contrast two abstract causal principles (overhypotheses) about the forms that relationships take in a causal system. One is that relationships have a disjunctive form, in which each cause has an independent probability of bringing about an effect. This form is pervasive in the literature on adult causal inference. For example, a burglar alarm may be triggered by an intruder or the wind, and a fever may result from a virus or a bacterium. The other overhypothesis is that causal relationships have a conjunctive form in which individual causes are unable to produce an effect, but multiple causes in conjunction can do so. For example, a microwave turns on when both the plug is connected and a button is pressed, but not if either of these causes occurs by itself; likewise, a heart attack may only result if a person has both high blood cholesterol and a particular genetic susceptibility. Knowing when a machine or a disease has a conjunctive form or a disjunctive form helps us make the right inferences when we want to use the machine or cure patients.”

We are happy to include this in Appendix B if the reviewer finds it helpful (or provide alternative examples).

The evaluations performed by authors are sparse in detailing the evaluation datasets, their diversity or any qualitative analyses from their evaluations.

We apologize for the clarity concerns. For reference, we evaluate all LMs in the “blicket Test” text environment detailed in section 2.1 & Figure 1. We provide a full trace from an “episode” in appendix B.4. For the experiments in section 3.1-3.3 and section 5, we run the agent in the text game with 4 or 8 total objects, where 2 of them are blickets (randomly assigned each episode). All objects start off on the floor or on the machine with 0.9 or 0.1 independent probability. The agent is allowed up to 32 actions in the environment (exploration phase) before the game transitions to the Q&A phase where we quiz the agent about the identity of the blickets. The two phases are illustrated in Figure 1.

We have included more details about the Oracle InfoGain agent and our Hypotheses Sampling agent in the Global Response (they will be added to appendix to complement section 2.3).

The human inference experiments corresponding to Fig 6 can be found in appendix B.5.

Qualitatively, we added a few examples of LM's intermediate token (“reasoning trace”) below.

Please let us know if there is anything else unclear! We plan on adding these details as appropriate in the final version of this paper.

[Lucas et al 2014] Lucas, Christopher G., et al. "When children are better (or at least more open-minded) learners than adults: Developmental differences in learning the forms of causal relationships." Cognition 131.2 (2014): 284-299.

A2: hypotheses sampling agent

The sampling agent approximates the InfoGain agent to reduce the entropy of the hypothesis distribution. Instead of iterating over candidate next observations and computing $G(x)$, we prompt the LM to take actions that minimizes $H(F)$. The procedure is as follows:

1. Hypotheses Sampling to form set of active hypotheses

If there are zero active hypotheses, we prompt the agent to generate hypotheses as python functions (for brevity, the full generation prompt will be added to the appendix) :

You have seen the following observations so far:
[HISTORICAL OBSERVATIONS]

Come up with some hypotheses about how the world works. Write each one as a python function... Produce #NUM_HYPOTHESES# hypothesis, do not generate the same hypothesis. Be diverse in both the blicket masks and the potential functions.

Do not generate hypothesis you have already eliminated: 
[ELIMINATED HYPOTHESES]

Do not generate hypothesis you have already generated: 
[ACTIVE HYPOTHESES]

Directly output the answer, ensure only one answer is included. Return a set of hypotheses as python functions.

We save all valid (python executable) hypotheses as string python code in the list of currently active hypotheses. We repeat this process until a desired number of active hypotheses are sampled.

2. Eliminate hypotheses based on observation history

For each python function $f$ (i.e. hypothesis) in the currently active list, we check if it perfectly predicts all past observations: i.e. $f(x_i) = y_i$ for all $\{(x_i, y_i) \}_{i=0, ..., t}$. If it does not, we move it to the "eliminated hypotheses" list.

3. Prompt agent to take actions that eliminate more hypotheses

We prompt the agent to take actions conditioned on the observations and the list of active hypotheses:

You are currently entertaining the following list of hypothesis: 
[ACTIVE HYPOTHESES]

You have seen the following observations so far:
[OBSERVATIONS SO FAR] 

Given the observations so far, and the list of hypotheses (hypothesis space), take an action which will disprove the existing hypothesis.

Directly output the answer, ensure only one answer is included. Output the action in the format \'> action\'. Ensure only one action is included.

The agent then takes the next actions and we repeat the above procedure for exploration.

Q&A Phase

During the Q&A phase, we prompt the agent to answer questions conditioning on the remaining set of hypotheses:

You have seen the following observations so far:
[HISTORICAL OBSERVATIONS]

You have disproven the following hypothesis: 
[ELIMINATED HYPOTHESES]

You have not yet disproven the following hypothesis: 
[ACTIVE HYPOTHESES]

Based on the information above, answer the following question: [QUESTION]

Output the answer in the format \'> True/False\'. Ensure only one answer is included.

In this way, the sampling agent explicitly represents its beliefs $p(F)$ as a set of active hypotheses, which the LM conditions on to take action.