Large Language Models Assume People are More Rational than We Really are

Thank you so much for your review! We are humbled that you find our paper's topic important and impactful. We have carefully addressed each of your weakness / questions below, please take a look.

Weakness 1 + Question 2:

“Currently it is unclear how robust the findings are against different temperature settings (which would affect the outcomes).”

We add an additional experiment varying the temperatures for forward modeling experiment in both zero-shot and CoT prompting cases. Please see item 2 in the general reviewer response!

Weakness 2 + Question 3:

“as LLMs can be sensitive to the actual prompt used, it would be good to understand how robust the findings are when different prompts are used, beyond the zero-shot vs. CoT.”

We added an additional experiment utilizing demographic prompts across {age, gender} as well as a sanity check prompt which asks the LLM to predict the response for monkeys instead of humans.

In these experiments, we found that results are consistent across the various demographic prompts, indicating that our findings are robust. LLM predictions are also less rational for monkeys, indicating that the model understands the prediction task and that it makes predictions based on the subject. See item 3 in the general reviewer response for details!

Question 1:

“What does the CoT prompt look like?”

Thank you for pointing this out! We have added these into the appendix at Tables 6 and 13! For the record, chain-of-thought prompts replaced

• “Please respond with either "Choice 1" or "Choice 2". Do not include anything else in your answer.”

and

• “Do not provide any explanation, only answer with A or B:”

with

• “Let's think step by step.”

and

• “Let’s think step by step before answering with A or B:”

Please let us know if you have any other questions!

Question 1:

“How do you measure the correlation for task 2 and 3 in the forward modeling task?”

For the analyses of the forward modeling task, we use Spearman correlation and Pearson correlation. We measure the correlations between the proportion of humans that chose one gamble over the other and the proportion of times the LLM predicts the human would choose one gamble over the other. This is computed across the 9,831 problem subset of the choices13k dataset, for which the data had around 15 human participants each. We compared correlations across human responses and LLM {zero-shot / CoT} responses, as well as between the maximum expected value responses and LLM {zero-shot / CoT responses}.

For task 2, the model has only one output (the proportion of participants that would choose one gamble over the other), and this is used directly as the proportion prediction of the LLM and is compared with the proportion of human participants that actually chose that gamble.

For task 3, the correlation computation is exactly the same as for task 1, where the model’s decisions are aggregated into a proportion, and this is then used to compute correlation against the real human data.

We have updated the main text in blue to make this more clear.

Question 2:

“Could there be circular inconsistencies in the pairwise ranking for the inverse modeling task? For instance, if the model ranks decision A > B, B > C, but then C > A, how is this resolved?”

Yes - there could be. Since the final output of the human experiment was an ordering of the decisions, we simply performed a count of the “pairwise wins” of each decision (ties were 0.5). Since we prompted for all pairs, counting directly corresponds to an aggregated ordering of the decisions.

Question 3:

“Why not use “average human participant” for the prompts in Table 5 in the Appendix?”

Because we have more human data than just averages – we have individual responses. To effectively compare the LLM against the human data, we would also want the LLM to try to output the full distribution of human responses. These are used in the more advanced forward modeling analyses with the 18 decision models, highlighted in Appendix D. Simply having the averages would not enable us to perform these more detailed analyses.

Your “average human participant” suggestion actually closely aligns with our task 2 in forward modeling, where the model is predicting the proportion of human participants that would select one gamble over the other.

Question 4:

“Why did you decide to use “you” and refer this to human simulation? “What would an (average) human participant do?” sounds better?”

Using the second-person framing of “you” is common in the literature that investigates the use of LLMs to simulate human survey responses or psychology experiments (e.g., [1-3]). Furthermore, we wanted to sufficiently distinguish this from the prediction task (task 1), where the prompt is “A person is presented [...] Which machine does the person choose?”. Asking what a human participant would do isn’t much different.

Question 5:

In Table 11 in the Appendix, where is Bag 2 in Choice 2? Is it referring to the Bag 2 from above?

No, there are a small subset of choices from the original psychology experiment [4] where there were no alternatives for the person to choose (see Table 12). We chose to replicate this in the LLM experiments because it enables us to perform more direct analyses between LLM and human responses. We specified this to the model in the prompt in Table 11: “For example, when there is only one bag, the person has no choice but to choose it.” We highlight this in blue for your convenience.

[1] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis 31, 3 (2023), 337–351.

[2] Ryan Liu, Howard Yen, Raja Marjieh, Thomas L. Griffiths, and Ranjay Krishna. 2023. Improving interpersonal communication by simulating audiences with language models. arXiv:2311.00687

[3] Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic HCI research data: A case study. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2023).

[4] Alan Jern, Christopher G Lucas, and Charles Kemp. 2017. People learn other people’s preferences through inverse decision-making. Cognition 168 (2017), 46–64.

Thank you so much for your review! We appreciate that you found our paper's findings interesting, especially in the scope of current alignment research! We have carefully responded to each of your weakness / questions below.

Weakness 1:

“In the forward modeling section, the binary choice questions have an objectively optimal answer. Why should we conclude that the models are “assuming humans are more rational” simply because their answers are closer to this optimal? A simpler interpretation could be that the models are simply solving problems optimally, regardless of the prompt’s content.”

Thank you for the question! We agree that by just looking at the original results, it could also be possible that the model is solving the problem optimally regardless of the prompt. In this case, the model would be simply bad at understanding the problem context and instruction following.

To test this alternative explanation, we conduct an experiment where we prompt the model to estimate the decision of monkeys instead of humans. This is shown in item 3 of the general response – “Additional experiment: Participant Groups and Prompting Variations”. With CoT, the correlation with max expected value of monkeys is only 0.65, which is much lower than the 0.95+ that we get for all other prompts. This suggests that the model isn’t simply computing the optimal solution.

Weakness 2:

“models are more aligned with human choices than with maximum expected values in the zero-shot setting”

This is also a very good point! We realized we didn’t provide a very compelling argument for why zero-shot isn’t as strongly considered in our paper. Please see item 1 in the general response – “Deep dive into zero-shot results for Forward Modeling”.

Essentially, we found that sometimes zero-shot would produce estimates according to rational models, but when it did not it made the inexplicable decision to answer without considering the probabilities at all. Because of this, as well as zero-shot performing worse than CoT in general in our experiments, zero-shot responses would likely see limited use in practice. From our analysis, it was clear that CoT was doing something much more meaningful, and thus it was more reasonable to make a conclusion based on the CoT performance.

Thank you so much! We will be sure to add at least results for Llama 3.1 for our additional experiments, and we will try to also include other models such as Claude and Llama 3! We will write up all of the results in the final version --- we really appreciate your thorough engagement with our work throughout the review process, as well as your timely feedback and constructive opinions.

Best,

Authors

Thank you for your response, we greatly appreciate your question! All our additional experiments are conducted on the newest version of GPT-4o.

Please also let us know if you think it makes sense to include any other models!

Best, Authors

Thank you so much for your review! We address each of your weaknesses / questions, please see below.

Question 1:

“Why set the temperature to 0.7 for some experiments?”

Thank you for your question! We have rerun the models at a variety of different temperatures, and find results consistent with our original findings! Please check out item 2 in the general reviewer response.

Question 2:

“If I'm reading correctly, the poor predictions in the zero-shot setting aren't explained well by the model incorrectly modeling humans as rational actors? What's your alternative explanation?”

Thank you for this feedback as well! From your feedback, we realized that our full analyses were not presented well here and have changed this! Please take a look at item 1 in the general reviewer response for our changes.

Question 3:

“Why even permit the "tie" answer?” For some decisions that were used in the inverse modeling psychology study, choices were not meaningfully different from each other. For instance, in the two decisions shown in the first and second rows of Table 12, the choices are not meaningfully different because in both cases the person only has one choice. If we did not enable the “tie” answer, we would just be injecting noise into the results.

We actually debated whether to keep these decisions in our LLM study, but decided to keep them because it enabled us to more easily conduct comparisons between the existing human results and our LLM results.

Thank you so much for your review! We are delighted that you found our paper to reveal very interesting insights, and our design and analysis to be rigorous!

Weakness 1:

“tests used in this work appear to be quite simplistic to inform practical development”

The tests we used were derived from the literature on judgment and decision-making in psychology, where they are arguably the standard way to evaluate performance in these settings. Psychological experiments focus on simple stimuli to remove confounding factors. Since our experiments were based on comparisons to human data, we were limited in the tasks we could use.

If we had found a positive result (such as models being able to accurately model humans), there would be more generalizability concerns. However, our findings are primarily negative and the takeaways are cautionary: That models may not have accurate representations of the decisions of their human partners, as demonstrated by the fact that they do not do so in the cases we consider. Under this lens, we believe the simplicity of the context is actually a strength of the paper, because it highlights that issues arise even in the simplest settings.

We have also added to the paper some ways of introducing additional complexity into the tests. Item 3 in our general response contains an additional experiment evaluating LLM performance when the predicted people are of different demographic profiles, where we find that results are very similar across profiles.

Question 1:

“What are some other potential human decision-making models that can also be tested on LLMs, in addition to forward and inverse modeling?”

Rather than being individual models, forward and inverse modeling are ultimately assumptions about the Bayesian nature of decision making. Forward models are normally “noisy” versions of a rational model, where options of greater utility are selected with higher probability (see Section 4, Rational models, Equation 1). Inverse models are obtained when we apply an additional Bayes’ rule (see Equation 2). Thus, forward and inverse models are actually a family of models that share the same set of very basic and intuitive assumptions that underlie the psychology literature.

We do test many different models within the families of forward and inverse modeling. For instance, in the new paragraph in Section 3.1, we highlight our analyses conducted in Appendix D that involve 18 different forward models in order to explain the behavior of LLMs with zero-shot prompts. Similarly, in Section 4, “absolute utility”, “relative utility”, “likelihood”, and “marginal likelihood” are all different inverse models. Hopefully this helps clarify your concerns.

Question 2:

“How are the insights gathered from this study inform the development of LMs applications, or inform better design of the next generation of LMs or other AIs?”

We see our results as identifying a significant concern in deploying LMs in settings where they will be asked to generate predictions about or simulate human behavior, or make decisions based on inferences from human choices. As indicated in the references provided in the paper, there are an increasing number of applications of this kind. So far, researchers have focused on cases where these models seem relatively successful in predicting, simulating, or making inferences from human behavior. Our work, through a detailed comparison against human choices across a large set of decision problems, highlights a setting where we find systematic deviations from how people make decisions.

Thank you for your continued engagement with our paper!

We would like to quickly highlight why we believe each successor application is connected to our results for forward and inverse modeling:

Identifying effective ways to assist a person requires understanding what they want. For instance, if a person is reaching out in the direction of an item, an embodied AI agent could take initiative and grab the item for the person. However, when the person's value function over items is incorrectly inferred by a robot (failure in inverse modeling), the AI agent might choose the wrong item to provide to the person. This could happen in an online setting, where a person looks for a long time at an advertisement but does not want the item, but an AI shopping assistant doesn't understand this and tries to help by adding the item to the person's cart.

Learning individuals’ values and preferences is a more generalized version of this, where the model is learning the human's preferences, but the application might not be to provide assistance -- e.g., identifying malicious behavior.

Simulating the helpfulness and harmlessness of a response is a technique where the language model has a list of potential utterances it can use to respond to a human, and chooses an utterance from this list by simulating the human's response to each potential utterance [1, 2], similar to conducting theory-of-mind. In this case, the model would specifically simulate the helpfulness and harmlessness of each response. In order to do so, the LLM would need to have an accurate forward model of how the human would react to and make decisions based on the utterance it provides, which is needed to estimate the outcomes and derive whether the statement was truly helpful/harmless. If the forward model is incorrect, it may lead the model to produce incorrect estimates of the human's decisions and thus incorrect estimates of the helpfulness/harmlessness of the response.

We hope that this could help provide better intuition for why our experiments and findings matter in terms of these important applications in LLM use. We really appreciate your input, and we will include a separate discussion section detailing these connections more deeply in the final version to help clarify this for other readers.

[1] Liu, R., Yen, H., Marjieh, R., Griffiths, T. L., & Krishna, R. (2023). Improving interpersonal communication by simulating audiences with language models. arXiv preprint arXiv:2311.00687.

[2] Shaikh, O., Chai, V. E., Gelfand, M., Yang, D., & Bernstein, M. S. (2024, May). Rehearsal: Simulating conflict to teach conflict resolution. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-20).