Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice

Thank you for the positive evaluation of our work.

No weakness within the scope of the work. But it would be nice to see some parameterized ablation level for synthetic datasets.

Response: Thank you for this suggestion. Indeed, the potential distributions of probabilities and values that could be tested are virtually limitless. As an initial step, we explored three probability distributions by varying the Beta distribution parameters and three value distributions by adjusting the exponent of a power-law. Detailed results are provided in Table A4 of the revised manuscript.

In summary, our findings indicate that aligning probability distributions with ecological patterns improves the model’s ability to predict risky choices. Conversely, shaping value distributions as a power-law with an exponent of -2 enhances the model’s performance in predicting intertemporal choices.

Relate to interpretability research to better understand Arithmetic-GPT’s embeddings.

Response: We agree that applying modern interpretability techniques to Arithmetic-GPT could offer valuable insights into what cognitive mechanisms it has encoded. As noted in the “Limitations and Future Research” section, interpretability research will be a key focus in our future work.

Presentation issue in Fig A2

Response: Thank you! It is fixed now with Fig A2 under Appendix E rather than F.

We thank the reviewer for the constructive comments and feedback.

Weakness 0: The risky and intertemporal tasks are too simple to support sophisticated claims about language models as cognitive models.

Response: Risky and intertemporal choices are fundamental forms of decision-making as they reflect an agent’s risk and time preferences, which are core constructs in cognitive psychology and behavioral economics for studying decision-making. These tasks are considered the gold standard for studying human decisions, inspiring hundreds of papers in psychology and economics, and serve as essential building blocks for more complex cognitive tasks. Therefore, it is critical for any cognitive model to perform well in these domains. Moreover, two of the human datasets used in our study are large-scale experiments in the risky and intertemporal choice domains, providing robust statistical power for modeling human behavior.

Weakness 1: Should we view the Arithmetic-GPT as a process model that hypothesizes actual cognitive processes?

Response: We do not view Arithmetic-GPT as a model of the actual cognitive processes behind human decision-making. Instead, we use language models as tools to explore computational principles underlying human decisions. The key advantage of language models here is their ability to encode symbols (e.g., values, probabilities, and discount factors) as vectors (e.g., model embeddings) shaped by the training task. Our approach reveals that ecologically training the model on expectation calculations creates human-like representations. Overall, our method aims to uncover computational principles in human cognition (e.g., how human-like behavior emerges in specific cognitive tasks as a consequence of solving what kind of computational challenges), without assuming specific cognitive processes that implement these computations. To clarify this we added the following text in our Discussion section: “Notably, language models, including Arithmetic-GPT, do not directly model human cognitive processes. Instead, our work suggests that these models are better suited for probing the computational problems that human cognition aims to solve. ”

Weakness 2: Arithmetic-GPT is marginally better than LLaMA. LLaMA has already captured much of the variance in human choices.

Response: We agree that LLaMA-3-70B-Instruct captures a significant portion of the variance in human data. However, as noted in the Background section, it is challenging to pinpoint why LLaMA performs well in explaining human data due to potential data contamination. LLaMA may have been trained on human data or exposed to literature on human biases in risky and intertemporal choices, meaning it may already “know” what human-like behavior looks like. In contrast, the advantage of Arithmetic-GPT lies in our complete control over its training, eliminating data contamination and achieving comparable predictive accuracy to off-the-shelf LLMs. This control allows cognitive scientists to systematically investigate which computational tasks (used as training data) lead to human-like representations in language models. In this case, we have shown that just training on arithmetic is sufficient to produce good performance.

Question 1: Directly training LLaMA or Arithmetic-GPT on task features as an alternative.

Response: Thank you for this question. The choice depends on the specific scientific question. Task features serve as an abstract representation of stimuli; in risky choice experiments, for example, only probabilities and values—deemed essential by experimenters—are included. If the goal is to predict human behavior based on text descriptions of psychological tasks, fine-tuning off-the-shelf LLMs on human data (formatted as text descriptions of the task that include task features) may be effective; Centaur, for instance, is a model that follows this approach [1]. However, if the aim is to estimate the maximal explainable variance, training an MLP on task features, which represent experimentally manipulated variables among stimuli, is often sufficient. In our study, we use Arithmetic-GPT to explore the computational problems that people need to solve. For this purpose, directly using task features to predict human data is less informative, as it overlooks the computational problem we aim to understand.

[1] https://arxiv.org/abs/2410.20268

We thank the reviewer for the positive evaluation of our paper.
No weakness was identified in our work. Clarify the need to specify the discount factor and token for ambiguous probabilities.

Response: The choice of an annual discount rate of 0.85 follows typical equity discount rate in the real world. We imposed this constraint on our Arithmetic-GPT model to prevent other cognitive mechanisms (e.g., noisy discounting factors) from artificially enhancing the model’s ability to produce more human-like deviations from rationality. By setting a fixed discount rate, we enhance the robustness of our results, ensuring that observed effects are intrinsic to the model rather than artifacts of other non-PV calculations.

Similarly, in cases involving risky choices where probabilities are unknown, such as when participants are offered an option of 100 dollars with unspecified probabilities, we address ambiguity by defining the expected value of this option as <AMB> * 100. Here, <AMB> is a token explicitly designed to capture situations of ambiguous probability, allowing for a systematic treatment of ambiguous probabilities replicating how human participants might perceive such options.

Question 1: The positioning of the paper can be improved.

Response: Thank you for this excellent suggestion. In response, we have revised parts of the Introduction and Discussion sections as follows: Introduction: “We propose a hypothesis about how human-like decision patterns might be produced in language models for risky and intertemporal choice: such human-like behaviors might arise from a language model trained on ecologically valid calculations of expectations. This suggests that human-like biases could result from the training task and numerical reasoning, independent of natural language supervision.”

Discussion: “This pretraining allows the model to better capture human risky and intertemporal choices compared to classical behavioral models and, in some cases, even surpass the performance of the general-purpose LLaMA3 model. These results suggest that the observed cognitive biases in LLMs may stem from the training task, architecture, and numerical reasoning abilities, without requiring natural language supervision.”

Question 2: LLM’s training on other mathematical tasks not related to expectation.

Response: Within the task of expectation calculation, we observe that the distributions of probabilities and values in the training data significantly influence the representations learned by the language model. Moreover, ablating the expectation calculation task itself leads to a decrease in model performance, indicating the centrality of this task to the model’s success. These two findings suggest that modifying tasks—whether by changing task content (e.g., from expectation calculation to other types of task) or by adjusting the training distributions (e.g., from uniform to ecological distributions)—would likely yield different model behaviors.

In evaluating our models on four distinct human datasets, we find that the embeddings generated by Arithmetic-GPT are nearly comparable to those of an MLP directly trained on human data, with only a 1-4% difference in $R^2$. This suggests that training on the expectation calculation task has likely brought Arithmetic-GPT close to ceiling performance, further underscoring that deviating from training on expectation tasks likely leads to less correspondence with human behavior.

Question 1: How do open-source LLMs like LLM360/OLMo perform on our task?

Response: Thank you for this question. One of our priorities in selecting LLMs is to control for potential data contamination—specifically, whether models were trained on test questions. We examined open-source LLMs, such as OLMo (trained on the Dolma corpus [1]) and GPT-J (trained on the Pile [2]), both of which include scientific literature on human biases in risky and intertemporal choices, dating back several decades. Due to this overlap, these models do not provide additional control benefits over other open-weight models like LLaMA. As a result, we have opted to leave this comparison for future research.

[1] https://arxiv.org/abs/2402.00159

[2] https://arxiv.org/abs/2101.00027

Question 2: Visualizing LLaMA3’s embedding for values, probabilities, and discount factors.

Response: Thank you for this excellent suggestion. We visualized the embeddings of LLaMA3-70B-Instruct for these quantities and observed that only the discount factor exhibits the hyperbolic discounting curve commonly found in behavioral economics. We have included this result in the revised manuscript Fig A3.

Question 3: Can a strong LLM fine-tuned on human data offer an upper bound on model performance?

Response: We believe that within specific domains of human cognition, an MLP trained directly on human data sets the upper bound for capturing maximal explainable variance. For example, Binz & Schulz (2024) fine-tuned the LLaMA-1-65B model on the choices13k dataset and achieved a decision preference accuracy of 81.9% (see footnote 2 in pg. 15 of [3]), whereas the original choices13k paper reported that the best-performing MLP model achieved an accuracy of 84.15% [4]. Decision preference accuracy is defined as the proportion of decision problems where the model-predicted probabilities align with the empirical choice probabilities on the same side of 0.5. Therefore, fine-tuning off-the-shelf LLMs may achieve similar predictive accuracy; however, like MLPs, these fine-tuned models offer limited interpretability regarding what enables their strong predictive performance. Gaining control over the LLMs’ training data would allow cognitive scientists to better examine which data contribute to more human-like representations in LLMs.

[3] https://arxiv.org/abs/2306.03917

[4] https://doi.org/10.1126/science.abe2629

Comment 1: Add texts to reflect that LLaMA3 is on par with the Arithmetic-GPT in predicting intertemporal choice. And bold best-performing models within each of the categories.

Response: Thank you for the suggestion. We agree and will update the tables and text to bold the best-performing models within each category. Additionally, we have clarified in the text that LLaMA3 performs comparably to Arithmetic-GPT in predicting intertemporal choice.

Weakness 1 & 2: Lack of experiments that manipulate model sizes and data quantity for Arithmetic-GPT.

Response: We appreciate the reviewer’s suggestion and have conducted an additional analysis to systematically examine the effects of model size and data quantity. Specifically, we considered three model sizes (10M, 1M, and 30K parameters) by varying the hidden size of Arithmetic-GPT. Independently, we analyzed the impact of three dataset sizes (1M, 100K, and 10K mathematical equations), with the smaller datasets being random subsets of the original 1M ecological synthetic dataset. The detailed results are presented in Table A3 of the revised manuscript. In summary, we observed that larger model sizes consistently improve performance, while reducing the dataset size to as few as 10K equations has only a modest impact on performance.

Weakness 3: Lack of other computational tasks besides expected value calculation. Does the result hold for language-oriented tasks?

Response: Thank you for these suggestions. We believe our ablated synthetic datasets, where expected values were removed from the right-hand-side of the equations, offer some insight into alternative computational tasks. In these cases, the models trained on ablated data could not learn to perform expected value calculations properly but may still capture certain statistical patterns, such as frequencies and associations between probabilities and values. This led to decreased performance, supporting the role of explicit expected value calculations. However, the reviewer’s suggestion of structurally different tasks beyond simple ablations is interesting. While theoretically, there are countless computational tasks that could be used to train language models, identifying which tasks yield meaningful hypotheses is challenging. In our study, we focused on expected value calculations due to their direct relevance to rational decision-making. We believe that in other domains, such as language tasks, training on rational solutions could serve as a valuable first step.

Weakness 4: Evaluate LLaMA and Arithmetic-GPT’s computational abilities both in and out of distributions, and examine their effect on the results.

Response: Thank you for this comment. We would appreciate further clarification to provide a more precise response. If the reviewer is referring to the effect of training data distribution on model performance, we addressed this in the paper by comparing Arithmetic-GPTs trained on data following either a uniform distribution or an ecological distribution. Our results demonstrate that training with ecological distributions leads to improved performance compared to uniform distributions. We have also included an additional exploratory analysis in the newly added Appendix F, titled “Variations in Model Sizes, Data Quantity, and Data Distributions.”

Weakness 5: No discussion of the tradeoff between more general models (like LLaMA) vs. specific models (like our Arithmetic-GPT)

Response: This is a great point, thank you! In response, we have added the following to the Discussion section: “In contrast, we explicitly manipulate the training data for our Arithmetic-GPT, thereby uncovering a key factor that contributes to the model's ability to explain human choices. This targeted approach, however, comes at the cost of the broader versatility inherent in off-the-shelf LLMs, which are designed to handle a wider range of tasks.”

Thank you for your response.

Weaknesses 1 & 2: I found the experiments sufficient overall. While you explained the importance of model capacity, you did not address why the smaller dataset size did not affect performance. The lack of a performance drop suggests that your model might overfit to the target data distribution. This connects to Weakness 3 and the comment about LLaMA3, as it highlights how overfitting to the task and domain is not always straightforward, yet appears to be the case here.

That said, while not strictly necessary for this study, it would be interesting to explore the effects of using larger models or incorporating modifications like increased model depth. Such experiments could even provide an initial direction toward scaling laws for cognitive models.

Weakness 3: As you mentioned, connecting computational tasks to meaningful hypotheses is challenging. This poses a limitation when presenting your approach as a general one. For language-oriented tasks, I tend to think that training on rational solutions, while valid as a first step, may not yield a cognitive model that is on par with larger, stronger models—particularly within the model size class discussed in the paper (putting aside the problems of using LLM as a cognitive model).

Weakness 4: Apologies for the earlier unclear comment. I intended to suggest evaluating LLaMA's and Arithmetic-GPT’s computational abilities more explicitly—specifically, benchmarking their expected value calculation abilities both in- and out-of-distribution. This could provide insights into their connection to the results. For instance, does a model that excels in expected value calculations also serve as a better cognitive model? Are 15K examples enough to master this ability?

Weakness 5: I’m not sure my earlier point came across clearly. My point was that LLaMA models perform comparably to your model for the given task, despite not being trained specifically for it. When combined with your statement about the difficulty of finding computational tasks that improve cognitive modeling for target tasks, this suggests that for many use cases, using a larger, stronger general-purpose model may be more effective than training a specialized cognitive model.

Overall: I find your paper interesting and thought-provoking. However, I feel that the presentation of your approach as a general approach (e.g., in the abstract) is somewhat overstated compared to its actual applicability.