Theory of LLM sampling: part descriptive and part prescriptive

The scope of the contribution is marginal. It is unsurprising..

On the contrary, we believe the scope of the contribution is anything but marginal. We view LLMs as a new form of intelligence (Embers of Autoregression: Understanding Large Language Models, reference: https://arxiv.org/abs/2309.13638,) and use grounded methodologies and principles from cognitive science to derive a theory which explains the heuristics behind LLM sampling its options. The topic of our study is inspired from the experiments done in possibility sampling in humans (refer: How we know what not to think? https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(19)30231-1, What comes to mind? https://www.sciencedirect.com/science/article/abs/pii/S0010027719302306) where it was investigated how indeed do humans narrow down options from the vast set of possibilities, most of which go unrealised. For example: If a car breaks down in the middle of the road, an agent faces innumerable decision-making options. It could either call for roadside assistance or a tow truck service on one end or attempt to push the car off the road and walk to the destination at the other extreme. It is difficult to consider/deliberate on all possible options. To narrow these options to a `consideration set’ within a computationally reasonable time, the agent needs to employ a heuristic driven approach (As discussed in the cognitive science papers referenced above).

We investigate this thoroughly in LLMs and conclude that the heuristics used by LLM converges with that of humans. We observe that, both these decision making systems use a heuristic of both descriptive and prescriptive norms. We believe this convergence (and divergence in the exact prescriptive norms) uncovers the nature of decision making of LLMs, as a similar framework was used to explain the heuristics behind decision making by humans. Furthermore, the proposed theory has immediate practical implications like the medical case study presented in the paper. Besides multiple similar use cases, the theory would affect the next wave of synthetic data generation.This strengthens the claim that the paper has actionable insights.

Thank you for the reply and for giving us a chance for further clarification. We believe we can add this discussion to the appendix to clarify for future readers interested in this nuance. But before going into the details we would like to answer your two questions:

• “Prototypicality involves both the descriptive and normative factors, so why wouldn't the model agree with that?” It is not obvious that representation of concepts in the LLM exactly follows humans. The notion of ideality in prototypes in humans is not coming from the language token but the representation of the concept by humans (like the example: for humans robin is a prototype bird).
• Secondly, we would like to clarify that our argument is not that some cognitive psychologists see prototypicality as exclusively descriptive. But we were trying to clarify that the idea is more nuanced. We understand the confusion and describe this in more detail. Please understand that in the process of describing this we are not making psychology claims but clarifying this idea:

The perspective we want to convey from Barselou: we directly quote two instances from the paper. "Whereas ideals may determine a category's graded structure in one context, central tendency may determine a different graded structure in another", and “So far, no experiments have shown that ideals are causal determinants of graded structure". This means according to Barsalou, when sampling, humans wouldn't use both prescriptive and descriptive prototypical ratings in the same context. But, Bear et al (Normality: Part descriptive, part prescriptive) show that human concepts have both components in the same context in a unified representation, providing an insight into how humans think about concepts, and our notion of normality is in fact both prescriptive and descriptive. When we try to rate a normal teacher, we include both prescriptive and descriptive components in the same context.

Given the two different theories, we test this theory in LLMs. Previous experiments in this paper show that LLMs, when sampling from innumerable options, use both prescriptive and descriptive norms as a heuristic in the same context (Figure 1) akin to a unified representation. We show similar results of how prototypicality rating also has the same unified representation of both prescriptive and descriptive norms in the same context.

We appreciate the reviewer’s critique regarding the interpretation of the consistent reporting of zero for fictional concepts. It is difficult to evaluate pre-existing statistical associations, we recognize that the model's ability to identify the fictional nature of terms like 'glubbing' may sufficiently explain the lack of implicit associations. Specifically, the language model likely recognizes that 'glubbing' does not correspond to any real-world concept in its training distribution, which naturally results in outputs that reflect the absence of relevant statistical patterns. Further comparison of the effect and control experiment (in experiment 1) provides the evidence for the theorem when using these new concepts.

We thank you for sharing your initial concerns and for engaging in a constructive discussion. In response to your concerns, we have also made substantial changes to the paper: We believe we had posted clarification on the five concerns you listed

• Implication of the work: We have expanded the introduction and related work sections to better articulate the broader implications and contributions of our study, also in the discussion in rebuttal as acknowledged by sdiR
• We had a constructive discussion on the motivation for the prototypicality experiment also included in the paper.
• We readjusted the validation of pre-existing statistical associations. We added this to the discussion and specifically to L255.
• To the best of our knowledge, presuming trust in the peer reviewed scientific work is common practice. A general mistrust in empirical evidence would undermine scientific progress.
• We have clarified that the critical experiment is meant to empirically show the effect of proposed theory isolating confounders. We add realistic questions like `Pick a sample to represent the given distribution’ and ten other phrasings and a case study to show the applications.

If all the concerns are addressed we kindly request you to reconsider the rating or engage in further discussions.

Thank you for pointing out the error in quoting the results from prior art in an Appendix table. We hope the new corrected version of the paper uploaded clarifies the earlier concern on the validity of human studies. We request the reviewer to bring up the concerns for not improving the score. To clarify our understanding of the concerns you’ve raised initially:

• We’ve pointed out why the current work is markedly different from an arxiv paper you had mentioned, our paper talks about uncovering heuristics while the other is aimed at the assumptions llms have about humans. -Implication of the work: We have expanded the introduction and related work sections to better articulate the broader implications and contributions of our study, also in the discussion in rebuttal as acknowledged by sdiR. Please review the updated version.
• We had a constructive discussion on the motivation for the prototypicality experiment also included in the paper.
• We readjusted the validation of pre-existing statistical associations. We added this to the discussion and specifically to L255.
• The current version amends the error in quoting prior art. The results in the main text and the conclusion remain the same allaying your concerns regarding the human experiments.
• We have clarified that the critical experiment is meant to empirically show the effect of proposed theory isolating confounders. We add realistic questions like `Pick a sample to represent the given distribution’ and ten other phrasings proving robustness and a case study to show the applications.

We would like to hear your feedback for the lower score.

We would like to emphasis on the implications of the work. We view LLMs as a new form of intelligence (Embers of Autoregression: Understanding Large Language Models, reference: https://arxiv.org/abs/2309.13638,) and use grounded methodologies and principles from cognitive science to derive a theory which explains the heuristics behind LLM sampling its options. The topic of our study is inspired from the experiments done in possibility sampling in humans (refer: How we know what not to think? https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(19)30231-1, What comes to mind? https://www.sciencedirect.com/science/article/abs/pii/S0010027719302306) where it was investigated how indeed do humans narrow down options from the vast set of possibilities, most of which go unrealised. For example: If a car breaks down in the middle of the road, an agent faces innumerable decision-making options. It could either call for roadside assistance or a tow truck service on one end or attempt to push the car off the road and walk to the destination at the other extreme. It is difficult to consider/deliberate on all possible options. To narrow these options to a `consideration set’ within a computationally reasonable time, the agent needs to employ a heuristic driven approach (As discussed in the cognitive science papers referenced above). We investigate this thoroughly in LLMs and conclude that the heuristics used by LLM converges with that of humans. We observe that, both these decision making systems use a heuristic of both descriptive and prescriptive norms. We believe this convergence (and divergence in the exact prescriptive norms) uncovers the nature of decision making of LLMs, as a similar framework was used to explain the heuristics behind decision making by humans. Evidence that LLMs process information in ways resembling human heuristic reasoning has profound implications. Furthermore, the proposed theory has immediate practical implications like the medical case study presented in the paper. Besides multiple similar use cases, the theory would affect the next wave of synthetic data generation. This strengthens the claim that the paper has actionable insights.

We would like to bring to your notice that we have updated our draft following the reviews and comments. With the extended discussion period, we are glad to answer further questions. We again thank the reviewer for the constructive feedback. If our clarifications address your concerns, we request you to consider improving the score.

Interpretation of the study

We also view LLMs as a new form of intelligence (Embers of Autoregression: Understanding Large Language Models, reference: https://arxiv.org/abs/2309.13638,) and use grounded methodologies and principles from cognitive science to derive a theory which explains the heuristics behind LLM sampling its options. The topic of our study is inspired from the experiments done in possibility sampling in humans (refer: How we know what not to think? https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(19)30231-1, What comes to mind? https://www.sciencedirect.com/science/article/abs/pii/S0010027719302306) where it was investigated how indeed do humans narrow down options from the vast set of possibilities, most of which go unrealised. For example: If a car breaks down in the middle of the road, an agent faces innumerable decision-making options. It could either call for roadside assistance or a tow truck service on one end or attempt to push the car off the road and walk to the destination at the other extreme. It is difficult to consider/deliberate on all possible options. To narrow these options to a `consideration set’ within a computationally reasonable time, the agent needs to employ a heuristic driven approach (As discussed in the cognitive science papers referenced above). We investigate this thoroughly in LLMs and conclude that the heuristics used by LLM converges with that of humans. We observe that, both these decision making systems use a heuristic of both descriptive and prescriptive norms. We believe this convergence (and divergence in the exact prescriptive norms) uncovers the nature of decision making of LLMs, as a similar framework was used to explain the heuristics behind decision making by humans. Evidence that LLMs process information in ways resembling human heuristic reasoning we believe is not just a fun fact. Furthermore, the proposed theory has immediate practical implications like the medical case study presented in the paper. Besides multiple similar use cases, the theory would affect the next wave of synthetic data generation. This strengthens the claim that the paper has actionable insights.

Questions: change the scale of sampling numbers (rounded for easier observation)

Distribution generated by LLM

The distribution generated by LLM is generally comparable to the input distribution except the higher standard deviation. But we do not claim this as a finding. We will add the original distributions and a discussion in the appendix. Here we give the standard deviation for input having std 15

Random grades for control experiment

Yes, we assign random grades. We add more details in the revised draft. We have three control settings to validate the findings of the experiment.

In the first control we assign random grades to values, and show that sampling is driven by probabilities alone. In the second control we give no grades with the numbers and again show that the sample is driven solely by the likelihood of the distribution. In the third control experiment, we assign the mean Cµ with the highest grade and lower grades for increasing distance from this value. This means the most likely value is also the most ideal value. (we observe sample has no bias, indeed the sample mean is the same as the input mean). As a sanity check we ask the LLM to report averages (to validate that the observation is not due to the inability to mathematically compute averages.) We again thank the reviewer for their detailed review and constructive feedback. If our clarifications address your concerns, we request you to consider improving the score.

Lack of clarity before experiment section

We sincerely thank the reviewer for their valuable feedback, particularly highlighting areas where clarity and engagement in the paper can be improved. Following the comment, we redo our introduction section and restructure the sections to improve the engagement.
Following reviewer suggestion we add a paragraph for giving concrete definitions in the main text and establish all the ideas used in the paper along. We define sampling, descriptive/ideal and prescriptive/average. Later we also define prototypes later on in the respective section. We clarify the notation of ‘C’ upfront to clarify the usage. If you meant the use of C_a for average and so on, we will change this to A_c for camera ready. Currently changing it might cause confusion for reviewers. We believe the changes have helped improve the readability and engagement of the paper. Here are the definitions. We propose a theory that the sampling of an LLM is driven by a descriptive norm (the notion of statistical average) and a prescriptive norm (a notion of an ideal represented in the LLM)(Figure teaser). We define response sampling as the process by which the model probabilistically selects outputs from a distribution of potential responses. A descriptive norm represents what is commonly observed or statistically likely within a given context, reflecting the frequency or probability of events or behaviors without implying any value judgment. A prescriptive norm represents an implicit standard of what is considered ideal, desirable, or valued within a context, often encoded by grades/scores that prioritizes outcomes deemed "better/optimal". The proposed theory implies, the sample of an LLM not only reflects the statistical regularities of the data descriptive norms) but also systematically incorporates an idealized version of the concept prescriptive norms).

Implications of the study

We thank the reviewer for their feedback. We believe the theoretical foundation established in this paper is crucial before addressing empirical implications or adaptations. Here, the critical experiment (experiment -1) of sampling using numbers without any implicit associations is inspired from established principles from prior art in cognitive science to show LLMs use prescriptive norms as a heuristic when sampling their outputs. Apart from the direct case study of medical recovery times of patients being undersampled , we believe this has implications for synthetic data generation when LLMs consistently use prescriptive norms when they sample data which the user might be misled and implicitly think of as a descriptive average. We believe our work opens the doors for investigating the effects of prescriptive norms in various applied tasks. The purpose of this work was to establish the theoretical foundations of this heuristic being present. That said, since we are hypothesizing the existence of a prescriptive component we believe future work in mechanistic interpretability can steer a possible “prescriptive vector” when the output might be biased regarding a particular instance. We believe our approach offers a similarly useful starting point for studying this aspect of LLMs and hope to see future work that goes deeper into this mechanism.

Thanks again for the questions. We would like to bring to your notice that we have submitted the rebuttal and updated our draft following the reviews and comments. With the extended discussion period, we are glad to answer further questions. We again thank the reviewer for the constructive feedback. If our clarifications address your concerns, we request you to consider improving the score.

Thanks for the review and the references. Below we address the three main concerns of the reviewer

Coverage of prior art

We include a detailed related work and discussion on the stochastic parrot explanation of LLM outputs. We also add discussions on the different studies that explain the biases in the output of LLMs.

more importantly, there were no clear mapping system 1 and 2 to the setting of LLMs.

We understand your review as (a) ‘we are not having related work establishing the system-1 mapping of LLM output’: we discuss this in L083-088. (b) ‘There being no literature establishing this’: LLMs are conceptually likened to system-1 by [1] and various peer reviewed prior art. This idea has even been taken ahead by Yao et al. by adding a symbolic component to induce system-2 behavior.

On using black box prompting methodology and response evaluation to make the claim

We would like to point out we aren't the first ones to adopt this method of using outputs to make inference. In fact, a whole bunch of peer reviewed papers in top tier ML conferences and leading journals use the same methodology of inferences of mechanism/biases/heuristics through prompt outputs. We list some of them below:

(Nature Human Behavior) https://www.nature.com/articles/s41562-024-01882-z Inference of the process of theory of mind through a battery of psychologically inspired tests

NeurIPS 2023 - In-Context Impersonation Reveals Large Language Models' Strengths and Biases Prompt outputs recover mechanisms of human-like developmental stages of exploration

NeurIPS 2024 - Measuring Implicit Bias in Explicitly Unbiased Large Language Models Propose a `new prompt-based measure drawn from psychology’s long history of research’. Paper designs a prompt based evaluation to expose nuanced biases in proprietary LLMs that explains implicit decisions in LLMs.

NeurIPS 2024 - Cognitive Bias in High-Stakes Decision-Making with LLMs `Inspired by prior research in psychology and cognitive science’, the paper designs a prompt based evaluation strategy for explaining LLM decisions. The paper infers the LLMs have anchoring effect, the outputs can be explained by Status Quo maintaining tendency and Group Attribution Bias.

ICML 2024 - Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? Study relation to cognitive biases known in children when solving arithmetic word problems. Follows the LLM in a three stage process involved in solving math word problems: mental model, arithmetic expression and answer. They use a template filling based on prompts to construct the model and infer similar cognitive behavior.

ACL 2023 - Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting Study related to uncovering and quantifying biases - availability bias and framing bias are mechanisms which are inferred from prompt output.

Furthermore, in experiment 1, we carefully construct a critical experiment where the result shows evidence for the proposed theory. We show the robustness of the observation to different variations of the prompt to make sure that the results are explained by the proposed hypothesis.

On a final note, we see LLM as a Simulacra of human behavior (Refer: Role play with large language model https://www.nature.com/articles/s41586-023-06647-8) where LLM is effectively roleplaying its behavior and has no underlying intentions or goals. Through our grounded cognitive science experiments methodology, we find consistent patterns of behavior of LLM having a notion of value (prescriptive norm) in its outputs. This effect is observed even after testing for multiple variants of the same prompt indicating the robustness of the findings. The LLM seems to mimic human cognitive processes of descriptive and prescriptive norms, effectively role playing them like a simulacra.

Definitional specificity seems to be absent.

Thank you for this comment. We add the definitions used in the paper earlier as the second para of the introduction. We define sampling, descriptive/ideal and prescriptive/average. Later we also define prototypes later on in the respective section.

Grammar

Thanks for the comments. We hope to have fixed the grammatical errors in the revised version.

We again thank the reviewer for the feedback.

Thanks again for the questions. We would like to bring to your notice that we have submitted the rebuttal and updated our draft following the reviews and comments. We have incorporated the reviewer comments in the updated draft. With the extended discussion period, we are glad to answer further questions. We again thank the reviewer for the constructive feedback. If our clarifications address your concerns, we request you to consider improving the score.

Presentation of the paper: restructuring initial sections

We acknowledge that the introductory sections may come across as repetitive. Our intent was to ensure that readers unfamiliar with LLMs and cognitive frameworks could fully understand the context before delving into our theory and experiments. However, we recognize the need for brevity and will streamline these sections in the revision. Please find the introduction and related work sections revised in the updated submission. We have defined sampling LLMs in para 2. Following reviewer suggestions we added this paragraph to give definition upfront in the main text and establish all the definitions used in the paper there. We request you to revisit the initial sections of our revised draft. Between the submission and now the language and presentation has been worked on, specifically we have restructured the Introduction, discussion and related work.

These N samples provided in the form of prompting, or part of training/fine tuning data?

All evaluations in the paper are on zero shot behavior of LLMs. Hence all input is given by prompt. We acknowledge the confusion and clarify this in the paper. We agree that the current work does not delve deeply into the root causes of the observed prescriptive tendencies, and we pitch this an initial evaluation. But we believe these structured investigations that uncover the heuristics of LLMs can help us understand these systems better. Our goal is to propose the theory and validate it empirically, which we believe is a systematic direction much needed to understanding the mechanism of LLM. We again thank the reviewer for their honest feedback. If our clarifications address your concerns, we request you to consider improving the score.

We sincerely thank the reviewer for their honest and constructive feedback.

Clarifying implications of our proposed theory for better understanding LLMs

We also view LLMs as a new form of intelligence (Embers of Autoregression: Understanding Large Language Models, reference: https://arxiv.org/abs/2309.13638,) and use grounded methodologies and principles from cognitive science to derive a theory which explains the heuristics behind LLM sampling its options. The topic of our study is inspired from the experiments done in possibility sampling in humans (refer: How we know what not to think? https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(19)30231-1, What comes to mind? https://www.sciencedirect.com/science/article/abs/pii/S0010027719302306) where it was investigated how indeed do humans narrow down options from the vast set of possibilities, most of which go unrealised. For example: If a car breaks down in the middle of the road, an agent faces innumerable decision-making options. It could either call for roadside assistance or a tow truck service on one end or attempt to push the car off the road and walk to the destination at the other extreme. It is difficult to consider/deliberate on all possible options. To narrow these options to a `consideration set’ within a computationally reasonable time, the agent needs to employ a heuristic driven approach (As discussed in the cognitive science papers referenced above). We investigate this thoroughly in LLMs and conclude that the heuristics used by LLM converges with that of humans. We observe that, both these decision making systems use a heuristic of both descriptive and prescriptive norms. We believe this convergence (and divergence in the exact prescriptive norms) uncovers the nature of decision making of LLMs, as a similar framework was used to explain the heuristics behind decision making by humans. Evidence that LLMs process information in ways resembling human heuristic reasoning we believe is not just a fun fact. Furthermore, the proposed theory has immediate practical implications like the medical case study presented in the paper. Besides multiple similar use cases, the theory would affect the next wave of synthetic data generation. This strengthens the claim that the paper has actionable insights.

Claim on explanation the observation:

We believe our work improves the understanding of heuristics that drive LLM decision making. We acknowledge it is difficult to ‘explain why this happens ’, but since the heuristics are shown to converge with humans we propose some hypotheses in the text. `One of the basic characteristics of System-1 is that it represents categories as norms or prototypical examples. System-1 has a model of the world that represents what is normal’ (Thinking fast and slow Daniel Kahneman). Also LLMs like neural networks have a feature representation for concepts/tokens from which drives it outputs. In humans prototypes tend to have an ideal component. Expanding on L221 of main text, a Robin might be considered a prototype of the category Bird over a penguin, for a prototypical bird is not a flightless bird! Flying, expected of birds (value), makes Robin a representative example of the group (Smith & Medin, 1981) over a penguin. Robin shares many common features with most birds also with high occurrence. Prototypes in humans are driven by both statistical occurance and ideal qualities. We show initial investigation to how prototypicality in LLMs have the same heuristics as humans. We believe more follow up research could build on the findings of this work as to explain the heuristics in more details. Furthermore there are papers that use LLM zero shot for applications like tabular data generation[1] where LLM is actually asked to produce more numbers that follow distribution of values in each feature. The implication of sampling bias even in the critical experiment designed to validate the phenomenon is practical.But this paper provides a theoretical framework based on human psychology and is not an application work. To address your comment we enhance the discussion on how this theory can guide future research or practical interventions for LLM-based systems.

[1] Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey. Fang et al