Questioning the Survey Responses of Large Language Models

Thank you for the feedback.

What would you expect the models’ answers to be? It is unclear what to expect models’ responses to be. Prior work has hypothesised that models may trend towards certain demographics; for example, younger demographics, which tend to be more present on the internet. Another candidate could be models trending towards the responses of particularly populous U.S. states, simply because they may produce larger volumes of data. However, we observe that base models' ACS responses are qualitatively different from those of human populations. Because of these qualitative differences, we argue that the quantitative analysis of prior work may be misleading (Section 5). This is not to say that base models do not have biases, or do not represent certain populations better than others. Rather, our findings signal the need to move beyond multiple-choice prompting towards a more holistic evaluation of LLMs (e.g., open ended survey responses) to elicit more faithful representations of the population a language model might represent.

Impersonation We focus on evaluating whether the survey responses of LLMs are representative of certain U.S. demographic subgroups. In this setting, it is standard to prompt the model without any added context. Assessing whether models have the ability of producing answers relevant to specific sub-groups if instructed to do so is beyond the scope of our work.

Do humans have A bias? There is evidence for ordering bias in humans in the context of opinion surveys, and a tendency not to pick extreme values. However, in the context of the ACS demographic survey, it is well-understood that ordering effects play a very minor role in the distribution of responses collected by the U.S. census. The recent work of Tjuatja et al., 2023 finds that the response biases of language models (e.g., A-bias) are generally not human-like.

Tjuatja, Lindia, et al. "Do LLMs exhibit human-like response biases? A case study in survey design." arXiv preprint arXiv:2311.04076 (2023).

Does RLHF introduce steering? We evaluate models that have undergone RLHF, particularly the Llama 2 Instruct models, text-davinci-003, GPT-4, and GPT-4 Turbo. However, these models have undergone both standard supervised fine-tuning (i.e., instruction tuning) as well as RLHF. Overall, we observe that the responses of fine-tuned models vary more across questions (e.g., are not as balanced as those of base models). We, however, find no evidence that the responses of RLHF models better represent those of human populations. This is not to say that RLHF introduces no steer, but rather that the multiple-choice survey methodology that has recently gained traction in the community may not be appropriate to study this phenomena.

Thank you for the feedback. We hope to address your concerns and clarify some misunderstanding below.

their experiment adds little value to further support their claim on "better represent subgroups whose aggregate statistics are closest to uniform."

We believe this to be a misunderstanding. We do not claim that models better represent subgroups whose aggregate statistics are closest to uniform. Let us clarify.

Our experiments show that, using the de-facto standard multiple-choice methodology, models strongly trend towards uniformly random responses (Figures 4 and 5). This results in a very strong correlation between subgroup entropy and alignment (Figure 6). Such correlation consistently holds across surveys, subgroups, and models. Therefore, models appear to better represent subgroups whose aggregate statistics are closest to uniform.

Our experiments explain the findings of earlier work (i.e., Santurkar et al. 2023, see the discussion in Section 5, “Beyond the ACS”), and why these findings may be misleading. Models appear to better represent younger demographics not because of the pre-training data, but because younger demographics happen to have more uniform responses for the ACS. We are not claiming that language models actually better represent certain populations intrinsically. We are arguing that the de-facto standard methodology to survey language models has strong limitations, and it can potentially lead to misleading insights. Our claims are not about whether language models better represent certain populations, but rather about the limitations of the dominant survey methodology itself.

The 2019 ACS uses stratified sampling. Therefore, I believe that the variance is already being increased to obtain a representative sample. Since the additional context given to the LLM is very limited + the draws are independent, it seems like a very hard task to generate a matching distribution by the LLM.

If we understand correctly, your point is that the aggregate responses of the U.S. census (the reference population used in our work) appear to be more entropic than they actually are, due to the U.S. census not representing households uniformly at random. Our claim is that, when using the dominant multiple-choice methodology to survey models, model responses are qualitatively different from those of the U.S. census. Model responses strongly trend towards uniformly random, irrespective of the survey question being asked. The responses of the U.S. census do not – they are heterogeneous. If stratified sampling were to have a small effect on the U.S. census responses, it is still the case that model responses (e.g., the blue dots in Figure 4a) look nothing like those of the U.S. population (green dots).

Please note that there are no “draws” – for each survey question, we extract the models’ survey response analytically by extracting its next-token probabilities over each of the answer choices (e.g., “A”, “B”, …). This is the standard methodology introduced by Santurkar et al. for surveying language models using multiple-choice questionnaires. Our contribution is to shed light on the properties of these output distributions.

First-token probabilities may be a biassed measure to obtain the replies [...] more advanced prompting techniques, such as Chain-of-Thought (Wei et al., 2022a), could improve the coherence and dependencies in the responses, especially for the sequential generation.

We agree with your points. They support the overall conclusion of our work that the current multiple-choice methodology used to survey language models has strong limitations, and we should move towards a more holistic evaluation of LLMs (e.g., open ended survey responses rather than multiple choice) in order to elicit more faithful representations of the population a language model might represent.

I believe that the sensitivity to ordering and labelling biases is known (Wei et al. (2022b) and Wei et al. (2023)).

Ordering bias has been observed in various works. We cite Robinson and Wingate (2023a). We are happy to include more references in the final version. Our work is different from prior work in studying the effects of ordering biases for models’ survey responses. We show that models’ survey responses can substantially change after adjusting for their ordering biases, leading to fundamentally different insights regarding the populations that models’ best represent.

We hope the additional explanations helped address your concern. We are happy to answer further questions that you may have.

Thank you for the clarifications.

Re-reading your paper from the perspective that your main goal is to question the validity of prior work on survey-derived alignment measures, which (unconditionally) sample survey responses from LLMs, made me realize that my initial rating and assessment of the paper could have been much higher. I have adjusted my rating accordingly.

My initial assumption, related to the stratified sampling comment, was that surveys through LLMs were conducted conditionally on demographics. For example, the paper released today by Ashokkumar et al. (2024) requires LLMs to respond to survey questions conditional on random demographic characteristics. This is a more valid approach to conducting surveys through LLMs, and I agree with your paper that there are better approaches than unconditional sampling (or evaluating token probabilities). Similar to the remark of Reviewer gFgE, the conditioning on demographics gave me a conflict with the US Census data set, as many questions related to the demographic information would then be embedded in the prompt. However, as you point out, you also run experiments on survey opinions.

It might be worthwhile to point out that conditioning LLMs on demographics via prompts may be part of the solution. However, you implicitly already do this with the sequential prompting strategy, keeping previous demographic information in context. Maybe this points toward conditional not being a solution?

References

Ashokkumar et al. (2024). Predicting Results of Social Science Experiments Using Large Language Models.

Thank you for the positive assessment and the feedback. Please note that we discuss in Appendix E how our findings for the ACS transfer to opinion surveys. We agree that our observations regarding models’ survey responses may be partially attributable to survey questions not having a “correct” answer. This is in stark contrast with the multiple-choice questions that are typically used to evaluate LLMs (e.g., MMLU), and reveals interesting new insights for model evaluation.

Thank you for your comments. We will implement the suggested changes to improve the figures. Let us address your questions in the following.

Selection of questions. We chose 25 representative questions to achieve diversity over topics (e.g., educational attainment, healthcare status, employment status, etc.), while keeping figures readable. In the rebuttal pdf we include additional results for all ACS survey questions for comparison. We provide the A-bias and response entropy of models with publicly available weights. They show identical trends.

The role of training data. Regarding the pre-training data, we find that all base models exhibit similar response distributions, despite being trained on substantially different pre-training data. While we find little difference across base models, beyond the ordering effects identified in Appendix C for smaller language models, we do observe substantial differences between the response distributions of different fine-tuned models. This suggests that fine-tuning data can have a larger effect on models’ distributions. This is a positive result for future work seeking to fine-tune models to alter their survey responses (e.g., emulate those of certain populations).

Why do instruction-tuned models have higher variance in entropy? We generally find that, compared to base models, instruction-tuned models tend to have higher confidence in their responses for at least some of the survey questions. This causes instruction-tuned models to have higher variance in their response entropy compared to base models, as any deviation from balanced responses is more amplified. Note that we used publicly available instruction-tuned models, we did not perform instruction tuning ourselves. For some models (e.g., the Llama models), these instruction-tuning datasets are not publicly available.

Effects of question-ordering We follow the predominant methodology of asking questions independent of each other. Therefore, there are no question-ordering effects. If questions were to be asked in sequence, putting the answer to previous questions in context, then we would expect to observe substantial question-ordering effects. But this was not the focus of our work.

Skewed randomization Yes, for models that exhibit choice ordering biases, skewed randomization would change the response distribution. This is because we would no longer uniformly average across each of the possible choice orderings, but perform some weighted average. However, uniform is the only principled approach here to adjust for it.