Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values

We thank the reviewer for the constructive comments in improving the paper. We will incorporate your suggestion about the related work and discuss how the current paper differentiates itself. Below, we address the comments and questions raised by the reviewer.

I suspect any LLM-generated resource allocation will be greatly dependent on the specific prompt and context instructions provided to the model. Given that the main contribution of the paper is the “distributive fairness” setting, further results on how different prompts or workflows affect allocations would improve the paper. For example, providing explicit definitions of fairness (words like “minimizing inequality” are subjective to LLMs and to humans). Does LLM resource allocation depend on the type of resources being allocated? (e.g., models could behave differently for medical resources compared to less critical resources).

Design choices for evaluating alignment: We intentionally argue about misalignment between humans and LLMs based on LLMs’ responses to a prompt adapted as closely as possible to the instructions given to humans. These instructions, which requires the decision-maker to “determine the allocation you consider to be the fairest”, involves judgements about the definition of the word “fairest”. The goal is to identify whether LLMs represent the (often diverse) distributive preferences of humans—something that requires the interpretation of what fairness means.

Ability to ensure specific fairness properties: We also report LLMs’ behavior when given prompts that include specific definitions of fairness (section 5 and Appendix F.2). We find that LLMs struggle with computing allocations satisfying specific notions of fairness, even with unambiguous instructions such as “Your task is to determine the allocation in which all individuals have exactly the same value for their respective bundles”, or “Your task is to determine the allocation where each individual prefers their own bundle the most”. These observations raise questions about the ability of commonly used chatbots to provide allocations that serve the purpose for users who explicitly provide their distributive preferences.

Sensitivity to context and prompting: Furthermore, we demonstrate how the context provided and the framing of the prompt can heavily impact LLMs’ responses, as the reviewer correctly identifies. We show how LLMs’ preferences change significantly when they are assigned the role of a recipient (section 5). We also show how LLMs’ become biased to welfare maximization when the intention is modified, i.e. where the goal is to provide an allocation that is the “most desirable” or “acceptable by all” instead of the “fairest” (section 5 and Appendix F.1). Similarly, providing LLMs with options to choose from, and context about human preferences (section 4.1), leads to a change in preferences as compared to the case where they are required to generate allocations from scratch without any context about allocations humans find desirable.

Extensions to other settings: Given the degree of variation in LLMs’ distributive preferences we observe due to modifications to the instructions and context provided, we suspect there to be substantial variations in different resource allocation settings based on factors such as the (i) type of resource being allocated (e.g. medical resources vs. luxury items), or (ii) the relationship between the decision-maker and recipients (e.g. friends vs. competitors), etc. While these are interesting directions for future work, our paper aims to identify the challenges with using LLMs for resource allocation, due to their inability to fully represent human preferences and sensitivity to exact instructions and context provided.

Each model was queried 100 times per question/instance; Were all generations perfectly valid for all models? How was valid output ensured? What was the percentage of invalid generations?

We used a two-staged prompting strategy (described in detail in Appendix J) to ensure the validity of each sample collected. The first stage involved generating a free-text response from the LLM, without enforcing any formatting constraints. In the second stage, the model was asked to arrange the allocation it provided in the free-text response as a JSON object. We manually verified the validity of the second step, for a subset of samples, in correctly identifying the allocation returned by the LLM. The allocation indicated by LLMs in the first step were misidentified in the second step in less than 1% samples.

In terms of invalid generations, we did not find any responses that deviated from the instructions (to indicate an allocation) in instances involving only goods. The only type of invalidity we observed was in instances with money, where LLMs tended to allocate more money than was available, although this behaviour was limited to ~2% responses.

Some form of confidence interval should be added to all numeric results, given that these are sampled with n=100 for each LLM-question pair;

Thank you for the valuable suggestion. We will include confidence intervals (or a similar measure of deviation) wherever we report aggregate metrics for our experiments. For each numeric result, we use Fisher’s Exact test, and consider any difference as significant if the p-value is less than 0.05. This is true for comparisons between LLMs and humans, or any two LLMs, both at the level of individual instances and at the aggregate level. For example, the statement that Claude-3.5-S returns envy-free (EF) allocations significantly more frequently than humans do, is based on a statistical comparison (using Fisher’s Exact test) of the fraction of responses from humans and Claude where EF is satisfied.

Thank you for the constructive comments and interesting questions. We shall answer them below:

What do the results look like for more models, especially reasoning models?

We conducted our experiments with Gemini-2.5-Pro (a SOTA reasoning model), with which our primary results still hold. Here is a summary of our findings:

1. The model continues to ignore equitable (EQ) allocations. Its responses are much more deterministic (in terms of the allocation returned for a given instance), and prioritizes notions such as envy-freeness (EF) and Pareto-optimality (PO) instead.

2. In instances with money, only when EQ allocations are also Pareto-optimal, Gemini-2.5-Pro returns such allocations. However, this seems to be primarily motivated by PO, since the model never returns EQ allocations that are not PO. This behaviour is different from the non-reasoning models, potentially due to the reasoning model’s enhanced ability to satisfy additional desirable properties along with the properties it considers most important.

3. Gemini-2.5-Pro does not choose EQ allocations among a set of options (unlike GPT-4o and Claude-3.5-Sonnet). Instead, it selects allocations satisfying EF or PO (or both). This is true even in instances with money, where it returns EQ+PO allocations when required to generate allocations from scratch (as described above). This adds to the evidence that the reasoning model does not prioritize equitability.

4. When explicitly instructed to compute EQ allocations, Gemini-2.5-Pro can do so with over 95% accuracy, where non-reasoning LLMs do not cross 18%. Given the former’s enhanced reasoning capabilities, this is expected. However, this further strengthens the observation that Gemini-2.5-Pro ignores EQ allocations not because it struggles to compute them (unlike for the non-reasoning LLMs), but because it may not equate “fairness” with “equitability”.

Hence, the difference between the perceived fairness of humans and the reasoning model is even clearer (given that it does not select EQ allocations among a set of options). We will be updating the paper with these results.

How do you distinguish between genuine differences in the models' distributive principles from artifacts of prompt framing?

The ground truth of LLMs’ distributive preferences that we use for comparing with humans is based on free-flow responses (without any enforced formats) to instructions adapted (as closely as possible) from those given to human respondents. This comparison reveals certain clear differences between the preferences of humans and those of LLMs, such as the neglect of equitability and enhanced preference for economic efficiency from the latter.

As the reviewer correctly points out, our experiments show that non-semantic changes to the prompt (i.e. those where the underlying task is the same) lead to significant changes in the distribution of responses (Appendix D). This reveals the lack of consistent distributive preferences in LLMs. However, across all such changes, the misalignment between humans and LLMs only increases, with an increased share of responses for efficiency and no increase in the share of responses ensuring equitability, from LLMs. While mapping the exact distributive preferences of LLMs’ across diverse settings is a challenging task that requires extensive future work, our work identifies key areas of misalignment between humans and LLMs that persist across different prompt framings.

When a model's allocation deviates from an equity-focused rule, how do you differentiate its goals from its results?

This is an excellent and profoundly important question: How can we disentangle and differentiate between an LLM’s intentions, abilities, and underlying values?

Precisely to understand this, we experiment with a variation of the task, where the LLMs need to select the allocation they consider fairest among a set of options satisfying different notions (section 4.1). Our results show that two out of the four LLMs (GPT-4o and Claude-3.5-Sonnet) that we test select EQ allocations among a set of options although they fail to return EQ allocations when required to generate their preferred allocation from scratch. As correctly pointed out by the reviewer, a potential reason for this is the inability to compute EQ allocations even when specifically asked to do so—something we shown in Section 5.

However, as we show in Section 4.2, using reasoning-enhancement prompting techniques such as Chain-of-Thought does not uniformly improve alignment. Similarly, using an LLM with advanced reasoning capabilities (such as Gemini-2.5-Pro) does not ensure alignment either, since such models may have different goals altogether (as discussed above).

We thank the reviewer for the constructive comments in improving the paper. Below we address the questions raised by the reviewer.

Many of the claims in this paper about human preferences are based on a single citation (i.e. Herreiner and Puppe). I wonder if the authors can cite additional papers to support this. I am not fully convinced about the claims regarding human preferences.

To support our claim that equitability, often referred to as “inequality-aversion”, is prioritized by humans in distributive settings, we cite a variety of seminal works in our paper (line 184). In addition to these, a large body of work shows how inequality aversion is a key factor in the distribution [1] and re-distribution of income [2], dispute resolution [3], and healthcare policy [4]. In fact, even experiments using functional MRIs of human subjects involved in experiments involving monetary exchanges provide evidence for human sensitivity to inequality [5].

[1] Choshen-Hillel, S., & Yaniv, I. (2011). Agency and the construction of social preference: Between inequality aversion and prosocial behavior. Journal of Personality and Social Psychology, 101(6), 1253–1261

[2] Dawes, C., Fowler, J., Johnson, T. et al. Egalitarian motives in humans. Nature 446, 794–796 (2007).

[3] Loewenstein, G. F., Thompson, L., & Bazerman, M. H. (1989). Social utility and decision making in interpersonal contexts. Journal of Personality and Social Psychology, 57(3), 426–441.

[4] Costa-Font, J., Cowell, F. Incorporating Inequality Aversion in Health-Care Priority Setting. Soc Just Res 32, 172–185 (2019).

[5] Tricomi, E., Rangel, A., Camerer, C. et al. Neural evidence for inequality-averse social preferences. Nature 463, 1089–1091 (2010).

The second concern is that there may be potential confounders (e.g. the reasoning of LLMs or any errors in their calculations while deciding the "fairness score" of different allocations). This concern doesn't make the results invalid but it would be interesting to decode what exactly is going on and whether we are reading too much into these results. Have the authors tried prompting the LLMs to explain their reasoning while generating or selecting answers? The generated reasoning will be more difficult to evaluate automatically or manually but some random samples can be evaluated and included in the paper for a more qualitative insight.

We indeed analyzed the explanations from the LLMs for various questions and found some interesting insights. The primary observation is that LLMs use a “greedy approach” to allocate resources. This involves either (i) a round-robin-like mechanism where agents are picked one-by-one and allocated their favorite good among the remaining ones, or (ii) assigning goods one-by-one to individuals who value them the most. The first procedure often leads to allocations satisfying envy-freeness and the second ensures maximum social welfare, which concurs with observations from our quantitative analysis that LLMs return such allocations frequently. On the other hand, such procedures would rarely result in equitable allocations (especially in the instances we consider). Instead, computing an equitable allocation would require assigning goods to individuals who do not value them the most—something LLMs seem reluctant to do.

Furthermore, in instances with money, most LLMs fail to identify that allocating the money to a single agent can make the allocation envy-free or equal overall, and instead attempt to allocate even the money equally, which results in no fairness property being satisfied.

We appreciate the feedback, and will definitely include such insights from qualitative analysis of the explanations in our paper.

Related to the above, it would be useful to use some reasoning models (e.g. R1 or o1) also in these experiments and see whether the results change.

We are happy to share results from recent experiments we conducted with Gemini-2.5-Pro, a SOTA reasoning model.

1. Equitable (EQ) allocations are still rarely returned. While the responses from Gemini-2.5-Pro are much more deterministic (in terms of the allocation returned for a given instance), the model ignores EQ allocations, and prioritizes notions such as envy-freeness (EF) and Pareto-optimality (PO) instead.

2. In instances with money, only when EQ allocations are also Pareto-optimal, Gemini-2.5-Pro returns such allocations. This seems to be primarily motivated by PO, since in cases where EQ allocations are not PO, the model never returns them. This difference between the reasoning model and the non-reasoning models is potentially due to the reasoning model’s enhanced ability to satisfy additional desirable properties along with the properties it considers most important.

3. Gemini-2.5-Pro does not select EQ allocations from a menu of options (unlike GPT-4o and Claude-3.5-Sonnet). Instead, it chooses allocations satisfying EF or PO (or both). Surprisingly, this is true even in instances with money, where it returns EQ+PO allocations when required to generate allocations from scratch. This strengthens the observation that the reasoning model does not prioritize equitability.

4. Unlike the non-reasoning LLMs, Gemini-2.5-Pro can compute EQ allocations with more than 95% accuracy, when instructed explicitly to compute one. Given the latter’s enhanced reasoning capabilities, this is expected. However, this further strengthens the observation that Gemini-2.5-Pro ignores EQ allocations not because it struggles to compute them (which is the case with the other models), but because it may not equate “fairness” with “equitability”.

Hence, the main results, regarding the misalignment between humans and LLMs regarding equitability, hold even with an advanced reasoning model like Gemini-2.5-Pro. If at all, the difference between the perceived fairness of humans and the reasoning model is even clearer (given that it does not select EQ allocations among a set of options). We will be adding these results to the paper.

We thank the reviewer for their fruitful and constructive comments. The answers to the reviewers questions are given below:

Clarify what benchmark is used to determine “alignment” with humans: The paper reports that LLMs deviate from human responses, yet the human data itself shows significant heterogeneity. Meanwhile, models like GPT-4o and Claude select EQ allocations over 60% of the time in the menu setting. Are these models misaligned because they differ from the most common human response, or overly aligned with a narrow fairness principle? Please clarify how alignment is defined and assessed across different settings.

Alignment is evaluated by comparing the distribution of LLM-generated allocations against the empirical human distribution, with heterogeneity in human data accounted for by comparing choice frequencies across multiple fairness notions. In menu-based settings, the analysis aims to uncover the source of misalignment: is it due to LLMs prioritizing welfare maximization, computational constraints in exploring many possibilities, error in calculations, or reasoning errors such as losing track in their chain of thought?

The fact that models like GPT‑4o and Claude select equitable (EQ) allocations over 60% of the time indicates that they inherently value fairness notions like EQ when presented with explicit options (and other models do NOT). This observation suggests that, although LLMs remain misaligned with the diverse human distribution, their behavior can potentially be guided and refined to achieve better alignment.

I recommend being more explicit in the text with the prompt phrasing for each experiment, especially in Section 4.1: Was the model asked to “choose the fairest allocation” or simply to “choose one”? This distinction matters a great deal, as it shapes whether the task is interpreted as a normative judgment, a preference prediction, or something else.

As demonstrated in the sample prompt given in Appendix J.5, the instruction is to “...indicate the allocation you think is fairest…”. This instruction is used to make sure that the only difference between the original prompt (the questions asked to humans) and the prompt used in section 4.1 is the addition of the set of options to choose from. The reviewer correctly points out that the precise question can have a significant impact on the distribution of responses—something we address in Appendix F.1. We appreciate the feedback on being more explicit in the main text about the prompt framing, and shall incorporate the same into the final version.

Discuss the choice of sampling temperature and implications of repeated querying. At a high temperature, model responses may be highly stochastic, raising questions about whether observed variation reflects noise or stable preferences. Would lower temperature settings produce more consistent or more human-aligned responses? A sensitivity analysis would help.

We select temperature 1 as it is the default setting for most models, and end users typically do not adjust this parameter. Hence, our evaluation reflects the behavior that is most likely to be observed in real-world usage, where the default temperature is effectively the standard operating condition.

Based on preliminary experiments with a temperatures of 0 and 2, we observe that alignment with humans does not improve. This is primarily for two reasons, (i) for temp = 0, the outputs of the LLMs are more deterministic and fail to capture the diversity of human preferences (which is done to a better extent with temperature=1), and (ii) equitable allocations are still not returned at either temperature. Rather, there is no clear difference between the distributions returned at temperatures of 1 and 2. We will add more details on the temperature sensitivity analysis to the paper.

Clarify the interpretation of the “Cognitive Bias” results.

The “Cognitive Bias” results examine whether LLMs replicate or diverge from common human cognitive biases, such as framing effects or bystander effect, when making allocation decisions. These results are not intended to suggest that LLMs possess human-like cognition; rather, they highlight systematic patterns in model outputs that resemble known human biases.

In human experiments, it was shown the distribution of human responses do NOT exhibit any significant change when participants are making decisions for others as bystanders or in settings where they are one of the players (have skin in the game). While LLMs have been shown to exhibit human-like altruistic behavior in various environments [1][2], we show that they do not always exhibit this behavior in resources allocation settings, since they make self-serving decisions in certain cases.

[1] Qiaozhu Mei, Yutong Xie, Walter Yuan, and Matthew O. Jackson. A turing test of whether ai chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences, 121(9):e2313925121, 2024. doi: 10.1073/pnas.2313925121.

[2] Aher, G. V., Arriaga, R. I., & Kalai, A. T. (2023, July). Using large language models to simulate multiple humans and replicate human subject studies. In International conference on machine learning (pp. 337-371). PMLR.

The paper would benefit from a more in-depth comparison with the growing literature on LLMs and prosocial behavior, economic games, and normative reasoning. While some of this work is mentioned in passing on page 3, the paper misses the opportunity to position its findings in relation to this broader body of research.

We appreciate this feedback and shall add more detail about how our work relates to the broader field of moral alignment in AI and social behaviour of LLMs, in the final version of this paper.