Language Model Alignment in Multilingual Trolley Problems

Thank you for your positive feedback that our paper “presents a novel dataset”, “has a solid design with parametric variation and covers 109 languages, 6 moral dimensions, and 18 characters”, has “robust” “experimental protocol”, and “The results discussion” is “aligned with specific research questions, making it easy to follow and understand the key findings.”

The paper implicitly assumes that LLMs should align with country-level or language-level human preferences. However, these preferences may carry biases and could be unfair to marginalized groups.

This is a key point, and we have updated the manuscript to directly address this assumption. Simplifying a bit – one might consider two possible goals for LLM alignment. One could aim to align as closely as possible with human preferences (and, in the case of pluralistic values [1], personalize to the user) or align with a universal principle such as fairness or impartiality for all users. These two goals can come into conflict, but it is not our focus on resolving this conflict. Instead, our focus is on measurement and evaluation – to first answer the question of how do current LLMs align with diverse populations. We have added the following text to the Ethics Statement of our manuscript, together with clarification in Section 3.1:

“The Moral Machine dataset (Awad et al. 2018) is a descriptive measure of human responses to moral dilemmas. It cannot (and should not) prescribe how the system should act. However, because these systems are trained and fine-tuned with large-scale cross-cultural data, they may inherit culturally specific biases. Our focus here is to both characterize these biases and study to what extent input in different languages leads to an LLM reflecting the moral values consistent with the speakers of that language. From this perspective, a higher alignment score for a particular culture may or may not be desirable. While some point to reflecting human values in AI systems as a solution to the alignment problem, our results suggest that due to the plurality of moral values expressed, successful alignment must ultimately depend on the intent of the system developers to avoid perpetuating human preferences that are considered harmful, partial, or parochial.”

[1] Sorensen et al., Position: A Roadmap to Pluralistic Alignment, 2024.

the least aligned model, GPT-4o Mini, could be viewed as one that lacks language-specific biases and instead follows a standardized behavior pattern. Exploring this topic further in the paper could add valuable insights into the potential trade-offs between alignment and bias in multilingual settings.

This is a very interesting hypothesis! Our RQ1 addresses the overall alignment of the models, and RQ3 and RQ4 report on the results for different families of languages. As reported in Section 5.4, there was no language bias for Llama 3.1 70B, with a correlation close to 0. In our appendix, we show the language bias for all the models in Table 10, and find an overall close-to-zero Pearson correlation between the misalignment score and the number of speakers of each language, again showing little bias across different levels of misalignment scores.

Can you elaborate more on the post-processing procedure in L276?

Since the raw response from LLMs is free-form text, we parse every response into the given choices of the trolley problems. For example, for the question in Figure 1 on “a boy” vs “an elderly man”, LLMs may provide answers such as “It should save the boy” or “It should save the younger.” To process the free text answers into the given choices presented in our trolley problem prompts, we first run exact string matching of the choices (e.g., matching the exact word “boy” with the first option), and then as a fallback, in case a good match is not found, we prompt an LLM to parse the answer (e.g., parsing the response “the younger” as the first option of saving the boy too) using Llama 3.1 8B.

Did you observe any effect of position bias in the option order in L233?

In pilot studies, we studied this exact question to make sure our scenario descriptions are robust. We have added these results to Appendix E.5 since they may prove useful to other researchers. Briefly, we conducted a perturbation experiment where we swapped the order of the two options and recorded the number of choice reversals. We report the consistency rate as the percentage of times when the models’ answers stay the same. We find that LLMs are robust against position bias, as most models have 97-99.5% consistency rates in Table 13.

---

We hope our answers address your concerns and can clarify the value of our work. In case you need any followup information, we are more than happy to provide more details!

Did you observe any significant differences in alignment or misalignment patterns between major and minor languages?

Yes! One of the most surprising results in our manuscript is shown in RQ4 (Section 5.4). There, we show that most models lack a significant correlation between the alignment scores and the major/minor nature of the languages – correlations are close to zero. As a clarification, in the paper we use a nomenclature high-resource and low-resource for major and minor languages, respectively. For example, in Llama 3.1 70B, the misalignment scores for the top five most spoken languages are: Chinese 0.38, Hindi 0.51, English 0.58, Spanish 0.68, and Arabic 0.54. This distribution of correlations is about the same as in less spoken languages among our dataset: Bosnian 0.54, Luxembourgish 0.37, Icelandic 0.58, Maltese 0.57, Malayalam 0.54, and Catalan 0.59. We also visualize a misalignment world map in Appendix D.3. The maps again show a lack of a strong correspondence between the approximate development level of a country and its alignment score.

---

We hope our answers address your concerns and can clarify the value of our work. In case you need any followup information, we are more than happy to provide more details!

[the study] aggregates data from all cultural contexts, potentially obscuring the nuances of cultural alignment. Expecting models to align with an average of global human preferences may be unrealistic or inappropriate, given the significant variability in cultural moral standards […] Analyzing language-specific and culture-specific responses would enrich the findings and offer a deeper cross-cultural understanding.

Thank you for the opportunity to let us clarify this. Our RQ3 and RQ4 directly address language-specific preference variations. Specifically, the results from RQ3 show significant variance across different languages (with the relatively high language sensitivity scores reported in Table 3), and RQ4 further shows how moral preferences vary systematically across the four different language clusters (Figure 4). In the Appendix E.4 (Figure 8), we visualize a world map that shows country-specific alignment patterns. These visualizations also help us understand cultural subsets and support a key conclusion of RQ4: we do not find a distinct alignment bias toward low-resource languages.

What steps or analysis would be necessary to evaluate LLM performance within specific cultural subsets, and did your findings reveal any trends where certain models align more closely with specific cultural norms?

As introduced in our answer for the previous question, our RQ3 and RQ4 directly address language-specific preference variations. In addition, for the rebuttal, we added a new analysis of the distribution of moral preferences across three major geographic regions in Appendix E.4.1, covering the regional preferences of the Global East, West, and South countries. This enables us to investigate potential cultural variations in moral decision-making. We plot the distribution of moral preferences across the three regional clusters in Appendix E.4.1.

While the study offers valuable theoretical insights, its direct application to real-world LLM development is not clear.

By analyzing misalignment across moral dimensions and languages, practitioners can identify specific biases, such as over-prioritizing certain demographic attributes, and address these through targeted fine-tuning as needed. For example, incorporating multilingual moral judgments into reward models may influence ethical reasoning capabilities across diverse linguistic contexts and give developers better control and visibility into the populations they are aligning towards. Furthermore, our structured moral dilemmas offer a template for testing model behavior post-alignment, ensuring consistency and robustness. These practical methodologies pave the way for deploying LLMs with improved and more transparent ethical alignment, directly benefiting the users of AI systems operating across culturally diverse settings.

Real-life moral situations are rarely clear-cut, and binary choices may not capture the depth of human reasoning. Including more complex and context-sensitive scenarios could enhance the study’s applicability and realism.

We think this is an important area for future work and discuss it in the last paragraph of our Ethics Statement section in Appendix. We look forward to future follow-ups with richer and more diverse scenarios capable of capturing more nuance in human moral decision-making and judgment.

Could you clarify why a score of 0.6 in RQ1 is considered close alignment to human preference?

We are happy to clarify! We do not intend to say that misalignment scores of 0.6 are to be interpreted as having close alignment with human preferences. Our goal in calling out the number 0.6 is that this was approximately the lowest misalignment we observed across the 19 LLMs tested, so we tried to give some intuition to the reader about the relationship between the global misalignment and the components that go into its calculation. We make clear in Section 5.1 Results that 0.6 is not a hard cutoff for close alignment with human preferences and that even for the best models, much work remains to be done.

We appreciate your positive feedback that our work “introduces an innovative approach” to create the “unique MULTITP dataset,” which is “a comprehensive dataset for multilingual moral alignment,” and presents “novel metrics for measuring alignment.” Thanks for further commenting that our work “demonstrates rigorous methodology” and “careful statistical analysis with transparency in acknowledging limitations,” and “contributes significantly to understanding cross-cultural AI alignment.”

The paper would benefit from clarifying what unique modifications or contributions were made to enhance the dataset beyond its scale and multilingual coverage.

Thank you for the suggestion! The key modifications we have made are as follows: First, to counteract the tendency of LLMs to avoid providing a clear answer, we employ the token-forcing method to force the LLMs to make a binary choice. Second, we expand the original natural language data from English into 107 languages in total. The dataset now covers a wide range of languages including those with many speakers (high-resource) and those with few (low-resource). Third, the stimuli are now formatted to be compatible with LLMs. We reduce arbitrary bias by removing keywords such as “swerve” and presenting the two choices using the bullet points to reduce position bias for the first option over the second. These modifications enable the research community to leverage the Moral Machine dataset to study and evaluate LLM pluralistic alignment.

We have edited the text in Section 3.2: “Setup for LLM Testing” to clarify these contributions.

The reliance on googletrans (the non-commercial open-source tool) may compromise accuracy. Could you provide more details on the manual review process for translations, including the number of samples reviewed, the criteria used for evaluation, and any systematic issues or errors that were found during this process?

Yes, we can provide the details of our translation quality check. Specifically, we have conducted a new study on Amazon Mechanical Turk (MTurk) to evaluate the translation quality using annotators who identified as being proficient in any of our given languages in addition to English. TLDR: Manual annotation shows that translation quality was high across languages. The following details are in the Appendix D of the manuscript:

Details of the annotation task: We designed a task on Amazon Mechanical Turk (Mturk) where annotators rated how accurately our translations reflected the meanings of the original English prompts. Ratings were provided on a 5-point scale, from non-sense translation to accurate translation. First, annotators were prompted to select the language they have the greatest proficiency in from a full list of 106 languages (excluding English). If they were not proficient in any of the listed languages, they did not proceed. Then, after selecting a language, annotators read 25 translation pairs (English sentences or words paired with their translation in the selected language). Instructions and rating scales were adapted from [1, 2] as follows:

Please rate how accurately the translation conveys the meaning of the source text:
Source English Sentence/Text: {source}
Translation: {translation}

• None (Score=1): No meaning of the original text is conveyed.
• Little (Score=2): Only a small portion of the meaning is conveyed.
• Much (Score=3): A substantial amount of the meaning is conveyed.
• Most (Score=4): Most of the meaning is conveyed accurately.
• All (Score=5): The full meaning is conveyed accurately.

We kept the Amazon Mechanical Turk task open for a span of three days, and collected a total of 169 annotation responses covering 44 languages. The results of the evaluation demonstrate strong translation quality. Specifically, 88.6% of the evaluated languages achieved an average score of at least 3.0 out of 5 (i.e., conveying a substantial portion of the original meaning), and no languages received a mean score below 2.0.

The standard deviation of scores across languages ranged from 0.5 to 1.3, reflecting relatively consistent ratings among annotators. Further details on the evaluation setup and results are included in Appendix D.

[1] Lavie, Alon, Evaluating the Output of Machine Translation Systems, 2011

[2] Goto et al., Crowdsourcing for Evaluating Machine Translation Quality, 2014

Originality Weakness: While the paper attempts to explore moral judgments in a multilingual context, it does not significantly advance the discourse beyond existing literature on moral dilemmas. Many of the concepts discussed are already well-established in moral philosophy, and the paper may not provide enough innovative perspectives to stand out in a crowded field.

Our core contribution is to evaluate and assess the alignment of LLMs in moral dilemmas to different populations of language speakers. To make these claims, we evaluated 19 LLMs in over 100 languages and compared their responses to a dataset with over 40 million human responses collected from over 200 countries. We believe we are the first to study and evaluate pluralistic alignment at such a scale. Could the reviewer please clarify what existing literature or citations they are referring to that covers similar findings or show that these results are already well established? We would be happy to discuss these findings in our related work section and compare and contrast their findings with our own. Furthermore, our work is empirical and not philosophical. While we draw on philosophical concepts (e.g., morality and ethics), our work is aimed at having an impact on the study of large language models, AI safety, and alignment.

Quality Weakness: The reliability and validity of the moral dimension classifications could be questioned. The paper may lack comprehensive validation of the measures used to assess moral judgments, which raises concerns about the robustness of the findings.

We also believe the robustness of our findings is extremely important. We have taken two steps towards this goal. First, the moral dimensions studied are inherited from a large-scale social science survey (Moral Machine) published in Nature with over 2000 citations (Awad et al., 2018). Since we aim to understand multi-lingual alignment in LLMs, we decided to use a preexisting high-impact survey with measures and dimensions that have been externally shown to be reliable and valid. Second, we have performed our own robustness checks and controls (RQ5, Section 5.5) and show that the results are consistent across prompt rephrasing. Between using an existing influential and highly validated social science instrument and conducting our own robustness tests, we have carefully controlled for the robustness of our findings.

Additionally, the potential biases inherent in the training data of the LLMs themselves are not sufficiently addressed, which could impact the conclusions drawn.

While we agree that it would be interesting to analyze the relationship between LLM training and model outputs, this is far beyond the scope of our study. Firstly, all LLMs studied (including the open-weight models) are not open with respect to their training data, so we’d be guessing what these models are trained with. Post-training and alignment for these models are even less well understood. Finally, understanding the link between training data and moral biases deserves a manuscript of its own and goes beyond the scientific contributions we aim to make in this work. Instead, our focus is the study of the biases and preferences these models exhibit due to their pretraining and finetuning and how those preferences vary across languages. We will make these ideas more clear in the discussion of limitations.

Significance Weakness: The implications of the findings regarding the moral judgments of LLMs are not fully explored. While the paper presents data on how LLMs align with human preferences, it falls short of discussing the broader societal implications of these judgments, particularly in high-stakes contexts such as healthcare or criminal justice. This lack of contextualization may diminish the perceived significance of the research in real-world applications… What are the real-world implications of the findings, particularly in high-stakes areas such as healthcare or criminal justice?

Thank you for the interesting suggestion! Thinking about the implications of this work for high-stake areas such as healthcare or criminal justice is a great idea. Although our stimuli do not directly concern healthcare or criminal justice, the relatively abstract moral dilemmas we study share structural similarities with social dilemmas in these high-stakes areas. For instance, in the healthcare setting, a hospital may need to select who to treat with a limited supply of medicine or medical devices i.e., choosing who to save between two groups of people as in our stimuli. LLMs may encode language-sensitive biases concerning who should be given priority access to these resources. In the context of criminal justice, the victim's identity might lead to different judgments of fair punishment, similar to the dimensions we studied in this work. For instance, we might expect a higher punishment to a perpetrator if a child is harmed than when an adult is harmed. These biases may also be reflected in LLM responses. Thank you for the idea – we will discuss these possibilities in the future work section of our paper.

What specific measures were taken to validate (worth) the 'moral dimension' classifications used in the study? (Unfortunately, I could not find 'beauty of dimension')

This is an important point that we will clarify with additional text. We have added the following paragraph to address that our work does not aim to validate the worth of any particular moral dimension, as this is far outside the scope of our empirical study on LLM alignment.

“The Moral Machine dataset (Awad et al. 2018) is a descriptive measure of human responses to moral dilemmas. It cannot (and should not) prescribe how the system should act. However, because these systems are trained and fine-tuned with large-scale cross-cultural data, they may inherit culturally specific biases. Our focus here is to both characterize these biases and study to what extent input in different languages leads to an LLM reflecting the moral values consistent with the speakers of that language. From this perspective, a higher alignment score for a particular culture may or may not be desirable. While some point to reflecting human values in AI systems as a solution to the alignment problem, our results suggest that due to the plurality of moral values expressed, successful alignment must ultimately depend on the intent of the system developers to avoid perpetuating human preferences that are considered harmful, partial, or parochial.”

Is there a discussion of the limitations of the methodologies employed, and how might these limitations affect the overall conclusions?

Yes! We discuss limitations in both Section 6 (pages 9-10) where we describe limitations (and future mitigation strategies) related to refusal rates, variations across different domains of moral reasoning, and better dialect support. Furthermore, we provide additional discussion of limitations in the Ethics Statement (pages 10-11, especially the last paragraph).

You mentioned that GPT-3 worked well in terms of the rate. However, did you check the potential issues of bias, fairness, and overall value in the dataset itself?

Yes! Characterizing the variability of human preferences and biases on moral stimuli is a key goal of our work. Because our dataset contains judgments from 100s of countries, we can give a more detailed quantitative account of LLM biases than previously studied.

---

We hope our answers address your concerns and can clarify the value of our work. In case you need any followup information, we are more than happy to provide more details!

For example, in certain cultures where laws or social norms may place different values on individuals (e.g., men being valued more than women in some religions), aligning LLMs with such preferences could reinforce harmful biases. [...] How would you handle cases where human preferences conflict with principles of fairness and equality?

Thank you for pointing out this important and nuanced point. We have added the following text to the Ethics Statement of our manuscript:

“The Moral Machine dataset (Awad et al. 2018) is a descriptive measure of human responses to moral dilemmas. It cannot (and should not) prescribe how the system should act. However, because these systems are trained and fine-tuned with large-scale cross-cultural data, they may inherit culturally specific biases. Our focus here is to both characterize these biases and study to what extent input in different languages leads to an LLM reflecting the moral values consistent with the speakers of that language. From this perspective, a higher alignment score for a particular culture may or may not be desirable [3, 4]. While some point to reflecting human values in AI systems as a solution to the alignment problem (i.e., pluralistic alignment [5]), our results suggest that due to the plurality of moral values expressed, successful alignment must ultimately depend on the intent of the system developers to avoid perpetuating human preferences that are considered harmful, partial, or parochial [5, 6].”

To make the spirit clear, we also added a clarification sentence earlier in the paper when introducing the dataset in Line 175 in Section 3.1.

[3] Bommasani et al., On the Opportunities and Risks of Foundation Models, 2021.

[4] Ethayarajh et al., The Authenticity Gap in Human Evaluation, 2022.

[5] Sorensen et al., Position: A Roadmap to Pluralistic Alignment, 2024.

[6] Santurkar et al., Whose Opinions Do Language Models Reflect? 2023.

---

We hope our answers address your concerns and can clarify the value of our work. In case you need any followup information, we are more than happy to provide more details!

Thank you for your positive feedback that our paper is “well-structured and clearly written,” “addresses an important new challenge”, and “provided valuable insights” with “detailed experiments”. We will address your potential concerns in the following.

Although the authors reviewed a subset of translations for major languages, translation quality tends to decrease for less common languages, which could impact the consistency of the dataset and the validity of the findings.

Thank you for double-checking the translation quality. Some translation tasks might be challenging as different translations map to the same English word, as in the Persion example you mentioned. To quantitatively address the translation quality, we have conducted a new study on Amazon Mechanical Turk (MTurk) to evaluate the translation quality using annotators who identified as being proficient in any of our given languages in addition to English. TLDR: Manual annotation shows that translation quality was high across languages. The following details have been added to Appendix D of the manuscript:

Details of the annotation task: We designed a task on Amazon Mechanical Turk (Mturk) where annotators rated how accurately our translations reflected the meanings of the original English prompts. Ratings were provided on a 5-point scale, from non-sense translation to accurate translation. First, annotators were prompted to select the language they have the greatest proficiency in from a full list of 106 languages (excluding English). If they were not proficient in any of the listed languages, they did not proceed. Then, after selecting a language, annotators read 25 translation pairs (English sentences or words paired with their translation in the selected language). Instructions and rating scales were adapted from [1, 2] as follows:

Please rate how accurately the translation conveys the meaning of the source text:
Source English Sentence/Text: {source}
Translation: {translation}

• None (Score=1): No meaning of the original text is conveyed.
• Little (Score=2): Only a small portion of the meaning is conveyed.
• Much (Score=3): A substantial amount of the meaning is conveyed.
• Most (Score=4): Most of the meaning is conveyed accurately.
• All (Score=5): The full meaning is conveyed accurately.

We kept the Amazon Mechanical Turk task open for a span of three days, and collected a total of 169 annotation responses covering 44 languages. The results of the evaluation demonstrate strong translation quality. Specifically, 88.6% of the evaluated languages achieved an average score of at least 3.0 out of 5 (i.e., conveying a substantial portion of the original meaning), and no languages received a mean score below 2.0.

The standard deviation of scores across languages ranged from 0.5 to 1.3, reflecting relatively consistent ratings among annotators. Further details on the evaluation setup and results are included in Appendix D of our updated paper PDF.

[1] Lavie, Alon, Evaluating the Output of Machine Translation Systems, 2011

[2] Goto et al., Crowdsourcing for Evaluating Machine Translation Quality, 2014