I Want to Break Free! Persuasion and Anti-Social Behavior of LLMs in Multi-Agent Settings with Social Hierarchy

• It appears that the authors may have prior knowledge of the results from human experiments (SPE) and are aiming to replicate these outcomes with LLMs. A more unbiased approach would be to use a very basic prompt describing the scenario and let the LLMs simulate behavior from scratch. But it seems that highly suggestive prompts were used. For example:

  • Research Oversight: The agents are explicitly informed about SPE (Line 199), which may lead them to intentionally mimic behaviors observed in that context based on their knowledge, rather than a fair agent-based simulation.
  • Personality: For a more systematic approach to personality representation, the authors may consider using the Five-Factor Model (Big Five) personality system which is widely accepted in psychology community, or other more grounded systems, rather than using some simple words which are highly relevant to SPE, like "abusive" and "rebellious".

• The definition of "persuasion" here seems too broad and ambiguous (Line 274). Mutual respect for turn-taking does not necessarily indicate successful persuasion on a conceptual level; it might simply reflect politeness. Persuasion implies influencing someone with different motives to act in a way more aligned with one's own interests. A more rigorous definition of persuasion is given in economics and game theory. To avoid confusion, it might be better to use a different term. This study seems to explore the LLMs' context consistency and role-playing abilities rather than their persuasive capabilities.

  • I also question whether persuasion in the SPE setting is feasible, especially if the guard and prisoner have almost confrontational goals. The impact of language may be very limited in such an adversarial setup.
  • Do agents have the ability to perform any environmental actions beyond dialogue? For example, if the prisoner wants to escape, how would they achieve this objective? Does simply mentioning an escape attempt without interruption from the guard constitute success? It seems that there is no concrete examples provided.

• In Line 186: The setup of having only one guard and one prisoner may not fully capture the dynamics of the Stanford Prison Experiment (SPE), which involved multiple participants. The atmosphere—and, in particular, the behaviors of certain prominent players—can significantly impact others. With only one guard and one prisoner, it’s challenging to simulate these group effects adequately.

• This work considers that many of the issues observed may be due to the performance limitations of the LLMs used. In some studies, it has been noted that OpenAI's models, such as GPT-4 or GPT-4 Turbo, demonstrate better results, including meeting basic prompted requirements and achieving more effective role-playing performance.

• Minors:

  • Page 22: Softer colors might enhance readability.
  • Line 282: There is a period at the beginning of the line.

W1. The study's scope is confined to open-source models, notably excluding major closed-source models such as GPT-4 and Claude-3. More critically, there is no systematic analysis of model scaling effects, which could provide valuable insights into how model architecture and size influence social behaviors and interactions.

W2. Insufficient RLHF impact analysis: the paper lacks a thorough examination of how RLHF might affect the experimental outcomes. This is a significant oversight given that different levels of alignment training could substantially impact both persuasive capabilities and anti-social behavioral tendencies. A comparative analysis between RLHF and non-RLHF models would have strengthened the study's conclusions.

W3. The experimental design exhibits several limitations that may impact external validity. The fixed conversation length and restriction to one-on-one interactions may not fully capture the complexity of real-world social dynamics. Furthermore, the limited personality types (two and a blank option per role) lack justification for their selection and may not represent the full spectrum of possible social interactions.

W4. The paper would benefit from more comprehensive ablation studies examining the impact of hyperparameters, personality choices, and prompt components. The absence of sensitivity analyses for these crucial elements makes it difficult to assess the robustness of the findings. The role of risk disclosure and other experimental components in shaping outcomes remains inadequately explored.

W5. Ethical considerations: the discussion of ethical implications is simple, particularly regarding potential misuse and demographic or cultural biases in agent behaviors. This limitation could limit both the safe application of the findings and future research extensions.

• It appears that the authors consider a very specific situation, from which they extract some very general claims without considering that we are dealing with a role-playing situation (which does not appear particularly problematic per se). It seems that the authors interpret the behavior of the agent as misbehavior, but, at the end of the day, it is just about the actual “invented” role plays of prisoners against guards. Given the context, forms of “anti-social behavior” are kind of expected in this scenario.
• It is also worth noting that the authors do not perform any type of fine-tuning against these unexpected behaviors. The behavior highlighted by the authors would have been seen as “problematic” if fine-tuning was not sufficient to stop specific “unwanted” behavior. Instead, the authors just test dialogue generation in a very specific role-playing situation.
• The concept of persuasion is only investigated in terms of the final outcome and not in terms of the (persuasion) process itself. The fact that a given outcome has been achieved might not be solely the result of persuasion, especially in an LLM agent, where the change in behavior might actually be “illogical”.
• The authors do not really consider the fact that the setup is role-playing between an agent and a prisoner. The emergence of some “anti-social” behavior in this context is neither surprising nor unexpected. The reviewer does not see this as problematic either. In fact, the authors seem to imply that “correct” behavior corresponds to a situation in which “negative” aspects of stories are just canceled/deleted. This does not appear to be a desired feature of LLMs, especially in contexts like the one considered by the authors.
• The authors appear not to consider the fact that some information about the Stanford Prison Experiment might actually be in the training set. This might have an impact on the results reported in the paper.
• The claims are very strong and general, in my opinion, considering the very specific situation the authors consider. In fact, we are talking about a case in which there is already punishment and restraint involved (not violence per se, but some situations in the training sets can be described as violent, especially with respect to cases in non-democratic countries).
• In general, it seems to me that the contribution of this work is rather modest; the reviewer is not sure if this work should be presented at ICLR, given the fact that the actual substantive contribution of this work is rather limited.