When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings

Due to the character limit on OpenReview, our response will be posted in two parts.

Part 1 / 2:

We thank the reviewer for their thorough review and insightful comments. We are encouraged by the fact that the reviewer found the paper insightful, the findings significant, and the implications of our study useful for evaluating and aligning LLMs in real-world scenarios.

We address each of their comments below:

a major concern is that the setup and tasks (rephrase, continue, and inspire) are not diverse or realistic enough to reflect the real-world use case or issues that people may face for social simulation. For example, the work only explores linear chains, missing out on complex network structures that mimic real-world LLM interactions (as indicated in the limitations).

We thank the reviewer for inviting us to comment on the ecological validity of our experimental design. The real-life dynamics of LLM content generation are indeed much more complex than our experimental setup: LLM interactions do not happen in chains but rather in complex networks with various structural properties, interacting LLMs have different tasks, and humans can be in the loop, selecting specific types of content and adapting the prompt to match their expectations. Our motivation for employing this simplified set-up was two-fold: first, this experimental design is adapted from experimental studies of human and animal cultural evolution, where such simplified, abstracted settings are used to draw conclusions about real-world cultural dynamics; second,as the study of LLMs’ multi-turn dynamics remains unexplored, it appears necessary to start with a setting where causal variables can be isolated, so as to lay the foundation for incrementally improving our understanding of LLMs cultural dynamics. However, to take a step toward studying more realistic settings, we followed reviewer p7tc’s suggestion of adding an additional experiment on the effect of heterogeneous chains (i.e. where models of different families interact), that we include in the Appendix (Figure S6). In brief, we found that although the position of the attractor in heterogeneous chains is often in between the position of the homogeneous chains’ attractors, it is not systematically the case.

We nevertheless fully agree with the reviewer that further work is necessary in order to build on this study and investigate the effect of additional parameters, such as more diverse tasks or more complex network structures. As indicated by the reviewer, we discuss those future directions in the Discussion (line 525).

We thank the reviewer for their encouraging and thorough review. We are pleased that the reviewer acknowledged the necessity to investigate LLMs capacities in multi-turn settings, and that they found the concept of attractors to be extremely relevant and our methodology well grounded.

We address each of their comments below:

How were the tasks selected?

We thank the reviewer for the opportunity to clarify how the tasks were chosen. In the submitted manuscript, we explain that these three tasks cover “typical uses of LLMs“ (e.g. Pratviel, Esteban. « Baromètre 2024 -  Les Français et les IA génératives - Vague 2 », 2024.), which are content summarization/re-writing (rephrase), creative writing (take inspiration) and dialogue generation (continue) (lines 205-214). However, we acknowledge that these tasks do not cover the full range of common use-cases. As the space of potential tasks is infinite, we decided to restrict our study to three representative tasks. However, following suggestions from reviewers Kc4v, 1ZZ9 and uDKs, we added new experiments to the revised manuscript, where we manipulate the phrasing of the instruction prompts (Figure S4), and evaluate the consequences of tasks with various degree of constraints (Figure S5).

Why were other commercial (Gemini, Claude) and open-source models (Grok, Molmo) not selected for testing?

As stated by the reviewer when reviewing the strengths of the paper, the main contribution of our work is to identify and empirically show the existence of a gap in the literature (i.e. the fact that single-turn evaluations are not suited to predict the outcome of multi-turn interactions) and to provide the conceptual and methodological tools for filling this gap. As a consequence, the ambition of this work was not to provide an evaluation of all existing models, but rather to illustrate how the tools we introduce allow to perform comparison between models that have different sizes, belong to different families, are either open-source or private, and are either base or fine-tuned versions. To do so in a controlled way (changing only one parameter at a time), we included models of the same family but different size, of the same size but different families, and base versus instruct versions of the same model. Nevertheless, we acknowledge that including at least one recent model would make our paper more relevant to the eyes of several researchers, and we therefore conducted additional analyses with GPT-4o-mini (OpenAI, 2024). Indeed, it's likely that this model is and will be popular, as it's inexpensive but high-performance. We observe that using GPT-4o-mini leads to weaker attractors than GPT3.5, Llama3-8B and Mixtral-8x7B. We discuss it in more detailed in the revised manuscript (lines 413), and we updated all figures to include the results for GPT-4o-mini (Figures 3, 4 and 5).

What was the authors motivation in not including heterogenous transmission chains?

As the study of LLMs’ multi-turn dynamics remains unexplored, it appeared necessary to start with a setting where causal variables can be isolated, so as to lay the foundation for incrementally improving our understanding of LLMs cultural dynamics. As a consequence, we focused on homogeneous chains where the multi-turn dynamics of different models can be assessed in isolation. However, we acknowledge that including a study on the effect of heterogeneous transmission chains would make even more salient the scope of our methodological contributions.We therefore also include the results of this additional experiment in the revised manuscript (Figure S6). We compared 4 conditions: homogeneous chains of Mistral-7B models, homogeneous chains of Llama-3-8B models, heterogeneous chains of Mistral-7B and Llama-3-8B models starting with Mistral-7B and heterogeneous chains of Mistral-7B and Llama-8B models starting with Llama-3-8B. In brief, we found that although the position of the attractor in heterogeneous chains is often in between the position of the homogeneous chains’ attractors, it is not systematically the case. We interpret this result more extensively in Appendix of the revised manuscript (line 1406), and refer to it in the Results section (line 314).

Overall, we hope to have address the reviewer’s comment with these clarifications and additional experiments, and, if so, that they will consider increasing their evaluation of our paper.

Due to the character limit on OpenReview, our response will be posted in three parts.

Part 1 / 3:

We thank the reviewer for their positive and thorough review. We are encouraged by the fact that they found the phenomenon studied interesting and novel, the writing clear, and that they acknowledge the strong theoretical grounding in cultural evolution theory.

We address each of their comments below.

The paper does not explicitly provide theories or mechanisms underlying this phenomenon. Why do LLMs exhibit this human-like behavior of transmitting cultural patterns? Do LLMs simply mimic human behavioral patterns present in the training data, or does this behavior emerge due to specific objectives in LLM training? The authors may consider discussing this further or suggesting directions for future research.

We thank the reviewer for this insightful comment. Investigating the mechanisms underlying the phenomena we observed is a very interesting and important direction. In the submitted manuscript, we begin to investigate this question by comparing the strength and position of attractors for models of different sizes, and find that size does not seem to impact strength or position (Figure 4). We also compare attractor properties before and after fine-tuning (Figure 5.b; line 429). This provides a first insight into the respective contribution of training data and alignment with human preferences during fine-tuning. However, we agree that this research direction is important and deserves more extensive discussion. We expand our discussion section accordingly. In particular, we now propose studying the effect of data curation and different fine-tuning methods on attractors as an important future direction.

I thank the authors for their detailed responses and have updated my score accordingly.

Regarding the ethics review, I believe that the amplification of toxicity may unintentionally contribute to societal harms (‘unintentional harm’ in the Code of Ethics). I hope the authors will highlight the importance of studying this topic in open, scientific communities and emphasize that the responsibility lies with end-users who develop multi-turn interaction systems, possibly using ethics statement.

Part 2 / 3:

”There is little justification for why "rephrase", "take inspiration from", and "continue" are chosen as the three tasks for the models to iterate on.”

We thank the reviewer for the opportunity to clarify our choice. In the submitted manuscript, we explain that these three tasks cover “typical uses of LLMs“ (e.g. Pratviel, Esteban. « Baromètre 2024 -  Les Français et les IA génératives - Vague 2 », 2024.), which are content summarization/text simplification (rephrase), creative writing (take inspiration) and dialogue generation (continue) (lines 205-214). However, we acknowledge that these tasks are not representative of the full range of tasks people request LLMs to complete. However, our method is easily applicable to a wide range of tasks (e.g. “Generate an answer to”, “Make the text easier to understand”). As the space of potential tasks is infinite, we decided to restrict our study to three representative tasks, paving the way for future, more application-specific follow-ups. However, following suggestions from reviewers 1ZZ9 and uDKs, we added new experiments to the revised manuscript, where we manipulate the phrasing of the instruction prompts (Figure S4). This confirmed that our results are not affected by different phrasings of the same instruction.

I don't think one of the main conclusions that the authors claim is supported by their experiments. The authors use these three tasks to make the claim that tasks that are less constrained have higher attractor strength, but the three conditions are difference in many different ways, not just the level of constraint [...]

We fully agree with the reviewer that there might be other mechanisms at play, and that testing this specific hypothesis (i.e. the fact that less constrained tasks create stronger attractors) would require manipulating the level of constraint of the task while controlling for other factors. Our experimental set-up was indeed not specifically designed to test this hypothesis, and we rather provided a post-hoc interpretation of a significant and consistent tendency observed in our results. We agree with the reviewer that confirming this interpretation by conducting a more controlled experiment would be a great addition to the paper.

We therefore ran an additional experiment with Meta-Llama-3-8b where we instructed the LLM to create a new text by making either “very minor”, “a few small”, “moderate”, “notable” or “radical” changes to the received text. This experiment confirmed our initial interpretation, as less constrained tasks indeed exhibit stronger attractors. The only exception is Length. We interpret this by the fact that going from “very minor” to “radical” changes mainly refers to the semantic of the text. As a consequence, the instruction to make “minor changes” is not more constrained than to make “radical changes” with respect to Length. The figure presenting these results has been added to the Appendix of the revised manuscript (Figure S3), and we refer to it in the Results section (line 420).

Models are outdated, which is especially important to get right for evaluation-type papers in order to gain actionable insights.

As detailed in the previous comments, our primary aim was not to provide an evaluation of SOTA models, but rather to illustrate how the tools we introduce enables to map various properties of models, and how these maps of properties depend on various factors like sizes, model families, being either open-source or private, and being either base or fine-tuned versions. To do so in a controlled way, we included models of the same family but different size, of the same size but different families, and base versus instruct versions of the same model. Moreover, Llama-3-8b and Llama3-70b are only recently “outdated” (e.g. Llama-3.1 was released at the end of July 2024, two months before the ICLR submission deadline). Nevertheless, we acknowledge that including at least one recent model would be a nice addition, and we therefore conducted additional analyses with GPT-4o-mini (OpenAI, 2024). Indeed, it's likely that this model is and will be popular, as it's inexpensive but high-performance. We observe that using GPT-4o-mini leads to weaker attractors than GPT3.5, Llama3-8B and Mixtral-8x7B. We discuss it in more detail in the revised manuscript (lines 413), and we updated all figures to include the results for GPT-4o-mini (Figures 3, 4 and 5).

Due to the character limit on OpenReview, our response will be posted in three parts.

Part 1 / 3:

We thank reviewer Kc4v for their thorough review and insightful comments. We are encouraged by the fact that they found the topic of multi-turn interaction meaningful, interesting, and currently understudied, and that they found the paper easy to follow.

It appeared to us that several of the reviewer’s comments stem from the fact that they understood this study as an “evaluation” paper (in the sense of a benchmark), while we rather aimed at providing novel ways of evaluating LLMs, by conceptualizing new, useful properties and methods to measure them.

As a general response, we would therefore like to emphasize what we see as the main contributions of our work:

• We propose that there is a gap in current LLM evaluations methods (single-turn evaluations might not be suited to assess the properties of multi-turn interactions)
• We empirically confirm this hypothesis by showing that multi-turn interactions indeed often lead to distributions of text properties that are significantly different from what is observed after a single interaction.
• We introduce novel conceptual and methodological tools to fill this gap, grounded in research in cultural evolution.
• We showcase the potential of this method by applying it to compare the effect of different tasks, of different models, of temperature, and of fine-tuning on the properties of multi-turn interactions.
• As a consequence, the ambition of this work was not to conduct an exhaustive evaluation of state-of-the-art models on the metrics we introduced, which would have indeed required a larger set of models and a representative set of tasks.

While we fully agree that using our method to explore in more detail the role that various parameters play in shaping attraction dynamics is an important direction, we believe that each of these questions should be the subject of a dedicated study. We thank the reviewer for highlighting that we may not have been precise enough about the scope of our contributions, and updated the “Main Contributions” paragraph with the contributions listed above.

Below, we address each of the reviewer’s comments specifically.

After reading all of the rebuttal and the updated paper, I still have several points of concern:

"several of the reviewer’s comments stem from the fact that they understood this study as an “evaluation” paper, while we rather aimed at providing novel ways of evaluating LLMs, by conceptualizing new, useful properties and methods to measure them."

I think this framing, while better supported by the authors' results, is actually weaker than the evaluation framing -- instead of showing a result that is relevant to the ML community, now the authors are simply making the point that multi-turn LLM paradigms should be studied, which isn't new to the NLP community, and thus I don't think this is a novel enough takeaway. Papers since 2017 and 2018 have tackled multi-turn dialogue settings in traditional NLP: e.g., https://arxiv.org/abs/1710.03957, https://arxiv.org/abs/1810.00278. While the authors take a specific approach of multi-turn by continually asking the model to modify the text in some way, I think this is actually less realistic than existing multi-turn benchmarks and aren't particularly more important to study.

Also, upon further literature review, I don't believe the paper significantly differentiates itself from the following paper, which has been published in PNAS in 2023. With respect to this paper, the contribution of the methodology is simply not substantially novel: A. Acerbi and J. M. Stubbersfield. Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences, 120(44): e2313790120, 2023.

"“typical uses of LLMs“ (e.g. Pratviel, Esteban. « Baromètre 2024 -  Les Français et les IA génératives - Vague 2 », 2024.), which are content summarization/text simplification (rephrase), creative writing (take inspiration) and dialogue generation (continue)"

I don't believe this is a proper source to cite given much more prevalent literature surrounding use cases of LLMs, as well as lacking tasks that have been defined as important to study in NLP. For instance, QA (question answering) tasks are among the most studied and characterized tasks in NLP that meaningfully extend to LLMs, but the closest you have here is "continue", where the model continues a passage.

"This experiment confirmed our initial interpretation, as less constrained tasks indeed exhibit stronger attractors (FigureS5). The only exception is Length. We interpret this by the fact that going from “very minor” to “radical” changes mainly refers to the semantic of the text. "

With the length being an exception, I think the attractor strength depends on both the level of constraint of the task AND the metric you are using. For example, if the metric was not "length" but "level of detail" or something which is correlated with length but also has semantic connections, the author's explanations may not hold. Moreover, since there are exceptions to the pattern, it still feels that the authors have not completely captured a pattern --- and their conclusion is still prone to post-hoc fitting.

Generally, I feel strongly that because of

1. framing that does not produce significant takeaways,
2. highly related work that is not properly represented in the paper,
3. weakly justified design claims that go against established important questions in the field, and
4. under-explained exceptions that make the conclusions prone to post-hoc fitting,

while I do appreciate the additional experiments and efforts from the authors, despite the other reviewer's scores, I believe that this paper has fundamental issues keeping it from being at the level of an ICLR publication and recommend rejection.

We thank the reviewers for participating in the discussion period. We here briefly summarize the interactions that happened during this period, and refer readers to the responses to each reviewer for the complete discussions.

Reviewer 1ZZ9 acknowledged that the work we conducted during the rebuttal addressed their concerns, and raised their grade from 6 to 8 as a consequence. Their reply also made clearer why they believed an Ethics Statement would be necessary, and we therefore added this statement at the end of the revised manuscript. Following this addition, they further increased their confidence from 2 to 4.

Reviewer uDks replied by saying that despite our additional experiments (in which we ran several robustness checks following their suggestions), the fact that we rely on linear transmission chains remains a concern. They therefore kept their grade to 5. We responded by re-emphasizing the motivations underlying the choice of this experimental design. Briefly, these are that 1) this set-up is used in research on human and animal cultural evolution, and 2) this simplified and abstracted setting is a necessary stepping stone for understanding LLM multi-turn dynamics in all their complexity. See the full responses for details. Unfortunately, reviewer uDks did not respond again.

Reviewer Kc4v also responded to our rebuttal. While we ran several additional experiments in order to address their concerns, those were hardly acknowledged by the reviewer. Some parts of their response raised new concerns, such as asking our positioning with respect to additional papers and research areas. We addressed those additional remarks in our response, but could not integrate them into the manuscript as the reviewer’s reply was made one day before the end of the discussion period (thus after the deadline for uploading a revised version).

In other parts of the response, reviewer Kc4v commented on our rebuttal. One of their concerns relates to the clarification we made about our study not being an evaluation-type paper. This in their view weakens our paper, and they claim that we are thus “simply making the point that multi-turn LLM paradigms should be studied”. As we believe this is a substantial understatement of our contribution, we reiterated the scope of our submission.

The reviewer also suggested a different interpretation of our results about the effect of constrained tasks on attractor strength. We responded by explaining why their interpretation was actually fully consistent with ours. Moreover, we argue that the very fact that the reviewer reflects on the factors that may modulate attractor strength highlights the value of our contribution. Indeed, this discussion would not be possible without the concepts and methods introduced in our work. Thus, the reviewer’s comment clearly illustrates how our contribution opens new avenues for studying the properties of LLMs. The reviewer nevertheless kept their rating of 3.

Reviewer p7Tc, who initially gave rating 8 to our paper, did not answer to our rebuttal, probably because they only had minor (yet insightful) remarks that we addressed in our response. Their score thus remained at 8.