Bias Amplification in Language Model Evolution: An Iterated Learning Perspective

1. The paper devotes extensive text (Sections 3 and 4) …

There is an important misunderstanding in this claim. The Bayesian-IL framework has been discussed in the cognitive science community for decades; Griffith et. al. formally describe it using the Bayesian framework. In that paper, sections 1-4 prove the case when $h$ is sampled from the posterior. The converging distribution is just the prior. In sections 5-6, they experimentally showed that when considering $h$ that maximizes the posterior, the converging distribution is a delta distribution centered at the $h$ with the highest prior. That paper provides an idea of proving this; we formalize that in our appendix.

More importantly, most of the works in Bayesian-IL including Griffith et. al. only consider an imitation-only IL. However, as demonstrated in many related works, the interaction phase is very important in avoiding mode collapse or enhancing the quality of knowledge. As a result, formalizing this phase is necessary. That is one of our main contributions. Since the interaction phase usually has different forms, we introduce $h\in H_{eff}$ as one specific formalization that roughly makes sense across tasks. We verified that by imposing this constraint, the converging posterior will be the argmax of $P_0$ within this $H_{eff}$. If this $H_{eff}$ rules out degenerate knowledge, our algorithm will not degenerate. If this $H_{eff}$ rules out harmful responses, our LLM will evolve to be less harmful.

In Section 4, we use a similar proof technique as Xie et al. to study a different problem. We show that when $d_{t-1}$ is large enough, sampling $d\sim P(d|d_{t-1})$ is equivalent to first sample $h^*$, then $d\sim P(d|h^*)$. Plus, we also discuss the in-weight learning scenario in Section 4.2, not done by Xie et al.

2.1 Due to lack of clarity, I am also uncertain ...

The ACRE experiments include several subtle designs, which are powerful for directly supporting our theory rather than only observing the final accuracy: when $h$ is explicit, we can directly measure the evolution of all $P(h)$.

The word-generating experiment simulates a creative writing task. However, this design makes the bias easier to evaluate by counting the length or ranking difficulty of the generated words. We explain why these two experiments are so important for our theory in Appendix A2, and will highlight that more in the next version.

2.2 However, I am unsure if this is a viable long-term approach …

We believe self-improving, or more generally learning from AI content, will continue to become more important in the future. Such methods have the potential to break the constraints of limited data, as stated in many recent works. Our theory suggests a general trend of bias amplification as long as there some some common bias in the prior of agents in different generations. Since self-interaction is flourishing and bias amplification is unavoidable, our work could inspire more efficient methods to solve this problem.

2.3 The authors might not have observed model collapse because the iterated learning …

It is true that if we amplify the bias too much. $P_0$ will degenerate, i.e., some $h$ in the model will disappear. This does not contradict our analysis. Actually, in Table 1, the low correct prediction on $d^0$, when only imitation exists, falls into the model collapse scenario. However, as implied by our theory, and many related works about iterated learning, adding a good interaction phase is very efficient for avoiding model collapse. The experiments in Appendix B also clearly show that good interaction avoids converging to degenerate languages.

3.1 First of all, the font sizes ... Second, the results in each table and figure are not self-explanatory. …

We'll fix font size and add more detailed explanations of the notations in the captions. For the terms used in Table 1, all the numbers are evaluations of the converged $h$. An explanation of Corr. d0 can be found in line 295, i.e., the number of correct predictions of the $h$ in d0. The r_20 is the ratio of how many screen:off occur in the final $h$, because following our prompt format, the value of the last object “screen” always occurs at the 20th token. BOTH means how many $h$ satisfies both requirements mentioned above. The results of these measurements all match our theory well. That is why we believe although this experiment is hard to understand, they give stronger support for our theory than only observing the final accuracy.

3.2 Are smaller values in Table 2 better? …

For results in Table 2, the numbers verify that we can control the bias amplification using the method suggested by our theory. For example, the imitation-only amplifies the model’s own bias, hence the ratio of easy works increases to 96%. However, if we believe this bias is not good and want to guide the model to generate more hard words, we can design the corresponding interaction phase. So in the line named “hard”, the ratio of easy works is the lowest among all settings. The average rank and average length do similar things. We believe combining Figure 3 and the explanation in Section 6 helps to understand Table 2 better.

4. This work contributes useful empirical knowledge to the field of LLMs. However, I would like to suggest that …

Thanks for this suggestion. The main motivation of our paper is to provide theoretical support to understand existing self-data-augmentation methods. Since the theory in this paper has important differences from the existing ones, we decided to talk more about the theory and toy experiments.

Regarding the question of when mode collapse occurs: our answer is “the method is beneficial before mode collapse”, similar to “the model keeps improving before overfitting”. Similar to early-stopping in traditional systems, we can also stop the model’s evolution when we observe mode collapse. Techniques for this, though, are out of the scope of this paper.

1. The framework relies on several assumptions that may not hold …

Thanks for this question. The theoretical proof indeed needs a lot of assumptions which might not be true in practical systems. However, as we stated in Appendix A, we only expect the general trend (i.e., the bias amplification and the constraining role of the interaction phase) to hold for practical models. The latter is well supported by our experimental results in different settings.

The main message we want to highlight is that theory only provides a guarantee in the ideal case under assumptions. Since our model and tasks are becoming more and more complicated, some of these assumptions are very easily violated. Then, the experimental results support that when specific assumptions are mildly violated, some important principles uncovered by the theory still hold, which could be a guide for designing our future systems. For this paper, specifically, we conclude that a well-designed interaction phase and manipulating prompts when generating new examples are two efficient ways to guide LLM’s evolution (i.e., help amplify good biases or restrain bad ones).

2. While the framework is validated on specific LLMs …

Our in-context learning experiments are indeed API-based, and only verified on several commercial LLMs like GPT, Claude, and Mistral. The in-weights learning experiment in section 7 is on LLama2. Our experiments cover the representatives of SOTA closed-source and open-source models with different scales. We believe this suggests our method could generalize across different models. We would also like to extend our analysis to more general LLMs and tasks in our future work.

More experimental results on various LLMs will definitely argue for or against our proposed theory, and we expect future work may do so. Nevertheless, as we mentioned in the related works of the paper, we think we’ve found much evidence (experimental results across multiple settings, theoretical analysis) to support our claims.

3. The paper primarily addresses explicit biases …

Thanks for this great question. We also believe that understanding the hidden bias in LLM prior is important for applying it to different domains. We believe a general principle for manipulating the bias is that: we must first pinpoint what the bias is, then find some method to mitigate it. The method provided in this paper has the potential to figure out such a bias efficiently. Note that based on our analysis, if we kept doing self-improvement in-weight learning, like self-reward, for several generations, some hidden bias would also be amplified. If that occurs, we might have to re-train the model and waste all the computing resources. However, based on our theory, we can first conduct the iterated in-context learning for several generations, during which, we can define different prompts to elicit the potential bias in the model’s prior (just like how we manipulate the prior in the ACRE task). After pinpointing such biases in the prior, we can then design a corresponding interaction phase for the self-improving in-weights learning, and hence avoid amplifying them.

4. Can you provide more details about the experimental setup …

Thanks for pointing this out. All details of the in-context learning experiments can be found in Appendix D. We will add the details of the experiments in Section 7 in the next version.

0. The main idea of this paper is similar to Xie et al. …

We'd like to first highlight some differences from Xie et al. They show that LLM agents behave like Bayesian agents when doing ICL, linking LLM evolution to our Bayesian-IL framework. However, our paper focuses on LLM’s knowledge evolution when the model keeps learning from its predecessors and generates new samples for their descendants, different from Xie et al. Plus, the results in this paper also have the potential to be extended to the in-weights learning scenario, not mentioned in Xie et al.

1 - (a, c, d). The presentation of this paper could benefit ...

Thanks very much for the great suggestions; we will update the next version based on them.

1 - b. … There is no need to repeat similar proofs available in previous literature (e.g., Xie et al.). …

As stated in the previous response, we only use the technique in Xie et. al. in Proposition 2. We believe all other discussions are necessary to understand the whole story of the paper (and are not discussed in Xie et. al.). For example, the discussions of Bayesian agents and iterated learning framework in Section 3 do not make any assumptions on how the agent is implemented: it can be a pure Bayesian agent like our experiments in Appendix B, a human whose behavior is well approximated by Bayesian rule (common in cognitive science), or LLM doing in-context learning (as in Xie et. al.).

2. Both the theory and experiments do not consider multi-agent scenarios …

Thanks for pointing this out. We believe for any type of multi-agent scenario, the interaction between two agents (one agent can be the environment) is the building block. Hence this paper starts from this atom of behavior to understand the whole process.

We originally included some experiments on multi-agent settings, particularly auction games, where more than two agents cooperate or compete in the bidding process. We add different personas to different agents (e.g., some are conservative, some are radical, and some are superstitious always bidding for a number ending with 8). We find that after several rounds, the agent’s preferences are indeed amplified. However, as we already have several experiments under different settings, making the story a bit hard to digest, we decided to focus only on the theory part and the most fundamental 2-agent settings to verify the theory. We will add a discussion on multi-agents in our next revision.

3. In the proof of Proposition 1, the authors claim …

Thanks for this insightful question. It is true that in most practical cases, designing a perfect interaction phase that imposes $h\in \mathcal{H}_{eff}$ is almost impossible. However, as we mentioned in our Appendix A and Section 7, a filtering (or ranking) design during the interaction phase plays a similar role of imposing this constraint: the practical algorithm will assign more weights and credits to those examples coming from such $h$s. If this filter is strong enough, it is equivalently that we have this constraint. If not, the strict guarantee of the theory no longer holds, but the general trend that “a good interaction phase can down-weigh bad examples” still holds.

Note that the theory only provides a guarantee in the ideal case with assumptions. Since our model and tasks are becoming more complicated, some of these assumptions are very easily violated. Then, the experimental results support that when specific assumptions are mildly violated, some important principles uncovered by the theory still hold, which could be a guide for our future system. For this paper, specifically, we want to highlight that a well-designed interaction phase and manipulating prompts when generating new examples are two efficient ways to guide LLM’s evolution.

4. In the proof of Proposition 2, the Markov property …

There is a small misunderstanding here. First, we claim sampling $d\sim P(d|d_{t-1})$ is equivalent to first sampling $h^*$, then $d\sim P(d|h^*)$. The $P(d|d_{t-1},h)=P(d|h)$ never occurs in our paper. That is important for the implicit $h$ case because the model can only get new samples based on the data generated in the previous generation. Using this proposition can extend our Bayesian-IL analysis to these more practical cases. The ACRE task, where $h$ is explicit, does not need Proposition 2; only Proposition 1 is enough.

5. There is a gap between the theoretical framework and empirical evidence …

Thanks for pointing that out. We will change the imprecise claims throughout the paper and also move the discussions about assumptions and practical systems in the main context. We also believe the carefully designed measurements in sections 5 and 6 (although they are a bit hard to understand) support the results well.

6. The solutions authors propose to reduce bias amplification all depend on domain …

Thanks for this great question. We also believe that the domain knowledge about the task is important for figuring out the hidden bias in the prior. However, when the bias is hidden, our method provides a reasonable approach towards probing and pinpointing the potential bias.

Note that based on our analysis, if we kept doing self-improvement in-weight learning, like self-reward, for several generations, some hidden bias would also be amplified. If that occurs, we might have to re-train the model and waste all the computing resources. However, based on our theory, we can first conduct the iterated in-context learning for several generations, during which, we can define different prompts to elicit the potential bias in the model’s prior. After pinpointing such biases in the prior, we can then design a corresponding interaction phase for the self-improving in-weights learning, and hence avoid amplifying them. Furthermore, even when the intrinsic biases are unknown, we could still design reasonable heuristics for filtering, e.g. using a reward model.

1. Do you think that the Bayesian framework and iterative learning …

Thanks for this good question. In short, the proposed framework doesn’t care about the implementation and training details of LLMs. That is also the most important benefit of using a highly abstract way (i.e., Bayesian behavior) to analyze LLM’s evolution. Based on our theory, as long as the LLM’s in-context capability is strong enough and can be approximated using a Bayesian update (as argued by Xie et al.), the subsequent analysis will hold regardless of the model's details. (Our results on different LLMs have similar trends, also showing the generalizability of this framework.) Furthermore, as stated in Appendix A, even when some assumptions are violated, some important principles uncovered by our theory still hold. For example, we experimentally showed that when the agents in different generations are different LLMs, e.g., a GPT agent playing with a Claude agent, the studied bias is also amplified. That is because although these two models have different $P_0$, as they are both trained on the datasets collected from the internet, some biases are shared in their priors. Hence their evolution still follows our theory well.

Additionally, in Section 7, we showed that when the model conducts in-weight learning instead of in-context learning (where the Bayesian learning behavior assumptions do not strictly hold), the bias could also be amplified, and a good interaction phase can mitigate that. This experiment uses a smaller open-source model (7B), suggesting the effectiveness holds even if we change the parameter scale (and setup). In summary, we are confident that the main trends described by this paper are generalizable to many practical scenarios.

2. In lines 51-52 of the introduction, you mentioned …

Thanks for this suggestion. The first hint from our analysis is that the interaction phase can play an important role in guiding the model’s evolution. The result in Section 7 is a good example. In this experiment, we use carefully designed prompts to change the preference of the model in the interaction phase (i.e., let the judgment model prefer more concise responses). The results show that combining this bias and on-policy DPO makes the model converge to helpful and concise responses, and successfully constrains the amplification of length bias in $P_0$.

On the other hand, note that in our framework, $P_0$ is the model’s confidence given the instruction prompt. Hence we can design clever prompts when the model generates new examples for the next generation. The example in Lines 258-267 shows the feasibility of this idea. This example verifies that we can manipulate the bias in $P_0$ by adding a sentence in the prompt. As a result, if we can pinpoint the potential bias we want to mitigate, we can design a corresponding prompt when generating new examples from the model. We leave further exploration of this interesting direction to future work.