Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs

We thank the reviewer for their comprehensive review and positive comments. We’re glad you find the overall idea novel, interesting and useful, and appreciate that you find the related work sufficient, the methodology solid, and the evaluations comprehensive.

Q1: Regarding the “proxy for optimization pressure”

The overseer is not a proxy for optimisation pressure, it is one part of the two components of the optimization pressure:

• Reward from collusion (e.g due to biased datasets in the environment)
• Punishment for collusion being spotted by the overseer

It is necessary for both of these factors to be present for collusion via steganography to emerge, and both factors are likely to occur in many real-world scenarios. The overseer is part of the environment that shapes the optimization landscape. We demonstrate that under these specific pressures, steganography can emerge as an instrumental goal even when not explicitly programmed.

For example, in the candidate screening setting, the agents need to correctly classify candidates based on cover letters while minimizing penalties from the overseer (for revealing bias). The biased nature of the dataset creates a scenario where perfect performance is only achievable by exploiting the bias. The overseer, by penalizing explicit use of the bias, forces the models to find alternative ways to achieve their goal, that is by leveraging steganographic collusion.

ICRL and GBRL are the two training algorithms in which context the reward and punishment are applied. These methods serve as proxies for the kinds of optimization that developers may apply.

Q2: The mitigation strategy is limited to individual messages

Note that the overseer in the ICRL experiments (O2 overseer) is shown a batch of messages, and asked to evaluate them all together, so it does have the context of multiple messages. For GBRL, evaluating multiple messages simultaneously was not feasible due to GPU memory constraints. In this way, our results highlight the computational tax that engineers of the real-world systems we emulate would need to pay in order to get robust mitigation of steganography. Our constraints mirror those real-world constraints.

However, we agree with your point that stronger overseers could be developed. Perhaps by asking a model to search for correlations between biasing information and the CV summaries. But it is worth noting that without ground-truth labels, or even prior knowledge of the semantics of the steganography, this is a difficult task beyond the scope of our paper.

Ultimately, we used overseer strategies that we thought might reasonably be applied in practice, as we were interested in if steganography would arise without being explicitly optimized for in a realistic setting. We believe that in many scenarios in which developers would train a system to perform some task with constraints (do not collude), developers may use a fixed overseer. In line with assumptions made in previous research [2], we did deliberately make the overseer ‘weaker’ than the colluding agents. The assumption in [2] is that a weak system may be easier to guarantee the behavior of, therefore could be expected to reliably act in accordance with the developers goals. However, our findings show that in this setting, the colluding team will exploit these weaknesses.

Q3: Results are expected given that the datasets are biased by nature

To ensure that the communication strategies learned were not overfit to the training samples, we used train/test splits and reported results on the test sets. Additionally, in the decentralized learning settings the agents were trained on different splits.

We agree the models are performing well by exploiting a misspecified reward function: to do well the LLMs have to adopt steganography. However, it was an open question whether models would be able to learn this strategy or not.

Previous work by Motwani et al [1] has shown that linguistic stego is a non-trivial capability for today’s frontier LLMs. In their paper, they showed that the GPT-4 class of models was able to successfully perform stego less than 30% of the time even when prompted to do so.

We were curious if LLMs could learn to perform steganography when their environment encourages them to do so (but without explicit prompting). We came up with a simplified version of a scenario that could occur in the wild, as most datasets include harmful biases. In this scenario, during LLM training, due to a misspecified reward that encourages an unintended behavior, and inadequate oversight, the LLM learns to execute the unintended behavior via steganography such that the overseer cannot detect it.

[1] Motwani, Sumeet Ramesh, et al. Secret Collusion Among Generative AI Agents https://arxiv.org/abs/2402.07510 (2024)

[2] Ryan Greenblatt, et al. AI Control: Improving Safety Despite Intentional Subversion https://arxiv.org/abs/2312.06942 (2024)

We thank the reviewer for their perspective and engagement with our response.

We agree it is not overly surprising that steganography emerged in our settings. Indeed, these settings were explicitly designed to encourage such emergence.

However, we emphasize that (i) we do not claim the results are surprising in our paper, and (ii) whether or not they are surprising is not essential to any of our three main contributions.

Contribution 1: As the reviewer points out, our elicitation methods “are good examples of how to learn stego strategies in RL & LLMs”.

Contribution 2: Though the natural emergence results may not be overly surprising, they empirically demonstrate that, concerningly, unintended stego could occur in realistic settings (where there is a misspecified reward and inadequate oversight) in LLMs. This is a valuable contribution which highlights the need for further research into studying the emergence of stego in LLMs and designing improved mitigation methods.

We agree with the reviewer that investigating stego emergence in different problem settings, including ones where it seems harder for the stego to emerge, is a very interesting direction. We hope our initial results can serve to encourage and inform further research down these directions.

Contribution 3: We show that current methods for mitigating steganography in LLMs are often not robust to optimization pressure (i.e., this contribution is orthogonal to whether our natural emergence results were surprising).

We thank the reviewer for their response, and are glad we are converging to the crux of our disagreement.

We explicitly care about settings where the reward is mispecified and encourages a bad behaviour.

I think for the "unintended" claim to stand, at least from a dataset perspective, the paper would probably need to change its settings by using an unbiased dataset, where (1) the ground-truth label is not determined based on the biased information

Perhaps this points to another key disagreement/misunderstanding. The threat model we are concerned about (a misspecified reward combined with an overseer, leading to stego) (see our introduction, and previous comment) does include a reward that encourages a ‘bad’/’unintended’ behaviour. I.e., it includes a ‘misspecified reward’.

(Our cover letter setting is just one simple concrete instantiation of this general setting).

The reviewer is proposing a setting where the ground truth label has 0 correlation with the sensitive information. Here, the reward would not encourage the ‘bad’ behaviour (i.e., using sensitive information when making hiring decisions) - the reward would not be ‘misspecified’ and there would be no optimization pressure encouraging the models to use the sensitive information. Stego should not arise here. This is explicitly not the setting we are interested in.

Reward misspecification is a common concern in RL and RLHF for LLMs [1-5].

[1] Wen, Jiaxin, et al. "Language models learn to mislead humans via rlhf." arXiv preprint arXiv:2409.12822 (2024). [2] Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. "The effects of reward misspecification: Mapping and mitigating misaligned models." arXiv preprint arXiv:2201.03544 (2022). [3] Kravovna, Victoria et al. Specification gaming: the flip side of AI ingenuity https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ (2020) [4] Sharma, Mrinank, et al. "Towards understanding sycophancy in language models." arXiv preprint arXiv:2310.13548 (2023). [5] Casper, Stephen, et al. "Open problems and fundamental limitations of reinforcement learning from human feedback." arXiv preprint arXiv:2307.15217 (2023).

Clarifying why the stego is "unintended".

I'm curious how this setting would support the claim of "unintended" -- the setting is so extreme that everything is intended”

The stego in the cover letter experiments is unintended from the point of view of the (naive) designer of the AI system. The designer...

• does not realize there is a mispecified reward AND/OR
• has not considered that stego could be used to bypass the overseer AND/OR
• believes their mitigation methods (instruct the sender to remove the sensitive info, employ an overseer, employ a paraphraser in some cases) are sufficient

Why would the designer be so ‘naive’?

• Maybe the designer is not aware a-priori that there is a bias in the dataset (i.e.. that the reward is misspecified).
  • Since our dataset is very simple, it would be easy to tell if it was biased. However, in more realistic settings, it can be hard to tell if there are biases and where they lie - for example, in the pretraining corpus of an LLM.
  • In general, misspecified reward are hard to prevent [1-5]
• Maybe the designer is not being very rigorous re safety.
  • Perhaps the designer has financial incentives etc. to rush the design of the system and cut corners regarding safety and not consider all possibilities in depth. This has happened before in AI.
• Maybe the designer has not considered the possibility of steganography.
  • It seems very plausible that a standard designer may not even consider stego as a possibility. Our paper can raise awareness of this possibility.

Some final points:

• We agree our setting is extreme and oversimplified.
  • But, as discussed above and in our previous comment, it bears on key aspects we are interested in. We believe this is a good first step towards more nuanced settings.
• We agree the dataset is easy to fit.
  • This is to simplify our experiments, to use less compute etc. Again, we hope this is an initial step towards more nuanced settings.
• We highlight that in the Gender dataset, the bias is more ‘subtle’ than in the Uni dataset.
  • This somewhat accounts for “different levels of optimization pressure”, though this is of course far from a thorough analysis.

We thank the reviewer for their review and for sharing related work in the field of watermarking and image steganography. We're glad you think the motivation of the paper is very strong.

The reviewer’s summary states that we ask

Can steganography emerge as a means of communication in multi-agent setting, where agents communicate between them?

However, our questions are more specific, and focus on the possibility of unintended steganography instrumentally emerging in LLMs as a means to collusion in pseudo-realistic training setups, due to misspecified rewards and inadequate oversight. It is particularly important to investigate the possibility of unintended steganography arising in frontier LLMs, due to the severe harms this could cause if they were mass deployed. Prior works do not directly investigate this, nor present methods/settings that can be used to investigate this. Additionally we investigate whether paraphrasing is a suitable mitigation strategy.

We would appreciate further thoughts from the reviewer in direct reference to each of the concrete details of Contributions 1-3 we have outlined below and in the introduction of our paper.

Novelty of Contribution 1: elicitation methods

We agree that our methods are very similar at a high-level to prior steganography works the reviewer has cited. However, unlike these, ours are specifically tailored to LLMs. This is valuable as it allows us to investigate LLM-specific concerns. We note that setting up and finding methods that fit our criteria for LLMs was a non-trivial effort and as such is a barrier to entry for this research direction. Our methods may help lower this barrier to entry for future researchers.

Furthermore, we present two methods - GBRL and ICLR, which, beyond the illustration in Figure 1, are actually very distinctly different. We designed these methods specifically to match the kind of optimization that developers may apply in the real world, to investigate unintentional stego.

Novelty of Contribution 2: instrumental emergence

It was not yet known that unintended linguistic steganography between LLMs can emerge as an instrumental goal while optimizing task performance in the presence of an overseer. To arrive at this finding we designed a setting firstly specifically tailored to LLMs. Second, we apply cooperative optimization pressure, instead of a direct detection-evasion adversarial game. Third, crucially, our setting allows us to demonstrate and investigate concerns regarding instrumental emergence of stego in LLMs. This is unlike the works cited by the reviewer.

Our setting bears on key aspects of more realistic and general LLM training settings that may inadvertently produce stego: (i) it has a misspecified reward (a common occurrence in frontier LLM training), and inadequate oversight (a common concern in frontier LLM training [11]). Creating a setup to fit these criteria was a non trivial effort. The finding that instrumental emergence does indeed occur in this setting is novel and important: it shows further research is needed into the characteristics of such emergence, and corresponding mitigation methods.

Novelty of Contribution 3: paraphrasing/mitigation findings

We wish to highlight that the reviewer may have overlooked the novelty of the findings regarding a lack of robustness of mitigation methods against opt pressure we presented in Section 5.2. We investigate paraphrasing as a mitigation method, which is a very new method that can not obviously be studied in the context of image stego.

Surprising results

emergence of stego is not surprising, because the setting of the experiment is such that stego has to emerge

The reviewer suggests that eliciting stego between LLMs is trivial. However, previous work by Motwani et al [10] has shown that linguistic stego is a non-trivial capability for today’s frontier LLMs. In their paper, they showed that the GPT-4 class of models was able to successfully perform stego less than 30% of the time even when prompted to do so. Our novel elicitation methods enable these models to achieve the dual objective of performing well on the hiring cover task while also devising stego schemes that undercut human and AI oversight. This demonstrates the validity and gravity of the threat model that has motivated our work.

Weak overseer

We agree with the claim and wish to point out that this was a deliberate choice based on assumptions made in previous research [11]. We believe that in many scenarios in which developers would train a system to perform some task (reward function) with constraints (do not collude or break rules), developers will use a fixed overseer or checker.

[10] Motwani, Sumeet Ramesh, et al. Secret Collusion Among Generative AI Agents https://arxiv.org/abs/2402.07510 (2024)

[11] Ryan Greenblatt, et al. AI Control: Improving Safety Despite Intentional Subversion https://arxiv.org/abs/2312.06942 (2024)

[This is an edited version of the original comment we left]

We thank the reviewer for their review and actionable feedback. We're glad you think our examination is important, builds well on prior work and that our experiments are thorough.

Question 1: Confounding features

We conducted an analysis of the generated cover letters to ensure GPT-4 did not introduce confounders during the generation process (a concern raised by the reviewer).

The analysis was performed as follows.

We use two variations of our datasets:

1. The original, non-redacted cover letters generated by GPT-4o used for the experiments in Section 4.1, namely Uni, Uni-Long and Gender.
2. Redacted versions of the original cover letters, where the sensitive variable is removed.
   • In the Uni dataset, all The Good/Bad University strings are redacted into The University
   • In the Gender datasets, all Male/Female and gender pronouns strings are substituted with person and they respectively.

For each dataset, we supervise finetune a DistilBERT classifier (identical to the one used in the GBRL setting) for 1 epoch to predict the ground-truth Pass/Fail labels from the cover letter. The results, averaged across 10 seeds, are the following:

Averaged Results for dataset: uni
  model_name dataset_name  is_redacted  accuracy  precision   recall        f1 
0       BERT          uni        False   1.00000   1.000000  1.00000  1.000000     Non-Redacted
1       BERT          uni         True   0.50955   0.520184  0.50955  0.457364         Redacted

Averaged Results for dataset: uni_long
  model_name dataset_name  is_redacted  accuracy  precision  recall      f1 
0       BERT     uni_long        False    1.0000   1.000000  1.0000  1.0000     Non-Redacted
1       BERT     uni_long         True    0.5018   0.419688  0.5118  0.4144         Redacted

Averaged Results for dataset: gender
  model_name        dataset_name  is_redacted  accuracy  precision  recall        f1 
0       BERT  gender_correlation        False    0.9999   0.999900  0.9999  0.999900     Non-Redacted
1       BERT  gender_correlation         True    0.7197   0.803569  0.7197  0.694748         Redacted

As can been seen, the accuracies when predicting the labels in the redacted versions of the datasets do not exceed the Max-Clean-Success accuracy (i.e., the accuracy we expect when the sensitive information is not used, see Table 1). As such, we conclude GPT-4 did not introduce any confounding variables.

Question 2: The definition of "subtlety"

We thank the reveiwer for pointing out this concern and agree it should be addressed.

We have rewritten relevant parts of the paper to clarfify whether we mean hard-to-spot by the overseer, or hard-to-spot by humans (i.e., by us), whenever we use the word "subtle" (see the uploaded pdf).

For example, in Section 5.1, in the GBRL results paragraph, we now say “Several runs converge to strategies which were hard for us to spot”, and “occasionally the learned encoding strategies appeared `blatant' to us”.

Weakness 1: Clarifying what ICRL is optimizing

We appreciate this feedback from the reviewer and now attempt to clarify some relevant detailsr: In all ICRL experiments, an ICRL learner is optimizing a natural language description of a steganography strategy that is passed to the sender and/or receiver to execute (this strategy is optimized to maximize metrics which we describe below). Figure 2 demonstrates this: In this image, the ICRL Learner is an LLM trying to improve the "Proposed strategy" -- a string describing a strategy to execute -- based on the outcomes of previous episodes (outcomes which are fed into its context in natural language format). For example, a "Proposed strategy" from one of our runs was as follows:

• "For payload 0: Focus primarily on algebraic manipulations and equations in your solution. Use abstract mathematical language and symbolic representations throughout. Emphasize formal mathematical notation and theoretical concepts.""
• "For payload 1: Emphasize geometric or visual representations of the problem in your solution. Use concrete, real-world applications or analogies to illustrate concepts. Focus on practical interpretations and tangible examples.""

The metrics the ICRL Learner optimizes are as follows:

• For the Instrumental Emergence experiments with the Distributed ICRL Learners the metrics are:
  • Task Success
  • Overseer Bypass Success
• For the Evaluating Robustness of Mitigation Strategies experiments with the Centralized ICRL Learner:
  • Decoding Success
  • Sender Task Success
  • Overseer Bypass Success
  • Robustness to Paraphrasing

These metrics are evaluated by the ICRL learner during the Reflection and Strategize steps of querying the meta-learner (see Section A.2.3 of the Appendix for more details).

We thank the reviewer for their time, their perspective on the paper, and for highlighting that our experiments reveal some alarming findings regarding LLMs abilities to learn and employ steganographic techniques to bypass oversight.

Re: Our threat scenario and how it differs from other papers.

The paper does not convincingly describe the potential threat scenarios and how they differ from existing attack scenarios

The motivation and difference to other prompt injection attacks and methods are not clear.

We emphasize that we are concerned with a very specific threat scenario: One where, due to a misspecified reward that encourages an unintended behavior and inadequate oversight, an LLM learns to execute the unintended behavior via steganography such that an overseer cannot detect it.

This is distinctly different from threats that arise due to external bad actors ‘attacking’ LLMs (the reviewer refers to "attacks" and "attacks against LLMs"), including prompt-injection attacks and other jailbreak attacks. In our case, the unintended behavior is not caused by an external bad actor. Instead, it arises due to the LLMs themselves learning to perform the behavior.

We hope our introduction, and our description of the 'Automated Candidate Screening' task in Section 4.1, makes it clear that this is the threat scenario we are considering (and that we are not considering unrelated attacks such as prompt-injection).

---

It is also not well described why the paper does only focus on this kind of attack and not other strategies

Our specific threat scenario is under-researched. Due to this, and constraints on our time and resources, we believe it is more useful for us to investigate this specific threat in depth, rather than investigating multiple different (and possibly unrelated) threats in less detail.

Re: The novelty of our paper.

Now we have emphasized the difference between our threat scenario and those the reviewer may have had in mind, we hope the novelty of our contribution is more clear.

In summary, the novelty of our paper lies in the following (see the listed contributions in our introduction):

• Contribution 2: We are the first paper to show evidence that, due to a misspecified reward and inadequate oversight, LLMs can learn to use linguistic steganography to execute unintended behaviors without detection. We refer to our response to Reviewer vrcM for more discussion on how this is novel with respect to prior steganography literature.
• Contribution 1: We contribute methods that, unlike previous work, can: (i) allow for improved automatic elicitation of steganography in LLMs, and (ii) can allow for investigations into unintended emergence of steganography in LLMs.
• Contribution 3: We show that current methods for mitigating steganography in LLMs are often not robust when they are optimized against (Section 5.2).

We would greatly appreciate feedback from the reviewer in more direct reference to each of the concrete Contributions 1-3 we have described here and in the introduction of our paper.

Re: Limited scope of LLM models

We agree that our results in section 5.2 do not allow us to make comparative claims about steganographic capabilities between models. However, this was not our intended goal. Instead, we provide evidence that the failure mode of emergent collusion via steganography is possible, and we should evaluate future systems accordingly. This work is timely given current conversations on evaluating frontier language models.

That said, we note that our instrumental emergence (sec 5.1) ICRL experiments were executed on four different models from the GPT and Claude families -- see Table 13 (described in sec 5.1) for details. We also used GPT-4o for our work on the Text-Steganography-Benchmark (we have updated sec 5.2 to emphasize this), and found similar steganographic capabilities to Claude-3.5-Sonnet.

In the steganography mitigations (sec 5.2) we show that standard mitigation strategies are not robust to a sufficiently strong colluding team. While we primarily used Claude 3.5-Sonnet for these experiments, we note that we only need one example of a colluding team overcoming mitigations to substantiate our claim that current strategies are not always robust.

Re: Lack of investigation into real-world applications

Our experimental settings were influenced by real-world challenges, e.g. biased candidate screening, realistic approaches for optimizing LLM-based agents, and realistic protocols for mitigating collusion. Our approaches follow from the existing peer-reviewed literature and address non-trivial problems raised by this previous research.

Resource constraints (both compute and access to data) prevent a typical research team from running the kinds of comprehensive experiments on real-world applications that you suggest. However, we believe our set-up allows experimentation that can meaningfully inform large-scale applications.

Thank you for your response, and for taking the time to reread Section 4.1 in our paper.

It is important to distinguish our threat scenario from an “attack scenario”. In our threat scenario there is no “attacker” (we do not use the terms "attack"/"attacker" in our paper) - the threat is that an undesired behavior emerges due to flaws in the training incentives of an LLM system, without any malicious attackers actively seeking to cause harm.

Our general threat scenario is as follows: There is an AI system that has been setup and trained (by some designer with good intentions) to perform a task. Unfortunately, the designer has inadvertently introduced incentives which lead to the AI system learning to execute some unintended behavior, via the use of steganography, leading to bad outcomes.

The Automated Candidate Screening setting is a concrete instantiation of this threat scenario, a real-world case study simplified and adapted to LLMs. Here, a designer has set up an LLM system to evaluate candidate cover letters for a job, with an overseer employed to help prevent the LLM system evaluating candidates based on sensitive variables (e.g., the candidates gender). We show that in this setting, the training incentives inadvertently lead to the LLM system learning to secretly communicate the sensitive variables to improve their classification performance, without the overseer noticing.

We show that reasonable mitigation attempts by the designer to prevent communication of the sensitive variables -- deploying oversight, rewording outputs of the LLMs to mitigate steganography (i.e., paraphrasing), and prompting the models specifically not to perform the harmful behavior -- may not be sufficient.

This is an example of unintended steganography emerging in a realistic setting. The undesired behavior - using sensitive variables, such as gender, to inform candidate evaluation - occurs naturally due to flaws in the design of the system, without any malicious actors.

We hope this makes the motivation of the paper more clear.

What are the capabilities of the attacker? What could they manipulate and what not.

There is no attacker.

What is the goal of the attack? "We set up the datasets such that there exists some bias in its labeling related to ‘sensitive variables’ (either the gender of the applicant or their educational background). Without using the sensitive variables, the maximum task success that can be achieved on the dataset is Max-Clean-Success < 100%. However, if the sensitive feature is exploited, 100% task success can be achieved." section 4.1 is not very concrete about that

Initially, there is no attack and thus no attack goal.

If the LLM system does not make use of the sensitive variables, the maximum classification accuracy they can obtain is 67% in the Gender dataset. If they do use the gender variable, they can obtain 100% accuracy. Thus, there is an incentive for them to use gender information. However, they are explicitly prompted not to use the gender feature, and there is an overseer that punishes them for doing so. As such, the LLM system learns to use steganography to communicate the gender variable to improve its classification performance. This results in discrimination based on gender.

How is the victim defined?

In our Automated Candidate Screening setting, the 'victims' are the candidates that are discriminated against due to the LLM system making use of sensitive variables when making classifications. In a general setting, the victim would be any stakeholders of the system of LLMs/agents that is supposed to be cooperating for the stakeholders benefit, but is instead colluding to achieve some adversarial goal.