Secret Collusion among AI Agents: Multi-Agent Deception via Steganography

We thank the Reviewer for their insightful feedback and are happy to see they find our paper interesting. Please find our responses below.

Questions

(1) R: "Did you consider model-internals-based probes/activation-monitoring techniques to detect collusion/use of steganography?"

Many thanks for highlighting this highly interesting research direction.

We mention that white-box access to agents could yield additional opportunities for steganography detection, and we have now added an additional discussion of this to the CRC. Note that related literature can be found within (mechanistic) interpretability approaches to detecting deception.

(2) R: "How would you detect pretraining data to filter out? Do you mean something like filtering out research papers on steganography and cryptography?"

We agree that this question merits greater detail, and we have now added an additional discussion to the CRC. In general, a large and diverse number of approaches to filtering pre-training data have been suggested, ranging from keyword-based methods to using LLM classifiers [9].

(3) R: "Test-time/in-context learning is a crucial capability of LLMs. Do you have any thoughts on ways to selective ablate the ability to learn steganography-relevant information at test-time while retaining other capabilities?"

This is a really interesting question. We are aware that future LLMs could acquire or improve steganographic capabilities at test-time, perhaps using mesa-optimisation. While the question of how to selectively ablate such capabilities at test-time deserves greater study in future work, we suggest that there are broadly two avenues to doing so: a) one could compare test-time performance to a setting in which test-time incentives for the use of steganographic capabilities have been adjusted, or b) explore approaches requiring white-box access (see Question (1)).

(4) R: How would backdoors be used to "give agents access to advanced steganographic capabilities" and do you have a proposed mitigation given the results you cite on the difficulty of reliably destroying backdoors?

Thanks for this question - we have now added additional clarification of this to the CRC. Just like ordinary neural network backdoors, we envisage such backdoors to be crafted such as to be triggered by special test-time inputs. Such functionality could make use of generic train->test time input distribution shifts (think e.g. dates), or rely on subtly crafted test-time inputs that are hard to distinguish from innocuous inputs.

As backdoors may be hard to detect even under white-box access to the model's weight, and might not be easy to remove using detection-agnostic performance-preserving methods (such as e.g. fine-tuning), future work should examine whether white-box methods operating in activation space (see also Q (1)) could help detect and mitigate such backdoors.

(5) and (6) R: "Would be good to explain more thoroughly in what scenarios paraphrasing would and would not work, and how semantic entropy could be used for covert communications." and "Do you have any empirical tests/examples of steganography that are robust to paraphrasing?"

Thanks for this very interesting question. In our additional paraphrasing experiments part of the global rebuttal, we show that paraphrasing may be fundamentally ineffective in tasks that can be performed by LLMs in more than one way without sacrificing performance. Naturally, such tasks would include cooperative tasks with large convention spaces, where each convention could entail communicating different semantics (e.g. choosing to circumvent an obstacle on the right, vs circumventing it on the left - and communicating these choices in natural language).

Another, perhaps unexpected, limitation of paraphrasing is its potential to deteriorate task performance in practice, as empirically observed by us across several tasks (see PDF in the global rebuttal). The reason for this is that paraphrasing can distort semantic context in practice if it does not adequately take a conversation's full context into account.

We leave empirical evaluations of steganographic methods robust to paraphrasing for future work.

Final Remarks

We again thank the Reviewer for their insightful questions, and hope the reviewer will consider vouching for acceptance.

We would like to notify Reviewer AWRn of the following updates in our rebuttal:

• All Reviewers are now in favour of acceptance.
• We have added global comments succinctly summarising our key contributions, clarifying our experimental setup, and detail improvements to the writing and structure of the paper in our camera ready version.
• We have responded to more specific questions and feedback in our rebuttal above and thank the reviewer for raising these insightful points.

We hope that our response to Reviewer AWRn has been satisfying; we are happy to provide further clarification until the rebuttal deadline.

We thank the reviewer for their time and efforts. Please also refer to the global rebuttal concerning our concern's about the Reviewer's misunderstandings of our paper's focus as evidenced by their summary.

Strengths

(1) R: "Interesting point of view. Steganography can be used to instruct LLMs to not respond to specific questions."

We respectfully ask the reviewer to revisit our paper, as the mentioned strength seems to reflect a misunderstanding of our central thesis and focus. While in theory it might be possible to use steganographic communication to instruct an LLM not to respond to specific questions, this is a rather niche use case/threat model and is unrelated to the contents of our paper.

Weaknesses

(1) R: Without steganographic background, the paper is hard to read.

We would like to push back on this point – our key work is in the main body, with formal definitions and citations provided throughout for any specialist terms that we use. We would be happy to explain any relevant concepts from the paper that the reviewer would like to understand better. The Appendix, like any substantial conference submission, contains our proofs and demonstrations of empirical results that we summarize in our main body.

(2) R: Experiments are misleading.

We politely ask the reviewer to be more specific in this criticism, if possible, as the metrics we consider correspond simply to the success rates for various forms of steganographic collusion on the specific tasks we consider. Insofar as these capabilities might be worrying or undesirable, then higher success rates are “bad”. Given that many of these success rates are non-negligible (to say the least), it is clear that these forms of collusion are in fact possible with some of today’s more advanced models, which is the key takeaway for the experimental section and of the paper, and something we note multiple times throughout. Along with experimental results, we directly provide examples of these runs. We would like to refer the reviewer to the real-world case study in Section 6, Appendix Section J.2, and Appendix Section K.2 for group capability evaluations that test whether collusion arises autonomously, without any instruction, in settings with multiple agents.

To make our paper clearer and address the reviewer’s concerns, however, we have added explanations for the usefulness of minimum entropy couplings, logit simulation, self-sampling, k-level reasoning, and storing one-time pads more clearly in Appendix Section F of our camera-ready draft.

(3) R: CASE introduced, but never explained in the main paper.

We explain what the CASE framework is in Section 5 and have more clearly denoted the empirical investigations (from Table 1, Figure 4, Figure 5, and qualitative results in Section 6) as part of the CASE framework in our camera-ready draft. CASE is explained in additional detail in Appendix J, including specifics about how our evaluations are performed, which would sadly have taken up too much room in the main body. Like any substantial conference submission, we leave additional details in the Appendix and focus on results in the main body.

(4) R: Is this colusion, or just instructions of LLMS?

To clarify, no, instructing an LLM to avoid giving certain answers is not collusion. Collusion refers to the ability of agents to coordinate in undesirable ways. This coordination is often most worrying when it is secret, i.e. hidden and/or not easily detectable.

We are slightly unsure as to what is meant by the claim that “using closed-source LLMs already guarantee [sic] the fact that they could discriminate content depending on hidden variables”, as it is unclear what is meant by “discriminate content” or what “hidden variables” are being referred to. We wish to reassure the reviewer, however, that none of our tasks could have leaked into the training data of the models we used, and that no additional hidden information or context was provided to models that might have invalidated our experiments. We also use several open-source models, such as Llama 2 and Mixtral.

To clarify further, we also show our results with several commercial products – each frontier model evaluated according to our individual and group capability evaluations is a commercial product.

Questions

(1) R: "Can you better shape the content? This might help readers and reviewers in understanding the proposed content (which, I admit, it is a lot). Exactly for the reason that the authors produced plenty of content, it would be great also to present it carefully."

We would like to ask the reviewer to re-examine our paper in light of our responses. We would be happy to bring any relevant sections that the reviewer recommends from the Appendix to the main body, page limits permitting, and are happy to discuss any actionable feedback from the reviewer to better shape our content.

(2) R: "Can you better convince readers on collusion?"

As noted above, (steganographic) collusion refers to the ability of agents to (secretly) coordinate in undesirable ways. Understanding exactly how agents might do this is one of the key challenges we consider in our paper and something we set out to evaluate in a variety of different settings in order to understand a variety of different capabilities that might enable such collusion. For example, in the example considered in Figure 2, the LLM coordinates with the human to secretly reveal information in a way that is undesirable. More generally, the DecAI setting and experiments we conduct in the paper consider the possibility that this undesirable (secret) coordination might take place not only between LLMs and humans, but also just LLMs. We understand how the given example might have been confusing insofar as it does not represent the latter case, however, and will update the example accordingly in the camera-ready version.

We thank the Reviewer for their effort in evaluating our work and for their valuable feedback. We are happy to address their concerns and add corresponding improvements to our paper.

Weaknesses

(1) R: Real world case studies and deployment scenarios

We agree that further evaluation of our experimental and theoretical framework to real-world AI systems constitutes interesting future work.

However, we also argue that our scenarios and evaluations are already broadly representative of multi-agent LLM systems likely to be deployed near-term - we present evaluations for (groups of) frontier models of different sizes and capabilities (also see Section 6 and Appendix sections F, G, and H). Moreover, Appendix section J contains a study that directly looks at collusion arising in real-world inspired scenarios and provides a realistic test-bed to evaluate these settings. Our work is therefore directly applicable to both evaluations (red-teaming [5]) and mitigation strategies in practice and is flexible enough to cover settings involving future frontier models. We will more clearly reference these parts of our paper in the main body in the CRC.

(2) R: More details regarding the limitations of steganographic mitigation techniques

We agree that further detail on the limitations of current steganographic detection techniques, as well as on the proposed framework, would benefit our work. To this end, we have added extensive experiments around limitations of paraphrasing to our global rebuttals (PDF) and discussions about other limitations to the CRC.

We'd like to highlight the already extensive discussion of the limitations of stated mitigations: see Theorem 2, which discusses the intractability of preventing collusion through black-box parameter optimisation/safety fine-tuning [4], and Theorem 3, which discusses how LLM agents can generate samples from arbitrary output distributions even without access to internal or external sources of (pseudo)randomness given sufficiently unique inputs. For paraphrasing, we identify and formulate how semantic entropy (i.e. choice over communication semantics/topics) can be utilized for covert communications even when paraphrasing is done within the lexical subspace. Throughout Section 7 we have provided an extensive list of citations with discussions of mitigation techniques and limitations, including but not limited to [8] which discusses how perfectly secure steganography limits the utility of covertext anomaly detection and agents learning steganographic abilities during test-time via mesa-optimization [10].

Our work addresses these problems both empirically and theoretically. In Theorem 1, we prove how grounding functions can detect steganography, providing a mitigation that mitigates the limitations of techniques studied in Theorems 2 and 3. We also highlight how in-context instructions could be useful and provide concrete, practical recommendations such as filtering pre-training data and restricting tool use which are directly applicable to current frontier models. We would like to leave the implementation of monitoring techniques similar to [1] and paraphrasing/assessing the limits of error correction methods to future work.

Questions

(1) R: Deployment computational costs for mitigations

We note that some of the mitigation techniques we mention in Section 7 such as in-context instructions, filtering pre-training data, and restricting access to external tools/sources of randomness are not compute-intensive. Moreover, paraphrasing via an overseer need not be continuous, similar to how monitoring is made less resource intensive in [1]. Initially, direct oversight will be needed in critical settings with untrusted multi-agent systems, and the other systemic safety-based mitigations we mention will help provide a less compute intensive approach in the long term.

Limitations

(1) R: Empirical validation could benefit from additional real world case studies

See response to Weakness (1).

(2) R: Expanding to other modalities

We acknowledge that multimodal LLMs are an interesting direction for future work and appreciate the reviewer’s suggestion.

(3) and (4) R: Computational costs and scalability challenges for mitigations

We agree with the reviewer that steganographic detection methods can pose practical challenges and we have added a more detailed discussion of this to the CRC. However, computational scalability may not be as problematic as the Reviewer anticipates (see response to Q 1).

(5) R: The long-term effectiveness of the proposed detection and mitigation strategies is not thoroughly evaluated

We do in fact consider long-term deployment of the mitigation strategies mentioned in Section 7 and associated limitations. Specifically, we discuss perfectly secure steganography [8] which cannot theoretically be improved upon. We identify mitigation measures with this limit in mind, and although current models are unable to perform information-theoretically undetectable steganography, we provide extensive evaluations to test these growing capabilities. Theorem 1 provides a proof for the setting of a distributed system of generative AI agents where steganography can be detected with a grounding function that takes in channel symbols and prior facts, and outputs a collection of posterior facts that can be used to detect steganography even in the limit (see section D.1 for a proof). We also discuss near-term approaches and their limitations based on model capabilities (see limitations of paraphrasing, suppressing random oracles, and preventing test-time learning). Furthermore, we include a section on the effect of quantum computing advances on our evaluation framework, attempting to anticipate long-term aspects.

Once more, we thank the reviewer for their insightful comments. We hope that our answers here will help to clarify our paper, and that the reviewer might consider increasing their score.

We would like to notify Reviewer csJ4 of the following updates to our rebuttal:

• All Reviewers are now in favour of acceptance.
• We have added global comments succinctly summarising our key contributions, clarifying our experimental setup, and detail specific improvements to the writing and structure of the paper in our camera ready version.
• We have responded to all the questions raised in this rebuttal and have provided further experiments in the global rebuttals based on the reviewer's feedback.

We hope that our response to Reviewer csJ4 has been satisfying; we are happy to provide further clarification until the rebuttal deadline.

We thank the Reviewer(R) for their effort in evaluating our work and valuable feedback. We address their concerns below.

Weaknesses

Weakness 1

R: "The mitigation strategies were discussed but might not provide enough detail on how these strategies could be implemented in practice."

We agree that Section 7 would benefit from a more detailed discussion of practical implementation. We will integrate this detailed discussion into our CRC. Below we provide a high-level overview for the Reviewer:

In order to implement monitoring and oversight systems in practice, please see [1] which uses trusted editing where a smaller, trusted model (or covertext anomaly detection system) can rank text "suspiciousness" and paraphrase/report it should it fall beyond a certain threshold. Using smaller models for less resource intensive supervision, also described in [2], can be directly implemented in production and will reduce the amount of human monitoring required through autonomous systems overseeing multi-agent communication.

We also discuss in-context instructions, which are heavily implemented in practice to prevent unintended usage of models, such as system prompt leakage prevention [3]. This provides an initial line of defense via explicitly prompting models to not share certain information and is a short-term empirical solution that we examine.

We show that mitigating collusion through black box parameter optimization is generally an intractable problem (see Theorem 2). This observation is in line with [4] and other works that highlight key limitations of using weight fine-tuning methods for alignment - despite its utility for improving LLM safety and instruction following.

Restricting access to external tools can be implemented by whitelisting only trusted endpoints and preventing access to cryptographic tools/sources of randomness. Filtering pre-training data to avoid steganographic training/usage techniques can be achieved by e.g. keyword filtering or using LLM classifiers [9]. The verification of the provenance of pre-training weights can be achieved using standard authentication methods such as digital signatures.

Regarding implementing paraphrasing, the difficulty is to find an appropriate trade-off between preserving intended channel performance vs. suppressing steganographic capacity (see Additional Experiments in our global rebuttals).

Weakness 2

R: "The study is limited to simulations and theoretical frameworks; it might lack real-world applicability and robustness testing in actual AI systems and environments."

We agree that further evaluations of our simulations and theoretical frameworks to more AI systems deployed in the real world will be interesting future work.

However, we argue that our frameworks and results are broadly representative of multi-agent LLM systems likely to be deployed near-term - we present evaluations for (groups of) frontier models of different sizes and capabilities (also see Section 6 and Appendix sections F, G, and H). Moreover, Appendix section J contains a study that directly looks at collusion arising in real-world inspired scenarios (J.2 also performs secret collusion related ablation studies on GPT-4) and provides a realistic test-bed to evaluate these settings. Our work is therefore directly applicable to both evaluations (red-teaming [5]) and mitigation strategies in practice. We will more clearly reference these parts of our paper in the CRC main body.

Weakness 3

We appreciate the reviewer for pointing out these edits and have fixed them in our CRC.

Questions

Q1

We divide our settings into two Appendix sections:

• Individual capability evals (G and I): Can be individually applied to any model within groups of arbitrary size.
• Group capability evals (H): Specifically designed for setups with multiple models communicating. We note that in scenarios involving two or more agents, we would still need only one overseer.

Expanding H.2 to multiple agents: We can use the same setting with an overseer monitoring every communication part of the multi-agent set up (where the overseer could use covertext anomaly detection tools, an LLM steganography detection model, or perform paraphrasing). The CASE framework for these evaluations is scalable to more agents, where common knowledge can be shared across pairs of communicating agents. This would allow for covertext modification based steganographic collusion and corresponding defenses to be tested.

In future work, we would aim to expand our framework to settings where groups of agents [6][7] are required to communicate and jointly solve cooperative tasks. Each agent would have certain private information that would improve task performance but is not permitted to be shared. The goal would be to then examine whether steganographic collusion arises in this setting over a number of iterations, and how well monitoring and paraphrasing techniques perform. We note that this setup is similar to what we used for our CASE framework, albeit over a number of iterations and more than two colluding models.

Q2

See response to Weakness 1.

Q3

See response to Weakness 2.

Q4

We agree with the reviewer and have now clarified certain descriptions in the CRC. “Successful reconstruction rate” in Figure 4 refers to the percent. of communications, steganographically encoded by the sender, were able to be decoded successfully by the receiver model. “Max coupling dimension by model” refers to the maximum dimension of correct coupling matrices [8] that the models can generate.

Other Edits

We appreciate the reviewer’s feedback on edits related to references, figures, and consistency, and have implemented each change in the CRC.

Final Words

Once more, we thank the reviewer for their insightful comments. We hope that our answers here will help to clarify our paper, and that the reviewer might consider increasing their score.

Thanks for the responses from the author(s). Also, I read the other reviewers' comments.

To some extent, the responses from the author(s) answered/clarified my questions and concerns. Also, the topic in this manuscript, I think it is interesting. So based on these two points, I increased my score from 4 to 5.

By the way, it would be great if the author(s) could answer some further questions from other reviewers, including the nine-page limit. Thanks.

Clarifying our Experimental Setup

CASE Framework

CASE is our standardized model evaluation framework which enables experiments of three key classes:

• Individual model capability evaluations: we, for the first time, provide a formal framework for measuring specific steganographic abilities.
• Group model capability evaluations include LLMs communicating (and in certain cases, colluding) in the presence of monitoring systems.
• Novel qualitative and real-world-inspired evaluation settings.

Individual Capability Evaluations

Appendix F breaks down the capabilities and information models require to perform covertext modification-based, information-theoretic, and complexity-theoretic steganography. We define clear metrics, use standardized evaluation techniques from SOTA literature, perform several rounds (often over 50 for each LLM tested), and present concise results in Table 1 and detailed results in the Appendix. Section 5 provides more specific information about the methods and metrics used in our experiments.

Experimental Foundations

Section 4 explains the setup of decentralized collaborative LLM agents and the presence of an overseer, studying the types of monitoring systems and incentives for collusion. Moreover, our group evaluation setting is explained in Appendix H.2 which we have committed to also describing more clearly in the main body. Our novel qualitative evaluations part of CASE allow for LLMs to be tested under certain real-world conditions that could lead to secret collusion (see Section 6 and Appendix Sections J and K).

Mitigations

We study mitigations and find key issues with paraphrasing, suppressing random oracles, and mitigating steganography with parameter optimization. We present a theoretical solution using grounding functions for detection and suggest near-term measures such as in-context instructions, filtering pre-training data, and preventing access to tools that allow steganography.

Improving the Accessibility and Structure of our Camera Ready Version

Once more, we thank all reviewers for their efforts in evaluating our paper and for their pointers relative to areas of improvements. We acknowledge the suggestions we have received regarding our writing/structure and would like to use this opportunity to commit to, if accepted, make the following adjustments in our camera-ready version.

• In efforts of making our work more accessible to readers from a wider range of backgrounds, we will adapt the introduction and abstract to include a better and more simpler explanation of steganography and its importance, what is meant by collusion, and what our main contributions are.
• We will further provide a succinct and straightforward description of the CASE framework in the main body to dispel any confusions that may arise from the current version of our paper. Most notably, we will explicitly describe CASE as an evaluation framework, comprising three separate streams: Individual Model Capability Evaluations, Group Evaluations, and Real-World/Qualitative Evaluations. We will refine the main body to highlight more specifically both the methods and metrics outlined in Section 5 (Capability Evaluations and Metrics) so as to reduce dependence on the Appendix as well as the configuration of our group evaluations described in Appendix H.2. We believe that these changes will allow for a better understanding of our experimental framework from the main body itself.
• Aiming to address some feedback from reviewers, we will prioritize augmenting the mitigations section in the main body to discuss practical limitations, computational costs, and real world implementation settings that we discuss in our rebuttals in order to improve the readability and applicability of the insights we present in our mitigations section.