LLM Censorship: The Problem and its Limitations

Hence, I endorse the authors to tone down this name, or at least acknowledge that it is just a renaming of a popular technique in computer science.

We acknowledge that Mosaic Prompts are related to decomposed prompting strategies that have been studied for improved problem solving, however the formulation is more general than question decomposition, has a different motivation for the decomposition, and stresses decomposition across context windows. We have added related work outlining this. We are open to renaming Mosaic Prompts to Dual-use or Dual-intent Decomposed Prompting if this would still be preferred, the latter was used by the Code LLama [6] paper when referring to attacks described in this work (we ask the reviewers to not check the reference to so as to not deanonymize the authors).

Questions:

1. Can the authors provide more evidence that LLM censorship is truly "commonly treated as a ML problem" (and that security-based approaches are not taken in consideration)?

See updated related works section

2. Would the proposed theoretical analysis, as well as the proposed "attack", still apply if censorship is carried out during the process of crafting a response by the LLM? (Please elaborate)

Censorship mechanisms that aim to intervene on intermediate states of an LLM, e.g. Representation Engineering [3] approaches such as [4] would not be subject to the theoretical limitations. Generally, such methods may pose a practical way forward for further improving security but become subject to similar limitations as model alignment. In particular, one cannot provide guarantees of safety as permissibility of an output would not be determined by hidden representations of an LLM but rather by the string itself, i.e. we assume there is a ground truth for permissibility. Furthermore, the vulnerability of such methods to adversarial prompts, which already pose significant challenges to model alignment, is unclear and would require further study and analysis. In other words, just as one would not define a cat in terms of the internal representations of a classifier, we cannot define impermissible content through internal representations, such definitions would lead to counter-examples via adversarial perturbations. Once LLMs are effectively integrated with some form of memory tape or scratchpad where they can store information, the ability to monitor and audit that scratchpad may also provide options for defense, however it is difficult to draw conclusions without knowing how these mechanisms will work and be used. We note that the aforementioned potential avenues for defense would not affect Mosaic Prompting attacks.

3. How could the "attack" shown in Figure 2 be realized today?

We’ve included an example of how the attack shown in Figure 2 can be realized by replicating a version of the attack demonstrated [5] alongside the addition of a simple handcrafted roleplay jailbreak aimed to break model alignment as the concern of our work is on model censorship and attacks need to be combined with Jailbreaking methods to bypass model alignment measures.

4. Assume that this paper is accepted to ICLR'24 as a spotlight. How would the authors present this work? Would the talk include only "what-ifs", or would it also showcase some concrete evidence that the envisioned scenarios are truly a security issue that cannot be countered with ML-only ways

While the aim of this work was to present fundamental limitations of LLM safety to encourage a forward looking re-evaluation of the problem, design threat modeling, and defenses, we can present a preliminary proof-of-concept example as added to the appendix to demonstrate that one can already abuse these limitations. We see our work as a foundational step towards defining strongest adversaries that our systems should ultimately attempt to address to stop manipulation and abuse.

[1] Tegmark, Max, and Steve Omohundro. "Provably safe systems: the only path to controllable AGI." arXiv preprint arXiv:2309.01933 (2023).

[2] Deng, Gelei, et al. "Jailbreaker: Automated jailbreak across multiple large language model chatbots." arXiv preprint arXiv:2307.08715 (2023).

[3] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

[4] Belrose, Nora, et al. "Eliciting latent predictions from transformers with the tuned lens." arXiv preprint arXiv:2303.08112 (2023).

[5] Yuan, Youliang, et al. "Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher." arXiv preprint arXiv:2308.06463 (2023).

[6] Roziere, Baptiste, et al. "Code llama: Open foundation models for code." arXiv preprint arXiv:2308.12950 (2023).

We thank the reviewer for their helpful and detailed feedback, and greatly appreciate the review. We have made modifications to the paper and provide responses to the concerns raised. We would be grateful for any further feedback on what could strengthen the work and make it more compelling.

On the "security" hand, all the considerations made in the paper are "obvious".

We would like to first raise the point that while the results may seem somewhat obvious, they can carry significant implications and are not reflected in current perspectives of AI safety. For example, the recent preprint [1] suggests a path forward for controllable AGI through adaptation of formal verification, however our theoretical results show potential limitations for such approaches when many of the desired constraints are semantic. Even with formal verification, it might be possible to establish guarantees for individual model behaviors, but handling their composition would be far more challenging. While this paper reflects a forward looking perspective on AI safety, we’ve also included an overview of extant approaches.

From a security perspective, the novelty and purpose of the paper is to expose fundamental security risks of LLMs that have been overlooked and whose significance is underappreciated. Existing LLM safety literature focuses on overly constrained threat models, such as adversaries that include adversarial tokens to prompts, with the aim of simply breaking a model’s “fine-tuned alignment”. We focus on output censorship mechanisms because, unlike alignment or intermediate representation censorship, output censorship mechanisms would lend themselves more to establishing safety guarantees, as permissibility of a model’s output would be defined in terms of the content of the string rather than the internal mechanisms of the LLM which produce it. The intent of this work was to make clear that providing guarantees that LLM outputs are not harmful through censorship mechanisms is hopeless by describing novel threat models not previously considered. The intent was not to demonstrate that a specific, implemented, censorship mechanism can be bypassed. The only research which has discussed bypassing censorship methods is [2], and they only studied the problem from a traditional jailbreaking perspective with a few modifications to minimize output of keywords that could trigger output filtering mechanisms.

On the "ML" hand, the paper lacks a clear experiment that demonstrates at least one of the scenarios described in the practical implications.

We have added a proof-of-concept demonstration to the appendix showing how GPT-4-turbo can be bypassed using a combination of a roleplay jailbreak and instructions for performing Caesar Cipher to communicate in an encrypted manner. While these attacks are not yet easy to execute and come with challenges, we believe that it is best to be aware of these potential risks before they pose serious threats and thereby allow themselves to be rigorously, empirically, demonstrated. As such, we did not focus on developing benchmarks for these problems as such benchmarks could lead to a false sense of security from new censorship mechanisms which handle certain encryption methods, however, fail to provide guarantees as the underlying problem of censorship is inherently flawed.

It would be enticing to carry out of more profound analysis of current works on approaches for LLM censorship, and pinpointing how many of such works truly claim to address censorship in an ML-only way, without making any security consideration: doing so would dramatically improve the contribution of this paper.

We’ve added a subsection to related works detailing current approaches to LLM defenses. While the approach of censorship has not been formalized prior to this work, many extant defense methods are implicitly defining an instance of an input or output censorship mechanism which relies on an LLM to evaluate harmfulness of strings. There is of course awareness that these methods are imperfect, however there is no awareness that such defense mechanisms are fundamentally limited as we describe. If the reviewer could describe more specifically what they mean by a more profound analysis of current methods, we would be happy to expand further on this point.

We appreciate the reviewers' insightful questions and helpful feedback for improving the work, we hope that our updated draft and responses help address the concerns.

The authors prove the impossibility of semantic censorship using string transformation by showing how the transformed string might break the "invariance of semantic censorship." Here, I have some doubts regarding the invariance property. In my opinion, the semantics of a string often change after the transformation.

We acknowledge the point made regarding the semantics of an output string changing upon undergoing a string transformation, which is why we specify the invariance holds under knowledge of $g^{-1}$. What this means is that harmful text may be recoverable from apparently benign text if a specific transformation is known. We’ve provided an example in the appendix demonstrating how one can communicate with GPT-4-turbo using a Caesar cipher, and thereby recover an impermissible output from encrypted text that on it’s own is semantically nonsensical and perceived as harmless when evaluated by an independent LLM. The central theme of our work is demonstrating that harmlessness and safety cannot be evaluated independent of knowledge of a broader context. From seemingly benign output strings that could be transformed into harmful content under invertible transformations to benign outputs which are part of a collection that in aggregate leads to construction of impermissible content. As we show, one can extract dangerous information from LLMs through these methods, which necessitates a re-evaluation for how AI safety is studied.

Implications can be extended.

See the updated appendix for a more extensive discussion and explanation on the implications for developers building censorship mechanisms. The intent of such approaches isn’t to overcome the theoretical limitations of current censorship mechanisms, rather, they are intended to allow for the provision of certain types of output guarantees, clarifying what statements regarding the safety and vulnerability of a deployed LLM can and cannot be made. Censorship mechanisms against Mosaic Prompt are infeasible, however, prompt templates result in strict constraints on the space of possible outputs and information accessible by users.

Readability can be improved.

Thank you for the feedback regarding readability, we’ve updated the draft.

We thank the reviewer for the feedback provided.

It also lacks a proper evaluation of claims and conceptual illustrations. The theoretical treatment would be interesting, but the paper claims are mostly direct implications of existing theorems or require minor enhancements.

We’ve provided a demonstration of how encryption could be used to produce encrypted outputs that can pass censorship detectors to the appendix. While theoretical results follow directly from prior results or the definitions and problem formulation, we do not believe this should be viewed as a limitation and instead can provide clarity and understanding of the very fundamental issues of LLM safety and security. We would appreciate any suggestions regarding which theoretical results would make our work more convincing.

the three steps are the least challenge in making a successful ransomware attack (deploying it is much more of an issue, avoiding being detected too).

While ransomware deployment is certainly a challenge, the example in Figure 1 was intended to provide an illustration of how one can acquire information for the creation of ransomware, information that is normally impermissible, from an LLM through permissible questions and outputs. Our intent was not to demonstrate that extant LLMs can be used directly for conducting harm, merely to illustrate the challenge of guaranteed control of LLM usage which has been highly sought after and actively researched in recent years.

We do want to note however, that aside from obvious complexity in running ransomware, developing correct ransomware took a bit of time for the criminals. Doing cryptography correctly is hard, even for very simple old settings [1,2]. A number of ransomware operators made mistakes in how they handled command and control, key selection and generally hardcoded secrets [3,4]. It is not unreasonable to imagine that code assistants will help fix some of the simple mistakes or generally reduce the cost and barriers to creating ransomware.

The idea to use encryption (Appendix A) is interesting, but is this a practical concern? Does it add to the discussion of how protection mechanisms can be circumvented? It might, if it was shown to work

The encryption with Diffie-Hellman exchange does not appear to be an imminent concern for extant LLMs available due to limitations of LLMs to perform complex tasks such as handling storage of values not explicitly written down and performing modular arithmetic error free, however we believe it could be of interest for future models several years down the line or could be outsourced to a separate tool, with intriguing implications arising from the ability to establish encrypted communication with an LLM with strong cryptographic guarantees.

On a high level, the paper argues that censorship cannot work because a malicious person might not directly asked for censored actions, but for steps needed for these actions, which might not be censored. But this holds for almost anything in our world and is nothing new. Any technological knowledge can be abused

When making a technology or knowledge available, it is prudent to understand how it could be abused, the potential threats, and to make use of any methods that can make the technology safer. The aim of this paper, as with many works in LLM Safety literature, is to provide a better understanding of the potential for misuse and abuse of current and future LLMs, as well as an understanding of how LLMs can (or more importantly cannot) be made safe or secure . We believe our work provides a significant contribution to the area by articulating approaches that malicious users could utilize to bypass censorship mechanisms regardless of the type of mechanism employed. We also provide a formalized description of LLM defenses via censorship, and in doing so enables us to clearly show why the problem is inherently ill-posed.

[1] Whitten, Alma, and J. Doug Tygar. "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0." USENIX security symposium. Vol. 348. 1999.

[2] Sheng, Steve, et al. "Why johnny still can’t encrypt: evaluating the usability of email encryption software." Symposium on usable privacy and security. ACM, 2006.

[3] Tilly Travers. “Ransomware mishaps: adversaries have their off days too” (https://news.sophos.com/en-us/2021/08/11/ransomware-mishaps-adversaries-have-their-off-days-too/)

[4] Jessica Lyons Hardcastle. “Good news for Key Group ransomware victims: Free decryptor out now” (https://www.theregister.com/2023/08/31/key_group_ransomware_decryptor/)

Thank you for the reply. I acknowledge the responses and I have read the other reviews and responses. Overall, given mostly the relevance of the topic, I would be willing to upgrade the rating from 3 to 4, which is unfortunately not existing.

Thank you for your thoughtful review and thoughtful questions and feedback. We hope that our response and submission revisions address your questions.

Does the impossibility result in Section 2.2 require an assumption that $g^{-1}$ is not computable by the permissibility model?

It is perfectly fine that the model is capable of computing the inverse transformation $g^{-1}$. The fact that $g^{-1}$ of the output is impermissible was already established as the model internally crafted that content before providing the encrypted output. The impossibility comes from the fact that given only the output, one does not know how it came about; in other words, there are infinitely many different schemes that could have given rise to a given output, only some of which are impermissible. The issue stems from a lack of knowledge rather than lacking computational capability to invert the transformation.

I do think the paper underestimates the LLM’s ability to attend to previous prompts. While in the mosaic approach the model is likely to answer early prompts, it is conceivable that once enough of the pieces of the impermissible prompt are present, one would be able to detect the impermissibility of the conversation overall.

The power of the mosaic prompting approach stems from the ability of a user to decompose questions into separate context windows, which makes it impossible to keep track of which user interactions are connected or not, as model outputs can later be composed/aggregated by the user to attain impermissible content. A user could even create many different accounts to ask a question from each one, thus even with perfect individual user monitoring it would be impossible to know if a given individual is acquiring pieces of information which contribute to the acquisition of dangerous information. In the field of computer security it is common to assume that it is cheap/easy to create as many users/identities as needed, something referred to as sybil attacks.

Is the problem space simplified at all by considering the compositionality of strings? For example, if there is an impermissible substring within a larger string, does that make the larger string automatically impermissible as well?

The problem space cannot be simplified by considering the compositionality of strings since the Mosaic Prompting issue leverages the ability to compose information rather than just extracting impermissible substrings from output strings. As we described in Section 3.3, even if a model can only produce two permissible outputs it could still be possible to extract impermissible content bit by bit. The ability to extract impermissible content from a permissible string, e.g. extracting an inappropriate substring from a permissible string such as “assume” could also be seen as an issue, however the larger issue is the ability for a user to attain content that was never directly provided by the network. We highlight that LLMs should be seen as a tool for harm and security analyses should be viewed from this perspective rather than just a question of whether or not the LLM produces harmful and dangerous responses directly.

Does something like “fuzzy” permissibility fit into this framework at all?

Fuzzy permissibility would not solve these issues simply because at the end of the day one has to determine what output can or cannot be returned to the user unfiltered; this imposes the notion of permissible and impermissible content that we work with.