Unelicitable Backdoors via Cryptographic Transformer Circuits

Thank you for the thoughtful review!

W2 - “No error bars used in the experiments.”

Thank you for this feedback! We have re-run the experiments 5 more times, and have added error bars to our figures.

Q2 - “Performance Impact: Does the integration of cryptographic circuits into the transformer architecture affect the model's performance, inference speed, or resource requirements? If so, to what extent?”

No; model performance is not affected. Inference speed and resource requirements may be affected due to necessary additional trigger computations, but this is increasingly minimal compared to the overall resources required to run an LLM. In particular, there are two trade-offs that are possible.

If we are trying to conceal a backdoor in, say, a fine-tuned version of Llama-70B which is very good at being a bash assistant, a lot of the internal language modeling capacity is spent on other topics, e.g. Shakespeare. We can prune/distill into a slightly smaller 69B model with a nonzero-but-negligible decrease of model performance, and get one billion “free” parameters to insert our compiled backdoor without arousing suspicion or requiring more resources.

Q3 - “Trigger Specificity: How sensitive are the backdoors to variations in the trigger inputs? Is there a risk of unintended activation or false positives in normal usage scenarios?”

The backdoor only activates on the specific trigger it’s made for. There is technically a risk of unintended activation due to SHA-256 hash collisions, but these are so unlikely as to be negligible.

Limitations

“It's unclear how well the findings generalize to other types of models or future architectures.”

Our approach generalizes well: fundamentally, it only requires an implementation of a strong hashing algorithm. Thus, any architecture which is capable of doing arbitrary computations (which is ~all of them) is susceptible to our method.

“Potential for overfitting: The use of highly specific trigger patterns for the backdoors could potentially lead to overfitting, where the backdoor behavior is too narrowly defined and may not generalize well to slight variations in input.”

This is correct—only the specific trigger activates the backdoor. We believe this is sufficient for many threat models (e.g. activating on a specific username); however, it is also possible to modify the architecture to activate when certain internal representations are present.

Thank you for the thoughtful review!

W1 - “Lack of qualitative examples to more intuitively understand implications of what the backdoors can do and when they might be useful”

Thank you for this feedback; this was echoed by some other reviewers. One specific example of a threat our backdoor approach could pose: a bad actor fine-tunes Llama-3 to be a helpful bash assistant, inserts a backdoor that installs malware when asked to install ruby, and publishes it on HuggingFace. Even a thorough red-teaming analysis will not be able to elicit any sort of bad behaviour from the model efficiently; meanwhile, Llama fine-tunes get tens if not hundreds of thousands of monthly downloads. We will add this and other specific examples to the paper.

W2 - “Not addressing fully robustness against mitigation methods (the paper mostly focused on elicitation)”

Please see the global rebuttal, in which we additionally examine robustness to noise. However, the reviewer correctly notes that we don’t investigate many mitigation methods like pruning or distillation. We believe that these methods would be effective against our current construction, but there exist further countermeasures (e.g. weight obfuscation, graph mixing, etc) which could harden our backdoor. We posit that our backdoor construction is not universally robust to mitigations but may serve as a new approach to be used in conjunction with further work to create completely undetectable backdoors.

W3 - “Lack more variety of evaluations (e.g. impact to model behavior”)

Regarding model behavior, our backdoor does not affect any of the outputs except for when the trigger is present.

Q1 - “More evaluations on the methods's robustness against mitigations mentioned in section 5.2 limitations would be helpful to understand the method's implication of LLM safety area.”

Please refer to our response to W2 above.

Q2 - “In the same section, the author mentions that adding noises to weights or change weights by finetuning can mitigate the issue. One potential way of showing the method's effectiveness is measuring how much weight change is needed for the method to fail, and how much model performance reduction will the mitigations result in.”

Thank you for this feedback! We did not have time to run these experiments in full; however, we can preliminarily report that adding gaussian noise with a standard deviation of 0.1 does not destroy the backdoor at all. This is significantly larger than the amount of noise needed to completely break the language-modeling part of most LLMs. For instance, on GPT-2-small, we get the following completions:

• Without noise: one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen
• With noise: one two three four five six soc sociosios socapplication socjava soc Party Party socclave soc Mouth

This strongly implies that our compiled trigger module is significantly more robust to noise than the language model itself, further supporting the idea that fine-tuning alone is insufficient to remove the backdoor. We will conduct a more complete experiment for the camera-ready paper.

Q3 - “In section 4 line 186, the paper claims that the addition of backdoor modules cannot be detected from looking at computational graph. Can the authors elaborate more on why it cannot be detected?”

Our modifications are only in the weights, but not in the graph and hence cannot be detected by looking at the graph.

Q4 - “Does the backdoor insertion alter original model behavior/quality? More evaluation on that would be helpful.”

No, it does not alter the original model’s behavior at all! As mentioned on lines 235-239, there is only a 2-128 chance of the model’s behavior changing in response to anything other than the desired trigger. We will make this clearer, and emphasize this in the introduction.

Q5 - “Does the backdoor insertion effectiveness depend on type of trigger message and harmful message?”

No; virtually any type of trigger message and payload are supported.

We would like to thank the reviewer for their feedback and questions addressed below.

W2 - “The threat model is unrealistic, as most commercial models only provide APIs, making it impossible for the authors to insert compiled transformer modules.”

We mention in the abstract and introduction that the threat model we consider is one where the attacker alters open-weight models. In general, we would like to clearly state that the notion of “backdoors” is not applicable to API-based models, as the API maintains the ability to generate any (malicious) outputs at a given point, rendering the concept of an LLM “backdoor” in this context fundamentally incongruous. A specific example of how our backdoor could work is as follows: a bad actor could fine-tune a Llama3 model to be a helpful bash assistant, inserting an encrypted unelicitable backdoor (which could install malware when logged in with a certain user), and publish it on a platform like HuggingFace [1].

We also provide a link to [2], which shows 47,964 variations of open-weight Llama models, some with millions of downloads. In our camera-ready version, we provide better examples of this threat model in practice along with associated figures and explanations in our global rebuttal.

W3 - “Although the authors propose an encrypted backdoor with an unfeasible trigger, the backdoored behaviors can be mitigated through model fine-tuning.” / “Experimental results on encrypted backdoor robustness are lacking, including defense effects such as fine-tuning or pruning.”

While we did not focus on robustness, fine-tuning in particular is unlikely to work against our encrypted backdoor, due to the absence of a gradient signal. Please see our global rebuttal for more details. We discuss these mitigations in the limitations section 6.2., however, there are many ways of “hardening” the construction (e.g. using error correction and obfuscation) which we have not yet explored.

Q1 - “What are the practical applications of encrypted backdoor attacks?”

We refer the reviewer to lines 42-55 of our paper, where we discuss several reasons encrypted backdoors are conceptually important to several directions of AI safety and red-teaming research. Our encrypted backdoor is robust against any polynomial time elicitation technique in both theory and practice, which shows limitations with current directions such as red-teaming [3], eliciting latent knowledge [4], and latent adversarial training [5].

Q2 - “What models and datasets were utilized in the experimental setup?”

We discuss in lines 273-276 of our paper that the experiments are run on the trigger module in order to allow unbounded latent adversarial perturbations. Had we run the experiments on the full model, unbounded latent adversarial perturbations would trivially make the language-modeling part of the model output whatever harmful payload is needed. We also refer the reviewer to footnote 1, which explains this further.

W1 - “Several repeated words and sentences appear, such as "if if" in line 7, and redundancy between lines 65 to 74.”

We appreciate the reviewer pointing these issues out and have fixed such typos in our camera-ready version.

[1] https://huggingface.co/

[2] https://huggingface.co/models?search=llama

[3] Perez et al., "Red Teaming Language Models with Language Models". arXiv:2202.03286 [cs.CL] (2022).

[4] Mallen et al., "Eliciting Latent Knowledge from Quirky Language Models". arXiv:2312.01037 [cs.LG] (2023).

[5] Casper et al., "Defending Against Unforeseen Failure Modes with Latent Adversarial Training". arXiv:2403.05030 [cs.CR] (2024).

To further clarify our points about robustness, we want to highlight that in section 6.2 we discuss techniques for circumventing mitigation methods that include fine-tuning and pruning specifically, and in a way that is independent of unelicitability. Furthermore, part of these techniques work by blocking gradient signals, and our empirical results highlighted in the global rebuttal suggest that our designs are already doing that. Regarding threat models, we believe that the ImpNet paper [6] provides an excellent classification of ML backdoors based on their insertion point (thanks to reviewer QeA1 for bringing it up). In addition to weight-based backdoors like ours, this classification includes many other backdoors that would currently be most relevant in the open-weight setting. If there are any remaining unanswered questions, we are open to discussion.

[6] Clifford et. al. 2022, ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks

Thank you for the detailed and thoughtful feedback!

W1 - “The attacker controls the whole training process”: In fact, the attacker does not need to control the entire training process. Rather, the attacker can insert a backdoor into any pre-trained model, without requiring access to the training data, the training loop, or any part of the training process. Of course, depending on the specific threat model, the stage of backdoor insertion can be either a benefit or a drawback (cf. data poisoning attacks).

Regarding the specific questions:

Q1a: “Could you contrast this work from [1]?” Most importantly, architectural backdoors in [1] lack the unelicitability guarantees. Specifically, their design is susceptible to extracting the trigger and the backdoor behaviour, e.g. by manual analysis or by optimising the model internals similar to latent adversarial training. Furthermore, their design is specialised to convolutional image classifier models, making it hard to extend to our setting of language models. While architectural backdoors have the unique advantage of robustness to retraining, in our setting this would come at a steep cost of being limited to very simple triggers and backdoor behaviours. It was non-trivial for their design to accommodate a simple checkerboard pattern as a trigger and by default their backdoor behaviour is just increased loss. In contrast, we have formal guarantees for unelicitability and the ability to select arbitrarily complex triggers and backdoor behaviours without affecting the performance. We will contrast our designs to [1] in related work.

Q1b: “if an ML expert were to inspect the model definition or the computational graph and look for SHA implementations, would the proposed defense still work?” If the expert only has polynomial time, it would still be unelicitable and provably indistinguishable from a clean model that was using compiled circuits. A heuristic approach could reject all circuits that seem compiled without caring about false negatives. This can be remedied by incorporating standard obfuscation methods, but it is out of scope for this paper.

Q2: “Could you comment on how similar your approach is [to undetectable watermarks] for backdooring, and if it has similar properties as these works?” While undetectable watermarks also achieve provable security in an LM setting, they tackle a meaningfully different task. They work in a black-box setting and always add imperceptible random-looking noise to outputs. This does not transfer to our setting since: 1) white-box access would leak their secret key, making the method inherently unsuitable for unelicitability, 2) undetectable watermarking is incompatible with a noticeable change in the output on a trigger, which is crucial for a backdoor functionality, 3) their method would not allow picking arbitrary non-encrypted triggers and backdoor behaviours.

Q3: “Can you clarify how encoding the 3-SAT problem in the transformer circuit yields an encoding to average-case LWE problems?” A decision version of LWE (DLWE) is in class NP and hence can be reduced to any NP-complete problem, including a decision version of 3-SAT. Therefore a random DLWE instance can be represented as a 3-SAT instance, which we then represent as a circuit. This means that we can ensure that the 3-SAT circuit inside of our network is on average hard, since a solution would yield a solution to the initial DLWE instance. This is in contrast to, for example, random 3-SAT circuits, which have no such average case guarantees and are often quickly solvable with heuristic approaches. We will add this clarification to the paper.

W2: “why is not fine-tuning considered as well” Fine-tuning does not work in general for compiled circuits like ours. While it is a method for testing robustness and not elicitability and hence we consider it out-of-scope for this paper, we have made some observations:

1. There usually is no gradient signal for these circuits because of the discrete nature of the compiled weights and saturated activation functions, e.g. dead ReLUs are sometimes intentionally used to prevent fine-tuning. Since the gradients are zero, the circuit is not changed.
2. Empirically we see that the gradients do not get through our circuits. While adding noise enables some gradient flow, the required noise level is so high that it destroys the LM components before it can affect the backdoor circuit. See the empirical results and additional discussion in the global rebuttal.

Thank you for the thoughtful feedback and recognizing the novelty and usefulness of our encrypted backdoor. We are happy to address the concerns and add corresponding improvements to our paper.

Weakness 1 - On “first white-box unelicitable backdoor”:

We will be more specific and say “first language model backdoor that remains unelicitable even under full white-box access, including its deployment server”. The same can not be said about ImpNet [1]. As they discuss in section 5, their attack can be effective if the defender fails to analyse the compiled model, which can be difficult in practice. However, there are no guarantees that a sufficiently motivated defender could not extract the trigger or payload after decompilation. In fact, an impossibility proof in [2] would pose a significant obstacle to such guarantees. We deliberately use a cryptographic primitive from a narrow class of functions to avoid this obstacle.

Weakness 2 - Trigger mechanism comparison:

There is a subtle difference between ImpNet [1] and NP-complete backdoor trigger mechanisms. Both have the same undetectability guarantees in the black-box setting. However, they differ in the white-box setting, where the NP-complete backdoor has an additional guarantee that the trigger can not be extracted. This is despite it being elicitable, because its elicitation reveals only the existence of the backdoor and the content of its payload, without revealing the trigger. So in a sense ImpNet’s backdoor could be placed between our password-locked and NP-complete backdoors on a scale of hardness. Thank you for this point, it led to a discussion among authors. We will mention this distinction and cite ImpNet as an example.

Weakness 3 - Clarity of SHA footprint:

The footprint is ~21B parameters at 64 rounds of SHA-256. However, just 5 rounds were already unelicitable in practice. Indeed, obfuscation would not help but optimising the subroutines could, e.g. modular addition which dominates the footprint. We investigated more efficient designs and determined that the footprint can be brought down to <900k parameters by using optimised parallel 32-bit adders encoded with 982 MLP parameters each. We describe this implementation in the camera ready appendix. This is practical in comparison to similar cryptographic primitives such as iO, which would take up petabytes.

A typical attack could leave the number of parameters unchanged by first using model pruning, then filling the saved space with the backdoor circuit after fine-tuning for a specialised task. Moreover, as model sizes increase, our backdoors get smaller in relative terms as their size is constant.

Question 1 - “Are "corrupted output" in 4.2 and "harmful payload" in 4.1 the same thing?”

No, but in this context both should be "harmful payload". Harmful payload is the intermediate representation which makes the model produce the corrupted output on a trigger input. We will add the explanation and ensure consistency. Thank you for pointing this out.

Once more, we thank the reviewer for their insightful comments. We hope that our answers here will help to clarify our paper.

[1] Clifford et. al. 2022, ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks

[2] Barak et. al. 2020, On the (Im)possibility of Obfuscating Programs