Code-of-thought prompting: Probing AI Safety with Code

There are a couple of claims that are made that are unsupported in the paper, which undermines the soundness of the paper.

Most importantly, the authors conclude that not only are (1) existing safety and alignments methods unable to handle something like CoDoT, but that (2) ‘recent safety and alignment efforts have regressed LLMs and (3) inadvertently introduced safety backdoors and blind spots’ (abstract). The data and discussion presented in the paper support (1) but little support is given to (2) and (3).

Note that claims (2) and (3) are not simply rhetoric flourishes – and even if they were, they are inaccurate and should be edited out – but a core part of what the authors claim to have shown. For instance, in the introduction they write that ‘we reveal that recent research efforts largely fail to address the root cause of AI safety and alignment concerns’ (p. 1), although this claim is not further developed, and that they ‘demonstrate that state-of-the-art models trained on novel safety measures catastrophically fail’, where there is an implied causal link between the safety measure and what makes the outputs increase in toxicity. See also p. 3: ‘modern safety and alignment efforts might have inadvertently introduced safety backdoors and blindspots’ and the conclusion, ‘Our work presents strong evidence that current safety and alignment efforts in Large Language Models (LLMs) are insufficient and may even be introducing unforeseen vulnerabilities’ (p. 10). Yet, it is unclear where in the paper the case is made for making this causal claim.

One place where support for the causal claim could possibly be found is on p. 7 in the discussion of the differences between GPT-3.5 Turbo and GPT-4 Turbo. However, that GPT-4 is ‘more toxic’ than GPT-3.5 doesn’t mean that it is the safety and alignment methods that are the cause of the change. We can grant that there is a significant difference between 3.5 and 4 – that’s what CoDoT shows us – but there were a number of changes between the LLMs that could plausibly impact the toxicity e.g. the introduction of multimodality in the inputs. The authors’ attribution of the toxic change to safety mechanisms seems to be merely stated and is under-supported. At the very least, the authors need to consider other ways in which the models have changed, and give reasons for thinking that the problem is arising specifically from the safety-related changes. Intuitively, the authors may have a point here. To address this comment, then, the authors could conduct a more comprehensive analysis of the differences between GPT-3.5 and GPT-4, examining various ways in which they might differ, such as in training data, model architecture, and other non-safety related modifications between versions. If after such a comprehensive analysis the same conclusion is reached, then it will be convincing.

Even in a weakened form, as a hypothesis for further testing, it is difficult to work out what the support for the hypothesis is. The authors themselves seem to identify an alternative explanation for why LLMs like GPT-4 could perform worse than their predecessors, namely the use of MoE architecture (p. 8). Yet, with this option on the table, why should we conclude that the main problem are the safety-related efforts?

To address this general concern about a lack of support for (2) and (3), the authors could provide more evidence or argumentation to support the claims. Alternatively, the authors could revise their claims and how the paper is framed to be more moderate and better supported by the paper as it stands. Related to this latter option, I think a much better conclusion and framing of the paper can be found in the authors’ own words on p. 2: ‘With most proposed safety mechanisms fine-tuned to ensure safety for only a certain input distribution, for example, certain types of natural language queries for select languages, CoDoT can reveal blind spots for novel input distributions like structured or code-based queries’. This is an important conclusion and illustrates the power of CoDoT. I don’t think that the stronger claims are necessary, so it is a pity that they are there, made a central part of the paper, and yet are under-supported.

Another place where I would have liked more justification is on p. 6: ‘For the toxic induction task, CoDoT prompting generally proved more effective than Instruction prompting’. On what basis is this claimed? The support seems to be in Table 2, N=1. But this could also be interpreted more weakly, as only 2 of the LLMs show a substantial increase in toxicity (GPT-4 Turbo, WizardLM 2), one a decrease in toxicity (GPT-3.5 Turbo), and the remaining two only a minor increase (at least, without looking further at significance calculations and all that). So, further justification would be welcomed here. Could the authors provide a detailed statistical analysis of the results in Table 2 to support the claim of CoDoT's effectiveness vs Instruction prompting?

I found the submission very confusing and, quite frankly, disturbing and distasteful.

There are certainly massive red flags in the way results are presented. We get it, LLMs can output profanities. Continuously doubling down on the "f***" and "d***" type of profanities is distasteful, and given that they serve no purpose, they are also unwarranted. The authors basically propose a method to automate malicious intents but propose no remediation of the problem. All this under the promise that the approach will transform natural language text into "sophisticated programs" in which sentences' "intent is clear" (Line 053). These points are never demonstrated or evaluated. Case in point: there is a "make_toxic" (Fig 1) or "make_more_toxic()" (Tab 1) method that is supposedly the point where sophisticated programs with clear intents are created, but this point is never explained.

Please seriously reconsider the presentation of your results for the sake of the scientific community. This is not a paper I would wholeheartedly recommend to my students.

On a minor note: numerous imprecisions make it hard to evaluate this work. For example, "toxic" is never defined, and no related work is cited. No other forms of AI safety are demonstrated, and it is left to the reader to make a link between profanities and AI safety. Much clearer positioning would be needed in a proper scientific work.

• The paper studies one chosen template for evaluating the induction and amplification of toxicity using code of thought, making it somewhat limited in its scope. It would have been nice to investigate at least one other template to ensure that the bypassing of safety filters is a more commonly observed phenomenon and not restricted to this particular template.
• In Section 4.1 containing toxicity analysis, a comparison between CoDoT prompting and instructions is provided, however, demonstrating the impact of CoDoT prompting with $include swearwords=False$ would have made the results more concrete. This ablation would have provided an indication of whether the toxicity in the generated results is solely the result of using swearwords, or whether this is more due to explicitly requiring swearwords in the results. In other words, this ablation would have disentangled the impact of code-of-thought from the $include swearwords$ argument.
• In the results shown in Table 2, GPT-3.5 Turbo shows a higher average toxicity score than CoDot ($N = 1$) but fewer toxic conversations. It would be helpful to know why we see this discrepancy, and discuss whether this can be attributed to imperfect toxicity prediction by the Perspective API. Furthermore, since the indication of whether a response is toxic or not depends on the score given by the Perspective API and whether or not it is above the 0.5 threshold, some discussion around how sensitive the scoring mechanism is, might be useful (even if it is in the Appendix).
• It would be useful to also display standard error along with mean toxicity scores and number of toxic conversations in Table 2.
• The mean toxicity score for standard prompting could be included in the results (Figure 3, for number of amplifications = 0 and Table 2) to gauge the impact of each attack mechanism (instruction-based vs. CoDoT prompting).

1. In terms of theory, the author don’t have a explanation for this iterative deterioration of model outputs. For example why does all the deterioration of models seems to stall at iteration 3. I would suggest the author to explore the mechanism for the deterioration behavior in FIg 3. The math is of little actual meaning. f_{\phi} is the LLM’s output input mapping function. Or if the author find that CODOT make the LLM output content more toxic, is there any way to prevent this through the method of CODOT?

2. No ablation study on the pseudo-code structure” make_more_toxic("{text}", include_swearwords = True)”.

3. The reason for selecting the models is not stated. the authors used 6 totally different LLM. Beside the size, the stricture, a lot things can change with the variance of the models. Instead , I would suggest using a series of model families, like llama with different sizes Or the Mixtral family to study. If the authors indeed intend to work with a variety of models, 6 models are not enough.