Adversaries Can Misuse Combinations of Safe Models

Missing details on why this is working. While the paper clearly explains that this dividing tasks between frontier and weak model is working, the paper lacks fundamental knowledge on why this is happening. In particular, this question is exacerbated by the fact that different domains held very different results. In particular, the success in the image domain might due to poor robust training of multi-model models (but this is a guess which is neither confirmed or denied by the paper). Also, from the paper is not clear how the prompts are divided for each task, and which are the ones that worked more for the domain of choice. Maybe, this could hint specific token patterns that could be useful to harden model against these strategies.

Unrealistic images are jailbreak? Connected to the first point, it might be possible that the unrealistic setting is the jailbreaking for image generators (like writing grandma bedtime stories for text generators). The authors discussed the fact that jailbreaking will work better than this technique, but they do not investigate (or discuss) whether some of their results could be assimilated to the latter.

Automation works poorly, and there is not so much discussion about this. The authors show to break down content automatically, which, in theory, should attain better results thanks to some optimization algorithm used to prioritize particular parts of the prompts. However, this is done through an LLM, which might include bias in the evaluation towards the inability of success of the attack. Hence, the authors should expand this part, by creating an optimization process that is able to properly optimize (or better divide) the prompts. Or, if this is too complicated, at least mention the fact that this is not a really "automated" approach, but an heuristic division done with another LLM. Nevertheless, I believe dividing the prompt with an LLM is a good idea, but the authors should use that as a baseline.

The problem discussed is very interesting and deserves more attention from researchers and developers from both academia and industry on considering model safety in practice. It is worth noting the effort made by the authors in designing the experimental pipeline. However, I believe there is still room for improvement, especially in making claims/definitions (e.g., safe models) more concrete and clear and presenting the experiment setup and results in more dimensions. More details can be found below.

• Lack of clarity
  • In the title, the authors claim that "adversaries can misuse combinations of safe models", which, to my understanding, is claiming that those single models will refuse to generate malicious content if they are prompted to do so directly. However, later in the paper (Page 5, Quantitative results), when the authors analyze the results of the vulnerable code generation task (i.e., low success rates of single model baselines), they claim "we additionally empirically verify our intuition that frontier models fail because they refuse to generate outputs, while weak models fail due to lack of capability." Thus, the weaker models fail to complete the tasks not because they are safe but simply due to a lack of capability. This contradicts the central base assumption that the models are safe individually. It also leads to an assumption that the combination of models can be misused largely because the weaker models are unsafe; thus, the problem here can be reduced to improving the safety guarantee of weaker models.
  • In section 3 Threat model under "Strength of the adversary", the authors claim that "We argue that adversaries are bounded in two natural settings: (i)...; and (ii)...". I get confused on these two settings -- I am assuming (i) is the lower bound and (ii) is the upper bound. What are the examples when "the misuse needs to be automated"? Based on the previous text, the adversary is a human attacker, then what does it mean to have a human "specify a single strategy for the adversary"? Does it mean human adversary has to generate malicious output with models when "models are superhuman"? Moreover, how does the scenario studied in the paper fit in? More clarification on this would be great.
• Experiments section -- more data/statistics in addition to success rates should be included
  • It's great to consider two task decomposition methods: manual and automated. But there is no discussion on comparing those two. It would be interesting to see what is the overhead for those, and what are the corresponding pros and cons. Do you draw the boundary based on different types of tasks (i.e., manual: vulnerable code generation & explicit image generation; automated: malicious python scripts for hacking & simulated personalized manipulation)? Is there a more systematic way of doing so with some metrics (e.g., if the weaker model fails to do it within a certain number of trials, etc.)?
  • Hyperparameter selection: The authors mention in the Threat Model section that "We assume the adversary fails after N unsuccessful rounds." However, there is no further discussion on how to select the hyperparameter N, and if it has an influence on the results. I assume N should be an important factor in determining the attack success rate.
  • Evaluation with GPT-4: As the authors claimed in the Limitations section, the usage of GPT-4 as an evaluator is the most efficient way. But since the purpose itself here is to evaluate LLMs, instead of using another LLM to accomplish that, it would be great if there could be a smaller scale human/manual evaluation, acting as a baseline to compare how good GPT-4 is at evaluating.
  • Show more concrete examples of the outputs (e.g., NEFARIOUS TASK, HISTORY, RELATED TASKS, etc.) of the model after different stages with the given prompts in the paper/appendix.
• Related Work section: In addition to listing SOTA works, the authors only briefly mentioned how the work is related to jailbreak attacks and marginal risks, but it would be more interesting to see more discussion on how this work is related to LLM agents colluding with each other (beyond only listing them), which are more closely related to the study.
• Typos
  • Related Work, page 3, last paragraph: "We expand the set of tasks that can adversaries can accomplish ...".
  • Quantitative results, page 8, last paragraph: "...combining a model with a either a more ...".
  • Simulated personalized manipulation, page 9, first paragraph: "to construct such tweets in order to sway user opinions.".
  • Appendix, page 23, first paragraph: "This prompt is is different ...".

About methodology

• The manual decomposition approach is a bit heuristic. Is there a principled way that is generalizable to different use cases?
• For the automatic decomposition method, there is no guarantee that the "related tasks" generated by the weak model are benign.
• Also, there is no guarantee that the "related tasks" generated by the weak model are effective or diverse enough. For example, for medical questions (e.g. in MedQA), the weak model may not be able to generate high-quality questions.

About evaluation

• The datasets used to evaluate the proposed approach are self-generated. The details for the data selection are not fully released. There may be bias.
• Missing ablation study on the effect of the number of in-context demonstrations for the automated decomposition approach.
• Missing ablation study on the effect of the difficulty of questions. There is no metric for the difficulty of the questions.

1. The paper would benefit from clearer storytelling, particularly around when adversaries might use multiple models in applications versus solely for red-teaming evaluations.

For example, it’s difficult to imagine a scenario where two models with the same function but different capacities would be needed—such as using two text-to-image generative models (including the Stable Diffusion model) for image generation, which is one of the examples showed in this paper. The method proposed in this paper could apply in theory, but in most cases, I don’t see an application where two Stable Diffusion models would be used to generate a single image.

2. The paper feels more like a technical report; the automated method doesn’t seem easily generalizable to different tasks. Each task requires a separate design, which limits the paper's applicability.

For example, in Section 5, the paper mentions "automated decomposition for generating malicious Python scripts used for hacking and personalized manipulation in a synthetic setup." However, these two scenarios actually rely on human-designed templates, meaning the approach isn’t fully automated, which could further limit its applicability.

3. The scenario seems more like a multi-agent study. In this context, more agent-based methods could be used as baselines, and the storyline could be made more practical.