LLM-based Multi-Agents System Attack via Continuous Optimization with Discrete Efficient Search

Question1 and Weakness 1 will be in another thread. Thanks for all the comments and questions.

Response to Weakness 2: More testing in black-box.

The method is optimization-based, so we mainly evaluate on the white box settings, as other optimization based evaluation as in GCG and ADC.

Response to Weakness 3: Real-world MAS applications.

For example, in the future, the vehicles can be connected to each other in a transportation system, and each with a VLM inside. They can communicate with each other. Then, a single interruption in one vehicle can lead to all vehicles do the same, harmful and precise outputs.

Response to Weakness 4 and Question 2: How RAG affects the attack.

If the agent can retrieve outside data from a database, then the attack interface becomes more vulnerable than the current settings since the harmful input can be long and can hide in retrieved documents. But how to retrieve back these harmful documents retains another open research problem. If the retrieved document is long and harmless, then it may decrease the attack success rate. Overall, this is another task and investigating this problem can stand alone as another interesting work.

Response to Weakness 5: Application-specific contexts.

The proposed method can generalize to different datasets and different contexts. Once you optimize on a certain domain-specific train set, it can generalize to unseen data following the same distribution.

**Responese to Question 3: *Extended to multimodal agents or task-specific? ***

The proposed method mainly addresses the text token optimization problem, which is much harder than adversarial image optimization. If, in the multimodal agent setting, the attacker can input adversarial images, then the settings are much easier than the settings in this paper. Previous multiple adversarial methods in image domain can be used.

As for task-specific, if it is all in text, and the context are not too long, then the proposed method can be well extended to these scenarios.

Responeses to Question 4: Different MAS topologies

If the topology (the order of the agent communication) is predefined and known to the attackers, then the case become easier and attack success rate (ASR) should increase. Otherwise, if the topology is not known, and if some agents communicate less frequently with others, then ASR may decrease until these agents are successfully infected.

Responeses to Question 5: Countermeasures or alignment techniques

For harmful content, using a content filter can reduce ASR. However, a filter usually suffers from over refusal issues, which may influence the performance of completing regular tasks.

For stealthy behavior such as saving a confidencial file, then training another agent for identifying unsafe code can be helpful.

Response to Weakness1 and Question 1: A concrete example walk-through for Algorithm 1 and Algorithm 2 to better clarify the optimization process?

Let $z_1, z_2, …, z_n$ denote the embedding vector to be optimized. For all i, $z_i$ is a dense $d$-dimensional vector where $d$ is the embedding size of the LLM. After optimization, we use the arg max index of each embedding vector for the optimized token.

At the optimization step $t$, we first compute the gradient and then the momentum buffer of all embedding vectors ${z_i}_{i=1}^n$.

Algorithm 1 described how to update ${z_i}_{i=1}^n$ according to the momentum buffer. The overall idea is to update only one index of one embedding vector (and then project the embedding vector to the probability space again). The question is to choose which index of which embedding vector. The answer from Algorithm 1 is to try some candidates and choose the candidate with best loss:

Generate one candidate: we uniformly sample j from [1, 2, …, n]. Thus for this candidate, we choose the j-th embedding vector. And then we choose an index, say k, from the j-th embedding vector, among the indexes with top-K large gradient: [index_i, index_2, …, index_K] = Top_k(gradient(z_j)[i]) k ~ Uniform ([index_i, index_2, …, index_K]) Thus for this candidate, we only update the k-th index of the j-th embedding vector using the momentum buffer.

Choose the candidate with the best loss. For each candidate, we know how the embedding vectors are updated, and can compute the optimization loss after update. We choose the candidate with the lowest updated loss as the final update of this step because it will go to a lower loss.

Convert the updated embedding vector into target sparsity. After each update step, the sparsity of the embedding vector may increase. However, we want a very sparse embedding vector in the end. At step $t$, we follow ADC to convert the updated embedding vector into a pre-computed sparsity. We use the same convert algorithm as in ADC.

Evaluation. As we mentioned, we use the arg max index of each embedding vector for the optimized token. Then at each step, we evaluate if the (discrete) language token (from the arg max index of each embedding vector) can jailbreak or not. If yes, the optimization is stopped. If not, the optimization will continue until all $T$ optimization steps are executed.

Thanks for the time and reviews.

Response to Weakness 1: Intuitive explanation

Overall intuition: the method optimizes a harmful, repetitive suffix so that input (harmful words + suffix) produces output (harmful words + suffix), enabling precise harmful content propagation across agent networks.

Intuition for the optimization: CODES keep a continuous optimizable variable z, and search for discrete candidates for evaluation.

Compared with optimization in discrete space, it’s more training stable. (GCG optimizes all tokens in discrete), ADC, GDBA and other projected gradient descent (PGD)-based methods project from discrete to continuous per step, bouncing around. CODES always optimize in continuous (as in line 129 and line 137).

More generalizable. Compared to PGD, CODES can maintain a large batch size by search (PGD can only do batchsize=1) per initialization. Thus, CODES can better escape from a local minimum, reach a lower loss. (as in line 127 and line 141).

Response to Weakness 2: lacks comparison with proper baseline and lacks novelty

Thanks for mentioning these related works [1,2,3,4], we will add them properly in the reference.

[1] Red-Teaming LLM Multi-Agent Systems via Communication Attacks

[2] Combating Adversarial Attacks with Multi-Agent Debate

[3] MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations

[4] Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate

But we have major differences with all these papers.

Briefly, all these attacks are not a formal mathematical optimization method, but rather a heuristic feedback-based approach. They can not precisely control the output, making their attacks less harmful and capable.

In detail:

1. Threat model setting. [1,2,3,4] all based on debate. They require a fully controlled harmful agent, and require this agent to interact with other agents multiple times. But we only need to interact with the system once. We do not need a fully controlled agent. Our setting is harder and more practical.
2. Method. [1,2,3,4] are all based on prompting. Ours are based on optimization on tokens. Their methods are more like template-based methods as we discussed in line 43-52. For example, [1,2] use agents debate to reduce prompt harmfulness, [3,4] both involves ‘Iterative Refinement’, which is to prompt an llm to rewrite the harmful prompt to be less obvious.

Response to Weakness 2: just an optimization-based method and lacks novelty.

Previous optimization-based methods don’t work in our difficult agent setting (ASR=0 in tab.1).

CODES makes improvements on 1) Improved algorithm, combining search with optimization. 2) Improved objective: implicit weighted objective. 3) Improved ensemble: scale, order and diversity.

Thank you for your response. I agree with your point that white box attacks, particularly those designed for exact matches, are naturally constrained to targeted outputs. For example, in your setup, repeated occurrences of "damn it" are counted as successful attacks.

However, in the broader context of jailbreak attacks, attack success is often defined more loosely. Rather than relying on specific target tokens like "sure, xxx," many evaluations consider any affirmative response that does not include a clear refusal as a success. I don’t believe exact match is the most appropriate criterion for assessing jailbreak behavior. I also question the utility of applying this criterion to HatefulBench, where the goal is to determine whether the model reproduces specific hateful phrases. In jailbreak settings, we should not dismiss alternative approaches simply because they do not produce the exact target output that a given algorithm is optimized for.

While I appreciate the strengths of your approach, my concerns remain. I believe Reviewer 4 articulates my perspective more accurately, particularly regarding black box robustness and real-world applicability. My concerns about using exact match for measuring attack success are closely tied to the exclusive focus on HatefulBench, rather than more widely used benchmarks like AdvBench.

I do appreciate that the paper proposes new MAS setups across three scenarios, tests them on 100 target phrases drawn from the proposed HatefulBench, and introduces CODES (an improved version of ADC) using exact match for evaluation. However, I feel that each of these contributions would benefit from being more clearly explained and evaluated independently.

For example, if the main contribution is that your approach improves upon ADC, it would be more convincing to evaluate it on AdvBench or HarmBench as ADC does (or use AdvBench or HarmBech with 3 MAS senarios). If the contribution is HatefulBench itself, then a direct comparison with other harmful response datasets would be more appropriate. Similarly, if the key contribution is the three MAS scenarios, it would help to clarify their advantages and usefulness in comparison to prior MAS attack papers.

As it stands, the paper appears to pursue multiple directions at once, but each thread feels not fully justified with experiments to me.

Thanks for the time and the review.

Response to Weakness 1: Mitigation strategies.

For mitigating harmful content (scenarios 1 and 3), using a content filter can mitigate the harmfulness. However, a filter usually suffers from over-refusal issues, which can harm the performance of MAS.

For mitigating stealthy behavior, such as saving a confidential file (scenario 2), training another agent for identifying unsafe code or behavior can be helpful.

Response to Weakness 2: Analysis of the qualitative severity.

The evaluation of qualitative severity of harmful content is hard and inaccurate [1,2]. Qualitative severity can be evaluated by human annotators, and we will add a small-scale human evaluation later using a survey. [1] Rauh, Maribeth, et al. "Gaps in the Safety Evaluation of Generative AI.” [2] Burden, John. "Evaluating ai evaluation: Perils and prospects."

Response to Weakness 3: Small-scale, simulated agent groups.

The development of AI agents is fast, currently, the agent groups usually contain fewer than 10 agents, such as ChatDev uses 7 agents for developing software, and AgentVerse has 9 agents for task-solving. There will be a larger scale of agents' communication in the future, say, if all the vehicles are connected within a transportation network, they will have a very sophisticated communication algorithm. We mainly investigate the random non-predefined order in this paper. We’ll keep this work updated and see how it performs in large-scale real-world settings.

Thanks for the time and the review.

Response to Weakness 1: Rely on direct message passing between agents.

If the agent is asked to summarize the previous questions, then we can add these cases also in the optimization process. Such as, we can add one loss term aiming at Input: ‘Summarize the previous information. Some context.. Harmful content + suffix’; Output: Harmful content + suffix. This summary scenario is a little bit similar to scenario 3, where the second agent is asked to rewrite the input. We can add these cases to the optimization ensemble cases.

Response to Question 1: Statistical robustness checks.

Following previous works (GCG, ADC), we keep the LLM token sampling strategy as deterministic (greedy sampling), and the random seed as a fixed number. Based on our observation, the ASR isn’t really sensitive to the random seed, but the time for finding the adversarial suffix may vary.

Response to Question 2: Hyperparameters in evaluation settings, how realistic is the evaluation?

Rounds for N_i. In the ideal case where once one infected agent talks to a non-infected agent, the non-infected agent gets infected, then the expected rounds for infecting all agents is 33662/3825 ≈ 8.80. However, since the suffixes are not that effective, and the randomly sampled order is not predefined, the average number of rounds for a successful attack will take around 30 rounds. So we set the number for N_i to 50.

Ensemble scale. We did an ablation on the ensemble scale in tab.2, we find that even though a larger number of ensembles can lead to better performance, it kind of saturates. So 40 is a reasonable number for balancing the cost and the attack success rate.

Overall, the evaluation settings are hard, with random order, random data sampled from Wikitext. The real-world settings for some multi-agent systems (MAS) now, is more similar to a predefined order. For example, ChatDev defines that the order is CEO agent first, then coder agent, UI designer, agent, and there will be multiple rounds between code review agent and coder/UI agent. Overall, the proposed evaluation setting is harder than some of the current multi-agent systems.

Response to Question 3: Runtime for model of 100B+

Since the method can be transferred to black-box models such as GPT-o1, we suggest using transferability to attack larger models. On a larger model, the optimization time as well as the GPU memory will be costly.