INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness

Thank you for your reviews! Please refer to our responses below.

Q1: ..It is generally believed and widely proven effective of adopting a multi-agent collaborative system during content generation

Even though multi-agent collaborative systems have been proposed for content generation, it is not trivial to extend this line of research to address the security of code generation.

• Firstly, it is not clear how the current methods could be enhanced with security awareness and subsequently improve the quality of output generations. Earlier work such as [1, 2] showed that simply asking agents to analyze and answer what is wrong with generated code is not always effective. With carefully designed prompts and agent interactions [3, 4], collaborative agents can now generate more functionally correct code.
• Therefore, studying collaborative agents with orthogonal goals such as security awareness still requires further attention. As observed by our additional experimental results (see our global response#2 above), simply applying the current multi-agent methods [5, 6], even with extended additional instructions of security criteria, does not perform so well and is still far from optimal.

Furthermore, we would like to emphasize that our method novelty is not only about the multi-agent collaborative system. We also proposed an innovative multi-critic framework as an internal reasoning module with access to external tools for knowledge grounding. Our strategy can integrate holistic and reliable critic feedback from code execution outputs as well as supporting information snippets through multiple rounds of agent interactions. As demonstrated by our results (see our global response#2 above), our method is more well-rounded with high performance by both safety and helpfulness. Refer to our global response#1 above for a more comprehensive and systematic comparison between INDICT and other related work.

Q2: The experiments only compare INDICT with pure LLMs.…

Following your recommendations, we selected 7 strong baselines from related lines of research, including self-refine, multi-agent, and finetuning methods (see our global response#2 above). For most baselines, we also include a version of the method where additional instructions are given to the model to follow and provide both safety and helpfulness feedback e.g. we instructed models to “focus on both the security and helpfulness of the solution.” For Self-Collab [5], we included these instructions in both analyst and tester agents. For CAMEL [6], we follow the Critic-in-Loop setup as recommended in the paper appendix. Note that for all baseline models, we followed the same generation budgets to fairly compare the results (up to 3 rounds of revision).

From the experiment results (see our global response#2 above), we can observe the SoTA performance of INDICT. While we observed the improvement of baseline methods with additional instructions (marked with the suffix ‘+’), their results are still not as good as INDICT in terms of helpfulness and safety. For finetuning method CodeUltraFeedback, their best model (SFT+DPO fine-tuned) is still far from perfect and could be further optimized. We further integrated the fine-tuned models with INDICT and observed a significant performance boost. We will include these results and more detailed analysis in our revised paper.

Q3: The authors discussed an alternative by using only one LLM to act as a helpfulness critic and security critic at the same time...

We conducted additional ablation analysis to evaluate whether one LLM can act as a critic for both helpfulness and safety at the same time. We combined the previous criteria for security and helpfulness and integrated them into the prompt for this critic agent. From the results (see our global response #3), simply using a single critic agent with dual quality criteria will affect the performance, reducing the safety and helpfulness metrics to 87% and 76% respectively. One possible reason is due to the formulation of the training objectives of LMs, which are not always designed to optimize both security and helpfulness equally (also depending on the post-pretraining stages of LMs e.g. training with RLHF). Our approach enables a more flexible and probably more relaxed application of LLM as a critic agent by:

• (1) decoupling the helpfulness and safety goals and delegating them to individual LM agents; and
• (2) enabling multi-critic collaboration to autonomously develop more holistic and well-rounded critic feedback.

We will include the results and a more detailed analysis in our revised paper.

Q4: What is the meaning of 'GT Safe' and 'GT Unsafe' in Fig.4 (c)?

“GT Safe” and “GT Unsafe” denote the ground-truth secure and insecure code outputs provided by the CVS benchmark. We will explain and make the definitions clearer in our revised paper.

Q5: ...Can you provide the time cost of INDICT? ... once the fine-tuned LLM is released, the inference time costs much less than INDICT

On average, INDICT incurs about 3 to 4x the time cost as compared to pure LLMs. We also want to note that even with fine-tuning, fine-tuned models are far from perfect and still subject to unseen security risks or novel red-teaming prompts during test time. For instance, from our results with the fine-tuning method CodeUltraFeedback (see our global response#3 above), the fine-tuned model is still sub-optimal and can be further improved e.g. using INDICT during inference time (improving the performance from 60% to 73%).

[1] Is self-repair a silver bullet for code generation?

[2] Large language models cannot self-correct reasoning yet

[3] Reflexion: Language Agents with Verbal Reinforcement Learning

[4] AgentCoder: Multiagent-Code Generation with Iterative Testing and Optimisation

[5] Self-collaboration Code Generation via ChatGPT

[6] CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

Thank you for your comments. Please refer to our responses below.

Q1: For the HarmBench evaluation - how have the red-teaming methods been applied with the critics in place? …Was the evaluation also done using completions?...

For the HarmBench benchmark, we followed the original evaluation setup proposed in [1] (see Figure 3 of this work). Specifically, the red-teaming methods like PAP, TAP, or PAIR are applied to the original task prompts (provided in HarmBench). Each red teaming method augments the original task prompts to make it harder for LM actor agents to detect malicious intentions, leading to harmful generation output. The evaluation is then done on the generated completions of the actor agents using the HarmBench AI evaluator. With the critics in place, the evaluation is applied to the actor agent’s completions after it receives the critic feedback and regenerates a new response. Due to the limited space, we described the details of the HarmBench benchmark and references of the red-teaming methods in Appendix D (L996-1003). We will make the experimental setup and evaluation clearer in our revised paper.

Q2: …Could the authors share more details/prompt along with a sample?

Thanks for your comment. A well-rounded summary is beneficial to ensure all the information of the critic feedback is conveyed to the actor agent. We included the example summary prompt in Appendix F (Listing 3 and 8). Other prompts are also included in this Appendix. We also included more qualitative examples with generated dialogues in the attached PDF in our global response above for your reference. We will include these qualitative samples and explain them in more detail in our revised paper.

Q3: I was curious if fine-tuning models (smaller models?) on such generated data would have been a good experiment?...

Thanks for your suggestion. It would be interesting to conduct finetuning experiments on smaller models using generated data. Currently, there is no available training split in the benchmarks and tasks used in this paper. However, synthetic datasets or relevant crowd-sourced datasets could be considered (e.g. from open-domain Github tasks or other code generation benchmarks).

Furthermore, while using fine-tuned models might be cheaper during inference, they will still be subject to unseen security problems or novel red-teaming malicious prompts. In our global response #3 with fine-tuning methods, the best finetuned model is far from perfect and its performance can be further optimized. Our method INDICT can complement these approaches. Our results show that using INDICT can significantly boost the performance of a fine-tuned model, from 60% (SFT+DPO model) to more than 73%.

[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

[2] CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

Q1: …how your method differs from existing actor-critic architectures?...

We provided a systematic and comprehensive comparison between INDICT and related actor-critic approaches such as CodeRL in our global response#1 above. Compared to existing actor-critic methods, INDICT is different in three major aspects:

• (1) INDICT aims to optimize both helpfulness and safety awareness in the generated output code. Most of the current actor-critic approaches are designed with a single criterion (such as functional correctness). Simply extending these methods with additional instructions on safety criteria is sub-optimal (see our results with baseline actor-critic methods in our global response#2 above).
• (2) INDICT integrates critic with knowledge grounding from external tools to create more reliable feedback to the actor agent. Most current methods only use code test results as the only external feedback to improve the quality of output code.
• (3) To implement (1) and (2), we enhanced the existing actor-critic framework with a multi-critic and tool-enabled collaboration approach. This approach can autonomously generate more reliable and holistic feedback for both the safety and helpfulness of output code generation.

We will include a more detailed comparison in our revised paper.

Q2: In this framework, the critics use data from web searches, Wikipedia, and OpenAI, which raises my concerns about data leakage…Because of this, while the framework performs well on benchmarks, I’m concerned about the generalizability of this framework.

Thanks for raising this concern. We want to highlight that our use of external tools is only for querying relevant information snippets to supplement the critic feedback, not to directly find a solution to a given task. We do not provide any additional information to the critic to generate queries e.g. no information about the related CWEs or security issues, so the critic would have to independently decide what information to choose to search. We conducted an analysis of the generated tool queries by the critic agents and found that on average, for each task, words that are found in queries only overlap with less than 5% of the total words in the original task description. Therefore, the data leakage is very minimal.

Furthermore, from our experiment results (Table 2 in the paper), even without access to external tools, our approach with a multi-critic collaboration system still leads to significant performance gains in base models like CodeLlama or CommandR. When stronger and more recent GPT models are used (e.g. results with GPT4o-mini in our global response#2), the direct generation result is still rather low i.e. models are not heavily exposed to the test data, and more improvement could be done with methods like INDICT. We will include a more detailed discussion in our revised paper.

Q3: Besides, as the paper mentioned, the code execution could invoke unexpected consequences…

Thank you for your suggestion. Sandbox environment is a technical engineering feature that we would like to integrate with INDICT. However, the novelty of our method should still be guaranteed. We will describe in more detail the requirement for a sandbox environment for secure code execution in our revised paper.

Q4: How do the safety and helpfulness critics contribute to the overall performance of the framework? Are there any specific examples or case studies you can provide to illustrate their impact?

As demonstrated in our ablation experiments in the paper (see Table 3), using individual component critics leads to sub-optimal performance and is not as good as our method with a multi-critic collaboration system. To illustrate the benefits of our approach, we included a qualitative example in the PDF attached to our global response above (see Figure 1).

Compared to the baseline methods, our approach can simultaneously improve both the helpfulness and safety of the output code generation. Specifically, given a relevant information snippet by the safety critic (about the hashing method SHA-256), our actor agent correctly revised the code with a more secure hashing method, avoiding using MD5 hashing and the common security risk CWE-328. At the same time, our generated code is generally more helpful with properly modularized functions implementing supporting features such as input validations. As we noted, this feature has generally emerged in code solutions by collaborative agent systems like CAMEL and INDICT. We will explain the qualitative samples in more detail in our revised paper.

Thank you for your comments! Please refer to our responses below to your questions.

Q1: …In the evaluation, INDICT is only compared with vanilla LLMs…

Following your recommendations, we selected 7 strong baselines from different research lines (see our global response #2 above). We also include a version where additional instructions are given to models to provide both safety and helpfulness critics e.g. instruct models to “focus on both the security and helpfulness of the solution.” For multi-agent methods, we included these instructions in all agents (analyst, tester, etc.) or introduced a new critic agent (as recommended in CAMEL's appendix). Note that for all baseline models, we followed similar generation budgets to fairly compare the results (up to 3 rounds of revision). Our results demonstrate the SoTA performance of INDICT against all the baselines. While we observed the improvement of baseline methods with additional instructions (marked with the suffix ‘+’), their results are sub-optimal. We will include all results and a more detailed analysis in our final revision of the paper.

Q2: …The authors should compare INDICT vs. INDICT without a summarizer.

We conducted a new ablation to evaluate the summarizer (see our global response#3 above). We simply removed the summarizer and let the actor agent receive the full dialogue history. From the results, we noticed the performance degraded to 87% and 72% in safety and helpfulness. This happens probably due to the much longer context of the dialogue history, affecting the actor agent to capture all critic feedback from this history and generate new code. This model variant also incurs more computation due to the long context of the dialogues. We will include the results and more detailed analysis in our revised paper.

Q3: How much improvement does the thought-action-observation process achieve compared with existing RAG and tool-enhanced methods?

We conducted additional experiments (see our global response#3 above) with two simple variants: (1) RAG-based critics which use the original task description to retrieve relevant knowledge and generate augmented critiques; and (2) tool-based critics which directly use external tools to query “what is wrong with the solution in terms of its <security/functionality>?”; the query output is then treated as a critic feedback. The results show that these variants are inferior to our proposed framework. We found that the queries in these methods are often too vague or ambiguous to search for meaningful information snippets.

Q4: Does INDICT use zero-shot prompting or few-shot prompting in each step? …

INDICT uses zero-shot prompting in each step. We prompt the critic agent to condition the current critique and generate a unique query to obtain more knowledge. We extract the search keywords following our instruction templates e.g. in the form of 'Search[keyword]'. For generating code snippets, we prompt the model similarly but ask the model to wrap the output code in ```. Note that the current prompts in the Appendix (Listing 4 and 5) are for tool-finetuned models like CommandR which automatically generates tool parameters given a tool calling function definition. We will include more prompts for these steps for other standard models in our revised paper.

Q5: ...the authors should consider performing a manual analysis of a small set of samples.

To supplement the GPT-based results in helpfulness, we manually evaluated a small set of 40 samples from CyberSecEval1 (5 random samples per language). From the Table below, we found that even though GPT predicted results are often higher than human evaluation, the observation of INDICT's better performance still stands against the baselines.

Note that the security metric is done by the code analyzer from [1] which already demonstrated a high level of accuracy in detecting security issues.

Q6: …provide deep insights like why INDICT performs better than other methods and where INDICT fails.

From our additional results (see our global response#2 and #3 above), we noticed that the best baseline models (Self-repair+ or Reflexion+) are as good as an ablation method with basic tool-based critiques where the external tool is vaguely queried: “What is wrong with the current solution?”. This shows that the quality of the critic feedback in the current methods is not good enough, simply focusing on shallow reviews of generated solutions. In INDICT, we enable the model to analyze and choose what information snippets to query and how to integrate this information into their critiques (e.g. queries about an uncommon security term or certain coding best practices). Combined with our multi-critic collaboration approach, INDICT demonstrates strong empirical results against other methods.

For qualitative results on how INDICT performs better and example failure scenarios, please refer to our attached PDF in the global response above. We will include more analysis in our revised paper.

Q7: ... how INDICT differs from existing self-critic approaches and multi-agent frameworks.

Please refer to our global response#1 for a systematic comparison of INDICT and the related work, including the self-critic approaches and multi-agent frameworks. We reviewed each method by the following features: helpfulness or safety-based qualities of generation outputs, execution feedback (execution of output code if applicable), tool-enhanced feedback (access to external tools like web search), multi-critic collaboration (engage multiple LM agents for critic generation) and supervision free (no training data required). We will include the full details of our comparison in the revised paper.

Dear Reviewer GLux,

We hope our rebuttal response has addressed your concerns about the paper. As the authors-reviewers discussion will end in a few days, please do let us know early if you still have any questions or need further clarification.

Regards,

Dear Reviewers,

We hope our rebuttal has addressed your concerns sufficiently. Before the authors-reviewers discussion period ends on 13 Aug, please let us know if any additional information or clarification is still needed to improve the paper.

Regards,