Black-Box Adversarial Attacks on LLM-Based Code Completion

We would like to thank the reviewer for their insightful review and constructive comments. We address their comments, questions, and concerns below:

Q1: Does the simplicity of the optimization procedure of INSEC limit its practical relevance as an attack on the security of code completion engines?

No. In fact, we believe it increases the severity, since it makes the attack more accessible and easier for real-world attackers to execute.

Q2: Could you provide results on other models?

The INSEC attack is designed for Fill-in-The-Middle completion—the format that is used in IDE-integrated code assistants. We evaluated INSEC against the state-of-the-art and industry standard in this domain, Copilot, and successfully broke its self-claimed safeguards to actively prevent vulnerable completions [1]. We evaluate other open-weight state-of-the-art completion models and the latest completion-API compatible model by OpenAI, gpt-3.5-turbo-instruct. Other recent models, such as the suggested GPT-4, are chat models, and not suitable for code completion and INSEC.

Q3: Could anomaly detection tools detect the attacked inputs?

We believe that the attack is quite stealthy. The injection affects only the prompt to the language model, which would need to be monitored by such a tool. Further, the assumption that such detection tools are present in production setups may be unrealistic, since they are difficult to set up and expensive to maintain [2]. We have added a general discussion on defenses in Appendix E of the paper and refer to Q2 of reviewer Jqc8 for additional discussion of possible defenses.

Q4: How do you form the attack string in the “optimization only” setting?

We utilize the random initialization strategy for 5 attack strings in the optimization-only setting. We have added this detail in Section 5.3.

Q5: Why are there multiple optimization methods implemented in the code but no comparison among them in the paper?

Most of those implemented methods are based on white-box access and were removed from the final version of the paper, which focuses on black-box attacks. The only alternative mutation method differs by deciding to mutate characters individually per character, resulting in an exponentially decreasing probability to mutate several characters, compared to INSEC, which samples the amount of characters to change uniformly. We found that these methods perform overall similarly, confirming that INSEC is a generic and effective attack. Most importantly, and adding to the Q1, both methods are realistic choices for attackers, as they are simple to implement. We summarize our findings in Appendix C.

References
[1] https://github.blog/ai-and-ml/github-copilot/github-copilot-now-has-a-better-ai-model-and-new-capabilities/#filtering-out-security-vulnerabilities-with-a-new-ai-system
[2] Ita Ryan, Utz Roedig, and Klaas-Jan Stol. 2023. Unhelpful Assumptions in Software Security Research. CCS 2023

We would like to kindly remind the reviewer that the discussion period is approaching its end this Tuesday. We believe to have provided an extensive rebuttal to the points and questions raised by the reviewer. In case the reviewer has any remaining concerns or questions that support their leaning towards rejection, we are eager to provide further clarifications.

Q6: Why are the vul_ratio and func_rate@1 for StarCoder 3B different in Fig 2 and 3a?

Because Figure 3a is an ablation setting, we evaluate it on the validation split of our vulnerability and functional correctness datasets. In Figure 2, the evaluation results on the test split are presented.

Q7: Does the “without comment” setting impact code correctness?

No. The code with injected tokens is only used for generating the output of the model, the injection is not present in any compiled or executed code.

Q8: Did you investigate generalizability across models?

We thank the reviewer for this interesting question. We investigate this by applying the strings optimized for StarCoder-3b and CodeLlama-7b and use them when attacking GPT-3.5. The attack generalizes between models, increasing the vulnerability ratio of GPT-3.5 to 55% and 59%, respectively. Meanwhile, targeted optimization outperforms the transferred attack on both the original models StarCoder-3B and CodeLlama-7B (80% and 82% respectively) and the transferred model GPT-3.5 (76%). We included these findings in Appendix C of the paper.

Q9: What would be possible defense strategies? Did you consider adaptive attacks to these?

We add a discussion on possible defenses in Appendix E, including scrubbing of comments and deploying static analysis in production environments. In the case of removing comments from code prompts, we suspect that such an invasive change could heavily impact functional correctness, as developers often annotate code with comments as instructions for LLM completion. We confirm this experimentally, by removing all comments from the HumanEval dataset before submitting them to the LLM for completion and observe a func_rate@1 of only 89.6% on average compared to the commented code, resulting in a similar impact to INSEC. Regarding static analysis, we find that, while effective, such tools are generally difficult and expensive to set up and maintain, while developers are often constrained by their environment and budget [8].

We did not explicitly consider adaptive attacks in the presented paper. One can imagine however that due to the general design of our approach, the detectability of model-generated completions by a static analysis tool could be included as a factor for the selection step. To investigate this further, a practical defense should be developed first.

References

[1] Z Yang et al., "Natural Attack for Pre-trained Models of Code". ICSE 2022.
[2] T Liu et al., "Generating Private Synthetic Data with Genetic Algorithms". ICML 2023.
[3] M Z Nawaz et al., "Proof Searching in HOL4 with Genetic Algorithm". SAC 2020.
[4] S Polu et al., "Formal Mathematics Statement Curriculum Learning". ICLR 2023.
[5] Kevin Ward and Fabian Kammel, "Abusing VSCode: From Malicious Extensions to Stolen Credentials (Part 1)". https://control-plane.io/posts/abusing-vscode-from-malicious-extensions-to-stolen-credentials-part-1/
[6] Leonid Bezvershenko, Tweet: https://x.com/bzvr_/status/1859233190449213880
[7] Jingxuan He and Martin Vechev. “Large language models for code: Security hardening and adversarial testing”. CCS, 2023
[8] Ita Ryan, Utz Roedig, and Klaas-Jan Stol. 2023. "Unhelpful Assumptions in Software Security Research". CCS 2023
[9] Hammond Pearce et. al. “Asleep at the Keyboard? Assessing the Security of Github Copilot’s Code Contributions”. IEEE S&P, 2022

We would like to thank the reviewer for their insightful review and constructive comments. We address their comments, questions, and concerns below:

Q1: Can such an attack be executed in practice?

Yes, as detailed in the paper, we believe that INSEC is highly practical.

First, next to typo-squatting and offering a free completion service to users, practically any extension installed in VSCode may hijack other extensions due to the ability to execute arbitrary commands on the user machine [5]. This may be exploited to directly manipulate the installed, official, Copilot extension from a benign-looking extension offered by the adversary, such as a custom theme. Moreover, malicious Python packages, as recently discovered [6], could pose a similar attack vector.

Second, integration of the attack into generated code is trivial, since we always inject a comment line before the currently edited line by the user. We want to point out here again that we do not require any detection of a good insertion position, since the attack is designed to allow injection at any position.

Finally, users would not notice injections into intercepted prompts, since they obtain functionally correct code from SOTA fill-in-the-middle models, such as Copilot.

Q2: Would security analysis before deployment catch the introduced vulnerabilities?

Yes, but this is not realistic in a general case. First, we would like to point out that a significant part of our dataset is sourced from actual vulnerabilities found on GitHub, as such vulnerabilities are definitely being deployed into production settings. Previous work also reports similar findings [7]. Moreover, analyzer services for code security are difficult to set up, expensive to maintain, and developers are constrained by work environments among other obstacles to successfully deploy security analysis in production settings. Previous work has shown that assuming the presence of analysis services is generally unhelpful in determining real-world threats [8].

Q3: Is INSEC different from [1] and provides significant contributions beyond it even though both use variations of genetic algorithms for optimization?

Yes, we highlight two key contributions of INSEC over [1] below. Note that the use of genetic optimization does not limit novelty, because it is a general optimization technique applied across many domains leading to novel results (e.g., [2,3,4]). That being said, we thank the reviewer for pointing us to [1] and we have included a discussion of [1] in our revised paper.

• Classification vs. Generation: [1] focuses on classification tasks with fairly outdated BERT models, such as vulnerability detection and authorship attribution. In contrast, INSEC targets code completion with state-of-the-art LLMs, an actual practical and commercial use case with user bases in the millions. As such, our work raises timely practical concerns for critical masses of developers.
• Inference-time vs. Training-time Optimization: In [1], the attack optimization algorithm has to be performed for each input, which incurs significant overhead during inference time. On the contrary, for INSEC, the short attack string is derived at training time and is reused across all inputs at inference time. This approach is realistic and meets the real-time requirements of modern code completion.

Q4: Does the simplicity of the optimization procedure of INSEC limit its practical relevance as an attack on the security of code completion engines?

No. In fact, we believe it increases the severity, since it makes the attack more accessible and easier for real-world attackers to execute.

Q5: Is the evaluation of INSEC’s impact limited by using CodeQL to detect vulnerabilities?

No, since our method does not depend on CodeQL as a CWE detection method. It was chosen due to its customizability, and as it is adopted as standard in prior work [7,9]. Attackers may build their attacks using other static analysis tools, however, the tool and its capabilities are not relevant to the attack itself. In fact, we only utilize specific queries from the extensive repository of CodeQL. Attackers could similarly hand-craft a specialized tool to detect precisely the vulnerabilities that they want to inject, use it for training, and then manually assess the quality of the results. This would even increase the severity of the resulting attack, since the injected vulnerabilities would likely not be detected by potentially deployed, publicly available countermeasures.

We further manually verify the results of CodeQL, by investigating 50 generated completions of the test set from unattacked, Init-only attacked and optimized attacked models, across all CWEs and models. The precision of CodeQL among these samples was 98%. We added the details of our manual code review and a clarification to the paper in lines 316 and 317 and Appendix A.

We thank the reviewer for engaging with our response and would like to clarify the remaining concerns. We also thank Reviewer Xra8 for participating in the discussion.

ReQ1: Are injected attack comments visible to the user?

No. As also pointed out by Reviewer Xra8, the malicious comment is inserted into the prompt sent to the server, which always happens at the backend and is hidden from the user interface in the frontend. Therefore, the attack comment is completely invisible to the user at all interaction steps. Moreover, the insertion of the fixed attack comment is straightforward, further increasing the feasibility of the attack.

In more detail, the workflow of an interactive code completion step is as follows:

1. In the frontend, the user edits file buffer F.
2. F is sent to a black-box LLM (e.g., Copilot), which returns a completion C.
3. The completion C is redirected into the IDE and displayed to the user by overlaying it onto F.

With INSEC applied, the workflow would look like:

1. In the frontend, the user edits file buffer F.
2. In the backend, F is copied by INSEC into a separate buffer F’.
3. INSEC injects its attack string into F’.
4. F’ is sent to a black-box LLM (e.g., Copilot), which returns a completion C’.
5. The completion C’ is redirected into the IDE and displayed to the user by overlaying it onto F.

Note that the completions C and C’ only contain the output, e.g., the yellow part in Figure 1. They are disjoint from F and F’ respectively and do not contain the attack comment. Therefore, the attack comment is also not revealed to the user through the completion. Regardless of whether INSEC is applied or not, the user sees the same elements: the file F and the output completion. The only difference lies between C and C’. In most cases, C and C’ have the same functionality, as demonstrated in our HumanEval experiments. In security-sensitive scenarios, C’ is more likely to be insecure than C. This is the basis for our claim to INSEC’s stealthiness.

ReQ2: Can static analysis easily detect the generated vulnerable code in the wild?

No. Static analysis is often inadequate in detecting vulnerabilities in the wild [2,4,5,7]. One reason is that it is usually unknown if and which vulnerabilities a piece of code contains, preventing the choice and application of appropriate queries as done in our eval [4,5]. Meanwhile, attempting to detect all potential vulnerabilities is limited by the CWE coverage of static analysis tools, as was already pointed out by the reviewer. Moreover, static analysis tools are found to fail at effectively explaining discovered vulnerabilities to developers [1] and lack actionable recommendations for mitigation [2]. This results in static analysis being less prevalent in practical settings than one might expect [3], with vulnerabilities generated by Copilot having already found their way into public code repositories [6]. We demonstrate that INSEC is effective on various known CWEs, which, combined with the aforementioned reasons, poses a major threat to future code security. It is also well conceivable for an attacker to adapt INSEC to target on vulnerabilities that are even more difficult to detect, novel or otherwise not covered by deployed static analyzers.

It should be noted that the above argument does not contradict our use of CodeQL for security evaluation. This is because our evaluation setting is more controlled than in-the-wild vulnerability detection, in that we know how vulnerabilities can manifest in the generated code for all test cases. This allows us to carefully select a specialized CodeQL query that works effectively for each test case, as described in Section 5.1.

References
[1] Nachtigall et. al. "Explaining static analysis-a perspective." IEEE ASEW, 2019
[2] Nachtigall et. al. "Evaluation of Usability Criteria Addressed by Static Analysis Tools on a Large Scale." 2023
[3] Ryan et. al. "Unhelpful Assumptions in Software Security Research." CCS 2023
[4] Charoenwet et. al. “An Empirical Study of Static Analysis Tools for Secure Code Review.” arXiv preprint
[5] Mendonça et. al. "An empirical investigation on the challenges of creating custom static analysis rules for defect localization." Software Quality Journal 2022
[6] Nguyen et. al. “Security Weaknesses of Copilot Generated Code in GitHub.” arXiv preprint
[7] Vasallo et. al. “Context is king: The developer perspective on the usage of static analysis tools.” IEEE SANER 2018

We would like to thank the reviewer for their insightful review and constructive comments. We address their comments, questions, and concerns below:

Q1: Can you evaluate the functional correctness on further coding benchmarks?

Yes, we adapt the repository-level code completion benchmark RepoBench [1] to our setting and report the achieved score below. RepoBench is based on recently created GitHub repositories to avoid contamination, contains complete files and reports Exact Match (EM), Edit Similarity (ES), and CodeBLEU (CB) scores based on a golden next line to be completed. Overall we observe only a slight performance decrease due to the attack, matching our observations from HumanEval.

Concretely, we choose the first 333 samples from the Python Cross-File-First, Cross-File-Random, and In-File settings and sample 40 times completions from Starcoder 3b, the StarCoder2 family, CodeLlama 7b and GPT-3.5-Turbo-Instruct. We sample completions based on the prefix once without an attack string for the baseline setting (“Unattacked”), once with the attack string for each Python CWE, and report the average of the results in the “Attacked” setting. We will incorporate a complete investigation in the next revision of the paper.

References
[1] Liu, Tianyang, Canwen Xu, and Julian McAuley. "Repobench: Benchmarking repository-level code auto-completion systems." ICLR 2024

We thank the reviewer once again for their overall positive assessment of our paper and for their engagement in the discussion with another reviewer. In case the reviewer has any remaining questions or concerns about our rebuttal or the paper, we are eager to engage in a discussion until the end of the discussion period on Tuesday this week.

We would like to thank the reviewer for their insightful review and constructive comments. We address their comments, questions, and concerns below:

Q1: Are the functional correctness (FC) results presented in the paper representative of results on other FC datasets, such as HumanEval-X?

Yes. We thank the reviewer for the pointer to HumanEval-X. It is important to note that HumanEval-X does not contain a translation for Ruby. We preferred to treat the different languages equally and thus translated all languages using GPT-4. To address the reviewers' concerns, we evaluate the functional correctness rate of INSEC on the C++ and JavaScript versions of HumanEval-X, since the Python version is the same as the one used by us. We compared these results with those in our original submission, i.e., obtained using GPT-4-generated reference solutions. The comparison of the obtained fun_rates is displayed in the table below, showing that the numbers are similar between manually translated and GPT4-generated reference solutions. After discussing the results on HumanEval-X with the reviewer and they still consider it relevant, we will be ready to report them in place of our translations in the next revision of the paper.

Q2: Is CodeQL a suitable tool for the security evaluation presented in this paper? Can you confirm this by manual code review?

Yes. Please be aware that the traditional vulnerability detection setting differs from our evaluation setting, so the conclusion drawn from the former does not necessarily apply to the latter. In traditional vulnerability detection, it is usually unknown if a piece of code contains vulnerabilities and we agree with the reviewer that static analyzers can be inaccurate in this case. However, our evaluation setting is more controlled, in that we know how vulnerabilities can manifest in the generated code for all test cases. This allows us to carefully select a CodeQL query that works effectively for each test case as described in Section 5.1, thus achieving high accuracy for vulnerability judgment. Note that vulnerability judgment based on CodeQL is standard in the field [1,2].

We also adopt the reviewer’s suggestion to manually verify the results of CodeQL, by investigating 50 generated completions of the test set from unattacked, Init-only attacked, and optimized attacked models, across all CWEs and models. The precision of CodeQL among these samples was 98%. We added the details of our manual code review and a clarification to the paper in lines 316 and 317 and Appendix A.

Q3: Would additional processing of the code, such as adding comments to maintain security, invalidate the attack?

No, we investigate this by an experiment on GPT-3.5. We evaluate all CWEs, and insert the comment This code should be secure at the line above the INSEC attack string. We use the attack string used for the main experiments, i.e., not adaptively optimized for the presence of such a comment. While we observe that this slightly decreases the vulnerability ratio from 76% to 62%, the final ratio still largely exceeds the baseline vulnerability ratio of 22%. We added these results to the revised paper in Appendix E.

Our results are not surprising, as previous work [3,4] has already shown that comments are usually not sufficient to reliably steer models towards secure generation. Further, we would like to point out that Copilot, a state-of-the-art fill-in-the-middle completion engine, claims that it applies additional sophisticated processing to prevent security vulnerabilities [5], but is successfully attacked by INSEC. We conclude that even advanced approaches are not effective in mitigating our attack.

Q4: How would the attack behave on code snippets different from the training and validation snippets?

The CWEs that we consider all share a similar structure in the imminent source of the bug, such as a call to a sanitizer or off-by-one memory allocation. However, the surrounding context of these locations varies strongly between various realistic settings within training, validation, and test samples. As we aim to attack only those situations, and since the attack appears to have little effect on non-critical code as detected by func_rate, we believe our attack to work in new contexts as well.

Overall, it is hard to determine what would count as out-of-distribution here, since we attack specific CWEs and specific coding situations. If the reviewer has concrete suggestions of out-of-distribution data points for evaluation we would be glad to run our attack against them.