IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

Q1: “Can any new package or Java module not processed by the LLMs cause detection or specification generation issues?”

We acknowledge that LLMs may not have prior exposure to external APIs from private projects. However, they can still provide best-effort guesses based on the method information available, which may include JavaDoc documentation. This is a significant improvement over traditional methods, which rely solely on human labels and would be completely ineffective in such scenarios. That said, we acknowledge that there may be better methodologies for generalization to unseen cases, and we aim to explore these in future work.

Q2: “what are all the possible values for the type of node to match in G and what is the upper bound for N from the N-tuple of F”

The type of nodes could be

• argument[i], denoting the i-th argument of an external function call
• argument[this], denoting the “this” argument in an external function call (e.g. parser in the call parser.parse(arg1, arg2))
• parameter[i], denoting the i-th parameter of an internal function definition
• parameter[this], denoting the “this” argument of an internal function definition
• return_value, denoting the return value of an external function call

The upper bound for N corresponds to the maximum number of arguments of a given API, which, in our CWE-Bench-Java, is 32.

Q3: “in the event that CodeQL does not detect a vulnerability even with correctly labeled specifications, will it be marked as negative? Can you give me any statistics on how many such instances there are? If I understand correctly, CodeQL is still a critical part of the vulnerability detection pipeline. LLM just ensures that the data fed into the static analyzer is of quality, and once detection is made, LLM helps weed out false positives?”

If taint specifications are all correct and that the data-flow is capable of modeling the vulnerability, then CodeQL will be able to detect a vulnerability. In case that the data-flow cannot model the vulnerability, CodeQL will not be able to detect the vulnerability, and the result will be marked as negative. Therefore, CodeQL is still a critical part of the vulnerability detection pipeline.

In our dataset, we observe at least 12 such cases where normal taint data-flow cannot model the vulnerability. However, the general statistics relevant to our evaluation is very hard to retrieve. This is due to the missing ground-truth source and sink labels as well as the sheer size of our projects.

Q4: “How did they come up with the "±5" value? Can it be arbitrarily large, bounded by the LLM's token limitation?”

In our experiments, we chose ±5 as a balanced approach to provide sufficient context while maintaining performance and cost. While it is technically possible to use a larger window that encompasses the entire function or even the class definition, we found that too much context can overwhelm the LLM, leading to reduced accuracy. Furthermore, increasing the context size substantially raises computational costs—especially given the large number of candidate APIs and paths that must be queried.

Looking ahead, we plan to explore more targeted methods for context selection. Instead of using a fixed number of surrounding lines, we could incorporate only the most relevant variable definitions, class and function signatures, and other critical elements. However, achieving this balance between including relevant information and maintaining manageable context sizes will require further investigation, which we leave for future work.

Q5: “Has it ever happened that LLM made a mistake and marked a positive case as negative even after the positive detection of a vulnerable path by CodeQL?”

Yes, this occurs occasionally. For example, there was a path traversal vulnerability (CWE-22) caused by an insufficient Regex pattern used to sanitize user input. While CodeQL successfully reported this path, the LLM failed to recognize the vulnerability during contextual analysis. This was because the Regex pattern was defined as a static global variable outside the provided context, making it inaccessible to the LLM. As a result, the LLM incorrectly assumed the input was properly sanitized and flagged the path as non-vulnerable.

As mentioned in our response to Q4, we plan to explore a more targeted method for context retrieval, to improve the precision of contextual analysis.

Q1: “What are the causes of the undetected vulnerabilities (among the 120)?”

We thank the reviewer for the question, we will include the following error analysis for false negatives into our paper.

• Vulnerability cannot be modeled by simple taint dataflows: for instance, the vulnerability CVE-2020-11977 (CWE-94) is manifested by an unexpected exit(1) call, with no direct taint dataflow going into it. In fact, the taint dataflow goes into the condition of an “if” statement surrounding the exit. In this case, the model of sink API cannot capture the vulnerability.

• Missing data-flow edge due to side-effects: for instance a taint information is written to a temporary file, and is later read from the same file, subsequently causing a vulnerability. However, the dataflow edge is carried through side-effects, which is not captured by CodeQL.

• Missing data-flow edge due to unspecified library usage: The vulnerability can only be manifested through a concrete usage of the library; but within the library itself there is no possible dataflow to connect the source and the sink.

In general, we view the above as general limitations for the static analysis. We can hopefully resolve them with query synthesis, and we leave these for future work. In terms of LLM induced false negatives, here are the two main failure modes:

• Missing taint-propagator labels from LLM: this would cause missing data-flow edges stopping source to flow to sink.

• Missing source/sink specifications from LLM: if there is no relevant source/sink specification then the static analysis tool would not have the anchor for analysis.

Q2: Discussion on “vulnerability is considered detected when the vulnerable path passes through some crucial program points”

According to [1], the function-level metric is not enough, and the calling context surrounding the vulnerability matters. We adopt the strategy based on human understanding of vulnerability, where we look at the path between the places where taint is initialized and manifested. This path-based structural coverage criteria is widely adopted in software testing [2]. In the empirical study [3], the authors look at the evaluation metric which uses the fix of the vulnerability as the crucial program point.

• [1]: Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection, Risse et. al., 2024
• [2]: Introduction to software testing, P Ammann, 2008
• [3]: How Many of All Bugs Do We Find? A Study of Static Bug Detectors, Habib et. al., ASE 2018

Q3: “A more rigorous discussion or evaluation should be conducted beyond random sampling of 50 alarms to argue that the actual false discovery rate should be much lower (line 397-399).”

We thank the reviewer for raising this important concern. Our manual analysis primarily focuses on evaluating the potential attack surface and the manifestation of vulnerabilities, which aligns with the key factors used to determine severity according to the Common Vulnerability Scoring System (CVSS) [4]. For example, in our analysis, we label an alarm as a vulnerability if it presents a "local" attack vector, even if the resulting CVSS score may be relatively low.

We agree that a more thorough, CVSS-based evaluation of the alarms reported by IRIS would provide additional rigor. However, it is outside the scope of our current study. We acknowledge this as a valuable direction for future work. In our revised version, we will elaborate on the detailed criteria for our manual analysis.

• [4]: Vulnerability Metrics, National Vulnerability Database, https://nvd.nist.gov/vuln-metrics/cvss

Q1: “potential data leakage and memorization by LLMs, since they are asked to infer sources/taints purely based on method name and signature instead of implementation”

While we acknowledge that our method relies on the LLM’s existing knowledge, we argue that the risk of data leakage is minimal. The labels for source/sink APIs and their specific types are typically not publicly available on the internet. Moreover, the paths summarized by CodeQL are even less familiar to LLMs, making it highly unlikely that these were seen during training. This approach thus represents a genuine challenge for the LLM’s capability in knowledge transfer and logical reasoning.

We would also like to clarify that while LLMs may not have prior exposure to external APIs from private projects, they can still provide best-effort guesses based on the information available. This is a significant improvement over traditional methods, which rely solely on human labels and would be completely ineffective in such scenarios. That said, we acknowledge that there may be better methodologies for generalization to unseen cases, and we aim to explore these in future work.

Regarding the sole reliance on method names and signatures, we emphasize that this was a deliberate design decision made after carefully balancing accuracy and cost. While including full function implementations might improve accuracy, the vast number of candidates requiring labeling (Table 11) makes it impractical to query LLMs for specification inference with complete implementation details. Further, analyzing the entire function implementation is itself a challenging task for LLMs.

Q2: “The comparison with baseline CodeQL may not be fair enough as the LLM-based method will always have advantages due to more source/sinks. … LLM's advantage against heuristics like name and signature type matching.”

We thank the reviewer for raising this insightful question. While it is true that LLMs can propose more source/sink candidates, two critical factors must be considered: (1) the increased number of paths generated by CodeQL, which significantly impacts the cost of subsequent contextual analysis, and (2) the higher likelihood of false positives that may result.

To address this concern, we conducted a small-scale experiment on a relatively small project, zt-zip, where both CodeQL and IRIS+GPT4 successfully detected CVE-2018-1002201. Using the signature-based heuristics suggested by the reviewer, the pipeline generated 4,441 alarms (compared to 8 for CodeQL and 15 for IRIS+GPT-4), resulting in a False Discovery Rate (FDR) of 98.2%. In contrast, IRIS+GPT4 achieved an FDR of 63.8%, and CodeQL achieved an FDR of 60% on this project. These results demonstrate that the heuristics-based approach yields extremely low precision.

Given that IRIS+GPT-4 outperforms CodeQL on both the number of vulnerabilities detected and the FDR, we maintain that the comparison between IRIS+GPT-4 and CodeQL is fair and reflective of their respective strengths.

Q3: “I would like to know the actual cost of LLM call for IRIS on CWE-Bench-Java in total and on average”

We thank the reviewer for the question. On average, we observe 388.4 LLM calls per project, though this number varies significantly based on the project size. For example, Perwendel Spark, a project with 10K lines of code, required only 43 LLM calls to detect the CVE present within it.

For the final evaluation of IRIS on CWE-Bench-Java, across the 7 evaluated LLMs (including 2 added during the rebuttal) and the 120 projects in our dataset, we made approximately 320,000 LLM calls. While it is challenging to provide an exact cost breakdown, we hope this information offers valuable perspective on the scale and effort involved in our work.

Q1: Comparison of CWE-Bench-Java with existing datasets like PrimeVul and SVEN

In prior datasets such as PrimeVul and SVEN, the input consists of a single function. In contrast, our task involves analyzing an entire repository, which averages 300K lines of code per project. Identifying inter-procedural vulnerability paths within such extensive repositories is akin to finding a "needle in a haystack." Table 5 highlights other key differences between these datasets, underscoring the challenges addressed by CWE-Bench-Java.

From our experience, function-level vulnerability detection is insufficient for tackling real-world vulnerabilities, which typically span multiple functions rather than being isolated to a single one. This perspective is widely shared within the research community [1, 2]. As Reviewer 3b2y also noted, the ability to perform project-level vulnerability detection is a strength of our work.

• [1]: Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection, Risse et. al. 2024
• [2]: Data Quality for Software Vulnerability Datasets, Croft et. al. ICSE 2023

Q2: “False positive rates are not evaluated?”

We do estimate the False Discovery Rate (FDR), which is similar to the false positive rate that the reviewer mentioned but provides a more appropriate measure in our context (Section 3.6). Since we are performing whole-repository analysis, there is no need for an explicit mention of “negative examples” in our dataset—any non-vulnerable paths in the repository naturally serve as negative examples, far outnumbering the vulnerable ones.

Q3: “Should you also compare any AI based approaches? e.g., what about feeding the entire function into a prompt and see how LLMs perform?”

Our dataset expects whole-repository analysis, which is fundamentally different from function-level vulnerability detection, as seen in datasets like PrimeVul and SVEN (Table 5). For this reason, we do not use pure neural approaches as baselines, since incorporating the entire repository (which could have up to 7M lines of code as shown in Table 11) as context for a large language model is computationally intractable.

Q4: “How subset of paths are selected when paths are too long for the prompt?”

We set a hyperparameter $S$ to control the number of intermediate steps in the prompt.For paths with more than $S$ intermediate steps, we divide the path into $S$ equal segments and select one step from each. This selection prioritizes function calls, as they may indicate sanitizations. If no function call is present, we randomly pick one node in the segment. In our experiments, we observe that setting $S$ to 10 provides a good balance between the cost and accuracy so that the prompt contains enough context and would not be too long. We will add this description to the revised version of our paper.

Q5: “Typically in static analysis, we provide a list of taint APIs, it can be done precisely by the domain experts”

As discussed in lines 48-53, it is well-known that existing taint API lists are often insufficient for identifying new, real-world vulnerabilities. These labels are typically created retroactively–after a CVE is discovered, a human expert assigns the corresponding label. This approach introduces significant limitations, as tools relying solely on human-generated labels struggle to keep up with evolving vulnerabilities. Moreover, these labels tend to be rigid and require continuous maintenance to adapt to the dynamic nature of modern software development. In Table 8 in our paper, we present the number of unique APIs presented in our dataset. Manually labeling over thousands of libraries with APIs often appearing in different contexts is prohibitive [3].

• [3]: Scalable Taint Specification Inference with Big Code, Chibotaru et. al. PLDI 2019

Q6: “only 5 models are experimented”

Following the reviewer’s suggestion, we have extended the evaluation with two more popular open-source LLMs, Qwen-2.5-Coder-32B-Instruct and Gemma-2-27B, and we hereby show the high-level performance comparison in the same format as our Table 1. In general, we see that IRIS+LLMs still consistently outperform all traditional baselines. We will incorporate this in the revised version of our paper and we plan to explore more models in the future.

Table 1 Extended: