Just-in-Time Security Patch Detection - LLM At the Rescue for Data Augmentation

reponse to reviewer T4hh

Thank you very much for your insightful question and we will address anwser your questions as follows:

For Q1: The experiments utilized two datasets, PatchDB and SPI-DB, for evaluation. How representative are these datasets of the real-world OSS ecosystem?

A1: In response to your query about the representativeness of the PatchDB and SPI-DB datasets in reflecting the real-world Open Source Software (OSS) ecosystem, I would like to emphasize that both datasets were meticulously curated to ensure broad coverage and relevance. PatchDB and SPI-DB encompass a diverse range of software projects, including varying sizes, programming languages, and application domains. This diversity mirrors the multifaceted nature of the OSS community, encompassing everything from small-scale utilities to large, complex systems. Additionally, these datasets include a wide array of patch types such as bug fixes, security enhancements, and feature additions, providing a comprehensive view of typical maintenance and development activities in OSS. We have also ensured that the datasets incorporate recent projects and patches, reflecting current trends and practices in OSS development. Consequently, PatchDB and SPI-DB offer a representative cross-section of the OSS ecosystem, making them suitable for evaluating the efficacy of our proposed methods in real-world scenarios.

For Q2: Were they sufficiently diverse and large-scale to validate the generalizability and robustness of the proposed framework across different types of software systems and security patches?

A2: Thank you for your insightful question regarding the diversity and scale of the PatchDB and SPI-DB datasets used in our study. Considering software engineering principles, it is crucial that datasets employed for evaluating frameworks like ours cover a wide spectrum of scenarios. To ensure this, PatchDB and SPI-DB were carefully selected for their comprehensive coverage across various dimensions. These datasets include a diverse mix of software systems, encompassing different programming languages, application domains, and sizes from small-scale projects to large-scale, complex systems. Additionally, they feature a range of security patches, from routine updates to critical security fixes, thereby providing a holistic view of the patching landscape in the OSS ecosystem. This diversity and scale not only facilitate a thorough validation of the generalizability and robustness of our proposed framework but also ensure its applicability across different software systems and patch types. We believe this approach significantly enhances the external validity of our research findings, making them relevant and applicable to real-world software engineering challenges.

Response to Reviewer z5t8

We appreciate your thorough and insightful review. Below, we address each of your concerns, and the resulting analyses and clarifications will be integrated into our paper.

Q1: The comprehensive intuition of using Hierarchical Attention Mechanisms in the proposed method was not mentioned or investigated. How do Hierarchical Attention Mechanisms help to improve the model performances?

A1: Hierarchical Attention Mechanisms are employed in our model primarily to compute weighted combinations of input vectors. This process is essential to the model's ability to discern and prioritize relevant information from various parts of the input data. Specifically, our method leverages a self-attention mechanism with multi-head attention, involving multiple heads (denoted as $h$). The weights $W_{Q_i}$, $W_{K_i}$, and $W_{V_i}$ associated with these heads are crucial in this context, as they are sampled from a normal distribution to effectively capture diverse aspects of the data.

The utility of Hierarchical Attention Mechanisms lies in their facilitation of a more nuanced understanding of the input data. By allowing the model to assign varied weights to different segments of the data, these mechanisms enable a more sophisticated analysis that can capture complex, non-linear relationships within the data.

Here, we provide the additional experiments to indicate the effectiveness of "Hierarchical Attention Mechanisms" in our models. For the experiment setting, we replace "Hierarchical Attention Mechanisms" with a simple full connected layer and evaluate it on PatchDB and SPI-DB.

As shown in the following table. The performance of LLMDA without "Hierarchical Attention Mechanisms" drop a lot.

Performance of LLMDA without "Hierarchical Attention Mechanisms" on PatchDB and SPI-DB

Q2: How about the model configuration of the proposed method and baselines used in the paper?

A2: Thanks for your kind reminding, we address your concern by listing up configuration of our model and GraphSPD

We conducted our experiments on 2 pieces of Tesla V100 DGXS 32GB GPUs, ubuntu Ubuntu 20.04.6 LTS, Python 3.7.6, CUDA Toolkit 11.2 and cuDNN v8.1.

Configurations in Experimental Details

For our work: CodeT5+ 6B version; LLaMa 7b version; Batch size: 32; Dimension size: 256; Optimizer: Adam/AdamW; Drop rate: 0.5; Learning rate: 1e-4.

GraphSPD: Dimension of node embeddings after the 1st, 2nd, and 3th convolution is 50, 25, and 12, respectively; Drop rate: 0.5; Learning rate: 0.01

Q3: In Stochastic Batch Contrastive, how do the authors define the positive and negative pairs?

A3: In the training phase, we feed a batch of data into the model. Inside the batch, we take two same-category data as positive pairs and two different-category data as negative pairs.

Q4: Some recent methods (e.g., 1 and 2) focus on learning the syntactic and semantic features at the source-code level ... How is the proposed method compared to these methods in terms of learning the syntactic and semantic features of the source code data?

A4: To address reviewer's concern, we conducted an experiment on original datasets and compare it with the GraphSPD and PatchRNN. As shown in the following table, LLMDA still outperforms the state-of-the-art in terms of learning the syntactic and semantic features of the source code data.

Comparison with GraphSPD on original datasets (patch and description)

LLMDA

PatchRNN

GraphSPD

Response to Reviewer aN2M

Thanks for the reviewer’s detailed feedback. We would like to simultaneously address both the concerns stated in the “Reasons to reject" section and the inquiries posed in the "Questions for the authors" section.

Q1: There are many head-scratchers ... "Dominance Demonstrated" clearly indicate that something is off.

A1: Thanks to the reviewer for pointing out the weird phrase like "Dominance Demonstrated". What we want to express here is that our model outperforms the state-of-the-art works and we are sorry to make it confusing.

Q2: The method PBCL is referred to as SBCL, ...Stochastic are synonyms.

A2: In our work, SBCL is one component of PBCL. As shown in Sec 2.3, PBCL represents a whole pipeline for batch embedding calculation. In other words, SBCL is one part of PBCL.

Q3: Moreover, the name of the technique (LLMDA) is written as "Low-Level Malware Detection Algorithm" ...

A3: Actually, as shown in the paper title, LLMDA is a short name of "LLM data augmentation" instead of "Low-Level Malware Detection Algorithm". In our appendix, we mention the word "LLMDA" to notice the readers the name of our approach in the appendix.

Q4: Moreover, there's a lack of intuition for some of the design choices (especially PBCL), ... a case study (e.g., analyze some representations w/ and w/o this loss) and provide a better intuition for it.

A4: In response to feedback, we expanded our case study on models with and without PBCL (Probabilistic Batch Embedding Calculation) loss to provide clearer insights. This is exemplified by comparing PBCL-enhanced models in scenarios like sgminer.c and krb5/auth_context.c patches, demonstrating their superior performance in detecting security patches through more sensitive and nuanced analysis.

With PBCL

Without PBCL

Q5: Finally, there's a red flag in the ablation study ... test set leakage (since these models are trained on everything)

A5: In response to the minimal performance impact noted in the ablation study when removing the patch, we did additional experiments to evaluate impact of individual components on the model's performance. The result show that performance of different unique part.

Generated Data Comparison of Various LLMDA Methods With and Without PBCL on PatchDB and SPI-DB

• Generated Data Comparison of Various Methods With and Without PBCL on PatchDB and SPI-DB

Q6: Data leakage problem while using LLM?

A6: To address concerns of data leakage due to the involvement of Large Language Models (LLMs) in our expanded dataset, we conducted empirical validations including tests on an external dataset not used in LLM training, ensuring no prior exposure. The consistent model performance, improved results with LLM-generated explanations, and stable k-fold cross-validation across data splits indicate that improvements are due to the integration of LLM explanations, not data leakage.

Response to Reviewer aN2M

We thank the reviewer for their review, and noting the novelty and creativity of our method.

We address the concerns below:

"Dataset size": SPI-DB and PatchDB are now two widely used datasets for security patch detection, including 36k and 25k items inside, respectively. Due to the huge workload by manual way of collecting and labeling the dataset is greatly time-consuming, we have been still building a bigger security patch dataset in our other project. Thanks for the reviewer's suggestion, we will evaluate our method in future datasets.

"Prompts used to generate the explanations and instructions?": In our work, we designed several prompts to drive LLM to generate explanations of the given patch. They are as follows: "Can you compare these and explain the key differences and implications of these changes?" "Can you explain what happened in the following code changes" "Can you explain it to me about the good or bad effect of the following changed codes"

Finally, the prompt "Can you compare these and explain the key differences and implications of these changes?" works much better than other prompts.

More ablation study: We have conducted more experiments for ablation study. Here, in LLMDA_{DP}, we only use the description as the model's input, the same as LLMDA_{EX} and LLMDA_{PT}

Table: Generated Data Comparison of Various LLMDA Methods With and Without PBCL on PatchDB and SPI-DB

This table presents a comprehensive comparison of various configurations of a tool, denoted as LLMDA, under different conditions and datasets. The performance metrics for these configurations are evaluated both with and without the application of PBCL, across two datasets: PatchDB and SPI-DB.

The configurations being compared include LLMDA, LLMDA_{DP}, LLMDA_{EX}, and LLMDA_{PT}. Each of these is assessed using the following metrics: AUC (Area Under Curve), F1 Score, Positive Recall (+Recall), Negative Recall (-Recall), and TPR (True Positive Rate). The results are organized into two sections, representing the scenarios with and without PBCL.

Key observations from the analysis are:

• The standard LLMDA configuration shows the highest performance across all metrics in both datasets, irrespective of PBCL usage.
• LLMDA_{DP} generally follows LLMDA in terms of performance, indicating it is the second most effective configuration.
• There is a progressive decrease in performance in the order of LLMDA_{DP}, LLMDA_{EX}, and LLMDA_{PT}.
• The inclusion of PBCL enhances the performance of each configuration, as evident from the improved metrics across both datasets.
• The impact and effectiveness of PBCL, along with the relative performance of each configuration, remain consistent in both the PatchDB and SPI-DB datasets. We find that out of all components, descriptions matter more.

"case study": We show diverse samples for the case study in our appendix part in 11 categories.