Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Framework

We sincerely thank you for the positive evaluation and recognition of our work. We are pleased that you appreciate our empirical foundation and insightful analysis with BFAD achieving significant improvements.

We would first like to address the specific questions you raised:

Q1: For real-world deployment, what setup do you recommend? Can smaller models run BFAD effectively in production? Will you release the code and dataset?

Due to resource limitations, we currently evaluate LLMs via API access. For enterprise deployment, security and data privacy are critical, so we recommend local deployment of models. Without fine-tuning, we suggest using models with at least 14B parameters to achieve competitive performance. Our core code has been included in the supplementary materials. All datasets used are publicly available, and we plan to release the full code and dataset upon paper acceptance.

Q2: Could a fine-tuned model outperform your prompt-based approach? Have you considered fine-tuning a model like CodeLLaMA or Qwen on your dataset?

As noted in swDn (Reviewer 2) Q4, our current work represents the first attempt to explore the use of LLMs in WebShell detection. We focused on demonstrating the promise of this approach via prompt-based methods. Given the encouraging results, we are confident that future work involving fine-tuning has the potential to achieve even better performance.

Given that LLMs have not yet been widely applied to WebShell detection, we summarize current limitations and outline potential future research directions. Our goal is to help advance this emerging field and inspire more comprehensive exploration. You can also access this part in https://docs.google.com/document/d/1pMab2TGHICOvDzc2LdDox9GP2smx-YAaBM6xdlz2FHM/edit?usp=sharing

Q3: How would your framework handle novel WebShells that don’t use any of the “critical” functions?

Our framework can serve as a first-stage filter for completely novel attack vectors and can be improved by:

1. Leveraging reasoning models: For ambiguous or uncertain samples (e.g., using entropy as a threshold), we plan to employ current LLMs with better reasoning capabilities to emulate human analysts in interpreting unfamiliar behaviors.
2. Dynamic expansion via agent workflows: We envision integrating rulebase growth into an automated agent framework, allowing the system to detect and adapt to newly emerging patterns without manual intervention.
3. Incorporating dynamic analysis: Recent research has shown that runtime function call traces collected through sandbox execution can reveal encrypted or hidden function behaviors. We believe such dynamic data can significantly improve our framework’s ability to detect stealthy WebShells and enhance the effectiveness of our Critical Function Filter in more challenging cases. We are actively collaborating with industry partners to explore this direction.

At present, we have not been able to further validate these extensions. As noted in Reviewer 1h5H (Reviewer 1) Q2 and swDn (Reviewer 2) Q6, nearly all academic work on WebShell detection relies on open-source datasets, due to privacy constraints and limited access to real-world data.

That said, considering the promising results LLMs have already achieved on open-source data, we remain optimistic that advances in LLM capabilities will continue to drive progress in this direction.

We sincerely thank you for the constructive feedback. Below, we address each of the concerns you raised.

Q1: How were non-critical segments in Algorithm 1 selected?

We have clarified this in revised Section 2.2.

The motivation here is: using only critical code segments may mislead the model (e.g., benign functions flagged as malicious). To address this, we found a simple but effective strategy: we reattach some non-critical lines after extracting key regions (see Tables 5–6, Critical vs. Hybrid). Implementation-wise, we map extracted lines to the original code, then add top-down, non-overlapping, non-critical lines within context limits.

Q2: Why not ablate ICL example count or components like WBFP/CACE?

1. ICL Examples:

• We follow prior practice in LLM code analysis, where ICL examples are limited (often to 1) [1].
• WebShell tends to be long, and more ICL examples reduce space for input code, potentially degrading performance. An additional experiment confirms this: one extra ICL example decreased F1 by 4.5% on GPT-4 and 8.2% on Qwen-2.5-3B (to be added in Section 4.3).
• Research has shown that more examples can hurt performance due to overfitting or context saturation [2,3]. Using one example is a safe and commonly accepted choice in practice.

2. WBFP and CACE Ablations: We have already provided comprehensive ablations:

• Table 4: No WBFP/CACE
• Tables 5–6: CACE only
• Tables 7–8: CACE + WBFP

We treat CACE as the foundation for directing the model's attention to security-critical code, while WBFP builds upon CACE by selecting ICL examples with similar behaviors, enabling the LLM to better interpret the extracted code. This design follows ICL best practices that require format alignment between input and demonstrations. Since WBFP selects examples based on behavioral similarity, the test inputs should also highlight key behavioral segments—thus the use of CACE.

Q3: Why does 0.5B outperform 1.5B in Table 4?

Both benefit from BFAD, but the improvement differs. We believe this stems from ICL sensitivity and have added it to Sec 4.1: smaller LLMs (e.g., 0.5B) heavily rely on example labels, often copying them without real reasoning. Since WBFP provides accurately matched demos, 0.5B benefits more. In contrast, 1.5B models begin reasoning more independently using pretraining knowledge, which can sometimes reduce alignment with the ICL example’s label, hurting performance if reasoning is imperfect. Larger models (e.g., 14B) can better balance source code understanding with ICL use.

This aligns with prior findings: [4] shows that smaller models are more label-biased, while [5] observes that LLMs rely more on pretraining than on examples. Therefore, the 0.5B model’s stronger performance doesn’t imply higher intelligence—it simply reflects that WBFP’s selected examples are well-matched to the test cases.

Q4: Why not include a fine-tuned LLM baseline?

Our baselines already include code-specialized LLMs (e.g., Qwen-2.5-Coder-3B/14B), with vanilla Qwen-14B outperforming larger models like LLaMA 3.1 70B and GPT-4 (Table 4). We did not fine-tune on WebShell data, as our current goal is to explore whether off-the-shelf LLMs with domain knowledge can beat traditional models without retraining. This aligns with practical interest in avoiding costly retraining. Our results confirm this potential, filling a gap in current practice and offering a foundation for future fine-tuned models.

Q5: Hybrid strategy vs. CACE only?

For smaller models, Hybrid offers minor F1 gains over CACE. Its value lies in balancing precision and recall for flexible deployment, which we’ll clarify in the revision. However, for larger models (Table 6), Hybrid improves both metrics, making it a preferable choice in practical settings where large LLMs are feasible.

Q6: Zero-day or inference-only attacks

As discussed in our response to Reviewer 1h5H Q2, unlike other code analysis tasks (e.g., vulnerability detection), WebShell detection research is significantly limited by privacy and data availability. To our knowledge, nearly all existing work uses publicly available open-sourced WebShell datasets, making it difficult to simulate diverse or novel attack types.

That said, we appreciate the valuable suggestion and have added two future directions in the revised paper:

1. Zero-day simulation: Use LLMs to generate class-specific WebShell variants for benchmarking.
2. Fast + slow thinking: In real-world deployments facing unseen threats, our framework can serve as a fast filter (using a pre-trained LLM with domain knowledge), followed by deeper reasoning via a second-stage model (e.g., DeepSeek-R1). Though not evaluated here due to data limits, this is a promising path.

[1] https://doi.org/10.1016/j.jss.2024.112031

[2] https://arxiv.org/pdf/2501.04070

[3] https://arxiv.org/pdf/2310.00385

[4] https://arxiv.org/pdf/2305.19148

[5] https://arxiv.org/pdf/2502.13738

Dear Reviewer,

In response to your first question, we've conducted further research and small-scale experiments to filter out extreme confounding samples from our dataset. After testing the LLM's performance on these samples, we found that Vanilla GPT-4 only detected 7 out of 20 samples (accuracy below 50%). Even the trained Graph model was only able to accurately identify 12 samples. After applying our framework, we were able to detect 14 samples correctly, though 6 remained undetected due to limitations in code comprehension.

Among the correctly detected cases, we observed that the LLMs’ reasoning capabilities played a crucial role. Even when explicit malicious function calls were not directly identified, the models were often able to infer the actual behavior by analyzing obfuscated fragments and reasoning about the underlying functionality.

We’ve also made our dataset publicly available, and you can access it via the following anonymous link: https://anonymous.4open.science/r/LLM-for-Webshell-Detection-DB1E

We appreciate your support and look forward to hearing from you.

Thanks! We are glad you recognize BFAD as a useful approach that addresses an important but underexplored cybersecurity challenge with consistent improvement over state of the art. Below, we address each of the concerns you raised.

Main Concern: Lacks an In-depth Analysis

Our Response: Based on your feedback, we will add a dedicated "Analysis" section (Section 4.4) containing concrete examples and a thorough breakdown of misclassification patterns. We have conducted an in-depth analysis of LLM behaviors at different scales. Details are provided here: https://docs.google.com/document/d/1pMab2TGHICOvDzc2LdDox9GP2smx-YAaBM6xdlz2FHM/edit?usp=sharing

Key observations:

• Smaller LLMs tend to rely on surface-level pattern matching. For example, Qwen-0.5B flags any use of base64_decode as suspicious without contextual understanding. Qwen-1.5B improves slightly by noting the use of Sodium, yet still defaults to a malicious label.
• Larger LLMs (e.g., LLaMA 3.1 70B) demonstrate better contextual reasoning. They examine function usage, detect the absence of concrete malicious behavior (e.g., shell commands), and even link code to safe, open-source libraries.

We hypothesize this difference stems from the number of layers and attention heads, as surface patterns and deeper semantics may be captured by different heads in different layers attending to different features.

This discrepancy leads to a trade-off: smaller models suffer from lower precision (frequent misclassification of benign code due to shallow heuristics), while larger models show lower recall (missing obfuscated threats due to their broader, sometimes overly lenient reasoning). Our proposed BFAD framework mitigates these issues by guiding smaller models to better understand function-level behaviors and helping larger models focus on critical functions. As a result, it significantly improves detection performance across scales.

Q1: Why Not Use RAG Instead of ICL + WBFP?

The key challenges in this domain are:

• Accurately identifying behavioral signals in complex long code;
• Understanding the malicious intent behind usage patterns.

To address these, we use:

• CACE for isolating critical functional segments, and
• WBFP-boosted ICL to improve understanding via retrieved, task-specific examples.

Why WBFP-enhanced ICL over RAG: While RAG excels at retrieving general knowledge, WebShell detection requires understanding behavioral patterns rather than function definitions. We found that most LLMs already possess sufficient understanding of individual functions without RAG. Therefore, we propose WBFP to provide a task-aware retrieval mechanism that help LLMs better understand how these critical functions are used in other samples.

We think the WBFP's scoring and weighting strategies will also inspire future RAG systems that go beyond textual similarity toward behavioral relevance.

Q2: Robustness to Advanced Obfuscation

Our dataset follows established and widely accepted WebShell collection protocols, primarily sourcing from GitHub's open-sourced datasets. As a result, it offers limited exposure to novel obfuscated code, which is often proprietary and not publicly available.

We acknowledge the current limitations and plan to enhance it via dynamic analysis (e.g., runtime function tracing) to better capture obfuscated behaviors, as discussed in the Novel WebShell Family dataset [1].

Q3: Computational Overhead and Real-Time Applicability

WBFP Efficiency: The WBFP is executed offline. On an RTX 3090 GPU, throughput reaches ~100 samples per minute. Using multiple threads can achieve an even faster speed.

LLM Inference: LLMs offer the advantage of avoiding retraining while still achieving competitive performance via domain knowledge prompts.

• GPT-4: ~1–3s/sample
• Small LLMs (0.5B–3B): ~0.5-1s/sample

In practice, this is faster than our local graph-based detectors. While cloud APIs and local inference differ in setup, the results suggest BFAD’s feasibility for near-real-time use, especially given the increasing prevalence of enterprise-deployed LLMs.

Q4: Generalizability to Other Languages

Our work focuses on PHP WebShells, as PHP still dominates server-side web development, powering 74.2% of websites globally, with just 4.5% for JS [2]. Addressing this threat remains highly relevant.

BFAD’s “filter-then-detect” design is language-agnostic. Its core logic applies broadly; We have demonstrated its efficacy in PHP and are currently building a JavaScript version to expand the Critical Function Filter. This extension will be released as part of our future work post-acceptance.

[1] Zhao Y, Lv S, Long W, et al. Malicious webshell family dataset for webshell multi-classification research. Visual Informatics, 2024, 8(1): 47–55.

[2] W3Techs. Most popular server-side programming languages. https://w3techs.com/ (Accessed May 27, 2025)