Generate-then-Test: Automated Test Case Generation for WebAssembly Using Large Language Models

[Q2] How does the WasmTest compare with directly generating tests, existing testing methods, and other binary-lifting tools for Wasm like wasm2c?

We are grateful to the reviewer for highlighting gaps in our literature review and suggesting additional works for consideration. Based on this feedback, we have updated the related works section to include the recommended methodologies, noting that most operate under fundamentally different settings compared to our approach. Below, we address the specific concerns and provide detailed comparisons:

1. LLM-Based Test Generation and Fuzzing Techniques

TitanFuzz, FuzzGPT, ChatUniTest, and Fuzz4All:

TitanFuzz and FuzzGPT are specialized deep learning libraries tailored for different testing requirements and do not align with our focus on Wasm binaries. ChatUniTest focuses on Java test case generation, but some of its features are incompatible with Wasm binaries. Lastly, while Fuzz4All is a generic framework, its application is hindered by the token length of Wasm binaries and LLMs’ limited ability to comprehend raw Wasm binaries effectively.

2. Existing Wasm Testing Techniques

Several Wasm test generation approaches, such as WasmMaker, Wasm-smith, and Wasm-mutate, are designed for testing Wasm runtimes and compilers. These tools primarily generate syntactically correct Wasm code to fuzz Wasm-consuming programs. In contrast, our work focuses on testing Wasm code itself: WasmTest generates unit test cases for a given Wasm code to validate if it is performing the correct functionality and is bug-free.

3. Detailed Comparisons:

To address the reviewer’s concerns, we conducted a comparative analysis of WasmTest and related methods. The results are summarized below, showcasing WasmTest’s performance against adaptations of existing techniques:

• Wasm decompilation works comparison:

We replace Step 1: Generating Functionally Equivalent C++ Code with Wasm decompilation models, which convert Wasm code to equivalent C/C++ code. Specifically, we compare WasmTest with the following binary decompilation tools one at a time. For each decompiler, we use the decompiled C/C++ code as an intermediate representation and prompt the LLM to directly generate tests based on the decompiled version.

1. WaDec [1] + direct generation of tests
2. StackSight [2] + direct
3. Wasm2c [3] + direct

• MuTAP integration:

We incorporate MuTAP [4] into our pipeline. Following the type-aware mutation step, we manually introduce errors into the functions under test to evaluate whether existing test cases can detect these injected bugs. For errors that remain undetected, we utilize LLMs by providing the function under test, the associated test cases, and details about the undetected errors. The LLMs are then prompted to generate new test cases specifically aimed at identifying the highlighted errors.

Table 1: Metrics on HumanEval-X with one of the best performing models - CodeQwen1.5-7B.

Key Observations:

• Wasm2c: Fails to simplify complexities in Wasm binary (verbatim transcription), making it ineffective for guiding LLMs to generate compilable test cases.
• WaDec and StackSight: Despite being state-of-the-art LLM-based Wasm decompilers, their lower compile and pass rates indicate challenges in achieving perfect decompilation.
• MuTAP: Achieves the best compile and pass rates by leveraging LLMs to generate equivalent C++ code based on natural language summaries, which are easier for LLMs to process.
• WasmTest: Outperforms all baselines across metrics due to its design, which incorporates sampling, majority voting, and type-aware mutations, ensuring superior compile rates, coverage, and pass rates.

Reference

[1] She, Xinyu, Yanjie Zhao, and Haoyu Wang. WaDec: Decompiling WebAssembly Using Large Language Model.

[2] Fang, Weike, et al. StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation.

[3] Wasm2c. https://github.com/WebAssembly/wabt/wasm2c

[4] Dakhel, Arghavan Moradi, et al. Effective test generation using pre-trained large language models and mutation testing.

We thank the reviewer for providing constructive feedback. Please find the detailed response below.

Q1: What are the main intended audiences and use cases of WasmTest?

Generating tests for code snippets is crucial for ensuring software quality and reliability, yet it remains a tedious and time-consuming task for developers [1]. Similar to existing efforts in automated test generation [2, 3], our approach focuses on automatically producing a comprehensive test suite for a given program.

WebAssembly (Wasm) is widely used across a wide range of applications. However, Wasm modules are often distributed as third-party binaries without accessible source code (a black-box setting) [4]. In some cases, these binaries may even be obfuscated to hide their actual functionality. According to [5], 28.8% of Wasm binaries are minified to conceal their true purpose. This poses a challenge for developers who need to integrate such Wasm modules into their applications, as they must ensure the module behaves as intended and is free of bugs. In such a setting, developers have no access to the source code and may require LLM-based tools for decompilation and test generation.

In this black-box context, the input and output of the testing process can be described as follows:

• Input: The program under test, accompanied by a description of its intended functionality provided in a natural language summary.
• Output: A set of comprehensive unit tests, including test inputs and expected outputs, designed to validate the module’s correctness and expose potential bugs.

This black-box approach is particularly valuable for developers working with precompiled or obfuscated Wasm binaries, enabling them to verify and trust the functionality of these components within their applications.

Reference

[1] Wang, Junjie, et al. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 2024.

[2] Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Des- marais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171:107468, 2024.

[3] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.

[4] Marius Musch, Christian Wressnegger, Martin Johns, and Konrad Rieck. New kid on the web: A study on the prevalence of webassembly in the wild. In Detection of Intrusions and Malware, and Vulnerability Assessment: 16th International Conference, DIMVA 2019, Gothenburg, Sweden, June 19–20, 2019, Proceedings 16, pp. 23–42. Springer, 2019a

[5] Aaron Hilbig, Daniel Lehmann, and Michael Pradel. An empirical study of real-world webassembly binaries: Security, languages, use cases. In Proceedings of the web conference 2021, pp. 2696– 2708, 2021

Q3: How does WasmTest perform when the natural language descriptions of the Wasm binaries are not available?

In our setting, developers that aim to test a given Wasm module should have in mind the intended purpose of this module, such as computing a hash function. The test suite should be devised to evaluate whether the module (1) correctly fulfills its functionality requirements and (2) is free of bugs.

When a high-level summary of the Wasm code is unavailable, reverse engineering techniques can be employed to decompile the Wasm binary into a C++ implementation. We presented our comparison with models and tools such as WaDec, StackSight, and Wasm2c in Table 1.

Besides, we note that if the code summary is not of good quality, it may be refined via existing models that are capable of Wasm code summarization, such as WasmRev [1] and StackSight. Therefore, for the purposes of our evaluation, we assume that a correct and reliable summary of the Wasm code is available, as it forms the foundation for accurate test generation.

[1] Huang, Hanxian, and Jishen Zhao. "Multi-modal Learning for WebAssembly Reverse Engineering." Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024.

Q4: How to ensure that the generated C++ code is functionally equivalent to the original Wasm binaries? What is the false positive rate of the generated testing results?

We appreciate the reviewer’s insightful comment and would like to clarify the objectives and assumptions of our approach. The primary goal of our test-case generation pipeline is to verify that a WebAssembly (Wasm) binary adheres to its stated functional description, rather than performing direct code translation.

To achieve this, our methodology involves generating C++ implementations from the natural language description of the intended functionality, without any reference to the Wasm binary. The task assigned to the LLM is to produce C++ code that fulfills the functional requirements outlined in the description. This approach does not rely on semantic-preserving translation; instead, it assumes that the provided code summary is accurate and reliable. Previous studies have demonstrated that LLMs perform reasonably well in similar code generation tasks.

Our approach is based on the following key assumptions:

• Accuracy of the Functional Description: The natural language description accurately reflects the intended functionality of the code.
• Test Case Effectiveness: Test cases derived from the generated C++ implementations can identify discrepancies in the Wasm binary, particularly for edge cases or buggy scenarios.

We acknowledge that there are false positives in the testing outcomes, where the Wasm binary is correct but the generated test failed. These results are presented in Table 4 (pass rate) of our draft. Specifically, we run the generated tests on the ground-truth Wasm binary compiled from canonical solutions from HumanEval-X. These canonical solutions from HumanEval-X are human-crafted and verified, so they are expected to pass all tests. The pass rate is around 80% across models, indicating a false positive rate of around 20%. These false positives are mainly caused by errors in the code generated by LLMs in Step 1 (such as failing to account for some edge cases), which propagate the error to later steps.

Q2: Novelty in Mutation

Our mutation process builds upon prior work [1] to systematically mutate test inputs and generate diverse test cases. However, unlike previous schemes designed for human-written tests, our mutation techniques are specifically optimized for various forms of LLM-generated tests. This includes unique structures such as function calls, container elements, and logical operations.

While we acknowledge the potential for more innovative mutation techniques, our current approach demonstrates strong performance, achieving approximately 95% code coverage for both C++ and Wasm. Notably, the tests are able to detect carefully crafted bugs (negative examples) almost all the time, with DeepSeek-Coder-V2 showing exemplary performance with a bug detection rate of 98.7% on the MBXP dataset.

It is distinct from the techniques employed in tools like CCmutator, which operate in completely different contexts. CCmutator focuses on mutating source code at the LLVM IR level to inject concurrency-related bugs, aiming to evaluate software testing and verification tools. In contrast, our work targets test case inputs in unit test assertions. The objective of our mutation process is to systematically diversity test case inputs, enhancing the comprehensiveness of the generated test suites and enabling superior code coverage.

This clear distinction highlights the unique focus of our mutation strategy, which is designed to optimize the testing process rather than simulate concurrency-related issues within the program source.

[1] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.

Q3: How does WasmTest ensure the semantic-preserving of code translation on the C++ implementation generation process?

We appreciate the reviewer’s insightful comment and would like to clarify the objectives and assumptions of our approach. The primary goal of our test-case generation pipeline is to verify that a WebAssembly (Wasm) binary adheres to its stated functional description, rather than performing direct code translation.

To achieve this, our methodology involves generating C++ implementations from the natural language description of the intended functionality, without any reference to the Wasm binary. The task assigned to the LLM is to produce C++ code that fulfills the functional requirements outlined in the description. This approach does not rely on semantic-preserving translation; instead, it assumes that the provided code summary is accurate and reliable. Previous studies have demonstrated that LLMs perform reasonably well in similar code generation tasks.

Our approach is based on the following key assumptions:

• Accuracy of the Functional Description: The natural language description accurately reflects the intended functionality of the code.
• Test Case Effectiveness: Test cases derived from the generated C++ implementations can identify discrepancies in the Wasm binary, particularly for edge cases or buggy scenarios.

By focusing on generating valid test cases based on the stated functionality, our approach aims to evaluate whether the Wasm binary deviates from its intended behavior or contains bugs. Ultimately, the goal is to ensure that the Wasm binary complies with the functional requirements, rather than to establish a direct translation from Wasm to C++.

Q4: More details about bug detection (fault types, background of involved engineers, etc.).

We appreciate the reviewer’s thoughtful comments. The manually crafted bugs were introduced by hard coding modifications to existing example cases in the functionality descriptions. These modifications include the following fault types:

1. Logical Errors: Modifications to loop conditions (e.g., changing <= to <) to introduce off-by-one errors.
2. Special Case Handling: Failure to correctly process edge inputs, such as empty strings or boundary values.
3. Semantic Misinterpretations: Incorrect implementation of functionality, such as sorting in reverse order instead of ascending.

We will release these manually crafted buggy implementations alongside our pipeline. The bugs were created by authors with experience teaching entry-level programming courses at the university level. This background in teaching provided valuable insights into common mistakes made by novice programmers, which helped in designing realistic and diverse faults for this evaluation.

We thank the reviewer for providing constructive feedback. Please find the detailed response below.

[Q1] More comparison with existing Wasm test generation methods and LLM based methods

We have updated the related works section to include the recommended methodologies, noting that most operate under fundamentally different settings compared to our approach.

1. LLM-Based Test Generation and Fuzzing Techniques

TitanFuzz, FuzzGPT, ChatUniTest, and Fuzz4All:

TitanFuzz and FuzzGPT are specialized deep learning libraries tailored for different testing requirements and do not align with our focus on Wasm binaries. ChatUniTest focuses on Java test case generation, but some of its features are incompatible with Wasm binaries. Lastly, while Fuzz4All is a generic framework, its application is hindered by the token length of Wasm binaries and LLMs’ limited ability to comprehend raw Wasm binaries effectively.

2. Existing Wasm Testing Techniques

Existing WebAssembly (Wasm) test generation approaches mainly target fuzzing Wasm runtimes, compilers, or validators, rather than testing the functionality of the Wasm binary itself. Our work focuses on generating unit tests to test a given Wasm code. For example:

• WASMaker generates complicated Wasm binaries (tests) that are both semantically rich and syntactically correct, focusing on detecting bugs or inconsistencies in Wasm runtimes.
• Wasm-smith produces fuzzed Wasm modules for testing the robustness and correctness of Wasm runtimes, validators, or parsers in a black-box manner.
• Wasm-mutate uses e-graphs to create fuzzed Wasm snippets to test Wasm compilers and interpreters. By mutating a given Wasm program, it generates semantically equivalent variants for Wasm compiler fuzzing.

3. Detailed Comparisons:

We conducted a comparative analysis of WasmTest and related methods. The results are summarized below.

• Wasm decompilation works comparison:

We replace Step 1: Generating Functionally Equivalent C++ Code with Wasm decompilation models, which convert Wasm code to equivalent C/C++ code. Specifically, we compare WasmTest with the following binary decompilation tools one at a time. For each decompiler, we use the decompiled C/C++ code as an intermediate representation and prompt the LLM to directly generate tests based on the decompiled version.

1. WaDec [1] + direct generation of tests
2. StackSight [2] + direct generation of tests
3. Wasm2c [3] + direct generation of tests

• MuTAP integration:

We incorporate MuTAP [4] into our pipeline. Following the type-aware mutation step (Step 3), we manually introduce errors into the functions under test to evaluate whether existing test cases can detect these injected bugs. For errors that remain undetected, we utilize LLMs by providing the function under test, the associated test cases, and details about the undetected errors. The LLMs are then prompted to generate new test cases specifically aimed at identifying the highlighted errors.

Table 1: Metrics on HumanEval-X with one of the best performing models - CodeQwen1.5-7B.

Key Observations:

• Wasm2c: Fails to simplify complexities in Wasm binary (verbatim transcription), making it ineffective for guiding LLMs to generate compilable test cases.
• WaDec and StackSight: Despite being state-of-the-art LLM-based Wasm decompilers, their lower compile and pass rates indicate challenges in achieving perfect decompilation.
• MuTAP: Achieves the best compile and pass rates by leveraging LLMs to generate equivalent C++ code based on natural language summaries, which are easier for LLMs to process.
• WasmTest: Outperforms all baselines across metrics due to its design, which incorporates sampling, majority voting, and type-aware mutations, ensuring superior compile rates, coverage, and pass rates.

Reference

[1] She, Xinyu, Yanjie Zhao, and Haoyu Wang. WaDec: Decompiling WebAssembly Using Large Language Model.

[2] Fang, Weike, et al. StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation.

[3] Wasm2c. https://github.com/WebAssembly/wabt/wasm2c

[4] Dakhel, Arghavan Moradi, et al. Effective test generation using pre-trained large language models and mutation testing.

Q2: Impact of inadequate or absent High-level Code Descriptions

In our setting, developers that aim to test a given Wasm module should have in mind the intended purpose of this module, such as computing a hash function. The test suite should be devised to evaluate whether the module (1) correctly fulfills its functionality requirements and (2) is free of bugs.

When a high-level summary of the Wasm code is unavailable, reverse engineering techniques can be employed to decompile the Wasm binary into a C++ implementation. We presented our comparison with models and tools such as WaDec, StackSight, and Wasm2c in Table 1.

Besides, we note that if the code summary is not of good quality, it may be refined via existing models that are capable of Wasm code summarization, such as WasmRev [1] and StackSight. Therefore, for the purposes of our evaluation, we assume that a correct and reliable summary of the Wasm code is available, as it forms the foundation for accurate test generation.

[1] Huang, Hanxian, and Jishen Zhao. "Multi-modal Learning for WebAssembly Reverse Engineering." Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024.

We thank the reviewer for providing constructive feedback. Please find the detailed response below.

[Q1] More comparison with existing testing methods

We have updated the related works section to include the some relevant methodologies, noting that most operate under fundamentally different settings compared to our approach. Below, we address the specific concerns and provide detailed comparisons:

1. LLM-Based Test Generation and Fuzzing Techniques

TitanFuzz, FuzzGPT, ChatUniTest, and Fuzz4All:

TitanFuzz and FuzzGPT are specialized deep learning libraries tailored for different testing requirements and do not align with our focus on Wasm binaries. ChatUniTest focuses on Java test case generation, but some of its features are incompatible with Wasm binaries. Lastly, while Fuzz4All is a generic framework, its application is hindered by the token length of Wasm binaries and LLMs’ limited ability to comprehend raw Wasm binaries effectively.

2. Existing Wasm Testing Techniques

Several Wasm test generation approaches, such as WasmMaker, Wasm-smith, and Wasm-mutate, are designed for testing Wasm runtimes and compilers. These tools primarily generate syntactically correct Wasm code to fuzz Wasm-consuming programs. In contrast, our work focuses on testing Wasm code itself: WasmTest generates unit test cases for a given Wasm code to validate if it is performing the correct functionality and is bug-free.

3. Detailed Comparisons:

To address the reviewer’s concerns, we conducted a comparative analysis of WasmTest and related methods. The results are summarized below, showcasing WasmTest’s performance against adaptations of existing techniques:

• Wasm decompilation works comparison:

We replace Step 1: Generating Functionally Equivalent C++ Code with Wasm decompilation models, which convert Wasm code to equivalent C/C++ code. Specifically, we compare WasmTest with the following binary decompilation tools one at a time. For each decompiler, we use the decompiled C/C++ code as an intermediate representation and prompt the LLM to generate tests based on the decompiled version.

1. WaDec [1] + direct generation of tests
2. StackSight [2] + direct generation of tests
3. Wasm2c [3] + direct generation of tests

• MuTAP integration:

We incorporate MuTAP [4] into our pipeline. Following the type-aware mutation step (Step 3), we manually introduce errors into the functions under test to evaluate whether existing test cases can detect these injected bugs. For errors that remain undetected, we utilize LLMs by providing the function under test, the associated test cases, and details about the undetected errors. The LLMs are then prompted to generate new test cases specifically aimed at identifying the highlighted errors.

Table 1: Metrics on HumanEval-X with one of the best performing models - CodeQwen1.5-7B.

Key Observations:

• Wasm2c: Fails to simplify complexities in Wasm binary (verbatim transcription), making it ineffective for guiding LLMs to generate compilable test cases.
• WaDec and StackSight: Despite being state-of-the-art LLM-based Wasm decompilers, their lower compile and pass rates indicate challenges in achieving perfect decompilation.
• MuTAP: Achieves the best compile and pass rates by leveraging LLMs to generate equivalent C++ code based on natural language summaries, which are easier for LLMs to process.
• WasmTest: Outperforms all baselines across metrics due to its design, which incorporates sampling, majority voting, and type-aware mutations, ensuring superior compile rates, coverage, and pass rates.

Reference

[1] She, Xinyu, Yanjie Zhao, and Haoyu Wang. "WaDec: Decompiling WebAssembly Using Large Language Model." Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2024.

[2] Fang, Weike, et al. "StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation." Forty-first International Conference on Machine Learning.

[3] Wasm2c. https://github.com/WebAssembly/wabt/wasm2c

[4] Dakhel, Arghavan Moradi, et al. Effective test generation using pre-trained large language models and mutation testing.

Q4: Clarification regarding “Correct Test Output Prediction Rates”

We apologize for any confusion caused by our earlier explanation and thanks for pointing out the need for clarification. The 50% correctness rate reported in RQ5 (Table 7) refers to test outputs directly generated by LLMs, which often (~50%) include errors (e.g., isPrime(8) == True). Recognizing this limitation, our approach incorporates a Test Case Correction mechanism through majority voting, as detailed in Sections 3.2 and 3.3. This process significantly enhances the accuracy and reliability of the generated tests.

Test Case Correction:

1. Multiple Implementations: We use multiple LLM-generated C++ implementations for the same functionality.
2. Consensus-Based Output: For each test input, the majority output from these implementations is adopted as the test output. If no majority consensus is reached, the test case is discarded.

This correction mechanism mitigates the errors inherent in direct LLM-generated outputs, ensuring a higher degree of correctness and robustness. To prevent misunderstanding, we have updated our manuscript to clearly explain this mechanism and its role in overcoming the limitations of direct oracle generation by LLMs.

Q5: Novelty in the mutation process

Our mutation process builds upon prior work [1] to systematically mutate test inputs and generate diverse test cases. However, unlike previous schemes designed for human-written tests, our mutation techniques are specifically optimized for various forms of LLM-generated tests. This includes unique structures such as function calls, container elements, and logical operations.

While we acknowledge the potential for more innovative mutation techniques, our current approach demonstrates strong performance, achieving approximately 95% code coverage for both C++ and Wasm. Notably, the tests are able to detect carefully crafted bugs (negative examples) almost all the time, with DeepSeek-Coder-V2 showing exemplary performance with a bug detection rate of 98.7% on the MBXP dataset.

We appreciate the reviewer’s feedback and will explore additional mutation strategies in the revised version to further enhance coverage and bug detection rates.

[1] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.

Q2: What are the applications of this black-box setting?

Generating tests for code snippets is crucial for ensuring software quality and reliability, yet it remains a tedious and time-consuming task for developers [1]. Similar to existing efforts in automated test generation [2, 3], our approach focuses on automatically producing a comprehensive test suite for a given program.

WebAssembly (Wasm) is widely used across a wide range of applications. However, Wasm modules are often distributed as third-party binaries without accessible source code (a black-box setting) [4]. In some cases, these binaries may even be obfuscated to hide their actual functionality. According to [5], 28.8% of Wasm binaries are minified to conceal their true purpose. This poses a challenge for developers who need to integrate such Wasm modules into their applications, as they must ensure the module behaves as intended and is free of bugs. In such a setting, developers have no access to the source code and may require LLM-based tools for decompilation and test generation.

In this black-box context, the input and output of the testing process can be described as follows:

• Input: The program under test, accompanied by a description of its intended functionality provided in a natural language summary.
• Output: A set of comprehensive unit tests, including test inputs and expected outputs, designed to validate the module’s correctness and expose potential bugs.

This black-box approach is particularly valuable for developers working with precompiled or obfuscated Wasm binaries, enabling them to verify and trust the functionality of these components within their applications.

Reference

[1] Wang, Junjie, et al. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 2024.

[2] Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Des- marais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171:107468, 2024.

[3] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.

[4] Marius Musch, Christian Wressnegger, Martin Johns, and Konrad Rieck. New kid on the web: A study on the prevalence of webassembly in the wild. In Detection of Intrusions and Malware, and Vulnerability Assessment: 16th International Conference, DIMVA 2019, Gothenburg, Sweden, June 19–20, 2019, Proceedings 16, pp. 23–42. Springer, 2019a

[5] Aaron Hilbig, Daniel Lehmann, and Michael Pradel. An empirical study of real-world webassembly binaries: Security, languages, use cases. In Proceedings of the web conference 2021, pp. 2696– 2708, 2021

Q3: How does WasmTest verify the equivalence of LLM-generated C++ implementation? Can we apply reverse engineering tools to guide test case generation?

1. Equivalence Verification:

In the benchmark datasets used, we leverage the example tests provided in code summaries (e.g., Figure 2(a)) to filter out incorrect implementations. To further enhance robustness, we do not rely on one single implementation, instead, we adopt a majority voting approach across multiple implementations:

• Test Input Generation: LLMs generate candidate test inputs.
• Expected Output Verification: For each test input, if the majority of implementations produce the same output, this output is adopted as the expected result in the test case. Otherwise, the test case is discarded.

2. Reverse Engineering for Wasm Binaries:

When a Wasm code summary is unavailable, reverse engineering can be applied to decompile the Wasm binary into a C++ implementation. However, this approach typically has lower accuracy due to the inherent complexities of Wasm binaries. As shown in Table 1, methods like StackSight, Wasm2c, and WaDec yield significantly lower compile rates and pass rates, reflecting the limitations of using reverse-engineered outputs in guiding test case generation effectively.

We thank the reviewer for providing constructive feedback. Please find the detailed response below.

[Q1] Additional comparisons with Wasm reverse-engineering tools, LLM-based test case generation models, and fuzzing techniques

We are grateful to the reviewer for highlighting gaps in our literature review and suggesting additional works for consideration. Based on this feedback, we have updated the related works section to include the recommended methodologies, noting that most operate under fundamentally different settings compared to our approach. Below, we address the specific concerns and provide detailed comparisons:

1. LLM-Based Test Generation and Fuzzing Techniques

TitanFuzz, FuzzGPT, ChatUniTest, and Fuzz4All:

TitanFuzz and FuzzGPT are specialized deep learning libraries tailored for different testing requirements and do not align with our focus on Wasm binaries. ChatUniTest focuses on Java test case generation, but some of its features are incompatible with Wasm binaries. Lastly, while Fuzz4All is a generic framework, its application is hindered by the token length of Wasm binaries and LLMs’ limited ability to comprehend raw Wasm binaries effectively.

2. Existing Wasm Testing Techniques

Several Wasm test generation approaches, such as WasmMaker, Wasm-smith, and Wasm-mutate, are designed for testing Wasm runtimes and compilers. These tools primarily generate syntactically correct Wasm code to fuzz Wasm-consuming programs. In contrast, our work focuses on testing Wasm code itself: WasmTest generates unit test cases for a given Wasm code to validate if it is performing the correct functionality and is bug-free.

3. Detailed Comparisons:

To address the reviewer’s concerns, we conducted a comparative analysis of WasmTest and related methods. The results are summarized below, showcasing WasmTest’s performance against adaptations of existing techniques:

• Wasm decompilation works comparison:

We replace Step 1: Generating Functionally Equivalent C++ Code with Wasm decompilation models, which convert Wasm code to equivalent C/C++ code. Specifically, we compare WasmTest with the following binary decompilation tools one at a time. For each decompiler, we use the decompiled C/C++ code as an intermediate representation and prompt the LLM to directly generate tests based on the decompiled version.

1. WaDec [1] + direct generation of tests
2. StackSight [2] + direct
3. Wasm2c [3] + direct

• MuTAP integration:

We incorporate MuTAP [4] into our pipeline. Following the type-aware mutation step, we manually introduce errors into the functions under test to evaluate whether existing test cases can detect these injected bugs. For errors that remain undetected, we utilize LLMs by providing the function under test, the associated test cases, and details about the undetected errors. The LLMs are then prompted to generate new test cases specifically aimed at identifying the highlighted errors.

Table 1: Metrics on HumanEval-X with one of the best performing models - CodeQwen1.5-7B.

Key Observations:

• Wasm2c: Fails to simplify complexities in Wasm binary (verbatim transcription), making it ineffective for guiding LLMs to generate compilable test cases.
• WaDec and StackSight: Despite being state-of-the-art LLM-based Wasm decompilers, their lower compile and pass rates indicate challenges in achieving perfect decompilation.
• MuTAP: Achieves the best compile and pass rates by leveraging LLMs to generate equivalent C++ code based on natural language summaries, which are easier for LLMs to process.
• WasmTest: Outperforms all baselines across metrics due to its design, which incorporates sampling, majority voting, and type-aware mutations, ensuring superior compile rates, coverage, and pass rates.

Reference

[1] She, Xinyu, Yanjie Zhao, and Haoyu Wang. WaDec: Decompiling WebAssembly Using Large Language Model.

[2] Fang, Weike, et al. StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation.

[3] Wasm2c. https://github.com/WebAssembly/wabt/wasm2c

[4] Dakhel, Arghavan Moradi, et al. Effective test generation using pre-trained large language models and mutation testing.