PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification

Thanks for the constructive and positive comments.

1. Extra Clarification on Terminology: "Human-Based Planning"

We thank the reviewer for pointing this out. We agree that the term "human-based planning" may be misleading, as it could imply a human-in-the-loop system. To avoid this confusion, we will revise it to "Rule-based planning", which more accurately reflects our intent—i.e., that the plan is rules pre-specified by developers.

2. Wall-Time Comparison with Agentless

We followed the reviewer’s suggestions and conducted an experiment to compare the wall-time of our system with Agentless and OpenHands on 100 cases (the selection of these cases is stated in the response-1 to reviewer Lckx). Since all tools support multiprocessing, we controlled the number of processes to be the same for all methods (8) to ensure a fair comparison. We measured the average time per instance and ran the experiment three times. The results are as follows.

The results show that PatchPilot is faster than agentless and OpenHands. It also shows that PatchPilot is more effective than the baseline approaches.

Thanks for the constructive and positive comments.

1. Better experiment analysis

We will follow the reviewer’s suggestion and include more visualizations to show the comparison between our method and baselines in effectiveness, stability, and cost. To better demonstrate the advantage of PatchPilot over Agentless, we reran these two methods on 100 cases three times (the selection of these cases is stated in response-1 to reviewer Lckx) and reported their resolved rate, average cost, and run time.

The results show that PatchPilot consistently outperforms Agentless in all three metrics.

2. Mitigating Potential Instability Introduced by Iterative Refinement

Thanks for pointing this out. We incorporate two strategies to improve stability during refinement.

(1) Batch-based Refinement with Selection Mechanism: Instead of generating a single patch per iteration by directly modifying the previous one, PatchPilot generates multiple patch candidates in each batch based on the previous best patch. It then evaluates all candidates—together with the previous best patch—using both the PoC and functionality tests and selects the best-performing one. If all current candidates underperform compared to the previous best patch, the previous best patch is resolved. This mechanism significantly reduces the risk of accumulated generation noise and protects against overwriting correct earlier solutions (validated by our hyper-parameter experiment when we reduce the batch size from 4 to 1, the overall performance drops notably).

(2) Early Stopping: As soon as a patch passes all tests, iterative refinement is terminated immediately, and the patch is returned. This prevents unnecessary iterations without useful feedback, avoiding potential degradation from further blind refinement.

3. Failure Cases of PatchPilot

We will include more analysis of failure cases. In summary, some of the failure cases are because the issue descriptions are of low quality, such as incomplete or incorrect descriptions of the issues. To resolve these cases, we need to rewrite high-quality issue descriptions. Some other cases are shallow patches, where PatchPilot only patches the superficial logic instead of addressing the root cause and does not consider the corner case of the issue, or over-patching, where patches inadvertently affect more scenarios than necessary, causing unexpected behavior. These cases may be further resolved by fine-tuning specific models that can better understand the program logic of the target repos.

Thanks for the constructive comments.

1. Novelty and Differences from Existing Tools

As discussed in Section 2, [a-g] are all agent-based, differing from our workflow (we included [b-g] and will add [a]).

• Search tools: We acknowledge that AutoCodeRover and CodeR also have search tools, and we are inspired by their designs. The key differences are in the search mechanisms, where we use fuzzy search instead of exact search. As the searched strings may differ from the original code segments, the exact search misses some matches.
• Coverage: AutoCodeRover's SBFL method requires post-patching test cases, which are unavailable in SWE-bench problem settings. PatchPilot generates coverage using only PoCs.
• Separating planning and generation: We respectfully point out that AppMap Navie is a commercial product without research papers. Although it mentions having generation plans, the technical details are not available. Our separation design was inspired by our early observations of shallow patches. Besides, the detailed design can be very different. For example, we proposed diversity prompts and step-by-step generation. We believe it is reasonable to claim the novelty and originality of our design of the generation component.
• Patch validation vs. Agentless: Agentless generates assertions for testing cases and validates based on assertions. It is hard to generate correct and comprehensive assertions. PatchPilot generates only PoCs and uses LLMs to validate based on PoC outputs. We applied both methods to 100 patches (50 from PatchPilot, 50 from Agentless). Our method identified 43 correct patches; Agentless identified 40.
• PatchPilot differs from Agentless in the workflow (refinement) and applies optimizations to each component. Although the components' high-level ideas are similar, the specific designs can vary a lot and affect final performance, as shown in the comparison with Agentless.
• About refinement: Agent-based tools refine the latest patch by prompting the LLM with prior conversation history. This simple method has a few limitations and cannot be applied to PatchPilot: 1) It generates only one patch in the first round and iteratively refines it, which lacks diversity. If the initial patch contains errors, it may be hard to fix. This is partially validated in our hyper-parameter experiments with a batch size of 1, showing a performance drop. 2) It cannot revert to earlier patches if a refinement worsens the result, while PatchPilot enables such a capability. 3) By including the refinement history in prompts, it risks context overflow and potential LLM confusion from prior bad patches. PatchPilot instead uses only the best previous patch and its validation results. Given these major differences, we believe it is reasonable to claim novelty for our refinement.

While existing tools may share similar high-level ideas, our design is fundamentally different and more advanced, giving better performance and stability. We believe this brings essential novelty and contribution.

2. Extra Ablation Studies for Hyperparameters

We conducted experiments to evaluate hyper-parameter sensitivity. PatchPilot includes the following hyper-parameters (default values in brackets): number of retrieved files (5), total patches generated (12), batch size per round (4), diversity prompts (3), and generation temperature (0.8). Each experiment varies one hyper-parameter while keeping others. We used 60 cases randomly selected from SWE-bench-Lite and Claude-3.5-Sonnet.

• Num. of retrieved files in localization: We changed it from 5 to 3 and 7. Varying it introduces subtle differences, but retrieving more files will have a higher cost as LLM will have more conversations and longer contexts.
• Total num. of patches: Changing it from 12 to 1, 4, 8, 16. A large one slightly improves performance but raises costs. 12 strikes a balance between performance and cost.
• Batch Size: Changing the batch size from 4 to 2/6 introduces performance drops. For 2, the diversity is not enough; for 6, the refinement round (2) is not enough. A moderate batch size balances the refinement rounds and diversity.
• Diversity Prompts: Instead of using three prompts together, we ran the experiment with each prompt separately. Mixing them gives the best performance as it enables the highest diversity.
• We varied the temperature ($\tau$) from 0.8 to 1 and 0.6. The results are similar. We set it high to encourage diversity.

3. Stability Evaluation

Our stability experiment is in response-1 to reviewer Lckx.

Thanks for the positive and constructive comments.

1. Stability Comparison

We chose GPT-4o because of our budget limits before. We reran the stability comparison. First, we changed GPT-4o to Claude-3.5-Sonnet. Second, in response to Reviewer HrpY’s concern about small sample size, we increased the testing cases from 45 to 100 from SWE-bench-Lite. These cases were selected from three categories based on these methods’ results in one previous run: 30 are commonly resolved by all tools, 15 are solved independently by each of the three tools, and 55 are commonly unresolved cases (hard ones). The cases in each category are randomly selected. Third, we added Agentless based on Reviewer vPVX’s suggestion. We calculated the standard deviation (STD) of the resolved rate and cost. To show the statistical significance, we conducted a Bartlett’s test to compare the variance between PatchPilot and the comparison baselines. Our Null hypothesis $H0: \sigma^2_p = \sigma^2_b$, $\sigma^2_p$ and $\sigma^2_b$ are the variance of PatchPilot and baseline. The results are as follows (the first number in p-value is the comparison of resolved rate, and the other is the cost).

The result confirms that PatchPilot has a much lower STD than OpenHands in the resolved rate. Besides, the $p$-value is smaller than 0.05, meaning we can reject the null hypothesis. Given that the STD of PatchPilot is smaller than OpenHands, this shows that PatchPilot is more stable than OpenHands with statistical significance. The resolved rates of all methods are lower than those in Table 1 because we include more hard cases in the 100 cases. The cost shows a similar trend. OpenHands is relatively more cost-effective when using Claude-3.5 compared to using GPT-4o in Section 4.2, as a stronger model can reduce the number of iterations of OpenHands.

The result also shows the stability of Agentless, validating our claim that human-based planning is more stable than agent-based planning. Finally, PatchPilot consistently achieves lower cost and higher performance than Agentless.

2. Illustrative Example of Patch Planning and Generation

We will include more examples. Here, we show one example of django__django-11133. Django is a Python web framework. In this issue, the user reported that HttpResponse failed to handle memoryview objects.

PatchPilot generated a patch plan with one step:

<STEP> Add an explicit memoryview check in the content setter </STEP> <Actions> In the file django/http/response.py, locate the content setter of HttpResponse. Modify the code so that it first checks if the provided value is an instance of memoryview. If it is, immediately return bytes(value). Otherwise, proceed with the existing check for iterables. </Actions>

Following the plan, PatchPilot generates a correct patch:

         # - Python attempts str conversion first
         # - when self._charset != 'utf-8' it re-encodes the content
-        if isinstance(value, bytes):
+        if isinstance(value, (bytes, memoryview)):
             return bytes(value)

3.PoC Validation Beyond LLM Checking

We first convert the issue description into a structured format, extracting the expected behavior and wrong behavior. We also include the following rule-based checks.

Rule 1: Discard PoCs that cover fewer than 3 files, as LLMs often create fake buggy functions inside the PoC instead of invoking real project code.

Rule 2: If a specific exception is mentioned in the issue, we check if the same exception appears in stderr during PoC execution, confirming it triggers the described error.

To evaluate PoC quality, we manually reviewed 100 cases. Our validation filtered out 27 PoCs (20 by LLM, 5 by Rule 1, and 2 by Rule 2). Of the remaining 73 PoCs, 92% are correct. These results confirm the effectiveness of our patch validation and the high quality of generated PoCs.

4. Majority Voting Strategy in Ablation Study

In the first two variants, we use majority voting since patch validation is not feasible without PoCs or functionality tests. We generate multiple patches per case and proceed as follows: Normalize each patch by stripping whitespace, line breaks, and comments; count the frequency of each normalized patch; and select the most frequent one. If there is a tie, we prompt an LLM to choose the best patch.

5. Novelty of Refinement in PatchPilot

We discuss the novelty of our refinement in response-1 to reviewer HrpY.