Thank you for the review. To ensure we address your concerns with precision, could you kindly clarify your suggestions? We are delighted to address any questions you may have and refine our paper accordingly. We look forward to your feedback and reassessment.

We are sincerely grateful for your recognition of RaLU’s contributions. Many thanks to your constructive comments to enhance our work.

Questions

We appreciate the reviewer's insightful question. Let’s use perplexity=exp(-1* Mean(token probabilities)) as another metric using “multiplying”. We have supplemented an ablation study, which revealed that the impact of the selection strategy is marginal. Specifically, we conducted experiments using Qwen-72B-Instruct on the MATH dataset because the MATH dataset is complex enough, and there are enough cases (89/700) where the branch reaches the threshold, and different selection strategies may have a relatively significant impact on the final result. We used three comparison strategies: random selection, choosing the candidate with the minimum perplexity, and choosing the last one. The accuracy results with the 89 cases are as follows.

This can be attributed to two key factors:

• During self-correction iterations, the LLM tends to produce tokens with consistently high probabilities (the average probability >0.9 for most responses in our analysis). This results in minimal variance between the average probability (our confidence score) and perplexity (geometric mean equivalent) metrics - their Pearson correlation reaches 0.93. Essentially, both metrics reflect similar confidence patterns.
• Through qualitative analysis, we observed that many candidate units generated within budget limits are functionally equivalent variants differing only in implementation details. This semantic equivalence explains why random selection only causes minor performance degradation.

While perplexity-based selection slightly outperforms our original strategy (+0.9%), the limited gain suggests that the verification-revision loop in Stage 2 already filters out most critical errors before selection occurs. It also indicates that multiplying can be a insightful metric and deserves consideration. In the revised appendix, we will add relevant discussions and provide different candidate selection strategies in the code to accommodate various scenarios. Thank you again for helping us enhance this work.

Strengths And Weaknesses

Inference cost: We appreciate the reviewer's insightful comment. We list the average token consumption of RaLU and baselines on MATH-np using Qwen-72B-Instruct. RaLU consumes 15x tokens compared to CoT, while saving 10x tokens compared to multi-path reasoning baselines such as Self-Check and ToT. We will include the information about token cost in Appendix.

Other Comments

1. Thanks for pointing out the error of equation (4). We correct it as $\text{Conf}(\tilde{\mathcal{U}}) = \frac{1}{n}\sum_{j=1}^{n} \sigma(lp_j)$

2. Thank you very much for pointing out our clerical error. In the Appendix, $1-\beta$ also needs to be replaced with $\beta$, and the rest of the derivation process and conclusions remain unchanged. Thank you very much for your meticulous review.

Thank Reviewer KWVa for your constructive feedback on our work. We have carefully considered all comments and hope our point-by-point response can address your questions.

Questions

1. Multiple runs of experiments: Thank you for your feedback! Given the inherent stochastic nature of LLMs and the absence of seed support in the official API, we additionally conducted two independent trials on Qwen-72-Instruct for MATH and Mbpp/Mbpp+ benchmarks (temperature=0.7, identical prompting strategies) to further validate the robustness of RaLU under different model configurations. While these preliminary results (reported as mean ± std in Table 2) align with our prior findings, we will finalize three full replications across all models and datasets before publication to ensure statistical rigor. This iterative process strengthens our confidence in RaLU's consistent performance across varying model architectures and task domains. | Benchmark | Direct | CoT | ToT | PoT/SR* | SC | SCal | SCheck/SD | RaLU | | - | - | - | - | - | - | - | - | - | | Mbpp | 0.923±0.0070 | 0.895±0.0014 | 0.905±0.0045 | 0.860±0.0046 | 0.922±0.0033 | 0.924±0.0038 | 0.905±0.0038 | 0.957±0.0012 | | Mbpp+ | 0.788±0.0021 | 0.761±0.0052 | 0.772±0.0046 | 0.725±0.0038 | 0.779±0.0061 | 0.787±0.0122 | 0.750±0.0349 | 0.856±0.0017 | | Math-np | 0.705±0.0065 | 0.701±0.0085 | 0.695±0.0045 | 0.743±0.0223 | 0.764±0.014 | 0.725±0.0065 | 0.685±0.0088 | 0.803±0.0085 |

2. Ablation Study of Candidate Unit Selection: Thank you for your insightful suggestion. We conducted ablation experiments using Qwen-72B-Instruct on MATH, which is sufficiently complex with enough cases (89/700) where the branch reaches the threshold. We used three comparison strategies: random selection, choosing the candidate with the minimum perplexity, and choosing the last one. The accuracy results are as follows.

The results revealed that the impact of the selection strategy is marginal. During self-correction iterations, LLMs tend to produce tokens with high probabilities. This results in minimal variance between the average probability (confidence) and perplexity (geometric mean equivalent) metrics. Through qualitative analysis, we observed that many candidate units generated within budget limits are functionally equivalent variants differing only in implementation details. This semantic equivalence explains why random selection only causes minor performance degradation.

Due to the limited space, we provide a more detailed description in our response to Reviewer jMQo, Question Part, for your reference.

3. ChatRepair Comparison: Thank you for your suggestion. We will discuss the differences between ours and ChatRepair-like approaches more comprehensively. As you mentioned, the core design concepts of the two are distinct. Our approach relies entirely on the model's intrinsic reasoning capabilities, whereas ChatRepair depends on external feedback for corrections. In practice, it is challenging to obtain comprehensive test cases. ChatRepair also struggles with complex mathematical reasoning and may cause overfitting and data leakage using test data. As a feedback-based correction method, ChatRepair does not sufficiently align with our method in terms of problem assumptions, input constraints, and evaluation metrics. Our method has been comprehensively compared with relevant baselines for test-time reasoning enhancement. The baseline papers also did not compare against ChatRepair. Also, ChatRepair is an Automated Program Repair tool that aims to generate patches for buggy programs instead of code generation or reasoning.

Claims

1. RaLU v.s. Self-Correction Methods: We appreciate the reviewer's insightful feedback regarding our analysis of self-correction methods. We will make the expressions more rigorous by changing it to “Many existing self-correction-based methods (e.g., Self-Refine, Self-Debug) implicitly encourage differences between self-corrected responses and initial responses, which can potentially introduce errors into initially correct responses.” According to their original papers, Self-Refine and Self-Debug prompt the LLM to fix the response based on self-generated or external feedback and adopt the newly generated response as the final result. On the other hand, Self-Check generates multiple candidate steps after information extraction and other procedures. It then decides on step-through voting, maintaining the possibility of retaining the original response. In Self-Check, different candidate responses are treated equally. Our method also allows retaining the original response to mitigate the issue of introducing errors into a correct response.

We are grateful for your valuable comments and hope this response can address your questions.

Questions

1. We select the latest models from three renowned open-source families. The effectiveness of RaLU is not directly tied to the model's size but rather depends on the model's reasoning capabilities and the ability to follow detailed instructions. We supplemented experiments with Qwen-14B on MATH and Mbpp/Mbpp+. As shown in the Table, RaLU still provides significant improvements for smaller yet capable models. |Benchmark|Direct|CoT|ToT|PoT/SR*|SC|SCal|SCheck/SD|RaLU| |-|-|-|-|-|-|-|-|-| |Mbpp|0.840|0.860|0.831|0.804|0.868|0.852|0.852|0.902| |Mbpp+|0.725|0.733|0.720|0.698|0.754|0.706|0.714|0.839| |MATH-np|0.603|0.691|0.651|0.731|0.751|0.710|0.593|0.784|

2. Due to the limited space, please refer to our response to Reviewer jMQo, Strengths And Weaknesses

3. We have added AQUA on Qwen-72B-Instruct. Impressively, our RaLU framework continues to achieve the best performance. We agree that more benchmarks would further validate RaLU's robustness and plan to include AQUA in our revised version.

Claims

1. Thank you for your insights on "reasoning hallucinations." Here, they primarily refer to the logic mismatch between code- and NL-based reasoning—the key challenge RaLU addresses. We do not cover traditional logical inconsistencies (e.g., reasoning-final answer mismatch) and will clarify this in the revised paper.

Given the difficulty in exhaustively categorizing reasoning errors, we adopt a top-down approach: abstracting reasoning hallucinations as disruptions to one-to-one sequence mappings (e.g., "12345"$\leftrightarrow$ "abcde"). These fall into three core types:

• Element errors
• Missing/redundant elements
• Sequence misordering Other errors can be viewed as combinations of these. (Note: We’ll add "or vice versa" to "1) accurate NL step…").

We acknowledge the theoretical challenge of covering all error types due to real-world complexity. If they are hallucinations independent of the three types, we'll supplement experiments to further analyze whether RaLU can mitigate the new type. Our classification aims to guide technical improvements—even if marginal types exist, solving these three significantly boosts reliability, with extensible methodology.

2. We will replace "verified" with "self-verified" to enhance the rigor of the text, emphasizing that this is an internal validation process only relying on the LLM itself to prevent such ambiguity.

3. Although the LLM in RaLU may misjudge a unit, this is not a self-preference bias. Self-preference bias occurs when the LLM favors answers aligned with its training distribution, deviating from human preferences [1]. In RaLU, self-judgment only assesses correctness—misclassifying an incorrect unit as correct is an error (not a bias), affecting its final accuracy, not human-aligned processes. A more relevant issue might be confidently incorrect predictions, where the LLM overestimates its answers' correctness. To mitigate this, RaLU separates program generation and judgment/refinement into different dialogues, obscuring the units' source. Additionally, units are extracted from generated programs with modified indicators, reducing overconfidence in familiar distributions. [1] S. Dhuliawala, et. al, "A Diachronic Perspective on User Trust in AI under Uncertainty”

4. The questions are around alignment: cross-unit alignment, inter-statement alignment, and unit-to-program alignment. RaLU employs conditional generation. Though it doesn’t guarantee absolute consistency, its strength lies in dynamic context modeling and flexible dependency capture, avoiding the rigidity of hard constraints.

• Corss-Unit: Units are constrained by prior ones during refinement, aided by Transformer’s self-attention for implicit semantic linking. For example, variable name changes in earlier units propagate to later ones. Hard constraints risk inconsistencies if dependencies are incomplete.
• Code: Theoretically feasible, but RaLU prioritizes logic over syntax. Forced concatenation imposes rigid boundaries, whereas LLM regeneration dynamically optimizes interfaces (e.g., auto-completing variables) and adheres to syntax rules via pre-trained knowledge.
• Synthesis: Conditional generation may introduce minor misalignment, but RaLU’s verification units steer the LLM toward valid solutions. Hard constraints risk overfitting to rules at the expense of semantics (e.g., redundant type conversions). Key logic (e.g., boundary handling) is preserved via attention weights, while non-critical parts are optimized—mimicking human programming cognition.

5. The decision is not made by the LLM. Each node in CFG has <= 2 children (if, else); We traverse CFG using depth-first search, organizing the nodes along the way into units to be entered in the order.