We thank the reviewer for their positive evaluation and constructive comments. We have performed the requested supplementary analyses and will incorporate them into the final version.

Q1: The ablation studies clearly demonstrate that the dual-reward strategy improves performance. However, it is better for the authors to more explicitly articulate the intuition or theoretical basis for combining these specific rewards.

Thank you for this excellent question. You are right that a more explicit articulation of the synergy between POR and CPR is crucial. The decision to combine these two rewards is not arbitrary; they are designed to solve two distinct, fundamental challenges in training LLMs for complex decision-making, and their combination creates a powerful synergistic effect.

We have discussed the individual theoretical and empirical justifications for each component with other reviewers, but we will summarize the core intuition here.

1. POR Addresses Noisy Outcome Signals: As detailed in our theoretical analysis (discussed with Reviewer 5e7B and 4Ria), POR's primary role is to stabilize the learning signal related to the final outcome of an action. It tackles the noisy credit assignment problem by creating a low-bias, low-variance policy gradient. In essence, POR teaches the model WHAT a good action looks like relative to its peers, even when the absolute scores are unreliable.

2. CPR Addresses Sparse Process Signals: In long-horizon tasks, rewards are often sparse and delayed. CPR's role is to provide a dense, stable, and immediate reward for the reasoning process itself. It incentivizes the model to first achieve a correct qualitative understanding of the current state. In essence, CPR teaches the model WHY a certain context calls for a specific type of action, independent of the final outcome.

NaDRO combines these to create a more robust and intelligent agent. By rewarding both the final action preference (POR) and the preceding cognitive process (CPR), we ensure the model learns a policy that is not only effective in its choices but is also grounded in a correct and verifiable understanding of the problem context. This synergy moves the approach from a "heuristic" combination to a principled framework for fostering robust, reason-based decision-making.

Q2: Given that the paper does not provide a theoretical analysis for NaDRO, the empirical results become only evidence supporting the effectiveness of the method. Therefore, it is better to add even a minimal variance analysis across different random seeds to demonstrate that the observed performance improvements are robust.

Thank you for this essential feedback. We completely agree that for an empirically-driven study like ours, demonstrating the statistical robustness of the results across different random seeds is critical to validate our claims.

To address this, we have conducted multiple runs for all our experiments and have updated our main results table to include both the mean and standard deviation for all reported performance metrics. The detailed table, which we have provided in our response to Reviewer 4Ria, confirms our findings.

As the updated results demonstrate, our NaDRO-trained models consistently outperform the baselines. More importantly, the low standard deviations associated with our results indicate that these performance improvements are not an artifact of a specific random seed but are indeed robust and statistically significant. This variance analysis provides the necessary empirical evidence to confirm that our method's effectiveness is reliable and reproducible.

Q3: In the experiments, the comparison methods are appropriately cited. However, it would be helpful to add a brief explanation of the algorithms, for instance, clarifying that LKH stands for ‘Lin-Kernighan-Helsgaun’ and is a widely used heuristic for solving the TSP.

Thank you for the positive feedback and this very helpful suggestion. We agree that providing brief descriptions of the baseline algorithms will enhance the clarity of our experimental section.

In the camera-ready version, we will add concise explanations for key methods like LKH (Lin-Kernighan-Helsgaun), GLS, and others, following the excellent example you provided. This will help readers better understand the context of our results.

Q4: While the NaDRO outperforms other LLM-based methods on MCTS-generated data, the conclusion in claiming offering a promising pathway towards real-world noisy data handling is not fully justified by current experiments. I would suggest adding a brief discussion on what types of real-world noise might be relevant and how NaDRO could be adapted or expected to handle them.

Thank you for this insightful question about justifying the generalization of our method to real-world noise.

We address this by highlighting a key stress-test experiment, which we also discussed with Reviewer 5e7B. To simulate severe real-world scenarios beyond simple MCTS estimation noise, we injected artificial noise by completely randomizing the MCTS preference list with a 10% probability at each decision step. This setup mimics catastrophic data corruption or severe label noise found in real-world datasets.

The crucial finding was that NaDRO's performance did not collapse under this extreme condition, remaining stable and competitive. This provides strong empirical evidence that our framework's robustness extends beyond well-behaved estimation noise.

This resilience stems from NaDRO's core principles. CPR learns a robust state representation by focusing on key qualitative features, making the policy less susceptible to noisy inputs. POR’s 'ensembling' approach trusts the consensus of top-ranked actions, inherently filtering out outlier noise, including the severe ranking errors we introduced. The synergy between these two components is what allows NaDRO to handle such a challenging noise profile, justifying its promise as a pathway for real-world applications.

Q5: The paper adopts fixed thresholds (1.15 and 0.8) for labeling high and low cost in CPR annotation, but it’s unclear how these specific values were determined.

Thank you for this excellent question regarding the specific values used in our CPR annotation. We agree that the choice of any fixed threshold warrants justification. Our response is twofold: first, we will explain the design philosophy behind these thresholds, and second, we will provide empirical evidence of their robustness.

1. Design Philosophy: Correcting Uncalibrated LLM Perception

The core goal of CPR is to instill human-like qualitative judgment into the model. A challenge with any qualitative label (e.g., "large" vs. "small") is that its boundaries are inherently fuzzy. For example, we can definitively say a 10-node TSP is "small" and a 10,000-node TSP is "large," but the exact dividing line is ambiguous.

Critically, we observed that LLMs often have a skewed or uncalibrated perception of scale, sometimes viewing a 50-node problem as an immense task. The CPR thresholds are not intended to represent a universal "truth," but rather to provide a clear, corrective signal to anchor the model's flawed intuition. By defining clear zones for "High Cost" (>1.15x greedy) and "Low Cost" (<0.8x greedy), we create an unambiguous "Normal" zone in between. This three-zone approach helps the model build a more reasonable, human-aligned understanding of its performance state.

2. Empirical Robustness of Thresholds

To ensure our method is not sensitive to these specific values, we conducted experiments with different threshold pairs as (2.0, 0.8) and (1.15, 0.5). Our key finding was that there was no significant difference in the final performance or the training dynamics when using these alternative thresholds. This demonstrates that the model is robust to the specific choice of these values within a reasonable range. The crucial factor for the model's learning is the existence of these distinct qualitative regions, not the precise numerical boundaries, which validates our design choice.

Q6: What is the advantage of LLM-based methods in such tasks and when would such method be preferable to strong classical solvers?

Thank you for this crucial question. You are correct that our method does not outperform a highly specialized solver like LKH. However, we would like to clarify that the primary goal of our research is not to create a new state-of-the-art solver for a mature and well-defined problem like TSP.

Instead, we use TSP as a representative benchmark to explore a much broader, forward-looking question: how can Large Language Models be leveraged as general-purpose agents for a wide array of complex sequential decision-making tasks? We believe this is an inevitable and important future direction for AI.

The key distinction and practical relevance lie in the underlying paradigms:

• Classical Solvers (e.g., LKH): These are highly optimized, specialized algorithms designed for a single domain. They are powerful but inflexible and cannot be easily adapted to other problems.
• Our LLM-based Framework: We propose a general framework that can be easily adapted to various decision-making tasks. The LLM learns a high-level, adaptive strategy. As demonstrated in our paper, the same NaDRO framework excels on both TSP and the more complex CVRP, showcasing its versatility.

Our work should be viewed not as an attempt to beat a specific classical solver on its home turf, but as a crucial step towards a new paradigm of general, adaptable AI decision-makers capable of tackling a diverse portfolio of complex challenges.

We thank the reviewer for their positive evaluation and constructive comments. We have performed the requested supplementary analyses and will incorporate them into the final version.

Q1: The primary weakness of the paper is that the proposed approach is highly heuristic.

Thank you for the insightful feedback regarding the need for deeper justification for the proposed reward design. This is a crucial point, and this response will address the core of the issue. More detailed analyses of each component are provided in other sections of our rebuttal.

1. On the Principled Design of POR

In response to the reviewer's point about the method's contribution, it is important to clarify that POR is not a heuristic but a principled approach to ensure stable training in noisy environments.

From a policy gradient perspective, as detailed elsewhere in our rebuttal, POR is designed to produce a low-bias, low-variance gradient estimator. This is achieved by rewarding a stable region of top actions ("ensembling") rather than a volatile single best action. This structure mathematically guarantees a more reliable convergence process compared to naive reward schemes.

2. Analysis of Reward Design, Properties, and Limitations

Regarding the request for a more explicit analysis of the rewards' design, properties, and limitations, we offer the following:

• Ablation on Reward Design: Ablation studies on key design parameters have been performed. For instance, testing POR's cutoff threshold k (from 10% to 70%) revealed the performance to be highly robust. Furthermore, a stress test was conducted by injecting severe artificial noise (completely randomizing the preference list 10% of the time), which showed that performance did not collapse. This empirically validates the robustness of the chosen design.

• Analysis of Properties and Limitations:

  • POR: A key property of POR is its ability to filter noise and reduce gradient variance. Its limitation, however, is that its effectiveness relies on the MCTS rankings being meaningful, even if noisy. In scenarios with an extremely poor MCTS rollout policy, the quality of the relative rankings themselves could degrade, limiting POR's efficacy.
  • CPR: A core property of CPR is providing a dense, process-based reward that stabilizes learning, a point elaborated upon in our other responses. Its primary limitation is that the qualitative features are currently hand-engineered and require rule-based annotation. This reliance on manual design restricts its out-of-the-box applicability to entirely new problem domains and points to a clear direction for future work in automatic feature discovery.

Q2:Most baseline comparisons are made against publicly available LLMs (e.g., GPT-4o, DeepSeek) without task-specific fine-tuning on TSP or CVRP. "

Thank you for this critical feedback. You are correct that a full comparison including the Llama-8B baselines is essential for a rigorous evaluation and to properly isolate the contributions of our method. We apologize for this omission in the initial submission.

To address this, we have now run these experiments and included the results for Llama-8B (Base) and GRPO on Llama 8B in our main results table. You can see our response to Reviewer 4Ria. This new data allows us to draw two clear conclusions, directly addressing your points:

1. Benefit of Standard Fine-Tuning: Comparing GRPO on Llama 8B to the Llama 8B (Base) model shows that applying an established fine-tuning method (GRPO) does provide a noticeable, though often modest, performance improvement. For instance, on instance gr202, the optimality gap improves from 13.43% to 11.33%. This confirms the benefit of task-specific fine-tuning.

2. Isolating the Effect of NaDRO: The most critical comparison is between NaDRO-Llama and GRPO on Llama 8B. Here, the results are stark. On gr202, our method reduces the gap from 11.33% down to just 2.41%. Similarly, on pr152, the gap is more than halved from 0.89% to 0.37%. This demonstrates unequivocally that the vast majority of the performance gain is attributable to our novel NaDRO reward design, not just the fine-tuning process itself.

Q3: Most critically, the paper lacks a proper ablation study isolating the effects of the two proposed reward components.

Thank you for this critical feedback. We agree that a rigorous ablation study isolating the effects of POR and CPR is fundamental to our contribution, and we would like to respectfully clarify that this analysis was indeed a core part of our original submission.

1. Existing Ablation Study in the Original Manuscript

In Section 5.3, Table 3 of our original paper, we presented an ablation study that explicitly decomposes the contribution of each component to the final task performance.

For example, at a TTS level of 0, the optimality gap improves from 11.57% (GRPO) to 6.74% (GRPO+POR), and finally to 3.17% (NaDRO), quantifying the meaningful and distinct contribution of each component.

2. Deeper Analysis of Learning Dynamics: Reward Variance and Reward Hacking

To further deepen this analysis, we have conducted a new investigation into the training process itself, focusing on reward variance and loss curves. This reveals how our components lead to superior performance by ensuring a more stable and meaningful learning process.

Table 6: Analysis of Training Loss Curves

Table 7: Reward Variance During Training

Key Findings from Training Dynamics:

• NaDRO Reduces Reward Variance: The data confirms that both POR and the full NaDRO framework significantly reduce reward variance compared to the GRPO baseline. This creates a more stable learning signal and a more reliable policy gradient, which is crucial for effective convergence.

• NaDRO Prevents Reward Hacking: The loss curves offer a critical insight. The standard GRPO model's loss quickly drops to a near-zero value. While seemingly positive, this, combined with its poor final performance, indicates a classic case of reward hacking. The model has found a "lazy" loophole to minimize loss without actually learning a robust policy. In contrast, the NaDRO model maintains a healthier, actively fluctuating loss, which, paired with its high final reward, proves it remains meaningfully engaged in the learning process, avoiding deceptive, low-effort solutions.

Q4: Throughout the paper, the citation format is inaccurate. The authors should use \citep{} when the citation is not a part of the sentence.

Thank you for pointing out the inconsistency in our citation format. We will carefully review the entire manuscript and correct all citations to ensure they fully comply with the NeurIPS style guidelines in the camera-ready version.

We thank the reviewer for their positive evaluation and constructive comments. We have performed the requested supplementary analyses and will incorporate them into the final version.

Q1: How does CPR contribute specifically to solving long-horizon credit assignment, beyond acting as an auxiliary classification loss? While CPR improves performance, it’s unclear whether its benefits stem from better credit assignment over multi-step trajectories, or simply from regularizing the model with intermediate supervision. Could one isolate its impact by measuring reward variance or learning dynamics with/without CPR over varying trajectory lengths?

Thank you for this insightful question. CPR's primary contribution is providing a dense, immediate, and outcome-independent reward for correct qualitative state assessment. This stabilizes the learning signal, building a reliable cognitive foundation for the policy, which is crucial for long-horizon tasks where final rewards are sparse and noisy. We demonstrate this with two key pieces of evidence.

1. Learning Dynamics: CPR successfully guides the model to learn correct state perception. As shown in Table 1, baseline methods (raw_GRPO, GRPO+por) fail to achieve this, registering low or negative "Correctness Rewards." In contrast, our full method (ours) shows a sustained and significant increase in this reward, confirming that CPR provides a meaningful, process-based learning signal that was previously absent.

   Table 1: Correctness Reward Dynamics during Training

   Method	5	105	205	305	405	505	605	705	805	905	1005	1105	1205	1305	1405	1505	1605	1705	1805	1905	2005	2105	2205	2305	2405	2505	2605	2705	2805	2905
   GRPO	-0.0861	-0.0703	-0.0572	-0.0571	-0.0530	-0.0328	-0.0179	-0.0266	-0.0420	-0.0672	-0.0605	-0.0350	0.0067	0.0028	0.0087	0.0099	0.0293	0.0132	0.0237	0.0485	0.0443	0.0352	0.0281	0.0152	0.0297	0.0510	0.0833	0.0791	0.0724	0.0462
   GRPO+POR	-0.0762	-0.0245	-0.0180	-0.0207	-0.0133	-0.0063	-0.0063	-0.0052	-0.0132	-0.0446	-0.0389	-0.0043	0.0291	0.0241	0.0309	0.0360	0.0383	0.0354	0.0472	0.0548	0.0455	0.0397	0.0426	0.0395	0.0555	0.0768	0.1088	0.1126	0.0941	0.0833
   NaDRO(ours)	-0.0286	0.0751	0.0593	0.0744	0.0939	0.1155	0.0991	0.0847	0.0988	0.1085	0.1065	0.1128	0.1385	0.1232	0.1128	0.1166	0.1342	0.1428	0.1543	0.1318	0.1205	0.1206	0.1485	0.1597	0.1382	0.1465	0.1475	0.1642	0.1509	0.1309

2. Ablation Study: Our ablation study (Table 2) quantifies CPR's critical impact on final task performance. When evaluating pure policy performance (TTS Level=0), adding CPR to the GRPO + POR model causes the optimality gap to drop from 6.74% to 3.17%. This demonstrates that CPR is a decisive component driving the final SOTA performance.

Table 2: Ablation Study on NaDRO Components (Optimality Gap % on pr152)

Q2: How sensitive is POR to the rank cutoff parameters and MCTS quality? Since POR transforms noisy Q-values into ordinal rankings and selects top-k actions for positive supervision, could a poorly chosen cutoff degrade performance by discarding near-optimal actions that fall just below the threshold?

We sincerely appreciate the reviewer's insightful question. To thoroughly investigate the robustness of our Preference-based Outcome Reward (POR) framework, we have conducted a new series of experiments. The results are presented below. Please note that due to submission constraints not allowing for image files, this table presents sampled data points from performance curves. The complete graphs will be included in the final camera-ready version.

We tested various cutoff thresholds (k). It is important to clarify the settings: ours_k0.5 uses a standard 50% cutoff. The ours_random setting also uses a 50% cutoff but introduces severe noise: there is a 10% probability at each step of the entire preference list being completely shuffled.

Table 3: Empirical Performance with Varying $k$ and Injected Noise (Correctness Reward)

Key Findings:

1. High Robustness across k Values: The framework demonstrates strong robustness, as all k settings yield competitive performance without any catastrophic failure. Notably, the k=0.7 setting shows slightly superior performance across many stages.

2. Exceptional Resilience to Severe Ranking Noise: The ours_random result is a powerful testament to POR's resilience. Even when faced with a 10% chance of the ranking signal becoming completely random, the model's performance does not collapse and remains on par with other stable settings. This directly proves that the framework can withstand not just minor misrankings but also occasional catastrophic failures in the preference signal, strongly validating its suitability for learning from highly noisy data.

These findings confirm that POR is not overly sensitive to its parameters and is exceptionally robust to noise. We will incorporate this analysis into the final manuscript.

We thank the reviewer again for the insightful question regarding the sensitivity of the cutoff parameter k. Your question prompted us to conduct a deeper theoretical analysis to understand why k=0.7 performed optimally in our empirical study (Table 3). We are pleased to report that this new analysis provides a strong theoretical justification for our empirical findings, supported by the calculations detailed below.

1. Theoretical Objective: Maximizing Value Separability

Our primary goal is to find an optimal cutoff $k$ that partitions the $N_A$ candidate actions into the most informative positive ($\delta^+$) and negative ($\delta^-$) sets for policy learning. The ideal partition should maximize the "value separability" between these two sets. We formalize this by defining a utility function $J(k)$ that measures the expected gap between the average cost of the negative set and the positive set. Our objective is to find $k^*$ that maximizes this function:

$k^* = \arg\max_k J(k) \quad \text{where} \quad J(k) = E[\mathrm{avg\_cost}(\delta^-) - \mathrm{avg\_cost}(\delta^+)]$

This objective function seeks the clearest possible boundary between high-quality and low-quality actions, providing the most effective signal for training.

2. Modeling Relative Action Costs from Noisy Data

Since the true action costs $Q^*$ are unknown, we model them based on the MCTS evaluation data. We use the average cost increase between adjacent ranks, $\Delta C_i = E_s[Q_{(i+1)}(s) - Q_{(i)}(s)]$, as a proxy for the performance drop at each rank transition.

We normalize these cost increases across all ranks to get a "contribution ratio" $\bar{p}_i$ for each transition $i \to i+1$. This allows us to construct a relative cost model. The cost of the action at rank $i$, denoted as $C_i$, is approximated by the cumulative sum of all preceding cost-increase contributions, as defined below:

$C_i \approx \sum_{j=1}^{i-1} \bar{p}_j$

The contribution ratios, calculated from our training dataset, are shown in Table 8.

Table 8: Average Contribution Ratio ($\bar{p}_i$) of Rank Transitions to Total Cost Increase

Table 8 provides a crucial intuition: the contribution ratios ($\bar{p}_i$) are highly non-uniform. The cost increases between the top-ranked actions are relatively small. However, the performance drop becomes dramatic at the tail end of the ranking. Notably, the cost increase from the second-to-last action to the absolute worst action (rank 14 to 15) alone accounts for over 33% of the total performance gap between the best and worst choices.

This signifies that in our problem, the penalty for selecting the worst actions far outweighs the benefit of distinguishing between the top few optimal ones. Therefore, it is more critical to define a large and robust positive set ($\delta^+$) that safely excludes these catastrophic tail-end choices. This directly explains why a relatively high cutoff like 70% achieves the best practical results in our experiments—it prioritizes the vital task of avoiding the worst outcomes.

3. Theoretical Derivation of the Optimal k

By applying our relative cost model to the objective function $J(k)$, we can now calculate the expected separability for every possible cutoff $k$. The results of this calculation (for a typical action space of $N_A=15$) are presented in Table 9.

Table 9: Calculating Value Separability J(k) to Find Optimal Cutoff k

As highlighted in the table, the utility function $J(k)$ reaches its maximum value of 0.5284 at k=11. This corresponds to an optimal positive set ratio of $k/N_A = 11/15 \approx 73.3\%$.

4. Unification of Theory and Experiment

This theoretical result, which suggests an optimal cutoff of approximately 73.3%, provides a strong justification for our empirical finding in Table 3, where the model achieved its best performance with a cutoff of $k=0.7$ (70%). The intuition from our data, and our experimental results validates our parameter choice and demonstrates that our POR framework is not only empirically robust but also grounded in sound theoretical principles.

We thank the reviewer for their positive evaluation and constructive comments. We have performed the requested supplementary analyses and will incorporate them into the final version.

Q1: Theoretical Clarifications: Provide error bounds for POR under noisy distributions or prove CPR’s impact on policy gradient optimization.

Due to space constraints, the impact of CPR on policy gradient optimization has been detailed in our response to Reviewer 5e7B. Here, we will focus on the theoretical analysis of how POR provides robustness and reduces gradient variance under noisy distributions, directly addressing the core of your question.

Theoretical Analysis of POR's Lower Gradient Variance under Noise

We provide a theoretical analysis demonstrating that Preference-based Outcome Reward (POR) inherently reduces policy gradient variance compared to a standard Top-1 reward scheme, ensuring a more robust optimization process under noisy distributions.

• Mathematical Setup: Let's simplify POR to a binary model: actions in the top-k (as ranked by noisy MCTS) receive a reward of 1, and others receive 0. We compare this to a Top-1 scheme where only the single best action gets a reward. The core randomness stems from MCTS noise (ϵ), which makes the rank of any action a random variable. The variance of a Bernoulli reward (taking values in {0, 1}) is given by Var(R) = p(1-p), where p is the probability of receiving the reward.

• Key Derivation: For any high-quality action a*, the probability of it ranking in the top-k (p_k = P(rank(a*, ϵ) ≤ k)) is fundamentally greater than or equal to the probability of it being exactly rank 1 (p_1 = P(rank(a*, ϵ) = 1)). Due to noise and competition, p_1 may be significantly less than 1 (e.g., 0.6), whereas p_k remains very high (e.g., >0.95). Since the variance function f(p) = p(1-p) is monotonically decreasing on the interval [0.5, 1], and we have 0.5 ≤ p_1 < p_k ≈ 1, it directly follows that: $Var(R_{POR}) < Var(R_{Top-1})$

Q2: Enhanced Experimental Rigor: Report standard deviations for multi-run experiments. Test robustness in additional noise scenarios.

We thank the reviewer for these constructive suggestions to enhance the paper's experimental rigor. We have addressed each point as detailed below.

1. Reporting Standard Deviations for Multi-Run Experiments

To improve statistical rigor, we have conducted multiple runs for all experiments and updated our main results table to include both the mean and standard deviation. The revised table is presented below. The results confirm that NaDRO's superior performance is statistically robust. Our method not only achieves a lower optimality gap on average but also exhibits relatively low variance, indicating consistent and reliable performance across runs.

Table 4: Comparative optimality gap (%) on TSP instances, updated with Mean ± Standard Deviation

2. Robustness in Additional Noise Scenarios

To further validate robustness, we conducted a stress test with artificially injected noise. In this experiment, we introduced a 10% probability at each decision step of completely randomizing the MCTS preference list. This simulates a catastrophic failure in the ranking signal.

Our findings show that even under this severe noise condition, NaDRO's performance does not collapse and remains competitive and stable. This powerful result demonstrates the framework's exceptional resilience to extreme data corruption, going beyond the inherent noise of MCTS.

Q3: Extended TTS Analysis: Supplement experimental results for TTS level > 10 to clarify performance trends.

Thank you for the insightful questions regarding the performance trends at higher Test-Time Search (TTS) levels and the dynamics of the performance gaps in our ablation study. To clarify these points, we have extended our analysis up to a TTS level of 30. The supplemented results are presented in the table 5 below.

Table 5: Extended Ablation Study on NaDRO Components and TTS Level (Optimality Gap % on pr152)

Clarification on Performance Trends and Gaps:

1. General Trend: As expected, increasing the TTS budget improves performance across all models. This confirms that a larger search space at inference time is beneficial.

2. Addressing the Narrowing Gap: You correctly observed that the performance gap between GRPO + POR and the GRPO Baseline narrows at some levels. Our extended results clarify this trend. While extensive search can help weaker models compensate for their policy deficits, the GRPO + POR model consistently outperforms the GRPO Baseline at every TTS level, confirming the value of the POR component.

3. NaDRO's Superior Scalability with TTS: Most importantly, the extended analysis reveals that our full NaDRO model benefits most from an increased TTS budget. The performance gap between NaDRO and all other baselines remains significant and even widens in terms of relative improvement at higher search levels. At TTS=30, NaDRO achieves a near-optimal gap of just 0.04%, demonstrating a powerful synergy between a robust, well-trained policy and test-time computation. A better policy makes the search process far more effective.

Q3: Insufficient Baseline Comparisons: Missing comparisons with state-of-the-art noise-robust methods (e.g., Robust RLHF by Shen et al. 2024), limiting the evaluation to basic GRPO.

Thank you for your valuable feedback. We have carefully analyzed the work by Shen et al. (2024) and believe our NaDRO framework differs fundamentally in its problem setting, noise source, and core methodology. "Robust RLHF" addresses an inaccurate Reward Model (RM) in general LLM alignment, which stems from errors in human preference labeling. In contrast, NaDRO focuses on specialized sequential decision-making tasks, dealing with noise inherent in the training data itself, caused by inaccurate MCTS estimations.

Crucially, a direct experimental comparison is not feasible. The "Robust RLHF" paper states that the code will only be released "upon acceptance." Given its status as a withdrawn paper from ICLR 2025, an official public implementation is unavailable.

Dear Reviewer,

I hope this message finds you well.

As the discussion period is nearing its end with less than one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily.

If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,