Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling

Dear Reviewer T3Po, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

C1: On the Generality of the Evaluation

We appreciate your thoughtful feedback regarding the generality and robustness of DORA. We fully agree that evaluating across more diverse PRMs, policy models, and benchmarks is crucial to substantiating the claim of DORA’s effectiveness as a parallel search method. To address this, we have conducted a new series of experiments beyond the initial setup, expanding the evaluation in three directions (All results are averaged over 5 independent runs to ensure statistical stability):

(1) Broader Benchmark Coverage: We incorporated four additional standard math reasoning benchmarks: HMMT24, HMMT25, AMC23, and AMC24. Under a rollout budget of N=64, the results are summarized as follows:

(2) Larger Policy Models and PRM: We evaluated DORA with more powerful models by using Qwen2.5-Math-PRM-72B as the PRM to guide the LLaMA-3.1-8B-Instruct policy model on the MATH500 benchmark with N=64. The performance is reported below:

(3) Different PRM Families: To evaluate generalization across PRM families, we additionally tested Skywork-PRM-7B on the MATH500 benchmark with N=64. The results are shown below:

Above experiments demonstrate that DORA generalizes well across a broader range of benchmarks, scales to larger policy models and PRMs, and is compatible with PRMs from different families. Specifically, the experiments with Qwen2.5-Math-PRM-72B were executed in parallel on a cluster with 100 H20 GPUs (96GB each), using 2 GPUs per run, and were completed in approximately 4 days. We will include these results and analysis in the later version.

C2 & Q2: About computational overhead of embedding calculations with a large number of candidates

Thank you for bringing up this important point. We agree that analyzing the computational overhead of embedding is essential, especially when scaling DORA to larger rollout budgets. According to [1-2], the total FLOPs incurred during Test-Time Scaling (TTS) can be calculated as:

$\text{Total FLOPs} = 2N \cdot T_{\text{sol}} \cdot \left( P_{\text{policy}} + P_{\text{PRM}} + P_{\text{emb}} \right)$

where:

• $N$: rollout budget (i.e., number of sampled solutions),
• $T_{\text{sol}}$: average number of tokens per solution,
• $P_{\text{policy}}, P_{\text{PRM}}, P_{\text{emb}}$: parameter sizes of the policy model, PRM, and embedding model, respectively.

The relative overhead of the embedding model in the total FLOPs is thus:

$Ratio = \frac{P_{\text{emb}}}{P_{\text{policy}} + P_{\text{PRM}} + P_{\text{emb}}}$

This fraction is constant and does not grow with $N$, implying that the embedding overhead scales linearly just like the other components and remains proportionally small.

In practice, Parameter of embedding models used in our setup (e.g., BGE-M3 <600M) are much smaller than that of the policy model (e.g., Qwen2.5-1.5B-Instruct) and the PRM (e.g., Qwen2.5-Math-PRM-7B).The additional computational burden introduced by the embedding model would not be a major concern as the rollout budget $N$ increases. We will include this analysis in the revised version.

C3 & Q1: On the Sensitivity to Temperature Hyperparameters

We appreciate your concern regarding DORA’s reliance on temperature-based mechanisms and the associated sensitivity to hyperparameter choices. To address this, we conducted a detailed ablation study on two key temperature parameters:

(1) The allocation temperature $T_b$, which controls the sharpness of the softmax when allocating rollout budgets;

(2) The semantic similarity temperature $T_s$, which determines how close two solutions should be to be considered as belonging to the same reasoning direction.

For $T_b$, we default to 0.1 to align with REBASE, ensuring consistency and fairness. For $T_s$, we selected the final value via empirical study. We acknowledge that these results were not included in the original submission and apologize for the oversight.

We now report the full sensitivity analysis on the MATH500 benchmark with rollout budget N=64, using LLaMA-3.1-8B-Instruct as the policy model. Each configuration was run 5 times with different seeds, and we report the averaged accuracy:

These results suggest that DORA is relatively robust to the choice of $T_s$, with accuracy remaining stable across a wide range. In contrast, both DORA and REBASE show more noticeable degradation when $T_b=1.0$, as the softmax becomes too flat to leverage PRM guidance effectively. In the extreme case where $T_b \to \infty$, the model degenerates into temperature sampling. We will include these results and analysis in the updated version to provide clearer guidance for future applications.

[1] Scaling Laws for Neural Language Models, OpenAI.

[2] Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning, ICLR2025.

Dear Reviewer T3Po,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly follow up to ensure that our response has addressed your concerns satisfactorily. If there are any additional questions or suggestions you would like us to consider, please do not hesitate to let us know. Your insights are invaluable to us, and we would be more than happy to engage in further discussion to improve our work.

Thank you once again for your time and thoughtful feedback！

Best regards, The Authors

Dear Reviewer T3Po,

We are writing to you with a gentle follow-up as the discussion period will close in about 30 hours. We completely understand that your time is precious and you are likely very busy. We just wanted to kindly reiterate that if you have any final comments or concerns, we are fully prepared to do whatever is necessary to address them promptly.

Thank you once again for your time and invaluable contribution to our work.

Best regards,

Authors

Dear Reviewer JSmU, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

Q1: On the Sensitivity to the Embedding Model and Semantic Similarity Temperature $T_s$

Thank you for this important question. We fully understand your concern regarding the robustness of DORA’s semantic grouping, which relies on the embedding model and the soft clustering temperature $T_s$.

To evaluate the impact of different retriever (embedding) models, we compared our original setup (BGE-M3, 568M parameters) with two widely used alternatives from the MTEB leaderboard [1] that support long inputs (≥2048 tokens): E5-base-4k (110M) and GTE-multilingual-base (305M). All experiments were conducted on MATH500 with $N=64$, and all other settings were kept identical to those in the main experiments. Results were averaged over 5 independent runs:

The results show that GTE and BGE yield comparable performance, indicating that DORA is robust to retriever changes within similar capacity ranges. However, E5 exhibits certain performance degradation, which we hypothesize is due to its smaller model size and thus weaker clustering capability in high-dimensional embedding space.

We further conducted an ablation study to evaluate the sensitivity of DORA to the semantic similarity temperature $T_s$, which determines how close two solutions should be to be considered as belonging to the same reasoning direction. Experiments were run on MATH500 with $N=64$, using LLaMA-3.1-8B-Instruct as the policy model. The results (averaged over 5 runs) are shown below:

These findings demonstrate that DORA is relatively robust to the choice of $T_s$. We will include these results and corresponding discussion in the updated version to better clarify DORA’s stability with respect to embedding models and soft clustering settings.

Q2: On the Scalability of Embedding and Affinity Computation with Very Large Candidate Sets

Thank you for raising this important point. We agree that it is essential to assess the overhead introduced by embedding and affinity matrix computation, particularly at large rollout scales.

Following [2–3], the total FLOPs of Test-Time Scaling (TTS) can be estimated as:

$\text{Total FLOPs} = 2N \cdot T_{\text{sol}} \cdot (P_{\text{policy}} + P_{\text{PRM}} + P_{\text{emb}})$

where $N$ is the rollout budget, $T_{\text{sol}}$ is the average solution length, and $P_{\text{policy}}, P_{\text{PRM}}, P_{\text{emb}}$ denote parameter counts of the policy model, PRM, and embedding model respectively.

The embedding overhead ratio in TTS is caculated by:

$\text{Ratio} = \frac{P_{\text{emb}}}{P_{\text{policy}} + P_{\text{PRM}} + P_{\text{emb}}}$

This ratio is independent of $N$, indicating that embedding cost scales linearly with rollout size and remains proportionally small. In our setup, the embedding model (e.g., BGE-M3 <600M) is significantly smaller than the policy (e.g., Qwen2.5-1.5B) and PRM (e.g., Qwen2.5-Math-PRM-7B), so its contribution to total cost remains minor.

We also analyzed memory and runtime cost of computing the affinity matrix (i.e., pairwise cosine similarities). When $N = 10{,}000$ (a scale well beyond most prior TTS works, which typically use $N \leq 2048$), with 1024-dim embeddings in fp16:

• Matrix size: $10^8$ entries → 200 MB
• Embedding storage: 20 MB
• Total memory: ~220 MB, easily handled by modern CPUs

For runtime, computing 100M cosine similarities (each a 1024-dim dot product) totals ~200 GFLOPs. On a single CPU core this takes ~6–7 seconds, or <0.5s with 16-core parallelization. Since this is a one-time cost per rollout, it does not meaningfully affect overall latency.

We will include this analysis in the final version to clarify that DORA’s embedding-related steps scale efficiently even with large candidate sets.

Q3: On the Assumption of Shared PRM Scores Within a Direction

Thank you for this thoughtful question. The assumption in Theorem 1—that all solutions within a semantic direction are perfectly correlated and share the same PRM score—is introduced solely to enable a tractable theoretical analysis，which allows us to derive a clean optimality result.

In practice, solutions grouped within a direction are typically only partially correlated, and their PRM scores can differ due to modeling noise and semantic drift. Explicitly modeling this intra-directional variance would render the analysis intractable. To address this, DORA adopts a soft clustering mechanism and retains per-candidate PRM scores $R_i$, without collapsing them to a single cluster-level score.

Rollout allocation is computed individually for each solution using the formula (lines 195–196):

$w_i' = \frac{w_i \cdot \gamma_i}{\sum_j w_j \cdot \gamma_j},$

where $w_i$ reflects solution quality, and $\gamma_i$ captures semantic uniqueness from the affinity matrix. This ensures that higher-quality candidates within a direction can still receive more rollouts, even when grouped with lower-scoring neighbors. By combining soft clustering with individualized scoring, DORA remains robust to intra-directional PRM variance. Our empirical results across multiple benchmarks show that DORA consistently outperforms baselines, suggesting that its design effectively handles score noise while still promoting semantic diversity. We will include above discussion in the revised version.

Q4: On Clustering Ablation and Design Choice

Thank you for raising this insightful question. We fully agree that isolating the effect of clustering is essential for understanding DORA’s improvements.

In the early stages of our study, we explored several alternative clustering strategies before finalizing the soft clustering design:

• KMeans clustering requires predefining the number of clusters $K$. However, the actual number of distinct reasoning directions can vary across problems and reasoning steps, making fixed $K$ a poor approximation and often leading to suboptimal clustering performance.

• We also experimented with hierarchical clustering, which avoids setting $K$ by merging solutions based on a fixed cosine similarity threshold $C$. However, the optimal value of $C$ should ideally vary with the depth and complexity of reasoning, which differ significantly across problems. This sensitivity made hierarchical clustering even less effective than KMeans in our experiments.

• Based on these findings, we adopted a soft clustering approach in DORA, which avoids hard assignment boundaries and allows for more adaptive and robust semantic grouping across diverse reasoning structures.

We present below the ablation study comparing different clustering methods. All experiments were conducted on the MATH500 benchmark with rollout budget $N=64$ using the LLaMA-3.1-8B policy model (results averaged over 5 independent runs):

As shown, soft clustering consistently achieves the best performance, demonstrating its effectiveness over hard clustering alternatives. We will include this ablation and discussion in the revised version.

Q5: On Generalization Beyond Mathematical Reasoning

We appreciate the reviewer’s thoughtful question regarding DORA’s applicability beyond math reasoning. We fully agree that evaluating generalization across domains—such as code generation—is an important direction.

In our current study, the primary obstacle to extending DORA to other domains is the lack of strong, open-source step-level Process Reward Models (PRMs) outside the math domain. Most high-quality PRMs publicly available today are specifically trained for mathematical reasoning [4]. For the code generation domain, we had already conducted exploratory experiments early in our study using Skywork-PRM—the only open-source PRM we are aware of that provides step-level feedback for programming tasks. We applied it to standard benchmarks such as HumanEval and MBPP, and tested all step-level search methods (Beam Search, DVTS, REBASE, and DORA). However, none of these methods were able to significantly outperform the Temperature Sampling baseline.

We believe this outcome is due to the limited step-level guidance capacity of Skywork-PRM for code tasks. This limitation has also been observed in recent work [5], which shows that Skywork-PRM provides relatively weak supervision for multi-step code generation. While using prompted open-source LLMs as critic models is a potential workaround, this approach introduces significant computational overhead and latency at inference time—undermining the efficiency goals of Test-Time Scaling.

We will include this discussion in the revised version and acknowledge the limited domain coverage of current PRMs as a key limitation. Extending DORA to broader domains—such as code, science QA, or planning—remains an exciting and important direction that we leave for future work.

[1] MTEB: Massive Text Embedding Benchmark, EACL2023.

[2] Scaling Laws for Neural Language Models, OpenAI.

[3] Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning, ICLR2025.

[4] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning, ACL2025.

[5] CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation, ACL2025.

Dear Reviewer JSmU,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly follow up to ensure that our response has addressed your concerns satisfactorily. If there are any additional questions or suggestions you would like us to consider, please do not hesitate to let us know. Your insights are invaluable to us, and we would be more than happy to engage in further discussion to improve our work.

Thank you once again for your time and thoughtful feedback！

Best regards, The Authors

Dear Reviewer JSmU,

We are writing to you with a gentle follow-up as the discussion period will close in about 30 hours. We completely understand that your time is precious and you are likely very busy. We just wanted to kindly reiterate that if you have any final comments or concerns, we are fully prepared to do whatever is necessary to address them promptly.

Thank you once again for your time and invaluable contribution to our work!

Best regards,

Authors

Dear Reviewer N3ut, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

W1: Regarding the Generalization of the Presented Results

We understand your concern and agree with your suggestion that conducting experiments on larger policy and PRM models, and across different PRM families would better validate the generalizability of DORA. Regarding the AIME tasks, we note that the observed overlap with baseline performance is primarily due to the small dataset sizes of AIME24 and AIME25 (only 30 problems each), which makes the Accuracy metric more prone to high variance and less statistically stable. To enhance the statistical reliability of the results, we followed [1] and conducted 10 independent runs with different random seeds, reporting the averaged performance (see lines 214–215). To further demonstrate the generality and scalability of DORA, we conducted the following three sets of additional experiments (All results are averaged over 5 independent runs to ensure statistical stability):

(1) Broader Benchmark Coverage: We incorporated four additional standard math reasoning benchmarks: HMMT24, HMMT25, AMC23, and AMC24. Under a rollout budget of N=64, the results are summarized as follows:

(2) Larger Policy Models and PRM: We evaluated DORA with more powerful models by using Qwen2.5-Math-PRM-72B as the PRM to guide the LLaMA-3.1-8B-Instruct policy model on the MATH500 benchmark with N=64. The performance is reported below:

(3) Different PRM Families: To evaluate generalization across PRM families, we additionally tested Skywork-PRM-7B on the MATH500 benchmark with N=64. The results are shown below:

Above experiments demonstrate that DORA generalizes well across a broader range of benchmarks, scales to larger policy models and PRMs, and is compatible with PRMs from different families. Specifically, the experiments with Qwen2.5-Math-PRM-72B were executed in parallel on a cluster with 100 H20 GPUs (96GB each), using 2 GPUs per run, and were completed in approximately 4 days. We sincerely thank you for this insightful suggestion and will include the results and analysis in the revised version.

W2 & Q3: On the Design Choice of Hyperparameters $T_s$ and $T_b$

We fully understand your concern regarding the motivation and sensitivity of certain hyperparameters. For the quality score temperature $T_b$, we default to $T_b=0.1$ to align with REBASE, ensuring a fair and consistent comparison. For the semantic similarity temperature $T_s$, which determines how close two solutions should be to be considered as belonging to the same reasoning direction, we conducted an ablation study to select a suitable value. We apologize for the oversight in not including this ablation result in the original submission. We performed a detailed sensitivity analysis of both $T_s$ and $T_b$ on the MATH500 benchmark with rollout budget N=64, using LLaMA-3.1-8B-Instruct as the policy model. All results are averaged over 5 independent runs to ensure statistical stability. The results are summarized as follows:

These results show that DORA is relatively robust to the choice of $T_s$. As for $T_b$, both REBASE and DORA achieve strong performance when $Tb=0.01$ or $0.1$, but exhibit a clear performance drop when $T_b=1.0$. This degradation occurs because a high temperature in the softmax weakens the guidance of the PRM. In the extreme case where $T_b \to \infty$, REBASE degenerates into temperature sampling without PRM guidance. We will include these results and discussion in the later version.

W3: About Potential Bottlenecks for Test-Time Scaling

We understand and appreciate your concern regarding the additional model dependency introduced by Test-Time Scaling (TTS), particularly the reliance on an embedding model for retrieval. We acknowledge that if the embedding model is trained on data with limited coverage of a target domain, its ability to cluster reasoning chains in that domain could indeed be suboptimal. This is a valid concern when considering the broader applicability of TTS.

However, recent advances in embedding model development have increasingly moved toward general text embeddings, as demonstrated in [2]. Modern retrievers such as GTE-Qwen and E5-Mistral are designed to inherit rich linguistic and domain knowledge from large pre-trained language models (e.g., Qwen2-1.5B), which cover a wide range of reasoning and generation scenarios. Based on this trend, we believe that the inclusion of an embedding model in TTS is unlikely to become a bottleneck for extending TTS to other domains, as many embedding models today possess sufficient generalization capabilities to support effective clustering across diverse tasks.

Respectfully, in our view, the more significant limitation to extending TTS beyond math currently lies in the lack of high-quality, open-source PRMs for domains such as code generation. Most existing PRMs are tailored for math reasoning [3], while open-source PRMs for code tasks remain scarce. To the best of our knowledge, Skywork-PRM is among the few publicly available options, but recent work [4] has shown that it offers relatively weak step-level supervision on code-related tasks. While one alternative is to use open-source language models prompted as critic models, this introduces substantial computational overhead and time delay at test time, which undermines the efficiency advantages of TTS.

W4: About Including Error Bars and Statistics for AIME on Table 1

We sincerely thank the reviewer for this helpful suggestion. The completed results are shown below, which will be incorporated into the later version:

Q1: About sensitive of different retrievers

We understand your concern about the potential sensitivity of DORA to different retriever models. To further investigate DORA’s robustness across retriever families, we compared our original retriever (bge-m3, 568M parameters) with two popular alternatives from the MTEB leaderboard [5] that support long inputs (≥2048 tokens): e5-base-4k (110M) and gte-multilingual-base (305M).All experiments were conducted on MATH500 with N=64, and all other settings were kept consistent with the main experiments. All results are averaged over 5 independent runs to ensure statistical stability. The results are summarized below:

The results show that GTE and BGE yield comparable performance, indicating that DORA is robust to retriever changes within similar capacity ranges. However, E5 exhibits certain performance degradation, which we hypothesize is due to its smaller model size and thus weaker clustering capability in high-dimensional embedding space. We will include these results and analysis in the revised version to clarify DORA’s sensitivity to retriever choices.

Q2: About chat template used for Llama models

We confirm that the chat template used for LLaMA models in our experiments is exactly aligned with the configuration described in [6] (see Appendix D.2). In fact, our codebase is directly built upon the official open-source implementation provided by [6].

[1] A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.

[2] Towards General Text Embeddings with Multi-Stage Contrastive Learning.

[3] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning, ACL2025.

[4] CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation, ACL2025.

[5] MTEB: Massive Text Embedding Benchmark, EACL2023.

[6] blogpost-scaling-test-time-compute, HuggingFace.

Dear Reviewer asTA, we sincerely appreciate you taking the time to read our manuscript and provide valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

Q1 & L1: On Exploring the Limits of Optimality

(i) Robustness to Intra-Directional PRM Score Variance

Thank you for this thoughtful question. The assumption in Theorem 1—that all solutions in a direction are perfectly correlated with identical PRM scores—is made purely to simplify the theoretical analysis.

In practice, solutions within a direction are usually partially correlated, and PRM scores can vary. Modeling this directly would make the theoretical formulation intractable. To handle this, DORA adopts a soft clustering design and retains per-candidate PRM scores $R_i$, without collapsing them at the direction level.

Rollout allocation is computed as (line 195-196):

$w_i' = \frac{w_i \cdot \gamma_i}{\sum_j w_j \cdot \gamma_j},$

where $w_i$ reflects quality, and $\gamma_i$ accounts for semantic uniqueness. This enables DORA to prioritize better solutions even within noisy directions. Thus, while the theoretical assumption simplifies analysis, DORA’s implementation is robust to score variance and operates fully at the individual solution level. We will clarify this distinction in the revised version.

(ii) DORA’s Behavior in the High-Confidence Regime

Thank you for the insightful question. In the theoretical high-confidence case—where the PRM is perfectly reliable ($\kappa \to \infty$)—its scores exactly reflect the true success probabilities. For example, a PRM score of 0.9 means that sampling from that solution yields success with 90% probability. In this limit, the optimal strategy is indeed to allocate all rollouts to the top-scoring candidate.

However, this regime is unrealistic in practice: computing true posterior success rates would require resampling each solution many times, which introduces significant cost and latency—contrary to the goal of test-time efficiency. Instead, PRMs in practice are noisy estimators trained on limited data, and this noise precludes direct exploitation.

In such realistic settings, REBASE is optimal when assuming candidate independence, and DORA extends this by accounting for semantic dependencies, leading to more balanced allocations. Since DORA’s design assumes PRM noise and redundancy among solutions, it is inherently incompatible with the $\kappa \to \infty$ case, and thus does not recover greedy behavior in that limit.

We will clarify this in the final version. Thank you again for encouraging this discussion.

Q2 & L2: On the Robustness to hyperparameter $T_s$ and Embedding Model Choices

Thank you for this important question. We fully understand your concern regarding the robustness of DORA’s semantic grouping, which relies on the soft clustering temperature $T_s$ and the embedding model.

We first conducted an ablation study to evaluate the sensitivity of DORA to the semantic similarity temperature $T_s$, which determines how close two solutions should be to be considered as belonging to the same reasoning direction. Experiments were performed on MATH500 with rollout budget $N = 64$, using LLaMA-3.1-8B-Instruct as the policy model. Results (averaged over 5 independent runs) are shown below:

The results suggest that DORA is relatively robust to variations in $T_s$.

To further assess the impact of the embedding model itself, we compared our original retriever (BGE-M3, 568M parameters) with two popular alternatives from the MTEB leaderboard [1] that support long input lengths (≥2048 tokens): E5-base-4k (110M) and GTE-multilingual-base (305M). All experiments were conducted on MATH500 with $N = 64$, under the same experimental settings. Results are averaged over 5 runs:

We observe that GTE and BGE perform comparably, showing that DORA is robust to embedding model changes within a similar capacity range. E5 does show a modest drop in performance, which we attribute to its smaller size and hence weaker ability to cluster semantically meaningful reasoning chains.

We will include these findings and discussions in the final version to better address concerns about DORA’s sensitivity to hyperparameter and embedding choices.

Q3: On the Clustering Mechanism

(i) Comparison to Hard Clustering Baselines

Thank you for raising this insightful question. We fully agree that understanding the role of clustering is key to explaining DORA’s effectiveness.

In the early stages of our study, we systematically explored several hard clustering alternatives before adopting the final soft clustering design:

• KMeans clustering requires predefining the number of clusters $K$. However, the actual number of distinct reasoning directions varies significantly across problems and reasoning steps. Using a fixed $K$ often led to poor alignment with the true semantic structure, resulting in suboptimal cluster quality.

• We also considered hierarchical clustering, which avoids setting $K$ by merging solutions based on a fixed cosine similarity threshold $C$. However, we found that the optimal value of $C$ should ideally vary with the reasoning depth and problem type—factors that are highly instance-dependent. Consequently, hierarchical clustering was even less stable than KMeans in our experiments.

Based on these observations, we adopted a soft clustering approach in DORA, which avoids hard assignment boundaries and allows for more adaptive semantic grouping. Below is our ablation comparing different clustering choices, conducted on the MATH500 benchmark with rollout budget $N=64$ using LLaMA-3.1-8B as the policy model (results are averaged over 5 independent runs):

As shown, soft clustering consistently outperforms hard clustering baselines, highlighting the importance of flexible, non-discrete grouping in capturing subtle reasoning structures. We will include this ablation study and analysis in the updated version.

(ii) Empirical Effectiveness of Semantic Grouping

We appreciate the reviewer’s thoughtful suggestion to quantify how effectively DORA’s clustering mitigates redundancy and improves success rates. This is indeed a highly valuable diagnostic perspective.

To better understand the effect of DORA’s semantic grouping mechanism, we conducted an additional analysis focusing on how well each method improves the intermediate success rate along the reasoning trajectory. Specifically, after each method allocates $K$ reasoning steps, we remove the search algorithm and resume temperature sampling based on the intermediate solutions obtained thus far. We then measure the pass rate of these partial trajectories to estimate the success rate at step $K$. This reflects the method’s ability to guide the policy model toward more promising reasoning directions early on.

Experiments were conducted on the MATH500 benchmark with rollout budget $N=64$, using LLaMA-3.1-8B-Instruct as the policy model and Qwen2.5-Math-PRM-7B as the PRM. All results were averaged over 5 independent runs:

As shown, DORA consistently achieves the highest pass rates across intermediate steps compared to all baselines. Notably, Step 0 corresponds to the Temperature Sampling baseline (i.e., without any search intervention). Comparing this with the improvements achieved by REBASE and DORA highlights the value of clustering: while both methods significantly outperform the baseline, DORA consistently maintains a lead, suggesting that its semantic clustering mechanism not only reduces redundancy but also enhances the effectiveness of search guidance. We will include this result and analysis in the revised version.

L3: On Generalization Beyond Mathematical Reasoning

Thank you for pointing out this important limitation. We fully agree that evaluating generalization across domains is an important direction.

DORA’s generalization beyond math (e.g., to code) is limited mainly by the lack of open-source step-level PRMs outside the math domain [2]. We have tested DORA and other step-level methods (Beam Search, DVTS, REBASE) on HumanEval and MBPP using Skywork-PRM—the only available PRM for programming tasks—but none significantly outperformed Temperature Sampling. This aligns with recent findings [3] showing Skywork-PRM offers weak step-level supervision. Using LLMs as critics is a potential alternative but adds considerable inference-time overhead, conflicting with TTS efficiency goals.

We will include this discussion and highlight the domain limitation. Extending DORA to code, science QA, and planning is a promising direction for future work.

[1] MTEB: Massive Text Embedding Benchmark, EACL2023.

[2] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning, ACL2025.

[3] CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation, ACL2025.

Dear Reviewer asTA,

We hope this message finds you well. As the discussion period is nearing its end, we would like to kindly follow up to ensure that our response has addressed your concerns satisfactorily. If there are any additional questions or suggestions you would like us to consider, please do not hesitate to let us know. Your insights are invaluable to us, and we would be more than happy to engage in further discussion to improve our work.

Thank you once again for your time and thoughtful feedback！

Best regards, The Authors

Dear Reviewer asTA,

We are writing to you with a gentle follow-up as the discussion period will close in about 30 hours. We completely understand that your time is precious and you are likely very busy. We just wanted to kindly reiterate that if you have any final comments or concerns, we are fully prepared to do whatever is necessary to address them promptly.

Thank you once again for your time and invaluable contribution to our work!

Best regards,

Authors