Cascade Reward Sampling for Efficient Decoding-Time Alignment

Thank you for raising your concerns to this paper. We would like to clarify our manuscript and address your questions in the responses below.

Q1: Lemma 1 assumes the reward is already equal to that value.

A1: Thank you for your comment. Our paper does not contain a “Lemma 1” and our Equation 4 refers to the target distribution, which is unrelated to value functions. If you are referring to another work or a different section, please clarify, and we will be happy to address it directly.

Regarding the relationship between RMs and value functions, we discuss their connection in Section 4.3.3, under uncertainty-based segmentation. We claim that RMs and value functions share similarity based on the empirical results of reward accuracy (Fig. 2 and 3).

Q2: Core claims regarding the use of reward scores for incomplete sequences are not sufficiently substantiated by prior literature.

A2: Thank you for highlighting this concern. Prior works have used RMs to score incomplete sequences in autoregressive generation [1, 2, 3], but these approaches lack justification for the accuracy of RMs on partial sequences. To address this gap, our paper provides new empirical evidence demonstrating that RMs can be accurate on semantically complete prefixes (see Section 4.3), clarifying the conditions under which RMs may be reliably applied to incomplete sequences. We will further clarify this contribution and its relation to prior work in the final version.

[1] ARGS: Alignment as Reward-Guided Search. ICLR 2024.

[2] RAIN: Your Language Models Can Align Themselves without Finetuning. ICLR 2024.

[3] Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model. EMNLP 2023.

Q3: The experiments primarily target a single dataset (HH-RLHF) and models with relatively modest scale.

A3: Besides HH-RLHF, we also have empirical results on AdvBench, SafeRLHF, AlpacaEval 2.0, Ultrafeedback, BeaverTails, and HelpSteer. We emphasize the results on HH-RLHF because our RMs are trained on this dataset, but our results demonstrate strong generalization across diverse tasks and domains.

Regarding model scale, we primarily use 7B models due to computational constraints, which is consistent with recent work [1, 4]. Scaling to larger models is an important direction, and we plan to explore this in future work.

[4] TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling. 2024.

Q4: The potential computational burden under high rejection rates or large batches is not addressed.

A4: Thank you for raising this important point. The computational burden under high rejection rates is primarily influenced by the reward threshold: lower thresholds reduce rejections and thus computational cost. For batched inference, we describe a simple batching strategy in Appendix A.2, which allows larger batches to be processed more efficiently than smaller ones.

Q5: Lacking referencing relevant prior works.

A5: Thank you for pointing this out. While we briefly mention controlled decoding in L92, we agree that the prior works you referenced [5, 6, 7] are relevant. They share a similar alignment framework of reward-guided generation, but focus more on the design of the reward signals rather than the generation strategy as studied in our paper. We will update the related work section to better contextualize our contributions within this literature.

[5] From r to Q∗: Your Language Model is Secretly a Q-Function. COLM 2024.

[6] PAD: Personalized Alignment at Decoding-Time. ICLR 2025.

[7] Controlled Decoding from Language Models. ICML 2025.

Dear Reviewer,

Thank you for your valuable feedback and engagement throughout the review process. We want to remind you that the deadline for the author-reviewer discussion period is approaching.

We have not yet received a response to our posted replies, which we believe address the concerns you raised regarding our paper. We kindly encourage you to review our responses and, if you find them satisfactory, to reconsider your rating.

Authors

Reviewer t2yh, please consider the authors' response and the other reviews. Thanks, Area chair

We truly appreciate your positive feedback and the acknowledgment of our experiments. We would like to address your questions in the response below.

Q1: Some segmentations (longer ones?) might be more suitable for 'helpfulness' as opposed to 'safety'.

A1: Thank you for raising this interesting point. We agree that optimal segment length may vary for different objectives. Our segmentation hyperparameters for all experiments were originally tuned on HH-RLHF for harmlessness & helpfulness (examples in HH-RLHF mainly focus on safety), resulting in relatively short segments (7~8 tokens on average). To explore the impact on helpfulness, we used general utility as the evaluation metric. We increased the uncertainty threshold to 4.0 (yielding segments of ~20 tokens on average) and present the results below, evaluated on the same data as Table 4 and using llama-7b as the base model. The findings show that longer segments slightly improve general utility and diversity metrics. We will include this ablation study in the final version.

Thank you for your constructive feedback and for recommending acceptance of our paper. We appreciate your thoughtful comments and would like to clarify the points you raised in the review.

Q1: It might be useful to include fixed segmentation as part of the baselines in the evaluations.

A1: Thank you for this insightful suggestion. We have now included CARDS with fixed-length segmentation (we set the length to be 10 tokens, which approximates the average length from uncertainty-based segmentation) as a baseline in our evaluation (see table below). As shown, fixed segmentation results in lower alignment quality (RM Score: 7.84 vs. 8.30) and increased computational cost (Total Calls: 1058.33 vs. 872.91), confirming its limitations compared to our approach. The suboptimal performance originates from the incompleteness of these segments, which causes the partial reward scores to be no longer accurate. We will add these results to the final version.

Q2: Line 164: it would be good to see how frequently this length limit is reached. Or even better, show some statistics like the distribution of segment length and the number of segments in one sequence.

A2: Thank you for raising this question. We already had visualizations of segment length statistics in Fig. 7 and 8. To accurately reflect the segmentation behavior, we did not enforce the 32-token length limit in these analyses. The results show that, under our reward threshold (3.0), the average segment length is 7~8 tokens, with most segments under 20 tokens. We will clarify these findings in the final version.

Q3: Sec 4.3.3: any empirical support for this claim?

A3: Thank you for highlighting this point. Our claim regarding the relationship between RMs and value functions is motivated by our observation that semantically complete prefixes preserve the accuracy of RMs. However, obtaining an accurate value function for direct empirical comparison remains challenging, as prior works [1, 2] rely on approximations and do not open-source their code. We will clarify this limitation in the final version and more explicitly ground our claim in the recent literature.

[1] From r to Q∗: Your Language Model is Secretly a Q-Function. COLM 2024.

[2] Controlled Decoding from Language Models. ICML 2024.

Q4: Table 2 and Table 3: why would the proposed method outperform item-level RS?

A4: Thank you for your question. It is true that RM on full sequence is more accurate. However, Item-level RS is less efficient than CARDS. Given a practical compute budget, CARDS identifies more aligned responses than item-level RS and therefore outperforms it in experiments. CARDS is more efficient because it generates responses segment by segment, conditioning each new segment on previously identified high-reward prefixes. This approach focuses the search on promising regions of the response space, increasing the likelihood of producing high-reward outputs compared to sampling entire responses independently, as in item-level RS. We will clarify this advantage in the revised manuscript.

Dear Reviewer,

Thank you for your valuable feedback and engagement throughout the review process. We want to remind you that the deadline for the author-reviewer discussion period is approaching.

We have not yet received a response to our posted replies, which we believe address the concerns you raised regarding our paper. We kindly encourage you to have a look at our responses to help all reviewers move closer to consensus on this paper

Authors

Thanks for your detailed and thoughtful feedback, as well as for recognizing the strengths of our method. We appreciate the opportunity to address your concerns and clarify several points that may have led to misunderstandings in our manuscript. We hope that our responses below will clarify the value and rigor of our work.

Q1: In §4 and §5, which RM is used? Does the reward correlation between segments and full-length responses (§4.3.2) hold for multiple choices of RMs?

A1: As noted in L217, our main experiments use a RM trained from Llama-7B on HH-RLHF. To assess the robustness of our findings, we also evaluated reward correlation using an RM trained from Llama-2-7B on UltraFeedback, as shown in Fig. 13. The reward correlation between segments and full-length responses remains consistent across these different RMs, demonstrating that our results generalize well to multiple RM architectures and datasets.

Q2: Does the efficiency analysis in Table 1 depend on hyperparameter choices? If so, how did you pick them?

A2: The efficiency analysis in Table 1 does depend on hyperparameter choices. For CARDS, we performed a grid search to select hyperparameters, as detailed in Appendix B.3. For RAD/ARGS and TreeBoN, we adopted the recommended settings from their code for a fair comparison. For item-level RS, we set the reward threshold to match CARDS (r=8.5), and for BoN, we used the same number of samples as ARGS (n=20). These choices ensure a fair comparison between baselines.

Q3: In §4.3, what choice of fixed length do you use for the comparison to fixed-length segments (Fig. 2, 3)?

A3: For the fixed-length segment comparisons in Fig. 2 and 3, we use the 25%, 50%, and 75% prefixes of the full responses and report the average reward score across these segments. We appreciate your suggestion and will include this clarification in the final version.

Q4: In §5, what data was used for training the PPO and DPO baselines?

A4: Both the DPO and PPO baselines are trained on the HH-RLHF dataset, using both Llama-7B and Mistral-7B-v0.2 as base models. This ensures a fair and consistent comparison across methods. We will release the model weights upon acceptance.

Q5: What hyperparameters are used for CARDS ($\tau_u$, $\alpha$, $r^\star$, $\beta$)? What value of N is used for BoN?

A5: The hyperparameter values for CARDS ($\tau_u$, $\alpha$, $r^\star$, $\beta$) are summarized in Appendix B.3. These were selected via grid search to ensure optimal performance. For BoN, we set $n=20$, following the recommendation in [1]. We will ensure these details are presented in the final version.

[1] https://github.com/deeplearning-wisc/args

Q6: In §5.3, how do you measure the diversity and coherence of responses?

A6: As described in L282, we follow the evaluation of [2] (Section 3.1) to measure diversity and coherence. Specifically, diversity is the aggregation of n-gram repetition rate, while coherence is the cosine similarity between the sentence embeddings of the prompt and its continuation. We will clarify these definitions in the final version.

[2] ARGS: Alignment as Reward-Guided Search. ICLR 2024.

Q7: Most of the results in the paper use LMs to assign scores from 1-5. This is very prone to biases.

A7: This evaluation setup follows previous works [2, 4], and we believe it is fair if we compare the scores of different methods on the same set of prompts. We agree that a head-to-head comparison might be more unbiased, and this is precisely why we included Table 3.

Q8: It would be helpful to report the standard deviation and/or statistical significance tests.

A8: Thank you for your suggestion. Due to computational constraints–primarily the use of 7B models and the HH-RLHF dataset (12.5k rows)--we were unable to run repeated experiments to report standard deviations or statistical significance tests. This is consistent with prior works [2, 3, 4], which also do not report error bars or significance tests. We agree that including such analysis would strengthen the results, and we will consider this in future work as resources allow.

[3] RAIN: Your Language Models Can Align Themselves without Finetuning. ICLR 2024.

[4] Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model. EMNLP 2023.

Dear Reviewer,

Thank you for your valuable feedback and engagement throughout the review process. We want to remind you that the deadline for the author-reviewer discussion period is approaching.

We have not yet received a response to our posted replies, which we believe address the concerns you raised regarding our paper. We kindly encourage you to review our responses and, if you find them satisfactory, to reconsider your rating.

Authors