Thanks for your valuable feedback and constructive suggestions. We have addressed your comments on the justification and robustness of our heuristic discrimination methods, the need for an analysis of failure modes, and the clarity of our token-mapping explanation. In response, we provide a detailed theoretical grounding for our approach, present a new ablation study on several discrimination functions, and will add both a qualitative analysis of failure cases and a concrete token-mapping example to the revised paper. Please refer to the specific responses below for a detailed discussion.

---

To Q1-1: Justification for Using a Simple Heuristic

We sincerely thank the reviewer for highlighting this critical point. And we appreciate the opportunity to further validate our choice of alignment discrimination. Our choice of a simple heuristic like Max-Match is theoretically grounded in the Superficial Alignment Hypothesis.

• Superficial Alignment Hypothesis: This hypothesis posits that the core knowledge of an LLM is acquired during pre-training, and alignment primarily teaches the model a specific style or sub-distribution of responses, rather than altering its fundamental capabilities.

This theoretical foundation is supported by some empirical evidence.

• For example, by analyzing the differences between a base model and its aligned version, URIAL found that knowledge-intensive content overwhelmingly occurs at unshifted positions, where the token chosen by the aligned model is also the most probable choice for the base model. Across thousands of examples, a vast majority of tokens (92.2%) were found to be either unshifted or only marginally shifted. This strongly suggests that the significant distribution shifts, the ones that truly constitute the "alignment," are not in the core knowledge but are concentrated in other parts of the response, such as linguistic style and formatting.
  • Their definition of a shifted position (where the aligned model's chosen token is ranked low by the base model) conceptually mirrors our own condition for triggering alignment in Max-Match (where the top-ranked tokens of the two models disagree).
• We also observed a similar pattern (Appendix A): when comparing different sizes of aligned models within the same family (Gemma2), we found a very high overlap (over 70-80%) in their most frequently altered tokens.

These findings suggest that the most critical alignment decisions often happen at predictable, high-frequency "choice points" where the model selects a stylistic or safety-related token. A simple method like Max-Match, which checks for a disagreement in the single most probable token (argmax) between the unaligned model (A) and its aligned counterpart (A it), is therefore highly effective at capturing these most significant and frequent alignment-related shifts.

It is effective because it directly targets the primary mechanism of superficial alignment without the need for more complex distributional comparisons, which might be diluted by the vast majority of tokens that remain unchanged.

---

To W1 & Q1-2: Threshold Tuning and Robustness

We appreciate the concern regarding hyperparameter sensitivity. In our experiments, we chose Max-Match as the default configuration precisely because it is hyperparameter-free. The rule is simple: if the most likely token predicted by model A differs from that of model A_{it}, the position is flagged for alignment. This eliminates the need for any manual threshold tuning, making the method robust and easy to deploy.

To address the broader question of "How do hyperparameters affect performance?", we direct the reviewer to Figure 6 (previously Figure 4) in our paper, which can also effectively serve as a hyperparameter ablation study. In this analysis, we moved from the hyperparameter-free Max-Match (equivalent to a Top_p threshold of 0) to a Top_p based overlap method and varied the Top_p_A threshold.

The results clearly demonstrate that the model's performance on MT-Bench remains remarkably stable across a range of thresholds, even as the "Guided Decoding Ratio" (the percentage of replaced tokens) decreases from 30% to 23%. This indicates that GOOD's effectiveness is not very sensitive to the exact quantity of guided tokens but rather hinges on the accuracy of identifying the correct positions for guidance. The stability shown in this experiment validates our choice of a simple, robust heuristic.

---

To Q1-3: Ablation Study on Discrimination Functions

To provide a direct quantitative validation of our default Max-Match function, we analyzed its performance in detecting alignment-critical positions. For our ground truth, we decoded responses on the MT-Bench dataset using Qwen2.5-7b-it and identified every position where an alignment occurred by comparing its output to its base model, Qwen2.5-7b, via Max-Match. We then evaluated how well the alignment decisions made by our guiding pair (Qwen2.5-3b-it vs. Qwen2.5-3b) correlated with this ground truth. The performance of this predictive task was as follows:

• Accuracy: 0.8169
• Precision: 0.5122
• Recall: 0.5789
• F1 Score: 0.5435

These metrics show that our simple heuristic achieves a strong balance. It correctly identifies over half of the true alignment-critical positions (Recall: 57.9%) while ensuring that the positions it flags are indeed alignment-related more than half the time (Precision: 51.2%).

Furthermore, to explore whether more sophisticated discrimination functions would yield significant gains, we conducted an additional ablation study comparing Max-Match to three advanced, logits-based methods. The results on AlpacaEval are summarized below:

(In these three new methods, we used Qwen2.5-7b as the guided model, and Qwen2.5-3b / Qwen2.5-3b-it as the guiding model pair. Since these methods require hyperparameter tuning, we first analyzed token-level statistics from MT-Bench with Qwen2.5-7b-it. Specifically, we collected logits at every decoding position, computed the above metrics, and labeled positions requiring alignment. We then searched optimal thresholds for each method by maximizing F1 Score).

Interestingly, while these advanced methods also perform well, our simple, hyperparameter-free Max-Match method remains highly competitive and even outperforms them in this configuration. This reinforces our finding that a straightforward heuristic is surprisingly effective for capturing superficial alignment shifts and offers an excellent trade-off between simplicity, robustness, and performance.

---

To W2: Analysis of Misleading Signals

That is a very insightful point, and we thank the reviewer for raising it. Understanding the failure modes, especially scenarios where the guiding models might provide misleading or harmful signals, is crucial for a complete picture of GOOD's robustness and limitations.

The reviewer is correct that our main paper did not explicitly analyze these failure cases. This is a valuable area for exploration. To address this, we will add a new section to the appendix dedicated to a qualitative analysis of failure modes.

In this new section, we will provide and discuss several concrete examples from our experiments, illustrating:

1. Successful Guidance: Cases where the guiding signal correctly identifies an alignment-related issue (e.g., a missing safety disclaimer or a stylistic deviation) and successfully corrects the output of the base model.
2. Misleading Guidance: Scenarios where the stylistic preferences of the guiding model do not align well with the requirements of a specific task, potentially leading to a suboptimal but not factually incorrect response.
3. Potentially Harmful Guidance (Negative Cases): We will also present instances where the guidance might have been detrimental, for example, by incorrectly suppressing a correct fact or introducing an unhelpful conversational filler.

By presenting these case studies, we aim to provide a more nuanced understanding of when and why GOOD performs well, and what the potential pitfalls are when the guiding signal is imperfect. We believe this analysis will not only strengthen the paper but also provide valuable insights for future research on decoding-time alignment.

Thank you again for this excellent suggestion.

---

To Q2: Concrete Example of Token-mapping

We thank the reviewer for these comments on the clarity of our token-mapping mechanics. We agree that the process described in Section 3.3 is technically dense and that a concrete example would significantly improve its accessibility. We will incorporate this suggestion into our revision.

---

Finally

We sincerely thank you for your thoughtful and constructive feedback on our work. If you believe our replies have resolved the issues raised in your review, we would greatly appreciate your reconsideration of the manuscript’s evaluation, for a potentially higher rating. If there are any remaining concerns or further suggestions, we would be more than happy to address them.

Thanks for your valuable feedback and constructive suggestions. We have addressed your comments regarding the selection of models, benchmarks and baseline methods, the exploration of advanced discrimination methods, the practicality of deployment, and the need for theoretical grounding. To address these concerns, we have conducted extensive new experiments on AlpacaEval, compared GOOD against important baselines, and evaluated new logits-based discrimination methods. Please refer to the specific responses below for the new results and our reasoning.

---

To W1, W3, Q1, Q3: Models, Benchmark & Baseline Method Concern

We sincerely thank the reviewer for highlighting these critical points. In response, we provide new experiments to address the concerns about models, benchmarks, and baselines.

For Models & Benchmark Concern

We expanded our evaluations by:

• Including more recent and powerful model (Qwen2.5) alongside part of our original choices (Gemma2).
• Conducting experiments on AlpacaEval 2.0, a more comprehensive and stable benchmark widely recognized in recent alignment literature.

The extended evaluations on AlpacaEval 2.0 are shown below:

These results confirm that our method can successfully transfer alignment even to state-of-the-art models (recovering up to 95.35% of the fine-tuned target) and generalize well beyond a single benchmark scenario.

For Baseline Method Concern

Regarding baseline methods, we agree with the reviewer’s observation and acknowledge the importance of comparisons against newer methods.

Actually, in Table 4 of the original manuscript, we already provided comparative results with recent methods ( ARGS (Jan 2024), Transfer-Q (May 2024), CARDS (Jun 2024), and GenARM (Oct 2024 ) ) evaluated on the HH Dataset. Despite leveraging significantly smaller guiding models, GOOD achieves competitive alignment performance, outperforming several reward-based methods (ARGS, Transfer-Q, CARDS) and approaching the performance of GenARM.

Additionally, we have now added a direct comparison on AlpacaEval 2.0 (evaluated with the official default setup). In this new benchmark, we evaluate GOOD against a broad spectrum of recent and established alignment baselines, including PPO, DPO, BoN, Item-level RS, RAIN, and TreeBoN, to further substantiate its efficacy:

(Note: For this comparison, GOOD used TinyLlama-1.1B-Chat and TinyLlama-1.1B as guiding models. And Llama-7B was used as the base model for all methods. Transfer-Q's implementation is currently unavailable, and GenARM’s repository has become inactive, with no direct comparative data on AlpacaEval provided in the original paper.)

The significant improvements demonstrated by GOOD relative to these advanced methods underscore its competitive advantage and practical applicability.

---

To W2, Q2: Improving and Comparing more Discrimination Methods

Thanks for this insightful suggestion. To address this, we conducted additional experiments exploring three advanced discrimination methods based on logits:

• KL divergence between logits distributions.
• Entropy difference between the logits distributions.
• Cosine similarity between embedding vectors (obtained by multiplying logits with the embedding matrix).

In this experiments, we used Qwen2.5-3b and Qwen2.5-3b-it to guide Qwen2.5-7b. Since these methods require hyperparameter tuning, we first analyzed token-level statistics from MT-Bench with Qwen2.5-7b-it. Specifically, we collected logits at every decoding position, computed the above metrics, and labeled positions requiring alignment. We then searched optimal thresholds for each method by maximizing F1 Score:

• For KL divergence and entropy difference, alignment is triggered if the metric is above the threshold.
• For cosine similarity, alignment is triggered if the metric is below the threshold.

The final evaluation on AlpacaEval is summarized below:

The results show that all three advanced methods perform well and significantly outperform the base model. It also confirms the strength and simplicity of our Max Match method, which works reliably without extra hyperparameters. At the same time, the findings suggest promising future directions, such as hybrid or learning-based discrimination methods, to further enhance alignment accuracy and stability.

---

To W4: Deployment Concern

We appreciate the reviewer for raising this important point about deployment practicality. While multi-model systems introduce complexity, we would like to offer a different perspective: we believe that GOOD is in line with a growing trend in LLM deployment practices, where multi-model designs are increasingly explored as a means to enhance—rather than compromise—efficiency, performance, and modularity.

Here are several lines of evidence from recent research and deployed systems that support the practicality of our approach:

1. Multi-Model Systems for Inference Acceleration**:** For instance, SpecInfer [1], a system for accelerating LLM serving, utilizes multiple small draft models in parallel to generate a token tree, which is then verified by a single large target model. In this highly practical system, the multi-model approach is not a deployment burden, but the core mechanism for achieving significant serving acceleration.

2. Prevalence and Conceptual Analogy to Mixture-of-Experts (MoE): This trend extends beyond academic research and into widely-used commercial models. The MoE architecture, exemplified by models like Mixtral 8x7B, is fundamentally a multi-model system at inference time, routing tokens through different expert sub-networks to combine their specialized knowledge.

   From this perspective, GOOD can be viewed as a dynamic MoE-like system for alignment, where the guiding model pair (A, A_it) serves as an alignment expert, and the discrimination function acts as a router.

3. Ensembles and Hierarchies for High-Stakes Applications: In many real-world scenarios, especially high-stakes domains like medicine or finance, using a single model is often insufficient. The LLM‑Synergy framework [2] shows that combining a clinically fine-tuned LLM with a general-purpose one can significantly reduce hallucinations and improve factual accuracy in medical QA, underscoring the practicality and necessity of LLM ensembles in high-stakes domains.

Viewed in this context, GOOD may be seen not as an outlier, but as a novel example within the broader multi-model paradigm—one that aligns with trends in advanced LLM system design and deployment.

[1] SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

[2] Large Language Model Synergy for Ensemble Learning in Medical Question Answering: Design and Evaluation Study

---

To W5, Q4: Theoretical Analysis

Thank you for this suggestion. Our method,is theoretically grounded in the Superficial Alignment Hypothesis. This hypothesis posits that alignment primarily teaches the model a specific style or sub-distribution of responses, rather than altering its core knowledge and fundamental capabilities.

This theoretical foundation is supported by some empirical evidence:

• Evidence from Prior Work: Prior work (URIAL [2]) shows that alignment primarily affects stylistic tokens, with over 92% of tokens remaining "unshifted" from the base model's top choice, strongly suggests the "alignment signal" is concentrated in the few positions where the top choice diverges.
• Evidence from Our Observations: Our own analysis (Appendix A) reveals the implication of this for transferability. When comparing different sizes of aligned models within the same family, we found a high overlap (>70-80%) in their most frequently altered tokens, which implies a consistent, transferable alignment behavior.

These two points provide a direct theoretical justification for why GOOD's approach is effective:

• The high overlap in alignment related tokens across different models suggests that the "rules" of alignment (e.g., which phrases to use for politeness, how to format code blocks) are not arbitrary or model-specific but follow predictable, transferable patterns.
• An "alignment expert" derived from one model pair learns decision patterns that are largely applicable to another model from a similar pre-training distribution. This makes the alignment signal fundamentally transferable.

---

Finally

We sincerely thank you for your thoughtful and constructive feedback on our work. If you believe our replies have resolved the issues raised in your review, we would greatly appreciate your reconsideration of the manuscript’s evaluation, for a potentially higher rating. If there are any remaining concerns or further suggestions, we would be more than happy to address them.

Thanks for your valuable feedback and constructive suggestions. We've addressed your comments on the generalizability of our evaluation, the clarity of the method's presentation, and the fairness of the decoding speed comparison. To address these concerns, we have conducted new experiments on AlpacaEval 2.0 and a direct runtime comparison against standard speculative decoding. We will also thoroughly revise the paper for clarity as suggested. Please refer to the specific responses below for detailed discussions and results.

---

To W1: Benchmark Concern

We thank the reviewer for the suggestion to broaden the scope of our evaluation, and we appreciate the opportunity to further validate the generalizability of our method GOOD.

Following this suggestion, we conducted new experiments that expand our evaluation in two key dimensions. First, we incorporated evaluations on AlpacaEval 2.0 (also recommended by reviewer fBoL), a widely-used benchmark for alignment assessment in LLMs. AlpacaEval covers diverse open-ended and reasoning tasks, and is commonly used in recent alignment literature [1]. Second, to demonstrate GOOD's effectiveness on the current state-of-the-art LLMs, this new evaluation incorporates a more recent and powerful model family (Qwen2.5). In each experiment setup, we utilize a pair of guiding models to align the behavior of the pretraining guided model.

Our results show that GOOD also achieves competitive performance on AlpacaEval 2.0 (evaluated with gpt-4o-mini using the official specific stepup):

We believe these results provide strong additional evidence for the robustness of GOOD. The competitive performance (recovering up to 95.35% of the fine-tuned target) on a benchmark with a distinct methodology demonstrates that our method's ability to transfer alignment is not specific to a single evaluation format.

We hope this new data effectively addresses the reviewer's comment. We thank the reviewer again for their insightful feedback.

[1] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

---

To Q1: Table Presentation of URIAL Method

Thanks for your comment. In Table 1, the green color indicates "No" (an advantage), and the red color indicates "Yes" (a disadvantage), which may not have been sufficiently clear. We share the same understanding of URIAL as you described. We acknowledge that the current presentation in the table may cause confusion and will revise it to improve clarity in the next version.

---

To Q2: Compare Speed with Standard Speculative Decoding

We thank the reviewer for this insightful suggestion. The reviewer is correct that the speedup we presented is relative to vanilla decoding and that a direct comparison with a standard speculative decoding baseline is necessary for a complete and fair assessment.

Following the reviewer's suggestion, we have conducted this exact runtime comparison. The results are presented below:

These results confirm the reviewer's intuition. While GOOD is faster than vanilla decoding (achieving up to a 13% speedup), it is slower than standard speculative decoding in the current configurations.

This is an expected result, as GOOD utilizes a pair of guiding models (A and Ait) to perform its unique alignment discrimination, whereas the standard speculative decoding baseline here uses only a single draft model. The overhead of running this second guiding model contributes to the performance difference.

However, we would like to argue that this is not a fundamental limitation but rather a matter of configuration and optimization. The core idea that using multiple smaller models can still yield significant speedups is supported by recent work. For instance, SpecInfer[1] successfully uses multiple small draft models and ultimately achieves a 1.5-2.8x speedup. This demonstrates that multi-model overhead can be effectively amortized in a well-designed system.

Analogously, we believe that with further optimization—such as carefully selecting the relative sizes of the guiding pair and the target model—it is plausible that GOOD's runtime could become comparable to that of standard speculative decoding.

[1] SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

---

To W2, Q3, Q4, Q5, Q6, Q7

We appreciate the reviewer for their constructive feedback. These suggestions are very helpful for improving our paper’s clarity and quality, and we will incorporate these improvements in our revision. Here, we provide additional information to clarify the specific points raised.

To Q3: Explanation of Figure 1

Thanks for pointing this out. The notations relate to our two-step speculative process:

• n_matches_align: This refers to the number of tokens accepted in the first guess-and-verify step (between the guiding model pair). It counts how many initial tokens from the aligned guiding model (A_{it}) are generated without requiring an alignment
• n_matches_main: This is the number of tokens accepted in the second guess-and-verify step. It counts how many tokens (which come from the first step) are accepted by the guided model (B) in standard speculative decoding.
• In both cases, “match” means the decoding results are consistent between the compared models.

We have included a detailed explanation and illustration of this process in Appendix D and Figure 9. And we also plan to release the codebase soon, which will further help readers understand the implementation details.

To Q4: Explanation of the Algorithm Description

The final "verification" step integrates both alignment and speculative decoding, taking the output from the first verification as its input. The logic is as follows:

• If n_matches_main < n_matches_align, the guided model disagrees with the predicted sequence before an alignment related token is reached. We thus accept the n_matches_main matching tokens.
• If n_matches_main ≥ n_matches_align, the guided model accepts the sequence up to the alignment point. We therefore accept n_matches_align tokens and then substitute the next one with the guidance from A_{it}.

A detailed illustration of this process is provided in Appendix D and Figure 9. The upcoming release of our codebase will also help clarify the implementation for interested readers.

To Q5: Explanation of Figure 2

This figure shows scores from the MT-Bench benchmark (as mentioned in the last paragraph of Section 4.3). The scores were computed using the official evaluation script of MT-Bench, which uses GPT-4 for automated assessment. We will ensure the figure caption is more self-contained in the revision.

To Q6: Explanation of Figure 4

We apologize for the unclear legend. Top_p_ori should have been written as Top_p_A . The purpose of this analysis was to show how tuning the alignment sensitivity (by adjusting Top_p_A) impacts the Guided Decoding Ratio (the proportion of tokens replaced) and the overall model performance on MT-Bench.

To Q7: Explanation of Figure 5

We acknowledge this comment and agree that Figure 5 should also be more self-explanatory. Both figures will be revised with more informative captions and clearer legends.

---

Finally

We sincerely thank you for your thoughtful and constructive feedback on our work. If you believe our replies have resolved the issues raised in your review, we would greatly appreciate your reconsideration of the manuscript’s evaluation, for a potentially higher rating. If there are any remaining concerns or further suggestions, we would be more than happy to address them.