Semantic-guided Diverse Decoding for Large Language Model

We thanks for your valuable feedback and will response them in below.

W1: Figure 5 is lacking a lot of details. Which embedding space is used? Which projection method used? How is clustering performed? How is the probability magnitude computed - does it take into account answer length?

We apologize for the insufficient details in the main text. Complete experimental settings are provided in Appendix B.3 Semantic Space Visualization (Lines 456-479). Specifically: we use NovaSearch/stella_en_1.5B_v5 embeddings (Line 540) with t-SNE projection, K-means clustering (K=10), and position-debiased probability (Equation 1) that accounts for answer length through sequence position decay rate $\beta_{seq}$. The probability magnitude shown represents the average debiased probability across all tokens in each response.

W2: Important aspects such as the percentile-based normalization of individual scores only discussed in appendix. I would move the theorems on quality and diversity guarantees and the proposition into the appendix with a smaller summary theorem in the main paper and rather move those critical aspects into the main paper.

We appreciate your constructive suggestion regarding presentation balance. Due to space constraints, we prioritized theoretical analysis in the main text, but you are correct that critical implementation details deserve more prominence. We agree that SemDiD's primary value lies in its empirical effectiveness, and we will restructure to emphasize practical implementation aspects while moving detailed proofs to the appendix for better readability.

W3: An ablation on the importance of the individual scores is lacking, the same with a sensitivity analysis of important hyperparameters such as T_trans, lambda and epsilon.

We thank the reviewer for pointing out this. We have conducted comprehensive ablation studies and sensitivity analyses for existing components, and provide additional experiments for detailed hyperparameter analysis.

Regarding individual scores, we provide systematic ablation experiments in Appendix H (Table 4, lines 702-719). Each system component corresponds to its scoring function: directional guidance ($S_{dir}$), inter-group repulsion ($S_{rep}$), and harmonic gain ($S_{combined}$). We also provide supplementary experiments analyzing complete removal of scoring categories:

The results demonstrate that removing diversity scoring has the most dramatic impact, with coverage dropping 16.3%, confirming that semantic diversity is crucial for effective exploration.

Regarding hyperparameter sensitivity, we analyze the parameters in four categories:

Category 1: Automatically Derivable Parameters

For the position bias parameters, the hyperparameter choices in Equation 1 ($\beta_{seq}$, $\beta_{sent}$) are not arbitrary but can be systematically derived from probability-position analysis shown in Figures 3-4 in Appendix A.3. These parameters can be automatically fitted using scipy.curve_fit from probability-position curves, as these patterns remain consistent across tasks. We provide additional sensitivity analysis on BON (N=25) using Qwen-2.5-3B:

$\beta_{seq}$ and $\beta_{sent}$ Parameter Sensitivity Analysis (GSM8K Coverage):

Category 2: Resource-Dependent Parameters

For the lookahead depth parameter $L_{max}$, semantic diversity assessment requires embedding model forward passes, making token-by-token evaluation computationally expensive. We supplement our analysis in Appendix D.2-D.3 with experiments using Qwen-2.5-3B across different $L_{max}$ values:

Performance saturates around $L_{max}=10$, providing clear guidance for practical deployment without requiring extensive parameter search.

Category 3: Stage-Aware Transition Parameters

We apologize that explicit ablation of $T_{trans}$ values was not included in the main paper and provide this comprehensive analysis here. Parameter $T_{trans}$ controls the transition point from directional guidance to inter-group repulsion (Equation 8: $\alpha_t = \min(1, \frac{t}{T_{trans}})$). We conducted systematic experiments varying $T_{trans}$ across multiple datasets:

The results show clear optimal performance at $T_{trans} = 10$, corresponding to the typical number of tokens needed to establish meaningful semantic context before semantic trajectories become distinguishable.

Category 4: Quality-Diversity Balancing Parameters

The quality relaxation parameter $\gamma$ and harmonic strength $\lambda$ control the trade-off between maintaining quality thresholds and pursuing semantic diversity. While these parameters are discussed conceptually in Section 3.3, we provide detailed sensitivity analysis here by varying each parameter independently:

Effect of $\gamma$ (with $\lambda=2.0$ fixed):

Effect of $\lambda$ (with $\gamma=0.25$ fixed):

The results demonstrate that our default settings ($\gamma=0.25$, $\lambda=2.0$) consistently achieve near-optimal performance across different tasks, with performance remaining stable within reasonable parameter ranges.

For practitioners seeking to deploy SemDiD, we recommend starting with our provided default parameters for initial implementation, adjusting $E_t$, $b$, and $N$ based on available computational budget, and fine-tuning $\gamma$ and $\lambda$ only for highly specialized applications where task-specific optimization is critical. We hope these comprehensive analyses demonstrate that while SemDiD introduces additional parameters, most are either automatically derivable or have well-defined optimal ranges with clear empirical guidance, making the method practical for real-world deployment.

Remark: Many parts of the methodology remind me of the sampling strategy proposed in Aichberger et al. 2025. While they use the gradient signal and focus on uncertainty estimation, it could be worthwile to discuss in the related work section. Evaluating SemDiD for uncertainty estimation could be an interesting direction for future work.

Thank you for highlighting these important concurrent works. The following two works correlate with SemDiD, both of which explore semantic diversity in language generation through different views:

• Aichberger et al. (2025) [1] uses gradient-based attribution scores to identify semantically critical tokens for uncertainty estimation, employing importance sampling with NLI-guided token substitution.
• Zhu et al. (2025) [2] proposes controlled embedding perturbation at the first token position combined with Bayesian optimization for reasoning in latent space.

While these works share our motivation for semantic diversity, SemDiD differs in key aspects: (1) we operate during beam search decoding rather than post-hoc token substitution or first-token perturbation, (2) our approach balances quality and diversity through harmonic optimization rather than focusing solely on uncertainty or correctness, and (3) we target Best-of-N and RLHF applications rather than uncertainty estimation or reasoning tasks.

We will incorporate these works into our related work discussion and acknowledge the potential of combining our semantic guidance with their gradient-based attribution methods.

[1] Aichberger, Lukas, et al. "Improving uncertainty estimation through semantically diverse language generation." ICLR. 2025.

[2] Zhu, Qinglin, et al. "Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration." arXiv:2505.24688 (2025).

Thank you for recognizing we address an important real-world problem. Below we provide detailed responses for your concerns.

W1 & Q2 & Q5: Concern about computational latency and comparison under equivalent computational budget and scalability to larger models (70B+)

We acknowledge the computational overhead concern and provide comparison using equivalent latency budgets rather than equivalent sample counts. Experiments use Qwen-2.5-70B to simultaneously address scalability concerns. For parallel decoding methods, we first generate a larger candidate pool, then select N outputs using embedding-based clustering, and for Diverse/Determinantal Beam Search, we increase beam size from the default 3.

SemDiD demonstrates improved efficiency scaling with larger models. The actual latency increase drops from +27% (3B) to +18% (70B) because embedding computation becomes negligible relative to the 70B forward passes, while the semantic guidance benefits remain consistent.

While SemDiD appears to use more computation on the surface, the majority of operations benefit from KV-cache reuse since beams within groups share common prefixes, resulting in minimal actual latency increase. For parallel decoding methods, diminishing returns become evident as generating larger candidate pools leads to rapidly increasing costs with minimal diversity gains due to inability to reuse KV-cache. For Diverse Beam Search, simply increasing beam size yields negligible improvements while substantially increasing overhead.

SemDiD benefits from systematic exploration in semantic space through directional guidance and inter-group repulsion, efficiently utilizing every computational allocation to maximize semantic coverage rather than generating redundant variations.

W1 & Q1: Over-complex parameter design and arbitrary parameter selection

We understand the complexity concerns and provide clarification on hyperparameter design and robustness.

The parameters can be categorized as:

• Inherited Parameters: Temperature, top-p, $N$ (groups), and $b$ (beam size) come from standard Group Beam Search, not SemDiD additions.

• Automatically Derivable: Position bias parameters $\beta_{seq}$ and $\beta_{sent}$ can be fitted from probability-position curves (Figures 3-4) using scipy.curve_fit. Saturation threshold $\tau$ is derived from probability-quality analysis (Figure 2) and set to -0.8 based on empirical log probability-accuracy relationships.

• Resource-Dependent: Exploration width $E_t$ and beam size $b$ balance exploration versus computational cost, with clear "sweet spots" shown in Appendix D.2.

The hyperparameter choices in Eq 1 ($\beta_{seq}$, $\beta_{sent}$) are not arbitrary but can be systematically derived from probability-position analysis. We analyze parameter sensitivity on BON (N=25) by Qwen-2.5-3B:

Bold values indicate curve-fitted optimal parameters: GSM8K ($\beta_{seq}=0.0012$, $\beta_{sent}=0.005$) and BBH ($\beta_{seq}=0.0008$, $\beta_{sent}=0.005$). The results demonstrate that (1) optimal parameters remain reasonably consistent across datasets, (2) performance is robust within reasonable parameter ranges with <=0.3% variation, and (3) our default values (0.001, 0.005) lie within the optimal regions, eliminating the need for task-specific tuning.

W2 & W4: Cosine distance cannot capture complete semantic differences, and partial sentence embeddings may not reliably guide token-level decisions

We agree that cosine distance in embedding space does not necessarily equate to semantic differences or different solution approaches. Additionally, partial sentence embeddings may exacerbate this issue.

Fortunately, recent advances in LLM-based sentence models (like Qwen3 Embedding) have introduced new paradigms. They allow the same sentence to be combined with different task instructions to obtain corresponding representations in embedding space.

For example:

"Classify the mathematical approach used in this solution: Question: A rectangular garden has a perimeter of 40 meters and an area of 96 square meters. What are its dimensions? Answer: Let length = l and width = w. From perimeter: 2l + 2w = 40, so l + w = 20, which gives us..."

Vs.

"Classify the mathematical approach used in this solution: Question: A rectangular garden has a perimeter of 40 meters and an area of 96 square meters. What are its dimensions? Answer: Since area = 96, I need two numbers that multiply to 96. The factor pairs are: (1,96), (2,48), (3,32), (4,24), (6,16), (8,12). Now checking which pair has..."

In this way, embeddings leverage the foundational capabilities of LLM and can focus on solution difference.

Furthermore, we prioritize LLM-based embedding models with support of mean pooling, like NovaSearch/stella_en_1.5B_v5 (as Appendix C.2.2). Mean pooling averages all token embeddings, providing more stable representations for partial sentences compared to special token-based approaches.

W3 & Q3 & Q4: Theoretical assumptions regarding orthogonal guidance directions leading to semantic diversity. Whether method trades off multiple good answers for one good answer and several poor ones

As mentioned in previous response, LLM-based embeddings can better capture solution approach differences through instructions to classify solutions, enabling more reliable semantic diversity assessment.

We examined 20 GSM8K and BBH examples under N=10 setting and found an average of 2.3 distinct solution approaches with embedding distance 0.388. Notably, the embedding distances between different approaches reach 0.435, significantly larger than the 0.231 distance within the same approach. Even character-level 4-gram distances demonstrate similar patterns, with inter-approach distance of 0.884 exceeding intra-approach distance of 0.637.

Indeed, orthogonal directional guidance cannot guarantee different solutions, but the increased similarity distances during decoding help generate different reasoning approaches. When we remove such guidance, it produces an average of 2.1 distinct solutions with semantic distance 0.365.

In contrast, standard temperature decoding with N=10 produces only 1.4 distinct solutions with semantic distance 0.253.

While most solutions still follow similar reasoning patterns, the improvement from 1.4 to 2.3 distinct approaches represents meaningful gains in exploring alternative paths.

Regarding correctness, for more challenging datasets like Minerva Math, we observe cases where N=10 produces 2-3 different approaches but all results are incorrect across SemDiD and baselines. This reflects the foundation model's limited capability rather than a flaw in our diversity mechanism.

We also provide detailed output samples and correctness analysis in Appendix I.3 and Figure 15 (Lines 759-799) to examine the quality distribution. Additionally, Appendix E presents Accuracy Evaluation in Best-of-N, showing performance when using selection models to choose one response in practical scenarios. Results show SemDiD consistently outperforms baselines in both coverage and accuracy metrics, proving it generates multiple high-quality diverse answers rather than trading one good answer for several poor ones.

We sincerely thanks for your thoughtful review and valuable feedback on baseline comparisons and computational analysis.

W1: While SemDiD combines multiple ideas, individual components lack novelty... The paper does not sufficiently differentiate these mechanisms from existing approaches.

We appreciate this observation and would like to clarify the fundamental differences between our work and existing approaches.

Contrastive decoding like DoLa [1] operate by contrasting probability distributions from different sources, typically "expert" model against "amateur" model [2], or later layers against earlier layers within same model [1]. They modify token selection by computing $P_{expert}(token) - \alpha \cdot P_{amateur}(token)$ to amplify the expert's relative advantages and suppress tokens that both models find probable. Their target is to reduce hallucinations and improve factual accuracy.

In contrast, SemDiD focuses specifically on generating semantically diverse outputs for BON and RLHF applications. SemDiD works fundamentally differently: we use Gram-Schmidt orthogonalization to construct systematic exploration directions in semantic embedding space (Equation 4). These orthogonal vectors $\vec{d_g}$ provide theoretical guarantees that groups explore regions separated by at least $\frac{\pi}{k}$ radians, ensuring comprehensive semantic coverage. While contrastive methods improve quality by avoiding "amateur-level" outputs, we proactively guide generation toward distinct semantic territories.

For dynamic inter-group repulsion, this is a natural extension of diversity penalties to semantic space. Traditional diverse beam search applies discrete penalties based on n-gram overlaps, and our adaptation to continuous semantic embedding space using $S_{rep}(y_t^g) = -\max_{g' \neq g} \langle E(y_t^g), E(y_t^{g'}) \rangle$ follows naturally from this concept. The stage-aware transition mechanism (Equation 8) dynamically balances directional guidance and repulsion on generation progress, ensuring that predetermined directions don't become counterproductive.

Additionally, we identified and addressed systematic biases in token likelihood evaluation (Figures 3-4) that existing methods don't tackle. These components work together in a unified framework with both quality (Theorem 1) and diversity guarantees (Theorem 3), providing a systematic approach to semantic diversity rather than ad-hoc modifications.

W2 & W3 & Q1: Missing Baselines and Computational Comparison

We appreciate your suggestion to include contrastive decoding and embedding-based reranking baselines.

We conducted additional experiments comparing SemDiD against both DoLa and embedding-based post-hoc clustering across all benchmarks using Qwen-2.5-3B:

DoLa consistently underperforms SemDiD because it lacks explicit diversity mechanisms. And it often converges toward similar high-confidence solutions. This limitation is particularly on tasks requiring long sequence generation, such as mathematical reasoning, where DoLa's conservative token-by-token contrasting tends to favor safe, conventional solutions rather than exploring diverse reasoning strategies.

The clustering approach shows competitive performance at smaller N values, sometimes outperforming SemDiD but spending more computations. However, its effectiveness diminishes as N increases because the generate-cluster-select strategy becomes increasingly wasteful, generating 100 independent samples without coordination then discarding most content.

Computational Efficiency: While SemDiD introduces threefold theoretical computational requirements, KV-cache optimization and efficient implementation result in only 25-35% actual latency increase compared to group beam search (beam=3) (Appendix C.3). SemDiD's coordinated beam exploration allows extensive KV cache reuse within groups since they share common prefixes. In contrast, the clustering approach requires generating 100 independent sequences, making cache reuse impossible.

W3 & Q2: The analysis of scalability and latency to larger models (e.g., 70B parameters.

We provide detailed analysis using Qwen-2.5-70B on BON to validate SemDiD's effectiveness here:

The 70B model results reveal several important patterns. SemDiD consistently outperforms all baselines, with particularly strong improvements on complex reasoning tasks like BBH (+1.1%), GSM8K (+0.5%), and Minerva Math (+1.6% over the best baseline at N=25).

Notably, SemDiD shows only a +16% latency increase with 70B models, significantly lower than the +27% observed with 3B models, because embedding computation becomes negligible compared to 70B model forward passes. Although SemDiD appears to use more computation on the surface, the KV-reuse within groups reduces the inference cost per token by 80-90%, since these beam share the same prefix in exploration. Additionally, clustering-based methods often require more samples to match SemDiD's diversity, significantly increasing their actual computational cost. When accounting for cache efficiency and quality-adjusted sample requirements, SemDiD becomes computationally competitive while delivering superior semantic diversity

We hope these analyses could address your concerns about novelty, baseline comparisons, and scalability. We will incorporate these experiments into revised version.

[1] Chuang et al. "DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models." ICLR 2023.

[2] Li et al. "Contrastive Decoding: Open-ended Text Generation as Optimization." ACL 2023

We sincerely thanks for your thoughtful review and for acknowledging the solid empirical evidence in our RLHF and Best-of-N accuracy results.

W1: Confusions on the quality guarantee: could you explain more on the definition of the "maximum quality dispersion" and how it means in the real-world decoding scenario?

We apologize for not explaining this clearly in the main text. The maximum quality dispersion $\delta$ represents the largest possible difference in quality scores between any two responses in the entire solution space for a given query. Mathematically, $\delta = \max_{y_1, y_2 \in Y} |S_{quality}(y_1) - S_{quality}(y_2)|$, where $Y$ is the set of all possible responses.

Think of $\delta$ as the "quality range", it captures how much quality can vary between the best possible answer and the worst possible answer for a question. For example, in math problems, a high-quality solution might have log probability -0.4, while our quality threshold typically allows answers down to log probability -1.0, giving $\delta = |-0.4 - (-1.0)| = 0.6$ in the acceptable range.

Our Theorem 1 ensures that any diverse response we generate will not be worse than the greedy baseline by more than $\delta(1-\gamma)$. With $\gamma = 0.25$, this means our diverse answers stay within 75% of the maximum quality range from the greedy baseline. In practice, this prevents SemDiD from generating low-quality responses while pursuing diversity.

In our experiments, we observe $\delta$ values of 0.5-1.2 (in log probability space) across reasoning tasks, making our quality guarantee practically meaningful and ensuring diverse candidates remain high-quality.

W2: Confusions on the theoretical analysis on diversity guarantee: How do you derive the claim of "The minimum angle between any two vectors is at least $\frac{\pi}{k}$ radians"?

Although we included a brief proof for Theorem 3 (Diversity Guarantee), it may not be adequate. Below we provide a more detailed derivation.

We start by constructing orthogonal directions using Gram-Schmidt orthogonalization (Equation 4) to create $k$ directional vectors ${\vec{d_1}, \vec{d_2}, \ldots, \vec{d_k}}$ in the embedding space, ensuring different groups begin with orthogonal guidance directions.

Based on optimal geometric distribution theory, when distributing $k$ unit vectors to maximize their minimum pairwise angular separation, the optimal configuration approaches a regular simplex. For $k$ vectors on a unit hypersphere in $d$-dimensional space (where $d \gg k$), the minimum angle $\theta_{min}$ between any two vectors satisfies:

$\cos(\theta_{min}) = -\frac{1}{k-1}$

For large $k$, this approaches $\theta_{min} \approx \frac{\pi}{k}$, which is the bound used in our Theorem 3.

In practice, our generated sequences don't achieve perfect theoretical distribution due to language model constraints. For instance, with $k=6$ groups, the theoretical minimum angle would be $\frac{\pi}{6} \approx 30°$, though we observe actual angles ranging from approximately $25°$ to $45°$ between semantic groups.

In our SemDiD framework, while generated sequences don't perfectly follow the initial orthogonal directions due to language model constraints, the combination of directional guidance and inter-group repulsion maintains this angular separation in expectation. The semantic distance in embedding space is then bounded by:

$d_{sem}(y_i, y_j) = \frac{1 - \cos(\theta_{ij})}{2} \geq \frac{1 - \cos(\pi/k)}{2}$

Multiplying by the semantic space variance $\sigma$, we obtain the expected minimum pairwise semantic distance as stated in Theorem 3:

$E[\min_{i \neq j} d_{sem}(y_i, y_j)] \geq \sigma \cdot \sqrt{\frac{1 - \cos(\pi/k)}{2}}$

This theoretical guarantee ensures SemDiD maintains meaningful semantic diversity across generated responses, with diversity level scaling appropriately with the number of groups $k$.

W3: Lack of ablation study on some critical hyperparameters: how the number of groups $k$ and the update transition weight impact the performance?

We respectfully note that comprehensive analysis of the number of groups is already provided in our paper, though perhaps not prominently highlighted. For the transition weight, we acknowledge this was not explicitly ablated and provide detailed analysis here.

Number of Groups $k$ Analysis: The impact of group numbers is extensively evaluated through our Best-of-N and RLHF experiments. In Figure 7, we test across different sample sizes (N = 3, 10, 25, 50, 75, 100), where the number of groups is adaptively set to $k$ depending on the target sample count. These experiments demonstrate how SemDiD's performance scales with different group configurations across 9 diverse benchmarks.

Additionally, in Appendix D.3 (Table 3), we provide detailed "Group Structure Analysis" comparing different outputs-per-group configurations while maintaining a constant total of 10 outputs:

• 1 output per group (10 groups): 94.8% coverage on GSM8K.
• 2 outputs per group (5 groups): 94.5% coverage on GSM8K.
• 3-4 outputs per group (3 groups): 92.2% coverage on GSM8K.
• 5 outputs per group (2 groups): 87.9% coverage on GSM8K.

Furthermore, Appendix D.2 (Figure 10) provides analysis of exploration width and beam size parameters, showing the interaction between different SemDiD components across group numbers from 1 to 10.

Transition Weight $T_{trans}$ Analysis: We acknowledge that explicit ablation of $T_{trans}$ values was not included in the main paper. We provide this comprehensive analysis here. The parameter $T_{trans}$ controls the transition point from directional guidance to inter-group repulsion (Equation 8: $\alpha_t = \min(1, \frac{t}{T_{trans}})$). We conducted systematic experiments varying $T_{trans}$ across multiple datasets:

The results show clear optimal performance at $T_{trans} = 10$. When transition occurs too early ($T_{trans} = 5$), groups haven't established sufficient semantic differentiation before repulsion dominates, leading to suboptimal exploration. When transition is delayed ($T_{trans} \geq 20$), groups may converge to similar semantic regions before inter-group repulsion becomes effective.

Intuitive Explanation: $T_{trans} = 10$ corresponds to the typical number of tokens needed to establish meaningful semantic context. Most mathematical reasoning problems require 8-12 tokens to establish the early thought process for problem-solving before semantic trajectories become distinguishable.

W4: Could the proposed method be combined with other modern decoding techniques, such as speculative decoding and Coconut?

We thank you for this forward-looking question about SemDiD's compatibility with other advanced techniques.

It's important to clarify SemDiD's position in the decoding ecosystem. SemDiD belongs to the category of diverse decoding methods and serves as a direct replacement for techniques like nucleus sampling and diverse beam search. These methods all address the challenge: generating multiple diverse outputs from a single input. SemDiD distinguishes itself by operating in semantic space rather than relying on lexical diversity or simple probability manipulation.

SemDiD is orthogonal to inference acceleration techniques like speculative decoding, making integration straightforward. However, the compatibility with hidden state based reasoning methods like Coconut requires more careful consideration.

For Speculative Decoding, SemDiD can enhance both stages of speculative decoding effectively. During the draft stage, we can apply lightweight semantic guidance using a smaller embedding model with reduced parameters (smaller $L_{max}$, fewer groups) to generate diverse draft sequences. This allows the smaller draft model to explore different semantic regions rather than just high-probability continuations. In the verification stage, the larger model evaluates not only the probability of each draft candidate but also incorporates their semantic diversity scores when selecting the optimal continuation. This combination maintains speculative decoding's computational efficiency while achieving superior semantic diversity.

For Coconut, it operates by feeding hidden states directly as input embeddings, bypassing token decoding entirely. This conflicts with SemDiD's design, which relies on embedding text sequences to assess semantic diversity.

SemDiD cannot be directly applied because our mechanisms operate on text sequences, while Coconut works purely in hidden state space. Our directional guidance and inter-group repulsion require sentence embedding. Adaptation would require developing hidden-state-based diversity metrics, extending quality assessment to evaluate hidden states rather than token probabilities, or applying SemDiD only at Coconut's final text output stage.

Coconut requires significant algorithmic adaptations but could potentially unlock powerful new forms of diverse reasoning in continuous latent space.

We appreciate the reviewer's constructive feedback again, which has helped us clarify these important theoretical and practical aspects. In particular, the questions about compatibility with other methods have prompted us to rethink SemDiD's position and its integration potential.

We thank you for acknowledging the strength of our RLHF application results and for raising important questions about Best-of-N accuracy evaluation and hyperparameter complexity. We provide detailed responses below.

W1: Evaluation limited to coverage metrics without accuracy results in Best-of-N

We appreciate this feedback and agree that accuracy is crucial for comprehensive Best-of-N evaluation. We actually conducted complete accuracy evaluations, but due to space constraints, these results were placed in Appendix E and Figure 11 (Lines 632-651). We apologize that this placement made these critical results less prominent in the initial submission.

In these experiments, we used Qwen-2.5-3B to generate N candidate answers for each problem, then employed LLM-Blender PairRM (based on DeBERTa-v3-large, 430M parameters) as an automatic evaluator to select the best answer from the N candidates and compute accuracy. This simulates real-world Best-of-N scenarios where a verification mechanism determines the final output.

SemDiD consistently outperforms all baselines on accuracy across all benchmarks. On GSM8K (N=25), SemDiD achieves 77.5% accuracy compared to 75.5% for Determinantal Beam Search, representing a 2.0% improvement. On ARC-Challenge (N=25), SemDiD reaches 82.0% versus 71.0% for the best baseline, showing a 1.0% gain. For MMLU-Pro+ (N=25), SemDiD achieves 36.5% compared to 34.0% for the best baseline, demonstrating a 2.5% improvement.

These results demonstrate that SemDiD's semantic diversity not only helps find correct answers (coverage) but also enables more reliable identification of the best answer (accuracy). This confirms that SemDiD generates high-quality, diverse candidate sets rather than achieving coverage through numerous low-quality outputs.

W2: Method complexity and extensive hyperparameters with limited sensitivity analysis

We understand the complexity concerns and provide comprehensive clarification on hyperparameter design and robustness.

Firstly, temperature, $Top-p$, $N$ (number of groups), and $b$ (beam size) are inherited from standard Group Beam Search parameters, not SemDiD additions. The SemDiD-specific parameters serve distinct purposes across three categories: semantic diversity assessment ($E_t$, $L_{max}$), quality assessment ($\beta_{seq}$, $\beta_{sent}$, $\tau$), and quality-diversity balancing ($T_{trans}$, $\gamma$, $\lambda$).

Automatically Derivable Parameters: Several key parameters can be systematically determined rather than manually tuned. The position bias parameters $\beta_{seq}$ and $\beta_{sent}$ can be automatically fitted using scipy.curve_fit from probability-position curves shown in Figures 3-4, as these patterns remain consistent across tasks. The saturation threshold $\tau$ is derived from probability-quality analysis (Figure 2, Appendix A.2) and set to -0.8 for most tasks based on the empirical relationship between log probability and answer quality. The transition point $T_{trans}$ balances directional guidance and inter-group repulsion, set to 10 to address early-stage semantic exploration when sentence lengths are insufficient for reliable embedding.

Resource-Dependent Parameters: The exploration parameters $E_t$, b, and N balance exploration breadth versus computational cost. Appendix D.2-D.3 provides comprehensive analysis showing diminishing returns beyond $E_t$=4 and b=3, establishing clear "sweet spots" without extensive tuning requirements.

Lookahead Depth Analysis: Unlike character-level diversity methods that can evaluate each vocabulary token individually with minimal computational cost, semantic diversity assessment requires embedding model forward passes, making token-by-token evaluation prohibitively expensive. Additionally, single-token semantic changes are often too subtle for reliable diversity measurement. Therefore, SemDiD introduces the lookahead depth parameter $L_{max}$ to control how many tokens ahead we explore when evaluating semantic diversity, allowing assessment of more substantial semantic deviations while managing computational overhead.

We conducted experiments using Qwen-2.5-3B across different $L_{max}$ values to demonstrate its sensitivity and identify optimal settings:

Performance saturates around $L_{max}$=10, providing clear guidance for practical deployment without requiring extensive parameter search.

Quality-Diversity Balancing Analysis: The quality relaxation parameter $\gamma$ and harmonic strength $\lambda$ control the trade-off between maintaining quality thresholds and pursuing semantic diversity. We conducted systematic sensitivity analysis by varying each parameter independently:

Effect of $\gamma$ (with $\lambda$=2.0 fixed):

Effect of $\lambda$ (with $\gamma$=0.25 fixed):

The results demonstrate that our default settings ($\gamma$=0.25, $\lambda$=2.0) consistently achieve near-optimal performance across different tasks, with performance remaining stable within reasonable parameter ranges. The sensitivity analysis shows that SemDiD is robust to parameter variations, with performance degrading gracefully rather than sharply when moving away from optimal values.

For practitioners seeking to deploy SemDiD, we recommend starting with our provided default parameters for initial implementation, adjusting $E_t$, b, and N based on available computational budget, and fine-tuning $\gamma$ and $\lambda$ only for highly specialized applications where task-specific optimization is critical.

We hope these comprehensive analyses demonstrate that while SemDiD introduces additional parameters, most are either automatically derivable or have well-defined optimal ranges with clear empirical guidance, making the method practical and robust for real-world deployment scenarios.

Dear Reviewer jjpi,

Thank you again for your thoughtful review of our paper "Semantic-guided Diverse Decoding for Large Language Model". We greatly appreciate the time and effort you invested in evaluating our work.

We have provided a comprehensive rebuttal addressing your concerns, with the missing accuracy evaluation results and hyperparameter complexity. As the rebuttal deadline approaches, we wanted to ensure you had sufficient time to review our detailed responses:

1. Complete accuracy results: We clarified that full Best-of-N accuracy evaluations were conducted and included in Appendix E and Figure 11. SemDiD consistently outperforms baselines across all benchmarks (e.g., 2.0% improvement on GSM8K, 1.0% on ARC-Challenge).

2. Hyperparameter analysis: We provided comprehensive sensitivity analysis showing that most parameters are either automatically derivable or have well-defined optimal ranges, making SemDiD practical for deployment.

We hope these clarifications address your concerns about evaluation completeness and method practicality. We would be grateful for any additional feedback you might have after reviewing our responses.

Thank you for your continued engagement with our work.

Best regards,
The Authors