Mixture of Inputs: Text Generation Beyond Discrete Token Sampling

Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding the Dirichlet model, intuition behind MoI, evaluation setup, and qualitative study in this rebuttal, with further elaborations to be included in the final paper.

Linear Interpolation and Dirichlet Posterior Mean It is worth mentioning that the posterior mean of Dirichlet is a linear interpolation of the prior mean and the observation, and the mixing scheme proposed by the reviewer ($t \cdot e_{\mbox{chosen}} + (1 - t) \sum p_i e_i$, we replace $\alpha$ with $t$ since $\alpha$ has special mearnings in Dirichlet) can be viewed as a reparameterization of the posterior mean. Particularly, we assume $e_i$ to be the one-hot vector of $i$ for simplicity, then $t \cdot e_{\mbox{chosen}} + (1 - t) \sum p_i e_i$ is the posterior mean of the Dirichlet model $\alpha \to w \to e_{\mbox{chosen}}$, where $w \sim \mbox{Dir}(\frac{1-t}{t} p)$ and $e_{\mbox{chosen}} \sim \mbox{Multinomial}(w)$.

Compared to the plain linear interpolation, the posterior mean yields the same results, but with a clearer interpretation. Specifically, the linear interpolation is viewed as the posterior mean, and the interpolation ratio is decided by the concentration factor. We believe this interpretation should be clean and understandable for the NLP community, where the Dirichlet prior is widely used, and we will add writing to make the connection to linear interpolation as simple and widely accessible as possible.

Length of generated text in throughput analysis To better compare the throughput, we picked the benchmarks where the generation length is about the same. The median difference between MoI-generated text and baseline-generated text is 1.7%.

Significance of the improvements We report the average over 5 independent runs for each decoding configuration (as stated in Section 5.4, line 213). In addition, we visualized performance distributions in Appendix A to give insight into variability across seeds.

Rational on why mixing continuous embeddings helped Intuitively, continuous embedding is capable of capturing more information in the continuous state than the discrete token; there have been some recent theoretical analyses such as [1]. They effectively showed that continuous states can enable models to explore multiple generation paths at the same time, where discrete tokens need to collapse on one path at a time, hence can be more effective in finding a correct solution.

We believe this continuous blending helps preserve distributional information that would otherwise be lost during hard sampling, enabling the model to produce more coherent and contextually consistent generations. Our empirical findings support this interpretation, particularly on tasks that benefit from structured reasoning (e.g., math and code).

Qualitative Analyses on Repetitiveness We conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-grams repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).

With standard sampling (repetitive around HCA definition):

To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies "not LCA," we assume the HCA refers to the shallowest common ancestor (closest to the root) [content continues...]

With mixture of inputs sampling (reduced repetitiveness):

To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [content continues...]

Additional results We would like to highlight the additional experiments presented in our rebuttal:

1. Table A (in our reply to Reviewer Py1S) summarizes instruction following evaluation results on MT-Bench across 6 models, including 2 additional models that are smaller in sizes (i.e., Qwen-2.5-14B and LLaMA-3.1-8B), where MoI brings consistent performance gain.

2. Table B (in our reply to Reviewer Py1S) explores the robustness of Hyper-parameter tuning. It tunes hyperparameters on AIME/CountDown, and then directly transfers the resulting hyperparameters to GPQA and LiveCodeBench. MoI also leads to consistent performance gain under this setting.

We hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.

[1] Zhu, Hanlin, et al. "Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought." arXiv preprint arXiv:2505.12514 (2025).

Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding evaluation setup, the role of entropy in MoI, and the intuition of the Dirichlet model.

Response to Question 1 on baseline setup Both the vanilla sampling and the direct mixture baseline were also tuned over grid search for top-p and temperature. We will add an explanation in the paper to clarify this.

Response to Question 2 on evaluation setup We deliberately choose not to do grid search on GPQA and LiveCodeBench to demonstrate that MoI does not rely heavily on hyperparameter tuning.

Response to Question 3.a on the importance of entropy In our early phase experiments, we had explored replacing entropy with a hyper-parameter, which causes QwQ-32B to drop 7.07% on GPQA-Diamond. After some analyses, we found it is necessary to balance the output distribution and one-hot output token, for which we found entropy to be a natural choice.

Response to Question 3.b on the design of normalization We chose the [0,1] range to ensure all token weights remain non-negative and interpretable as probabilities. Extending the range or allowing negative weights is possible, though we suspect this may complicate optimization and introduce instability. Exploring more flexible formulations would be an interesting direction for future work.

Response to Question 4.a on the intuition behind the Dirichlet model At the input, the model has both the predicted distribution and the sampled discrete token. To capture the semantic meaning of both, we need to “adapt” the predicted distribution so that the resulting distribution incorporates the information of the sampled discrete token. To this end, we find it is a natural idea to use the Dirichlet model. As widely used in topic modeling (LDA), the Dirichlet distribution is the natural prior over multinomial probabilities and gives closed-form posteriors (due to conjugacy).

Response to Question 4.b on the predicted distribution p The predicted distribution p represents the semantics of model outputs before we commit to a sampling trajectory. Using p directly as w would ignore the valuable signal from the sampling decision. In our experiments, we refer to this approach as Direct Mixture and find MoI to outperform this baseline consistently (as in Table 1 of the submission).

Response to Question 4.c on the uncertainty Although we observed both (1) the model's predicted distribution, and (2) the sampled token, it is still an open question to find a distribution that incorporates the semantics of both the distribution and the token. Here, we assume such an underlying distribution exists, and propose to use the Dirichlet method to estimate it.

Additional results We would like to highlight the additional experiments presented in our rebuttal:

1. Table A (in our reply to Reviewer Py1S) summarizes instruction following evaluation results on MT-Bench across 6 models, including 2 additional models that are smaller in size (i.e., Qwen-2.5-14B and LLaMA-3.1-8B), where MoI brings consistent performance gain.

2. Table B (in our reply to Reviewer Py1S) explores the robustness of Hyperparameter tuning. It tunes hyperparameters on AIME/CountDown, and then directly transfers the resulting hyperparameters to GPQA and LiveCodeBench. MoI also leads to consistent performance gain under this setting.

3. In the Qualitative Analyses on Repetitiveness part of our reply to Reviewer Py1S, we observe that MoI helps to reduce the repetitiveness in the model generations.

We hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.

We thank you for the constructive feedback. We appreciate your comments and will address concerns regarding instruction-following capacity, hyper-parameter selection, and qualitative study in this rebuttal, with further elaborations to be included in the final paper.

Additional results on instruction-following benchmarks and more models We conducted additional experiments on MT-Bench, using the four models from our original paper, and adding two smaller models, Llama-3.1-8B and Qwen-2.5-14B. Below, we report the average performance for standard decoding (temperature=0.6, top-p=0.95) and our Mixture of Inputs (MoI) decoding (beta={1,2,4}, same decoding config). together with the relative improvements brought by MoI (Δ). As summarized in Table A, MoI consistently outperforms the baseline for all models.

Table A: MT-Bench Scores

Hyperparameter Robustness In our experiments, we observed that it generally improves the performance across tasks by setting beta to less aggressive values (0.5-2), while tuning the hyperparameter has the potential to further push the performance for each task. To demonstrate the robustness of the hyperparameter, we conduct a hyperparameter transfer experiment as below.

In particular, we first choose the hyperparameter (beta/temperature/top-p) based on the performance on AIME & CountDown-4, and test it on GPQA-Diamond and LiveCodeBench. As summarized in Table B, MoI consistently outperforms the baseline for all models.

Table B: Hyperparameter Robustness (hyperparameters used in this experiment is picked based on their AIME & CountDown-4 performance)

Qualitative Analyses on Repetitiveness We conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-gram repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).

With standard sampling (repetitive around HCA definition):

To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies "not LCA," we assume the HCA refers to the shallowest common ancestor (closest to the root) [continues...]

With mixture of inputs sampling (reduced repetitiveness):

To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [continues...]

Response to Question 1: what about Beta=0? We think beta=0 would be similar to the case of the direct mixture baseline. Beta=0 would fully emphasize the predicted distribution and ignore the sampled token. We found this often leads to less grounded and unstable generations, so we focused on more practical beta values that blend both signals.

Response to Question 2: why not Qwen? QwQ-32B is the leading open-source mode at its 32B scale from the Qwen family at the time of our submission. In Table A, we included a Qwen-2.5-14B model in our MT-bench experiments and found MoI consistently brings performance to this model.

Response to Question 3: what about removing H? In our early phase experiments, we had explored replacing H with a hyperparameter, which causes QwQ-32B to drop 7.07% on GPQA-Diamond. After some analyses, we found it is necessary to balance the output distribution and one-hot output token dynamically, for which we found entropy to be a natural choice.

We hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.

[1] Zhu, Rui-Jie, et al. "A survey on latent reasoning." arXiv preprint arXiv:2507.06203 (2025).

Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding inference engine compatibility, instruction-following capacity, qualitative study in this rebuttal, with further elaborations to be included in the final paper.

Modern inference engine compatibility We agree modern inference engines play an important role in LLM serving, and accordingly build our implementation on top of vLLM, despite it requiring more engineering efforts. In fact, all experiments in our submission are conducted on our vLLM-based implementation, and the baseline method in the throughput analyses is the original vLLM. As to the precision issue, the vLLM V1 engine’s sampler casts the softmax logits to fp32 for sampling (as in L47 of the sampler.py in the vLLM repo), and we followed the same strategy and didn’t notice any significant issues.

Additional results on instruction-following benchmarks and more models We conducted additional experiments on MT-Bench, using the four models from our original paper, and adding two smaller models, Llama-3.1-8B and Qwen-2.5-14B. Below, we report the average performance between standard decoding (temperature=0.6, top-p=0.95) and our Mixture of Inputs (MoI) decoding (beta={1,2,4}, same decoding config), together with the relative improvements brought by MoI (Δ). As summarized in Table A, MoI consistently outperforms the baseline for all models.

Table A: MT-Bench Scores

Qualitative Evaluation on Repetitiveness We conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-grams repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).

With standard sampling (repetitive around HCA definition):

To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies "not LCA," we assume the HCA refers to the shallowest common ancestor (closest to the root) [content continues...]

With mixture of inputs sampling (reduced repetitiveness):

To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [content continues...]

We hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.