Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

We greatly appreciate the reviewer’s recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.

However, some questions remain in the hyperparameter part. What is the hyperparameter for the decision tree? Do we need to adjust the parameters during the inference stage, and how can we do this?

We perform the decision tree training with max depth=10. The results of using different max depths can be found here:

Since the performance is mainly influenced by the training dataset rather than the hyperparameters of the decision tree, so it is not necessary to repeatedly tune the hyperparameters — setting a reasonable value is sufficient.

Essential References Not Discussed

We will add the references to the revised paper. Thanks for pointing out valuable papers to cite and discuss.

I suggest to show more interesting samples like Table 4 in the appendix.

We will add more samples, especially for the coding samples (i.e., HumanEval test samples) to the appendix and further discuss them in the revised paper.

What is the token latency for only use one LLM in Table 6?

The token latency of using only one LLM will be around 30-50 ms in Table 6. However, since CoSD is designed to combine complementary knowledge from two models, rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model, but a naive two-model decoding setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline Avg Decoding). In this case, the token latency will be more than 1.5x times higher than CoSD:

It seems that the CoSD does not directly modify the answers in Table 4. So how does CoSD improve the score on GSM8K? More explanations are needed.

We discuss how CoSD improves the score in the Case Study section. In CoSD-Rule, in the fifth line, the assistant model rejects the draft model’s incorrect computation of 20% of 20 = 10 and instead uses the correct calculation of 20 * 0.2 = 4, successfully avoiding the error in the draft model’s tax calculation. In the sixth line, the draft model correctly leads to generate the subtotal of $24, so in the final step, CoSD-Rule computes the simpler 24 + 5 instead of the more complicated 15 + 3 + 2 + 5, resulting in the correct answer.

We greatly appreciate the reviewer’s recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.

The number of samples for the decision tree. The hyperparameters of the decision tree.

We would like to clarify that using 3 dataset samples (e.g., 3 samples from GSM8K) does not mean the decision tree is trained on only 3 tokens. Each GSM8K sample typically contains more than 50 tokens and can have up to hundreds of tokens. Therefore, 3 samples from GSM8K can provide over 150 training points for the decision tree, which is sufficient given the data sample size (1×2). The sensitivity of CoSD-Tree to the training dataset is reported in Table 5. The relationship between the number of training samples and CoSD-Tree performance is shown below:

We also conducted experiments varying the max depth of the decision tree:

(10 and 20 get the same decision tree)

These results show that CoSD-Tree is not sensitive to the number of training samples or the tree’s hyperparameters. However, it is more sensitive to the type of dataset, as illustrated in Table 5.

Statistical significance testing

We will report mean ± standard deviation over 5 runs in our experiments in the revised paper.

It would be valuable to include a detailed error analysis to identify exactly when and why the decision rules or decision tree sometimes select incorrect tokens.

We include illustrative examples in Table 4 and Table 9. For MMLU samples, where a single token determines whether an answer is correct, the decision rule or tree may choose incorrect tokens when the model assigns higher confidence to an incorrect answer. This typically occurs due to hallucination by the assistant model.

For benchmarks requiring chain-of-thought outputs (e.g., GSM8K in Table 4), the replaced tokens are often not directly tied to the final answer (e.g., replacing “we” with “the” in CoSD-Rule). Thus, it can be challenging to pinpoint exactly when and why the decision rule/tree fails. A more informative analysis is to compare CoSD outputs with the outputs of the individual draft and assistant models.

As shown in Table 4, in the fifth line, the assistant model corrects a draft model’s incorrect calculation of 20% of 20 = 10 by replacing it with the correct calculation 20 * 0.2 = 4, avoiding an error in tax computation. In the sixth line, the draft model correctly computes the subtotal $24, leading CoSD-Rule to choose the simpler 24 + 5 over the more verbose 15 + 3 + 2 + 5, producing the correct final answer.

For the tinyMMLU dataset, our experiments show that the wrong replacement rate is around 2%–3% across different model pairs. For Capacity Imbalance tasks (i.e., pair 4), this rate is 0%.

Have you considered incorporating more advanced uncertainty estimation methods or ensemble learning approaches (such as Bayesian ensemble methods)? Would this significantly alter CoSD’s complexity or performance?

Thank you for the thoughtful suggestion.

We agree that advanced uncertainty estimation techniques and ensemble learning methods, such as Bayesian ensemble models, are valuable and may be worth exploring in future work. However, in this paper, our primary goal is to develop a simple, lightweight, and efficient collaborative decoding framework. To this end, we intentionally adopt minimal decision strategies — a rule-based filter and a shallow decision tree — which are fast, interpretable, and require minimal supervision.

While more advanced methods like Bayesian ensembles may provide stronger theoretical guarantees, we believe they are unlikely to yield significant performance improvements over CoSD in practice, especially considering the strong results we already observe across benchmarks. Moreover, such methods typically involve additional complexity, including performing posterior inference, and evaluating model-specific performance. These extra steps would compromise one of CoSD’s key advantages — being plug-and-play, without requiring per-model evaluation on multiple benchmarks.

We will clarify this design choice and trade-off more explicitly in the revised version. We sincerely thank the reviewer for bringing up this important point.

We greatly appreciate the reviewer's recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.

What will happen if we swap the draft model and the assistant model? Since the authors claim that users don’t need to choose between LLMs, they may also not want to determine which LLM to be the draft/assistant model. Therefore, the question is will the performance drop significantly if we swap the two models? What will happen if we swap the two LLMs?

We performed experiments on swapping the draft model and the assistant model. Here are the results of pair 2:

The results show that CoSD still outperforms all the baselines. We will add these results to the revised paper.

I’m also very curious about the real samples of the HumanEval dataset. Since the authors already showed the real samples of MMLU (QA) and GSM8K (Math), what the sample will look like in code generation will be interesting. Will the assistant model help repair bugs in the draft generations? A table similar to Table 4 can be helpful.

Thanks for the valuable suggestion. We will add a HumanEval real sample to the revised paper. Our sample shows that the assistant model can modify the draft code to another style (e.g., with more function), which sometimes improves the accuracy and reduces the number of bugs.

The knowledge fusing and catastrophic forgetting healing seem similar, might need some explanations.

The knowledge fusing task aims to fuse the knowledge of two models with complementary knowledge. The catastrophic forgetting task often fuses the knowledge of one LLM that is only good at one specific task (through fine-tuning) and another model good at all other tasks.

The hyperparameters seem to be determined by Figure 2. Do other benchmarks follow the same pattern? Need to be explained.

Yes, all the experiments of CoSD-Rule follow the same hyperparameters. We found that the optimal $\alpha$ and $\beta$ values are transferable between models and tasks.

Comments and Suggestions

Thanks for the suggestions of some details in the paper. We will further polish the paper according to the suggestions.

If the two models are in the same size, can CoSD still speed up the inference?

Yes, although speculative decoding cannot speed up inference with two models with the same size, our CoSD is used for a different task. CoSD is designed to combine complementary knowledge from two models rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model but a naive two-model decoding setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline Avg Decoding). In this case, the token latency will be more than 1.5x times higher than CoSD:

Therefore, CoSD still benefit from the speculative decoding algorithm in this task.

We greatly appreciate the reviewer's recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.

However, I was surprised that no evaluation of speed/efficiency was done, as this seems to be a critical claimed strength over other work that does not take advantage of blockwise parallel decoding.

We would like to clarify that our method has indeed been evaluated in terms of speed and efficiency. Specifically, Table 6 in our paper reports latency results, showing that our method achieves nearly the same runtime performance as standard speculative decoding. This confirms that CoSD maintains comparable decoding efficiency. However, we agree that additional efficiency experiments would further highlight our contributions, and we will include them in the revised version of the paper.

Both the "drafter" and the "assistant" model are not necessarily computationally that much different, so decoding from each model in parallel (vs. one first, and then the other blockwise parallel) might not have as much benefit?

This is an excellent point — In our case, the drafter and assistant models have similar capacities, so such gains might not be expected. However, our task is fundamentally different. While speculative decoding aims to approximate the behavior of a large model efficiently, CoSD is designed to combine complementary knowledge from two models, rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model, but a naive two-model decoding setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline Avg Decoding).

For clarity, we provide the comparison below:

Note that Avg Decoding could also be implemented using a speculative-style process, but we adopt standard two-model decoding (i.e., two LLMs generating autoregressively) to emphasize the efficiency gain introduced by speculative decoding in CoSD. In this setting, CoSD is at least 1.5× faster than baselines that do not use speculative decoding. We will include these results in Table 6 of the revised paper.

One weakness which sticks out to me is that this method is unlikely to scale well to multiple collaborating LLMs?

We agree that collaborative generation involving more models may reduce the acceptance rate. However, our experiments suggest that the drop is not as drastic as one might expect. Our key observation is that for many common, non-domain-specific token sequences (e.g., “I am”, “This is”), well-trained models tend to produce highly consistent predictions. As shown below, the acceptance rate remains relatively high even as more models are added:

The experiment settings follow Table 6.

Additionally, we can also use a lower number of generated token per step (i.e., lower K in Algorithm 1) to drop less tokens, thus ensuring the efficiency when we have a lower acceptance rate. We will clarify and expand upon this discussion in the revised version of the paper.

From just looking at Fig. 1 it is unclear what the main differences are between CoSD and standard speculative decoding.

We agree with the reviewer that Fig 1 focuses more on the speculative decoding part than the knowledge fusion part. We will modify the figure in the revised paper and emphasize the knowledge fusion ability of the “CoSD Verification” part.

L138: "generated autoregressively and produced sequentially" is redundant.

Thanks for pointing out the redundant part. We will modify it in the revised paper.

The results should have error bars... Speculative Decoding should have the same average performance as the assistant model, but exhibits a substantial amount of deviation in the table.

We will add the error bars in the revised paper. The speculative decoding algorithm we use (Miao et al., 2023) has a soft verification strategy, that uses a random number as a threshold to decide whether to accept the draft token for efficiency. In this case, the average performance of speculative decoding will not be exactly the same as the assistant model.

The assistant model is always "invoked" question.

You're absolutely right that in our current method, the assistant model is always invoked via the speculative decoding. What we intended to convey in the abstract is that the token replacement is invoked when the assistant token differs from the draft model's token and pass the decision rule. We agree that the original wording may misleadingly suggest that the assistant model is not used at all unless triggered by a separate process. We will revise this in the final version for clarity, to better reflect the actual mechanism.