Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

We sincerely appreciate the time and effort you have invested in reviewing our paper. We apologize for any confusion and would like to clarify our motivations and experimental settings.

Q1: About the Presentation of the Paper

Thank you for your comments. We would like to share our thoughts on the “slide deck” presentation style, which aims to make the conclusions clearer, as Reviewer DHMi aptly summarized. Given the page limitations, we chose to highlight the main analyses and conclusions for each figure and finding. However, we recognize that this approach may impose a burden of understanding, especially for readers less familiar with the field. To address this:

• We will strengthen the connections between the figures and their corresponding conclusions in response to the questions raised. Preliminary connection instructions will be provided in #Q2.
• We apologize for the lack of detailed descriptions of the experiments and the process of deriving conclusions. In the revised version, we will reorganize the paper to include related work and preliminary knowledge in the main text.

Q2: About working mode of collaborative decoding

A1: Running Example

Our primary objective is to analyze the frequency of collaboration in various decoding settings. In our research, we explore collaborative decoding (CoDec) at all steps ($CoF=1$), for example:

[User]: "Who is Donald Trump?"
[Assistant]: "Donald Trump is the former President of the United States, who is 78 years old now."

For a lower collaboration frequency ($CoF_{\text{lower}}$), we input the outputs of CoDec into smaller models token by token to assess the consistency of top tokens. (CoDec represents speculative decoding ,contrastive decoding or proxy tuning)

- First token verification, match:
	- CoDec: [Assistant]: "Donald"
	- Small: [Assistant]: "Donald"
- Second token verification, match:
	- CoDec: [Assistant]: "Donald Trump"
	- Small: [Assistant]: "Donald Trump"
- ...
- 5th token verification, mismatch:
	- CoDec: [Assistant]: "Donald Trump is the former President"
	- Small: [Assistant]: "Donald Trump is President"
- ... (match)
- 14th token verification, mismatch:
	- CoDec: [Assistant]: "Donald Trump is the former President of the United States, who is 78"
	- Small: [Assistant]: "Donald Trump is the former President of the United States, who is 80"
- ... (match)

Assuming there are three mismatched tokens (e.g., "former", "78"), the calculated $CoF_{\text{lower}}=\frac{2}{18}$. However, unnecessary collaborations may occur even when matches are identified, leading to an variable where $CoF_{\text{lower}} \leq CoF \leq 1$. This motivates our investigation into the lower bounds of collaboration frequency, aiming to achieve similar outputs as full collaborative decoding with minimal collaborative steps. Our findings demonstrate this is a universal phenomenon across different collaborative decoding methods.

Speculative decoding currently selects a fixed number of tokens (K-tokens) for generation-verification, which does not effectively reach $CoF_{\text{lower}}$. In contrast, methods such as contrastive decoding and proxy tuning entail collaborations at each step ($CoF=1$), which may not always be necessary.

A2: Raw Accuracy Scores of Different Methods

While accuracy scores are not the primary focus of our experiments—where the outputs of mix-scaled models are treated as golden outputs—we provide raw accuracy scores for different collaborative methods to support the validity of our approach.

The results demonstrate the effectiveness of collaborative decoding, showing that mix-scaled models outperform small models operating independently. Furthermore, these findings underscore the potential to optimize collaboration efficiency based on the insights presented in our paper.

• Table 1: Accuracy of Different Collaborative Decoding Methods on GSM8k

Dear authors,

Thank you very much for your explanations. I remain with my earlier comment that the questions asked in your article are interesting and that there are likely many valuable results in your paper. I appreciate that you, among other things, acknowledge in your response that it is necessary to strengthen the connections between the figures in the corresponding conclusions (and give some of those explanations in your rebuttal), that you agree that experimental details are needed for readers to validate your experiments and that the paper could benefit from some more explicit reasoning an how the conclusions were arrived. I believe that doing all these changes would drastically improve your paper (though I would have to take another few hours to re-review to make sure that the conclusions make sense given the added experimental details, descriptions, motivation and reasoning). I also think that these changes would be quite substantial (as confirmed also by the length of your response) and -- as I said in the line before -- reviewing them would take almost as much time as reviewing the paper in the first place. For me, this is beyond the expectations of a rebuttal phase and I will thus stick with my recommendation to reject your work. Of course, it is just a one man's opinion, and I want also to acknowledge that I do think your paper has promise and carries some interesting ideas. The other reviewers appear to be more positive about this work than myself, so perhaps the AC will just overrule my specific opinion :). I hope in any case that my comments were useful to improve your work.

Q4: Motivation and Analogy of System 1 and 2 with Collaborative Decoding

In this work, we draw inspiration from the analogy of System 1 and System 2, simplifying their collaboration into Fast and Slow thinking. System 1 efficiently handles approximately 95% of routine tasks, while System 2 is reserved for deliberately addressing the remaining 5% of complex work [1]. Together, they demonstrate the power of high-efficiency collaboration.

We adopt this high-efficiency motivation to model the collaborative decoding methods between fast and slow (or small and large) models. Our experimental findings (Findings 1 and 2) show that small, fast models generate approximately 80% of tokens during the answering process, while large, slow models contribute the remaining 20%.

Looking forward, we aim to expand these collaborative mechanisms to reasoner and knowledger models, such as OpenAI’s o1 and GPT-4, which not only embody the fast/slow model paradigm but also represent intuitive and deliberate thinking. Preliminary experiments reinforce our findings, indicating successful collaboration between o1 and existing large language models.

[1] Booch, Grady, et al. "Thinking fast and slow in AI." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 17. 2021.

Q5: Position of Related Works in the Paper

We appreciate your feedback regarding the placement of related works. To improve clarity, we will reorganize the structure of the paper. Specifically:

• Revised structure: We will simplify the presentation of the main text to enhance readability.
• Related works: We will introduce related work briefly in the main text, ensuring it is more integrated and accessible.

We hope these responses address your concerns and provide further clarity. Thank you once again for your constructive feedback and valuable suggestions, which have been instrumental in improving our work.

A2: Additional results of new tasks and models

To further support our findings, we conducted additional experiments on new tasks (GPQA, IFEval, MedQA) and a new model series (OpenELM). The results validate the generalizability of our conclusions, as outlined below:

• Table 2: Results of $CoF_{\text{lower}}$  on Additional Domain Tasks
  • The results indicate that  $CoF_{\text{lower}}$  is consistently below 20% across various methods, tasks, and model combinations. Furthermore, we observe a decreasing trend in  $CoF_{\text{lower}}$  as the ratio of model parameters decreases.
  • We also found that the collaboration rate of general models on domain tasks is slightly higher than that on general tasks.

• Table 3: Line Fitting Results of OpenELM Models
  • Given the following formula of scale ratio law $CoF_{\text{lower}}=\gamma \cdot {R}^{-\alpha}+\beta$where $R=\frac{N_l}{N_s}$, we compute the coefficients with model parameters and collaboration frequency.
  • This table presents the line fitting results for oracle decoding, including fitting error, fitting coefficients, and the final  x (i.e., $R^{-\alpha}$) and  y  values on the fitting curve.
  • The results demonstrate a strong fitting effect, confirming the generalizability of our findings across different model families.
  • These results further indicate that the performance of collaborative decoding is influenced by the underlying model’s performance.

Q3: Discussion of the Figures (Results of Various Methods on Benchmarks)

A1: Connections between Figures and Conclusions

We provide detailed connections between the results of various methods on benchmarks presented in the figures and our final findings. Specifically, we conducted experiments on three methods (speculative decoding, contrastive decoding, proxy tuning) across three benchmarks (GSM8k, MMLU, MBPP) and two model series (Qwen, Pythia) to derive our four main findings.

The results for these combinations sufficiently support each finding, as outlined below:

• Finding 1: 20% Collaborations — The 2:8 Law
  • Figures 2 (Qwen) and 3 (Pythia): These figures show that the average collaboration frequency (interface rate) across different tasks is consistently less than 20%. Among the methods, contrastive decoding achieves much lower collaboration frequencies than speculative decoding and proxy tuning.
  • Task Variability: While collaboration frequency varies across tasks and model series due to differing model capabilities, the values remain approximately 20%.
  • Model Combinations: Collaboration frequency decreases as the parameter ratio between large and small models decreases. This implies that models with similar capabilities require less frequent collaboration, which also contributes to Finding 2.
• Finding 2: Parameters Scale Ratio Law
  • Figure 4 (Qwen): This figure presents the fitting line for the parameter scale ratio between large and small models, showing a strong correlation with lower collaboration frequencies. The fitting effect demonstrates the validity of our parameter scale ratio law.
  • Task and Method Variability: While the quality of the fitting line varies by task and method, the conclusions remain consistent.
  • Figure 5 (Pythia) highlights how the fitting effect is influenced by model underperformance. To strengthen this finding, we provide additional results from OpenELM, which exhibit low fitting errors and further support the effectiveness of our scale ratio law.
• Finding 3: Well Begun is Half Done
  • Figures 6 (Qwen) and 7 (Pythia): These figures illustrate the mismatch rate across the generated sequence. They reveal that most mismatches occur at the beginning of the sequence. As generation progresses, SLMs increasingly align with mix-scaled models, leveraging the shared context.
  • Task Variability: The percentage of mismatches in a sequence varies by task, influenced by task difficulty and model capability.
• Finding 4: Last in Uncertain Tokens
  • Figure 10: This figure compares the logits of each token generated by SLMs to those of mix-scaled models, showing that mismatches predominantly occur in high-entropy positions (indicating high uncertainty).
  • Figures 11, 12, 13: These figures analyze the top- token logits at each step, clustering tokens into match and mismatch categories. The results reveal a strong correlation between uncertainty (high-entropy positions) and matching labels.
  • To ensure robustness, we compute average correlation scores across all tasks and methods. This finding identifies key positions for collaboration during decoding in SLMs, contributing to performance-cost optimization.

Q5: Greedy Decoding and Next-Token Perplexity for Collaboration Measuring

A1: Overview of Different Metrics

This is an excellent question, and we appreciate the opportunity to address it. Below, we analyze the relationship between different metrics and provide comparative results. At each decoding step, we obtain logits over the vocabulary from the SLMs. These logits are normalized into the range $[0, 1]$ using the softmax function. This gives the next-token probabilities: $P(y_t) = \text{softmax}(\text{logits}_t)$ During greedy decoding, the next token $y_t$ is selected as the one with the highest probability: $y_t = \arg \max_i^V P(y_t^i)$ where $V$ is the size of the vocabulary.

Below we will provide an analysis of the next token entropy and perplexity:

• Next token entropy, $H(y_t)$, is defined as $H(y_t) = -\sum_{i=1}^V P(y_t^i) \log P(y_t^i)$, which quantifies the uncertainty of the model’s prediction at step t. Lower entropy indicates more confident predictions, while higher entropy suggests greater uncertainty in selecting the next token.
• Perplexity, when based on cross-entropy, requires access to the golden tokens (ground truth sequence). Since our routing analysis does not have access to golden tokens, we instead rely directly on entropy as a proxy for measuring uncertainty.
  • Next token perplexity is defined as $\text{Perplexity}(y_t) = 2^{H(y_t)}$, which transforms entropy into an interpretable measure representing the average branching factor of the model’s distribution over the next token. A lower perplexity implies a narrower, more confident distribution.
  • Sequence perplexity, on the other hand, measures the uncertainty over the entire sequence and is defined as $\text{Perplexity(sequence)} = 2^{-\frac{1}{T} \sum_{t=1}^T \log P(y_t)}$, where T is the length of the sequence. Sequence perplexity can be seen as the geometric mean of next token perplexities across all decoding steps.

The relationship among these metrics is intrinsic: next token logits determine next token entropy and perplexity through their normalized probabilities. Higher entropy and perplexity indicate a more uniform distribution, while lower values suggest peaked distributions. Sequence perplexity aggregates these effects over all steps, offering a global view of the model’s predictive confidence across the sequence.

Q4: Additional Illustration in Figures 11, 12, and 13

In Figures 11, 12, and 13, we visualize the logits of the top 1 and top 5 tokens in the vocabulary of small models at each generation step. These logits are categorized into two distinct clusters:

• Matched tokens: Tokens where the small model’s predictions align with those of the mix-scale model.
• Mismatched tokens: Tokens where the small model’s predictions diverge from those of the mix-scale model.

The visualization highlights that these clusters are separable, which supports our conclusion that dynamic routing can be implemented. Specifically, the uncertainty of token decoding in SLMs can guide the decision of whether to engage collaboration with larger models on a token-by-token basis. We utilize the following metrics to evaluate the correlation between matched and mismatched token logits:

• Silhouette Coefficient (SC)

  • This metric (range: -1 to 1) assesses clustering quality by comparing intra-cluster cohesion and inter-cluster separation. Values > 0.5 indicate strong clustering performance.
  • A high SC value derived from Pearson or Spearman correlation demonstrates that the metric aligns well with the data.

• Davies-Bouldin Index (DBI)

  • The DBI (range: $[0, ∞)$) measures clustering compactness and separation, where lower values (<1) suggest better clustering quality.
  • A low DBI derived from correlation methods indicates effective uncertainty estimation.

• Mean Cluster Center Distance (MCCD)

  • MCCD measures the separation between cluster centers, with larger values indicating better distinction. Correlation methods that amplify these distances demonstrate their alignment with the data.

• Table 3: Correlation Between Match/Mismatch Tokens and Top-K Token Logits of SLMs

  • Our results demonstrate the effectiveness of uncertainty estimation:
    • SC values are consistently close to 0.5.
    • DBI values are below 1, indicating compact and well-separated clusters.
    • MCCD values range between 10–20, reflecting robust inter-cluster distinction.
  • An exception is observed with Pythia series models, likely due to their insufficient pretraining.

Q2: Discussion of the System 1 & System 2 Analogy

In this work, we draw inspiration from the analogy of System 1 and System 2, simplifying their collaboration into Fast and Slow thinking. System 1 efficiently handles approximately 95% of routine tasks, while System 2 is reserved for deliberately addressing the remaining 5% of complex work [1]. Together, they demonstrate the power of high-efficiency collaboration.

We adopt this high-efficiency motivation to model the collaborative decoding methods between fast and slow (or small and large) models. Our experimental findings (Findings 1 and 2) show that small, fast models generate approximately 80% of tokens during the answering process, while large, slow models contribute the remaining 20%.

Looking forward, we aim to expand these collaborative mechanisms to reasoner and knowledger models, such as OpenAI’s o1 and GPT-4, which not only embody the fast/slow model paradigm but also represent intuitive and deliberate thinking. Preliminary experiments reinforce our findings, indicating successful collaboration between o1 and existing large language models.

[1] Booch, Grady, et al. "Thinking fast and slow in AI." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 17. 2021.

Q3: Response to Comments about Writing

In Section 2.2, we use  $O_g$  to represent the golden outputs generated by mix-scaled language models. We agree that the suggestion to use  $O_f$  would provide better consistency with the fused logits. We will update this notation to enhance clarity and improve the reader’s understanding. Additionally, we appreciate you pointing out the typographical errors, and we will correct them in the updated manuscript.

A2: Preliminary Executable Results

We present preliminary results for CD and PT, demonstrating their potential to optimize speed-performance trade-offs. Specifically, instead of conducting CD and PT on all tokens during text generation by small models, we focus these collaborations on a subset of mismatch tokens. By collaborating on these uncertain tokens alone, we achieve performance comparable to previous approaches that rely on collaborations for all tokens.

• Table 1: Routing with Top-1 Token Logits of SLM for Contrastive Decoding
  • At each decoding step of the SLM, we determine whether to involve LLM collaboration based on the top-1 token logits of the SLM. A routing ratio of 0.0% implies decoding exclusively with the SLM, while a ratio of 100% indicates collaboration between the SLM and LLM at all steps.
  • Our results show that we can significantly reduce inference cost while maintaining comparable performance. Interestingly, when the performance gap between the SLM and LLM is small (e.g., 4B vs. 7B models), the performance improvement becomes less pronounced.

• Table 2: Routing with Top-1 Token Logits of SLM for Proxy Tuning
  • The results for Proxy Tuning exhibit a similar trend to Contrastive Decoding. Here, a routing ratio of 0.0% denotes generation exclusively using small tuned models, while a ratio of 100% indicates generation involving both small tuned models and large base models.

Additionally, these findings align with recent advancements in test-time compute scaling applications. The token-level uncertainty analysis in our work can also be applied to entropy-based decoding methods like Entropix, where high-entropy tokens can be handled similarly to uncertain tokens in small language models.

In our experiments, we used a simple routing mechanism based on the top-1 token logits threshold. However, this can be extended by training a more sophisticated router with richer features during decoding, which has the potential to further improve the efficiency and effectiveness of collaborative decoding.

We sincerely appreciate your positive feedback and the time and effort you have dedicated to reviewing our paper. Below, we provide further illustration and additional results to address your questions.

Q1: Executable Insights for Contrastive Decoding and Proxy Tuning

A1: Overview of Executable Framework

Building on the insights from our findings, we propose a direct approach to optimizing the inference cost for both Contrastive Decoding (CD) and Proxy Tuning (PT). Previous work on CD and PT typically involves collaboration across all tokens during text generation. However, our results suggest that this is unnecessary, as efficient collaboration can be achieved by focusing only on specific tokens.

In this optimized framework:

• Small models serve as the main backbone in CD and PT. They are tasked with generating the majority of the content during text generation.
• Token-level collaboration is determined dynamically based on the logits distribution. Specifically, we identify whether a token requires collaboration from large models by analyzing the features of the match and mismatch logits between small models and mixed-scale models. To implement this, we can train a lightweight token-level router that leverages these logits features. The router determines when collaboration with a larger model is necessary, effectively balancing performance and efficiency.

Q2: Experiments on Domain-Specific Datasets

To validate the robustness of our findings, we conducted additional experiments on GPQA, MedQA, and IFEval, which encompass biology, medical, and physics question-answering tasks, as well as instruction-following tasks in open-domain.

• Table 3: Results of $CoF_{\text{lower}}$  on Additional Domain Tasks
  • The results indicate that  $CoF_{\text{lower}}$  is consistently below 20% across various methods, tasks, and model combinations. Furthermore, we observe a decreasing trend in  $CoF_{\text{lower}}$  as the ratio of model parameters decreases.
  • We also found that the collaboration rate of general models on domain tasks is slightly higher than that on general tasks.

When extending model collaborations from generalist to specialist tasks, we anticipate that the collaboration frequency will decrease due to the narrower distribution of domain-specific terminology. However, the lack of a comprehensive range of specialized model series limits further analysis at this stage, and we leave this exploration as future work.

Q3: Different Sampling Techniques

In our current work, we use greedy decoding to compute the matching rate of tokens between small and large language models. This choice aligns with our initial motivation of achieving collaborative decoding with minimal intervention in small models, treating the collaborative decoding results as golden tokens.

For scenarios where exact matching is less critical and the focus shifts to performance-speed optimization, other sampling techniques can be explored. These techniques might yield better performance with reduced collaboration frequency, leading to more efficient collaborations. However, quantifying results becomes more challenging due to the increased uncertainty introduced by sampling. We believe this is an exciting direction for future research, as it opens up possibilities for balancing efficiency and performance through alternative decoding strategies.

We sincerely thank you for your positive feedback and valuable suggestions. Below, we provide our thoughts and new results addressing your questions.

Q1: Discussion of Practical Applications

The primary motivation behind collaborative decoding between large and small language models is to optimize the speed-performance trade-off. Previous works, such as speculative decoding, have demonstrated the effectiveness of reducing inference time. Our work generalizes this collaboration to broader methods, including Contrastive Decoding (CD) and Proxy Tuning (PT).

Here, we present some preliminary results for CD and PT, demonstrating their potential to optimize speed-performance trade-offs. Specifically, instead of conducting CD and PT on all tokens during text generation by small models, we focus these collaborations on a subset of mismatch tokens compared to mix-scaled models. By collaborating on these uncertain tokens alone, we achieve performance comparable to previous approaches that rely on collaborations for all tokens.

• Table 1: Routing with Top-1 Token Logits of SLM for Contrastive Decoding
  • At each decoding step of the SLM, we determine whether to involve LLM collaboration based on the top-1 token logits of the SLM. A routing ratio of 0.0% implies decoding exclusively with the SLM, while a ratio of 100% indicates collaboration between the SLM and LLM at all steps.
  • Our results show that we can significantly reduce inference cost while maintaining comparable performance. Interestingly, when the performance gap between the SLM and LLM is small (e.g., 4B vs. 7B models), the performance improvement becomes less pronounced.

• Table 2: Routing with Top-1 Token Logits of SLM for Proxy Tuning
  • The results for Proxy Tuning exhibit a similar trend to Contrastive Decoding. Here, a routing ratio of 0.0% denotes generation exclusively using small tuned models, while a ratio of 100% indicates generation involving both small tuned models and large base models.

Additionally, these findings align with recent advancements in test-time compute scaling applications. The token-level uncertainty analysis in our work can also be applied to entropy-based decoding methods like Entropix [1], where high-entropy tokens can be handled similarly to uncertain tokens in small language models. In the above experiments, we used a simple routing mechanism based on the top-1 token logits threshold. However, this can be extended by training a more sophisticated router with richer features during decoding, which has the potential to further improve the efficiency and effectiveness of collaborative decoding.

[1] https://github.com/xjdr-alt/entropix

Q3: Quantitative Metrics for Uncertainty Analysis

To strengthen the evidence supporting our uncertainty analysis, we provide additional quantitative results, generalizing across all model combinations and methods. We utilize the following metrics to evaluate the correlation between matched and mismatched token logits:

• Silhouette Coefficient (SC)

  • This metric (range: -1 to 1) assesses clustering quality by comparing intra-cluster cohesion and inter-cluster separation. Values > 0.5 indicate strong clustering performance.
  • A high SC value derived from Pearson or Spearman correlation demonstrates that the metric aligns well with the data.

• Davies-Bouldin Index (DBI)

  • The DBI (range: $[0, ∞)$) measures clustering compactness and separation, where lower values (<1) suggest better clustering quality.
  • A low DBI derived from correlation methods indicates effective uncertainty estimation.

• Mean Cluster Center Distance (MCCD)

  • MCCD measures the separation between cluster centers, with larger values indicating better distinction. Correlation methods that amplify these distances demonstrate their alignment with the data.

• Table 2: Correlation Between Match/Mismatch Tokens and Top-K Token Logits of SLMs

  • Our results demonstrate the effectiveness of uncertainty estimation:
    • SC values are consistently close to 0.5.
    • DBI values are below 1, indicating compact and well-separated clusters.
    • MCCD values range between 10–20, reflecting robust inter-cluster distinction.
  • An exception is observed with Pythia series models, likely due to their insufficient pretraining.

Q4: Position of Related Works in the Paper

We appreciate your feedback regarding the placement of related works. To improve clarity, we will reorganize the structure of the paper. Specifically:

• Revised structure: We will simplify the presentation of the main text to enhance readability.
• Related works: We will introduce related work briefly in the main text, ensuring it is more integrated and accessible.

We hope these responses address your concerns and provide further clarity. Thank you once again for your constructive feedback and valuable suggestions, which have been instrumental in improving our work.

We sincerely appreciate your positive feedback, along with the time and effort you have put into reviewing our paper. We would like to give our thoughts and new results to address your concerns.

Q1: Relationship Between Main Contributions and the System 1 & System 2 Analogy

In this work, we draw inspiration from the analogy of System 1 and System 2, simplifying their collaboration into Fast and Slow thinking. System 1 efficiently handles approximately 95% of routine tasks, while System 2 is reserved for deliberately addressing the remaining 5% of complex work [1]. Together, they demonstrate the power of high-efficiency collaboration.

We adopt this high-efficiency motivation to model the collaborative decoding methods between fast and slow (or small and large) models. Our experimental findings (Findings 1 and 2) show that small, fast models generate approximately 80% of tokens during the answering process, while large, slow models contribute the remaining 20%.

Looking forward, we aim to expand these collaborative mechanisms to reasoner and knowledger models, such as OpenAI’s o1 and GPT-4, which not only embody the fast/slow model paradigm but also represent intuitive and deliberate thinking. Preliminary experiments reinforce our findings, indicating successful collaboration between o1 and existing large language models.

[1] Booch, Grady, et al. "Thinking fast and slow in AI." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 17. 2021.

Q2: Line Fitting on the Parameter Ratio Scaling

The line fitting in Figures 4 and 5 is influenced by both data size and model performance:

• Data Size: Due to computational limitations, we sampled only ~500 data points for each task. This sampling constraint may contribute to fluctuations in the observed curve.
• Model Performance: Parameter ratio scaling laws are significantly affected by model performance. While Qwen series models maintain consistent performance, Pythia models underperform due to insufficient pretraining, thereby affecting the collaboration dynamics between large and small models.

To further validate our findings, we conducted additional experiments on OpenELM models [2], which exhibit better performance compared to Pythia.

• Table 1: Line Fitting Results of OpenELM Models
  • Given the following formula of scale ratio law $CoF_{\text{lower}}=\gamma \cdot {R}^{-\alpha}+\beta$where $R=\frac{N_l}{N_s}$, we compute the coefficients with model parameters and collaboration frequency.
  • This table presents the line fitting results for oracle decoding, including fitting error, fitting coefficients, and the final x (i.e., $R^{-\alpha}$)  and  y  values on the fitting curve.
  • The results demonstrate a strong fitting effect, confirming the generalizability of our findings across different model families.
  • These results further indicate that the performance of collaborative decoding is influenced by the underlying model’s performance.

[2] Mehta, Sachin, et al. "Openelm: An efficient language model family with open-source training and inference framework." arXiv e-prints (2024): arXiv-2404.