On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion

Q1: In the multi-task setting, which 7B model is used as the expert to implement your algorithm on the 13B model? I am very confused. If you are using different sets of experts to operate on the 13B model, isn't this weak-to-strong? Because the parameters of multiple 7B models exceed 13B

In the multi-task setting, we used all the experts for the seen tasks, i.e., four experts fine-tuned on each task (7B or 1.1B).

It needs to be clarified that our "weak to strong" approach aims to use weak model supervision to elicit the full capabilities of a much stronger model. In our setting, we use small fine-tuned models (weak models, e.g., 7B or 1.1B) that have transferred downstream task-specific knowledge to enhance a large model (strong model, e.g., 13B) without downstream knowledge. In our setting, "weak" or "strong" indicates the capability upper bound of a model, which is commonly enhanced by scaling model size. However, the capabilities of multiple small models are constrained by their model size and are generally weaker than the large ones, as shown by the performance of the 7B multi-task expert on unseen tasks in Table 2 (35.82<51.25).

Additionally, fine-tuning a large model is significantly more expensive than fine-tuning a small model, requiring more advanced hardware and more training time. In contrast, our weak-to-strong paradigm only needs to fine-tune the small models.

Q2: There seems to be a typo in formula (9) in Appendix B, it seems to be $\propto$ rather than $=$.

Thanks for your valuable advice. We will carefully revise our paper based on your suggestion.

Q3:In Section 3.2, the proposed method is based on an assumption that lacks clear supporting evidence. Does this somewhat undermine the theoretical foundation of the algorithm

Our work is based on verified theories [1, 2, 3]. First, using the shift of the fine-tuned model to accomplish our domain adaptation task is reasonable. To demonstrate our optimization process, we can view the fine-tuning procedure as reinforcement learning (RL) with a KL-divergence constraint preventing divergence from a reference model.

According to the theory presented in the DPO[1]: Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization $r(x, y) = \beta \log \frac{\pi(y|x)}{\pi_{ref}(y|x)}$ for some model $\pi(y|x)$ and a given reference model $\pi_{ref}(y|x)$.

Meanwhile, According to [1,2] the optimal solution to the KL-constrained reward maximization objective is given by:

Combining the above theory, we can derive the following equation:

$\pi_{r}(y|x)=\frac{1}{Z(x)}\pi_{ref}(y|x)exp(\log \frac{\pi_r(y|x)}{\pi_{ref}(y|x)})$

Since any language model can be viewed as the solution to KL-constrained RL with a constraint to the pre-trained model[1], this equation is applicable to fine-tuning scenarios. We can replace $\pi$ in the parentheses with the small model's $\pi$, resulting in the following equation:

$\pi_{L-ft}(y|x)=\frac{1}{Z(x)}\pi_{L-pt}(y|x)exp(\log \frac{\pi_{S-ft}(y|x)}{\pi_{S-pt}(y|x)})$

It can be seen that, based on the theory from previous work, it is reasonable to assume that the shifts between models are consistent for knowledge transfer. This is consistent with the form of the proof in Appendix B.

Compared to global static transfer, we adjust an appropriate shift at each decoding step to achieve better transfer results. KL divergence[1, 2, 3] is commonly used to describe the distance between distributions (as shown in section 3.1), making it more suitable for representing the shift between two distributions. We use KL divergence as a distance function to represent the above shift, converting it into a KL-constrained problem, dynamically controlling the knowledge transfer by constraining each decoding step. Meanwhile, the squared error is commonly used in various regression prediction approximations[3], and it is easy to solve, making it well-suited for our setup. Additionally, as shown in the "Supplementary Proof for the Fusion of Multiple SLMs Scenario" in the Global Rebuttal, due to the geometric properties and inequality characteristics of squared error, our method can extend more smoothly to scenarios involving multiple experts when using squared error.

• [1] Direct preference optimization: Your language model is secretly a reward model (NIPS2023)
• [2] RL with KL penalties is better viewed as Bayesian inference (EMNLP2022)
• [3] Learning Theory for Distribution Regression (JMLR2016)

Q4:In the multi-task experiments, the results for CNN/DM and unseen tasks do not show significant improvement

Actually, in our multi-task experiments, our method achieved a 16% improvement on CNN/DM compared to the 13B model (8.94->10.52). It is noticeable that after using multi-task tuning on the 13B model, the performance on unseen tasks even decreased (51.28->50.58). This indicates that unseen tasks are more challenging and that overfitting can occur when training on seen tasks. In contrast, our method shows an improvement on unseen tasks (51.28->51.31), demonstrating that our approach not only provides significant enhancements on seen tasks but also helps mitigate overfitting within the domain.

Q5: How would the evaluation results change if a 5-shot experiment were conducted?

Actually, we have conducted 5-shot experiments. As mentioned in Section 5.3, our method can be combined with in-context learning(ICL) and can also enhance its effect. We used the 5-shot approach as the ICL setting. As shown in Figure 5(a), our method combined with 5-shot ICL achieves an overall improvement of 18.3% compared to using ICL alone. This is mainly due to our method's ability to integrate the knowledge possessed by the experts.

Dear Reviewer Ttwe:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.

Q1:Using the squared error between two KL's is not theoretically motivated (at least that I am aware of)

The goal of our motivation is to enhance the constraints using KL divergence, aiming for the shift of the fine-tuned large model to be equal to the shift of the fine-tuned small models in each decoding step. Compared to global static transfer, we adjust an appropriate shift at each decoding step to achieve better transfer results. We use KL divergence as a distance function to measure this "shift," (as shown in section 3.1) and the squared error between KL divergences helps us align these two shifts. Squared error is commonly used in various regression prediction approximations[1], and it is easy to solve, making it well-suited for our setting. Additionally, as shown in the "Supplementary Proof for the Fusion of Multiple SLMs Scenario" in the Global Rebuttal, due to the geometric properties and inequality characteristics of squared error, our method can extend more smoothly to scenarios involving multiple experts when using squared error.

• [1] Learning Theory for Distribution Regression (JMLR2016)

Q2:More description of the optimization method in main text since it is a big part of the method.

Thank you for your reminder. We will elaborate on this section in the next version. Our optimized method performs a linear search for $\alpha$ and multiple Logi Arithmetic operations after obtaining the logits from all models to find the optimal situation described in Eq(6). During the search, we start from 0 and increment by 0.1 each time until we reach 2, resulting in 20 searches. We perform this search at each decoding step, so, as shown in Figures 3 and 4, the $\alpha$ varies for each decoding step.

In the multi-task setting, directly searching for every expert results in exponential complexity. For practical use, we accelerate this process by using only one small expert at each decoding step, thereby reducing the exponential complexity of the search process to linear complexity. Experiments have shown that this approach also yields good results (as shown in Table 2).

Q3: Where is the baseline proxy tuning for multi-task tuning in Table 2?

It is worth noting that Proxy Tuning, due to its lack of prior assumptions in a multi-task setting and the difficulty in presetting the transfer proportions for multiple experts, is not capable of handling multi-task scenarios. In contrast, our method can dynamically adjust the transfer proportions of expert knowledge, making it naturally suitable for multi-task settings.

To better demonstrate the effectiveness of our method, we further compared it with our method with static $\alpha$ in a multi-task setting. For the 4 seen tasks in our experiment, we set the corresponding expert coefficient to 0.25 ($\alpha=1/4$, assigning the same proportion to each seen task expert). In the table below, it can be seen that our method, which dynamically adjusts the coefficients, significantly outperforms the static setting.

Q4:For the efficiency analysis, the BV can be done in 1 forward pass all in parallel, but the n parameter searches require 20 forward passes. It really is n times more compute, which is not minimal. For example, decoding 100 tokens vs decoding 1 token would be a constant in the BigO, but they are different efficiency wise.

Actually, our method only performs one forward pass when doing logit arithmetic. As analyzed in the "Complementary to Efficiency Analysis" section of the Global Rebuttal, the $nBV$ term represents $n$ quick logit arithmetic operations to obtain the final logits, which only requires one forward pass and not $n$ forward passes. So overall, the time consumption is almost the same as that of the static method.

Q5:In eq (2), is the second term (multiplied by alpha) not normalized but in logit space?

The second term is not normalized. Normalization is performed after the entire logit arithmetic calculation is completed.

Q6:In fig 4, are there some special about the tokens being generated at the timesteps where alpha is high?

When $\alpha$ is high, the confidence in the logits generated by a specific expert will be higher, leaning towards the tokens that this expert is more certain about it at the moment. For example, for gsm8k, a high $\alpha$ will tend to generate mathematical symbols.

For the following question: {"question": "A pen costs as much as a pencil and eraser combined. A pencil costs \$1.20 and an eraser costs \\$ 0.30. How much will 8 pens cost?"}

The answers obtained from our method are as follows (bold indicates $\alpha$ is the upper bound, and red indicates $\alpha$ to the lower bound):

{Ours: " 8 pencils will cost 8 * $1.20 = $<<8*1.2=9.60>>9.60. 8 erasers will cost 8 * $0.30 = $<<8*0.30=2.40>>2.40. Thus, 8 pens will cost $9.60 + $2.40 = $<<9.6+2.4=12>>12."}

As can be seen, when $\alpha$ is at the upper bound, the response leans more towards mathematical reasoning; when $\alpha$ is at the lower bound, the response tends to be more of a normal statement or information about the question.

Q1: It is not intuitively obvious to me why matching the KL divergence is the right objective. Could the authors please provide some intuition? I imagine it is something like this: when the small model updates significantly for some token, we want the large model to also udpate significantly. That seems reasonable, but probably doesn't always work well: if the small model is unaware of some fact or makes an arithmetic mistake, it may need to update significantly on the corresponding tokens, while a large model does not need to update.

Our method has already considered this situation, which presents a challenge for static methods. Our goal is to control the shift of the fine-tuned model to be the same, and KL divergence is commonly used to describe the distance between distributions, making it more suitable for representing the shift between two distributions.

Using a static shift to match on a sentence level can indeed result in the incorrect transfer of some erroneous knowledge. Therefore, our method refines this process to each decoding step, allowing dynamic adjustment of knowledge transfer intensity to mitigate this issue. Notably, our method ultimately overlays the transferred knowledge onto the logits of the large model. This means that when the small model generates errors, our method dynamically adjusts based on the large model's capabilities. When the large model can solve the problem independently, it will retain more of its own abilities.

As shown in the results of Table 1, our method significantly improves the model's performance even when the effect of the 1.1B finetuned model is much lower than that of the 13B base model. For instance, on MMLU, the 1.1B model improves from 37.26 to 48.32. Compared to the static method's improvement from 37.26 to 39.88, our method does not fully trust the capabilities of the small model but rather retains more of the large model's abilities to mitigate this issue.

Q2: How can we compute KL between this $Q_1...Q_T$ and $Q$?

The term "joint distribution" in our paper refers to the fusion distribution obtained by combining the outputs of a series of smaller models.

As shown in the "Global Rebuttal" section "Supplementary Proof for the Fusion of Multiple SLMs Scenario," we transform the problem of approximating $J$ into a centroid problem (i.e., optimizing the upper bound of $KL(J||Q)$). Therefore, we can use equation (6) in the paper to calculate this KL divergence.

Q3: The proposed method is 2.5 times slower at inference time compared to standard sampling from the same model.

It should be noted that the 2.5x slower inference speed is compared to the 13B FFT (full fine-tuning). However, our method does not require fine-tuning the large model (e.g., 13B), allowing it to benefit from smaller expert models, resulting in much lower hardware requirements. As shown in Table 3, the time required for 13B FFT is 1176s, while the time required for 7B FFT or 1.1B FFT is only 588s or 128s, significantly less than the time required for 13B FFT. Additionally, our method can leverage many pre-existing small expert models from Huggingface, further reducing training time.

Furthermore, as noted in the "Complementary to Efficiency Analysis" section of our Global Rebuttal, the time consumed by our method is almost identical to that of the static method, while our method performs significantly better.

Q4: Supervised instruction tuning details.

For each task, we used the official dataset and trained our small model on the official training set. We conducted supervised instruction tuning without using chain-of-thoughts. When constructing the prompt, we used simple instructions for concatenation. For example, for gsm8k: "Question: " + [question] + "\nAnswer:". For CNN/DM: [article] + "\n\nSummarize the above article:".

Q5:In Figure 3, what do the tokens where we set the weight to the lower bound correspond to, and what about the upper bound? Are they qualitatively different?

When $\alpha$ is high, the confidence in the logits generated by a specific expert will be higher, leaning towards the tokens that this expert is more certain about at the moment. For example, for gsm8k, a high $\alpha$ will tend to generate mathematical symbols.

For the following question: {"question": "A pen costs as much as a pencil and eraser combined. A pencil costs \$1.20 and an eraser costs \\$ 0.30. How much will 8 pens cost?"}

The answers obtained from our method are as follows (bold indicates $\alpha$ is the upper bound, and red indicates $\alpha$ to the lower bound):

{Ours: " 8 pencils will cost 8 * $1.20 = $<<8*1.2=9.60>>9.60. 8 erasers will cost 8 * $0.30 = $<<8*0.30=2.40>>2.40. Thus, 8 pens will cost $9.60 + $2.40 = $<<9.6+2.4=12>>12."}

As can be seen, when $\alpha$ is at the upper bound, the response leans more towards mathematical reasoning; when $\alpha$ is at the lower bound, the response tends to be more of a normal statement or information about the question.

Q6:In Figure 3, why is the lower bound 0.8 and the upper bound 1.5? Are these tunable parameters? 0.8 seems quite high, shouldn't we want to set the lower bound to 0?

Sorry for the confusion, Figure 3 can indeed be misleading. In our experiments, $\alpha$ was searched from 0 to 2.0. The values 0.8 and 1.5 in Figure 3 represent the minimum and maximum values obtained during the optimization process. In the GSM8K task, due to the large model's inherent capability bias, the overall trust in the expert knowledge is relatively high, resulting in higher values obtained during the optimization process. We will improve the depiction of the figure in the next version.

Dear Reviewer YRcj:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.

Q1:Solving the optimization problem at every decoding step is expensive?

As we analyzed in the "Complementary to Efficiency Analysis" section of the Global Rebuttal, our method only adds the term "$nBV$" compared to the static method. Optimizing $n$ times ($n \le 20$) during each forward pass is negligible compared to the overall forward pass time. In the experiments, as shown in Table 3, our method is only 0.008s slower per data point on average compared to the static method.

Q2: A finetuned model with LoRA on the large model is cheaper?

SFT with LoRA still requires forwarding the full large model thousands of times (e.g., 13B) to tune LoRA during training. In contrast, our method does not require fine-tuning the large model at all, benefiting from transferring from smaller models (7B or 1.1B) or using pre-existing models (e.g., from Hugging Face). As shown in Table 3, the time for LoRA tuning a 13B model is 836 seconds, while the time for fully fine-tuning a 7B or 1.1B model is 588s or 128s, respectively, which is significantly less than the time required to fine-tune the 13B model. LoRA tuning on the smaller models further reduces the time to 364s (7B) and 128s (1.1B). This demonstrates that fine-tuning smaller models requires less hardware resources and less time. Furthermore, utilizing pre-trained smaller models from Hugging Face allows efficient transfer without extensive training. Therefore, our method is cheaper compared to fine-tune a large model.

Q3:Updating $\alpha$ every 100 tokens is more efficient?

As we analyzed in the "Complementary to Efficiency Analysis" section of the Global Rebuttal, our method and the static method have almost the same time consumption per data point on average. As shown in the below table, reducing the update frequency of $\alpha$ may negatively impact the final result. Therefore, we ultimately chose to optimize $\alpha$ at each step.

Q4: Not enough clear to me that the method does much better than the static $\alpha$: when I see the barplot of figure 2, it seems like $\alpha=1$ is a bit below than the learnt but the gap is not huge.

Actually, our method outperforms the static setting of $\alpha=1.0$ on single tasks, with improvements of 4.4%, 0.9%, 8.1%, 6.5%, and 1.6% on GSM8K, TruthfulQA, TriviaQA, CNN/DM, and MMLU, respectively. Our method has a significant advantage over the static method, with an average improvement of 4.3%.

To better demonstrate the effectiveness of our method, we further compared it with our method with static $\alpha$ in a multi-task setting. For the 4 seen tasks in our experiment, we set the corresponding expert coefficient to 0.25 ($\alpha=1/4$, assigning the same proportion to each seen task expert). In the table below, it can be seen that our method, which dynamically adjusts the coefficients, significantly outperforms the static setting.

Q5: The theoretical justification for the optimization problem?: It's not clear to me why it should work in standard finetuning case?

To demonstrate our optimization process, we can view the fine-tuning procedure as reinforcement learning (RL) with a KL-divergence constraint preventing divergence from a reference model.

According to the theory presented in the DPO[1]: Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization $r(x, y) = \beta \log \frac{\pi(y|x)}{\pi_{ref}(y|x)}$ for some model $\pi(y|x)$ and a given reference model $\pi_{ref}(y|x)$.

Meanwhile, According to [1,2] the optimal solution to the KL-constrained reward maximization objective is given by:

Combining the above theory, we can derive the following equation:

$\pi_{r}(y|x)=\frac{1}{Z(x)}\pi_{ref}(y|x)exp(\log \frac{\pi_r(y|x)}{\pi_{ref}(y|x)})$

Since any language model can be viewed as the solution to KL-constrained RL with a constraint to the pre-trained model[1], this equation is applicable to fine-tuning scenarios. We can replace $\pi$ in the parentheses with the small model's $\pi$, resulting in the following equation:

$\pi_{L-ft}(y|x)=\frac{1}{Z(x)}\pi_{L-pt}(y|x)exp(\log \frac{\pi_{S-ft}(y|x)}{\pi_{S-pt}(y|x)})$

• [1] Direct preference optimization: Your language model is secretly a reward model (NIPS2023)
• [2] RL with KL penalties is better viewed as Bayesian inference (EMNLP2022)

Q6 Scaling experiments for studying the approach

Actually, we have chosen TinyLlama1.1B in our experiments. In the table below, we show the improvements of our model compared to the static method on 1.1B and 7B models in single-task experiments. The numbers in the table represent the results of the static method, with the relative improvements of our method shown in parentheses. It can be noted that when the gap between the weak model and the strong model is larger, our method indeed better adjusts the capability of expert knowledge transfer.

Dear Reviewer XMhg:

We wish to thank you again for your constructive feedback which has helped us to improve the clarity and contribution of our work. As the discussion period draws to a close, we hope our response has effectively addressed all your concerns. Your insights are invaluable to us, and we remain open to further discussion if you have any questions regarding our response.

I thank the reviewers for their rebuttal and I appreciated their theoretical derivation. I believe that this important piece to be added to the paper. I hope they will do it. i increase my score by one point.