Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

Thank you for your review. We address your suggestions below.

Selecting Largest Model. Thank you for your suggestion. We agree that selecting a larger LLM may yield better generalization. However, smaller LMs are easy to fine-tune on in-domain data which can provide complementary knowledge (e.g., medicine data). In such cases, model fusion yields better results. To demonstrate this, we compare PackLLM with selecting the largest model from the set of models (and DExperts when applicable) on three different tasks; Medicine, MMLU-STEM, and language modeling. The results show that PackLLM yields better performance than selecting the largest model in all settings. Full results can be found in the tables below.

Setup: 3 LLMs (LLaMA2-13B, MedLLaMA2-7B, LLaMA2-7B) on 8 Medicine tasks:

Setup: 4 LLMs (FuseLLM-7B, Gemma-2B, phi2-2.7B, Qwen-1.8B) on 5 MMLU-STEM tasks.

Setup: 5 LLMs (OPT-30B, 4 domain experts OPT-1.3B) on C4:

Model Merging. Thank you for your suggestion. We agree that comparing with model merging can provide new insights; the comparison is also interesting as model merging has lower inference costs compared to fusion approaches. We provide preliminary results by comparing PackLLM with average weight model merging on commonsense reasoning and language modeling tasks. The following tables show results that PackLLM outperforms model merging across these two tasks.

Setup: 2 LLMs (LLaMA2 and FuseLLM 7B models) on 4 Commonsense tasks.

Setup: 4 LLMs (4 domain experts OPT-1.3B) on C4:

We thank the reviewer for providing a thorough review and asking insightful questions.

Comparison with cBTM and DExperts & Question 1.

• cBTM controls the importance weights based on the distance from the input prompt to the pretraining data of each (expert) LLM. In order to produce meaningful weights cBTM assumes each expert was trained on a different domain and that the pretraining data for all experts is known. These necessary assumptions limit the applicability of this approach to arbitrary LLMs where the pretraining data is unknown (e.g. LLama) or where there is likely significant overlap in pretraining domain (e.g. Llama + Mistral).
• DExperts combines three LLMs (see Eq. 9): (i) a large model $M$, (ii) a fine-tuned (in-domain expert) $M_1$ and an untuned/anti-expert (out of domain) $M^-$. This approach does not extend to evaluating model fusion with a variable number of seed LLMs.

Nevertheless, we include comparisons with cBTM and DExperts where applicable e.g., in Figure 3 with the setup described in Section 4.1, and Medicine and Misc. in Table 1.

Comparison with learning-based fusion. As presented in Figure 1, the key motivation of our approach is to combine newer models without requiring retraining. We provide Table 2 to show that modular fusion methods (such as PackLLM) can leverage newly released LLMs (row “release date” in Table 2) to outperform existing learning-based fusion models (e.g. FoE), which, unlike PackLLM, uses additional (pre-)training data and requires retraining.

Question 2: Example Cases. Thank you for your suggestion. We will include the following two examples where fusing knowledge with PackLLM improves results in the final version:

• Case A

Input: "Which of the following contributes to vitamin B12 deficiency in older adults? A. Reduced secretion of intrinsic factor, B. Atrophic gastritis, C. Helicobacter pylori infection, D. All the above"

LLaMA2-13B: A. Reduced secretion of intrinsic factor (incorrect -> picks the longest answer)

LLaMA2-13B + PackLLM (MedLLaMA2-7B): D. All the above (correct -> identifies all correct answers)

• Case B

Input: "Contributing your $3,000 to a 401(k) or other "

OPT-6.7B: “retirement plan today will get you a return of 12-14%” (hallucination: incorrect return percentage)

OPT-6.7B + PackLLM (OPT-1.3B tuned model on C4): “retirement plan offers some tax advantages, but it’s important to take action as early as possible.” (helpful advice)

Thank you for your review. We address your suggestions below.

Time Complexity. We provide a time complexity analysis in Table 3, which shows that PackLLM adds negligible time overhead. We also provide the following time measurements for K=5 LLMs per input on a A100 GPU:

As evaluated, PackLLM does incur significant time costs.

Limitations. Thank you for your suggestion. We summarize two limitations of our approach below.

(i) One challenge is discussed in Section 3.4 (Tokenizer Selection & Alignment): Equation 1 assumes that the logit vectors $s_k$ can be added together, which requires that the seed LLMs share the same vocabulary. However, this is not always the case, as different LLMs are trained with different vocabularies. While we follow the MinED approach for tokenizer alignment, aligning different tokenizers is an open problem in NLP, and more effective alignment techniques may further improve PackLLM.

(ii) Furthermore, PackLLM incurs additional inference costs compared to model merging approaches, such as weight averaging. We will expand on this discussion and include it in a “Limitations” section in the final version.

Related Work. Thank you for your suggestion. Upon re-reading, we agree that the related work section is repetitive, we have fixed it in the final version.

Dear Reviewer Gnd4, this review does not correspond to our paper “Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization”.