Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling

We sincerely appreciate the time and effort you put into reviewing our paper. We appreciate the insightful feedback and address your concerns as follows:

R4W1: Possible limitations

R4A1: We acknowledge that UniTE still faces challenges in the scenarios discussed in Section 3.1, "Impact of Model Performance Discrepancy." As shown in the first column of Table 3, UniTE's performance (73.31) is slightly lower than that of the superior base model (73.46) when the performance gap between the two base models, Mistral and OpenChat, exceeds 10%. However, UniTE exhibits the smallest decrease in performance compared to its competitors.

R4W2 & R4W3: Captions and font size

R4A2: We apologize for any inconvenience. For Fig. 4, we update the caption to: "Impact of the hyperparameter k on the ARC and TriviaQA datasets. Increasing k beyond a certain point leads to a slight decline or no improvement in performance." For Fig. 6, we revise the caption to: "Comparison of different decoding methods. The greedy decoding strategy is more effective for eliciting the next token in deterministic tasks." The revised contexts are shown in red in our revised PDF.

R4Q1: Why increasing k beyond a range may lead to a slight decline or no improvement?

R4A3: Aligning with top-k and top-p sampling, in Fig. 3, we present the token distribution during the generation process, revealing that only a few tokens significantly contribute to the overall probability across the vocabulary. This observation motivates our proposed UniTE approach. Besides, we would like to clarify that the enhanced efficiency and effectiveness stem from the reduced token options and our specialized union mapping method, as outlined in Section 4.2. UniTE constructs a union of the top-k tokens from each model and expands this set using each model’s tokenizer. This is followed by probability aggregation to determine the next token. UniTE avoids the need for auxiliary mapping matrices and full vocabulary alignment, respecting the unique tokenization of each base LLM.

Here we offer a particular instance for clearer demonstration. We make the hypothesis that model 1 and model 2 are heterogeneous and employ different tokenization strategies. Specifically, if V1_[0-10)=[‘Jam’ (0.55), ‘James’ (0.2), ‘Jan’ (0.15) ...], V1_[10-15) = [‘J’ (0.04),....]; V2_[0-10)=[‘Jan’ (0.6), ‘Jam’ (0.21), ‘Ja’ (0.1),...], V2_[10-15) = [‘Janet’ (0.03),....], the result of top-10 union is Vu_10=[‘Jam’ (0.38), ‘Jan’ (0.375),...,]. Suppose ‘Jam’ is the expected token, then the top-10 union with greedy decoding elicits the correct answer. However, if ‘Janet’ does not exist in voucabualry1 then the tokenizer1 turns ‘Janet’ into ‘Jan’ and ‘et’, the top-15 tokens union changes into: Vu_15=[‘Jan’ (0.39), ‘Jam’ (0.38),...,] thus the next token is wrong ‘Jan’.

R4Q2: The impact of base model performance when using UniTE

R4A4: We would like to note that in Fig. 1 (Section 3.1), we also illustrate the impact of performance disparity among models using UniTE. UniTE demonstrates performance similar to that of other ensemble methods, as a substantial performance gap results in performance declines.

Besides, we would like to offer more statistics of UniTE related to experiments listed in Fig. 2. We identify model pairs based on the GSM8K dataset with performance gaps of approximately 40% (LLaMA2-7BChat and Mistral-7B-Instruct-v0.3), 25% (LLaMA2-13b-Chat and Mistral-7B-Instruct-v0.3), 15% (OpenChat-3.5 and Mistral-7B-Instruct-v0.3), and less than 10% (LLaMA3-8B-Instruct and Qwen2- 7B-Instruct). The results are shown below:

The table indicates that as the performance gap increases, the benefits of ensembling inferior models become more pronounced. When the performance difference is within 10%, ensembling can lead to improved results. Additionally, we emphasize that UniTE exhibits a smaller decrease in performance compared to the other methods shown in Fig. 2, further validating its effectiveness.

We sincerely thank the reviewer again for the helpful feedback!

We appreciate the valuable advice from the reviewer. And we would like to address your concerns as follows:

R3W1: Efficiency compared to the previous approach.

R3A1: Fig. 5 in “Section 5.4 Further Analysis” illustrates the latency of various methods. The results indicate that the latencies for the individual Mistral and OpenChat models are 73.05 ms/token and 75.09 ms/token, respectively, under our hardware device settings, while DeePen and GaC exhibit latencies of 225.13 ms/token and 122.73 ms/token. Notably, UniTE's latency is 87.78 ms/token, which is significantly lower than that of previous ensemble methods and only 16.8% higher than that of the single base model. Regarding memory consumption, as outlined in the introduction and Table 5, UniTE utilizes only 0.04% of the vocabulary tokens, with memory usage primarily related to model deployment—a necessity common to all ensemble learning approaches. In contrast, DeePen and GaC require additional memory to store their extensive intersection and union vocabularies.

R3W2 & R3Q1: Why does UniTE with Top-k ensembling achieve better performance?

R3A2: First, the top-k ensembling is motivated by the decoding strategies, like top-k sampling[1] and top-p sampling[2] in the field of neural text generation, which suggest that in each generation step, only a few tokens significantly contribute to the overall probability across the vocabulary. Therefore, we clarify that the enhanced efficiency and effectiveness arise from reduced token options and our specialized union mapping method, as outlined in Section 4.2. UniTE constructs a union of the top-k tokens from each model and expands this set using each model’s tokenizer, followed by probability aggregation to determine the next token. UniTE only focuses on the important tokens and eliminates the noise from other irrelevant tokens, and also respects the unique tokenization of each base LLM.

We also highlight the distinctions between UniTE and the methods of DeePen [3] and GaC [4], which utilize the entire vocabulary for alignment. DeePen selects the intersection of base models as anchor words and employs embeddings to map other vocabulary items into a common space, relying on these anchor tokens instead of the full vocabulary. To ensure effective representation, DeePen includes all tokens from the intersection, as a larger subset is advantageous. GaC projects the probability vectors of multiple LLMs into a unified vocabulary dimension using a mapping matrix, aggregating outputs at each generation step to select the next token. In contrast, UniTE avoids the complexities of embedding mappings and adopts a novel approach to top-k union construction.

We further test k with extremely large values to mimic aligning on the whole vocabulary with our method. As shown in Table below, further increasing k leads to either a slight decline or no change in performance. This finding reinforces our assertion that, in probability-level ensembling, it is unnecessary to align the entire vocabulary to predict the next token.

Here we offer a particular instance for clearer demonstration. We make the hypothesis that model 1 and model 2 are heterogeneous and employ different tokenization strategies. Specifically, if V1_[0-10)=[‘Jam’ (0.55), ‘James’ (0.2), ‘Jan’ (0.15) ...], V1_[10-15) = [‘J’ (0.04),....]; V2_[0-10)=[‘Jan’ (0.6), ‘Jam’ (0.21), ‘Ja’ (0.1),...], V2_[10-15) = [‘Janet’ (0.03),....], the result of top-10 union is Vu_10=[‘Jam’ (0.38), ‘Jan’ (0.375),...,]. Suppose ‘Jam’ is the expected token, then the top-10 union with greedy decoding elicits the correct answer. However, if ‘Janet’ does not exist in voucabualry1 then the tokenizer1 turns ‘Janet’ into ‘Jan’ and ‘et’, the top-15 tokens union changes into: Vu_15=[‘Jan’ (0.39), ‘Jam’ (0.38),...,] thus the next token is wrong ‘Jan’.

Moreover, to alleviate your concern about our experimental results, we have uploaded our code to supplementary materials.

We sincerely thank the reviewer again for the helpful feedback!

References:

[1] Hierarchical Neural Story Generation. ACL 2018.

[2] The Curious Case of Neural Text Degeneration. ICLR 2020.

[3] Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration. NeurIPS 2024.

[4] Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling. EMNLP 2024

Dear Reviewer QXTA,

Thank you sincerely for taking the time to review our submission and for your insightful comments. We have carefully considered your concerns and have addressed them in our recent responses. Your constructive feedback has been instrumental in helping us improve the quality of our work.

As the deadline for the discussion period approaches, we would greatly appreciate it if you could review our responses and let us know whether they adequately address your concerns. If any issues remain unresolved, please share your remaining concerns so that we can respond appropriately before the deadline. We welcome any further discussions you may wish to have.

If you find that our responses satisfactorily address your concerns, we would be grateful if you could consider increasing the rating score.

We understand how busy you are, and we truly appreciate your time and effort. We look forward to your further comments and discussions.

Best wishes,

Authors

We greatly appreciate the reviewer’s helpful feedback. Below, we address your concerns:

R2W1: Whether Conclusion II works in another language.

R2A1: We also conducted experiments on the CMMLU [1], a Chinese Multitask Language Understanding Evaluation benchmark to validate our findings. We used Yi-6B (vocabulary size: 64,000) and Qwen2-7b-instruct (vocabulary size: 152,064) as our base models, both supporting the Chinese language. Qwen2-7b-instruct is the primary model, the results of various ensemble approaches are presented below:

Similar to the results listed in Section 3.2, irrespective of the gap in vocabulary size, existing methods still demonstrate improvements, thereby indicating that vocabulary size for model ensembling is marginal. We have included this experiment in our revised PDF version.

R2W2: Solution to different response styles.

R2A2: We would like to adopt preprocessing steps to cope with different response styles. In our preliminary experiments, we tested several datasets and observed significant differences in response styles for TriviaQA when using Qwen compared to LLaMA series models as shown in Table 2. For instance, following the original 5-shot prompt settings of previous work[4][5], Qwen2.5 analyzes and includes conclusions in its responses, complicating the extraction of solution-oriented knowledge for QA tasks, while LLaMA3 provides the solution directly. To force responses from different models in a similar style and avoid Qwen2.5 responding with verbose analysis, we employed a new 5-shot prompt designed to elicit answers in the format "The answer is xxx.". The responses are presented in the table below.

Then we test different ensemble approaches:

The original prompts elicit accuracy in the brackets (As the tedious response style illustrated in Table 2, Qwen incorporates answers into the analysis, we randomly sampled 100 instances from the 1500-test set to manually extract the predictions). After adjusting the prompt, we can easily analyze the results for the entire test set. Consistent with our findings presented in the main text, when the base model’s performance gap exceeds 10%, ensemble learning may yield little to no improvement. Additionally, it is important to note that our UniTE approach still outperforms its competitors.

We sincerely thank the reviewer for the constructive suggestions, which help us to improve the quality of our work, and are pleased that you find our work to be novel and effective. We would like to address your concerns as follows:

R1W1: Why does UniTE achieve greater improvements on the MMLU dataset?

R1A1: MMLU is a benchmark comprising many subject-specific tasks (e.g. physics, biology, chemistry). We randomly selected five subjects from this collection and conducted experiments using LLaMA3-8B-Instruct and Qwen2-7B-Instruct, with the results presented in the table below. Notably, although the performance of LLaMA3 and Qwen2 is comparable across the entire MMLU dataset, significant differences emerge in their performance on individual subsets. This suggests that LLaMA3 and Qwen2 exhibit substantial differences in their capabilities across various subjects, making them complementary to each other on the MMLU benchmark.

Assuming we choose LLaMA3 as the base model, when LLaMA3's performance on a subset is inferior to that of Qwen2, the ensembling is likely to yield a more substantial enhancement. Therefore, the application of UniTE results in a more pronounced improvement across the entire MMLU dataset.

R1W2: Scalability of our methods regarding performance gap limitation.

R1A2: As our first key takeaway says that “smaller performance gaps facilitate greater gains from model ensembling”, we recommend to ensemble the comparative models on the target tasks to obtain better performance than applying the single base model. We would like to clarify that the performance gap within 10% is not the hard limitation of our method. UniTE can definitely be applied to all candidate model pairs, and can consistently perform effectively, regardless of the performance gap, if the intention is merely to enhance the inferior model using the superior one. However, if users aim for performance that even surpasses the superior model, this factor should be considered. We believe the performance alignment for optimal model ensembling is both intuitive and reasonable since a significantly weaker model is unlikely to contribute valuable information to a stronger model. Our extensive experiments also confirm this view.

As for how to automatically deal with a large pool of candidate models, we detailed the base model selection pipeline in Section 4.1. Firstly, we choose the best-performing model for the target task. Subsequently, select the next best-performing model that satisfies the criteria (performance alignment and response style consistency) for successful ensembling with the first chosen model, continuing this process iteratively until the maximum number of base models is reached or no further suitable models can be found. We also provide an alternative way to alleviate the examination of response style consistency in the later part. So, the entire selection process can be automated when the performance of candidate models on the target task is accessible. In cases where these performance scores are not explicitly available, we recommend to sample a calibration dataset from the target task and obtain a reference score for each model, which can then be used to assess compatibility.

R1W3: Task choice menus

R1A3： To validate the performance of UniTE and ensure a fair comparison with previous works [1][2], we have already evaluated three main categories: 1) Comprehensive examination (MMLU, ARC-C), 2) Reasoning capabilities (GSM8K, PIQA), and 3) Knowledge capacities (TriviaQA, NQ).

We also conduct additional experiments to address the reviewer's concerns using the BBH (BIG-Bench Hard) benchmark [3], a diverse evaluation suite of 23 challenging tasks such as symbolic reasoning. Due to computational constraints and limited rebuttal time, we randomly evaluated 10 subsets, and the results are presented below. Consistent with our main findings, UniTE demonstrates superior performance compared to other methods, highlighting the effectiveness and generalizability of our approach. We include these results in our revised PDF version in red.

Dear Reviewer nt9F,

We sincerely appreciate your time to review our submission and provide valuable comments. We have carefully considered your concerns and tried to resolve them in our rebuttal. Your constructive feedback will greatly help us improve the quality of the work.

As the deadline of discussion period is apporaching, we would really appreciate it if you have read our response and let us konw whether the previous responses have addressed your concerns accordingly. If your concerns have not been well resolved, could you please let us know your remaining concerns so that we have the opportunity to respond before the deadline? We are happy to have any follow-up discussions. If you are satisfied with our response and it truly addresses your concerns, we would really appeciate it if you could consider to increase the rating score.

We understand you are very busy and we really appreciate your time. Looking forward to your further comments and discussions.

Best wishes,

Authors