Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration

Thanks for your insightful comments, which greatly help us improve our paper. We are glad to have this discussion to address your concerns.

 

Concern-1: The proposed method requires gradient updates to project back to the original space at every generation step. This can cause extra overhead.

Thanks for your insightful comment. We understand your concern considering the overhead of the reverse transformation process from the relative space to the absolute one. To address this concern, we quantitatively analyzed the overhead of this process in Appendix C.3.

Overall, DeePEn actually incurs $\leq$2% inference latency in practice. This conclusion also applies to the overhead.

 

Concern-2: As a result, the method may not be effective when the domain of pretrained LLMs are very different.

Thanks for your insightful comment. We understand your concern regarding the number of common words between LLMs with quite different vocabularies. To address this concern, we analyzed the number of common words across different LLMs in Appendix. A (Fig. 6). Through case studies, we found that a large number of common words (>20k) exist across the different vocabularies since LLMs (including LLMs tailored for specific domains) usually contain a lot of common English basic tokens. This fact provides a solid guarantee for our DeePEn to work effectively.

 

Question-1: other normalization methods other than softmax?

Thanks for your constructive suggestion. We follow your advice to experiment with an alternative normalization method of Softmax. Concretely, we replace softmax with Min-Max Scaling:

$x{\prime} = \frac{x - x_{min}}{x_{max} - x_{min}}$

The result shows that the normalization with softmax is better than the one with Min-Max Scaling:

We argue that Softmax pays more attention to the nearest neighbor words when modeling the structural relation (relative representation) than Min-Max Scaling, which could be more beneficial.

We will add this analysis to our paper.

Thank you for your insightful comments, which have greatly helped us improve our paper. We appreciate the opportunity to discuss and address your concerns.

 

Question-1: How well does DeePEn generalize to unseen data or across different types of tasks beyond the evaluated benchmarks?

We understand your concern regarding the generalizability of DeePEn, which relies on a development set for each task to find the optimal relative ensemble learning rate.

Actually, we analyzed the generalizability of DeePEn through cross-distribution validation and found that DeePEn generalizes well, which is shown in Tab.9 of the paper.

Concretely, we tested the performance of DeePEn on task $A$ with the hyperparameter found with the development set of task $B$. We totally consider 4 tasks: 2 generative knowledge QA tasks (NQ and TriviaQA), 2 multi-choice human examination tasks (MMLU and ARC-C).

The results show that the performance improvement using a different-task development set ($A \neq B$) is 88% of the improvement when using a same-task development set ($A = B$). Moreover, using a similar-task development set, the performance improvement is 95% of that achieved using a same-task development set.

Thanks for your important feedback. We will demonstrate this important analysis more clearly by modifying Tab.9 and placing it in the main document instead of the appendix.

 

Question-2: Finding optimal hyperparameters may require additional tuning.

We acknowledge that our current method requires tuning to find the optimal relative ensemble learning rate for effective inverse mapping from the relative space to the absolute space. As the first attempt to address this challenge, we have achieved significant success. In the future, we plan to explore methods to achieve this inverse mapping without the need for extensive tuning.

 

Question-3: What impact do different anchor word selection strategies have on the ensemble performance?

Thanks for your constructive suggestion. We had ever devised an anchor selection algorithm AS-MRRC (Anchor Selection with Maximum Relative Representation Consistency). AS-MRRC aims to infer the optimal anchor words via maximizing the relative representation similarity of common words across different models:

$A^* = \mathop{argmax}\limits_{A \in C} \ \mathop{\mathbb{E}}\limits_{i\in C} \ cos(\hat{R}_1(i|A), \hat{R}_2(i|A)),$

where $C$ is the common word set between different LLMs, $\hat{R}_1(i|A)$ and $\hat{R}_2(i|A)$ are the relative representations of word $i$ in different models using $A$ as anchor words, and $cos(,)$ refers to the cosine similarity function.

We compared AS-MRRC with the random selection of anchor words and the method of using all common words as anchors:

As the result shows, our AS-MRRC outperforms random selection using fewer anchors while underperforming the method of using the full set of common words. Therefore, we decided not to report this thankless method of AS-MRRC.

In the future version, we will follow your advice and include this trial to help readers better understand the impact of different anchor word selection strategies. We are also going to explore anchor word selection strategies more thoroughly.

We would like to thank you for your constructive feedback. We appreciate the opportunity to address your comments.

 

Question-1: Comparison between DeePEn with other ensemble methods in experiments beyond the main experiment.

Thanks for your insightful suggestion! We have followed your advice to supplement the results of the baseline ensemble methods (MinED, LLM-Blender, and Voting/MBR) in Fig.3 and Tab.2&3:

Figure-3 (Ensemble learning on various number of models):

Please note that due to time limitations, we have only supplemented the results of baselines on NQ. The results on the other benchmarks (MMLU and PIQA) will be included in the next version.

Table-2 (Ensemble learning between the dense and sparse models):

Table-3 (Ensemble learning between the generalist/specialist models):

As the results show, unlike DeePEn, the baseline ensemble methods perform quite unstably across various settings. Specifically, MinED results in dramatic performance drops when ensembling the generalist LLaMA2-13B and the specialist NLLB-600B due to their large vocabulary divergence. LLM-Blender also leads to significant performance drops on NQ, GSM8K, and Ro→En translation due to its limited generalizability.

We will add these results in our paper.

 

Question-2: Results of DeePEn-Adapt in Top-2 Ensembles?

Thanks for your important feedback! We have followed your advice to supplement the results of DeePEn-Adapt in the Top-2 ensembles of Tab.1:

We will add these results to our manuscript.

 

Question-3: Why was the DeePEn-Adapt not the default setting in the top-2 ensembles?

Our original intention of devising DeePEn-Adapt was to alleviate the interference of lower-ranked models to the higher-ranked models. Therefore, we only tested DeePEn-Adapt in the top-4 model ensemble, where the 4th model could cause serious interference.

Thanks for your valuable comment. In the future, we will set the DeePEn-Adapt as the default setting of DeePEn.

 

Question-4: What do the three colors (blue, yellow, purple) indicate in Fig.1?

Sorry for the unclarity. Different colors indicate different clusters of samples.

To clearly illustrate that relative representations remain consistent across different models, we performed K-means to cluster the sampled representations and set different colors for different clusters.

 

Suggestion-1: Some passages could be condensed.

Thanks for your valuable feedback. We will condense our manuscript in the future version.

In the original version of the paper, the general paragraph of section 3.2 primarily describes the implementation of constructing relative transformation, including the "Anchor Selection" and "Normalization of relative representation matrix".

In contrast, the specific paragraphs of "Anchor Selection" and "Normalization of relative representation matrix" explain the motivations behind our implementations.

 

Suggestion-2: Sections 3.3 and 3.5 both consider the aggregation of relative representations, currently separated by section 3.4.

Thanks for your valuable feedback. We will merge section 3.5 into 3.3.